Bad bots are everywhere. We’re taking measures to protect servers

Most internet traffic doesn’t come from humans anymore. At this point it’s mostly bots.

On one hand, some of those bots are more useful than ever, but on the other hand the majority of these bots are doing more harm than good.

For anyone running a website, it means there’s a lot of bots out there hogging your website’s resources like bandwidth. You want to keep the useful ones and get rid of the useless ones. That’s what our latest bot protection work is trying to do.

Not all bots are bad, but a lot of them are

Lots of important internet services rely on bots.

For example, Google uses bots to crawl websites and gather content for search results. You won’t find many people opposed to that kind of bot. AI tools use bots to scrape information, monitoring services use bots to check if websites are working as intended, data tracking tools rely on bots checking for updated stats—basically anything that’s automated on the internet is being driven by a bot.

Generally, bots like these follow the directions given in a website’s robots.txt files, take what they need from a webpage, and then leave.

On the other hand, there’s a category of bot where the opposite is true: bots that are up to no good. The ones that we like to call “bad bots”.

They’re bad because they have little regard for how they interact with websites. Whether programmed by a human to act maliciously, or just designed poorly, these bots create problems.

Some bad bots follow every link they can find on a website, endlessly clicking and looping between pages. Others try to brute force URLs just to see what they can find. And usually they’re not doing anything useful—on the contrary, they can be causing a lot of harm. These bot “clicks” are milliseconds apart, eating up all your bandwidth, soaking up CPU, and crushing response times for all your real visitors.

It’s not a little problem

If the internet didn’t have any protection against bad bots, then bots would be the only ones receiving your fastest page loads, and real people visiting your site would get an inferior experience.

Over the years we’ve added lots of different measures to filter out bad bots. A new step we’ve started taking more recently is more drastic, but a lot more effective.

The goal is to ruthlessly block bad bot traffic and be completely invisible to everyone else.

So far our initial testing of our new toolset has been extremely positive, with one server banning over 1,700,000 bad requests in under 24 hours. For some servers we're seeing about 30% of all traffic being banned. During a spike, some of our servers peaked at a 66.3% ban rate. With more bad traffic kept at bay, your sites will spend less time and fewer resources serving unwanted bots.

Steps we’re taking

It’s a sophisticated system, but surprisingly easy to explain how it works.

It hinges on the idea that bots behave differently than humans. Bots do things like try to guess URLs on a site rather than following links. They jump between pages in milliseconds. They’re scripted and therefore behave methodically. If they’re not programmed to respect the rules of a site, or they’re explicitly programmed to abuse a site—they’re a bad bot.

But even if it’s easy to tell a bot from a human it’s still quite difficult to judge between good and bad bots. So we’re employing a “leaky bucket” approach to judge traffic on their behaviour.

Behind the scenes we've been running a mechanism that classes every visitor interaction as “good” (like humans, and well-behaved bots) or bad (poorly-designed, or straight-up evil bots). Every bad action (like quickly jumping to random URLs) fills the bucket with a little bit of water. A typical human visitor may unknowingly trip that alarm once or twice when checking out a site. Because it’s a “leaky bucket”, it will drain over time and the humans and good bots sail through without noticing anything was going on.

If the internet didn’t have any protection against bad bots, then bots would be the only ones receiving your fastest page loads, and real people visiting your site would get an inferior experience.

On the other hand, a bad bot will trip the alarm dozens of times in a few seconds, filling the bucket until it overflows. Visitors that overflow the bucket get banned, everyone else just gets to visit the site.

The word “ban” sounds dramatic. What we’re actually doing is putting an automated CAPTCHA challenge in front of visitors coming from banned IP addresses. This isn’t the type of CAPTCHA where you have to type those weird squiggly letters or try and spot bicycles on a busy street. Instead, it’s an automated CAPTCHA that the visitor’s browser should solve on its own without requiring you to click anything.

Not just banned on one server

When we’ve identified these suspected bad bot IPs, they don’t just get banned on one server: the bad bots IPs get banned across all of our protected servers.

The moment a bad bot is spotted causing trouble on one customer’s site, we ban that IP across the whole fleet of protected servers for a few hours, and even longer for repeat offenders.

Once we’ve done that, we share that complete list of banned IPs with a worldwide community network of other reliable providers, who also share their high-confidence blacklist with us. All these providers lists are combined into one massive list of IP addresses that we ban from all our protected servers.

A dynamic list

Combining our own ban list with the community list gives us an average of ~140,000 IPs banned at any given time. It’s a lot of IPs.

But bots can change IPs incredibly quickly. Making matters worse, cloud providers like Azure reassign IPs to other users once a bot is done with them. Banning IPs lists can lead to banning innocent human traffic, so it isn’t a permanent solution.

What we’re doing instead is using a ban list that’s living, evolving, and constantly changing. Addresses are removed from the ban list when the abuse stops. Fresh ones are added the moment an attack is spotted.

Keeping the good bots in

Because bots aren’t all bad, we’ve got to be cautious about who we’re banning. It’s not as simple as letting humans through, and banning everything else. Sometimes even good bots can accidentally fail the leaky bucket test and get flagged as bad.

If we accidentally blocked Google’s crawlers, it’d be an SEO disaster. Banning your own office would be even worse.

So just as carefully as we assemble our ban list, we also maintain carefully assembled “allow” lists.

Obvious ones like search engine bots like Google and Bing are exempted. Monitoring and uptime status tool, and of course our customer’s own list of trusted IP addresses all get waved straight through.

The goal is to ruthlessly block bad traffic and be completely invisible to everyone else.

What banned visitors see, if they have human eyes

The new system is doing a lot of work to block the bad bot traffic, but sometimes it’ll get things wrong and real users will get temporarily banned. We’re working hard to minimise this.

But there are some cases where it will happen, so we want you to be aware of what banned users (or even you or your customers) will be shown in that scenario. Remember that CAPTCHA challenge we mentioned earlier? Here’s where it comes into play.

• A page with a simple CAPTCHA challenge that tries to verify the authenticity of traffic. The user’s browser should automatically solve the challenge in a couple of seconds, but it’s designed to be computationally expensive for a bot to solve. When the CAPTCHA is completed, the user is taken to their requested website.
• If for some reason the CAPTCHA challenge is failed, or not available, the visitor will be taken to a page which explains what happened, gives links to more information, and provides some steps to take.

If you're accidentally being banned, or want to know more about our process, check out the MyHost Knowledgebase for more information.

The battle is never over

We’re planning on rolling this protection layer out across more services and servers in the coming weeks and months. We wanted to keep you in the loop for now, in case you or your customers see this CAPTCHA challenge.

This new toolset isn’t a silver bullet. But it’s also a lot more than just a band-aid. It’s another important layer in our evergrowing defensive effort that we’re continuously building and improving.

Photo by Nick Fewings on Unsplash.