Protecting Your Site From Bad Bots

By Graeme Caldwell — Graeme works as an inbound marketer for Nexcess

Bad bots are the band of many WordPress site owner’s. We discuss what bad bots are, and how WordPress site owners can protect their sites.

Some people are surprised to learn that about half of the traffic on the web isn’t generated by human users. It’s generated by bots: software that trawls the web downloading and interacting with web pages much as humans do. Approximately a third of non-human web use is generated by “good bots”, bots that stick to the rules and perform useful services — search engine web crawlers like Googlebot fall into this category. The rest are bad bots, malicious software created and deployed by individuals and companies for nefarious purposes.

Bad bots can be a serious problem for WordPress site owners. They consume resources — sometimes almost all of a site’s resources — and, if they manage to compromise a site, they can turn a productive part of the web into a source of malware and denial of service attacks.

There are many different types of bad bot, but let’s have a look at few of the main categories.

Scrapers are bots that crawl the web collecting information from pages. There are perfectly good reasons to do this sometimes, but bad scrapers are used by content thieves to steal the intellectual property of site owners.

Most WordPress site owners are familiar with spambots. They trawl the web looking for opportunities to post comments and other spam on websites, often for SEO purposes or to attract site visitors to malware sites. A related type of bot scours web pages for email addresses that can be added to spam lists.

Brute force bots are on the lookout for WordPress and other sites with weak login credentials. When they find a site, they try lots of different username-password combinations until they discover one that gives them access. Brute force bots are a particular problem because responding to and rejecting thousands of spurious login requests can consume a huge chunk of a WordPress site’s resources.

There are also more sophisticated hacker bots that know about the vulnerabilities of specific versions of WordPress, and are capable of carrying out complex attacks to take over a site.

Bots are a problem for every WordPress site owner, but there are plenty of solutions to the problem of bad bots.

Blackhole For Bad Bots

Blackhole for Bad Bots uses the robots.txt file in an interesting way. Robots.txt is a file site owners can use to give instructions to bots. Good bots, like Google’s crawler, abide by the content of robots.txt. Bad bots ignore it.

Blackhole For Bad Bots inserts a link in WordPress web pages that is invisible to ordinary users, but visible to bots. It also adds an instruction to the site’s robots.txt file to tell bots they shouldn’t follow that link. Good bots won’t follow it, and users can’t see it, so if any visitor to the site does follow the link, the plugin knows it must be a bad bot and can block it from further access.

Wordfence Security

Wordfence Security is a comprehensive security plugin that has many features, some of which are intended to make life difficult for bad bots. If the plugin knows about a malicious bot, it can block it outright. It also uses a number of heuristics to detect likely bot attacks and blocks those too.

If you find your site’s resources are being consumed by non-human users, either of these plugins is a powerful ally in the fight against bad bots.
About Graeme Caldwell — Graeme works as an inbound marketer for Nexcess, a leading provider of Magento and WordPress hosting. Follow Nexcess on Twitter at @nexcess, Like them on Facebook and check out their tech/hosting blog,http://blog.nexcess.net/.