WordPress Security and SEO Wars

Theory: Site performance issues can relate to Bad Bot traffic. Here we describe one WordPress Security plugin to solve this scenario.|Theory: Site performance issues can relate to Bad Bot traffic. Here we describe one WordPress Security plugin to solve this scenario.
Andrew Lundquist
Andrew Lundquist
July 26, 2013
February 15, 2024
Example Bad Bot Crawling from Ad|WordFence Firewall Settings

Ad Siphoning, Bad Bots, and Crawler Traffic Induced Performance Issues?

We recently started noticing some abnormal sluggishness on our public website and even on interior plugin store pages as well as the WordPress Dashboard. Thinking it might be related to standard Multisite plugin bloat (i.e. our busy-induced laziness) we went through a relatively thorough house cleaning process and threw some performance tests at it too. That seemed to help, but it wasn't conclusive. Getting serious about it, we fired up some monitoring with AlertFox and Yottaa (both fantastic!) and quickly observed interesting, non-patterns, of peaks and valleys in page load times as well as some sporadic micro-outages. What began as a performance troubleshooting issue quickly blended into the WordPress Security arena.

Another clue came in from some other sites we manage on a different, single hosting account that have been experiencing massive performance issues recently as well. Logs revealed that "googlebot" was pounding the sites in that account at an alarming frequency. Well, we certainly never asked Googlebot to work against its own nature which already handles crawl throttling fairly smartly, so we began to wonder if there was something else going on and started suspecting there might be substantial naughty bot traffic hammering away.

Enter Google Adwords and China

Ok, before I go further with my theories please feel free to call me crazy and explain why in the comments (naturally supported by data). We're still gathering data to validate what is going on in our scenario, so some of this might be circumstantial observation fueled by other anecdotal historical scenarios.

The facts so far:

WordPress Security Measures Against Ad Siphoning and Bot Resource Consumption

Theory: Our site's performance issues are highly related to substantial Bad Bot traffic consuming resources. We're testing a solution that will validate (or disprove) this and I will describe that more in a moment. Has the moment arrived when super tight security is just part of the game when running a business website? And I'm not referring to security against viruses or intrusions or site compromises due to hacking and cracked passwords. I'm talking about security to protect bandwidth, resource utilization, and user experience in this age of bot swarms and legions of automated crawlers and scrapers. Those are limited resources, and you could have the most highly optimized site, but those inhuman code spiders are almost never programmed to be considerate when those resources are especially scarce (like in shared hosting environments) you can't afford to lose any of them to virtual visitors that produce zero value for you and your business. And the thing is, you likely have no idea how much of that is constantly hitting your site.

More disturbing, however, are some interesting implications for this crazy search driven era. What happens when these bad bots follow your Google Ads? They're never going to convert, but they are happy to burn your paid clicks. Maybe Google has precautions against this? But from our logs it looks like bots can burn clicks just like humans. What if lots of bots burn lots of the clicks you paid for. I'm asking the question - and maybe someone has a definitive answer as to why my assumptions are incorrect. But this seems like a super wasteful possible outcome. In the meantime, Google has a never ending supply of automated free money.

Safety in the Power of Innovative Ecosystems like WordPress

So how do Innovative Ecosystems like the WordPress Community fight back? One of our measures to test and defend against bad bot traffic is the amazing plugin called WordFence Security. The existence of high caliber plugins like this for free blows my mind. We set this up primarily to use its firewall features and see if blocking / throttling our traffic can improve our site performance, but it is loaded with a ton of other excellent security features too. The log captured in this post's featured image is from the WordFence live monitoring capability, and shows what appears to be a crawler siphoning off one of our Google Ads and hopping in to crawl the site. You have to read it from bottom up in chronology since the live monitoring keeps the most recent activity at the top. Fire that plugin up and just watch the traffic in the Crawlers tab. You will be floored by the non-human activity on your site.

On top of this, there are some amazing innovations in the WP.org plugin library that were born in response to internet-wide threats to WordPress security (like the widespread botnet attacks on standard admin accounts). It's amazing to be a part of the self-defending and evolving Ecosystem of Innovation surrounding WordPress.

yellow hexagon