Mad in America has been under a low-grade attack by hackers for several weeks.
I first noticed an odd traffic pattern in our Google Analytics account, indicating that the front page was receiving more than ten views for each unique visitor. This means that some minority of accounts was reloading our front page over and over again. This is called a botnet attack, where a hacker uses a set of zombie computers, often compromised by malware, to perform repetitive tasks.
When this came to my attention I began setting up extra security software on the site. Alas, my first attempt ended somewhat disastrously in a totally crashed Mad In America that took almost an hour to recover.
Shortly thereafter, we began experiencing a major spike in traffic due to the viral sharing of Bruce Levine‘s Why Anti-Authoritarian’s are Diagnosed as Mentally Ill article. This was last Friday, May 3rd. Curiously, our referral traffic indicates that the hacker community was a major source of this legitimate wave of new readership!
As many of you noticed, the site became virtually unusable for logged in users. Our traffic spiked considerably on Wednesday. Over the past week we’ve served more unique readers than we usually do in a month — and we are still experiencing exceptionally high traffic on Bruce’s article, which now has a very broad base of fans including almost a thousand readers from Portugal!
With the site crippled, my first priority was to keep it operational and to appease our VPS web host, who was threatening to shut us down because the load was impacting other users on our physical server. This crisis prompted me to apply a number of important optimizations to our database and web server software. I also had to turn off several plugin features of the site in order or minimize the amount of code that was being run with each view.
I am very grateful to the Cloudflare service for its “I’m under attack” security setting, which allowed us to block almost all the illegitimate traffic instantly, and return the site to a functional state where the flood of new readers and returning users could all make use of Mad in America. While this setting was active, you noticed a five-second wait time before being able to access the site, during which Cloudflare expertly determined your humanness. Sometimes you may have had to answer a CAPTCHA challenge as well.
While effective, this inconvenience of the 5-second interstitial page is not a sustainable solution. I’ve never administered a site this size before, so I was feeling eager, and also out of my depth. I spent many long hours sifting through server logs and process lists, familiarizing myself with our various firewalls, trying to identify where this bad traffic was coming from and how to most effectively stop it, to little avail.
Last night I was finally able to setup security software compatible with our site, called Wordfence, and I was astonished. It’s one thing to understand an attack like this in theory, and another thing to watch it playing out before my eyes.
I knew by this point that turning off Cloudflare’s extra security would result in our server being flooded to the point of near-failure within minutes. But with Wordfence I am able to set rules that throttle the most problematic traffic while I used a beautiful real-time access log to manually identify and block the zombie bots once and for all.
This took a long time. It seemed like each time I blocked one, another one popped up in its place, like a game of whack-a-mole. At a certain point I was beginning to lose heart. Maybe this attacker had an army of thousands of zombies, and would just throw more at us for each one I blocked. Perhaps I would have to rely on strict automated rules that would still cause some strain on the server and inconvenience readers. Just then, after around two hundred blocked IPs, they stopped coming. I defeated the zombies.
Why are we Under Attack?
There is no easy way to determine the source of a botnet attack. One might suspect that this attack was personal in nature — a professional undertaking by somebody who feels threatened by our community.
On the other hand, these sorts of hacking attempts against WordPress sites are extraordinarily common. The larger we grow, the more likely we are to experience them. Hackers go after WordPress sites because, while quite secure, they have certain predictable vulnerabilities if not setup properly. These hackers hope to leave their mark, steal useful data, or simply gain experience in the field.
False Positives
In my zealous crusade last night, some legitimate readers were swept up the banning of IP addresses.
The reason for this is that the criteria I chose to ban was based on some odd behavior I thought to be exclusive to the zombies, but wasn’t. They would always try to load certain plugin files that have long since been deleted from the site. In fact, this is why Google kept showing them hitting our front page. Our site had a legacy script from our launch last year that was set to redirect all 404 – Page Not Found errors to our front page. This was meant to help people who had bookmarked or were following links directed to the old madinamerica.com blog, which has a different file structure than the current site.
So the bots’ attempt to load old files and poke around for vulnerabilities resulted in epic traffic redirected to our front page. Ironically, our front page is the most resource intensive page on our site to load, so in a way we were complicit in our own attack!
As it turns out, some of your browsers are also loading these old, non-existant files. I’m honestly not sure why yet. It may have something to do with your local cache. Last night when I noticed anybody loading the files I deleted months ago from the site, I just went ahead and banned them. I even banned the IP address Robert Whitaker was accessing the site from!
So, please, do not be too alarmed if your IP address was banned. Just shoot me a message at: [email protected] and I will remove the block right away.
Thanks for your patience, and for your ongoing support. Happy reading!