Don’t think that referer spam is a problem? What about when it goes crazy?

The Big Swede has has a lot of fun over the last couple of days as thousands of zombie machines based mostly in Japan have hit various pages on his server at the rate of about 4 times a second for the last 3 days. This has added up to 10GB (giga-freakin-bytes) of useless traffic caused by these spamming bastards. Right now I’ve pulled the ethernet cable and I’m watching the blinkin lights go insane on my DSL router. I’m hoping that an hour down will get them to stop, but it’s not looking good.

Most of these hits are dumping referers, but some are just malformed get requests that make no sense at all. Strangely enough almost all of them are from windows machines. Thanks Bill for that great operating system. Looks like we may have to hand this one off to my ISP and see what they can do to throttle or kill off the attacks upstream so we’re not getting so hammered on my modest DSL line.

Referer spam is a real problem, and it’s only going to get bigger.

UPDATE:

The attack has been sustained for 4 days now and it only getting bigger as the spammer bends new zombie machines to his will. We’ve seen over 12,000 unique IPs involved in the attacks, with about 1,000 of those doing the heavy lifting by delivering tens of thousands of hits over the past couple of days. We’ve ended up blocking a couple A class IP blocks as so many machines from the 218 and 219 machines have been compromised. We’re working on a solution that will collect unique IPs and add them to firewall rules for blocking, but in the mean time some legitimate traffic will be dropped. Sorry about that, but we need to staunch the bleeding.

I haven’t wanted to limit the number of apache processes running because I don’t want people coming from known good IPs to not see the site because of these jerks, but I may have to as apache is consuming all available memory on the machine as a result of the attacks.

Almost all of the requests are in the form of a GET or HEAD with a fully qualified domain as the request URI. I’m setting up an .htaccess that will kill any FQDN in {THE_REQUEST} that isn’t our own, but it doesn’t seem to be working. When I get off work I need to test mod_rewrite to ensure it’s working. While this will 403 most of the bad traffic from these spammers, it will still require Apache chomping memory and a little processing overhead.

What we’re ultimately working on is a script that will parse the access_log both for these FQDN requests and for frequency of hits and pass bad IPs off to IPFW to block the traffic at the firewall. Kind of an abstraction of this. His setup progressively punishes hosts who are falling into a honey pot. I don’t care about unblocking those IPs in an automatic way. As far as I’m concerned, I don’t care if the zombie IPs are blocked forever.

We’ll post any updates to the drama as they unfold