Our site got hit by a Denial-of-Service attack

From November 9 to November 19 two of our domains were unter attack by cyber-criminals. Due to a Distributed Denial-of-Service attack (DDoS) involving thousands of remote controlled zombie computers directed from a secret control centre, some of our sites were inaccessible for several days.

First we received an automated warning email from our webhost, which gets triggered if a certain amount of traffic per hour is exceeeded. I started blocking IP addresses of hosts with an excessive number of connections using iptables in Linux, but could not keep up: The server became unreachable. I was left with no choice but to pull the emergency brake, i.e. to replace the IP address of the server with a non-routable IP address such as 127.0.0.1 (loopback address).

I then moved the affected website to a backup server and reenabled it there. The new server was running a later Linux kernel than the old one. If you get DOSed, make sure you have Linux kernel 2.6, which is more suitable for reconfiguration to make it more resilient against such attacks.

After a number of days, other hosts names on our server that had not been disabled were also added to the list of attack targets.

As a result of the tweaks on the new server the sites stayed up most of the time, but the bandwidth usage was tremendous. During one hour the attacking bots generated more than 31 GB of traffic. On that peek day the traffic on that server came to 152 GB, even though we added over 4000 different IP addresses of attacking hosts to the blocklist.

Clearly, anyone who doesn’t have an unlimited traffic allowance for his hosting account would be in trouble with such huge numbers, even if the machine and operating system were able to keep up. Once they exhaust their monthly allowance they would either have to start paying for extra Gigabytes or the server gets disconnected, or the network speed gets throttled down, which would make the site virtually unreachable.

After 10 days the attacks started winding down. By that time we knew where the control center of the botnet was located. It was hosted by a company called AbdAllah Internet Hizmetleri in Turkey. Its upstream provider is TurkTelekom. The IP address range used by the hoster is listed by anti-spam site SpamHaus.org as being used for “Ukrainian/Russian cybercriminal hosting”.

During or shortly after the attacks against our servers, the same botnet also attacked the following sites:

  • newgeneration.lv
  • streamingvideosoftware.info
  • www.kety.org
  • anriintern.com
  • datingsoftware.org

This target list ranges from an anti-spam website (ours) over an evangelical church site to sites related to adult videos.

Distributed denial of service attacks are a mortal danger for any website. There are few effective countremeasures, except load sharing with many fast servers connected via fat data pipes, but even that is no match for some of the largest botnets such Storm. Attacks are used to intimidate, to silence or to extort “protection money”. Victims have little hope of getting effective help from law enforcement.

What needs to happen? First of all, the number of infected computers needs to decrease. Unsecured broadband hosts that come under criminal control are a public menace. Webhosts need to take effective action against botnet control centres. Unlike the actual bots, which are mostly running Windows XP, most of the botnet control centres run on Linux servers in data centres. Hosters must not turn a blind eye to this. If they do that because of money from criminals then their upstream providers must disconnect them.

Anti-Fraud sites under attack

Several of the main sites dedicated to fighting online scams are currently inaccessible because of a “Denial of service” (DoS) attack.

Fraudwatchers.com, aa419.org, 419eater.com and occassionally thescambaiter.com have been offline. Thescambaiter.com and 419eater.com are two of the oldest sites that fight “419” scams (named of the section in the Nigerian penal code that prohibits fraud). Fraudwatchers.org and aa419.org deal with a wider range of online scams, Nigerian scams as well as Escrow and commercial scams often run by Eastern European crime rings.

It is still unclear who is behind the attack. The selection of websites for this concurrent attack suggests Nigerian scammers, but technically the type of attack is more typical for Eastern European scammers. It may well be a sign of increased cooperation between both crime communities.

The exposure of websites to the danger of cyber attacks in a more and more net-centric world was highlighted earlier in the year when websites in Estonia were crippled for several days in large scale attacks, many of which originated from next-door Russia, with which Estonia has had a strained political relationsship.

Throughout this year criminals have been building the Storm botnet, a network of remote-controlled zombie computers infected with Malware that lets the criminal masters download and run any software on them that they chose. So far the Storm botnet has been used primarily fo sending pump-and-dump penny stock spams (see here). However, experts estimate the network as being comprised of between 1 and 10 million computers, far larger than needed to spam every computer on the planet. It’s probably the only peer-to-peer network comparable in size to eBay’s voice-over-IP giant Skype, which currently has 4 to 7 million concurrent online users.

Botnets have the potential to cripple the information infrastructure that countries increasingly rely on. Greater efforts need to be made to prevent infections, clean up or quarantaine infected computers and to track down the criminals who control them.

Denial-of-service attacks hit anti-spam sites

If you’re a regular vistor to some of our websites you may have noticed that the server was down for much of the last 48 hours. This was due to an online attack known as a ‘distributed denial of service attack’ (DDoS). In the first two hours alone computers with over 1000 different IP addresses worldwide were involved. At the peak 3.6 GB of requests per hour (i.e. 1 MB per second) were sent to the server, which was unable to keep up with the load and became unresponsive.

We took several countermeasures and managed to bring some websites online again. As of today it appears the attacks have ceased.

Concurrent with this attack on our main server several other anti-spam servers underwent similar attacks. The website of URIBL.com was offline for some time. Several servers that are of the SURBL project were affected by attacks.

The large number of IPs involved suggests that the attack involved a botnet, a large number of remote controlled zombie computers infected with malware. This criminal abuse of stolen internet resources illustrates the dangers that infected computers pose to others, against which there are few effective defenses.

It also shows that anti-spam tools such as SURBL and URIBL are effective against the spammers, or they wouldn’t be trying so hard to sabotage our legitimate efforts.

(Update 2007-06-12): SpamHaus was also affected by the attack, according to an article by Ryan Naraine (ZDNet), which quotes a usenet posting by Steve Linford of SpamHaus. According to this information the DDoS was carried out using a variant of the ”Storm” malware by the same gang that also launched a DDoS attack against BlueSecurity last year.