A few days ago I received email alerts notifying me of abnormally high web server traffic. Naturally my first thought was that this might be a Denial of Service attack (DoS): My sites have been attacked before, including by botnets consisting of several thousand rogue computers. This time, during a one hour period, traffic exceeded the same period one week earlier by 800 MB, which works out as 20 GB per day if sustained.
A search of the server logs showed numerous requests like the following, requesting up to 50-60 documents per second:
18.104.22.168 – – [09/Jan/2009:19:20:39 +0000]
“GET /emails/2008-06/30/00209262.117.htm HTTP/1.1” 200 12928 “-”
“Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)”
All originated from the same IP address in Russia (22.214.171.124) owned by Gigabase Ltd in Moscow. I found it very odd that this search bot listed the URL of a commercial UK website as its reference, but the company that operates the service does so from Russia. A visit to yanga.co.uk yielded little information – the UK website turned out to be little more than a placeholder page, with no UK street address and only a non-geographic phone number. I grew very suspicious at this stage.
A Google search found a thread on Webmasterworld.com in which “Alexey”, who introduced himself as the CEO of Yanga, responded to criticism. He didn’t give his last name, but perhaps he is Alexey Tarasov who is listed in the Gigabase Ltd WHOIS record.
While it is good to see that Yanga WorldSearch / Gigabase are concerned enough about their reputation to respond to public criticism, they would do well to take steps to raise fewer suspicions in the first place. They could definitely be more open about their identity and their purpose, as well as trying be a good citizen when visiting other people’s websites (e.g. complying with robots.txt, sticking to reasonable traffic levels). Trust is social capital that is hard to earn but quick to burn. No business can succeed on the web without it.