Bots Outpace Humans Online: Study

8 Linux Security Improvements In 8 Years

We're outnumbered. The rise of the bots has begun, and humans fall farther behind with each passing day. And that's the good news.

The 2015 Bad Bot Landscape Report, released by Distil Networks, said that the amount of traffic generated by "bad" bots" on the networks they monitor went down slightly between 2013 and 2014. The thing is, the amount of traffic with a human origin declined dramatically in the same period.

In 2013, human beings were responsible for roughly 55% of all network traffic monitored by Distil. By 2014, that had fallen to 41%. In the same time period, traffic from "bad" bots (those that are involved in espionage, fraud, theft, or cyber-vandalism) declined slightly, from 24% to 23%. What makes up the rest? Good bots.

[ What matters more: Technology or people? Read Technology Is A Human Endeavor. ]

According to Eli May, a spokesman for Distil Networks, "good" bots are those that perform benign or essential network tasks. "Think about the web crawlers that Google uses to index pages for its search engines," he said.

In an email exchange with InformationWeek, Distil Networks' CEO Rami Essaid further explained, "A good bot provides value back to the site. For example, Googlebot and Bingbot index your site so your prospects and customers can find you on the web. Clear value for your site. Also, if a bot doesn't respect robot.txt then we consider it a bad bot. Robot.txt tells bots which areas of your site they are allowed to access, crawl, or scan." Good bot traffic has increased in volume, but bad bot behavior has dramatically increased in sophistication — and in some networks the increase in volume has been in the hundreds of percentage points.

The report said that smaller companies are most at risk from the rise of bot traffic because they lack the ability to effectively block unwanted data flows. Even so, the company with the highest percentage of bot-generated traffic in its network was Amazon, with nearly 78% of all its traffic coming from machine-based sources.

Two facts from the report signal a more significant change in bot trends for coming years. The first is the rise in mobile-oriented (or originated) bad bots. According to Distil, mobile bots are now responsible for more than 8% of all web traffic. And while each of the 21 largest global wireless companies were a source of bad bot traffic, T-Mobile (USA) became one of the top 20 bad bot hosts for the first time in 2014.

The second important development is the increasing sophistication of bot behavior. Distil's report says that 41% of the bad bots they found are attempting to mimic human behavior in order to get into a web site's infrastructure. And 23% of the bad bots are so good at this mimicking that they are essentially unstoppable by technologies employed by most web application firewalls in use today.

For companies running public networks the lessons are clear: They must do more to try to stop the proliferation of bad bots, or risk choking their networks with machine-generated traffic to the detriment of human users. Organizations that depend on web-facing applications to transact business or reach an audience should step up their game in application-level defense, especially as more of their traffic comes from users via mobile devices.

[Did you miss any of the InformationWeek Conference in Las Vegas last month? Don't worry: We have you covered. Check out what our speakers had to say and see tweets from the show. Let's keep the conversation going.]