Bots Outpace Humans Online: Study - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
IT Leadership // Digital Business
Commentary
5/22/2015
08:06 AM
50%
50%

Bots Outpace Humans Online: Study

Humans were responsible for only 41% of network traffic monitored by Distil Networks; bots made up the rest. What does this mean for your applications?

8 Linux Security Improvements In 8 Years
8 Linux Security Improvements In 8 Years
(Click image for larger view and slideshow.)

We're outnumbered. The rise of the bots has begun, and humans fall farther behind with each passing day. And that's the good news.

The 2015 Bad Bot Landscape Report, released by Distil Networks, said that the amount of traffic generated by "bad" bots" on the networks they monitor went down slightly between 2013 and 2014. The thing is, the amount of traffic with a human origin declined dramatically in the same period.

In 2013, human beings were responsible for roughly 55% of all network traffic monitored by Distil. By 2014, that had fallen to 41%. In the same time period, traffic from "bad" bots (those that are involved in espionage, fraud, theft, or cyber-vandalism) declined slightly, from 24% to 23%. What makes up the rest? Good bots.

[ What matters more: Technology or people? Read Technology Is A Human Endeavor. ]

According to Eli May, a spokesman for Distil Networks, "good" bots are those that perform benign or essential network tasks. "Think about the web crawlers that Google uses to index pages for its search engines," he said.

In an email exchange with InformationWeek, Distil Networks' CEO Rami Essaid further explained, "A good bot provides value back to the site. For example, Googlebot and Bingbot index your site so your prospects and customers can find you on the web. Clear value for your site. Also, if a bot doesn't respect robot.txt then we consider it a bad bot. Robot.txt tells bots which areas of your site they are allowed to access, crawl, or scan." Good bot traffic has increased in volume, but bad bot behavior has dramatically increased in sophistication — and in some networks the increase in volume has been in the hundreds of percentage points.

'Bad bots' look more like humans with each passing year.
(Image: geralt via Pixabay)

"Bad bots" look more like humans with each passing year.

(Image: geralt via Pixabay)

The report said that smaller companies are most at risk from the rise of bot traffic because they lack the ability to effectively block unwanted data flows. Even so, the company with the highest percentage of bot-generated traffic in its network was Amazon, with nearly 78% of all its traffic coming from machine-based sources.

Two facts from the report signal a more significant change in bot trends for coming years. The first is the rise in mobile-oriented (or originated) bad bots. According to Distil, mobile bots are now responsible for more than 8% of all web traffic. And while each of the 21 largest global wireless companies were a source of bad bot traffic, T-Mobile (USA) became one of the top 20 bad bot hosts for the first time in 2014.

The second important development is the increasing sophistication of bot behavior. Distil's report says that 41% of the bad bots they found are attempting to mimic human behavior in order to get into a web site's infrastructure. And 23% of the bad bots are so good at this mimicking that they are essentially unstoppable by technologies employed by most web application firewalls in use today.

For companies running public networks the lessons are clear: They must do more to try to stop the proliferation of bad bots, or risk choking their networks with machine-generated traffic to the detriment of human users. Organizations that depend on web-facing applications to transact business or reach an audience should step up their game in application-level defense, especially as more of their traffic comes from users via mobile devices.

[Did you miss any of the InformationWeek Conference in Las Vegas last month? Don't worry: We have you covered. Check out what our speakers had to say and see tweets from the show. Let's keep the conversation going.]

Curtis Franklin Jr. is Senior Editor at Dark Reading. In this role he focuses on product and technology coverage for the publication. In addition he works on audio and video programming for Dark Reading and contributes to activities at Interop ITX, Black Hat, INsecurity, and ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Threaded  |  Newest First  |  Oldest First
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
5/22/2015 | 4:54:31 PM
Really, we can't outsmart the bots?
Isn't there some human-oriented test that would baffle even the 23% of bots that are good at mimicking people? Such as what is this -- with a picture of a birthday cake or some other object common to human experience but difficult for a bot to decipher. Maybe a picture with confusing multiple choice answers, such as, Who is this? with a picture of JFK or Marilyn Monroe. Most people can get that one right.
Joe Stanganelli
50%
50%
Joe Stanganelli,
User Rank: Author
5/23/2015 | 11:45:46 PM
Re: Really, we can't outsmart the bots?
They already have CAPTCHAs just like this -- and they are FREAKING ANNOYING for typical human users like me.  "Identify all the pictures of a burrito from the below pictures," it will tell me -- and I get stymied because I accidentally miss one because it kind of doesn't look like a burrito and looks more like a taco.

I mean, seriously.
Susan Fourtané
50%
50%
Susan Fourtané,
User Rank: Author
5/29/2015 | 4:54:05 AM
Re: Really, we can't outsmart the bots?
Joe, I find those CAPTCHAs equally annoying. I have missed some silly pictures as well because they are not clear enough. -Susan
Brian.Dean
50%
50%
Brian.Dean,
User Rank: Ninja
5/26/2015 | 4:11:16 AM
Re: Really, we can't outsmart the bots?
Image recognition software is progressing at a fast pace. Microsoft has an app that can analyze an image of a person and determine their age. It would just be a matter of time before someone creates a bad bot that can scan the internet to correlate an image with a word.
Susan Fourtané
50%
50%
Susan Fourtané,
User Rank: Author
5/29/2015 | 4:12:18 AM
Re: Really, we can't outsmart the bots?
Brian, that Microsoft app you said sounds horrible. Have you seen it in action? Do you know if companies using it and what for? -Susan
Brian.Dean
50%
50%
Brian.Dean,
User Rank: Ninja
5/30/2015 | 8:54:38 AM
Re: Really, we can't outsmart the bots?
@Susan, I have come across a few users that have input different images into the software and the results were both amazing as well as, a huge miss. An image of a +30 year-old individual was tagged as a 7 year-old.

The first use case of the application that I came across is from the video game industry. Developers have been using the software to modify their game characters and make the game look authentic. The matching of real faces with virtual faces could take virtual reality and storytelling to a whole new level. 
Susan Fourtané
50%
50%
Susan Fourtané,
User Rank: Author
5/30/2015 | 9:21:01 AM
Re: Really, we can't outsmart the bots?
Thanks, Brian. Very interesting. I wonder if this is a similar software that some stores wanted to implement some time ago as part of a marketing strategy. 

-Susan
Brian.Dean
50%
50%
Brian.Dean,
User Rank: Ninja
5/30/2015 | 10:03:43 AM
Re: Really, we can't outsmart the bots?
@Susan, I think you have nailed the value proposition on the head. I can't blame the on-ground retail stores that want to capture data about the age demographics of their foot traffic as long as it is anonymous because, online retail stores have had the advantage of studying their user-traffic for years. It is software like this that could prove to be the great equalizer.  
Susan Fourtané
50%
50%
Susan Fourtané,
User Rank: Author
5/30/2015 | 11:02:09 AM
Re: Really, we can't outsmart the bots?
Brian, yes, online stores have had some advantage in the past years in terms of data collection and analysis. Then it's also a question about prirvacy what comes to mind if we think of this kind of software being used in retail stores. You say, "as long as it is anonymous," but if a customer's face stays in the records, I am not sure how anonymous that could be. -Susan
Slideshows
10 Ways to Prepare Your IT Organization for the Next Crisis
Cynthia Harvey, Freelance Journalist, InformationWeek,  5/20/2020
News
IT Spending Forecast: Unfortunately, It's Going to Hurt
Jessica Davis, Senior Editor, Enterprise Apps,  5/15/2020
Commentary
Helping Developers and Enterprises Answer the Skills Dilemma
Joao-Pierre S. Ruth, Senior Writer,  5/19/2020
White Papers
Register for InformationWeek Newsletters
Video
Current Issue
Key to Cloud Success: The Right Management
This IT Trend highlights some of the steps IT teams can take to keep their cloud environments running in a safe, efficient manner.
Slideshows
Flash Poll