Sponsored By

If you use the "real-time short messaging service" Twitter, you know that over the last year there have been many performance issues. So many, in fact, that users have created a name for the outages, called the "fail whale." I started to poke around and began to wonder if bots running on the Twitter network could be causing some of the system performance issues on the service.

Allen Stern

October 29, 2008

2 Min Read

If you use the "real-time short messaging service" Twitter, you know that over the last year there have been many performance issues. So many, in fact, that users have created a name for the outages, called the "fail whale." I started to poke around and began to wonder if bots running on the Twitter network could be causing some of the system performance issues on the service.A bot basically performs a function or functions. In the case of Twitter, most of the bots either post links to content from across the Web or provide a reply service for certain phases.

To get an idea of just how many bots there are on the Twitter service, over the past couple of weeks I analyzed blog posts from more than 10 popular tech blogs. While this won't account for all of the reply service bots, it did provide good insight into the massive number of link-posting bots. For the purposes of this post, the bots listed below come from one post on the tech blog TechCrunch. Here's a list of the bots that reposted a link to the post on TechCrunch along with the number of subscribers (i.e., followers) to each one: top_post - 104 tc2tw - 25 techupdates - 171 RSS_TechCrunch - 350 techupdate - 31 TechFeed - 207 twittfeed - 116 RossRSS - 70 SocialMedia411 - 51 TechGlance - 449 top_blog - 112 nuws - 132 readmashcrunch - 96 wiredgnome - 43 techcrunchtweet - 47 balduaashish - 28 tinycomb - 31 readburner - 348 normman - 89 crazyengineers - 25 TwitLinksRSS - 392 seomasterlist - 207 simplynews - 56 tweetmeme - 643 memebot - 48 foobot - 42 readburnerRSS - 112 techgurus - 225 linksgoogle - 300 The bots listed above only represent one post from one tech blog. If you start to multiply this out across blogs and posts, you can see where this might present some level of performance issues for Twitter. Are these bots actually needed? How many are actually being used anymore? With so few subscribers on the bots listed above, why not offer one feed bot and reduce the server load overall? What's really interesting is that in the case of TechCrunch, its Twitter account has the full feed in the account, so none of these bots are even necessary for TechCrunch, although some of the bots listed also scrape feeds from other tech sites as well. As Twitter works on implementing a revenue model, I can only imagine that bot traffic could be the first hit.

Read more about:

20082008

About the Author(s)

Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.

You May Also Like


More Insights