Powered by InformationWeek Business Technology Network
Topics:
Digital Life : Web Tech
Twitter Struggles With Downtime, While Fending Off Irked A-List Bloggers
Twitter is coming off a rough couple of weeks. The service seemed to be down more often than up, and had to switch off services like SMS and instant messaging in order to throttle demand on its servers. If that wasn't enough, the Twitter team has had to fend off sniping from A-list bloggers like Robert Scoble -- who got the idea that Twitter was blaming him, personally, for the outage -- and TechCrunch. In a post on the Twitter Blog titled "It's Not Rocket Science, But It's Our Work," Twitter's managers talked to users on Saturday about the plague of downtime the service has been seeing. Twitter management described the system's current architecture. They use a single MySQL database for writes with multiple slaves for read queries. "As many know, replication of MySQL is no easy task, so we've brought in MySQL experts to help us with that immediately. We've also ordered new machines and failover infrastructure to handle emergencies." They're transitioning to a "simple elegant fileysystem-based approached rather than a collection of database. Until then, we are adding replication to handle the current growth and stresses, but we don't plan on ever relying on a massive number of databases in the future." The Twitter team was responding to a list of questions posted on TechCrunch earlier on Saturday. Some of the questions were startlingly snarky: Q: How long will it be until you are able to undo the damage [you] caused to Twitter and the community? After Twitter responded to TechCrunch's questions, TechCrunch added a postscript to its post: "Twitter continues to be annoyingly and constructively responsive to criticism.... Kind of takes the air out of the balloon when you can't get them riled up." In an earlier, May 22 post, "Twittering About Architecture"Twitter developer "Al3x" (a/k/a Alex Payne) said the service's problems are architectural: "Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as much as possible, but that's introduced a great deal of complexity and unpredictability," he wrote. He added, "Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed. The alternative -- scrapping everything for "the big rewrite" -- is untenable, particularly given our small (but growing!) engineering and operations team." A couple of Twitter users asked if the service's reliance on Ruby was the source of its problems. Al3x said: "We've got a ton of code in Ruby, and we'll continue to develop in Ruby with Rails for our front-end work for some time. There's plenty to do in our system that Ruby is a great fit for, and other places where different languages and technologies are a better fit. Our key problems have been primarily architectural and growing our infrastructure to keep up with our growth. Working in Ruby has been, in our experience, a trade-off between developer speed/productivity and VM speed/instrumentation/visibility." Now here comes the part that got Scoble's knickers in a twist. Al3x added: "The events that hit our system the hardest are generally when 'popular' users -- that is, users with large numbers of followers and people they're following -- perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that's behavior we have to limit on our side." Blaming "popular" users is code for pointing the finger at one man: Scoble, said MG Siegler, writing on the blog VentureBeat: Most users of Twitter will single out one person who this points to: blogger and Fast Company employee Robert Scoble. Scoble replied (on Twitter, of course): "Twitter blames me: http://tinyurl.com/6ot8gh Screw you Twitter!" He elaborated on his own blog, Scobleizer: "A business that blames its best users is one that's in trouble. Serious trouble." But everything ended in big hugs today. Scoble wrote: Yesterday Evan Williams (co-founder of Twitter, his Twitter account is here) wrote me an e-mail telling me he wasn't blaming me and trying to clear the air. I said "can I come over?" to talk more about these issues face-to-face. I've always found that dealing with unpleasant topics is always better face-to-face and not over e-mail. He wrote back and said to come on over." What an enormous amount of drama. I'm on Twitter, like, all the time, and yet I can't work up a head of outrage about its recent outages. I'd love it if Twitter manages to recover from its problems -- which they seem to be doing, uptime, and services have been much better over the last few days than they were over the previous weeks. On the other hand, if they dive into the deadpool, I'll shrug it off and move on. I've been involved with many online communities over the years, they come and go. And it's not like I'm paying for the service. Follow InformationWeek on Twitter InformationWeek. has several feeds for the magazine and individual editors: informationweek All InformationWeek article and blog headlines, with links to the full text, updated automatically and continuously. iwpicks Only the best InformationWeek articles and blogs, with links to the full text, updated manually. MitchWagner Executive Editor Mitch Wagner (that's me, the author of this blog post). MichaelSinger Michael Singer, West Coast News Editor. awolfe58Alexander Wolfe, senior executive editor, online. tecscoopSecurity blogger George Hulme. phonescooper Mobile blogger Eric Zeman. More Twitter news: Mars Probe Uses Twitter To Communicate With Earthlings « Untangle: Aptly Named | Main | Mars Probe Uses Twitter To Communicate With Earthlings » |
| Sign up now for the weekly InformationWeek Blog Newsletter. |