Never Miss a Beat: Get a snapshot of the issues affecting the IT industry straight to your inbox.
Twitter Struggles With Downtime, While Fending Off Irked A-List BloggersTwitter Struggles With Downtime, While Fending Off Irked A-List Bloggers
Twitter is coming off a rough couple of weeks. The service seemed to be down more often than up, and had to switch off services like SMS and instant messaging in order to throttle demand on its servers. If that wasn't enough, the Twitter team has had to fend off sniping from A-list bloggers like Robert Scoble -- who got the idea that Twitter was blaming him, personally, for the outage -- and TechCrunch.
6 Min Read
Twitter is coming off a rough couple of weeks. The service seemed to be down more often than up, and had to switch off services like SMS and instant messaging in order to throttle demand on its servers. If that wasn't enough, the Twitter team has had to fend off sniping from A-list bloggers like Robert Scoble -- who got the idea that Twitter was blaming him, personally, for the outage -- and TechCrunch.In a post on the Twitter Blog titled "It's Not Rocket Science, But It's Our Work," Twitter's managers talked to users on Saturday about the plague of downtime the service has been seeing. Twitter management described the system's current architecture. They use a single MySQL database for writes with multiple slaves for read queries. "As many know, replication of MySQL is no easy task, so we've brought in MySQL experts to help us with that immediately. We've also ordered new machines and failover infrastructure to handle emergencies." They're transitioning to a "simple elegant fileysystem-based approached rather than a collection of database. Until then, we are adding replication to handle the current growth and stresses, but we don't plan on ever relying on a massive number of databases in the future."
The Twitter team was responding to a list of questions posted on TechCrunch earlier on Saturday. Some of the questions were startlingly snarky:
Q: How long will it be until you are able to undo the damage [you] caused to Twitter and the community?
A: We're working extremely hard to keep the service stable and performing, as well as architecting a system that stands the test of time. We'd love to be able to tell you exactly how long this will take, but it's no easy task. It will take time, time well spent.
After Twitter responded to TechCrunch's questions, TechCrunch added a postscript to its post: "Twitter continues to be annoyingly and constructively responsive to criticism.... Kind of takes the air out of the balloon when you can't get them riled up."
In an earlier, May 22 post, "Twittering About Architecture"Twitter developer "Al3x" (a/k/a Alex Payne) said the service's problems are architectural: "Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency's sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we've tried to make our system behave like a messaging system as much as possible, but that's introduced a great deal of complexity and unpredictability," he wrote.
He added, "Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown. First and foremost amongst those requirements is stability. We're planning for a gradual transition; our existing system will be maintained while new parts are built, and old parts swapped out for new as they're completed. The alternative -- scrapping everything for "the big rewrite" -- is untenable, particularly given our small (but growing!) engineering and operations team."
A couple of Twitter users asked if the service's reliance on Ruby was the source of its problems. Al3x said: "We've got a ton of code in Ruby, and we'll continue to develop in Ruby with Rails for our front-end work for some time. There's plenty to do in our system that Ruby is a great fit for, and other places where different languages and technologies are a better fit. Our key problems have been primarily architectural and growing our infrastructure to keep up with our growth. Working in Ruby has been, in our experience, a trade-off between developer speed/productivity and VM speed/instrumentation/visibility."
Now here comes the part that got Scoble's knickers in a twist.
Al3x added: "The events that hit our system the hardest are generally when 'popular' users -- that is, users with large numbers of followers and people they're following -- perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that's behavior we have to limit on our side."
Blaming "popular" users is code for pointing the finger at one man: Scoble, said MG Siegler, writing on the blog VentureBeat:
Most users of Twitter will single out one person who this points to: blogger and Fast Company employee Robert Scoble.Scoble, with his 25,000+ followers and 21,000+ people he is following, is a beast on the service. I would consider myself a fairly heavy user of the service and I've sent 3,598 tweets (Twitter messages) -- Scoble has sent 12,318. This is clearly putting a strain on the service.
Scoble replied (on Twitter, of course): "Twitter blames me: http://tinyurl.com/6ot8gh Screw you Twitter!"
He elaborated on his own blog, Scobleizer: "A business that blames its best users is one that's in trouble. Serious trouble."
But everything ended in big hugs today. Scoble wrote:
Yesterday Evan Williams (co-founder of Twitter, his Twitter account is here) wrote me an e-mail telling me he wasn't blaming me and trying to clear the air. I said "can I come over?" to talk more about these issues face-to-face. I've always found that dealing with unpleasant topics is always better face-to-face and not over e-mail. He wrote back and said to come on over."When I arrived at Twitter Evan Williams met me at the door (I had Twittered that I was going to be there in a few minutes). We had an off-the-record conversation which wasn't, let's say, fun. But we both cleared the air and then the conversation started getting interesting and I asked "can I turn on my cell phone and start broadcasting this because I think a lot of people would be interested?"
What an enormous amount of drama. I'm on Twitter, like, all the time, and yet I can't work up a head of outrage about its recent outages. I'd love it if Twitter manages to recover from its problems -- which they seem to be doing, uptime, and services have been much better over the last few days than they were over the previous weeks. On the other hand, if they dive into the deadpool, I'll shrug it off and move on. I've been involved with many online communities over the years, they come and go. And it's not like I'm paying for the service.
Follow information on Twitter
information. has several feeds for the magazine and individual editors:
information All information article and blog headlines, with links to the full text, updated automatically and continuously.
iwpicks Only the best information articles and blogs, with links to the full text, updated manually.
MitchWagner Executive Editor Mitch Wagner (that's me, the author of this blog post).
MichaelSinger Michael Singer, West Coast News Editor.
awolfe58Alexander Wolfe, senior executive editor, online.
tecscoopSecurity blogger George Hulme.
phonescooper Mobile blogger Eric Zeman.
More Twitter news: Mars Probe Uses Twitter To Communicate With Earthlings
About the Author
You May Also Like
More Insights
