May 22, 2008

Twitter At Scale: Will It Work?

Nik Cubrilovic

162 comments »

Only two days ago the contact messaging application Twitter suffered another bout of downtime, leaving some users frustrated and others asking why the platform continues to suffer problems.

I have recently spoken to an individual who is familiar with the technical problems at Twitter as well as the challenges that lay ahead for the startup. He re-iterated his belief that the problems lay not with Blaine Cook (the former head of engineering who was shown the door), nor with ~~Joyent~~ NTT (their host) but with the early lack of understanding of how complex their problems would be.

The issue is that group messaging is very difficult to achieve at a grand scale. Other large sites such as Wordpress and Digg are mostly dealing with known problems, such as how to serve a large number of pages or a large number of images. Twitter is unique in that it needs to parse a large number of messages and deliver them to multiple recipients, with each user having unique connections to other users.

Social networks have similar complexity issues, but they only usually need to route a message to a single user (or at the most to a defined group). Even so, social networks like Friendster struggled for years with technical and scaling issues. Twitter is specifically dealing with text messages, and in most cases with active users those messages are very frequent and go out to hundreds of contacts (or followers, as they are referred to in Twitter). Every new Twitter user and every new connection results in an exponentially greater computational requirement.

Some of the best web applications are able to efficiently solve very complex problems to produce simple results for users (Eg. Google). The success of these applications is due to the innovative efforts by developers to solve large technical challenges, where they have often had to break new ground for solutions. For Twitter to reach a similar point of reliability they too will need a very comprehensive, ground-breaking solution.

The source that I spoke to also commented on how ill-prepared the Twitter team were and are for their current and future challenges. The small team contains a handful of engineers, with only a person or two committed to infrastructure and architecture. He goes on to point out that at Digg the team for network and systems alone is bigger than the total engineering team at Twitter, and that at Digg they are lead by well-known “A-list rockstars”.

The problems at Twitter are often attributed to their use of RubyOnRails, a web development framework. Twitter is almost certainly the largest site running on Rails, so fans of the framework and its developers have been quick to deflect the criticism and point it back at the engineers at Twitter. Utilizing a framework that has never conquered large-scale territory must certainly add to the risk and work required to find a solution. As an out-of-the box framework, Rails certainly doesn’t lend itself to large-scale application development, but was a big part of the reason why Twitter could experiment and release early.

Rails has enabled Twitter to prototype quickly, to quickly launch and then to easily iterate with new features. But the old adage of “Good, Fast, Cheap - pick two” certainly applies; and Rails would do itself no harm by conceding that it isn’t a platform that can compete with Java or C when it comes to intensive tasks. Twitter is at a cross-roads as an application and Rails has served its purpose very well to date, but you are unlikely to see a computational cluster built with Ruby at Apache any time soon.

What we see at Twitter today is a very useful and popular service, but one with very complex underlying technical challenges to overcome. Twitter will require not only a new architecture approach and a big injection of the best minds they can find ($15 million can help), but will also need a little patience from users and those of us observing.

minimize

CrunchBase Information

Twitter

Website:	twitter.com
Location:	San Francisco, California, United States
Founded:	March 1, 2006
Funding:	$5.4M

Founded in July 2006, Twitter is social networking and micro-blogging site that allows users to post their latest updates. An update is limited by 140 characters and can be posted through… Learn More

Information provided by CrunchBase

This entry was posted on Thursday, May 22nd, 2008 at 11:03 am and is filed under Company & Product Profiles. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Trackbacks/Pings (Trackback URL)

Comments

RSS feed for comments on this post.

dang son

May 22nd, 2008 at 11:07 am

said it before, and will say it again…. why cover such a crappy service. credibility issue alert
Faux Gentleman

May 22nd, 2008 at 11:10 am

I agree, this service is trash, yet it continues to receive coverage on TC
Gleb

May 22nd, 2008 at 11:11 am

Solving Twitter’s Tech problems is actually very easy.
Compare that to ennormous computing needs of Meebo for each user…
gerard

May 22nd, 2008 at 11:11 am

“with only a person or two committed to infrastructure and architecture.”

WTF? They’ve been at this for almost a year now. And they’ve only hired one or two people? Hope they’re fairly smart.
Tony Wright

May 22nd, 2008 at 11:12 am

FWIW, Scribd is a Ruby on Rails site that (according to Alexa) is quite a bit bigger than Twitter in terms of web traffic. Of course, the API might tell a different story.
David Cancel

May 22nd, 2008 at 11:17 am

Nonsense.

This is not that hard or unique scaling issue, especially how long they’ve been plagued with these issues. Similar Instant Messaging architectures are well documented and have scaled well beyond the scale of Twitter.

Patience? Twitter’s users are the most patient I’ve seen. These problems have been going on for over a year now. You can see their downtime and avg response times at Pingdom: http://www.pingdom.com/reports.....witter.com
Michael Kimsal

May 22nd, 2008 at 11:26 am

Mod me down (oh yeah, you can’t!) but I do not understand this notion of ‘delivering’ the messages to people. What is meant here? If I post a message to twitter, and I have 400 followers, are there 400 copies made and ‘delivered’ to each follower? Is the term delivery another way of saying SMS alerts?
Mathijs van Abbe

May 22nd, 2008 at 11:28 am

All external services which accesss the API cause quite a load on the Twitter servers. We saw that with one of our side projects… There are many external sites checking the feeds and calling functions…. This is one of Twitter’s strengths and also the Achilles heel….
chewbacca

May 22nd, 2008 at 11:30 am

7. I think you’re onto something, there’s something fundamentally wrong with what they’re doing.

PHP with Zend optimizations. That’s what they need.
Jason Reynolds

May 22nd, 2008 at 11:30 am

Why not decouple the computationally intensive pieces? Use C or something else for the high performance parts and keep it more agile with Rails on the front end. Seems simple, so maybe I’m missing something.

With regard to Scribd, does anyone know if they use Rails for transcoding docs to iPaper or just for the UI?
chewbacca

May 22nd, 2008 at 11:33 am

Woops, comment was trunc’d

7. I think you’re onto something, there’s something fundamentally wrong with what they’re doing.

That’s what they need. Though it must be a problem with _how_ they’re doing it.

PHP with Zend optimizations. I’m not impressed with Ruby, I’ve seen cases where it’s derailing before it even hits the track.
Cameron Watters

May 22nd, 2008 at 11:33 am

Methinks you spelled Blaine Cook’s name wrong.
Darren

May 22nd, 2008 at 11:33 am

But the old adage of “Good, Fast, Cheap - pick two” certainly applies and Rails would do itself no harm by conceding that it isn’t a platform that can compete with Java or C when it comes to intensive tasks.

or .net

I suspect the problem is rails and mysql. Maybe they should be looking at bigtable or a bespoke system.

The biggest problem is will they have enough time to get it right before people get annoyed and walk away for other solutions?

my take on an open twitter system http://darrenstuart.com/post/35530215
chewbacca

May 22nd, 2008 at 11:34 am

@10 - Digg and Facebook use C extensions for heavy parts. I think they’re relying completely on Ruby from what it sounds. You can only scale hardware enough where you need a fundamental software overhaul.
curmudgeon

May 22nd, 2008 at 11:35 am

agreed. this scaling issue is easier (much easier) than that of systems like phone switches. twitter management is at fault for not hiring the right dev talent all this time. even when they’re up their sys barely can keep up with message flow. clearly their initial sw architecting was incompetent. it’s embarrassing.
Peter Urban

May 22nd, 2008 at 11:35 am

I don’t know what’s all the fuss about. Twitter is a playground that keeps most of us distracted from our actual work. So I am actually thankful when twitter is down for a few minutes so we can get stuff done…

Remember it’s no rocket-surgery ass DHH says and Rails enabled twitter to come into existence in the first place. They would have never had the time, patience and knowledge to get it up in C++ and we would have never gotten that wonderful toy to distract us from work.

Now they might have to move on to build a rock-solid backbone infrastructure for a world-conquering system. By the way I’ve read that for a few years eBay had to re-write their back end systems on a yearly basis to keep up with the demand.

Are investors getting a bit nervous about the lousy reliability - probably yes. Should they make sure that the next round goes into a hardcore engineering team - yes they should. Is this all common sense… thats for you to answer.

By the way, maybe now would be a good time over at twitter HQ to have a strategy meeting over how to monetize the service (as in drafting out a business model). The implementation of such might dampen the growth a little - which appears to be just what they need right now. In turn if they had a way to monetize the growth scaling problems would actually be fun to have because they could hire more ‘rock-star’ engineers with all the incoming cash flow…

Back to our complaints: No-one (hopefully) tried to call the doctor via twitter to come to an emergency yet. And the 911 operators don’t have an twitter account yet either. So while the twitter guys are fixing their infrastructure problems we could just remember that there is a phone in case we really need to connect with a friend right now, that there is IM if we want to blabber with a few more at the same time and there always is email to spam everybody.

My humble advice to twitter:
- turn it into a business now, or it won’t survive no matter how big and reliable
- bite the bullet and re-write if you have to without complaining about the technology that enabled your success story.
- Don’t hire rock-stars, solid engineers sans ego-trip will do fine

Tip to the rest of us:
- Why not all of us go over to pownce for a week and see how they scale on a dime (would be a fun exercise)
- The thing buzzing in your pocket: its a friend that wants to talk to you

Peter
do you follow me @ http://twitter.com/peterurban -in case it’s up
Peter B

May 22nd, 2008 at 11:35 am

Facebook was dealing with a very similar issue when they were designing their chat system. The difference is that Facebook started their implementation with a clear goal to scale the system up to 70 million users early on.

Link to the article if anybody is interested in details: http://highscalability.com/new.....ing-erlang
Amanda

May 22nd, 2008 at 11:36 am

wah.
Pandrogas

May 22nd, 2008 at 11:36 am

This is why Twitter needs to be more of a decentralized platform to really hit up larger scale. I ranted about this a couple days ago http://www.socialbrew.com/prof.....ost%3A1501 Sorry if the rant isn’t terribly coherant, but the idea is to turn the micro-blog into a standard and use the internet as the computational network via multiple servers capable of hosting the standard API’s and allowing for tweaks, then just adjusting incoming and outgoing feeds.

No idea if that would work, but it would be neat if it did. Think Wordpress extremely light with social network capabilities.
Solacetech

May 22nd, 2008 at 11:37 am

Interestingly enough Twitter picked “fast and cheap”. In business you want to build customers as fast as possible and fix the bugs along the way. Check, they’ve done that and as loyal users we have stayed relatively loyal. The unfortunate problem for them is that they picked a platform that has a Drop Ceiling!!! If they can’t get more funding for more developers can they monetize what eyeballs they have left to pay for more developers? They have to act quickly before someone capitalize on this apparent weakness…
Thomas Vincent

May 22nd, 2008 at 11:37 am

Twitter’s problem is how it has attacked the problem. Twitter basically has the computational model of a stock exchange such as the NASDAQ not a web site. At a large bank you have millions of messages being passed around every second and actions needing to be taken on those messages. That is the same model as Twitter.
Nik Cubrilovic

May 22nd, 2008 at 11:37 am

I should have clarified this further in the post, but the issue certainly isn’t with creating new messages, the issue is when you check your own messages

For eg, if I am following 200 people, the query needs to check the new messages from all those people and sort them into date order. That query would be really intense

Now take that query and multiply it out for each user, and on top of that you have all the Twitter clients requesting the same query every x seconds

Caching only helps to an extent, since that query changes so often (ie. as soon as one of those 200 posts a new message, its a new result, hence a new (big) DB hit). It wouldn’t surprise me if the DB is falling over on those requests.

Also judging by message ID’s and what I have seen so far, their data is in no way segmented - so you have one BFT (big fucking table)

I agree that the delivery part can be scaled.
Yan

May 22nd, 2008 at 11:38 am

Performance DOES NOT equal scalability and FUD articles do no service to anyone. Java is faster than Ruby, and C is faster than Java, and direct binary machine code dominates them all, but none of them are any more scalable than the other.

Also, very little if any of twitter’s messaging system has anything at all to do with Rails. Rails is a web framework and has no relation to messaging queues.

http://skwpspace.com/2008/05/0.....es-of-fud/
beatofhawaii.com

May 22nd, 2008 at 11:39 am

There’s no excuse for terrible planning and management. Compassion doesn’t seem like the appropriate response when a product represents itself as public-ready, and very frequently fails to operate. Twitter down has become the joke of the office here.

It would seem that any savvy investor would have already realized that the underlying technologies were faulty and have rapidly undertaken re-engineering. I don’t think we’re dealing with impossible obstacles.

Aloha, Jeff
ravi karandeekar

May 22nd, 2008 at 11:39 am

Thanks for the explanation about Twitter’s problem in a simple language. I am a Twitter fan but do not know much about technology. Henceforth, i will be more patient. I am sure, Twitter will solve this issue soon.
Trevor Plantagenet

May 22nd, 2008 at 11:40 am

Actually, it’s much more complex than an instant messaging system, more akin to a group chat system, and those are notoriously hard to manage. A big part of the problem is exemplified by Robert Scoble. The amount of processing every time required anytime Scoble posts something or anyone following him posts something is staggering, compared to the average user. All of these super-followers are taking down the system - ideally Twitter should charge you if you follow over 200 people to discourage this, otherwise they’ll never manage the scale.
Charles

May 22nd, 2008 at 11:40 am

Big-O notation - ya heard of it?
Kristie Wells

May 22nd, 2008 at 11:42 am

Nik, I would like to correct this sentence: ‘nor with Joyent (their hosting company)’.

Joyent is NOT their hosting company. Twitter moved to NTT in January. The post reads as ‘current’ so I would like that noted. Please.
klub

May 22nd, 2008 at 11:43 am

Seriously? Another twitter post? Good God.
Fred Oliveira

May 22nd, 2008 at 11:45 am

I am flabbergasted at the amount of comments from people who are utterly clueless about computer science, scaling applications or development as a whole but that still are able to point fingers. You guys do have the right to state your opinion but keep in mind that just because you throw in a couple of buzzwords to the mix, it doesn’t make you right. This is true for most articles on the blogosphere that mention Twitter’s scaling issues.

#9: PHP with Zend optimizations doesn’t mean shit to Twitter because their growth problems are with their architecture. Actually, they’ve said so themselves, admiting their problems were with their database, not their frontend or API.

It’s pathetic to keep reading articles about Twitter and Ruby on Rails. If Twitter was implemented with a similar architecture and a different framework things would be no different. The problem they have right now is that in order to change their architecture to something scales effectively in terms of message delivery, they’d be redesigning (from a scientific perspective) the application all over again.

(PS: Nik: this is naturally not a rant about you or your article, although I believe you’re wrong about Rails being the issue here. Hope you’re doing well!)
Sam Stevens

May 22nd, 2008 at 11:46 am

Not all of Twitter is Ruby: http://twitter.com/ev/statuses/801530348
Plaxico

May 22nd, 2008 at 11:47 am

I recommend you read this blog post and a couple related posts on this site.

http://theabstracttruth.wordpr.....-problems/
Mike D.

May 22nd, 2008 at 11:50 am

Great post Nik. Seems to me that most people underestimate the complexity of what Twitter is delivering. It is NOT as simple as an Instant Messaging platform so a comparison to that is to compare apples and oranges. The messaging on an IM platform is mostly real-time (you might need to collect some IM’s while someone is offline) versus Twitter where each user has a history of tweets from their friends, their own message history, etc. that has to be served up to Twitter.com and via the API throughout the day. Also, think about the difference in scale. Most IM message traffic is 1:1 or you might have group chat with 10-20 people. When Robert Scoble or Leo Laporte send out a tweet it’s going to 20,000+ people at one time. And it’s not unusual for “average” users to have several hundred followers. Throw in the API support and the SMS layer and it gets even more complex.
Josh Fraser

May 22nd, 2008 at 11:50 am

Why doesn’t twitter talk to the guys at Google App Engine?

- Google’s infrastructure could handle the traffic w/o thinking twice
- “We made twitter scale” would be incredible marketing for App Engine
- Google engineers would probably pitch in to help them make the transition
- It would get twitter closer to an potential acquirer
Pablo Formoso

May 22nd, 2008 at 11:51 am

I’m sure that Rails isn’t the problem. Scribd is an example of this. They probably started Twitter as quick development with no specs and now they’re finding many issues realted to how to scale the db or kepping all the things together.
Rob DeMillo

May 22nd, 2008 at 11:59 am

This is an excellent article, and hits the nail on the head.

I was the SVP of Engineering at m-Qube back in the day when we were hooking up carrier-to-carrier text and data transmission. Our traffic growth was 23% every week…the traffic numbers dwarfed anything that Twitter has seen to date, but it took a lot of work, 24/7 vigilance and an engineering staff of 60 and Ops staff of 9 to keep up: Constant changes to databases, network infrastructure, SAN arrays, big iron in co-lo’s, changes to software infrastructure to support big traffic and rapid database I/O. We were making changes to the system and the architecture while we were in flight. By the time I left, we were handling north of 2B transactions a day (up from 17,000/day when we turned it on).

The system? Java. The appserver? JBoss.

Ruby one day might be the new Java (but I have first hand experience with Ruby - and personally, I doubt it), but that is a long way away. Ruby is - at best - a nascent technology. It is roughly in the same state of scalability and large scale system support as Java was in the very early 90’s. It makes great demo, but that’s about it.

Twitter is doing itself a serious disservice by not using that $15M-$20M to hire a new team of, as the article itself states, “rock stars” who understand large scale transaction systems and scalability, and also by not switching languages.
Ryan

May 22nd, 2008 at 11:59 am

It sounds like in Twitter’s case, the growth of the business results in exponential growth of traffic. That’s a losing game folks! More and more dollars chasing an ever-more demanding capacity/scaling challenge. Is that a smart investment?
Nik Cubrilovic

May 22nd, 2008 at 12:00 pm

Fred: your right that even if it was mod_php using a php database connector into MySQL, the query for latest posts would still be heavy (you might save ~5%). Im not sure what the reflection part of RoR adds as a cost overhead

In terms of Big O, the query for latest posts would be x^n, which is the problem as with twitter you have a large number of followers on average and also a large number of messages

Also interesting point that the top 0.1% of users probably do account for a huge % of load. It only gets worse with every extra follower they get and with an increased frequency of messages

Replacing MySQL with a distributed purpose-built DB (al BigTable, or sets of sqlite) is a solution

Also, all corrections made - thanks
Bob Wyman

May 22nd, 2008 at 12:01 pm

@21: The problem isn’t just “millions of messages…” It has to do with the specific pattern of messages. Traffic in a system like Twitter (and many other pubsub systems) is exceptionally bursty.

For example: TechCrunch has almost 18,000 followers on Twitter. That means that every time TechCrunch tweets, you need to
* Determine which of 18K users are following TechCrunch on XMPP and if they are online, send them the update.
* Determine which of the 18K users want SMS delivery and queue those deliveries.
* Update the “tweets received” list of all 18K users
* Update Techcrunch’s history, feeds, etc.
* Mark as “stale” any cached RSS/Atom feeds that might exist for the 18K users.
* Match the message against any of the “tracking” subscriptions that might exist (Some popular words could result in thousands updates…). Then, determine which subscribers are online and send messages as needed.
* various other things…

Now, do the above at the rate of many, many times per second for over a million users who each have anywhere from 0 to many thousands of followers… The system is exceptionally bursty and very unlike what you see in most other messaging systems that you might be familiar with.

There are not a great many folk out there who have experience with large scale publish/subscribe systems. Don’t be too quick to trivialize what Twitter is doing. There are methods to handle this stuff — but not too many people have ever even been close to needing to use them… (Note: But, I agree, they should be able to do much better than they are.)

bob wyman
Ryan

May 22nd, 2008 at 12:05 pm

Erlang. Enough said.
Aaron Brazell

May 22nd, 2008 at 12:07 pm

Two things that come to mind here:

1) Email
2) User Patience.

The fact that Twitter can’t handle complex algorithmic calculations and handoffs to worker bots, etc is not an excuse. Email is just as complex from a top-down perspective. Add in mail server configs of a rainbow variety and it may be more complex. The answer is distribution. Winer likes to claim he was the first one clamoring for this but I was long before him. If you treat Twitter as a standard protocol instead of an application, you a) alleviate concerns of downtime, b) distribute the cost (who needs $15M anyway), c) mitigate user data loss should the service ever cease to exist - which is inevitable at the rate they are going.

User patience. Look, we’ve come to rely on Twitter. You get what you pay for doesn’t cut it anymore. I would happily pay $20, $20, $40/mo to have a reliable service. The fact that people are pissed is not a reflection on user patience, but a reflection of the fact that the Twitter team is completely incapable of solving these problems. The fact that they can’t hire competent people to scale this thing with the funding they already have is a reflection of this. We had our share of problems at b5, but the company is in it’s first round of funding with an initial investment of $2M and they (formerly, we ) have hired smart people who have solved some of the scaling problems we had. You don’t need MIT PH.D’s to do this stuff -you just have to swallow your pride and hire people smarter than you.

To date, this article included, I have no confidence in Twitter. Yes I use it. Yes I get pissed when it’s down. But more so, I’m frustrated watching how they work.
Nathan Clark

May 22nd, 2008 at 12:09 pm

Nik - you just hit on the problem, except wouldn’t this be a factorial algorithm, rather than exponential? If I go from 40 to 41 isn’t a 41! process now required?

In either case, every time Twitter adds a small percentage of users, they could experience a doubling of infrastructure demand. We ran a group chat-like environment on an app and experienced a jump from 10m/s in traffic to 20-30m/s when our user count went from ~150 concurrent users to ~200. We found a way to strip out the bandwidth per transaction, but it’s still exponential for us. Imagining our problems on Twitter’s scale gives me plenty of sympathy for their downtime.
Matt

May 22nd, 2008 at 12:10 pm

@#5 two things wrong with your statement:

1.) “According to Alexa”

2.) “might tell a different story”… might???

The time to defend Rails (or any “Framework”) as a production tool is past. For prototyping? Sure, why not… proof that concept, raise some initial capital, and churn out a production level application with the moneys raised… (rather than giving executive raises in exchange for shorter work days and a big-ass party with free booze and expensive hookers*… Getting an investment should mean it’s time to work twice as hard, not half as much.)

* not saying twitter did this. but it’s the vibe of the valley in general. what they did not do was take their initial investments and turn them into a production level application. (they should not be on rails at this point, they simply should not. it’s got nothing to do with their bitchy users, it has everything to do with their gracious investors.)
Ryan

May 22nd, 2008 at 12:13 pm

#43, why is this even a Rails issue? If the web site is the root of their problems, then they are really lost.
JosefVirek

May 22nd, 2008 at 12:17 pm

These scalability issues are really relatively easy to fix if you’ve been there, done that…

Sure it would be easy to scale Twitter with a 100-node Oracle 11g RAC Cluster, but it could be done using MySQL (it’d be harder, but it can be done). The problem is not the tools (RoR, MySQL, etc.), it’s the architecture.

I’ll quote Jack Nicholson…

“I’m an artist, you give me a f%$@ing tuba and I’ll get you somethin’ out of it.”
Frank Church

May 22nd, 2008 at 12:18 pm

I use twitter, but I am certainly not reliant on it. It is interesting, and learning exactly how it works from using the site, 3rd party apps, and Pidgin (my IM client) has been really interesting. 98% uptime is more than enough for me. and I hate whiners.

But, what I REALLY hate is idiots that think they know anything about the technical challenges that face twitter. IT IS NOT SOME LANGUAGE OR FRAMEWORK ISSUE!

Real-time, cross-protocol, archiving, access-control (my twitter feed is not public), follower lists and followed lists> Just not quite as simple as you all think.
Ehud

May 22nd, 2008 at 12:19 pm

I’m going to write a greasemonkey script that hides every techcrunch article tagged with twitter.
Seriously, this is getting out of hand. Not only is twitter WAY over hyped, the amount of coverage it gets here is just staggering…
A Williams

May 22nd, 2008 at 12:28 pm

Please, quit blaming rails

As you know….wait, as you obviously don’t know (from the post, your comments show otherwise for some reason), the way you scale any web app (php, python, asp.net, rails, anything) is to separate the application from the database. At that point you can have as many app servers (ruby on rails) as you need, and you have to deal with the database.

It is true rails as is isn’t saleable out of the box, but that hardly means “rails isn’t scalable”, an hours work could get the majority of an application following a custom schema, whether that means partitioned, or slaved.
vishal

May 22nd, 2008 at 12:28 pm

I believe The SMS plug could be causing problems. Also, this is not a 1-1 chat but a one-to-friggen-many-many. thats what causes problems every time arrington or scoble send out power-tweets. the IM backbone, built primarily for one-on-one chat takes a hit. so my take is that they need to rebuild the backbone. its just like the cell networks going down during a tsunami or terrorists attack as everyone starts using them. *sigh*. probably need to rebuild the architecture ground up again, taking into account its a 1-2-friggen-many-many.
Nik Cubrilovic

May 22nd, 2008 at 12:29 pm

Nathan: you might be right that it is x!

For those not familiar with it, it is the traveling salesman problem:

http://en.wikipedia.org/wiki/T.....an_problem

So every additional user/follower adds more than just a proportional level of load. Twitter should publish their numbers so that users understand the problem more.

Going from push to pull is a good solution as well. I remember reading somewhere that a huge % of the traffic is API traffic.. so push makes sense. For web apps you would just need a callback URL, for desktop apps NAT traversal

I think it will end up being decentralized, either Twitter will come out and specify the standards or somebody else will do it and they will eventually build in support. It just feels like they are trying to do the equivalent of implementing all of email in a single central service

Also the emphasis of this post is about the core problem, not so much about the tools they used. I am saying that the solution they find for Twitter at this scale and beyond is likely not to involve RoR (or MySQL for that matter)
Peter Cooper

May 22nd, 2008 at 12:30 pm

Nik @22: You are right, but there is a way to optimize this without getting too advanced.

Feed Digest had the same scaling issue (collecting together the latest items from up to 100+ feeds, then ordering by date, sometimes filtering by search query, etc.). It never got to the point where I had to implement this, but..

Instead of relying on DB calls, when a new message is posted, you can add it to a queue. You can then go through this queue and add the new message(s) to user-specific queues for all of the posting user’s followers. This removes the DB query for the most common task.. reading the messages of all your followers. It’s all in a personalized queue for you. It means you end up with a much larger disk space requirement, but the CPU requirements go through the floor.

When you unfollow a user, that user’s items get removed from your queue. When you follow a user, that user’s latest items get added to your queue. Easy and non CPU intensive.

The only slightly intensive action with this arrangement is if someone you’re following /deletes/ a message (very rare on Twitter). If this happens, a queue manager would have to remove that item from all of the follower’s queues, but that’s hardly a big job since you now have all the spare CPU cycles to deal with it…
Peter Antypas

May 22nd, 2008 at 12:30 pm

Scalability is a function of architecture, not implementation. Give me a good architecture and a so-so implementation and I’ll deliver, albeit at a high cost. Give me a poor architecture and I will fail, no matter how “good” the implementation.

A complex pub/sub system like Twitter needs flexible partitioning and redundancy to achieve both scalability and availability with acceptable performance. “Scalability” means that as the number of users (and events) increases, new instances of pluggable components are added to handle the extra load without any changes to the architecture. “Availability” means that there’s at least 2 components capable of accomplishing the same (partitioned sub)task, and each one can handle 100% of the load. For twitter, 99% availability would be perfectly acceptable I think.

All this can be achieved with distributed services running in a cloud, written in any programming language and using any kind of storage (from mySQL to files). As long as they are well written and have a reasonable degree of fault tolerance (again, this is entertainment, not banking!!!) things will work and scale fine.

People whose idea of software architecture is Rails or J2EE or .NET are not equipped to handle this.
ivan s.

May 22nd, 2008 at 12:33 pm

Del.icio.us has been dealing with this problem for years. The network part of del.icio.us used to be pretty unreliable too.
sodapop

May 22nd, 2008 at 12:34 pm

HAHAHA you fools trashing Twitter. The only thing that’s important is that it’s popular and defining trends. If you don’t have the business sense to understand that, there is no help for you.

The real reason fo my comment. Use http://m.twitter.com/home it stays up.
Angelos

May 22nd, 2008 at 12:35 pm

The loyalty to Twitter is an interesting example of the market dynamics in the web economy. Consumers of tangible products are much less willing to hope and wait while developers spend money for a solution.

Part of the reason for Twitter’s surprising dominance is the “hype machine” other readers have mentioned in disgust. Couple the press saturation with users apparently fearful of changing to another service and we have our current situation.

Here is a simple solution…look to other services that provide the same functionality. Too bad web companies don’t clone each other
Peter Farber

May 22nd, 2008 at 12:37 pm

is this nik from omnidrive?
Greg

May 22nd, 2008 at 12:39 pm

Great article, probably the most reasonable look at the subject I’ve seen.
Jim

May 22nd, 2008 at 12:41 pm

You have to be very careful not to mess with wrong company. Twitter is like new internet hype or rockstar. They rating is huge. If you keep attacking Twitter. Twitter is going to kill you and destory techcrunch weakness.

How do you know Twitter have problem with RubyOnRails?
Dan Benyamin

May 22nd, 2008 at 12:41 pm

(ok third time’s a charm with this post)

Why hasn’t someone reverse-engineered twitter, released it under an open source license, and allow it to become naturally distributed (not to mention peer reviewed). Worked for Gnutella…
Tony

May 22nd, 2008 at 12:44 pm

I think the people blaming Rails AND the people saying Rails has nothing to do with the problem both need to take a step back.

The fundamental problem is the architecture, at the database level. At least, that’s what we hear from Twitter. So, in and of itself, the problem isn’t Ruby or RoR.

The problem lies in the fact that Rails puts you into a certain box when you develop. Yes, it’s a great development tool for anything that fits within that box. But once you want to do something different, something it wasn’t intended to do, you start having problems.

By choosing the Rails framework, Twitter ended up in a box. That box has proven to be a problem for them.
Matt Malinger

May 22nd, 2008 at 12:47 pm

First, as other people have noted, they need to drop RubyOnRails and replace it with something more appropriate for scaling, like multiprocessor ErlangWeb backed with the Java RTC/HDR distributed notification framework. MySQL is great for startups, but it’s just not up to this kind of job, you need *at least* an Oracle XMP/12C cluster; anything less will just fry the I/O queues (I’ve seen it happen and it’s not pretty.). And obviously none of this does any good without hardware that can actually drive it. The usual replicated Intel boxes will never keep up, they should spend a good chunk of that $15M round on planar blade racks with multicore protocol pipelines. And for Christ’s sake, RAID-ify the fallover disk shards, I can’t count the number of times people forget to do that and you’re back to square one.
Patrick Altoft

May 22nd, 2008 at 12:55 pm

9 mobile social networks to use when Twitter goes down: http://tinyurl.com/5gul2e bookmark for when the need arises.
Joe Stump

May 22nd, 2008 at 1:02 pm

@chewbacca We (at Digg) don’t use any customized C extensions for PHP (yet), but we do use other languages where it makes sense. We currently have Java, Python, Perl and PHP in production. I think this article finally makes a decent case for why Twitter has had problems. They prototyped a crazy idea they had in a great prototyping language, Ruby, and it worked beyond anything they ever anticipated. Despite the problems everyone needs to realize that it’s an absolute Herculean effort by Twitter’s technical team that it runs as smoothly as it does now. Kudos are in order for those peeps.

Blaine was, essentially, a one man show up until about a year ago. The fact that him, along with a handful of engineering and operations people kept Twitter up in the manner they have is a testament to their combined efforts in my opinion.

Also, these problems have been solved before. The telecom industry has been routing billions of messages a day for decades. Unfortunately, they’re not exactly open with their solutions and those solutions aren’t exactly cheap either if you do manage to get access to them. This means that Blaine, Twitter, et al have been left on their own to solve horribly complex problems that not many other Web 2.0 companies are dealing with on the scale that they are.

–Joe
Ryan

May 22nd, 2008 at 1:02 pm

#59, the SMS component is one reason. Twitter is subsidizing that part.
john

May 22nd, 2008 at 1:05 pm

all technical problems can always be traced back to fundamental cause: personal issues and egos.
Jon

May 22nd, 2008 at 1:05 pm

There are plenty of people in silicon valley who can solve this problem. As others noted, sites like Facebook, Digg, and Meebo (esp. Meebo) rely heavily on C to do the heavy lifting. If Twitter is 100% Ruby, that’s foolish beyond belief.

As for the queries mentioned by Nik, the complexity arises only if you think in terms of a fat, disk-based, RDBMS. There are better approaches for a system such as Twitter.

Bottom line: from all that I’ve heard, the underlying problem is that Obvious has been acting like a product innovation shop that turns out neat applications that are ready for someone else to acquire and scale.

A company that wants to build out a game-changing platform invests far more heavily in technology and engineering. And there’s no question that Twitter is fundamentally more complex than sites like Digg. What have those guys been thinking for the last few years? It’s hard to point to a lack of experience- Blogger obviously had similar scaling problems.
Ryan

May 22nd, 2008 at 1:08 pm

#63, I concur, although the phone company is dealing with 1-1 interactions, while Twitter is many-many. In some ways, Twitter looks much more complex.

In regards to the language debate, I think your blog post sums it up pretty well:

http://www.joestump.net/2008/0.....tupid.html
John

May 22nd, 2008 at 1:10 pm

As a guy who has built a pub/sub system that handles in excess of 6 billion messages a day I can say that not ONE person on this comment list has even come close to the solution that would solve a very basic, though completely unknown if you haven’t deployed, architectural challenge. And….the solution is pretty inexpensive!

It’s not rails, not the db flavor and so on and so on.

The one thing I would agree with is that the majority of people *TODAY* that are completely dependent upon and flogging themselves when Twitter goes down are the uber geeks of the world. Or bloggers that are looking for lots of comments and backlinking (nice work TechCrunch).
Mogilny

May 22nd, 2008 at 1:10 pm

Wow. It feels like i’m in a room of doctors debating over how to rescue their patient. Here is the thing. Twitter is not your patient and even if you are the greatest doctor, if you don’t know your patient, how good can your prognosis be? Feel free to start a TwitterOffRails or a TwitterOnErlang.

In twitters defense, their original use model was between friend. They didn’t expect attention hungry people to use it as their public announcement media. Maybe they should have limited people to have no more than 50 followers. Growth is great, but uncontrolled growth is what I call a malignant tumor.
Jason

May 22nd, 2008 at 1:12 pm

Just use .Net and SqlServer and watch all the problems go away
AW

May 22nd, 2008 at 1:13 pm

@59: why reverse engineer? It’s apparently easy to copy.

http://meow.appspot.com/ is one clone.
dave

May 22nd, 2008 at 1:17 pm

“…Rails enabled Twitter to be developed quickly, to get to launch quickly and then to improve with new features relatively rapidly also.”

dis iz good writin! techcrunch iz da goodest!
Don Wilson

May 22nd, 2008 at 1:17 pm

Facebook does it with the Facebook Feed.
MySpace does it with user statuses and their news feed.

Don’t tell me you can’t accomplish this with $4m+ in the bank. This must be a joke.
Jim

May 22nd, 2008 at 1:17 pm

@71. I saw someone else that idea. They release GPL for free. It’s too late.
Google is everywhere.
randy

May 22nd, 2008 at 1:23 pm

OK, this is getting RIDICULOUS.

This is not a joke anywmore! Can you PLEASE fork all this Twitter garbage into TwitterCrunch?
Peter Antypas

May 22nd, 2008 at 1:24 pm

Continued from #52:

The best thing that Twitter can do at this point is to spend some of their new VC money on acquiring top talent. They need an architect with hard-core infrastructure background (telecom perhaps), someone who really understands scale. There’s nothing to be invented here; all these problems have been solved before.
Giles Bowkett

May 22nd, 2008 at 1:25 pm

Twitter isn’t the largest site on Rails, you ridiculous MBA-having suit-wearing dumber-than-a-monkey Friends-episode-reject J-Crew-catalog muppetfuckers. It isn’t even #3, you scum-gobbling weasel-brained magic-eight-ball-flipping clueless fucking nimrods.

http://rails100.pbwiki.com/

I’d tell you to Google shit instead of making it up, but it’s obvious from your ad prices that making shit up instead of Googling it works really well for you, so by all means, keep bullshitting, you worthless, brainless, subhuman Republican worms. Just realize there are a lot of people who know more about tech than you do, and while you get paid for lying about it, we get paid for telling the truth about it, you cockroaches, you mutant rats, you empty-headed bimbo-plungers, you soulless fart-drippings.
owned

May 22nd, 2008 at 1:28 pm

World War 1: Twitter vs. Techcrunch
Yes, Twitter won and leaving techcrunch insult behind…

Techcrunch got OWNED
Dan Benyamin

May 22nd, 2008 at 1:39 pm

#64: build popularity in an open source, SMS-free approach (I bet the majority of twitter users have smartphones anyways…). Then carriers would have an incentive to host their own endpoints, since there’s a bunch to be made off of SMS fees….
vicaya

May 22nd, 2008 at 1:43 pm

Sorry, the article and most of the comments here are ill informed. Twitter’s pub/sub problem is a solved problem. Especially they don’t need to do it in real time (i.e. a minute or two update delay (not query latency) for the followers is no big deal. Even large scale real-time group messaging system is a solved problem (think about your real-time stock trading system). There is even an open source solution for that (google zeroc icestorm), which is probably already more than they need No Google like innovations need here, just solid design and implementation.

With their new round of funding, I’m sure they can find a few guys to fix that. No, it has nothing to do with their Rails front-end (which is great for rapid feature development) and everything to do with the messaging/db back-end.

BTW, Twitter is almost certainly NOT the largest rails site
JerkCrunch

May 22nd, 2008 at 1:52 pm

Jaiku is waaaay better than twitter
Lets all move to Jaiku and see if Google scales like a pro
izmir evden eve

May 22nd, 2008 at 1:56 pm

thanks.
Michael Kimsal

May 22nd, 2008 at 2:00 pm

@PeterCooper

Might you not just store a reference to one DB record in people’s queues? That would solve the deletion problem, and makes the reading of the queues just a select where ID in (ids from queue). I’m simplifying that some, but it’d also make the disk requirements go down (not as much duplication) and the storing in queues faster (only storing IDs, not full text).
Jan Erik Paulsen

May 22nd, 2008 at 2:07 pm

Why not have Letterman ask “Will it float ?”. Oh, noes. It did not float. Must be Ruby on Rails !
Gordon Joly

May 22nd, 2008 at 2:11 pm

Twas my first thought when I saw Twitter…. will it scale!
Adam

May 22nd, 2008 at 2:22 pm

Nik- Multiply does the same thing as twitter, only not as annoying - and it’s a social networking site. And it doesn’t crash on the reg
Lars Fischer

May 22nd, 2008 at 2:39 pm

Ever heard of Message Queue software? Use Java for this part and it will scale if the problem is the number of messages.

But I guess the bottleneck is the database. A good database design with scalability in mind is key.
Trevor Plantagenet

May 22nd, 2008 at 2:46 pm

I can only hope that Scoble and the other mega-followers get kicked from Twitter so it can become useful for regular people again. Stop using an unfair share of processing power, Scoble! Every time you tweet, a Twitter server dies.
dangrsmind

May 22nd, 2008 at 2:47 pm

Some good posts here on scalability issues for those that care to read between the fluff. The issue here seems to be more the perceived pain, cost, and upset of moving to a new architecture that might potentially be radically different from the one supporting the current service rather than simply the less than adequate QoS of the service.

Such global re-architecting is fraught with risk and any experienced software person would and should IMO approach it with due caution. The failure scenarios are pretty well known here. The risk doesn’t mean you don’t do a re-architecting, because given the observed problems and stated architectural deficiencies the move here seems inevitable. Twitter shoudl stop the hand wringing and get on with the tough job ahead. In the meantime its going to be a bumpy ride.
David Cancel

May 22nd, 2008 at 2:49 pm

@Trevor+Plantagenet and @Mike D.

Twitter is indeed very similar to a modern IM platform such as Jabber. Obviously the semantics aren’t 1 for 1 but all of these problems have already been solved, and are dealt with each and every day. And any “simple IM platform” dwarfs the scaling problems faced by Twitter, a relatively low traffic asynchronous application.

I believe Twitter’s messaging already runs on ejabberd an Erlang-based Jabber server.
Nay Say

May 22nd, 2008 at 2:55 pm

Which is worse news?
1. Twitter engineers are incompetent fools.
2. Twitter model simply cannot be scaled.
It could be that (2) is true, and Twitter would rather want people to believe in (1), for as long as they can until they cash out.
brian

May 22nd, 2008 at 3:00 pm

need to get themselves some Akamai. Doesn’t solve their backend issues, but helps front-end / network / failover. All the other social nets use Akamai.
Angelos

May 22nd, 2008 at 3:01 pm

I just started a campaign to switch the conversation from Twitter to another service. This is the main reason why people are not leaving…Twitter is for conversations, and we all know porting your network to another service is not easy. Here is the link if anyone is interested in getting a modest sized group to develop a conversation on another platform http://is.gd/kvT
Rob

May 22nd, 2008 at 3:23 pm

Twitter is pointless as far as I can tell. It seems perfect for the younger American culture, and their need/belief that they are popular enough to think people are paying that much attention to them. Who cares what you are doing minute to minute. My last Twitter announcement: Setting up Twitter, 3 days ago.
silicon valley dropout

May 22nd, 2008 at 3:29 pm

f**k twitter
it folks like them that made me lose my job in silicon valley
AW

May 22nd, 2008 at 3:34 pm

@94: Make this your new one: ‘rambling about how much I hate twitter on a blog hoping someone will reply to me’
sh

May 22nd, 2008 at 3:35 pm

IMHO, the Rails brain trust should be lined up at Twitter HQ’s door to make sure that the most visible flagship of their framework has all the architectural support it needs.

If Twitter is forced to drop Rails in order to better handle their traffic, even if it has NOTHING at all to do with Rails, I could easily imagine a lot of would-have-been-Rails projects using other languages instead.

That could mean fewer attendees to all those lucrative Rails conferences.
Peter Cooper

May 22nd, 2008 at 3:56 pm

@Michael (83): Sure, you could do that. That’s a great mid-way solution that retains some abstracting without going too far. That said, actually compiling out to as close to an intermediate format as possible could work better. Leave all DB related activity for the less common activities. Even inserting messages should be queued (I’d be surprised if Twitter isn’t doing this - it saves so much hassle).
Trevor Plantagenet

May 22nd, 2008 at 4:04 pm

@90 No, Twitter is not like IM, and I’m not sure why you keep saying it is, except maybe you’re getting confused by the SMS portion. Because Twitter is many-to-many, it’s more like a chat room, which are notoriously difficult to scale past a certain number of people in a single room, or like an email system where every user has their own mailing list. Assuming you actually are a programmer, go through the thought experiment of figuring out how you’d build Twitter yourself, and you’ll quickly realize the complexity of it, which is why there aren’t 10 good twitter clones right now.
mark.t

May 22nd, 2008 at 4:35 pm

ITS TIME TO START USING APACHE, PLEASE GET RID OF TRASH ON RAILS!!

Twitter doesn’t scale, it sucks!
cammo

May 22nd, 2008 at 5:09 pm

But does everyone commenting here think that in over a year of attempting to solve these issue’s that the lads at Twitter would not have looked at(and probably implemented) C extensions? Or other languages? Or other frameworks? If they had a year and re-wrote it in Erlang would that solve their problems, I’d have a guess and say no. Twitter’s issue is that it is a free service, with basically no revenue model, so they cannot pump money into dozens if not hundreds of servers and/or developers.

PS. mart.t, seriously learn something about the web bro…. Rails is not anything to do with Apache you idiot, Apache is a webserver, Rails is a framework for development of web based applications in the Ruby language, similar to Cake for PHP or Spring in Java. Apache != Rails you fool!
alan p

May 22nd, 2008 at 5:26 pm

Anybody worth half their salt knew all this ages ago - see our post here for example:

http://broadstuff.com/archives.....ailed.html
Mike D.

May 22nd, 2008 at 5:48 pm

@David Cancel (#90) As Trevor pointed out (#99), the IM gateway is just one piece of Twitter. I suggest you read the transcript from a recent Gillmor Gang where ex-Twitter engineer Blaine Cook talked about the scaling issues…

“if I were to build it from scratch to make it bulletproof, the messaging system, you know, I would take a lot of time looking at that. And I mean frankly, I don’t think it is a secret that the hard part of scaling Twitter is not the messaging part of it. That’s a fairly well understood thing. The hard part is well SMS and the second part is displaying archives; so, displaying your friends’ timeline.

There are some complicated things in there, I mean, I am not going to get into them on the phone call, but building to optimize that and make that scalable. You know I’d love it if there were - if [indecipherable] or Hypertable or any of these other alternative data storage were usable right now for large scale systems, they are not, they are moving very quickly and so hopefully they will be.”

Audio and complete transcript is available here:
http://gillmorgang.techcrunch......ng-051508/
Diogo Peixoto

May 22nd, 2008 at 6:30 pm

@Josh Fraser

I believe Google will not help Twitter to solve their problems. Google has bought it’s own micro blogging tool called Jaiku (jaiku.com)
Vijay

May 22nd, 2008 at 7:35 pm

I have to roll my eyes. It’s a free service; stop complaining!
113.com

May 22nd, 2008 at 7:47 pm

Not worthy of discussing, at all. The problem is not down or not down. (It is a problem but not the problem). The problem is their “love-me-love-my-dog” attitude. Period. Just find a better service.
BeenThereDoneThat

May 22nd, 2008 at 8:03 pm

There is nothing unique about Twitter’s problem from a technology perspective. Its been done before, many, many times. Internet Email. Internet Messaging. Internet Chat Rooms. Internet Social Networks. All of these systems have scaled to BILLIONS of messages/events/data items and perform well (or at least good enough given they are all free). It all boils down to some basic technical principles on how things are designed. Any buffoon rent-a-coder can throw together a website, but few can (re)design one to operate on the scale of Google, Yahoo, Facebook, etc. Twitter needs to hire 1 or 2 smart engineers from Google/Facebook/Yahoo/etc. that have Been There and Done That. The service could be rock solid within a month. It still amazes me that they haven’t done that.
YDRIVE

May 22nd, 2008 at 8:18 pm

@107 - right on.
YupYup

May 22nd, 2008 at 8:32 pm

@107 - Totally. Was about to write the same thing. They really should look at some MQ or maybe some federated XMPP architecture and then sprinkle their frontend on top. And yeah, it sounds like haven’t hired the right person or if they have him, they don’t listen to him.
Kevin Merritt

May 22nd, 2008 at 8:33 pm

I agree with a number of previous comments that twitter’s problems are not as unique as the post makes them out to be. twitter can learn a lot from IM, email and paging networks. Stock trading networks have orders of magnitudes more traffic with the need for 100% guarantees of message delivery.

As I said in a post earlier today, twitter’s problems are solvable but they do need to step up and start that process:

http://blog.blist.com/index.ph.....ver-again/
Remind me

May 22nd, 2008 at 8:47 pm

Can somebody clarify what _any_ possible business model would be for twitter? I mean they have shitty performance so I doubt anybody would pay them a monthly subscription, and there is unlikely to be a way for them to sell ads with their free service.

Of course TechCrunch is yet again covering this turd, but of course a company like Yahoo that has never lost money MUST SELL TO MICROSOFT RIGHT NOW(!!!!).

What a load of steaming crap this site is.
fredrick

May 22nd, 2008 at 8:54 pm

@69: more like a bunch of slashdotters discussing women!
Sqij

May 22nd, 2008 at 9:14 pm

All they need to do is re-architect message delivery by talking to Bram Cohen, and learning lessons from the BitTorrent distributed model.
AkitaOnRails

May 22nd, 2008 at 10:04 pm

“Rails would do itself no harm by conceding that it isn’t a platform that can compete with Java or C when it comes to intensive tasks”

As a good journalist you probably did your due diligence, right? So point me out where exactly does Rails states anything different than “web framework”, or better: point me out anywhere Rails states that it is a “system framework”.

It is obvious now that the author of this article is more preoccupied in generating fuzz using FUD. The Twitter guys don’t use just Rails. They have a mixed architecture and, from as far as I’ve heard, PHP, Erlang and C in the mix as well.

Notice that when their system fall down, it is not only the Rails based website that goes off, but all the other channels, including the APIs that dozens of clients use, the Jabber broadcast and so on.

They either have a much bigger architectural problem: which is never easy to solve whatever mix of languages and technologies you’re using; or there is a shortage in resources and they can’t afford more boxes or better technologies.

Bashing Twitter is ok. Mixing Rails in the same article without any hard evidence is asking to be called FUD and unreliable liar. Do your home work.
Gabe

May 22nd, 2008 at 10:42 pm

@112 - No shit! For some reason these Twitter articles encourage the most clueless comments. It may be that the right developer can re-architect Twitter to scale within a month, but I guarantee you it’s not one of this cocky armchair critics claiming how it’s “so easy” and “a solved problem”.
Jitendra Sharma

May 22nd, 2008 at 11:00 pm

Remember, failure is the best reason to success. Anyway this type of article is very very helpful to understand the problems that might occur in a web application for a newcomer like me. Thanks to open source community.

But i don’t think RoR is the problem for this issue. Although this has been proved that there is a technical issue but that does not mean that it is a technical failure. I think to fixed it is not so difficult. As there are applications running and doing good with similar or even with high traffic like Scribd, Yellowpages.com(Both are RoR application with high traffic that twitter, as per my knowledge, might be wrong).

The Reason is the Failure of Twitter Management. May be due to not hiring the sufficient engineers or may be they were unable to looking so forward.
anon

May 22nd, 2008 at 11:07 pm

I don’t understand why Twitter should be so difficult. Several search engines search the whole internet in a fraction of a second.

RoR is definitely not the right platform. RoR is much more suited to complex business logic, rapid iteration, etc. Not heavy duty message delivery.
AkitaOnRails

May 22nd, 2008 at 11:12 pm

Paraphrasing Richard Feynman “if you think you understand the Twitter problem, you don’t understand the Twitter problem”.

This is the best post I’ve found so far with a clearer scenario: http://www.25hoursaday.com/web.....blems.aspx

Bottom line: this has nothing to do with Rails. This has - probably - to do with the “Follower” feature and the way it is currently implemented. If you think it is trivial: I challenge you to show me a working piece that you did and is actually standing still a similar overload!
Anthony

May 22nd, 2008 at 11:42 pm

Perhaps the owners of twitter are just trying to keep the ship floating until it’s sold? Maybe they don’t care what you “geniuses” have to say.

With such great knowledge flowing in the comments here I am assuming they are all owners of largely scaled websites… no? Oh.
Mick Liubinskas

May 23rd, 2008 at 12:44 am

Nik, are you using a different account now?

The current one shows you haven’t sent a message in 9 months?

http://twitter.com/cubrilovic
pazarlama

May 23rd, 2008 at 2:03 am

Nik, are you using a different account now
hit

May 23rd, 2008 at 2:05 am

Rails would do itself no harm by conceding that it isn’t a platform that can compete with Java or C when it comes to intensive tasks
Paul Stacey

May 23rd, 2008 at 4:43 am

Hi guys,

In my opinion, scaling twitter is not as easy as it would initially seem. The main problem is that the information on a twitter page is all real time and determined from who you know. This is in contrast to Facebook’s news feed, which it is clearly done using bots to aggregate the information into tables that are optimised for a user so that the query would look something like.

SELECT * FROM news_feed WHERE userid=”this_user_id”

Now it is my guess that over 1000 machines running mysql would be used to store this information. Each userid would be mapped to a single machine (and likely backed up on another one or two)

So an initial query would ask which machine the user information is held and then another query would go and retrieve that information after the relevant connection has been made.

This approach was pushed by Brad Fitzpatrick who used this approach at Livejournal. It also allows you to cache the results as objects using memcache as everything is indexed with the userid

Now lets think of the twitter problem.

SELECT * FROM twitter_stream WHERE userid=”friend1″ OR userid=”friend2″ ….” OR userid=”friend5000″

Now this would still be a quick query if all the database was on one machine as userid is indexed. Unfortunately, doing this would require a very fast, very large machine and would be subject to the reliability problems that are associated with such an approach.

In my opinion there are a few solutions that I can think of (and i am by no means an expert!). One would be to increase the number of inserts that are done to replicate the information in all the relevant places when a twitter is added (this is not a good solution as it increases data storage requirements and it is hard to ensure it is done correctly e.g. if you add a new friend and you will need to go through your existing twitter stream and insert all their twitters into the relevant places).

Two would be to have some type of query distribution engine to distribute the initial query to the relevant machines and then reduce the results into the stream. This solution would be a great starting point and wouldn’t be that hard to set up and maintain.

Just some thoughts, let me know what you guys think?
mike

May 23rd, 2008 at 5:44 am

Seems like tech crunch should change their name to twitter crunch you guys must either own twitter or wish you did, not a day goes by without a Twitter article
Livecrunch

May 23rd, 2008 at 7:20 am

@125 Mike: That was funny!

NIK,

Why don’t they say they ware hacked? And still are? And still don’t want to make one of the attackers admin ? Why?
KwangErn

May 23rd, 2008 at 10:20 am

Spot on. It clearly isn’t a simple issue.

Similar technology like BigTable, e.g. Hadoop, may help. It’d be very horrendous to learn that the Twitter team only looks at *SQL, and not DBs like Vertica, or Xapian, etc.

Sure, Erlang can help, but it alone is not enough.

Depending on how they architected their physical infrastructure would play a huge difference as well. Not to mention the details of the server specifications and configuration!

With some years of experience and lots of funding, I’m VERY amazed that they STILL cant’ solve this issue. It’s a HUGE disappointment, embarrasment and disgrace, IMHO.
Marcelo Ruiz

May 23rd, 2008 at 11:11 am

I actually use Twitter for a ’serious’ purpose. I use it to ’save’ my status and instantly remember what I was doing when I get back from a distraction. Believe it or not, it really works! (http://twitter.com/marceloruiz)
Stuart

May 23rd, 2008 at 12:00 pm

Someone please send this link to the twitter guys, this problem was solved over 21 years go. WTF.

http://en.wikipedia.org/wiki/Publish/subscribe

Oh, and Nik Cubrilovic, please stop talking about stuff you don’t understand.
fredrick

May 23rd, 2008 at 4:51 pm

I think TechCrunch officially jumped the shark with this article and subsequent comments.
Automatt

May 23rd, 2008 at 6:15 pm

I made a list where you can rate why twitter is down
jeremy

May 23rd, 2008 at 7:32 pm

did you just compare rails to java and c?
anon

May 23rd, 2008 at 7:49 pm

@22, @123, why use a database at all? Just append messages to each follower’s text file and serve them statically.
Ivan

May 23rd, 2008 at 8:32 pm

Why don’t they use the real stuff for message queuing and delivery, like MQ. Starling WTF? Also, agreed that Ruby is not the language for building platforms (gateways, message systems). It is great for web apps, but lacks real platform tools, libraries and products that one can plug-in. J2EE should be a better choice for back end.
Matt K. (codemonkey)

May 23rd, 2008 at 11:40 pm

They will keep covering it because people keep using it. If twitter is that bad of a service, why don’t people move to a different service? Or why don’t you create your own? Pownce is a great service, but I don’t see it getting the number of users (especially from outside the geek universe) that Twitter receives. It’s just a playground like Peter said (@10). They will find a way to make it scale. In the meantime, enjoy that it’s there in the first place.

[Tech Stuff]
I agree they should maybe offload the heavy commands to C and segment data to avoid BFT (big fuckin tables). No, Rails has nothing to do with the scalability of Twitter (It’s what powers the html rendering, etc.). PHP, Rails, J2EE or not more scalable then the other (in terms of millions of users). They should really better organize their databases to provide maximum speed with minimal effort. This will all come in time, be patient.
Craig Baker

May 24th, 2008 at 12:01 am

Welcome back to the Internet Nik.

The most outrageous architectural solution goes to post 132, ‘why use a database at all? Just append messages to each follower’s text file and serve them statically’
jimmy dean

May 24th, 2008 at 8:05 am

Dear idiots (and, yes, that’s all of you, except the anon of @117)

You’re all the reason software sucks today, but I’m really here to give out my American Idiot awards:

WIN: KwangErn of @126. You’re classic, dude. Hadoop?! Indeed. I’m still laughing.

PLACE: JosefVirek of @45. Do you jerk off to RAC centerfolds?

Show: It was too close to call, so 3rd place goes to the anyone who mentioned anything technical, esp the pub/sub folks. Congrats all.