Swarm: Open Source Web App ScalingSwarm: Open Source Web App Scaling

A new open source project called Swarm bills itself as "a transparently scalable distributed programming language." It's been written to tackle one of the thorniest problems of today's cloud-centric world: How do you create applications that can scale up and out without driving yourself nuts?

The problem: You have a cloud -- a whole bunch of computers all merrily computing away. You also have a bunch of data strewn across said computers. And finally, you have a computing problem -- a search, for instance -- that you want to run on said data. Problem is, said data is a couple of terabytes, so moving the data between nodes is impractical to say the least. What's to be done?

Swarm's solution to this: "move the computation, not the data". You take the problem to be solved, and implement it in such a way that the problem itself -- including the state of its computation -- can be moved efficiently across multiple nodes. The really heavy lifting for this work is done thanks to the facilities of the Scala programming language, which was devised to help solve problems like this in the first place.

For a sanity check, I talked to my friendly local programmer. To my surprise, he wasn't as excited about it, and that led to a discussion about what's at the real heart of this problem.

He didn't see the issue as being one of this toolkit or that language. He saw the problem as being more that most programmers don't understand parallelism -- how to program for it, how to take advantage of it with existing software, how to make use of languages that exploit it. The real problem, as he saw it, was dealing with the critical-paths issue -- how to break big knotty problems into little coherent subproblems. That falls mainly to the programmer and his skill set, and is not something you can prepackage as a solution. (Analogy: Buying a copy of Photoshop does not automatically make you a better artist.) And he didn't seem confident that Swarm could accelerate the development of such things.

I see where he's coming from. At the same time I figure having that many more tools in the arsenal -- especially shared, open source ones -- will make it possible for programmers to accrue and share those skills in the first place. I don't expect a genie out of a bottle, but I'm keeping my eyes open.

The information/bMighty Data Centers For Growing Companies virtual event explores the latest technology and tools you can use to manage your growing IT needs. Oct. 21, 2009. Find out more and register.

Follow me and the rest of information on Twitter.