Cloud // Cloud Storage
Commentary
10/12/2009
10:47 AM
Serdar Yegulalp
Serdar Yegulalp
Commentary
Connect Directly
Google+
Twitter
RSS
E-Mail
50%
50%

Swarm: Open Source Web App Scaling

A new open source project called Swarm bills itself as "a transparently scalable distributed programming language." It's been written to tackle one of the thorniest problems of today's cloud-centric world: How do you create applications that can scale up and out without driving yourself nuts?

A new open source project called Swarm bills itself as "a transparently scalable distributed programming language." It's been written to tackle one of the thorniest problems of today's cloud-centric world: How do you create applications that can scale up and out without driving yourself nuts?

The problem: You have a cloud -- a whole bunch of computers all merrily computing away. You also have a bunch of data strewn across said computers. And finally, you have a computing problem -- a search, for instance -- that you want to run on said data. Problem is, said data is a couple of terabytes, so moving the data between nodes is impractical to say the least. What's to be done?

Swarm's solution to this: "move the computation, not the data". You take the problem to be solved, and implement it in such a way that the problem itself -- including the state of its computation -- can be moved efficiently across multiple nodes. The really heavy lifting for this work is done thanks to the facilities of the Scala programming language, which was devised to help solve problems like this in the first place.

For a sanity check, I talked to my friendly local programmer. To my surprise, he wasn't as excited about it, and that led to a discussion about what's at the real heart of this problem.

He didn't see the issue as being one of this toolkit or that language. He saw the problem as being more that most programmers don't understand parallelism -- how to program for it, how to take advantage of it with existing software, how to make use of languages that exploit it. The real problem, as he saw it, was dealing with the critical-paths issue -- how to break big knotty problems into little coherent subproblems. That falls mainly to the programmer and his skill set, and is not something you can prepackage as a solution. (Analogy: Buying a copy of Photoshop does not automatically make you a better artist.) And he didn't seem confident that Swarm could accelerate the development of such things.

I see where he's coming from. At the same time I figure having that many more tools in the arsenal -- especially shared, open source ones -- will make it possible for programmers to accrue and share those skills in the first place. I don't expect a genie out of a bottle, but I'm keeping my eyes open.

The InformationWeek/bMighty Data Centers For Growing Companies virtual event explores the latest technology and tools you can use to manage your growing IT needs. Oct. 21, 2009. Find out more and register.

Follow me and the rest of InformationWeek on Twitter.

Comment  | 
Print  | 
More Insights
Google in the Enterprise Survey
Google in the Enterprise Survey
There's no doubt Google has made headway into businesses: Just 28 percent discourage or ban use of its productivity ­products, and 69 percent cite Google Apps' good or excellent ­mobility. But progress could still stall: 59 percent of nonusers ­distrust the security of Google's cloud. Its data privacy is an open question, and 37 percent worry about integration.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest September 18, 2014
Enterprise social network success starts and ends with integration. Here's how to finally make collaboration click.
Flash Poll
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.