Swarm: Open Source Web App Scaling - InformationWeek
IoT
IoT
Cloud // Cloud Storage
Commentary
10/12/2009
10:47 AM
Serdar Yegulalp
Serdar Yegulalp
Commentary
Connect Directly
Google+
Twitter
RSS
E-Mail
50%
50%
RELATED EVENTS
Faster, More Effective Response With Threat Intelligence & Orchestration Playboo
Aug 31, 2017
Finding ways to increase speed, accuracy, and efficiency when responding to threats should be the ...Read More>>

Swarm: Open Source Web App Scaling

A new open source project called Swarm bills itself as "a transparently scalable distributed programming language." It's been written to tackle one of the thorniest problems of today's cloud-centric world: How do you create applications that can scale up and out without driving yourself nuts?

A new open source project called Swarm bills itself as "a transparently scalable distributed programming language." It's been written to tackle one of the thorniest problems of today's cloud-centric world: How do you create applications that can scale up and out without driving yourself nuts?

The problem: You have a cloud -- a whole bunch of computers all merrily computing away. You also have a bunch of data strewn across said computers. And finally, you have a computing problem -- a search, for instance -- that you want to run on said data. Problem is, said data is a couple of terabytes, so moving the data between nodes is impractical to say the least. What's to be done?

Swarm's solution to this: "move the computation, not the data". You take the problem to be solved, and implement it in such a way that the problem itself -- including the state of its computation -- can be moved efficiently across multiple nodes. The really heavy lifting for this work is done thanks to the facilities of the Scala programming language, which was devised to help solve problems like this in the first place.

For a sanity check, I talked to my friendly local programmer. To my surprise, he wasn't as excited about it, and that led to a discussion about what's at the real heart of this problem.

He didn't see the issue as being one of this toolkit or that language. He saw the problem as being more that most programmers don't understand parallelism -- how to program for it, how to take advantage of it with existing software, how to make use of languages that exploit it. The real problem, as he saw it, was dealing with the critical-paths issue -- how to break big knotty problems into little coherent subproblems. That falls mainly to the programmer and his skill set, and is not something you can prepackage as a solution. (Analogy: Buying a copy of Photoshop does not automatically make you a better artist.) And he didn't seem confident that Swarm could accelerate the development of such things.

I see where he's coming from. At the same time I figure having that many more tools in the arsenal -- especially shared, open source ones -- will make it possible for programmers to accrue and share those skills in the first place. I don't expect a genie out of a bottle, but I'm keeping my eyes open.

The InformationWeek/bMighty Data Centers For Growing Companies virtual event explores the latest technology and tools you can use to manage your growing IT needs. Oct. 21, 2009. Find out more and register.

Follow me and the rest of InformationWeek on Twitter.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
IT Strategies to Conquer the Cloud
Chances are your organization is adopting cloud computing in one way or another -- or in multiple ways. Understanding the skills you need and how cloud affects IT operations and networking will help you adapt.
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll