Swarm: Open Source Web App Scaling - InformationWeek
IoT
IoT
Cloud // Cloud Storage
Commentary
10/12/2009
10:47 AM
Serdar Yegulalp
Serdar Yegulalp
Commentary
Connect Directly
Google+
Twitter
RSS
E-Mail
50%
50%
RELATED EVENTS
Building Security for the IoT
Nov 09, 2017
In this webcast, experts discuss the most effective approaches to securing Internet-enabled system ...Read More>>

Swarm: Open Source Web App Scaling

A new open source project called Swarm bills itself as "a transparently scalable distributed programming language." It's been written to tackle one of the thorniest problems of today's cloud-centric world: How do you create applications that can scale up and out without driving yourself nuts?

A new open source project called Swarm bills itself as "a transparently scalable distributed programming language." It's been written to tackle one of the thorniest problems of today's cloud-centric world: How do you create applications that can scale up and out without driving yourself nuts?

The problem: You have a cloud -- a whole bunch of computers all merrily computing away. You also have a bunch of data strewn across said computers. And finally, you have a computing problem -- a search, for instance -- that you want to run on said data. Problem is, said data is a couple of terabytes, so moving the data between nodes is impractical to say the least. What's to be done?

Swarm's solution to this: "move the computation, not the data". You take the problem to be solved, and implement it in such a way that the problem itself -- including the state of its computation -- can be moved efficiently across multiple nodes. The really heavy lifting for this work is done thanks to the facilities of the Scala programming language, which was devised to help solve problems like this in the first place.

For a sanity check, I talked to my friendly local programmer. To my surprise, he wasn't as excited about it, and that led to a discussion about what's at the real heart of this problem.

He didn't see the issue as being one of this toolkit or that language. He saw the problem as being more that most programmers don't understand parallelism -- how to program for it, how to take advantage of it with existing software, how to make use of languages that exploit it. The real problem, as he saw it, was dealing with the critical-paths issue -- how to break big knotty problems into little coherent subproblems. That falls mainly to the programmer and his skill set, and is not something you can prepackage as a solution. (Analogy: Buying a copy of Photoshop does not automatically make you a better artist.) And he didn't seem confident that Swarm could accelerate the development of such things.

I see where he's coming from. At the same time I figure having that many more tools in the arsenal -- especially shared, open source ones -- will make it possible for programmers to accrue and share those skills in the first place. I don't expect a genie out of a bottle, but I'm keeping my eyes open.

The InformationWeek/bMighty Data Centers For Growing Companies virtual event explores the latest technology and tools you can use to manage your growing IT needs. Oct. 21, 2009. Find out more and register.

Follow me and the rest of InformationWeek on Twitter.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of IT Report
In today's technology-driven world, "innovation" has become a basic expectation. IT leaders are tasked with making technical magic, improving customer experience, and boosting the bottom line -- yet often without any increase to the IT budget. How are organizations striking the balance between new initiatives and cost control? Download our report to learn about the biggest challenges and how savvy IT executives are overcoming them.
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll