Software // Information Management
Commentary
11/9/2011
10:00 AM
Doug Henschen
Doug Henschen
Commentary
Connect Directly
LinkedIn
Twitter
Google+
RSS
E-Mail
50%
50%

IT's Next Hot Job: Hadoop Guru

JPMorgan Chase makes a case for the big data platform (and career track) of the future.

"We're hiring, and we're paying 10% more than the other guys."

Those were the first words from Larry Feinsmith, managing director, office of the CIO, at JPMorgan Chase, in his Tuesday keynote address at Hadoop World in New York. Who JPMorgan Chase is hiring, specifically, are people with Hadoop skills, so Feinsmith was in the right place. More than 1,400 people were in the audience, and attendee polls indicated that at least three quarters of their organizations are already using Hadoop, the open source big data platform.

The "and we're paying 10% more" bit was actually Feinsmith's ad-libbed follow-on to the previous keynoter, Hugh Williams, VP of search, experience, and platforms at eBay. After explaining eBay's Hadoop-based Cassini search engine project, Williams said his company is hiring Hadoop experts to help build out and run the tool.

Feinsmith's core message was that Hadoop is hugely promising, maturing quickly, and might overlap the functionality of relational databases over the next three years. In fact, Hadoop World 2011 was a coming-out party of sorts, as it's now clear that Hadoop will matter to more than just Web 2.0 companies like eBay, Facebook, Yahoo, AOL, and Twitter. A straight-laced financial giant with more than 245,000 employees, 24 million checking accounts, 5,500 branches, and 145 million credit cards in use, JPMorgan Chase lends huge credibility to that vision.

[ Want more on meeting high-volume data challenges? Read Hadoop Spurs Big Data Revolution. ]

JP Morgan Chase has 25,000 IT employees, and it spends about $8 billion on IT each year--$4 billion on apps and $4 billion on infrastructure. The company has been working with Hadoop for more than three years, and it's easy to see why. It has 150 petabytes (with a "p") of data online, generated by trading operations, banking activities, credit card transactions, and some 3.5 billion logins each year to online banking and brokerage accounts.

The benefits of Hadoop? Massive scalability, schema-free flexibility to handle a variety of data types, and low cost. Hadoop systems built on commodity hardware now cost about $4,000 per node, according to Cloudera, the Hadoop enterprise support and management software provider (and the organizer and host of Hadoop World). With the latest nodes typically having 16 compute cores and 12 1-terabyte or 2-terabyte drives, that's massive storage and compute capacity at a very low cost. In comparison, aggressively priced relational data warehouse appliances cost about $10,000 to $12,000 per terabyte.

Cloudera's another company that's hiring to provide service and Hadoop systems management capabilities to its fast-growing customer base. "Hadoop's a big deal," said Cloudera CEO Mike Olson. "It's not just a Web thing. Companies across a wide range of vertical markets are generating big data and need to understand that data in a way they never did before."

Previous
1 of 2
Next
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
HM
50%
50%
HM,
User Rank: Strategist
11/9/2011 | 5:31:25 PM
re: IT's Next Hot Job: Hadoop Guru
Great article Doug! However, instead of using conventional relational databases as mentioned by JPMorgan, might I suggest Roxie, the HPCC Systems massive data delivery engine, which handles real-time query processing. Roxie can deliver query responses in sub-second predictable latencies to thousands of concurrent users depending on the size of the cluster and the complexity of the queries. Great for when there's lots of query complexity or concurrent workloads!
More at: http://hpccsystems.com
The Agile Archive
The Agile Archive
When it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
Register for InformationWeek Newsletters
White Papers
Current Issue
InformationWeek Tech Digest September 24, 2014
Start improving branch office support by tapping public and private cloud resources to boost performance, increase worker productivity, and cut costs.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.