IT's Next Hot Job: Hadoop Guru - InformationWeek
Software // Information Management
10:00 AM
Doug Henschen
Doug Henschen
Connect Directly

IT's Next Hot Job: Hadoop Guru

JPMorgan Chase makes a case for the big data platform (and career track) of the future.

"We're hiring, and we're paying 10% more than the other guys."

Those were the first words from Larry Feinsmith, managing director, office of the CIO, at JPMorgan Chase, in his Tuesday keynote address at Hadoop World in New York. Who JPMorgan Chase is hiring, specifically, are people with Hadoop skills, so Feinsmith was in the right place. More than 1,400 people were in the audience, and attendee polls indicated that at least three quarters of their organizations are already using Hadoop, the open source big data platform.

The "and we're paying 10% more" bit was actually Feinsmith's ad-libbed follow-on to the previous keynoter, Hugh Williams, VP of search, experience, and platforms at eBay. After explaining eBay's Hadoop-based Cassini search engine project, Williams said his company is hiring Hadoop experts to help build out and run the tool.

Feinsmith's core message was that Hadoop is hugely promising, maturing quickly, and might overlap the functionality of relational databases over the next three years. In fact, Hadoop World 2011 was a coming-out party of sorts, as it's now clear that Hadoop will matter to more than just Web 2.0 companies like eBay, Facebook, Yahoo, AOL, and Twitter. A straight-laced financial giant with more than 245,000 employees, 24 million checking accounts, 5,500 branches, and 145 million credit cards in use, JPMorgan Chase lends huge credibility to that vision.

[ Want more on meeting high-volume data challenges? Read Hadoop Spurs Big Data Revolution. ]

JP Morgan Chase has 25,000 IT employees, and it spends about $8 billion on IT each year--$4 billion on apps and $4 billion on infrastructure. The company has been working with Hadoop for more than three years, and it's easy to see why. It has 150 petabytes (with a "p") of data online, generated by trading operations, banking activities, credit card transactions, and some 3.5 billion logins each year to online banking and brokerage accounts.

The benefits of Hadoop? Massive scalability, schema-free flexibility to handle a variety of data types, and low cost. Hadoop systems built on commodity hardware now cost about $4,000 per node, according to Cloudera, the Hadoop enterprise support and management software provider (and the organizer and host of Hadoop World). With the latest nodes typically having 16 compute cores and 12 1-terabyte or 2-terabyte drives, that's massive storage and compute capacity at a very low cost. In comparison, aggressively priced relational data warehouse appliances cost about $10,000 to $12,000 per terabyte.

Cloudera's another company that's hiring to provide service and Hadoop systems management capabilities to its fast-growing customer base. "Hadoop's a big deal," said Cloudera CEO Mike Olson. "It's not just a Web thing. Companies across a wide range of vertical markets are generating big data and need to understand that data in a way they never did before."

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
1 of 2
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
User Rank: Moderator
11/9/2011 | 5:31:25 PM
re: IT's Next Hot Job: Hadoop Guru
Great article Doug! However, instead of using conventional relational databases as mentioned by JPMorgan, might I suggest Roxie, the HPCC Systems massive data delivery engine, which handles real-time query processing. Roxie can deliver query responses in sub-second predictable latencies to thousands of concurrent users depending on the size of the cluster and the complexity of the queries. Great for when there's lots of query complexity or concurrent workloads!
More at:
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends for 2018
As we enter a new year of technology planning, find out about the hot technologies organizations are using to advance their businesses and where the experts say IT is heading.
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll