Why Sears Is Going All-In On Hadoop

The MetaScale Mission

Shelley is still CTO of Sears, but if his portrayal of all the things Hadoop can do sounds a bit rosy, keep in mind that he's also now CEO of MetaScale, a division that Sears is hoping will make money from the company's specialized big data expertise.

The rarest commodity that MetaScale offers is Sears' experience in bringing mainframe data into the Hadoop world. Old-school Cobol programmers at Sears were initially Hadoop skeptics, Shelley says, but many turned out to be eager and highly skilled adopters of the Pig language for running MapReduce jobs on Hadoop. Tasks that required 3,000 to 5,000 lines of Cobol can be reproduced with a few hundred lines of Pig, he says. The company learned how to load data from IMS (mainframe) databases into Hadoop and bring result sets back into mainframe apps. That's not trivial work because it involves a variety of compressed data format transformations, and packing and unpacking of data.

MetaScale's business model is to run Hadoop clusters for other companies as a subscription cloud service in Sears' data center. Or Sears will remotely manage clusters in a customer's data center, a setup that two early customers, one in healthcare and the other in financial services, both want for regulatory reasons. Monthly fees are based on the volume of terabytes supported, and customers can buy out deployments if they want to take them over and run them themselves.

MetaScale also offers data architecture, modeling, and management services and consulting. The big idea behind Hadoop is to bring in as much data as possible while keeping data structures simple. "People want to overcomplicate things by representing data and dividing things up into separate files," says Scott LaCosse, director of data management at Sears and MetaScale. "The object is not to save space, it's to eliminate joins, denormalize the data, and put it all in one big file where you can analyze it."

Counterintuitive Tactic

It's an approach that's counterintuitive for a SQL veteran, so a big part of MetaScale's work is to help customers change their thinking: You apply schema as you pull data out to use it, rather than take the relational database approach of imposing a schema on data before it's loaded onto the platform. Hadoop holds data in its raw form, giving users the flexibility to combine and examine the data in many ways over time.

"If in three years you come up with a new query or analysis, it doesn't matter because there's no schema," Shelley says. "You just go get the raw data and transform it into any format you need."

For all of Shelley's boldness about replacing legacy systems, he's careful to describe Hadoop as part of an ecosystem. Sears still uses Teradata and InfoBright, for example, when applications call for fast analysis. But Hadoop is the center of Sears' data management strategy, handling the large-scale heavy lifting, while relational tools take tactical roles.

So where should Hadoop adopters begin?

"You have to go fast and be bold without taking stupid risks," Shelley says. Start with a business need "that causes enough pain that people will notice and they'll see tangible benefits."

Sears itself still has a lot to prove with its own use of Hadoop to solve huge business problems, such as offering customers personalized promotions. Shelley cites plenty of conceptual uses of Hadoop, and he sprinkles in details on speed-and-feed gains, but he doesn't offer clear cases of tangible benefits the retailer has realized. The company is well along in adopting Hadoop and in developing specialized expertise that might benefit MetaScale customers--particularly those using mainframes--but will Hadoop really help turn Sears around?

Sears' latest results for the quarter ended July 28 show that earnings before interest, taxes, depreciation, and amortization were up 163%, to $153 million, from $58 million in the year-earlier quarter. But same-store sales were down 2.9% at Sears and 4.7% at Kmart. Sears' spin is that it's selling fewer items more profitably, which could be in part because of smarter targeting and promotion. But Sears can't shrink its way back to greatness. As Wal-Mart and Target gain share, their buying power and ability to press Sears on margins only grows.

Would-be MetaScale customers in other industries will face different challenges as they consider embracing Hadoop. Could quick analytical access to an entire decade of medical record data change how doctors diagnose and treat patients? Could faster processing spot financial services fraud more effectively? Companies are focused on choosing and building out the next-generation platforms that will handle those big data jobs. Will Hadoop be that platform, and will Hadoop help turn MetaScale into a successful pioneer? That's a story that has yet to unfold.

Write to Doug Henschen at dhenschen@techweb.com, or on Twitter: @dhenschen. Read more stories by him at informationweek.com/doughenschen.

Download a free PDF of InformationWeek magazine
(registration required)