Why Sears Is Going All-In On Hadoop - InformationWeek
Data Management // Big Data Analytics
04:40 PM
Connect Directly

Why Sears Is Going All-In On Hadoop

Sears pushes the cutting edge with some big data techniques, while trying to sell its big data services. Can emerging tech drive change in old-school companies?

The MetaScale Mission

Shelley is still CTO of Sears, but if his portrayal of all the things Hadoop can do sounds a bit rosy, keep in mind that he's also now CEO of MetaScale, a division that Sears is hoping will make money from the company's specialized big data expertise.

The rarest commodity that MetaScale offers is Sears' experience in bringing mainframe data into the Hadoop world. Old-school Cobol programmers at Sears were initially Hadoop skeptics, Shelley says, but many turned out to be eager and highly skilled adopters of the Pig language for running MapReduce jobs on Hadoop. Tasks that required 3,000 to 5,000 lines of Cobol can be reproduced with a few hundred lines of Pig, he says. The company learned how to load data from IMS (mainframe) databases into Hadoop and bring result sets back into mainframe apps. That's not trivial work because it involves a variety of compressed data format transformations, and packing and unpacking of data.

MetaScale's business model is to run Hadoop clusters for other companies as a subscription cloud service in Sears' data center. Or Sears will remotely manage clusters in a customer's data center, a setup that two early customers, one in healthcare and the other in financial services, both want for regulatory reasons. Monthly fees are based on the volume of terabytes supported, and customers can buy out deployments if they want to take them over and run them themselves.

MetaScale also offers data architecture, modeling, and management services and consulting. The big idea behind Hadoop is to bring in as much data as possible while keeping data structures simple. "People want to overcomplicate things by representing data and dividing things up into separate files," says Scott LaCosse, director of data management at Sears and MetaScale. "The object is not to save space, it's to eliminate joins, denormalize the data, and put it all in one big file where you can analyze it."

Counterintuitive Tactic

It's an approach that's counterintuitive for a SQL veteran, so a big part of MetaScale's work is to help customers change their thinking: You apply schema as you pull data out to use it, rather than take the relational database approach of imposing a schema on data before it's loaded onto the platform. Hadoop holds data in its raw form, giving users the flexibility to combine and examine the data in many ways over time.

"If in three years you come up with a new query or analysis, it doesn't matter because there's no schema," Shelley says. "You just go get the raw data and transform it into any format you need."

Bricks And Clicks
Sears Holdings includes Sears and Kmart stores (more than 2,600 stores) and Lands' End. This year, it spun off Sears Hometown and Outlet stores.
Digital Ties
Its Shop Your Way loyalty program offers points and promotions.
IT Operations
IT staff in the U.S. and India, with data centers in Chicago and Detroit areas.
Hadoop Startup
MetaScale will run Hadoop clusters--in Sears' data center or remotely--on subscription basis, plus do consulting.

For all of Shelley's boldness about replacing legacy systems, he's careful to describe Hadoop as part of an ecosystem. Sears still uses Teradata and InfoBright, for example, when applications call for fast analysis. But Hadoop is the center of Sears' data management strategy, handling the large-scale heavy lifting, while relational tools take tactical roles.

So where should Hadoop adopters begin?

"You have to go fast and be bold without taking stupid risks," Shelley says. Start with a business need "that causes enough pain that people will notice and they'll see tangible benefits."

Sears itself still has a lot to prove with its own use of Hadoop to solve huge business problems, such as offering customers personalized promotions. Shelley cites plenty of conceptual uses of Hadoop, and he sprinkles in details on speed-and-feed gains, but he doesn't offer clear cases of tangible benefits the retailer has realized. The company is well along in adopting Hadoop and in developing specialized expertise that might benefit MetaScale customers--particularly those using mainframes--but will Hadoop really help turn Sears around?

Sears' latest results for the quarter ended July 28 show that earnings before interest, taxes, depreciation, and amortization were up 163%, to $153 million, from $58 million in the year-earlier quarter. But same-store sales were down 2.9% at Sears and 4.7% at Kmart. Sears' spin is that it's selling fewer items more profitably, which could be in part because of smarter targeting and promotion. But Sears can't shrink its way back to greatness. As Wal-Mart and Target gain share, their buying power and ability to press Sears on margins only grows.

Would-be MetaScale customers in other industries will face different challenges as they consider embracing Hadoop. Could quick analytical access to an entire decade of medical record data change how doctors diagnose and treat patients? Could faster processing spot financial services fraud more effectively? Companies are focused on choosing and building out the next-generation platforms that will handle those big data jobs. Will Hadoop be that platform, and will Hadoop help turn MetaScale into a successful pioneer? That's a story that has yet to unfold.

Go to the sidebar:
Should Sears CTO Be Building A Tech Startup?

Write to Doug Henschen at dhenschen@techweb.com, or on Twitter: @dhenschen. Read more stories by him at informationweek.com/doughenschen.

3 of 3
Comment  | 
Print  | 
More Insights
Threaded  |  Newest First  |  Oldest First
Ellis Booker
Ellis Booker,
User Rank: Moderator
10/31/2012 | 3:07:19 PM
re: Why Sears Is Going All-In On Hadoop
This is a big story on a number of fronts. First, it clearly expresses the value of big data analysis for retailers. As one of the Sears executives puts it, "With Hadoop we can keep everything, which is crucial because we don't want to archive or delete meaningful data." Second, it addresses the oft-heard complaint that big data solutions are prohibitively expensive--in fact, Sears says it reduced mainframe costs by more than $500,000 per year. Finally, the installation moves the retailer closer to real-time analysis: "Sears can develop in three days interactive reports that used to take IT six to 12 weeks." --Ellis Booker, InformationWeek Community Editor
User Rank: Apprentice
10/31/2012 | 7:59:28 PM
re: Why Sears Is Going All-In On Hadoop
The final questions are extremely apropos and often cause the most confusion: "Could quick analytical access to an entire decade of medical record data change how doctors diagnose and treat patients? Could faster processing spot financial services fraud more effectively?" This is not what Hadoop does. It is not an analytics technology, as pointed out in page 1. Extracting this type of valuable insight from the data requires a new class of analytics technologies, and the more powerful the mathematical algorithms, the faster and more accurate the insight.
Soozy G. Miller
Soozy G. Miller,
User Rank: Strategist
12/5/2012 | 3:31:54 PM
re: Why Sears Is Going All-In On Hadoop
@srentner good point. It's great that Sears has a great big data solution; it's quite another to get the analytics to put that big data to use.
Soozy G. Miller
Soozy G. Miller,
User Rank: Strategist
12/5/2012 | 3:34:31 PM
re: Why Sears Is Going All-In On Hadoop
"But will companies be interested in buying big data cloud and consulting services from Sears?"

So, what, is Sears planning on entering a whole new market? Then again, The Gap started by selling music and jeans.
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
Top IT Trends to Watch in Financial Services
IT pros at banks, investment houses, insurance companies, and other financial services organizations are focused on a range of issues, from peer-to-peer lending to cybersecurity to performance, agility, and compliance. It all matters.
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll