What's Excel's Place In Big Data Age?

Vendor hype, helped along by us journalists, sometimes succeeds in obscuring reality. Big data is a case in point -- what's so great about having even more data that tells us nothing?

It was almost painful sitting in a Microsoft conference room Thursday listening to executives say that Excel -- Excel! -- was going to help the corporate masses put the capital letters on Big Data.

"Excel can now manipulate over 100 million rows of data in memory on the computer and give you analysis," we heard from Craig Hodges, the silver-haired head of Microsoft's New England district. "Two years ago you could only process 64,000 rows of data in Excel."

How trivial.

It's in vogue now to invoke the moon landing as the pinnacle of innovation, and wonder what's happened to us since. There is truth to this. But trivial things can matter. Our innovations tend to need lots of small steps to catch on.

[ For more on Microsoft's vision for big data, see Microsoft's Big Data Strategy: An Insider's View. ]

Case in point: Predictive analytics, the great hope represented by big data. "Regression's been around for 200 years and people have been making prediction for a long time," acknowledges Sham Kakade, a senior researcher at Microsoft New England Research. "What is interesting now is the promise of having automated tools that can do this. People are getting broader accessibility to these tools."

One problem is that analytics algorithms require peculiar skillsets not available to most companies. In Microsoft's favor, says Hodges: "We have an analysis tool on your desk so you can build your own algorithm."

It is true that most companies employ somebody who knows a lot about Excel. The question for CIOs is can they use Excel as an analytic engine for the masses?

Microsoft, ever agile with its public relations, offered up a customer in its TAP (Technology Adoption Partner) program: Beth Israel Deaconess Medical Center. John Halamka, CIO at BIDMC, needs a way to get a handle on unstructured data -- that is, data that isn't defined in a database or other structure that a computer can easily understand. For healthcare, unstructured data ranges from the obvious (doctor's notes about patients) to the arcane (audit logs for servers, needed for HIPAA compliance). Halamka's been thinking about the problem for a few years, as this blog post from 2011 shows.

Halamka is no stranger to those looking for innovation in healthcare IT. Under Halamka's technology leadership, BIDMC was named No. 1 among the InformationWeek 500 in 2012. See our related article, Beth Israel Deaconess Medical Center Embraces Analytics.

Halamka told me that big data was one of his five top priorities for this year, and pointed me to his point person on analytics, Ayad Shammout, principal business intelligence consultant at BIDMC.

Shammout has been working with Microsoft's version of Apache Hadoop, a popular analytics tool, since January 2012, months before Microsoft announced its support for Hadoop.

He says the tool is "neat" and has allowed BIDMC to analyze its SQL server transaction logs far more effectively, in part because it has excellent compression tools that can take, say, a 22 GB log file and compress it down to 120 MB. That's important because BIDMC can generate terabytes of SQL transaction data in just days, creating huge needs for storage and putting pressure on the network.

Global CIOs: A Site Just For You Visit InformationWeek's Global CIO -- our online community and information resource for CIOs operating in the global economy.

Shammout says BIDMC is building predictive analytical models in Excel. The project on audits is about 70% to 80% complete, and then he will move towards applying analytics to structured and unstructured patient-oriented data.

He says the combination of Excel and Hadoop on Microsoft has saved BIDMC's development team countless hours in developing models and will sharply reduce the learning curve for BIDMC employees as they start to integrate structured and unstructured data. "We have a lot of Excel users," he says.

The question is how many CIOs still have a cadre of advanced Excel users available to them? Rob Enderle, head of the market researcher Enderle Group, thinks Excel skills have languished at many companies. He says custom tools aimed at analytics, or cloud analytics services will prove better suited to the needs of most CIOs than Microsoft's integrated tools, with Excel as the front end.

Has Excel become an afterthought in corporations? For Microsoft, the answer is not trivial.

Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. Register today!