Microsoft has its own opinion. But do CIOs want to make Excel the analytics engine for the masses?
Vendor hype, helped along by us journalists, sometimes succeeds in obscuring reality. Big data is a case in point -- what's so great about having even more data that tells us nothing?
It was almost painful sitting in a Microsoft conference room Thursday listening to executives say that Excel -- Excel! -- was going to help the corporate masses put the capital letters on Big Data.
"Excel can now manipulate over 100 million rows of data in memory on the computer and give you analysis," we heard from Craig Hodges, the silver-haired head of Microsoft's New England district. "Two years ago you could only process 64,000 rows of data in Excel."
It's in vogue now to invoke the moon landing as the pinnacle of innovation, and wonder what's happened to us since. There is truth to this. But trivial things can matter. Our innovations tend to need lots of small steps to catch on.
Case in point: Predictive analytics, the great hope represented by big data. "Regression's been around for 200 years and people have been making prediction for a long time," acknowledges Sham Kakade, a senior researcher at Microsoft New England Research. "What is interesting now is the promise of having automated tools that can do this. People are getting broader accessibility to these tools."
One problem is that analytics algorithms require peculiar skillsets not available to most companies. In Microsoft's favor, says Hodges: "We have an analysis tool on your desk so you can build your own algorithm."
It is true that most companies employ somebody who knows a lot about Excel. The question for CIOs is can they use Excel as an analytic engine for the masses?
Microsoft, ever agile with its public relations, offered up a customer in its TAP (Technology Adoption Partner) program: Beth Israel Deaconess Medical Center. John Halamka, CIO at BIDMC, needs a way to get a handle on unstructured data -- that is, data that isn't defined in a database or other structure that a computer can easily understand. For healthcare, unstructured data ranges from the obvious (doctor's notes about patients) to the arcane (audit logs for servers, needed for HIPAA compliance). Halamka's been thinking about the problem for a few years, as this blog post from 2011 shows.
He says the tool is "neat" and has allowed BIDMC to analyze its SQL server transaction logs far more effectively, in part because it has excellent compression tools that can take, say, a 22 GB log file and compress it down to 120 MB. That's important because BIDMC can generate terabytes of SQL transaction data in just days, creating huge needs for storage and putting pressure on the network.
Shammout says BIDMC is building predictive analytical models in Excel. The project on audits is about 70% to 80% complete, and then he will move towards applying analytics to structured and unstructured patient-oriented data.
He says the combination of Excel and Hadoop on Microsoft has saved BIDMC's development team countless hours in developing models and will sharply reduce the learning curve for BIDMC employees as they start to integrate structured and unstructured data. "We have a lot of Excel users," he says.
The question is how many CIOs still have a cadre of advanced Excel users available to them? Rob Enderle, head of the market researcher Enderle Group, thinks Excel skills have languished at many companies. He says custom tools aimed at analytics, or cloud analytics services will prove better suited to the needs of most CIOs than Microsoft's integrated tools, with Excel as the front end.
Has Excel become an afterthought in corporations? For Microsoft, the answer is not trivial.
Attend Interop Las Vegas May 6-10 and learn the emerging trends in information risk management and security. Use Priority Code MPIWK by March 22 to save an additional $200 off the early bird discount on All Access and Conference Passes. Join us in Las Vegas for access to 125+ workshops and conference classes, 300+ exhibiting companies, and the latest technology. Register today!
6 Tools to Protect Big DataMost IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Big Data Brings Big Security ProblemsWhy should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.