Advanced Analytics: Teradata's CTO on Breakthrough Insight

Interest in advanced analytics is a good thing, says Stephen Brobst, CTO of data warehousing vendor Teradata. But don't expect much value from prebuilt analytic applications and simplistic spreadsheet analyses, he warns. To gain broader access to truly valuable insight, Brobst says we have to move away from walled-off warehouses and programming-driven analysis tools.

Intelligent Enterprise: There's growing interest in advanced analytics, but the shortage and expense of expertise has led many vendors to offering prepackaged industry solutions and applications. What are your thoughts on prebuilt analytic applications?

Stephen Brobst: Industry data models absolutely make sense. Customers and vendors are both moving in that direction because you don't need to build an industry data model from scratch. You can also prepackage analytics -- things like reports, key performance indicators and balanced scorecards -- but that's not where the value is. If look at truly innovative companies, let's take eBay as an example, 85 % of the resources they use for their data warehouse are for answering new questions. The answers to questions you already know are cheap because you can prepackage them and optimize the queries. You may have to keep running those report for regulatory purposes or just to keep your fingers on the pulse of the business, but once you run them a few times there are no new insight.

In my view, you want a set of tools that will let you ask questions you haven't asked before yet get answers efficiently without having preoptimized the query. If something is preoptimized, that inherently means that you already knew the question and the probable answers to the question.

Intelligent Enterprise: That assumes you have analytic experts available to ask new questions. Hasn't the scarcity and cost of expertise limited the use of advanced analytics?

Brobst: I think there's a cultural change going on whereby the MBAs coming up from the top universities are more quantitatively capable than they used to be. They are comfortable using analytic tools. The problem is that the tools they learn tend to be Excel and variations of Excel. It's better than the old style of qualitative decision making; they are data driven, but they don't really know how to do deep analytics. It's sort of spread-mart analytics, not tools that would enable you to do things like predictive forecasting beyond a linear regression you could do in Excel.

Excel is the most common BI tool in the world, but there are all kinds of governance challenges with spreadsheets. And though spreadsheets may be easy to use, they're not an example of best-in-class analytic capabilities. We have to raise the bar a little. The level of analytic sophistication is rising, but we have to get people out of the mindset that a spreadsheet is the answer. Intelligent Enterprise: I can't imaging practitioners are clamoring for more sophisticated tools? What's the biggest complaint you hear in the field?

Brobst: The biggest complaint I hear is that the people who own the data warehouse are often like prison guards who won't let other people into the system. There's all this data available, but the people who own the warehouse are often incented based on the stability and availability of that environment. That means they don't want the power users to touch their sacred databases. There's a culture change that has to take place whereby the power users with some level of training should be able to bring in their own data and investigate.

Let's say I'm a marketing person and I have some survey data from a third-party source and I want to use that with company data, but the DBA says I can't. What do I do instead of loading my small amount of data and using it with production data? I do an extract of the production data and create a data mart somewhere else. That's completely inefficient. We have to change the prison-guard culture and give sandbox environments to power users to allow them to be successful with the data warehouse environment. At the same time, you don't want them to have to pay a huge price to model and specify a new resource. What you want is load-and-go, self-provisioning of data in an analytic cloud environment.

Intelligent Enterprise: You're obviously alluding to Teradata's Elastic Mart Builder capability (for VMWare and Amazon EC2) , right?

Brobst: Well, yes, but the reason I bring it up is that the companies that are very successful, and I'll use eBay again as an example, will facilitate that kind of exploration. They will have their DBAs and solution architects work with the business people to help them be successful. Instead of being jail keepers, they are now the facilitators. At eBay, they were resistant to this approach at first, but then they changed their mind and decided [the instinct to explore data] was a good thing.

The funny thing is that when business people work with the DBAs, the DBAs start out thinking they are training the business people how to use the technology. In fact, the DBAs are learning a lot about how to use data from a business perspective. It's a two-way mentoring process, and business ends up stealing the really good DBAs that learn enough about the business. That's what was happening at eBay, but it turned out to be a good problem. It's like getting your guys behind enemy lines -- not that there should be such lines drawn between business and IT, but there often are. With this blending, you start building a culture that understands why it's important to do proper data management and you start promoting a data-oriented mentality within the business.

Intelligent Enterprise: Business-savvy DBAs are certainly desirable, but it's the deeper analytic wonks that are harder to find. Do you see any way around this skills shortage?

Brobst: The tools we make available should be easier to use. You shouldn't have to be a PhD in statistics or mathematics to do advanced modeling. The tools should be embedded in an analytic environment that doesn't require you to be a programmer. The reality is that a lot of the advanced data mining tools require programming. SAS code, for example, is not C or Java code, but it's still programming. We need to make it easier. The SAS Enterprise Miner tool is a step in that direction. There's an interesting cultural shift taking place in the SAS community because people who use Enterprise Miner are considered by some to be sissies -- it's like data mining with training wheels. In the old days of data warehousing, the attitude was that real users knew SQL; who needed these GUI interfaces? But now we accept that writing SQL is not a good way to drive analytics throughout an organization. The same thing has to happen in data mining for more advanced analytics. Intelligent Enterprise: What other hot topics are you hearing about from customers these days?

Brobst: Use of non-traditional data types is becoming a big issue. People want to get information out of Facebook, call logs and semi-structured data. There are privacy concerns in this area, but if you can get somebody to become a fan or a friend, for example, a whole new source of data is available to you.

I was at a concert recently and a retail chain was giving out a CD of that artist if you would become a fan of the store on Facebook. Most people don't realize this, but as soon as you do that, you've let them into your network. Now they have access to all your data, and most people with Facebook accounts will list their age, gender, marital status, education, geography -- way more information than you'd ever get through traditional market research. Companies are doing social-network analysis to figure out whether you are an influencer or a follower. If you can get people to become your friend, you can pay Facebook for access to the APIs and you can suck all that data out. If you then do sentiment analysis on that data, you can translate that data into something structured that you can analyze and use as the basis for making decisions.

Intelligent Enterprise: The trend toward sentiment analysis is clearly catching on, but how broadly will it be adopted?

Brobst: I'll give you some facts. One third of people that blog in some form -- and Facebook is essentially a personal blog -- comment on products or brands at some point. More than half of those who do comment on products or brands will talk about, for example, the quality of their mobile phone service. This sort of data is out there, and it's highly valuable. If you go on Facebook, you'll see that people comment that they are angry about an airline or they may write about the great service experience they had.

Big companies used to do focus groups and conventional market research surveys to get at this kind of insight, but that style of research is becoming less relevant. When my sister wants to be heard, she shouts on Facebook, and she is heard by all of her friends. There are all sorts of interesting analytics being developed to exploit these new data sources.