Our 2013 Big Data Survey shows we're not lacking facts, figures, or tools to wrangle them. So why do just 9% of respondents rate themselves as extremely effective users of data?
Lie 5: We know what data we need
In our survey, we asked about 10 internal and nine external data types. Internal sources include financial accounting applications, detailed sales and product data, CRM data, unstructured network data such as Office files and images, and unstructured data stored on end user devices. External sources include government statistics and other public records, geolocation data, data collected from sensors on company products and services, social network data (Facebook, Twitter), and unstructured data stored in the cloud (Office365, Google Docs).
Clearly, there's a lot of information out there. But when we asked who's driving data analysis ideas, we were surprised to find that only 5% of respondents have a centralized team to drive big data strategy; an additional 3% use a looser collaborative effort.
We're not the biggest fans of committees, but given the fact that the users of your data are likely spread far and wide, it makes sense to create a cross-functional group to identify new sources or elevate the importance of an existing stream. It's staggering to see some of the great data that's all but untouched.
Take CRM, phone, email, and Web analytics. These four data points cover most of the communications relationship with your clients. Tying them together isn't rocket science, especially if you have decent baseline customer data to start with. Not only can you determine the number of conversations your company typically has with customers, but you can also understand how email relates to phone calls and Web traffic. If you have an outside sales force looping in, your CRM data gives you a profiling capability to model everything from product rollouts to customer service problems.
All that intelligence exists today, yet few companies have this level of analysis integrated into their big data strategies. While 35% of survey respondents say their IT organizations include CRM in their integrated plans, only 29% include email, 22% Web analytics, and 14% phone logs.
Lie 6: We do something with our analysis
There's nothing more frustrating for an analyst than to work for days or weeks on a project, present the findings, have a great meeting with execs, then watch those recommendations die on the vine. Everyone focuses on the positive aspects of data analysis--helping find new customers or discover more productive logistics routes. But the reality is that big data analysis will find some negative things--about your sales team's effectiveness, your online presence, your true costs of operations. The slow economy of the last four years has weakened multiple parts of most companies. Adding data sources and a more holistic analysis will help find and prioritize the problems you need to fix.
IT Truth Tellers
Want to raise IT's profile as a business enabler? Step in and assume responsibility for data quality across the company. Here's a quick check of items IT should review today:
1. Is there a centralized data quality team? If not, set one up ASAP.
2. Does the team do regular or, at minimum, spot audits of various analyses? Does it regularly look to add new data sources?
3. Are critical external events annotated within your data warehouse or as part of your reporting process? For example, think about major system upgrades that would change the underlying data related to order flow.
4. Do you require statistical notes, including sampling statements?
5. When it comes to customer or vendor surveys, are sample sizes validated against your base customers or total market size to ensure accuracy?
6. Do you run regular "stress tests" for current data sets with cross-functional teams, challenging assumptions and sacred cows?
7. Do you look outward? Most respondents, 75%, report some public cloud use. Yet many companies aren't capturing associated data--think WebEx conferencing for customer behavior analysis or Google analytics for sales tracking.
The Agile ArchiveWhen it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
2014 Analytics, BI, and Information Management SurveyITís tried for years to simplify data analytics and business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.