2018 has been an important year for businesses across industries as they continue to digitize more of their operations. To do that well, they need good, quality data to improve the accuracy of their analytics and to train artificial intelligence (AI) applications.
Many organizations have been working hard to liberate data trapped in systems because they need access the datasets. While the problem has not been solved completely, companies continue to unearth new insights that enable them to better control costs, create new revenue streams and increase profitability.
According to MicroStrategy's 2018 Global State of Analytics Report, which surveyed 500 analytics and business intelligence professionals, 57% have achieved faster and more efficient decision making through data and analytics use. Nearly two-thirds (64%) plan to invest more in data and analytics talent going into the 2020s, and almost three in five (57%) already employ a chief data officer. The CDO role (or its equivalent) is now considered essential for overseeing everything associated with data, ranging from knowledge of data assets and their use to articulating a strategy for using data to achieve competitive differentiation.
Data privacy and security concerns affect use
The journey to realize a more intelligent enterprise is not without its challenges, however. This year, many enterprises had to scramble to ensure compliance with the European Union's General Data Protection Regulation (GDPR) and are now facing the additional overhead that will be associated with the California Consumer Privacy Act, slated to go into effect in 2020. In the MicroStrategy survey, 49% said data privacy and security concerns are holding their organization back from using data more effectively. In addition, 45% reported that less than half of their organization's data is governed, a sobering statistic. The lack of solid governance exposes enterprises to potential privacy and security risks that need to be managed more effectively moving forward.
Organizations are under tremendous pressure to understand the data assets they have and have access to, because without awareness of the inventory, the condition of data assets, data ownership, and the rules restricting use, enterprises have a difficult time driving maximum value from their data and managing the potential risks and liabilities.
AI requires a mindset shift
Meanwhile, there is a major push to supplement data and analytics capabilities with AI. More vendors are touting their latest generation tools that may or may not actually take advantage of AI, and more specifically, machine learning and deep learning. Sadly, enterprise buyers may be unable to discern the difference between some vendors' dubious claims and the actual capabilities of offerings because they lack a basic understanding of AI-related concepts.
In addition, there are risks associated with AI-powered systems that business and IT leaders need to consider, such as the bias that may be present in the datasets used to train those systems and -- in the case of deep learning -- the potential inability for the system to explain the result and the reasoning that went into the result. In addition, Gartner estimates that by 2020, 50% of new data transformation flows will integrate one or more machine learning algorithms resulting in erroneous interpretations of data.
Data quality and metadata matter
Data quality is hardly a new concept, but the need for it is vital, not only to ensure more accurate analytics, but also to ensure that AI training data is reliable. Enterprises need to place greater emphasis on data quality out of necessity or suffer the consequences since "garbage in, garbage out" can erode brand reputation and customer trust, or worse. AI requires lots of training data. If the data used to train the system is of poor quality, it negatively affects the reliability of the insights and recommendations the system renders.
Given the sheer volume of data, increasing regulation and the data-intensive competitive environment, organizations need metadata management tools to enable effective data governance, data management and regulatory compliance. Metadata management is essential for understanding data and also for recognizing important patterns in data. It's also necessary to achieve effective data security and target marketing. According to Gartner, by 2020, 50% of information governance initiatives will be enacted with policies based on metadata alone.
Our top 30 vendors
It's tough to choose just a sampling of vendors out of thousands when driving value from data requires so many tools. The following three categories of vendors were selected based on current market trends and their ability to address current and emerging enterprise requirements at scale. Note that some vendors may have products in more than one category, but are only listed once to allow us to acknowledge the leadership of a greater number of companies. Check out the hot vendors in other segments of IT in our full Vendors to Watch package.
Oracle (@Oracle) offers enterprise SaaS application suites, database PaaS and IaaS. Its analytics platform provides visual analysis, discovery advanced analytics, reporting, and forecasting capabilities.
Data Science and Machine Learning Platforms
Amazon Web Services (@awscloud) offersa broad array of products and services for processing, analyzing and visualizing data easily and cost-effectively. It also offers several AI products and services spanning frameworks and infrastructure, API-driven services and machine learning platforms.
Data Management and Governance
Learn about more hot vendors in our full Vendors to Watch package.
(Cover image: Billion Photos/Shutterstock)