How Innovation is Changing the Analytics Landscape

Cloud, automation, and emerging intelligent platforms are just beginning to reshape the analytics landscape. Cloud has become a formidable force altering business models, project delivery, architecture, and team structures. Automation has expedited and eliminated traditional processes. Intelligent analytics platforms powered by artificial intelligence have expanded the reach of analytics to the masses. The analytics world is rapidly evolving.

We live in an incredible era of extremely fast, disruptive innovation. Globalization, accelerated technology change, infinite cloud scale, ubiquitous connectivity, and an Internet of smart things is enabling a fourth industrial revolution, the digital transformation. The challenges data and analytics professionals are facing right now are not technology related. The challenges are related to adapting to constant change and keeping up with the frenetic pace of innovation to remain relevant.

Many small quick projects

As on-premises infrastructure continues to migrate to the cloud and machines begin taking on repetitive analytics work, humans are adjusting to the new normal of working on many small, quick projects with less people. The core idea of lean, agile analytics development is to eliminate or reduce non-value-added activities to increase overall value. The good news is that we should expect more analytical mind work and less busy work.

Data integration and data preparation platforms with pre-built industry models are becoming smart enough to plug-and-play data sources, self-repair when errors occur in data pipelines, and even self-maintain with minimal human interaction reducing the amount of ETL work in analytics projects. Self-service reporting has long been simplified. Now we are seeing new augmented analytics capabilities automate aspects of reporting.

Automated machine learning platforms with pre-packaged best-practice algorithm design blueprints are shortening project lifecycles and are shifting historical data scientist work away from technical algorithm expertise towards emphasis on business domain knowledge.

Analytics projects that used to take months to complete have been reduced to days or weeks, since cloud eliminates time-consuming server purchase, set up, software installation, configuration, and deployment work. In the analytics consulting world that lives at the forefront of industry transition, I have already witnessed on-site staffing models merge with blended virtual and 100% virtual delivery models. Historical project-based consulting work has been gradually shifting to on-demand staff augmentation and subscription-like support models.

Having managed teams in the past, I know it is easier to coordinate a few big projects than to juggle a lot of little projects. It also takes exceptional time management prowess to constantly switch context and remain productive as you frequently move between small projects. Depending on the type of work that you have done in the past, these changes might be intellectually exciting or somewhat unsettling.

Flexible, on-demand architecture

To amalgamate constantly changing, different data source realms, data catalogs, search, data virtualization, data pipeline orchestration, and hybrid analytics technologies have become key assets in a digital era analytics arsenal. These solutions bring much needed order to modern data chaos.  

Traditional data warehousing and analytics architecture is also switching from rigid, legacy data systems to nimble, flexible, on-demand cloud service designs that can get the most out of big data and analytics for the least recurring cost. The wide array of analytics cloud services, and, compute and serverless technologies, including but not limited to various database types, data lakes, Internet of Things (IoT), Lambda, streams, ingestion hubs, and microservices has become overwhelming to understand, piece together and estimate usage-based pricing. To design and manage modern analytics architecture, new cloud data architect and data pipeline engineer roles have emerged.

[Read more from Jen Underwood, including How to Break into an Analytics Career.]

Another key analytics landscape difference in the digital world is the required speed of automated decisions. In an interconnected, omnichannel digital world, timely intelligence becomes a need versus a want to prosper. The use cases for batch reporting are declining. In automated channels, you don’t have the luxury to pull a report or wait for a data refresh from last night, week, or month. To close the loop between insight and action in digital processes, analytics and predictive algorithms are increasingly being embedded directly within business apps and processes. Intelligent systems automatically decide or interactively guide humans during the decision-making process from machine-generated insights.

While there will always be work for humans in the analytics word, the nature of that work is already changing. It is more important than ever before to invest a little time each week to look beyond the comfortable, older and current analytics technologies that you already know or might be assigned to work on. If you do work in an organization that is not embracing newer technologies, you’ll likely need to burn the midnight oil to keep up with these changes.