Is Data Streaming Worth the Learning Curve?

Data has inherent business value because it allows organizations to make more informed -- and ideally more cost-effective -- decisions. When business services and applications can ingest, process, and react to data in real time, teams can accelerate decision-making and make smarter choices, too.

To realize this for their own organizations, technical leaders need to remove roadblocks that prevent developers from learning and grasping data streaming skills. Even for organizations with plenty of expertise in data and systems architecture, this can be difficult.

Developers working with traditional data systems that rely on batch processing can default to what they know: relational databases that store information as finite sets and object-oriented programming. To learn data streaming, they will need to set aside the familiar to learn a completely new approach.

Ensuring that entire developer teams are ready to leverage real-time data will require strategic planning and insight into the day-to-day operational challenges of data streaming. To make these initiatives successful, technical leaders must give developers the time to learn the basics of data streaming, the tools and resources to fully grasp them, and the organizational buy-in they need to take that innovation further.

Challenges Developers Face When Exploring Use Cases

Businesses generally face three challenges when adopting data streaming: introducing developers to unfamiliar technologies, teaching those developers how to solve problems in a new way, and maintaining the underlying infrastructure.

The first and third challenges aren’t new. The languages, toolkits, and platforms developers use often change, and learning new technologies is a career-long inevitability. At the same time, businesses are always evaluating whether it’s more cost-effective to hire in-house infrastructure operators or pay for managed services.

The bigger challenge is teaching developers to rethink their approach to problem-solving.

The phenomena of a new technology forcing developers to rethink how they solve problems is happening all around us, and often with good results. The introduction of React asked developers to rethink how they build user interfaces, resulting in dramatically simpler and less buggy front-end applications. GraphQL led developers to think differently about how they developed APIs, resulting in slimmer and more secure layers over data systems. Kubernetes asked developers to reconsider how they manage infrastructure, resulting in easier to understand, declarative environment models.

Data streaming is no different. In some ways, it makes problems simpler to solve, but it’s an approach that’s unfamiliar to many developers, so mastering it takes consistent practice.

Lay the Groundwork for Real-time Innovation

From generating reports on marketing campaign performance to monitoring IT security, waiting for data and information is no longer sustainable in any business function or industry. Often, the data needed to bring these solutions to life is already being generated and stored in real time. It’s just locked away in siloed databases and systems.

Where many engineering organizations go astray is in maintaining the traditional centralized data ownership model. Centralized data teams are then tasked with building and managing data pipelines. As a company matures and scales its data streaming usage, that team can quickly go from a strategic resource to a bottleneck.

If overwhelmed data platform teams take weeks or even months to set up new data pipelines, organizational innovation will stagnate. Then the developers tasked with learning data streaming will have little time or opportunity to learn the necessary new skills.

Technical leaders who want to leverage real-time data across the organization need to make decentralized, domain-driven data ownership part of the plan from the outset, according to KPMG. This means giving domain-specific teams the ability to manage and own data pipelines that publish to a shared data streaming platform, while the central data team focuses on providing platform services and controlling data governance.

This approach gives developers plenty of opportunity to test and expand their skills while making real-time data a true competitive advantage for the broader organization.

Unlocking the Business Value of Real-time Data

Despite the implementation challenges, event-driven architectures and real-time applications have become more widely adopted because they solve the three priorities at the heart of every business problem: saving costs, making money, and reducing risk.

To unlock the full value of real-time data, engineering organizations need to commit to a new way of problem-solving: treating business challenges and their solutions as ongoing chains of events. That often means designing new business processes (and redesigning old ones) as well as the IT infrastructure and data ownership model that underpins them.

Over the past 10 years, we’ve seen companies ingest and process data in real time to dream up creative -- previously impractical -- solutions to customers’ everyday problems. Services like ridesharing apps Uber and Lyft and real-time, peer-to-peer payments like Venmo, CashApp, and Zelle have displaced long-established solutions.

Before turning these ideas into real-time capabilities, developers that built these applications first had to learn the basics of data streaming platforms. At your organization, that all depends on whether you provide developer teams with the tools and support they need to experiment, fail, succeed, and advance.