Agile Development And The Red Line: At A Point Of No Return

We’ve reached a point of no return. In the cloud industry, a culture has been established where everyone thinks they must innovate and deliver new features. It’s not only developers who have this culture. Sales has a tendency to promise new features with the hope that it will help them expand and retain current customers. We’ve all heard stories where a sales team has to fix a customer issue before even thinking about selling.

While new features and customer service are important, what has been successful for our customers and us has been to change this culture and focus on refining and delivering high-quality features. To do this, we created what we call a red line approach.  I believe that our red line method could become an industry-wide standard.

The red line is built upon the agile development philosophies of customer satisfaction by early and continuous delivery of valuable software and continuous attention to technical excellence and good design.  I’ll explain by showing how we use the red line at my company.

Finding Your Red Line

A few years ago, our engineering teams implemented a metric that we call the red line. The red line helps us find the sweet spot balance for improving the quality of existing features and new feature innovations. It helps us understand how much unnecessary effort we cause ourselves in support costs and, more importantly, how satisfied or frustrated our customers get with various features.

As our teams are directly responsible for keeping the ServiceNow cloud functional and embedding greater features, the red line is an important quality control measure.

The way we determine the red line is for each engineering team is simple: It is the total number of issues for features or services developed and operated by a team divided by the total number of customers we have on a monthly basis. This means that every customer issue needs to be categorized and attributed to an engineering team.

For example, if there were 10 issues in a given month and you had 100 customers, the red line would be 0.1. If you had 1,000 issues, the red line would be 10.0. The lower the red line number, the better. A key point to note is that an issue is an issue, regardless of severity or priority that the customer raises or that we raise as part of running our operations.

Applying the Red Line to DevOps

As an engineering organization, we require monthly accountability to management on the red line number for each team. Teams, in return, watch the red line on a daily basis to keep the number low, making it real-time and actionable. They are able to make changes and push out updates, saving time trying to help customers with a work around. Because of this time savings, we are able to look at every issue that comes in, not just the big ones.

When we started measuring the red line, we noticed an interesting trend; many of the issues that arose were the same. Focusing each team on their assigned areas helped us to resolve these repeat issues quickly, and most issues quickly become non-issues.

Balancing Innovation with the Red Line

As an organization, we don’t prescribe a balance between innovation and fixing issues. However, the red line helps us to know where the team is on this continual balancing act.

If the red line is down or sloping downward, a team can spend a majority of their time innovating new services or features. If the red line slopes up, they need to fix issues. If the red line is flat, they are balancing quality with innovation. If there is a consistent and significant slope up for a few quarters, we may need to make changes like diverting some resources to fix issues before they become big problems.

What is key here is understanding the issues that are driving the red line for each engineering team. In some cases, the issues may be driven by poor documentation, resulting in a spike in “how do I…?” issues. Another case may be an engineering team deploying new software that causes performance issues for a series of customers. This red line helps measure and address these issues so that an organization can improve product quality and customer satisfaction.

Getting Buy-In On the Red Line

To get buy-in from our engineers, we communicate the philosophy and our commitment to the red line again and again. They quickly learn how important it is to the company and also learn how to balance between fixing issues and innovation.

Since we implemented the red line four years ago, our team has grown from 50 to 700 engineers. As we hire new talent, we indoctrinate them into the philosophy. By indoctrinating, I don’t mean we do a of training, but rather we communicate about it. We send out regular updates and have an ongoing dashboard that displays each teams’ red line and progress. Making sure that the red line is front and center in all of our decisions helps us to improve our offerings, add the right features at the right time and ensure they are high quality. The focus on the red line helps our customers to be successful and as a result, helps us to be successful.

Industry-Wide Red Line Adoption

Think of the impact industry-wide adoption of our red line philosophy could bring to organizations. The cloud industry must improve the quality of existing features so that our customers can realize the impact of the innovation that has taken place. The red line philosophy helps with this.

I understand that if you are launching a product, or are a startup, you have to focus on innovation, and that is okay. But as soon as you start getting traction, your engineers need to balance that innovation with fixing customer issues. This means putting the right testing in place, ensuring you have the right automation and improving features, alongside working on new features.

By employing the red line philosophy, we have been able to change our culture, setting the customer experience as the foundation of our business. If the industry adopts this same philosophy, we will change our collective culture and balance innovation of features with quality features that help our customers to be successful. We also define the limit of what we will tolerate as acceptable in terms of issues.

This agile development approach and the creation of the red line has helped us reach our point of no return.   If our red line approach, or a similar one, is adopted industry-wide a culture of excellence for both truly high-quality and innovative features can be achieved.

Allan Leinwand is chief technology officer at ServiceNow, the enterprise cloud company.  In this role, he is responsible for overseeing all technical aspects and guiding the long-term technology strategy for the company.

Before joining ServiceNow, Leinwand was chief technology officer – Infrastructure at Zynga, Inc. where he was responsible for all aspects of technology infrastructure used in the delivery of Zynga’s social games including data centers, networking, compute, storage, content distribution and cloud computing. Leinwand currently serves as an adjunct professor at the University of California, Berkeley where he teaches on the subjects of computer networks, network management and network design. He holds a bachelor of science degree in computer science from the University of Colorado at Boulder.