State Of DevOps: Big Gains Elusive

Architecture before automation

DevOps uses a circular methodology, whereby the output of the last iteration of the IT environment feeds the inputs of the next iteration. For example, when IT installs an application, the environment must open ports through the firewall. But say the app doesn't work because developers forgot to mention that the app needs to talk to Active Directory, and that requirement wasn't in the list of ports they wanted opened in the firewall. The iteration to fix that problem must be skewed toward spending more time in architecture and modeling of the processes and infrastructure so developers don't make that mistake again. Chef and Puppet tools provide configuration management for a variety of packaged and custom software, including operating systems, applications, and databases.

The real goal of these DevOps tools isn't to get the configuration files for these applications syntactically perfect. It's to serve the larger goals, such as keeping an e-commerce system running. Doing so requires managing the entire infrastructure that the configuration files create, and checking that the infrastructure they build matches the designed architecture.

For example, the architecture might describe how the environment should react to keep a system up when the database server fails because of a lost network connection, a failed drive, or even too many users accessing it. Yet when someone installs the configuration, the application uses the IP address of the database server instead of the virtual IP of the database cluster, so losing the primary server breaks the connection from app to database. If the configuration had matched the architecture, the failover would have occurred properly.

IT teams set up DevOps tools to respond to bad events in the infrastructure, such as outages from broken servers, but also to good events, such as increased demand for an e-commerce application. In either case, IT can automate creation of new environments -- such as spinning up duplicate virtual machines to handle overcapacity -- to immediately fix these common problems.

Of course, there's a catch: The very tools that automate the infrastructure, such as Puppet, Chef, or custom scripts, can drag down the infrastructure if they fail. One bad script could bring down your infrastructure, just as one bad patch update did before DevOps. An essential part of the IT architect's work is knowing the details of how the infrastructure's automation works in order to determine the impact an application change will have.

Not everyone is on board with DevOps. In our firm's consulting work, the arguments we hear most often against DevOps are that it dumps more responsibility onto developers, that developers should focus on coding and let the IT ops people deal with servers, and that having standard and staging servers offers enough reliability.

The doubts come from both sides: 23% of respondents adopting DevOps say developers are the ones resisting the methodology, 38% blame network/datacenter folks, and another 39% blame both. Certain IT pros also have a legitimate fear of losing their jobs: If existing engineers can't understand the architecture and all the infrastructure components, and then they write scripts to automate the DevOps processes, they won't fit in a DevOps environment. Although 59% of the respondents to our survey say they have the proper scripting expertise on their network and systems team to implement DevOps, 38% of those are just now developing the skill set.

Everyone on board: The security example

The DevOps name implies that only developers and IT operations can benefit from the methodology. Not true. We've seen how IT security, audit, and governance teams can work more effectively and respond to business change better using DevOps. Sadly, many people don't see things that way: Only 45% of tech pros adopting DevOps say they expect it to improve security; 32% see DevOps having no impact, good or bad, on security; and 7% think DevOps will make the IT operation somewhat less secure.

To sell the security team on DevOps, let them sit in on the deep technical discussions that team members must have to develop the associated architecture models. Those discussions are pure gold for spotting security risks.

For example, one large company was spinning up a DevOps-driven development project using a private-cloud provider, and it added a security engineer to the discussion. That engineer was shocked to learn that all the scripts the team planned to use for automated provisioning and network changes stored user names and passwords in the scripts and used domain-administrator-level accounts. That security flaw meant anyone who had access to the scripts could log in to the network and gain full access to all the servers and data. Was this just how DevOps had to be done? After poking around, the worried engineer learned that the company was storing domain-level accounts in all kinds of batch files, scripts, and application configuration files -- all in plain text accessible to pretty much all Active Directory users. He started a company-wide fix right away.

Let's face it, both security and operations teams generally have a "say no" culture. Getting those teams in the same room implementing the DevOps methodology will make deployments go more smoothly, because both teams can gain trust in the automated processes and tools. Working together, they can automate even complex tasks, such as separating data access by role, which also makes them easier to audit.

Nick Galbreath, former director of engineering at Etsy, a website for selling handmade crafts, says Etsy used the same configuration scripts that automate deployment of computing capacity to also assert security controls, such as requiring SSL or checking whether certain ports are open. Misconfigurations get reported automatically. Taken a step further, when a system is configured using a tool such as Puppet, the system could require an automatic vulnerability scan before putting the new apps on the production network.

You aren't Etsy or Facebook

Web companies such as Facebook, Netflix, and Etsy are big public advocates of DevOps. Etsy technical blogs are great at describing how DevOps tools can work at massive scale.

But you're not Etsy, so you can't just copy its model. You need to start at the architecture stage and model for your very different business needs, and you need to accept that the necessary cultural change will take time.

Web companies, for example, do a huge number of code changes but have very few applications -- the opposite of most enterprise IT organizations, which have a maze of legacy apps for which they try to minimize changes. For example, in a single month (January 2011), Etsy made 517 code deployments to a Web application production environment that served 1 billion page views. Compare that with the organizations represented in our survey; 96% typically move fewer than 60 application deployments into productions in an entire year. Conversely, Etsy has essentially one major application, while 23% of the organizations represented in our survey have more than 60 applications.

You probably have systems from the 1980s and software documentation that might as well say "Here Be Dragons," because no one really knows how those systems work and the mindset is just don't touch that code. Startup Web companies have the luxury of being able to set up their teams from the start for the integration of IT ops and development -- the entire IT life cycle -- to emphasize open communication, continuous development, and shared architecture knowledge.

Given these vast differences, why should traditional IT shops try to mimic the DevOps agility and automation of the Web giants? The reason is that both Web companies and traditional enterprises share a need for speed and efficiency. To those 46% of survey respondents who say they have more pressing technology or business priorities, or the 33% who see no business demand for DevOps, we say hogwash. Companies of all kinds want to give customers and employees new capabilities faster, and few companies have a culture whereby IT can respond reliably and quickly enough to business changes.

Agile development has solved only half the problem. DevOps is a powerful methodology to get all of IT to ditch broken processes and systems that slow down innovation and to spend more time on opportunities.

Michael A. Davis is CTO of CounterTack. Write to us at [email protected].