Netflix Cloud Architect: How To Succeed As Disruptor
Adrian Cockcroft, Netflix's former chief cloud architect, outlines how a changed business flowed out of the cloud.
8 Data Centers For Cloud's Toughest Jobs
(Click image for larger view and slideshow.)
Adrian Cockcroft, former chief cloud architect for Netflix, recently described an "evolution of business" away from familiar processes into a more digital world, where integration of teams and speed of development decide life and death for companies. During seven years at Netflix, Cockcroft became one of the chief advocates and explainers of what it means to move a business into the cloud. He was a frequent speaker at shows like Cloud Connect, and wrote about key cloud management techniques in his aptly named Adrian Cockcroft's Blog.
During his tenure, Netflix halted construction of its own datacenters in 2008 and moved its customer-facing operations onto Amazon Web Services. The move coincided with Netflix's decision to become a video-streaming business to a variety of consumer devices as well as a shipper of DVDs. In the process, it rapidly grew its customer base by millions and launched new ways of finding and ordering films online.
At the end of 2013, Cockcroft left Netflix to become a technology fellow at venture capital firm Battery Ventures. He spoke for 40 minutes April 1 at the InformationWeek Conference, collocated this year with UBM Tech's Interop Las Vegas. Cockcroft is a former distinguished engineer at Sun Microsystems in high-performance computing and a co-founder of eBay Research Labs.
The changes wrought by digital services replacing physical ones have led to changes in how businesses must organize their staffs, conduct IT, and produce software, he said at the InformationWeek Conference. Many of the techniques he spearheaded at Netflix supported streaming services as a replacement for shipping DVDs. He said the only datacenter capacity that Netflix retained after it moved into the cloud supports the systems for DVD shipping and billing. Everything else has moved onto AWS virtual servers.
He didn't use the term "agile" to describe Netflix's software development, but the speed and close collaboration of developers and business managers he described would easily fit into many agile-inspired processes. Only at the end of his talk did he use "agile" to describe the state of the corporation as a whole, if it follows his recommendations.
Adrian Cockcroft (Image: Battery Ventures)
The old process -- laboriously defining a software project, ordering hardware to support it, coding it over a period of months or a year, and finally launching an application into production -- takes too long for today's pace of business, he warned. Aiming merely to "optimize a system," making it cheaper, faster, and better, is no longer sufficient.
"There's a real sense of innovation," in a Netflix-style approach. "There's an evolution of products from custom to something more like utilities," that can be periodically added to and built out, he said. This type of development "is usually trying to invent something people have never done before."
Cockcroft then launched into the heart of his talk. "Let me tell you briefly what I learned at Netflix. I was there seven years, right from the beginning when streaming started. The first thing we learned, the goal of Netflix [in keeping up with] the growth that was driving all the things that Netflix could do, was how fast can we move? Netflix at the time was a very small player in the market. The competitive advantage that we had was to move faster and learn faster and get things done more quickly."
But doing things faster meant speeding up decision-making and producing software to support new services. To do that, Netflix had to "remove the friction from product development." Friction is produced when designers of a product talk to product management and engineering teams to describe what must be done, then send each team off to do its own work. Incomplete communication, varying interpretations of requirements, and changing requirements in a changing marketplace all lead to friction.
A company must ask, "What are the handoffs, what are the things that slow you down? We put a high-trust, low-process environment in place with few handoffs between teams. We consolidated teams... We put things together. We removed process debt (overhead) completely," he recalled.
Designers, developers, and product managers were "consolidated" into single teams, with a small coding team responsible for the software from conception through deployment as a production system. It's a key DevOps goal to get developers to spend more time listening to the operations side of the house. At Netflix, developers became the operations side of the house. They continued to have responsibility for the software after it went into production.
Netflix created "a very developer-oriented culture," Cockcroft continued. "We give freedom to employees to do things, but they're responsible for them. [Netflix executives] are just incredibly insistent, you have the freedom to do things. If employees do something stupid, they generally learn from it. If they consistently do stupid things, then you have to get rid of them. There isn't a sense you can do anything you want, that you have tenure... You're free, but you're responsible."
Pushing responsibility down several levels allowed Netflix to produce new services quickly and try them out with customers rather than spending months building something that customers may not want. Cockcroft described it has a virtuous cycle of develop, release, test, go back to development again.
"The question is, 'How fast can you go around that cycle?' because that's how you learn something. We were learning something new all the time," he said, as Netflix discovered that customers wanted to download movies to game consoles and tablets, as well as to their computers.
Going around the customer test cycle requires "high trust" among team members and "low process," or few slowdowns, because tasks need to
Charles Babcock is an editor-at-large for InformationWeek, having joined the publication in 2003. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive Week. He is a graduate of Syracuse ... View Full Bio
Multicloud Infrastructure & Application ManagementEnterprise cloud adoption has evolved to the point where hybrid public/private cloud designs and use of multiple providers is common. Who among us has mastered provisioning resources in different clouds; allocating the right resources to each application; assigning applications to the "best" cloud provider based on performance or reliability requirements.
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.