Photobucket Pictures Its Future On OpenStack

Photobucket is using a Cisco-built OpenStack cloud to better compete with the heavily financed Instagram, Pinterest, and Flickr as a lifetime destination for users' photos.

Photobucket stores personal photos, provides editing tools, and generates a link to users' photos so that they can be displayed in documents or on other websites without being uploaded to those destinations.

The 10-year-old, 74-employee company was once all the buzz. Now it has to compete with Pinterest, which wants to be everyone's lifetime scrapbook site for photos, videos, and documents. In late October, Pinterest garnered $225 million in venture capital to expand its services, following $200 million it received in February. Its estimated valuation is $3.8 billion.

Photobucket knows it lags behind Instagram, the instant sharing service that adds a telegram-like ability to deliver images wherever its users want. But Photobucket figures it has edged ahead of Flickr, because it hosts 4 billion transactions a day, has links to 3 million websites, hosts 20 million unique visitors a month, and stores 14 billion pictures.

 [Want more on how Photobucket uses mobile applications? See Help Wanted 2.0: SMB Recruiting Goes Mobile.]

Given the nature of the competition, Photobucket sought to operate in the most efficient manner possible. It knows that fractions of a penny saved in one part of its operations would support other operations as a long, competitive race runs itself out.

These challenges prompted Mike Clark, Photobucket's co-founder and CTO, and Jay Kistler, its director of operations, to look for the best way to reorganize its infrastructure in 2012. Kistler realized that server utilization was only in the 15%-20% range for 800 bare-metal servers. Virtualization would improve that, but how should it virtualize?

Key staff members began OpenStack pilots in what was still an early version of the open-source cloud software. The staff members "dove in" and found OpenStack a somewhat unruly amalgamation of code at the time, Kistler told us. "It's not for the faint of heart." Particularly baffling was the Quantum networking part of OpenStack (now renamed Neutron), which proposed a new way to do networking in the cloud. Kistler and Clark weren't too sure how its software-defined approach to virtualized networking was going to play out.

"We actually are a strong command-line company," Kistler said. IT operations were comfortable with Cisco's converged approach to racks of servers, network switches, and storage in its unified computing system (UCS) architecture, but they didn't want an all-proprietary Cisco cloud.

Figure 1: (Source: Jon Pallbo via Wikimedia Commons)

Cisco was happy to help Photobucket implement a Cisco version of OpenStack on top of UCS. Now, instead of 800 bare-metal servers, Photobucket operates MySQL databases serving content from 350-400 physical servers. Hadoop analytics (essential to understanding what users are doing and how they're using Photobucket servers) also runs unvirtualized on a bare-metal cluster. The rest of operations run on the OpenStack Cloud, which has cut an additional 350-400 servers down to two UCS clusters -- one running in Denver and one in Phoenix. Both are operated from one management console as a single OpenStack cloud.

The Cisco distribution of OpenStack includes a plugin for OpenStack's Quantum networking that works with the Nexus 1000 switch for multiple virtual machines and Cisco's accompanying networking fabric. Clark said choosing this option saved Photobucket the task of deciphering an emerging and rapidly changing part of the OpenStack cloud. "It requires some real technical acumen to stand up an OpenStack cloud."

Photobucket has gone its own way with the Salt open-source configuration and deployment engine, written in Python, the preferred language of the firm's developer teams.

Kistler and Clark were enticed by the Xen hypervisor, a version of which runs the Amazon Web Services cloud. But in the end, they leaned toward OpenStack's default hypervisor, KVM, which was more efficient in their virtualization bakeoffs than VMware's ESX Server or even Red Hat's version of KVM used in its Enterprise Virtualization suite.

"We stood up their version, but it didn't meet our internal benchmark as most performant," Clark said.

The Photobucket staff members are comfortable dealing with the update and compatibility issues of open-source code, and they like Cisco's commitment to support the OpenStack project. They liked the vigor of the project itself, with its many large-company backers and long list of contributors. "The DOM zero stuff in Xen" -- the need to establish Domain 0 with Xen before it can operate -- "looked out of date" in comparison, Clark said. Contributions to Xen looked "not as relevant as those coming in to KVM," and some of the staff's favorite Xen contributors had moved over to the KVM project.

Kistler and Clark still have reservations about the tendency of one OpenStack project to "solve the same problem" as another, leading to confusion over what should be implemented to rationalize operations. But again, they felt Cisco's OpenStack would have Cisco's guiding hand behind it, making choices for its UCS rack architecture.

Now Photobucket's IT staff is looking at deploying fresh OpenStack servers in two to three minutes, instead of the minimum two to three hours that it used to take. It is using only 60% of its UCS rack architecture, leaving 40% available to plug in more converged servers and networking. Meanwhile, it has consolidated 10-12 servers down to a UCS blade or 2u rackmount server.

Running Photobucket systems has gotten easier in the process. "Everyone's walking around with sort of a smile on their face, because they're not fighting to keep the infrastructure running," said Kistler. With his shop 60% virtualized today, he expects that figure to be 90% by this time next year, with more of it running on the OpenStack private cloud.

IT leaders must know the tradeoffs they face to get NoSQL's scalability, flexibility, and cost savings. Also in the When NoSQL Makes Sense issue of InformationWeek: Oregon's experience building an Obamacare exchange (free registration required).