PayPal Declares It's 100% OpenStack Cloud

7 Linux Facts That Will Surprise You

PayPal has completed a three-year migration from a typical mixed-enterprise data center to an OpenStack private cloud.

PayPal processed $228 billion in payments on its infrastructure last year, making it one of the largest financial services OpenStack clouds in production.

PayPal runs 4,100 standardized x86 servers under OpenStack to provide 162 million customers with information, mobile application support, website interactions, and payment processing. Stateless interactions (such as a PayPal's front-end presentations in response to customer requests for information) and mid-tier services supporting business logic are being processed on OpenStack.

"We have converted nearly 100% of our traffic serving, Web/API applications, and mid-tier services at PayPal to run on our internal private cloud," said Sri Shivananda, VP for global platform and infrastructure, in an interview with InformationWeek. (Shivananda will discuss data center strategy at the InformationWeek Conference, April 27 & 28 in Las Vegas. IT leaders can learn more and register here.) 

But Shivananda and PayPal spokesmen were also careful to note that a few legacy systems remain in place, without specifying which ones.

Financial services firms often decline to reveal much about their infrastructure, seeking to avoid the possibility of making it easier for malicious hackers to get inside. But it's well known that some of the world's largest financial institutions, including Bank of New York Mellon, State Street Bank, and Bank of America, have reorganized their infrastructure to function into a more uniform architecture with more automated processes.

In PayPal's case, the OpenStack transition wasn't only a move to a more automated infrastructure. It was an internal cultural change as well, said Shivananda. The change undertaken by the IT staff "goes well beyond server provisioning," he said.

PayPal knew it wanted to revamp its data center infrastructure in 2011, when OpenStack was little more than a young work-in-progress. It experimented with building out a more automated infrastructure, relying in part on VMware virtualization.

OpenStack caught its attention early on, however, and eventually "there were five to six versions of it" among the  experimental implementations that PayPal software teams were working with, said Shivananda. "After a while, we had to converge the stack," he said. "That helped us learn a lot about managing OpenStack."

PayPal's OpenStack Upgrade Process

PayPal has also been forced to upgrade OpenStack as a new release appears, roughly every six months. "It's an eye-opening experience to upgrade OpenStack," said Shivananda. "We have built a ton of experience around the upgrade process."

[Want to learn more? Read PayPal Finds Node.js Secret To Successful Makeover.]

The organization has adopted a set of processes and procedures around an OpenStack upgrade, which includes establishing a war room and appointing a commander of the process. According to Shivananda, it's important "to bring consistency to the table and take fleet-wide actions," keeping the 8,500 server mass "homogenous" and not allowing it to drift into segments operating under different versions of OpenStack. That means 180,000 data center assets – servers, top-of-rack switches, firewalls, load balancers, and storage volumes -- all function as part of the PayPal OpenStack cloud.

Doing so allows IT to operate PayPal data centers in a routine and automated way, Shivananda continued. If a data center server, switch, or storage volume failed in the past, it was common practice to send a staff member to fix the issue as soon as possible. Under OpenStack, the processing on a failing device is switched over to healthy ones. Mechanical failures are tolerated in the OpenStack cloud until it's time for a routine sweep that fixes or removes all types of failed devices and brings replacements online. Instead of technicians working on, say, 1% of stalled, troubled, and failed devices, they may be working on as many as 3% to 5%, under the periodic sweep method of operating, he said.

The PayPal cloud has its own sensing mechanisms to detect when hardware is acting up or about to fail.  

The main goal behind the automated method of operations is to provision PayPal development teams quickly when they need a set of servers. In the rapidly changing field of mobile payments, PayPal is keeping up by allowing frequent updates to dozens of applications.

"There's not an hour in the day when we're not rolling out software patches and updates," Shivananda said. "We're a high frequency of change environment." That would not be possible without the conversion to an OpenStack infrastructure. The uniformity of what makes up the infrastructure, and the predictability of how it will run, make it possible to impose frequent software changes.

The IT staff's next challenge is to incorporate use of Docker containers into its OpenStack cloud, Shivananda said.

Asked to clarify an old controversy, Shivananda said PayPal, once a VMware virtualization shop, has backed off its dependence on VMware and uses OpenStack for its compute virtualization. That means it relies on the OpenStack default, KVM open source hypervisor on many compute hosts, instead of VMware's ESX Server. However, he added, "VMware remains core in our network virtualization." Virtual machine provisioning and lifecycle management are now done under OpenStack.

Two years ago, Boris Renski, chief marketing officer for Mirantis, an OpenStack consulting firm that was doing work at PayPal at the time, said that PayPal would convert to OpenStack and parent company eBay would follow, replacing VMware in the process.

Shivananda didn't comment on eBay's infrastructure plans, other than to say both companies have their own plans and schedules to get to an OpenStack cloud. PayPal and eBay are in the process of becoming separate companies, and each has its own infrastructure. The separation is expected to become official in the second half of 2015.

Asked if PayPal used a private label version of OpenStack -- both Mirantis and VMware produce their own distributions -- Shivananda said it had developed its own staff expertise and installed its own version, without relying on an outside vendor's configuration.

Are you an IT leader wrestling with data center strategy? Come discuss this and other strategic IT issues with your peers at the InformationWeek Conference. You can email InformationWeek editor Chris Murphy with questions, or register now and save $200 off Conference passes with discount code CMBLOG. Other speakers include the CIOs of Walmart, AstraZeneca, the NBA, ConocoPhillips, Royal Caribbean, and more, along with 3D Robotics CEO Chris Anderson, discussing drones and the future of automation.