PayPal Declares It's 100% OpenStack Cloud - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IT Leadership
09:06 AM
Connect Directly

PayPal Declares It's 100% OpenStack Cloud

After a three-year migration, PayPal says it has implemented its own OpenStack cloud for close to 100% of its operations, making it one of the largest financial services OpenStack clouds in production.

7 Linux Facts That Will Surprise You
7 Linux Facts That Will Surprise You
(Click image for larger view and slideshow.)

PayPal has completed a three-year migration from a typical mixed-enterprise data center to an OpenStack private cloud.

PayPal processed $228 billion in payments on its infrastructure last year, making it one of the largest financial services OpenStack clouds in production.

PayPal runs 4,100 standardized x86 servers under OpenStack to provide 162 million customers with information, mobile application support, website interactions, and payment processing. Stateless interactions (such as a PayPal's front-end presentations in response to customer requests for information) and mid-tier services supporting business logic are being processed on OpenStack.

"We have converted nearly 100% of our traffic serving, Web/API applications, and mid-tier services at PayPal to run on our internal private cloud," said Sri Shivananda, VP for global platform and infrastructure, in an interview with InformationWeek. (Shivananda will discuss data center strategy at the InformationWeek Conference, April 27 & 28 in Las Vegas. IT leaders can learn more and register here.) 

But Shivananda and PayPal spokesmen were also careful to note that a few legacy systems remain in place, without specifying which ones.

Financial services firms often decline to reveal much about their infrastructure, seeking to avoid the possibility of making it easier for malicious hackers to get inside. But it's well known that some of the world's largest financial institutions, including Bank of New York Mellon, State Street Bank, and Bank of America, have reorganized their infrastructure to function into a more uniform architecture with more automated processes.

In PayPal's case, the OpenStack transition wasn't only a move to a more automated infrastructure. It was an internal cultural change as well, said Shivananda. The change undertaken by the IT staff "goes well beyond server provisioning," he said.

(Image: PayPal)

(Image: PayPal)

PayPal knew it wanted to revamp its data center infrastructure in 2011, when OpenStack was little more than a young work-in-progress. It experimented with building out a more automated infrastructure, relying in part on VMware virtualization.

OpenStack caught its attention early on, however, and eventually "there were five to six versions of it" among the  experimental implementations that PayPal software teams were working with, said Shivananda. "After a while, we had to converge the stack," he said. "That helped us learn a lot about managing OpenStack."

PayPal's OpenStack Upgrade Process

PayPal has also been forced to upgrade OpenStack as a new release appears, roughly every six months. "It's an eye-opening experience to upgrade OpenStack," said Shivananda. "We have built a ton of experience around the upgrade process."

[Want to learn more? Read PayPal Finds Node.js Secret To Successful Makeover.]

The organization has adopted a set of processes and procedures around an OpenStack upgrade, which includes establishing a war room and appointing a commander of the process. According to Shivananda, it's important "to bring consistency to the table and take fleet-wide actions," keeping the 8,500 server mass "homogenous" and not allowing it to drift into segments operating under different versions of OpenStack. That means 180,000 data center assets – servers, top-of-rack switches, firewalls, load balancers, and storage volumes -- all function as part of the PayPal OpenStack cloud.

Doing so allows IT to operate PayPal data centers in a routine and automated way, Shivananda continued. If a data center server, switch, or storage volume failed in the past, it was common practice to send a staff member to fix the issue as soon as possible. Under OpenStack, the processing on a failing device is switched over to healthy ones. Mechanical failures are tolerated in the OpenStack cloud until it's time for a routine sweep that fixes or removes all types of failed devices and brings replacements online. Instead of technicians working on, say, 1% of stalled, troubled, and failed devices, they may be working on as many as 3% to 5%, under the periodic sweep method of operating, he said.

The PayPal cloud has its own sensing mechanisms to detect when hardware is acting up or about to fail.  

The main goal behind the automated method of operations is to provision PayPal development teams quickly when they need a set of servers. In the rapidly changing field of mobile payments, PayPal is keeping up by allowing frequent updates to dozens of applications.

"There's not an hour in the day when we're not rolling out software patches and updates," Shivananda said. "We're a high frequency of change environment." That would not be possible without the conversion to an OpenStack infrastructure. The uniformity of what makes up the infrastructure, and the predictability of how it will run, make it possible to impose frequent software changes.

The IT staff's next challenge is to incorporate use of Docker containers into its OpenStack cloud, Shivananda said.

Asked to clarify an old controversy, Shivananda said PayPal, once a VMware virtualization shop, has backed off its dependence on VMware and uses OpenStack for its compute virtualization. That means it relies on the OpenStack default, KVM open source hypervisor on many compute hosts, instead of VMware's ESX Server. However, he added, "VMware remains core in our network virtualization." Virtual machine provisioning and lifecycle management are now done under OpenStack.

Two years ago, Boris Renski, chief marketing officer for Mirantis, an OpenStack consulting firm that was doing work at PayPal at the time, said that PayPal would convert to OpenStack and parent company eBay would follow, replacing VMware in the process.

Shivananda didn't comment on eBay's infrastructure plans, other than to say both companies have their own plans and schedules to get to an OpenStack cloud. PayPal and eBay are in the process of becoming separate companies, and each has its own infrastructure. The separation is expected to become official in the second half of 2015.

Asked if PayPal used a private label version of OpenStack -- both Mirantis and VMware produce their own distributions -- Shivananda said it had developed its own staff expertise and installed its own version, without relying on an outside vendor's configuration.

Are you an IT leader wrestling with data center strategy? Come discuss this and other strategic IT issues with your peers at the InformationWeek Conference. You can email InformationWeek editor Chris Murphy with questions, or register now and save $200 off Conference passes with discount code CMBLOG. Other speakers include the CIOs of Walmart, AstraZeneca, the NBA, ConocoPhillips, Royal Caribbean, and more, along with 3D Robotics CEO Chris Anderson, discussing drones and the future of automation.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Newest First  |  Oldest First  |  Threaded View
<<   <   Page 2 / 2
User Rank: Ninja
3/31/2015 | 9:45:23 PM
Re: PayPal's hard won expertise
@Charlie B,

From what I've heard, it's tough to get started with OpenStack but once it's running, with the right documentation, it works well.
Charlie Babcock
Charlie Babcock,
User Rank: Author
3/31/2015 | 1:49:49 PM
PayPal's hard won expertise
OpenStack has been described as hard to implement, requiring professional consulting, open source that leads to proprietary code-like expenses to install. That said, PayPal has found it a worthwhile investment to get OpenStack up and running as its new data center infrastructure. The hard won expertise will hopefully keep it running.
<<   <   Page 2 / 2
Learning: It's a Give and Take Thing
James M. Connolly, Editorial Director, InformationWeek and Network Computing,  1/24/2020
IT Careers: Top 10 US Cities for Tech Jobs
Cynthia Harvey, Freelance Journalist, InformationWeek,  1/14/2020
Predictions for Cloud Computing in 2020
James Kobielus, Research Director, Futurum,  1/9/2020
White Papers
Register for InformationWeek Newsletters
State of the Cloud
State of the Cloud
Cloud has drastically changed how IT organizations consume and deploy services in the digital age. This research report will delve into public, private and hybrid cloud adoption trends, with a special focus on infrastructure as a service and its role in the enterprise. Find out the challenges organizations are experiencing, and the technologies and strategies they are using to manage and mitigate those challenges today.
Current Issue
The Cloud Gets Ready for the 20's
This IT Trend Report explores how cloud computing is being shaped for the next phase in its maturation. It will help enterprise IT decision makers and business leaders understand some of the key trends reflected emerging cloud concepts and technologies, and in enterprise cloud usage patterns. Get it today!
Flash Poll