PayPal Declares It's 100% OpenStack Cloud - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
IT Leadership
News
3/31/2015
09:06 AM
Connect Directly
Twitter
RSS
E-Mail
100%
0%

PayPal Declares It's 100% OpenStack Cloud

After a three-year migration, PayPal says it has implemented its own OpenStack cloud for close to 100% of its operations, making it one of the largest financial services OpenStack clouds in production.

7 Linux Facts That Will Surprise You
7 Linux Facts That Will Surprise You
(Click image for larger view and slideshow.)

PayPal has completed a three-year migration from a typical mixed-enterprise data center to an OpenStack private cloud.

PayPal processed $228 billion in payments on its infrastructure last year, making it one of the largest financial services OpenStack clouds in production.

PayPal runs 4,100 standardized x86 servers under OpenStack to provide 162 million customers with information, mobile application support, website interactions, and payment processing. Stateless interactions (such as a PayPal's front-end presentations in response to customer requests for information) and mid-tier services supporting business logic are being processed on OpenStack.

"We have converted nearly 100% of our traffic serving, Web/API applications, and mid-tier services at PayPal to run on our internal private cloud," said Sri Shivananda, VP for global platform and infrastructure, in an interview with InformationWeek. (Shivananda will discuss data center strategy at the InformationWeek Conference, April 27 & 28 in Las Vegas. IT leaders can learn more and register here.) 

But Shivananda and PayPal spokesmen were also careful to note that a few legacy systems remain in place, without specifying which ones.

Financial services firms often decline to reveal much about their infrastructure, seeking to avoid the possibility of making it easier for malicious hackers to get inside. But it's well known that some of the world's largest financial institutions, including Bank of New York Mellon, State Street Bank, and Bank of America, have reorganized their infrastructure to function into a more uniform architecture with more automated processes.

In PayPal's case, the OpenStack transition wasn't only a move to a more automated infrastructure. It was an internal cultural change as well, said Shivananda. The change undertaken by the IT staff "goes well beyond server provisioning," he said.

(Image: PayPal)

(Image: PayPal)

PayPal knew it wanted to revamp its data center infrastructure in 2011, when OpenStack was little more than a young work-in-progress. It experimented with building out a more automated infrastructure, relying in part on VMware virtualization.

OpenStack caught its attention early on, however, and eventually "there were five to six versions of it" among the  experimental implementations that PayPal software teams were working with, said Shivananda. "After a while, we had to converge the stack," he said. "That helped us learn a lot about managing OpenStack."

PayPal's OpenStack Upgrade Process

PayPal has also been forced to upgrade OpenStack as a new release appears, roughly every six months. "It's an eye-opening experience to upgrade OpenStack," said Shivananda. "We have built a ton of experience around the upgrade process."

[Want to learn more? Read PayPal Finds Node.js Secret To Successful Makeover.]

The organization has adopted a set of processes and procedures around an OpenStack upgrade, which includes establishing a war room and appointing a commander of the process. According to Shivananda, it's important "to bring consistency to the table and take fleet-wide actions," keeping the 8,500 server mass "homogenous" and not allowing it to drift into segments operating under different versions of OpenStack. That means 180,000 data center assets – servers, top-of-rack switches, firewalls, load balancers, and storage volumes -- all function as part of the PayPal OpenStack cloud.

Doing so allows IT to operate PayPal data centers in a routine and automated way, Shivananda continued. If a data center server, switch, or storage volume failed in the past, it was common practice to send a staff member to fix the issue as soon as possible. Under OpenStack, the processing on a failing device is switched over to healthy ones. Mechanical failures are tolerated in the OpenStack cloud until it's time for a routine sweep that fixes or removes all types of failed devices and brings replacements online. Instead of technicians working on, say, 1% of stalled, troubled, and failed devices, they may be working on as many as 3% to 5%, under the periodic sweep method of operating, he said.

The PayPal cloud has its own sensing mechanisms to detect when hardware is acting up or about to fail.  

The main goal behind the automated method of operations is to provision PayPal development teams quickly when they need a set of servers. In the rapidly changing field of mobile payments, PayPal is keeping up by allowing frequent updates to dozens of applications.

"There's not an hour in the day when we're not rolling out software patches and updates," Shivananda said. "We're a high frequency of change environment." That would not be possible without the conversion to an OpenStack infrastructure. The uniformity of what makes up the infrastructure, and the predictability of how it will run, make it possible to impose frequent software changes.

The IT staff's next challenge is to incorporate use of Docker containers into its OpenStack cloud, Shivananda said.

Asked to clarify an old controversy, Shivananda said PayPal, once a VMware virtualization shop, has backed off its dependence on VMware and uses OpenStack for its compute virtualization. That means it relies on the OpenStack default, KVM open source hypervisor on many compute hosts, instead of VMware's ESX Server. However, he added, "VMware remains core in our network virtualization." Virtual machine provisioning and lifecycle management are now done under OpenStack.

Two years ago, Boris Renski, chief marketing officer for Mirantis, an OpenStack consulting firm that was doing work at PayPal at the time, said that PayPal would convert to OpenStack and parent company eBay would follow, replacing VMware in the process.

Shivananda didn't comment on eBay's infrastructure plans, other than to say both companies have their own plans and schedules to get to an OpenStack cloud. PayPal and eBay are in the process of becoming separate companies, and each has its own infrastructure. The separation is expected to become official in the second half of 2015.

Asked if PayPal used a private label version of OpenStack -- both Mirantis and VMware produce their own distributions -- Shivananda said it had developed its own staff expertise and installed its own version, without relying on an outside vendor's configuration.

Are you an IT leader wrestling with data center strategy? Come discuss this and other strategic IT issues with your peers at the InformationWeek Conference. You can email InformationWeek editor Chris Murphy with questions, or register now and save $200 off Conference passes with discount code CMBLOG. Other speakers include the CIOs of Walmart, AstraZeneca, the NBA, ConocoPhillips, Royal Caribbean, and more, along with 3D Robotics CEO Chris Anderson, discussing drones and the future of automation.

Charles Babcock is an editor-at-large for InformationWeek and author of Management Strategies for the Cloud Revolution, a McGraw-Hill book. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 2   >   >>
Charlie Babcock
50%
50%
Charlie Babcock,
User Rank: Author
4/3/2015 | 8:04:04 PM
PayPal private cloud equals 4,100 servers
The 8,500 servers originally referenced in the third graph have been scaled back by PayPal spokesmen to 4,100 devoted to PayPal opertions. The 8,500 number inadervertently included eBay's Marketplace servers. The separation of PayPal and eBay resources is still underway, as the two prepare to separate later this year.
batye
50%
50%
batye,
User Rank: Ninja
4/2/2015 | 1:27:50 AM
Re: PayPal's hard won expertise
@ChrisMurphy, thanks for sharing this info... other way I would miss it... thank you
batye
50%
50%
batye,
User Rank: Ninja
4/2/2015 | 1:26:46 AM
Re: Paypal Declares It's 100% OpenStack Cloud
@GAProgrammer interesting point... I think PayPal could afford it... how I see it... 
batye
50%
50%
batye,
User Rank: Ninja
4/2/2015 | 1:20:26 AM
Re: PayPal's hard won expertise
@pcharles same question here "Right documentation" ??? could you please share more... what is refered to...???
batye
100%
0%
batye,
User Rank: Ninja
4/2/2015 | 1:17:38 AM
Re: Paypal Declares It's 100% OpenStack Cloud
@zerox203, interesting observation, I think Paypal must change a lot of the thing to be competative in the market this days...
GAProgrammer
100%
0%
GAProgrammer,
User Rank: Ninja
4/1/2015 | 9:20:50 AM
Re: Paypal Declares It's 100% OpenStack Cloud
@Xerox - I agree with most of your post, except the last line - essentially, if PayPal can do it anyone can. Not everyone has the talent, the team size, or the deep pockets to pull this kind of move off. While it is easy for many of us in IT to see these types of benefits, the corporate suite isn't always on board. Something like this definintely takes vision by the CEO and not all companies have a CEO like that. I will agree that companies that have that trifecta should be doing it yesterday.
shakeeb
50%
50%
shakeeb,
User Rank: Ninja
3/31/2015 | 11:46:20 PM
Re: PayPal's hard won expertise
@pcharles – Were you referring to user manuals and technical manuals when referring to "Right documentation" ?         
shakeeb
50%
50%
shakeeb,
User Rank: Ninja
3/31/2015 | 11:45:05 PM
Re: PayPal's hard won expertise
@Charlie – It's great to see PayPal achieving the objective of moving to OpenStack, I hope that they could retain that status without any operational problems. 
ChrisMurphy
100%
0%
ChrisMurphy,
User Rank: Author
3/31/2015 | 11:18:33 PM
Re: PayPal's hard won expertise
Mr. Shivananda will be speaking about the future of the data center at the InformationWeek Conference, April 27 & 28 in Las Vegas. There's still time for IT leaders to join us there -- check InformationWeek.com/conference for details and registration. 
zerox203
100%
0%
zerox203,
User Rank: Ninja
3/31/2015 | 10:47:08 PM
Re: Paypal Declares It's 100% OpenStack Cloud
Thanks for the coverage, Charlie. We hear a lot about the steam being picked up by OpenStack, OpenDaylight, etc. implementations at larger and larger companies, but few could be as notable as this one. I would say when PayPal speaks up, others (particularly in their industry) are inclined to listen. Financial companies citing endless compliance concerns, legacy systems they just can't do without, and culture that just won't change, may find themselves running out of excuses. PayPal made all that work and, to hear them tell it, gained a positive net benefit in their culture and in their processes in ways we wouldn't typically consider directly related to their infrastructure. One would think they can and have applied that schema of patching and upgrading to other areas of their business.

Even though Paypal was a pioneer in online payments, their name still comes to mind when you think of stodgy giants that are loathe to change. I personally dreaded to see Paypal as the only available payment option for whatever I was buying once upon a time, but have noticed stark improvements in their  convenience (re: speed) and customer service in recent years. I suspect this not unrelated to all the developments Mr. Shivananda is talking about here. I'd call this a lesson on how an agility-oriented shakeup can have a direct benefit on customers and their perception of you. I'd also say that if someone in PayPal's position (big, monolithic, security-oriented) can do it, anyone can do it.
Page 1 / 2   >   >>
Commentary
Enterprise Guide to Edge Computing
Cathleen Gagne, Managing Editor, InformationWeek,  10/15/2019
News
Rethinking IT: Tech Investments that Drive Business Growth
Jessica Davis, Senior Editor, Enterprise Apps,  10/3/2019
Slideshows
IT Careers: 12 Job Skills in Demand for 2020
Cynthia Harvey, Freelance Journalist, InformationWeek,  10/1/2019
White Papers
Register for InformationWeek Newsletters
State of the Cloud
State of the Cloud
Cloud has drastically changed how IT organizations consume and deploy services in the digital age. This research report will delve into public, private and hybrid cloud adoption trends, with a special focus on infrastructure as a service and its role in the enterprise. Find out the challenges organizations are experiencing, and the technologies and strategies they are using to manage and mitigate those challenges today.
Video
Current Issue
Getting Started With Emerging Technologies
Looking to help your enterprise IT team ease the stress of putting new/emerging technologies such as AI, machine learning and IoT to work for their organizations? There are a few ways to get off on the right foot. In this report we share some expert advice on how to approach some of these seemingly daunting tech challenges.
Slideshows
Flash Poll