PayPal Says Linux Grid Can Replace Mainframes

A Linux grid is the power behind the payment system at PayPal, and it's converted a mainframe believer. Scott Thompson, the former executive VP of technology solutions at Inovant, ran the Visa subsidiary responsible for executing Visa credit card transactions worldwide. The VisaNet system was strictly based on IBM mainframes.

In February 2005, Thompson became chief technology officer at the eBay payments company, PayPal, where he confronted a young Internet organization building its entire transaction processing infrastructure on open source Linux and low-cost servers. Hmmmm, he thought at the time.

"I came from Visa, where I had responsibility for VisaNet. It was a fabulous processing system, very big and very global. I was intrigued by PayPal. How would you use Linux for processing payments and never be wrong, never lose messages, never fall behind the pace of transactions," he recalled in an interview.

He now supervises the PayPal electronic payment processing system, which is smaller than VisaNet in volume and total dollar value of transactions. But it's growing fast. It is currently processing $1,571 worth of transactions per second in 17 different currencies. In 2006, the online payments firm, which started out over a bakery in Palo Alto, processed a total of $37.6 billion in transactions. It's headed toward $50 billion this year.

Now located in San Jose, PayPal grants its consumer members options in payment methods: credit cards, debit cards, or directly from a bank account. It has 165 million account holders worldwide, and has recently added such business as Northwest Airlines, Southwest Airlines, U.S. Airways, and Overstock.com, which now permit PayPal payments on their Web sites.

Thompson supervises a payment system that operates on about 4,000 servers running Red Hat Linux in the same manner that eBay and Google conduct their business on top of a grid of Linux servers. "I have been pleasantly surprised at how much we've been able to do with this approach. It operates like a mainframe," he said.

As PayPal grows it's much easier to grow the grid with Intel-based servers than it would be to upgrade a mainframe, he said. In a mainframe environment, the cost to increase capacity a planned 15% or 20% "is enormous. It could be in the tens of millions to do a step increase. In [PayPal's] world, we add hundreds of servers in the course of a couple of nights and the cost is in the thousands, not millions," he said.

PayPal takes Red Hat Enterprise Linux and strips out all features unnecessary to its business, then adds proprietary extensions around security. Another virtue of the grid is that PayPal's 800 engineers can all get a copy of that customized system on their development desktops, run tests on their raw software as they work, and develop to PayPal's needs faster because they're working in the target environment. That's harder to do when the core of the data center consists of large Unix symmetrical multiprocessing boxes or mainframes. In neither case is it cheap to install duplicates for developers, he said.

PayPal "pays very close attention to the Linux kernel development process" lead by Linus Torvalds and the kernel maintainers because future capabilities are being debated and resolved through the process, he said.

PayPal has experimented with virtualization and is watching carefully developments in open source virtualization, still a young field. "One place we see the kernel process at work is in virtualization," Thompson said. VMware's ESX Server can run Linux, as can the open source Xen hypervisor; both work outside the Linux kernel but can be linked to its internal operations. A year ago, Torvalds approved the addition of a contributed Kernel Virtual Machine, which runs inside the kernel and makes use of the kernel's own memory management and other functions.

"If we could fully virtualize our middle tier, that would be another step of cost advantage," said Thompson. More fully virtualized data centers also would allow him to shift workloads across the grid, depending on time of day and traffic volumes, which would lead to additional savings.

"We'd love to shift processing capacity to workloads. That would be a tremendous benefit," and slow the need to buy and manage blade servers, even as PayPal continues to grow, he said.

Virtualization also would help him avoid building out another "power center" or new unit of the data center located close to cheap power. Avoiding added power costs is a much higher priority than it was a few years ago, he said.

But he's not ready to virtualize batches of servers here and there. He wants a plan that will allow "virtualization with a layer of intelligence on top of it" to manage virtual resources. He is experimenting with VMware, but before he implements virtualization throughout the data center, he'll wait until he sees the right intelligence materialize to make it manageable.

Does relying on Linux worry him when Acacia Research, through its subsidiary IP Innovation, filed suit in October against Red Hat and Novell for violating its patent portfolio? And Microsoft claims its patents are violated by Linux? Thompson said: "I'm not worried about those statements. I'm familiar with the issue but I'm not worried about it. We have people in our business who are on top of it."