Unlike cloud providers, who run customers' workloads inside a virtual machine, eBay is a Web-scale company that has moved away from reliance on just virtual machines into greater use of containers.
The online marketplace is actually a good candidate for large scale container implementation. While it must answer hundreds of thousands of queries daily and engage in transactions coming in over the Internet, eBay nevertheless is in strict control of what code it's running. That means the risk is less than in a public cloud environment. At eBay containers may run in close proximity in system memory without the harder logical boundaries of a virtual machine.
Google may launch two billion containers a week, but that's for its search engine and internal operations, not the Google Cloud Platform. A Google cloud customer's workload still runs in a virtual machine. So if a container-oriented company like Google has to stick with VMs in its public cloud, eBay's path to large scale container adoption may prove instructive. One of its chief elements, not surprisingly, is the Kubernetes container cluster builder, which originated at Google.
Kubernetes became available in June 2014 as Google launched an open source project based on its experience in producing large scale clusters for containers in production. EBay has taken Kubernetes and put it on top of it its own tool for container administration within the cluster, Tess-Master.
EBay is currently running seven Kubernetes clusters with about 100 physical servers in each, said Suneet Nandwani, senior director of cloud infrastructure and platforms at eBay. It manages all of them through a single administrative interface provided by Tess-Master. EBay's internal version of Kubernetes is called Tess.IO, so Tess-master is the management framework for Tess.IO, eBay officials explained.
"We've been very purposeful about adopting Kubernetes," said Nandwani. EBay's development teams like using Docker containers in dev and test, and Kubernetes supports use of Docker and can accept handoffs of Docker containers. In addition to dev and test, eBay has started running production applications in containers. Each application actually requires a series of containers for its various parts and dependencies.
"We have been running four applications on Kubernetes this year. We're seeing a lot of potential for application efficiency and developer agility," said Nandwani. And in fact eBay has a huge appetite for running applications on clusters in the cloud, included its advanced search, machine learning and Hadoop systems. For all uses, it's running about 2,000 containers at the current time, a number it expects to scale up into the tens of thousands in the near future, Nandwani said.
PayPal, before being spun out of eBay, had similar goals. See how it pursued agility in PayPal's Path To Independence: A Tale Of Cloud Innovation.
EBay needed its own tool to deploy containers on Kubernetes because it is one of the world's five largest implementation of an OpenStack cloud. Where others attempted to adopt OpenStack and found the going tough, eBay implemented it, customized it and kept going. In effect, eBay runs nine OpenStack clouds with nine control planes, but they are tied together via what's known internally as the C3 cloud. The nine OpenStacks appear to be one cloud with one shared user interface to eBay users.
EBay won't disclose how large its OpenStack cloud is exactly, but some observers compare it to the number of physical servers available through Rackspace Hosting or IBM's SoftLayer cloud.
Part of the goal of converting to OpenStack was to free up developers with self-provisioning and produce the next generation of applications that would keep the business competitive. But if developers found it convenient to develop, test and hand off new code in containers, "running Docker by itself was very challenging," Nandwani said.
As containers proliferated, one of the few ways to stay on top of their spread was through a container cluster management system, whether open source Mesos, Docker Swarm, or Kubernetes. At the time eBay started searching for an answer, Docker Swarm had yet to emerge, Mesos was a young open source code project (later supported and productized through the firm Mesosphere) and Kubernetes was just emerging from Google's halls. Early in 2016, having played the field, eBay settled on Kubernetes as the most mature option.
"We did an internal evaluation and found that the Kubernetes and Docker (containers) combination was the best… We found our infrastructure usage could be a lot more efficient with Kubernetes," said Nandwani. Points in Kubernetes favor included its application-centric and policy-based approach to running containers, points that eBay developers and operations managers liked.
But Kubernetes needed to be adapted to the eBay's custom OpenStack C3 and other parts of the eBay infrastructure. EBay knew it had to do that part of container management itself and developed TessMaster as a result. With TessMaster, eBay can deploy containers on C3 Kubernetes clusters as Kubernetes handled the details of cluster creation itself. Through the interface, it can add capacity to the cluster or shrink it. If the cluster develops operational problems, administrators can remediate them through TessMaster.
Any changes to containers can be tracked and recorded in the enterprise change management database (CMDB) system through TessMaster and also propagated to other parts of the enterprise infrastructure.
TessMaster also handles integration issues with eBay's OpenStack cloud. Part of eBay's need to produce TessMaster is based on the fact that it's operating a large scale OpenStack cloud, with customization features, and there are few companies that could offer feedback.
"While popular, OpenStack was not easy to run at scale. There weren't that many companies that we could reach out to," recalls Nandwani.
He won't disclose the exact size, but C3 consists of "tens of thousands of physical servers and hundreds of thousands of compute servers (virtual machines). I would say it's one of the more mature OpenStack clouds in operation… We haven't encountered anyone who's bigger," Nandwani said.
For good measure, C3 hosts one of the largest Hadoop clusters in operation as well, he said.
EBay started out with a simpler cloud, Stratus, in 2010, evolved to an OpenStack cloud in 2012, and now operates its nine OpenStack clouds as a combined C3, with the help of Docker containers, Kubernetes cluster building and TessMaster container management. As this combination governs more of the eBay cloud, Nandwani said it will be labeled eBay's third generation cloud.
It's current state of the art are seven 100-server Kubernetes clusters under TessMaster, which might be viewed as container test projects. "We're seeing a lot of potential for application efficiency and developer agility," those long term goals that brought eBay into the business of establishing its own cloud operations in the first place.
Come 2017, eBay will be making "a big investment" in TessMaster, Kubernetes and container operations, he said. At some point, it will make public the results of its efforts by open sourcing TessMaster. But he set no timeframe within which that might happen. One place to watch for word on that is eBay's cloud technology blog.
EBay appears to be trying to take its place alongside other leaders in container implementations, especially Google and Docker. If it can solve problems of integrating containers into production operation, then there's likely to one day be great interest in seeing TessMaster takes its place alongside Docker and Kubernetes in container management.