"Know thy game," intoned Leinwand, and it turned out that the Zynga staff hadn't known very well the pathways of the complicated interactions.
Other bottlenecks were found in the networks to storage systems, Internet traffic moving through Web servers, firewalls' ability to process the streams of traffic, and load balancers' ability to keep up with constantly shifting demand.
Zynga went a step further. Since some of its game activities require connections to Facebook applications or take place in Facebook applications, it sought locations for its zCloud data centers that were in the same region as Facebook's. Likewise, with its continued use of Amazon, it sought to identify "super regions" where the data centers of all could be in close proximity, reducing delays caused by geographic separation.
"We think of all of our operations in the super region as a really big data center," said Leinwand. A Zynga game data center is directly connected to an Amazon EC2 data center by fiber optic cable, allowing Zynga to shift workloads over a high-speed interconnect.
With both Amazon and Facebook, "latency within a super region is in the single digit milliseconds. They are very tightly coupled pieces of infrastructure," he said.
Zynga uses Citrix Systems CloudStack as its virtual machine management interface superimposed on all zCloud VMs, regardless of whether they're in the public cloud or private cloud.
Asked when he would complete his move out of the public cloud into zCloud, Leinwand paused, then said, "I don't know if I've thought about it that way. I want both in our toolbox. I want to launch games in zCloud and have the flexibility of bursting to Amazon."
In the interview, he was more explicit. "We own the base, rent the spike. We want a hybrid operation. We love knowing that shock absorber is there."
Charles Babcock is an editor-at-large for InformationWeek.
IT's jumping into cloud services with too much custom code and too little planning, our annual State of Cloud Computing Survey finds. The new Leap Of Cloud Faith issue of InformationWeek shows you what to be aware of when using the cloud. Also in this issue: Cloud success stories from Six Flags and Yelp, and how to write a SAN RFI. (Free registration required.)