How Amazon Kinesis Adds Speed, Resilience To Analytics

Three Kinesis users explain how they use AWS's real-time, streaming data capture and analytics service to support their businesses.

storage in AWS's S3 object storage, AWS's NoSQL system DyanamoDB, or AWS's Redshift data warehouse for reuse by longer-term analysis systems.

"All the edge server logs, all the real-time data flows are downloaded into Kinesis," Simmons said. In addition, a data stream from the bidding engine allows a Kinesis application to look at bids that won versus those that lost and look for events in the bidding stream that may be recurring. Perhaps DataXu can better position its bid the next time around.

Amazon's ability to supply "the durability, the availability of the data, and the throughput performance [in processing real-time data streams] are very critical to us," said Simmons.

Suhas Kulkani, VP of engineering and chief architect at Gree, the Japan-based builder of mobile games, such as Casino, Crime City, War of Nations, and Knights and Dragons, said his firm streams online game activity through Kinesis to examine the individual player experience, responses of thousands of players to specific events within a game, analysis of when players become buyers of the wares sold within games, and so forth.

Gree is an 1,800 employee company with studios in San Francisco, Vancouver, B.C., and Japan. While it started out on PCs, 90% of its user traffic is now from smartphones. It offers free versions of its game or makes them available as $1.99 downloads for iPhone and Android users.

"Gree's long-term success relies on game insights and game optimization. What is the load time, what is the experience of going from one screen to another, what is the response to a promotion within the game? We collect tons of information," said Kulkani.

Gree wants to know how new users fare in a Gree game and how quickly they progress into it. How do users respond to the new figures in the game or new twists to the game's flow? With such information, "we try to improve the overall experience," he said in an interview.

With such information, Gree has been able to maintain the popularity of its games in a competitive mobile gaming market. Modern War is two years old, long in the tooth for many mobile entries, but Kulkani says "it's still doing extremely well performance-wise." To accomplish that, the game must keep a healthy balance between the challenge it presents and "the win/lose experience of the player. We want a good win/lose ratio. It's very important for continued engagement," according to analysis of player activity on Kinesis.

Before moving its analytics onto Amazon Web Services, Gree relied on an in-house analytics platform, which was difficult to maintain as a highly available, highly resilient system. In addition to analyzing the player experience, Gree needs analytics to be available to marketing staff, game designers, and other groups in the company. Moving to Kinesis and other Amazon analytical services made results available throughout the company at all times.

Kinesis's near-real-time data analysis shows what happens when a game crashes, usually due to a hidden bug in the game logic. If it occurs with an early user, all the energy and attention that the user has invested in the game "is gone, and that player is not likely coming back," he said. The analysis provided by Gree applications on game activity illustrates that fact to game developers and acts as a spur to better programming and more thorough testing, he said.

Omnifone is another Kinesis user, though it first attempted to build systems in its own data centers to provide its Music Station platform to a wide variety of streaming music customers. It is a B2B music platform provider, with Sony Music Unlimited, Sirius XM, Guvera, and Rara among its customers.


Omnifone's Phil Sant, speaking at Re:Invent.

Phil Sant, founder and chief engineer, said he and his partners attempted to build a global firm without realizing the difficulty of meeting data center requirements for such an operation. Sony Music, in particular, lobbied Omnifone to improve its operations. If Omnifone was going to be Sony's supplier of streaming services, it needed to build a new data center with high reliability -- and a second data center as a disaster-recovery site. "I had to commit $15 million in capital expenses for those two data centers," Sant recalls.

Re-engineering a music service while streaming music in a rapidly growing business was like "rebuilding an airplane while it's in flight," he said. Omnifone sampled Amazon services in 2008 and began a steady transition into the cloud. Now he uses the Kinesis service for analytical applications and other parts of Amazon for compute and storage. "We sit on this cloud like a rising tide. We use 21 of their services," Sant said.

Omnifone is streaming data from its customers' music sites to see what new types of music are gaining popularity, what people prefer to hear on mobile devices, how musicians gain acceptance in different cultures, and what's being played in different parts of the world.

Sant has concluded from his experience that companies should be based on a scalable data capture and analytics system, such as Kinesis, rather than building their own. "There is nothing you should be running by yourself... If you're not rebuilding on Amazon, you will be killed by those who are."

Just 30% of respondents to our new Big Data and Analytics Survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives? Get the The Trouble With Big Data issue of InformationWeek Tech Digest today. (Free registration required.)