How Ancestry.com Manages Big Data - InformationWeek
IoT
IoT
Software
News
9/10/2012
03:01 PM
Connect Directly
Google+
RSS
E-Mail
50%
50%

How Ancestry.com Manages Big Data

Popular genealogy site uses Hadoop to manage 4 petabytes of family histories, including DNA data.

If you've ever had a yearning to uncover your family roots, you're probably familiar with Ancestry.com, the largest genealogy website with around 2 million paying subscribers. Not surprisingly, maintaining a data store of some 40,000 record collections, including birth, census, death, immigration, and military records, can be challenging.

"Ancestry's been dealing with big data for a long time. We've been around for 15 years," said Scott Sorensen, Ancestry.com's head of engineering, in a phone interview with InformationWeek.

The digitized information that Ancestry warehouses is quite varied. "We've got collections that are just postcards or yearbooks. So not only do we have a massive amount of data, but we also have a great diversity of data," said Sorensen.

DNA matching is a growing area of focus for Ancestry--and one that's very data intensive. "We'll send you a DNA kit. You spit in the tube and send it back to us," Sorensen said.

Ancestry then sequences the customer's autosomal DNA, the 22 matched pairs of chromosomes (one from your mother, one from your father) that are recombined each generation. An autosomal DNA match between two people indicates a potential genetic link.

"We'll take 700,000 snips--or locations--on that DNA, and use that to determine your ethnicity, and to match you against anybody else in our DNA database," said Sorensen.

[ See DNA On Microchips: Big Data's Future Storage Answer? ]

This, of course, requires a massive amount of processing.

"We've had such a demand for this DNA offering, and it requires us to process such a massive amount of data," Sorensen said. "You can imagine, 700,000 snips for a single person. And then we have to take every new test we get and match it against every other person in the database."

Today, Ancestry uses the Hadoop platform for its data management system. But before the arrival of Hadoop--which is only a few years old--Ancestry used a homegrown system that allowed it to do massively parallel processing of data. The in-house system did the job, Sorensen said, but Hadoop does it better.

After using Cloudera's implementation of Hadoop, Ancestry recently switched to MapR's offering. One reason was that Map R offers high availability of both name node and job tracker.

For instance, name node high availability is crucial to Ancestry's DNA-match service.

"We've had such a demand for this DNA offering," said Sorensen. "In order to keep up with the demand, we simply cannot afford to have the system go down. We might take a thousand samples on any given day and put it through our process to determine ethnicity, and to do DNA matching."

Despite Hadoop's attributes, the platform is hard to use. Sorensen would like to see easier data mining and analysis tools that allow less technical users to access Ancestry's massive data volumes.

There are great products and features we could deploy, if we could just unleash the power of this data," said Sorensen.

"Today, we might have a great idea, but it's got to get in the queue for the data scientists to work on," he said. "Because in order to leverage the data, somebody has to write MapReduce code, or maybe even do some machine learning algorithms."

Sorensen added, "If we had tools that allowed business analysts to do some of that work at a higher level, we could extract value from the data more quickly."

Better data management tools would also allow Ancestry to help customers who get stuck in their research--and perhaps even allow the company to do more genealogical work on the user's behalf. This, in turn, could broaden the service's appeal.

"There are only so many research-oriented genealogists out there," said Sorensen.

The ability to use DNA-match results to better leverage Ancestry's data store is another goal.

Said Sorensen, "We're going to tell you something about your family history, and leverage data in ways that allow us to provide our product and service to a broader audience."

Web-facing databases have a huge target on their backs. The easy way to secure these databases would be to take them off the Web, but a system that does not serve a business function is worthless to the company. Security pros must figure out ways to secure databases with limited resources while keeping business systems operational. In the Strategies For Protecting Web-Facing Databases report, we recommend several cost-efficient methods. (Free registration required.)

Comment  | 
Print  | 
More Insights
Comments
Threaded  |  Newest First  |  Oldest First
How Enterprises Are Attacking the IT Security Enterprise
How Enterprises Are Attacking the IT Security Enterprise
To learn more about what organizations are doing to tackle attacks and threats we surveyed a group of 300 IT and infosec professionals to find out what their biggest IT security challenges are and what they're doing to defend against today's threats. Download the report to see what they're saying.
Register for InformationWeek Newsletters
White Papers
Current Issue
2017 State of the Cloud Report
As the use of public cloud becomes a given, IT leaders must navigate the transition and advocate for management tools or architectures that allow them to realize the benefits they seek. Download this report to explore the issues and how to best leverage the cloud moving forward.
Video
Slideshows
Twitter Feed
InformationWeek Radio
Archived InformationWeek Radio
Join us for a roundup of the top stories on InformationWeek.com for the week of November 6, 2016. We'll be talking with the InformationWeek.com editors and correspondents who brought you the top stories of the week to get the "story behind the story."
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll