How Ancestry.com Manages Big Data

If you've ever had a yearning to uncover your family roots, you're probably familiar with Ancestry.com, the largest genealogy website with around 2 million paying subscribers. Not surprisingly, maintaining a data store of some 40,000 record collections, including birth, census, death, immigration, and military records, can be challenging.

"Ancestry's been dealing with big data for a long time. We've been around for 15 years," said Scott Sorensen, Ancestry.com's head of engineering, in a phone interview with InformationWeek.

The digitized information that Ancestry warehouses is quite varied. "We've got collections that are just postcards or yearbooks. So not only do we have a massive amount of data, but we also have a great diversity of data," said Sorensen.

DNA matching is a growing area of focus for Ancestry--and one that's very data intensive. "We'll send you a DNA kit. You spit in the tube and send it back to us," Sorensen said.

Ancestry then sequences the customer's autosomal DNA, the 22 matched pairs of chromosomes (one from your mother, one from your father) that are recombined each generation. An autosomal DNA match between two people indicates a potential genetic link.

"We'll take 700,000 snips--or locations--on that DNA, and use that to determine your ethnicity, and to match you against anybody else in our DNA database," said Sorensen.

[ See DNA On Microchips: Big Data's Future Storage Answer? ]

This, of course, requires a massive amount of processing.

"We've had such a demand for this DNA offering, and it requires us to process such a massive amount of data," Sorensen said. "You can imagine, 700,000 snips for a single person. And then we have to take every new test we get and match it against every other person in the database."

Today, Ancestry uses the Hadoop platform for its data management system. But before the arrival of Hadoop--which is only a few years old--Ancestry used a homegrown system that allowed it to do massively parallel processing of data. The in-house system did the job, Sorensen said, but Hadoop does it better.

After using Cloudera's implementation of Hadoop, Ancestry recently switched to MapR's offering. One reason was that Map R offers high availability of both name node and job tracker.

For instance, name node high availability is crucial to Ancestry's DNA-match service.

"We've had such a demand for this DNA offering," said Sorensen. "In order to keep up with the demand, we simply cannot afford to have the system go down. We might take a thousand samples on any given day and put it through our process to determine ethnicity, and to do DNA matching."

Despite Hadoop's attributes, the platform is hard to use. Sorensen would like to see easier data mining and analysis tools that allow less technical users to access Ancestry's massive data volumes.

There are great products and features we could deploy, if we could just unleash the power of this data," said Sorensen.

"Today, we might have a great idea, but it's got to get in the queue for the data scientists to work on," he said. "Because in order to leverage the data, somebody has to write MapReduce code, or maybe even do some machine learning algorithms."

Sorensen added, "If we had tools that allowed business analysts to do some of that work at a higher level, we could extract value from the data more quickly."

Better data management tools would also allow Ancestry to help customers who get stuck in their research--and perhaps even allow the company to do more genealogical work on the user's behalf. This, in turn, could broaden the service's appeal.

"There are only so many research-oriented genealogists out there," said Sorensen.

The ability to use DNA-match results to better leverage Ancestry's data store is another goal.

Said Sorensen, "We're going to tell you something about your family history, and leverage data in ways that allow us to provide our product and service to a broader audience."

Web-facing databases have a huge target on their backs. The easy way to secure these databases would be to take them off the Web, but a system that does not serve a business function is worthless to the company. Security pros must figure out ways to secure databases with limited resources while keeping business systems operational. In the Strategies For Protecting Web-Facing Databases report, we recommend several cost-efficient methods. (Free registration required.)