How Manages Big Data - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

03:01 PM
Connect Directly

How Manages Big Data

Popular genealogy site uses Hadoop to manage 4 petabytes of family histories, including DNA data.

If you've ever had a yearning to uncover your family roots, you're probably familiar with, the largest genealogy website with around 2 million paying subscribers. Not surprisingly, maintaining a data store of some 40,000 record collections, including birth, census, death, immigration, and military records, can be challenging.

"Ancestry's been dealing with big data for a long time. We've been around for 15 years," said Scott Sorensen,'s head of engineering, in a phone interview with InformationWeek.

The digitized information that Ancestry warehouses is quite varied. "We've got collections that are just postcards or yearbooks. So not only do we have a massive amount of data, but we also have a great diversity of data," said Sorensen.

DNA matching is a growing area of focus for Ancestry--and one that's very data intensive. "We'll send you a DNA kit. You spit in the tube and send it back to us," Sorensen said.

Ancestry then sequences the customer's autosomal DNA, the 22 matched pairs of chromosomes (one from your mother, one from your father) that are recombined each generation. An autosomal DNA match between two people indicates a potential genetic link.

"We'll take 700,000 snips--or locations--on that DNA, and use that to determine your ethnicity, and to match you against anybody else in our DNA database," said Sorensen.

[ See DNA On Microchips: Big Data's Future Storage Answer? ]

This, of course, requires a massive amount of processing.

"We've had such a demand for this DNA offering, and it requires us to process such a massive amount of data," Sorensen said. "You can imagine, 700,000 snips for a single person. And then we have to take every new test we get and match it against every other person in the database."

Today, Ancestry uses the Hadoop platform for its data management system. But before the arrival of Hadoop--which is only a few years old--Ancestry used a homegrown system that allowed it to do massively parallel processing of data. The in-house system did the job, Sorensen said, but Hadoop does it better.

After using Cloudera's implementation of Hadoop, Ancestry recently switched to MapR's offering. One reason was that Map R offers high availability of both name node and job tracker.

For instance, name node high availability is crucial to Ancestry's DNA-match service.

"We've had such a demand for this DNA offering," said Sorensen. "In order to keep up with the demand, we simply cannot afford to have the system go down. We might take a thousand samples on any given day and put it through our process to determine ethnicity, and to do DNA matching."

Despite Hadoop's attributes, the platform is hard to use. Sorensen would like to see easier data mining and analysis tools that allow less technical users to access Ancestry's massive data volumes.

There are great products and features we could deploy, if we could just unleash the power of this data," said Sorensen.

"Today, we might have a great idea, but it's got to get in the queue for the data scientists to work on," he said. "Because in order to leverage the data, somebody has to write MapReduce code, or maybe even do some machine learning algorithms."

Sorensen added, "If we had tools that allowed business analysts to do some of that work at a higher level, we could extract value from the data more quickly."

Better data management tools would also allow Ancestry to help customers who get stuck in their research--and perhaps even allow the company to do more genealogical work on the user's behalf. This, in turn, could broaden the service's appeal.

"There are only so many research-oriented genealogists out there," said Sorensen.

The ability to use DNA-match results to better leverage Ancestry's data store is another goal.

Said Sorensen, "We're going to tell you something about your family history, and leverage data in ways that allow us to provide our product and service to a broader audience."

Web-facing databases have a huge target on their backs. The easy way to secure these databases would be to take them off the Web, but a system that does not serve a business function is worthless to the company. Security pros must figure out ways to secure databases with limited resources while keeping business systems operational. In the Strategies For Protecting Web-Facing Databases report, we recommend several cost-efficient methods. (Free registration required.)

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
What Becomes of CFOs During Digital Transformation?
Joao-Pierre S. Ruth, Senior Writer,  2/4/2020
Fighting the Coronavirus with Analytics and GIS
Jessica Davis, Senior Editor, Enterprise Apps,  2/3/2020
IT Careers: 10 Job Skills in High Demand This Year
Cynthia Harvey, Freelance Journalist, InformationWeek,  2/3/2020
White Papers
Register for InformationWeek Newsletters
State of the Cloud
State of the Cloud
Cloud has drastically changed how IT organizations consume and deploy services in the digital age. This research report will delve into public, private and hybrid cloud adoption trends, with a special focus on infrastructure as a service and its role in the enterprise. Find out the challenges organizations are experiencing, and the technologies and strategies they are using to manage and mitigate those challenges today.
Current Issue
IT 2020: A Look Ahead
Are you ready for the critical changes that will occur in 2020? We've compiled editor insights from the best of our network (Dark Reading, Data Center Knowledge, InformationWeek, ITPro Today and Network Computing) to deliver to you a look at the trends, technologies, and threats that are emerging in the coming year. Download it today!
Flash Poll