Big Data's Surprising Uses: From Lady Gaga To CIA

Earth's polar regions aren't particularly hospitable to computers or humans. Building an advanced data management and storage system -- one that operates on bumpy flights in frigid weather, no less -- can be a formidable task.

Doing so, in fact, is akin to building "a supercomputer that can fly," said Richard Knepper, manager of a campus bridging and research infrastructure team for Indiana University's (IU) Research Technologies, a cyber-infrastructure and service center.

Knepper has a lot of experience in this area. For the past four years, his team at IU has provided IT support for one of several research groups participating in NASA's Operation IceBridge mission, an ongoing effort to help scientists study global climate change by collecting radar data on Earth's polar ice sheets.

[ Analytic proof that business trips take a toll? Read Travel Stress Quantified Using Big Data. ]

The IU team provides data management services for the Center for Remote Sensing of Ice Sheets (CReSIS), a National Science Foundation Science and Technology Center led by the University of Kansas (KU).

In a phone interview with InformationWeek, Knepper discussed the challenges of processing large amounts of data in real time aboard research planes, many of which fly bumpy, 12-hour missions over vast ice sheets covering Antarctica, Chile and Greenland.

The CReSIS team provides radar technology that measures physical interactions of ice sheets in polar regions. IU researchers are responsible for storing and processing multiple terabytes of data collected by these radar systems, both in flight during the missions, and later on the ground. CReSIS "has a computer that turns the analog signals into digital signals, and then we provide a storage solution that captures and verifies (data) as it comes off the instrument," said Knepper.

The data management system provides some processing capability in the plane, as well as additional processing on the ground for more in-depth analysis.

Knepper's team tested a new in-flight data copy system, one that allows real-time data processing and archiving, during a research flight last fall to Antarctica. In addition to saving time and money, the system gave researchers immediate access to their data, allowing them to determine in flight which ice sheets needed closer examination.

Falling prices of solid state drives (SSDs) have helped the IU team improve its processing and storage systems. "What we built last fall, and what we've reconfigured for the spring mission, is something we couldn't have done two or three years ago," Knepper said.

He added: "In the future, data reads are going to go up. We're looking at a field mission next fall that will collect half a petabyte of data. So we have to be prepared to collect and manage 500 terabytes, and have it be accessible to the team while they're working."

IU's system includes three servers, each with 24 solid state drives. SSDs function better than hard drives in harsh environments, particularly when planes are stored outside overnight in subzero temperatures.

The IU system includes a fiber channel array that does use mechanical drives for data backups, however.

An Operation IceBridge mission isn't for the easily nauseated.

"Most people describe it as gut-wrenching. It can be pretty wild," said Knepper. "When you're on one of these airplanes, it's not like a commercial flight. They're doing 30-degree bank turns. They fly in very bad weather sometimes."

One of the aircraft used is a P-3, a propeller-driven plane that "tends to have quite a bit of shock and vibration," he added.

E2 is the only event of its kind, bringing together business and technology leaders across IT, marketing, and other lines of business looking for new ways to evolve their enterprise applications strategy and transform their organizations to achieve business value. Join us June 17-19 for three days of 40+ conference sessions and workshops across eight tracks and discover the latest insights in enterprise social software, big data and analytics, mobility, cloud, SaaS and APIs, UI/UX and more. Register for E2 Conference Boston today and save $200 off Full Event Passes, $100 off Conference, or get a FREE Keynote + Expo Pass!