NASA Polar Ice Flyover: One Bumpy Big Data Project
Indiana University researchers use data feedback in flight to guide NASA's ambitious airborne survey of Earth's polar ice. But bring a barf bag if you come along.
Big Data's Surprising Uses:
From Lady Gaga To CIA
(click image for larger view and for slideshow)
Earth's polar regions aren't particularly hospitable to computers or humans. Building an advanced data management and storage system -- one that operates on bumpy flights in frigid weather, no less -- can be a formidable task.
Doing so, in fact, is akin to building "a supercomputer that can fly," said Richard Knepper, manager of a campus bridging and research infrastructure team for Indiana University's (IU) Research Technologies, a cyber-infrastructure and service center.
Knepper has a lot of experience in this area. For the past four years, his team at IU has provided IT support for one of several research groups participating in NASA's Operation IceBridge mission, an ongoing effort to help scientists study global climate change by collecting radar data on Earth's polar ice sheets.
The IU team provides data management services for the Center for Remote Sensing of Ice Sheets (CReSIS), a National Science Foundation Science and Technology Center led by the University of Kansas (KU).
In a phone interview with InformationWeek, Knepper discussed the challenges of processing large amounts of data in real time aboard research planes, many of which fly bumpy, 12-hour missions over vast ice sheets covering Antarctica, Chile and Greenland.
The CReSIS team provides radar technology that measures physical interactions of ice sheets in polar regions. IU researchers are responsible for storing and processing multiple terabytes of data collected by these radar systems, both in flight during the missions, and later on the ground.
CReSIS "has a computer that turns the analog signals into digital signals, and then we provide a storage solution that captures and verifies (data) as it comes off the instrument," said Knepper.
The data management system provides some processing capability in the plane, as well as additional processing on the ground for more in-depth analysis.
Knepper's team tested a new in-flight data copy system, one that allows real-time data processing and archiving, during a research flight last fall to Antarctica. In addition to saving time and money, the system gave researchers immediate access to their data, allowing them to determine in flight which ice sheets needed closer examination.
Falling prices of solid state drives (SSDs) have helped the IU team improve its processing and storage systems. "What we built last fall, and what we've reconfigured for the spring mission, is something we couldn't have done two or three years ago," Knepper said.
He added: "In the future, data reads are going to go up. We're looking at a field mission next fall that will collect half a petabyte of data. So we have to be prepared to collect and manage 500 terabytes, and have it be accessible to the team while they're working."
IU's system includes three servers, each with 24 solid state drives. SSDs function better than hard drives in harsh environments, particularly when planes are stored outside overnight in subzero temperatures.
The IU system includes a fiber channel array that does use
mechanical drives for data backups, however.
An Operation IceBridge mission isn't for the easily nauseated.
"Most people describe it as gut-wrenching. It can be pretty wild," said Knepper. "When you're on one of these airplanes, it's not like a commercial flight. They're doing 30-degree bank turns. They fly in very bad weather sometimes."
One of the aircraft used is a P-3, a propeller-driven plane that "tends to have quite a bit of shock and vibration," he added.
E2 is the only event of its kind, bringing together business and technology leaders across IT, marketing, and other lines of business looking for new ways to evolve their enterprise applications strategy and transform their organizations to achieve business value. Join us June 17-19 for three days of 40+ conference sessions and workshops across eight tracks and discover the latest insights in enterprise social software, big data and analytics, mobility, cloud, SaaS and APIs, UI/UX and more. Register for E2 Conference Boston today and save $200 off Full Event Passes, $100 off Conference, or get a FREE Keynote + Expo Pass!
6 Tools to Protect Big DataMost IT teams have their conventional databases covered in terms of security and business continuity. But as we enter the era of big data, Hadoop, and NoSQL, protection schemes need to evolve. In fact, big data could drive the next big security strategy shift.
Big Data Brings Big Security ProblemsWhy should big data be more difficult to secure? In a word, variety. But the business won’t wait to use it to predict customer behavior, find correlations across disparate data sources, predict fraud or financial risk, and more.
InformationWeek Tech Digest, Nov. 10, 2014Just 30% of respondents to our new survey say their companies are very or extremely effective at identifying critical data and analyzing it to make decisions, down from 42% in 2013. What gives?