As a planned six-month Mars rover mission has stretched into a decade, the Analyst's Notebook app has morphed to manage the increasing amount of data generated.
NASA's Next Spacesuit: Mars Fashion
(Click image for larger view and slideshow.)
Mission creep -- and the pain of dealing with unexpected demands on software -- is an inevitable part of every software developer's job. But Tom Stein, the computer systems manager at NASA's Planetary Data Systems Geosciences Node at Washington University, has experienced a level of mission creep that few other software writers can claim.
Stein is the primary author of Analyst's Notebook, a front-end application to help researchers review data on the unfolding of a planetary mission. When the Mars Exploration rovers, Spirit and Opportunity, landed on opposite sides of the Red Planet's surface in 2004, they had a hoped-for life expectancy of 90 -- or at most, 180 -- days. Spirit finally fell silent in March 2010. But 10 years later, Opportunity is still sending back data transmissions. In 2012, it was joined on the Martian surface by Curiosity, a Science Laboratory rover, adding to NASA's knowledge of Mars and to Stein's data management headaches.
With data collection growing more complex, Stein's unit had outgrown many types of data collection and retrieval that had sufficed for earlier missions. (The data from the 1975 Viking I and II missions to Mars are archived on a 42-CD set.) Through Analyst's Notebook's user interface, data from Spirit, Opportunity, and Curiosity can be accessed via a Web service. Behind the front end, Microsoft's SQL Server retrieves the data off of Washington University servers in St. Louis. One Notebook serves the Spirit and Opportunity data users; a second, Curiosity data users. Combined, they represent 7 million to 8 million data products collected from various instruments at various mission stages, said Stein.
"Early on, we expected between 2,000 and 4,000 images over the course of the planned 90- to 180-day mission. After 10 years we have several hundred thousand images," said Stein in an interview after a Silicon Valley event spotlighting ambitious applications.
Early on, he and the four other NASA staffers who work part time on the Notebook interface realized they needed a broader platform for the app they were trying to build. He originally equipped it with a drop-down list that lets a researcher select the day of the mission for which he wishes to retrieve data. That list started out with 90 days for Opportunity, and it has been modified to accommodate the ongoing mission, now numbering more than 3,700 days.
The Notebooks must make the data searchable on a timed-sequence and geospatial basis. Simply time-stamping data might mean several things: recording the spacecraft clock's time, UTC (or Coordinated Universal Time), or the time of day on Mars, whose cycle is 37 minutes longer than Earth's. In addition, the Notebooks seek to correlate the data with thousands of documents that describe mission operations and activities: Why was the rover backed up at this time? What terrain did it encounter? What did the iron detected by the Moessbauer spectrometer in the rock at the end of Opportunity's arm indicate? (Iron is often associated with mineral interactions with water, something all three were geared to search for.)
Analyst's Notebook started out with a map to show a rover's movement for up to 600 meters (less than half a mile), its expected maximum. Opportunity, however, has covered 24.8 miles, setting a record for a vehicle in motion on a planet other than Earth. "This greater distance has forced us to change the way we create the terrain map and how we present it to the user in an interactive manner," said Stein. A satellite image of the route, for example, can be linked to associated surface-level rover images on the route.
Initially, Analyst's Notebook presented images in sequence and enabled researchers to query for images based on the time period in which they were taken. But the extended nature of the rover's treks, and the desire to correlate what was visible in the landscape to what was revealed by instruments, required a new method of image viewing. "It wasn't easy to do image inspection. It was easy to lose track of where you were," recounted Stein.
To continue building out the application, his unit acquired the Telerik UI ASPNET AJAX application platform from Telerik in Waltham, Mass. With it, the images could be assembled into broader pictures of the Martian landscape, giving researchers more context for what they were viewing. The feature became known as "context mosaics" in the Notebooks.
The tool bars, drop-down menus, and other features of Analyst's Notebook have expanded to support more data and wider timeframes than initially allowed. Stein said the information has become so accessible that "planetary scientists from around the world use it. We average 300 to 400 Notebook users a day, several thousand per month." In addition, the information is available to the general public, part of NASA's effort to make the results of spending tax dollars available to taxpayers.
Stein's own understanding of planetary data hasn't hurt. He majored in geology as an undergrad before getting his Master's in information management. "He's an expert in the data itself," said Todd Anglin, executive VP of cross-platform and tools at Telerik. "It makes a difference in how the data is presented."
Find out how NASA's Jet Propulsion Laboratory addressed governance, risk, and compliance for its critical public cloud services. Get the new Cloud Governance At NASA issue of InformationWeek Government Tech Digest today. (Free registration required.)
Charles Babcock is an editor-at-large for InformationWeek, having joined the publication in 2003. He is the former editor-in-chief of Digital News, former software editor of Computerworld and former technology editor of Interactive Week. He is a graduate of Syracuse ... View Full Bio
The Agile ArchiveWhen it comes to managing data, donít look at backup and archiving systems as burdens and cost centers. A well-designed archive can enhance data protection and restores, ease search and e-discovery efforts, and save money by intelligently moving data from expensive primary storage systems.
2014 Analytics, BI, and Information Management SurveyITís tried for years to simplify data analytics and business intelligence efforts. Have visual analysis tools and Hadoop and NoSQL databases helped? Respondents to our 2014 InformationWeek Analytics, Business Intelligence, and Information Management Survey have a mixed outlook.
Join us for a roundup of the top stories on InformationWeek.com for the week of December 14, 2014. Be here for the show and for the incredible Friday Afternoon Conversation that runs beside the program.