Rubin Observatory Goes Open Source to Capture Galactic Data - InformationWeek

InformationWeek is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them.Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

IoT
IoT
Data Management
News
6/11/2021
08:00 AM
Connect Directly
Twitter
RSS
E-Mail
50%
50%

Rubin Observatory Goes Open Source to Capture Galactic Data

Optical observatory under construction using InfluxData to help it process data to be collected over time from the stars.

Vera C. Rubin ObservatoryCredit: RubinObs/NSF/AURA
Vera C. Rubin Observatory

Credit: RubinObs/NSF/AURA

Faced with a long-term project to gather and process vast amounts of visual data from the universe, the Vera C. Rubin Observatory in the mountains of Chile turned to an open source, time series database, InfluxDB, developed by InfluxData.

The observatory and its 8.4-meter optical telescope are being built to survey the region of space viewable from the southern hemisphere for 10 years, capture about 1,000 images of the sky on a nightly basis. The project, called the Legacy Survey of Space and Time, is expected to generate 500 petabytes of visual data astronomers should be able to use to better understand the cosmos.

The Rubin Observatory, funded by the National Science Foundation and the Department of Energy, will aim to gather data on some 37 billion stars and galaxies, and gain further insight on stellar phenomena such as dark matter, dark energy, and asteroid movement.

Operating complex astronomical telescopes requires a sound understanding of the instrumentation, says Frossie Economou, project manager for the Rubin Observatory Science Platform. Though the observatory is in Chile, scientists around the world have interest in the data, she says.

As the project progressed, Economou says the team realized they needed to focus on that intricate work rather than be tied up dealing with storing and processing a flood of instrument readings. When the 3-gigapixel camera, telescope, and other equipment are fully assembled, the observatory is expected to generate substantial data at a high frequency, she says. “The telemetry is high volume. Even without the telescope in full construction we are already collecting about a terabyte of telemetry a day.”

The team currently uses the open source version of InfluxDB, says Angelo Fausti, software engineer with the observatory, though they are updating to another tier. “We are currently planning the migration to InfluxDB 2.0,” he says. That migration will include a new user interface with new visualization capabilities for different scatter plots, heat maps, and histograms, Fausti says. “It’s a tool made for developers and we, as scientists and engineers, are also developers.”

The observatory made a prior attempt to build a traditional MySQL, relational database, Economou says, to store and analyze telemetry, but it was challenge. The team was already using Apache Kafka and InfluxData for a different use case at the time, she says, and recognized those resources could be used to collect data at a high frequency, volume, and throughput. “We realized that our telemetry was a very good fit for this,” she says. The observatory team then built their engineering facilities database using InfluxData and Kafka, an open-source platform for handling data feeds, to that end.

InfluxData has also been useful for troubleshooting the facility, Economou says. “You’re trying to understand the origin of a problem or behavior in your hardware,” she says. “Otherwise you’re flying blind.” The observatory, situated in the Chilean Andes at an elevation of 9,000 feet, requires an on-premise installation of InfluxData, Economou says, because of the potential for instability in the connections to the mountain summit. “There’s a lot of fiber between us and the telescope.”

Economou says the team uses Kafka to work with the large amount of telemetry data captured that needs to be replicated to a data facility at the National Center for Supercomputing Applications in Illinois. From there, the data can be aggregated as well as used to create statistical representations of the data, she says.

The observatory expects to commence survey operations in 2024, Economou says, and plans to generate chronographs via visualization through InfluxData, so data scientists and engineers can examine and interact with the data. “You’ll be able to see changes on a scale that has never been achieved before in astronomy,” she says.

Related Content:

The AI Ecosystem: Mapping the Future of Data Science

What CIOs Need to Know About Graph Database Technology

Capella Space Goes with AWS to Handle Satellite Downlinks

 

Joao-Pierre S. Ruth has spent his career immersed in business and technology journalism first covering local industries in New Jersey, later as the New York editor for Xconomy delving into the city's tech startup community, and then as a freelancer for such outlets as ... View Full Bio

We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
InformationWeek Is Getting an Upgrade!

Find out more about our plans to improve the look, functionality, and performance of the InformationWeek site in the coming months.

News
Becoming a Self-Taught Cybersecurity Pro
Jessica Davis, Senior Editor, Enterprise Apps,  6/9/2021
News
Ancestry's DevOps Strategy to Control Its CI/CD Pipeline
Joao-Pierre S. Ruth, Senior Writer,  6/4/2021
Slideshows
IT Leadership: 10 Ways to Unleash Enterprise Innovation
Lisa Morgan, Freelance Writer,  6/8/2021
White Papers
Register for InformationWeek Newsletters
2021 State of ITOps and SecOps Report
2021 State of ITOps and SecOps Report
This new report from InformationWeek explores what we've learned over the past year, critical trends around ITOps and SecOps, and where leaders are focusing their time and efforts to support a growing digital economy. Download it today!
Video
Current Issue
Planning Your Digital Transformation Roadmap
Download this report to learn about the latest technologies and best practices or ensuring a successful transition from outdated business transformation tactics.
Slideshows
Flash Poll