How Federal Agencies Use the Cloud To Power Real-Time Scientific Discovery

A project of the National Science Foundation and the DOE, the observatory is tracking changes in the night sky while also building “the widest, deepest view of the universe in its more static form — the distribution of galaxies and quasars in the distant universe, and the stars in our own Milky Way galaxy,” Bolton says.

The mission here is to share that information with scientists worldwide, in real time. That presents a massive data management challenge.

There’s the sheer volume of information, “and then there’s the complexity of the data as well,” Bolton says. “The core scientific data in some sense is homogeneous, but in detail — the supporting data, the metadata, the data systems that operate the observatory — that ends up being a very complex data problem.”

When it comes to scientific research, “cloud makes it significantly easier to manage and process these large, distributed data sets at scale,” says Arunachalam Balasubramanian Shrinivass, co-author of a recent paper on digital government transformation from the Association for Computing Machinery.

This, in turn, has the potential to speed scientific breakthroughs.

In the research community, “what we’re seeing right now is a real emphasis on breaking down barriers between different data sets, different disciplines,” says Richard Knepper, chair of the Coalition for Academic Scientific Computation and director of the Cornell University Center for Advanced Computing. With cloud-based approaches, agencies can help scientists to access the data they need “and then apply that to a discovery that’s impactful in the real world, that helps citizens be healthier, safer and more productive.”

Sharing the Data

The project team has tapped the power of Google Cloud to solve its data problems. Cloud has the needed capacity, as well as the flexibility, to meet the anticipated demand, with the observatory expected to soon be sharing data with 10,000 users.

“They may all show up at once, or they may all hit certain parts of the data at the same time,” Bolton says. A commercial cloud platform “lets us scale up and scale down to respond to that demand in a much more flexible and elastic way.”

The observatory is heavily dependent on software, and the “as a service” nature of the cloud supports the team’s needs in that arena as well. “It allows you to try things out and learn from experiments,” Bolton says.

“If one approach doesn’t work, you haven’t sunk a bunch of capital into some purchase that turns out to be the wrong call,” he says. “You can spin that down and spin up something else. It allows for a much more rapid iteration and improvement cycle.”

Tracking the Weather

The National Oceanic and Atmospheric Administration generates a vast amount of environmental information via satellites, radars, ships and weather models. The NOAA Open Data Dissemination uses commercial cloud partners such as Amazon Web Services (AWS), Google Cloud and Microsoft Azure to make that data readily available for both scientific discovery and for commercial users.

“Historically, NOAA’s data has been very valuable to businesses, to the public and to other government agencies — but it’s been very inaccessible, very hard to discover,” says retired Rear Admiral Timothy Gallaudet, who formerly served as both deputy administrator and acting administrator at NOAA.

“The big challenge is the volume of data. NOAA generates a terabyte of data a day, and it’s spread across multiple sites, organizations and geographic locations,” says Gallaudet, now the CEO of marine technology and management consultancy Ocean STL Consulting.

With vast infrastructure and scalable compute capabilities, cloud plays a key role in making that data “more discoverable and easily downloaded,” he says. “Now anybody can access any amount of data, anywhere — and at speeds that are much faster, because of the computing powerhouses that those cloud vendors provide.”

READ MORE: What’s next in the cloud?

Ready availability of NOAA’s weather data has a direct impact on the nation. From hurricane season on the East Coast to atmospheric rivers on the West Coast and drought in the Southwest, “having more data available for researchers is going to help us speed the research, improving models that predict those storms,” Gallaudet says.

The data also drives ocean mapping in the U.S. Exclusive Economic Zone, a 3.4-million-square-nautical-mile maritime area surrounding the nation’s coastline. “Mapping the ocean allows you to manage fisheries, to update nautical charts for safe navigation, to conserve coral reefs and monitor harmful algal blooms,” he says — and the power of the cloud is making that possible.

Observing the Earth

Beyond the weather, there’s a lot going on here on Earth that scientists want to understand, from shifting ocean currents to changes in the atmosphere, shrinking forests and soil erosion.

NASA’s Earthdata Cloud leverages AWS to share that data with the scientific community. Agency documents say the cloud enables users to work with larger data sets and collaborate more effectively on in-depth research efforts. Cloud supports open-source tools that benefit the scientific community, and it delivers enhanced data management capabilities.

A cloud-based approach also eases IT management for the agency, says Knepper.

“If you’ve got a data center that’s owned by NASA or another agency, then you’ve got to manage all the ins and outs of it,” he says. With Earthdata Cloud, AWS manages the infrastructure. Cloud also “helps solve the problem of scale,” with the capacity and flexibility to make vast amounts of data readily available to interested researchers.