It took about two weeks to fully restore the ship’s network in the 2022 outage, but ultimately, all of the data up to the time of the hardware failure was restored.
“We have to know that we can always either roll back or very quickly recover the data that was already in our applications,” Myers says. “It’s a key function, and one that we absolutely expect our network to be able to perform.”
How the Census Bureau Restored 81 Million Files Quickly
The U.S. Census Bureau maintains more than 90 petabytes of backup data, spread among on-premises and public cloud environments.
“Protecting the data that the Census Bureau collects and uses to report on our nation’s people and economy is critical to its mission,” says Jan Dickerson, chief of the enterprise backup branch in the agency’s Computer Services Division. “Backup is the last resort for getting data back, so it has to be right. Whether we are protecting from human error, faulty software releases, hardware issues, malware or cyberattacks, or natural disasters — all are important.”
DISCOVER: Resilient networks and systems protect key federal services from cyberattacks.
Dickerson says the Census Bureau has a “central, well-established” backup and recovery environment that uses a combination of cloud-native features and commercial, off-the-shelf software to protect data in both the agency’s own data centers and its resources in the public cloud. “This approach gives us operational consistency wherever our data resides, and it also allows us to recover our data on-premises to the cloud, or vice versa, or between cloud providers, or between on-premises sites,” she says.
The bureau, Dickerson says, seeks out backup and recovery tools with features including deduplication (which helps optimize the use of disk storage), auto image replication (which creates identical copies across multiple sites to aid with disaster recovery) and immutable backups (which remove cyberattackers’ ability to seize control over both production and backup environments simultaneously).
While the agency periodically tests its entire backup environment and processes, Dickerson says, her team fields daily requests from different Census Bureau program areas to restore everything from single files to entire systems. “It could be for any number of reasons,” she says. “A patch that needs to be rolled back or something deleted by accident.”
Over the past year, Dickerson says, the agency successfully restored 81 million files. “That is a good indicator that our backup environment is working end to end,” she says. “Those restore requests are our main priority. We get them done quickly. We want to make sure our users know we’re there for them.”
RELATED: Zero-touch configuration helped the Census Bureau modernize its IT.
How the USPTO Is Protecting Its Data
Annually, the U.S. Patent and Trademark Office takes in more than 600,000 new filings for patents alone, and the organization runs more than 7,000 servers on-premises and nearly 4,000 more in the public cloud. USPTO officials must not only back up all of that data, but also ensure uptime for the agency’s various systems — many of which are interdependent and some of which are public-facing.
In 2018, the organization faced an outage to its Patent Application Locating and Monitoring database. As a result, the agency had to temporarily shut down more than a dozen systems, including those that allowed patent applicants to pay fees. But thanks to effective backup tools and processes, the database was fully restored within several days.
“It was very important for us to make sure that we recovered all of that data and then restored that database to normal operations,” says Debbie Stephens, deputy CIO. “The good news is, we didn’t lose the data. But we lost time. That is our primary database for patent application data, so that was easily petabytes worth of information. It had dependencies for things like being able to submit payments, because the system goes to that database to check on the status of an application to allow someone to pay a fee.”
Stephens notes that USPTO uses Cohesity software for data and server backups while relying on Commvault to back up individual workstations. Both solutions feature immutable backups.
LEARN MORE: New cyber resilience solutions focus on zero trust.
As federal IT leaders design their backup and recovery environments, Stephens says, they must keep their overarching mission in mind.
“It seems very basic: What are the drivers of your work, what is important to the business?” she says. “The mission for us, which is constitutional, is to issue patents and register trademarks. So, you have to ask, what are your drivers and how do you protect them? You have to determine your most important driver, and whatever it is, that should be your No. 1 system to protect.”
