Down the Street

Natural disasters and disruptive events have made continuity of operations planning, with a focus on offsite backups, standard fare for federal IT departments.

The Northeastern Tribal Health System in Miami, Okla., knows well the importance of offsite backup. The healthcare facility, funded by the Indian Health Service, rents 150 gigabytes a month of hosted tape backup. “If there is a tornado that rips through here, I know I don’t have to worry,” says Bryan Moore, the health system’s CIO. “When you are an IT person, data is your life.”

Disaster recovery solutions range from occasional data backups to fully redundant data centers that mirror data every few seconds and have automatic rollover capability, says Shawn McCarthy, director of research for IDC Government Insights.

“Most agencies choose a path somewhere between the two,” McCarthy says. “The choice depends on available funding, importance of the data in question and the need for data timeliness.” The options range from hosted collocation services, such as that used by Northeastern Tribal Health, to established hot sites ready to come online should disaster strike and even duplicate remote facilities that can take over processing transparently.

Ensuring Availability

For Northeastern Tribal Health, the decision to develop offsite backup in another state was easy, Moore says: He was not taking chances with electronic health records for the 58,000 patient visits last year at the system’s three facilities located in the middle of Tornado Alley.

The agency uses the SafeGuard service from Network Partners. The decision to create a secondary disaster recovery location was made with the patient records in mind, Moore says — access must be immediate should disaster strike. All of the patient data is captured and stored electronically, making offsite storage on tape a necessity, he adds.

“We don’t have paper charts anymore,” Moore says. “This is all the data they have. You have to imagine the folks who don’t remember if they are allergic to something. Without the info, that is a shot in the dark to quality patient care.”

The Agriculture Department’s National Finance Center, based in New Orleans, has a similar need. The center processes the payroll for more than 500,000 federal employees across multiple agencies. It deployed its disaster recovery team right before Hurricane Katrina made landfall in 2005.

Senior staff monitored the storm for a week before it hit Louisiana. Two days before the storm struck, the disaster recovery team loaded more than 150,000 tapes on trucks headed to Philadelphia. Staff members set up operations at the offsite facility, enabling payroll processing to continue uninterrupted.

“We had excellent planning,” CIO Gil Hawk says. “When the time comes, you have to understand roles and responsibilities.”

Hawk’s team developed a plan that included logistical details and clearly identified roles of each individual on the disaster recovery team.

Not a single paycheck was delayed during the crisis, a transition that Hawk attributes to a solid COOP plan. The agency had practiced two disaster recovery drills, and the backup team knew their roles.

“I can’t even fathom if we didn’t have disaster recovery or COOP in place,” he says. “We would not have been able to continue operations. We had great confidence in our ability.”

McCarthy recommends that agencies take a risk management approach to disaster recovery when considering how best to protect data, systems and facilities. A successful COOP plan will keep agencies running with a minimal loss of data and productivity, he says.

“The loss of a facility is very rare; however, no facility can be totally protected,” McCarthy says. “Knowing that a facility could be lost means agencies need to plan accordingly.”

Day-to-Day Benefit

COOP planning ultimately creates a more efficient agency because a good alternative work plan illustrates that employees can work offsite effectively, notes Larry Orluskie, director of communications for the undersecretary for management of the Homeland Security Department. A contingency plan for telecommunications is essential to an agency’s disaster recovery plan, he says.

“In the event of COOP, you can have people function anywhere,” he says. “For key individuals to have the ability to work anywhere is beautiful.”

Even with two offsite backup facilities that mirror the department’s operational infrastructure and continually capture and replicate changes in files, DHS routinely tests its COOP plan by having employees work from home or remote locations, he says. Alternative communications vehicles, such as wireless phones, are also critical, he adds.

“You can’t rely on one location to store data,” Orluskie says. “Disaster recovery is a way of thinking and an understanding about what’s required to be prepared.”

Moore, whose health system has relied on its offsite backup service for a year now, knows firsthand that its disaster recovery process works — he has used the offsite backup to recover electronic documents that were lost, including health records. Relying on backup tapes with encrypted information never felt better, he says.

“I was able to find everything down to a single file,” he says. Confidence in your agency’s COOP readiness “all comes down to how you feel at the end of the day about your backups.”

Moore says placing a high value on being prepared for disasters outweighs the alternative because losing healthcare information could result in a loss of confidence from patients. Storing tapes offsite ensures that Northeastern Tribal Health System will remain operating with a minimal loss of data no matter the disaster.

“I have always been a stickler for redundancy,” Moore says. “You have to be mindful of patients’ data.”

Disaster Recovery Terms of Art

Recovery time objective (RTO)

The time period — measured in minutes, hours or days — in which an agency says its must recover from a systems disruption to avoid a break in continuity of operations

Recovery point objective (RPO)

The time back to which an agency has decided it must be able to retrieve data should disaster halt systems operations

Hot site

A disaster recovery facility equipped with the systems and network components necessary for an agency to restore data and bring systems live after a disruption

Emergency operations center

A location where response teams gather to manage the agency during a disaster or emergency