Data Center

Data Center Optimization Standouts Share the Details on How They Did It

A GAO report named agencies who had excelled at meeting DCOI mandates; DOJ, USDA and EPA tell their stories.

Twitter

Jacquelyn Bengfort is a freelance writer based in Washington, DC. A social anthropologist by training, she writes on topics from education to the military, gender to fictional post-apocalyptic worldscapes.

Since 2010, the Office of Management and Budget has asked agencies to cut back on data center duplication and costs. Its first project focused merely on closing redundant data centers; the current one takes a broader view, seeing optimization as more important.

Savings are nearly guaranteed through this process; in 2007, the government paid $450 million for the electricity needed to run about 1,000 data centers, according to the Environmental Protection Agency.

Today, the government runs more than 2,000 data centers, and many agencies are managing to meet energy use metrics for unspecified cost savings, according to an April report from the Government Accountability Office.

OMB’s current Data Center Optimization Initiative mandated that agencies reduce the cost of physical data centers by at least 25 percent between fiscal years 2016 and 2018, for a savings of about $2.7 billion. At almost $2.4 billion in savings, the GAO report states, the agencies nearly made the goal.

In that report, the GAO also highlighted several agencies that excelled on its recent DCOI audit.

The Justice and Agriculture departments led the field in closing outmoded data centers; the General Services Administration and the Commerce Department demonstrated impressive cost savings; and the Social Security Administration and the Environmental Protection Agency stood out when it came to optimization efforts.

Justice Department Shifts to Commercial Cloud

As of October 2019, the Department of Justice has closed 101 of its original 110 data centers, and plans to close another six in pursuit of DCOI goals. The agency is focused on cutting energy and real estate costs while increasing efficiency and security, says CIO Joe Klimavicz.

“Our data center story really starts, like a lot of federal agencies, with somebody buying a couple of workstations and a server and putting it in the corner of the office,” says Klimavicz, “which gradually grows to 110 locations across the country.”

In 2014, Klimavicz began to examine all of the data centers in DOJ’s stock and started with the obvious candidates for consolidation: those housed in obsolete buildings with aging infrastructure.

His first priority was to “lift and shift” to close data centers quickly. Meanwhile, DOJ reskilled employees whose positions evolved as smaller, older data centers closed.

It’s a long journey, and a lot of things come up along the way.”

Joe Klimavicz CIO, Justice Department

“We didn’t try to optimize and move at the same time. That’s very difficult to do,” he says.

The approach allowed for a rapid shift to commercial cloud. DOJ has 36 cloud providers, including Microsoft Azure, one of its largest, Klimavicz says. DOJ uses on-premises private clouds for its most sensitive data.

By the end of fiscal year 2020, DOJ will have just three data centers remaining from its original stock of 110. The final 11 closures are the most difficult. “It’s a long journey, and a lot of things come up along the way,” says Klimavicz.

He credits strong communication with helping the department move forward: “We have weekly meetings on this topic with different stakeholders. When you’re dealing with a large agency with a lot of suborganizations, you can’t communicate enough.”

How to Achieve Data Center Optimization

There’s no one path to data center optimization, says Carol C. Harris, director of IT acquisition management issues at GAO and DCOI audit lead.

“There are challenges to defining best practices that work at large, federated departmental agencies that would also be practical at smaller, more centrally managed agencies,” she says.

It is possible to define general best practices, she adds. The GAO, during the audit and through surveys sent to successful agencies, found six:

Obtain executive leadership support
Use experience and lessons learned to refine consolidation planning
Boost use of cloud and shared services
Emphasize closing data centers in line with OMB goals
Increase the use of virtualization
Employ organizationwide communications

USDA Embraces Virtualization as It Closes Data Centers

Like DOJ, USDA was highlighted in the GAO report for its success in closing data centers, taking down 2,233 of its 2,237 nontiered data centers and 29 of its 35 tiered data centers.

“We no longer have agency-specific data centers,” says Ed Reyelts, associate CIO for USDA’s Digital Infrastructure Services Center. “That workload has been consolidated to the department’s two enterprise data centers or to the commercial cloud. We do have special-purpose labs, and remote offices for local file/print services.”

Virtualization was a big part of USDA’s DCOI strategy. “It allowed us to rightsize and ingest large workloads from the closing data centers, all while scaling the virtualization infrastructure already present within our department’s private cloud for significant economies of scale,” says Reyelts.

$128 million

The amount of cost savings and avoidances the Justice Department achieved as of August 2018 through data center consolidation.

Source: GAO, “Data Center Optimization: Additional Agency Actions Needed to Meet OMB Goals,” April 2019

USDA leadership offered strong support for the DCOI efforts. In December 2017, Deputy Secretary Stephen Censky set ambitious goals to restructure and consolidate IT management.

“Less than two years later, the USDA has closed 31 of the 37 targeted data centers, with the remaining six scheduled to close within the beginning of fiscal year 2020,” Reyelts says.

The USDA also defined a process — planning, preparation, data migration, testing and application cut-over — that ensured uniform and efficient closures. “The focus was on creating realistic project plans with achievable milestones where project managers and mission area executives were accountable,” explains Reyelts.

EPA Leaders Supported Data Center Consolidation

Leadership had a key role in EPA’s ability to optimize data center performance, says agency spokesperson Andrea Woods.

“EPA’s CIOs, past and present, strongly advocated for consolidation and optimization of enterprise services and applications. This was fundamental to the ongoing success,” says Woods.

With offices across the country, the EPA in 2010 operated four tiered data centers and approximately 74 additional nontiered data centers in 66 buildings in 48 cities. The agency has cut that inventory in half through consolidation and virtualization, and it also plans to close one of its four tiered data centers in 2020.

“The progressive consolidation of email and collaboration, first to EPA’s tiered data centers in 2009 and then to cloud services in 2013, was a fulcrum in shifting focus away from localized operations,” Woods adds.

Virtualization was also an opportunity to introduce a degree of uniformity across a large and decentralized organization.

“EPA evaluated a variety of virtualization platforms and selected the vendor most closely aligned with enterprise requirements gathered from application platform operators across the agency,” says Woods. “The standards were adopted for use agencywide to support workloads in both tiered and nontiered data centers.”

The Future of DCOI in Government

DCOI has resulted in some big successes, GAO’s Harris says. “The biggest gains are probably the closure of about half of the reported data centers and the related benefit of roughly $4 billion in cost savings and avoidances,” says Harris.

She notes that agencies have also begun to share data center space — an unforeseen development when the OMB instituted its original data center optimization plan almost a decade ago.

“Looking forward, agencies have more work to do to make their data centers more efficient, in terms of better space utilization and more energy efficiency,” says Harris.

Photography by Gary Landsman