How Enterprise Security Operations Centers Strengthen Agencies’ Cyberdefenses

The Department of Health and Human Services is an inviting target for information thieves and other cybercriminals. Not only does the department house confidential information on patients, it also handles sensitive data on medical research and drug testing.

To prevent hackers from helping themselves to its data, HHS in 2009 built a Computer Security Incident Response Center (CSIRC) to centrally monitor and protect the entire department’s networks and computer systems.

“Who wouldn’t want data on a blockbuster drug before it hits the market? Or know what research is going on in the National Institutes of Health that you can commercialize or compromise? We also have personally identifiable information from patients that identity thieves want,” says Kevin Charest, the department’s chief information security officer. “Our security infrastructure is tested all day, every day, across all our enterprises.”

Security is an increasingly difficult responsibility for agencies, which have limited resources to combat widely dispersed, ever more sophisticated threats. Many departments are finding that different agencies under their aegis are developing very different security postures, with some more robust than others. To improve security across the entire department, several organizations have established enterprise security operations centers (ESOCs).

An ESOC can help agencies implement standard security products across the enterprise and ensure that every agency has effective tools. It can share information about cyberattacks and security vulnerabilities, both within a department and throughout government. When one agency finds a tactic that works well against a particular threat, informing others of such success can save them considerable trial and error.

“The biggest benefit is the rapid, real-time identification of vulnerabilities and any malware introduction into our systems,” says Rod Turk, CISO of the Commerce Department, which will begin building an ESOC this fiscal year. “The quicker we can identify problems, the quicker we can quarantine and contain them. Our hope with our ESOC is to share information, collaborate and provide all our operating units with early warnings of potential threats.”

Think Globally, Act Locally

HHS built its CSIRC with standardization, collaboration and rapid response in mind.

Located in Atlanta, the 24/7 operations center features a floor-to-ceiling video wall of HDTVs, allowing the 50-person IT security team to monitor the status of the department’s 11 operating divisions, get real-time views of network traffic and keep tabs on breaking news.

The 12,000-square-foot facility includes a number of components, such as the security operations center and separate forensics and containment labs, where analysts can use advanced tools to reverse-engineer malware and other vulnerabilities and use specialized workstations to perform deeper-dive analysis of compromised systems.

HHS standardized on security tools that are used by the CSIRC but also deployed at each of the department’s 11 operating divisions, whose own security teams provide a first line of defense. Smaller offices that don’t belong to any of the divisions, such as the Office for Civil Rights, are monitored and secured by the CSIRC.

The standard tools include firewalls as well as anti-virus, patch management, intrusion detection and prevention, and packet capture software, Charest says.

The data and logs from the security tools and networking equipment feed into central management software, called a security information and event management (SIEM) tool, which provides a holistic view of all the data. The SIEM tool analyzes the information, flags suspicious activity and sends out alerts.

“We have the same set of robust tools across the department. It’s kind of our version of ‘think globally, act locally.’ The local security teams can act locally, but at the same time, the alerts and metadata are transmitted to the CSIRC, so we have a global macro view,” Charest says.

As a result, CSIRC staffers can help its divisions fight off a cyberattack. They can also discover compromised systems before individual divisions do and can improve situational awareness by sharing threat information and security strategies throughout the organization.

Standardizing on technology allows every HHS organization to be interoperable, allowing for faster responses, Charest says.

“If the Centers for Disease Control is attacked in their environment, we can quickly search across the network to see if it’s happening anywhere else,” he says. “We can identify and provide mitigating action and neutralize it across our networks in near real time. That’s the power of the CSIRC.”

Technology is not the hard part of creating an ESOC; it’s building consensus, says Charest, who met with the IT leaders of each HHS operating division and included their input when purchasing security tools.

“It requires relationship-building and good, strong communications,” he says.

Recipe for Good Security

An effective ESOC requires three key ingredients: security tools, a team of well-trained IT security professionals and a standard set of security practices or workflow processes, says Anton Chuvakin, research director for Gartner’s Technical Professionals Security and Risk Management team.

Photo: Welton Doby III

Enterprise visibility will help agencies share security tools and information on threats, the Commerce Department’s Rod Turk says.

For example, ongoing monitoring of security systems is one process. When an alert arrives, that triggers the triage process, in which an analyst investigates whether the alert is a valid concern; if it is, that kicks off the incident response process, when a security team gets involved, Chuvakin says.

When the Agriculture Department began plans for an ESOC in 2009 after a series of security breaches, the agency had an IT compliance program but no enterprise- class operational security resources or policies. USDA built everything from scratch, CISO Christopher Lowe says.

“The first step was to find and hire great people,” he adds.

After a year of planning, the USDA began building the Agriculture Security Operations Center (ASOC) in June 2010 and launched it in April 2011. “In the beginning, it took a lot of effort to understand what normal traffic patterns were and what were malicious,” he recalls.

USDA’s security team has since developed a 290-page playbook for the ASOC, detailing security policies and workflow. “It explains what we do in every instance and how to respond to different sce­narios,” Lowe says.

To ensure continuity of operations, the 24/7 operations center is run out of two facilities — at USDA’s Washington, D.C., headquarters and at a Kansas City, Mo., data center. The 24-person ASOC staff communicates via video conferencing between the two sites. They also use classified networks for secure communications with law enforcement officials.

As for technology, the USDA merged seven security tools into a SIEM tool. Functions include tools for data loss prevention, malware detection, packet-capture and real-time analysis, intrusion detection and prevention, patch management and asset management.

Under Lowe’s direction, the USDA has integrated its enterprise security operations with IT compliance efforts. The agency uses continuous monitoring tools to provide a regularly updated risk assessment of computer systems. That, in turn, provides a wider view of the USDA’s security landscape.

USDA procured all the tools from commercial vendors except for network flow and traffic management software built by the Department of Homeland Security and the U.S. Computer Emergency Readiness Team (US-CERT) to examine network traffic and analyze its content.

The center houses storage-area networks in each site, storing several hundred terabytes of data for analysis. In total, the operation uses 200 to 300 physical servers located at the agencies that collect data, which is transmitted to secondary servers for failover purposes.

In the past, the FBI or US-CERT would alert the USDA if its systems were compromised. Now, the department knows immediately and can address attacks quickly.

“Now, the FBI or US-CERT tells us, ‘We saw this two weeks ago,’ and we often say, ‘We already saw it and fixed it,’ ” Lowe says.

Different Kinds of ESOCs

Other agencies, including the Energy Department, are taking different approaches with their ESOCs.

Within Energy, numerous national laboratories and other sites already operate their own security operations centers. Instead of replacing them, the department is augmenting them with its Joint Cybersecurity Coordination Center (JC3) in the department’s office of the CIO. Launched two years ago, its main goal is to assist existing centers with their security needs as well as to provide a central place for all Energy Department sites to share cybersecurity information, says Brian Varine, director of the center.

If an Energy Department site is dealing with a security issue, it can alert the JC3, which will share the information with other department sites.

The collaboration is a two-way street. Each SOC in the Energy Department can also assist other DOE sites that are dealing with security issues. “Our national laboratories have some of the most talented people in the world for cybersecurity. We want all of our sites to take advantage of those experts,” Varine says.

Through the JC3, the department is working to consolidate purchases of enterprise security tools and leverage the buying power of the entire department.

At the Commerce Department, the “OC” in ESOC will stand for oversight center, not operations center.

A handful of Commerce’s 12 organizational units have some level of security oversight capability. Some, such as the National Oceanic and Atmospheric Administration’s SOC, operate 24/7, while others are operational only during normal business hours, Turk says.

The Commerce Department’s planned ESOC will collect all the security information from the individual SOCs, and through a dashboard application, the department’s IT security team will get an all-encompassing view of the organization’s security posture.

“What we want to do is establish visibility into each one of these operation centers and try to collate the incidents and information from these incidents to get enterprise visibility, so our CIO and the CIOs in the different organizational units can see what is going on,” Turk says.

The goal is to create a central organization where IT leaders from the department and organizational units can share information on threats, share security tools and collaborate on any security situations that emerge, he says.

The Commerce Department’s ESOC is still in the planning stages, but security officials hope to launch it with initial continuous monitoring capabilities by September 2014.

To build its dashboard application, Turk plans to use commercial tools as well as tools created within Commerce and other agencies. If one organizational unit within Commerce has a good security tool, it will be leveraged and used throughout the department, he says.

“With declining budgets and increasing threats, we have a challenge to bring all of it together,” he says. “We have to leverage existing applications where we can and use shared services where we can. That will reduce the price and make it feasible.”