When the WannaCry Ransomware Attack Struck, Feds Were Ready
On a Friday morning in mid-May, David Nelson was driving to work when he was struck by a news report on the radio: A massive ransomware attack was under way, crippling computer systems worldwide.
The news was alarming. As CIO of the Nuclear Regulatory Commission, Nelson began to worry. After all, he oversees IT systems for 3,200 employees and manages thousands of servers and computers for an agency that is tasked with protecting the public health and safety when it comes to nuclear energy.
But Nelson didn’t panic. He had spent the past nine months preparing the agency for this day. When he arrived at the office at 7:45 a.m., he immediately met with his cybersecurity team. The team told him that WannaCry ransomware locked up computers and servers across the globe that ran older, unpatched versions of Windows. But they also shared that the NRC was in a strong position because it had installed a patch that Microsoft released two months earlier.
“We were well protected,” Nelson says. “We have an inventory of the patches we have installed, so we are in a position to know what risks we have in our systems. WannaCry was not one of them.”
SIGN UP: Get more news from the FedTech newsletter in your inbox every two weeks!
Agencies are investing heavily in patch management to bolster cybersecurity. Under scrutiny from Congress, many agencies have improved their techniques with patch management to protect themselves against critical vulnerabilities. According to a White House report from March, 90 percent of agencies met federal targets for vulnerability management in 2016, up from 70 percent in 2015. Security experts say patch management is a basic but ultimately critical component for agencies to establish a strong cyberdefense.
“Unpatched software is one of the primary vectors used to create breaches,” says Kelley L. Dempsey, senior information security specialist with the National Institute of Standards and Technology. “Patching is absolutely one of the most basic protections you can put in place.”
For agencies, this means deploying vulnerability scanning tools, such as BeyondTrust’s Retina or Tenable’s Nessus scanner, and automated patch management tools, such as IBM BigFix or Microsoft System Center Configuration Manager.
IT leaders must also implement a set of policies and documented procedures that everyone follows to ensure vulnerabilities are tracked and patches are applied as quickly as possible, Dempsey says.
HHS Gets into Gear with WannaCry Response
On the morning of the WannaCry attack, a harried scene played out at the Health and Human Services Department. Officials had previously updated the agency’s patching processes for servers and workstations as well as its emergency patch management procedures. Each HHS operating division maintains its own patching environment, but at the enterprise level, HHS uses tools for automated patching and patch validation, which verifies that patches are correctly installed. IT staffers test the patches first, and if they don’t notice any hiccups, they deploy them to production servers and workstations.
As news of WannaCry spread, the CIO office’s newly created threat analytics center — the Healthcare Cybersecurity and Communications Integration Center (HCCIC) — issued an internal alert, prompting former HHS Secretary Tom Price to mobilize teams to ensure HHS’s infrastructure and the country’s healthcare providers were protected.
The situation was perilous because many medical systems are not designed to be visible on the internet and thus are difficult to patch automatically.
As a result, HHS’s own hospital systems, as well as healthcare providers across the country, were vulnerable to the malware, says HHS Deputy CISO Leo Scanlon. To coordinate their response, HHS leaders met at the Secretary’s Operations Center, a command center that allows for real-time situational awareness, communications and information sharing.
Patching Rapidly in Response to Ransomware
At one point, the emergency response team hosted a call with 7,000 to 10,000 officials from large hospital groups, small and rural practices and healthcare associations. On the call, HCCIC officials provided advice and guidance on identifying and patching their vulnerable systems, Scanlon says.
Internally, department leaders had secured HHS’s enterprise business systems and high-value assets through ongoing patching. But as the attack spread, the CIO’s office collaborated with its operating divisions to patch and secure tens of thousands of workstations and servers.
“Within 24 hours of WannaCry’s appearance, a majority of department systems were patched or accounted for, and within 48 hours, all systems were patched or sequestered from the internet,” Scanlon says.
HHS fully staffed the operations center throughout the weekend and the early part of the next week. HD monitors on the wall displayed the status of internal and external systems.
“This was the first time that the Secretary’s Operations Center was deployed in response to a cyber event, and it was an unqualified success,” Scanlon says.
Invest in Tools to Help with Patch Management
At the NRC, patching software had also been a long-time priority. When Nelson joined the agency in August 2016, he invested in new tools and improved processes to make patching software more effective.
During the first four months of his tenure, he and his IT team rolled out new vulnerability scanning and patch management tools. They also purchased additional servers, so the agency could scale the tools and scan for vulnerabilities and install patches more rapidly.
Nelson and his team also developed new processes, such as running scans several times a week to provide the agency up-to-date information on patch levels and vulnerabilities.
“If you are not scanning on a tight schedule, then you can’t measure yourself and see how much progress you are making,” he says.
The agency learns of critical patches and vulnerabilities through vendors and intelligence monitoring services.
Stay on Top of Cybersecurity Vulnerabilities
Today, Nelson requires his staff to produce a daily security report, which keeps him abreast of the latest patches and the agency’s 10 most critical vulnerabilities. He meets with his security team three times a week to discuss the latest cybersecurity risks and patches and prioritize tasks.
“With our daily briefing report, we are able to react pretty quickly,” he says. “It provides situational awareness.”
The efforts have produced results: The agency has reduced the time it takes to roll out new patch bundles by half. It’s also reduced the number of vulnerabilities in computer systems by half.
A More Structured Environment
The Commodity Futures Trading Commission, which regulates futures and swaps markets, has also made strides with patch management during the past year by creating more repeatable processes.
Today, the commission uses asset management software to track inventory of devices that need patching, and runs a vulnerability scanner once a week. Then, through a software distribution tool, the IT staff schedules and automatically pushes out patches.
“The key thing is to know what you have in your environment and scan all of it, then see what the risk is. And based on that risk profile, prioritize,” says John Rogers, the agency’s CIO.
During the past year, the IT team began to store information after each scan and after installing patches so leaders have a status on each device.
Rogers has also created patch management teams that meet weekly to discuss vulnerabilities.
Before deploying patches, the commission runs extensive tests to make sure patches don’t break applications. If there are problems, the IT staff reaches out to software vendors to resolve them. Or instead of applying a patch, they may implement a compensating security control to make sure the specific vulnerability is not exploitable, says Naeem Musa, chief information security officer of CFTC’s policy and planning branch.
The commission also regularly updates its golden images for servers and computers, so that when new devices are introduced onto the network, the agency uses the most updated, secure versions of operating systems and software, Musa says.
IT leaders say they’ve made progress improving patch management, but because of the never-ending nature of the job, vigilance is needed.
“To do it well requires constant diligence,” Rogers says.
