Data Analytics

How Federal Agencies Can Simplify Big Data Complexities

Before federal agencies can draw insights from Big Data, they must find a way to process and store all of this new information.

BizTech Staff

The billions of bits of information that add up to create “Big Data” come from a variety of sources: the mobile devices that agency workers carry with them; connected IT systems; networked sensors in Internet of Things deployments; electronic forms completed by the citizens that an agency serves; and the vast number of inspection, compliance and other records kept by most agencies. Additionally, agencies can extract real-time insights from the massive amount of data generated by websites, business applications, social media platforms, application servers, hypervisors, traditional databases and open-source data stores.

With a single cross-country airplane flight capable of generating as much as 240 terabytes of data, most organizations won’t have to hunt too hard for rich sources of information. As any IT professional who has had to expand storage, computing and networking resources in recent years can attest, most organizations will have to deal with Big Data whether they’re leveraging that data to create new insights or not.

The larger challenge is harnessing all of this data in a coherent and organized manner, processing it to arrive at new conclusions about an organization’s operational model and how it delivers services, implementing an analytics system that can help to differentiate between relevant patterns and meaningless noise and then devising an action plan based on these new insights.

The complexity alone of Big Data solutions presents a significant challenge. Because each organization’s data analytics needs are unique, Big Data tools must be tailored to an agency’s specific environment in order to provide the highest possible value. Technical challenges create another hurdle. While many organizations already have the storage, computing and networking equipment necessary to support a Big Data initiative, others may need to expand or upgrade their resources, and tying all of these systems into one integrated solution can pose a challenge for even savvy IT departments. Finally, the additional regulations faced by government agencies can create challenges, as organizations must be careful not to run afoul of rules governing how to store and safeguard sensitive data.

While the challenges of getting a Big Data initiative off the ground are considerable, the potential payoff is enormous. These solutions can help to deliver a central, unified view of IT operations and services; detect patterns, highlight anomalies and pinpoint areas of impact; improve security through threat intelligence; analyze behavior for predicting attacks and threats; conduct analysis in real time; and create actionable intelligence.

To illustrate just one example: A Big Data analytics solution can automatically monitor and look for anomalies in IT user behavior, alerting administrators when rogue devices are connected to the network, when someone changes system configurations or when a user swipes an access badge two hours before he or she is supposed to report for work and then engages in a period of heavy downloading. This sort of monitoring can give IT administrators confidence in the integrity of their systems and can help to keep enterprise IT environments secure, while requiring little in the way of manual monitoring.

Learn about the building blocks of Big Data by downloading the white paper, "Making Sense of Big Data."

Real-World Applications

Several agencies are already making headway in Big Data applications.

Performance diagnostics: Pacific Northwest National Laboratory, a Richland, Wash., laboratory managed by the Energy Department's Office of Science, used Splunk IT Service Intelligence to develop its own dashboards that provide real-time visibility and contextual insights into key performance indicators. The laboratory is looking to use the tool to not only identify problems, but also to automate responses.

Improving fire response: The Federal Emergency Management Agency, the U.S. Fire Administration and the National Fire Incident Reporting System partnered to develop an analytical prototype and worked with four regional fire departments to use Big Data analytics to study approximately 225 million fire incidents. The effort helped the agencies to identify trends and patterns in incident types, equipment failures and firefighter casualties, and delivered new insights on how to improve training and reduce losses.

Helping consumers: The Consumer Financial Protection Bureau uses text and log analytics from consumers’ interactions with the agency’s website, social media and online forums to study their behavior patterns. Through this work, the agency was able to help convince FICO, a credit rating service, to change its credit score algorithms by reducing the weight of medical debt in scoring, which lifted scores for many consumers.

PashaIgnatov/Thinkstock