The billions of bits of information that add up to create “Big Data” come from a variety of sources: the mobile devices that agency workers carry with them; connected IT systems; networked sensors in Internet of Things deployments; electronic forms completed by the citizens that an agency serves; and the vast number of inspection, compliance and other records kept by most agencies. Additionally, agencies can extract real-time insights from the massive amount of data generated by websites, business applications, social media platforms, application servers, hypervisors, traditional databases and open-source data stores.
With a single cross-country airplane flight capable of generating as much as 240 terabytes of data, most organizations won’t have to hunt too hard for rich sources of information. As any IT professional who has had to expand storage, computing and networking resources in recent years can attest, most organizations will have to deal with Big Data whether they’re leveraging that data to create new insights or not.
The larger challenge is harnessing all of this data in a coherent and organized manner, processing it to arrive at new conclusions about an organization’s operational model and how it delivers services, implementing an analytics system that can help to differentiate between relevant patterns and meaningless noise and then devising an action plan based on these new insights.
The complexity alone of Big Data solutions presents a significant challenge. Because each organization’s data analytics needs are unique, Big Data tools must be tailored to an agency’s specific environment in order to provide the highest possible value. Technical challenges create another hurdle. While many organizations already have the storage, computing and networking equipment necessary to support a Big Data initiative, others may need to expand or upgrade their resources, and tying all of these systems into one integrated solution can pose a challenge for even savvy IT departments. Finally, the additional regulations faced by government agencies can create challenges, as organizations must be careful not to run afoul of rules governing how to store and safeguard sensitive data.
While the challenges of getting a Big Data initiative off the ground are considerable, the potential payoff is enormous. These solutions can help to deliver a central, unified view of IT operations and services; detect patterns, highlight anomalies and pinpoint areas of impact; improve security through threat intelligence; analyze behavior for predicting attacks and threats; conduct analysis in real time; and create actionable intelligence.
To illustrate just one example: A Big Data analytics solution can automatically monitor and look for anomalies in IT user behavior, alerting administrators when rogue devices are connected to the network, when someone changes system configurations or when a user swipes an access badge two hours before he or she is supposed to report for work and then engages in a period of heavy downloading. This sort of monitoring can give IT administrators confidence in the integrity of their systems and can help to keep enterprise IT environments secure, while requiring little in the way of manual monitoring.
Learn about the building blocks of Big Data by downloading the white paper, "Making Sense of Big Data."