Unpacking AI Data Poisoning

Watch Out for Data Poisoning Tactics

Data poisoning attacks can be categorized in two ways: by how much knowledge the attacker has and by which tactic they employ. When a bad actor has no knowledge of the data they seek to manipulate, it’s known as a black-box attack.

The other side of the spectrum is a white-box attack, in which the adversary has full knowledge of the model and its training parameters. These attacks, as you might suspect, have the highest success rate.

There are also grey-box attacks, which fall in the middle.

The amount of knowledge a bad actor has may also affect which tactic they choose. Data poisoning attacks, generally speaking, can be broken into four broad buckets: availability attacks, targeted attacks, subpopulation attacks and backdoor attacks. Let’s take a look at each.

Availability attack: With this breed of attack, the entire model is corrupted. As a result, model accuracy will be considerably reduced. The model will offer false positives, false negatives and misclassified test samples. One type of availability attack is label flipping, or adding approved labels to compromised data.

RELATED: Government increased its use of AI in 2023.

Targeted attack: While an availability attack compromises the whole model, a targeted attack affects only a subset. The model will still perform well for most samples, which makes targeted attacks challenging to detect.

Subpopulation attack: Much like a targeted attack, a subpopulation attack doesn’t affect the whole model. Instead, it influences subsets that have similar features.

Backdoor attack: As the name suggests, this type of attack takes place when an adversary introduces a back door — such as a set of pixels in the corner of an image — into training examples. This triggers the model to misclassify items.

How to Fight Back Against Data Poisoning

In the private sector, Google’s anti-spam filter has been attacked multiple times. By poisoning the spam filter’s algorithm, bad actors have been able to change how spam was defined, causing malicious emails to bypass the filter.

Now, imagine if something similar happened to an agency. Undoubtedly, the impact would be far worse.

The question then is, how can agencies prevent data poisoning from taking place?

To start, proactive measures must be put in place. Agencies need to be extremely diligent about which data sets they use to train a given model and who is granted access to them.

When a model is being trained, it’s crucial to keep its operating information secret. This high level of diligence can be enhanced by high-speed verifiers and zero-trust content disarm and reconstruction, tools that ensure all data being transferred is clean.

Additionally, statistical models can be used to detect anomalies in the data, while tools such as Microsoft Azure Monitor and Amazon SageMaker can detect shifts in accuracy.

Proactive measures are critical because data poisoning is extremely difficult to remedy. To correct a poisoned model, the agency would have to conduct a detailed analysis of its training inputs to detect and remove fraudulent ones.

DISCOVER: Agencies should be part of the AI proof-of-concept process.

As data sets grow, that analysis becomes more difficult, if not impossible. In such cases, the only option is to retrain the model completely, a time-consuming and expensive process.

Training GPT-3, for instance, carried a price tag of more than $17 million. Most agencies simply do not have the budget for that kind of correction.

As agencies embrace ML and new forms of AI, they must be aware of the threats that accompany them. There are numerous ways for an adversary to disrupt a model, from inserting malicious data to modifying existing training samples.

Preventing data poisoning attacks is particularly crucial as more agencies rely on AI to deliver essential services. For AI to truly live up to its potential, agencies must take the steps necessary to maintain model integrity across the board.