Watch Out for Data Poisoning Tactics
Data poisoning attacks can be categorized in two ways: by how much knowledge the attacker has and by which tactic they employ. When a bad actor has no knowledge of the data they seek to manipulate, it’s known as a black-box attack.
The other side of the spectrum is a white-box attack, in which the adversary has full knowledge of the model and its training parameters. These attacks, as you might suspect, have the highest success rate.
There are also grey-box attacks, which fall in the middle.
The amount of knowledge a bad actor has may also affect which tactic they choose. Data poisoning attacks, generally speaking, can be broken into four broad buckets: availability attacks, targeted attacks, subpopulation attacks and backdoor attacks. Let’s take a look at each.
Availability attack: With this breed of attack, the entire model is corrupted. As a result, model accuracy will be considerably reduced. The model will offer false positives, false negatives and misclassified test samples. One type of availability attack is label flipping, or adding approved labels to compromised data.
Targeted attack: While an availability attack compromises the whole model, a targeted attack affects only a subset. The model will still perform well for most samples, which makes targeted attacks challenging to detect.
Subpopulation attack: Much like a targeted attack, a subpopulation attack doesn’t affect the whole model. Instead, it influences subsets that have similar features.
Backdoor attack: As the name suggests, this type of attack takes place when an adversary introduces a back door — such as a set of pixels in the corner of an image — into training examples. This triggers the model to misclassify items.
How to Fight Back Against Data Poisoning
In the private sector, Google’s anti-spam filter has been attacked multiple times. By poisoning the spam filter’s algorithm, bad actors have been able to change how spam was defined, causing malicious emails to bypass the filter.
Now, imagine if something similar happened to an agency. Undoubtedly, the impact would be far worse.