Web-Scale Data Poisoning: Split-View Attack - AI Case Study
AI Case StudyMany recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The researchers show that many of these datasets are vulnerable to a "split-view" poisoning attack. The attack exploits the fact that the data viewed when it was initially collected may differ from the data viewed by a user during training. The researchers identify expired and buyable domains that once hosted data...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Resource Development appears in 4 case steps.
- 2Multiple attack methods. The case connects to 6 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Resource Development
Step 1
Datasets
The researchers download a web-scale dataset, which consists of URLs pointing to individual datapoints.
-
Resource Development
Step 2
Domains
They identify expired domains in the dataset and purchase them.
-
Resource Development
Step 3
Poison Training Data
An adversary could create poisoned training data to replace expired portions of the dataset.
-
Resource Development An adversary could then upload the poisoned data to the domains they control. In this particular exercise, the researchers track requests to the URLs they control to track downloads to demonstrate there are active users of the dataset.
-
Impact
Step 5
Erode Dataset Integrity
The integrity of the dataset has been eroded because future downloads would contain poisoned datapoints.
-
Impact
Step 6
Erode AI Model Integrity
Models that use the dataset for training data are poisoned, eroding model integrity. The researchers show as little as 0.01% of the data needs to be poisoned for a successful attack.
Mitigations
Defenses connected to the attack methods in this case.
AI Bill of Materials
An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities.
This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Control Access to AI Models and Data at Rest
Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.
Input Restoration
Preprocess all inference data to nullify or reverse potential adversarial perturbations.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.