APromptRiskDBThreat intelligence atlas
AI Case Study

Web-Scale Data Poisoning: Split-View Attack - AI Case Study

Many recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The researchers show that many of these datasets are vulnerable to a "split-view" poisoning attack. The attack exploits the fact that the data viewed when it was initially collected may differ from the data viewed by a user during training. The researchers identify expired and buyable domains that once hosted data...

View attack chainRead summary
Exercise10 web-scale datasetsResearchers from Google Deepmind, ETH Zurich, NVIDIA, Robust Intelligence, and GoogleResource DevelopmentImpact

Overview

Case steps6Steps described in the case record.
Techniques6Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. Resource Development appears in 4 case steps.
  • 2Multiple attack methods. The case connects to 6 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

Resource Development4Impact2

  1. Step 1

    Datasets

    Resource Development

    The researchers download a web-scale dataset, which consists of URLs pointing to individual datapoints.

  4. Resource Development

    An adversary could then upload the poisoned data to the domains they control. In this particular exercise, the researchers track requests to the URLs they control to track downloads to demonstrate there are active users of the dataset.

  6. Impact

    Models that use the dataset for training data are poisoned, eroding model integrity. The researchers show as little as 0.01% of the data needs to be poisoned for a successful attack.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

AI Bill of Materials

An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities. This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.

Adversarial Input Detection

Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.

Control Access to AI Models and Data at Rest

Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.

Input Restoration

Preprocess all inference data to nullify or reverse potential adversarial perturbations.

Limit Model Artifact Release

Limit public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production, or that are representative of those used in production.

Maintain AI Dataset Provenance

Maintain a detailed history of datasets used for AI applications. The history should include information about the dataset's source as well as a complete record of any modifications.

Model Hardening

Use techniques to make AI models robust to adversarial inputs such as adversarial training or network distillation.

Sanitize Training Data

Detect and remove or remediate poisoned training data. Training data should be sanitized prior to model training and recurrently for an active learning model. Implement a filter to limit ingested training data. Establish a content policy that would remove unwanted content such as certain explicit or offensive language from being used.

Use Ensemble Methods

Use an ensemble of models for inference to increase robustness to adversarial inputs. Some attacks may effectively evade one model or model family but be ineffective against others.

Validate AI Model

Validate that AI models perform as intended by testing for backdoor triggers, potential for data leakage, or adversarial influence. Monitor AI model for concept drift and training data drift, which may indicate data tampering and poisoning.

Verify AI Artifacts

Verify the cryptographic checksum of all AI artifacts to verify that the file was not modified by an attacker.

Sources

Original public records and references for this case.