AI Risk

Fine-tuning related (Fine-tuning dataset poisoning)

"A deployer can poison the dataset used during the fine-tuning process [98] to induce specific, often malicious, behaviors in a model. This can be performed without having access to the model’s weights. This poisoning can be difficult to detect through direct inspection of the dataset, as the manipulations may be subtle and targeted."

View related techniques Read profile

AI Risk2. Privacy & Security2.2 > AI system security vulnerabilities and attacks1 - Pre-deployment

Record summary

A quick snapshot of what this page covers.

Techniques35Attack methods connected to this risk.

Mitigations24Defenses that may help with related attacks.

Domain2. Privacy & SecurityThe broad risk area this belongs to.

Risk profile

How this risk is described and categorized.

Domain2. Privacy & Security

Subdomain2.2 > AI system security vulnerabilities and attacks

Entity1 - Human

Intent1 - Intentional

Timing1 - Pre-deployment

CategoryModel Development

SubcategoryFine-tuning related (Fine-tuning dataset poisoning)

Related techniques

Attack methods connected to this risk.

AML.T0099 - AI Agent Tool Data Poisoning

feasible

Methodtaxonomy_keyword_ruleConfidence75%

AML.T0018 - Manipulate AI Model

realized

Methodtaxonomy_keyword_ruleConfidence75%

AML.T0018.000 - Poison AI Model

demonstrated

Methodtaxonomy_keyword_ruleConfidence75%

AML.T0070 - RAG Poisoning

demonstrated

Methodtaxonomy_keyword_ruleConfidence75%

AML.T0059 - Erode Dataset Integrity

demonstrated

Methodtaxonomy_keyword_ruleConfidence74%

AML.T0010.005 - AI Agent Tool

realized

Methodtaxonomy_keyword_ruleConfidence74%

AML.T0008.002 - Domains

demonstrated

Methodtaxonomy_keyword_ruleConfidence74%

AML.T0108 - AI Agent

demonstrated

Methodtaxonomy_keyword_ruleConfidence73%

AML.T0064 - Gather RAG-Indexed Targets

demonstrated

Methodtaxonomy_keyword_ruleConfidence73%

AML.T0109 - AI Supply Chain Rug Pull

realized

Methodtaxonomy_keyword_ruleConfidence72%

AML.T0086 - Exfiltration via AI Agent Tool Invocation

realized

Methodtaxonomy_keyword_ruleConfidence72%

AML.T0079 - Stage Capabilities

demonstrated

Methodtaxonomy_keyword_ruleConfidence72%

AML.T0080.001 - Thread

demonstrated

Methodtaxonomy_keyword_ruleConfidence72%

AML.T0066 - Retrieval Content Crafting

demonstrated

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0110 - AI Agent Tool Poisoning

realized

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0043.004 - Insert Backdoor Trigger

demonstrated

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0080 - AI Agent Context Poisoning

demonstrated

Methodtaxonomy_keyword_ruleConfidence70%

AML.T0034.002 - Agentic Resource Consumption

feasible

Methodtaxonomy_keyword_ruleConfidence69%

AML.T0058 - Publish Poisoned Models

realized

Methodtaxonomy_keyword_ruleConfidence68%

AML.T0011.002 - Poisoned AI Agent Tool

realized

Methodtaxonomy_keyword_ruleConfidence68%

AML.T0104 - Publish Poisoned AI Agent Tool

realized

Methodtaxonomy_keyword_ruleConfidence67%

AML.T0094 - Delay Execution of LLM Instructions

demonstrated

Methodtaxonomy_keyword_ruleConfidence61%

AML.T0081 - Modify AI Agent Configuration

demonstrated

Methodtaxonomy_keyword_ruleConfidence59%

AML.T0020 - Poison Training Data

realized

Methodtaxonomy_keyword_ruleConfidence58%

AML.T0010.002 - Data

realized

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0008.005 - AI Service Proxies

realized

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0111 - AI Supply Chain Reputation Inflation

demonstrated

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0018.002 - Embed Malware

realized

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0010.000 - Hardware

feasible

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0019 - Publish Poisoned Datasets

demonstrated

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0084.001 - Tool Definitions

demonstrated

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0051.002 - Triggered

demonstrated

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0010.003 - Model

realized

Methodtext_similarity_sqliteConfidence54%

AML.T0005.000 - Train Proxy via Gathered AI Artifacts

demonstrated

Methodtext_similarity_sqliteConfidence54%

AML.T0041 - Physical Environment Access

demonstrated

Methodtext_similarity_sqliteConfidence53%

Suggested mitigations

Defenses that may help with related attacks.

Control Access to AI Models and Data at Rest

Business and Data UnderstandingData Preparation+2 more

LifecycleBusiness and Data Understanding + 3 moreCategoryPolicy

Validate AI Model

ML Model EvaluationMonitoring and Maintenance

LifecycleML Model Evaluation + 1 moreCategoryTechnical - ML

Code Signing

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

Sanitize Training Data

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - ML

Maintain AI Dataset Provenance

Data PreparationBusiness and Data Understanding

LifecycleData Preparation + 1 moreCategoryTechnical - ML

AI Telemetry Logging

DeploymentMonitoring and Maintenance

LifecycleDeployment + 1 moreCategoryTechnical - Cyber

Privileged AI Agent Permissions Configuration

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

Single-User AI Agent Permissions Configuration

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

AI Agent Tools Permissions Configuration

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

Human In-the-Loop for AI Agent Actions

Deployment

LifecycleDeploymentCategoryTechnical - ML

Restrict AI Agent Tool Invocation on Untrusted Data

Deployment

LifecycleDeploymentCategoryTechnical - ML

Segmentation of AI Agent Components

DeploymentBusiness and Data Understanding

LifecycleDeployment + 1 moreCategoryTechnical - Cyber

Input and Output Validation for AI Agent Components

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - ML

Model Hardening

Data PreparationML Model Engineering

LifecycleData Preparation + 1 moreCategoryTechnical - ML

Use Ensemble Methods

ML Model Engineering

LifecycleML Model EngineeringCategoryTechnical - ML

Input Restoration

Data PreparationML Model Evaluation+2 more

LifecycleData Preparation + 3 moreCategoryTechnical - ML

Adversarial Input Detection

Data PreparationML Model Engineering+3 more

LifecycleData Preparation + 4 moreCategoryTechnical - ML

Memory Hardening

ML Model EngineeringDeployment+1 more

LifecycleML Model Engineering + 2 moreCategoryTechnical - ML

AI Bill of Materials

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryPolicy

Limit Model Artifact Release

Business and Data UnderstandingDeployment

LifecycleBusiness and Data Understanding + 1 moreCategoryPolicy

Verify AI Artifacts

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - Cyber

AI Model Distribution Methods

Deployment

LifecycleDeploymentCategoryPolicy

Limit Public Release of Information

Business and Data Understanding

LifecycleBusiness and Data UnderstandingCategoryPolicy

Use Multi-Modal Sensors

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - Cyber

Source

Research source for this risk, when available.

Included resource

Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems

AuthorsGipiškis et al.Year2024TypeJournal Article

DOIhttps://doi.org/10.48550/arXiv.2410.23472 URLhttps://arxiv.org/abs/2410.23472

Original source

MIT AI Risk Repository

Open the public repository used for AI risk records and taxonomy fields.

Repositoryhttps://airisk.mit.edu/