AI Risk

Data-related (Insufficient quality control in data collection process)

"A lack of standardized methods and sufficient infrastructure, including the absence of quality control processes for collecting data, especially for high-stakes domains and benchmarks, can affect the quality and type of the data collected [173, 95]. This may include risks of dataset poisoning, inadvertent copyright violation, and test set leakages which invalidate performance metrics."

View related techniques Read profile

AI Risk2. Privacy & Security2.2 > AI system security vulnerabilities and attacks1 - Pre-deployment

Record summary

A quick snapshot of what this page covers.

Techniques29Attack methods connected to this risk.

Mitigations22Defenses that may help with related attacks.

Domain2. Privacy & SecurityThe broad risk area this belongs to.

Risk profile

How this risk is described and categorized.

Domain2. Privacy & Security

Subdomain2.2 > AI system security vulnerabilities and attacks

Entity1 - Human

Intent2 - Unintentional

Timing1 - Pre-deployment

CategoryModel Development

SubcategoryData-related (Insufficient quality control in data collection process)

Related techniques

Attack methods connected to this risk.

AML.T0018 - Manipulate AI Model

realized

Methodtaxonomy_keyword_ruleConfidence75%

AML.T0079 - Stage Capabilities

demonstrated

Methodtaxonomy_keyword_ruleConfidence75%

AML.T0018.000 - Poison AI Model

demonstrated

Methodtaxonomy_keyword_ruleConfidence74%

AML.T0020 - Poison Training Data

realized

Methodtaxonomy_keyword_ruleConfidence73%

AML.T0064 - Gather RAG-Indexed Targets

demonstrated

Methodtaxonomy_keyword_ruleConfidence72%

AML.T0099 - AI Agent Tool Data Poisoning

feasible

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0080.001 - Thread

demonstrated

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0008.002 - Domains

demonstrated

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0070 - RAG Poisoning

demonstrated

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0010.002 - Data

realized

Methodtaxonomy_keyword_ruleConfidence71%

AML.T0034.002 - Agentic Resource Consumption

feasible

Methodtaxonomy_keyword_ruleConfidence70%

AML.T0108 - AI Agent

demonstrated

Methodtaxonomy_keyword_ruleConfidence70%

AML.T0104 - Publish Poisoned AI Agent Tool

realized

Methodtaxonomy_keyword_ruleConfidence70%

AML.T0059 - Erode Dataset Integrity

demonstrated

Methodtaxonomy_keyword_ruleConfidence70%

AML.T0019 - Publish Poisoned Datasets

demonstrated

Methodtaxonomy_keyword_ruleConfidence70%

AML.T0086 - Exfiltration via AI Agent Tool Invocation

realized

Methodtaxonomy_keyword_ruleConfidence70%

AML.T0080 - AI Agent Context Poisoning

demonstrated

Methodtaxonomy_keyword_ruleConfidence69%

AML.T0066 - Retrieval Content Crafting

demonstrated

Methodtaxonomy_keyword_ruleConfidence69%

AML.T0010.005 - AI Agent Tool

realized

Methodtaxonomy_keyword_ruleConfidence69%

AML.T0043.004 - Insert Backdoor Trigger

demonstrated

Methodtaxonomy_keyword_ruleConfidence68%

AML.T0058 - Publish Poisoned Models

realized

Methodtaxonomy_keyword_ruleConfidence68%

AML.T0011.002 - Poisoned AI Agent Tool

realized

Methodtaxonomy_keyword_ruleConfidence67%

AML.T0109 - AI Supply Chain Rug Pull

realized

Methodtaxonomy_keyword_ruleConfidence67%

AML.T0010.004 - Container Registry

demonstrated

Methodtaxonomy_keyword_ruleConfidence58%

AML.T0007 - Discover AI Artifacts

demonstrated

Methodtaxonomy_keyword_ruleConfidence58%

AML.T0094 - Delay Execution of LLM Instructions

demonstrated

Methodtaxonomy_keyword_ruleConfidence58%

AML.T0017 - Develop Capabilities

realized

Methodtaxonomy_keyword_ruleConfidence56%

AML.T0110 - AI Agent Tool Poisoning

realized

Methodtaxonomy_keyword_ruleConfidence55%

AML.T0097 - Virtualization/Sandbox Evasion

realized

Methodtaxonomy_keyword_ruleConfidence55%

Suggested mitigations

Defenses that may help with related attacks.

Control Access to AI Models and Data at Rest

Business and Data UnderstandingData Preparation+2 more

LifecycleBusiness and Data Understanding + 3 moreCategoryPolicy

Validate AI Model

ML Model EvaluationMonitoring and Maintenance

LifecycleML Model Evaluation + 1 moreCategoryTechnical - ML

Code Signing

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

Sanitize Training Data

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - ML

Maintain AI Dataset Provenance

Data PreparationBusiness and Data Understanding

LifecycleData Preparation + 1 moreCategoryTechnical - ML

Limit Model Artifact Release

Business and Data UnderstandingDeployment

LifecycleBusiness and Data Understanding + 1 moreCategoryPolicy

AI Bill of Materials

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryPolicy

Verify AI Artifacts

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - Cyber

AI Telemetry Logging

DeploymentMonitoring and Maintenance

LifecycleDeployment + 1 moreCategoryTechnical - Cyber

Privileged AI Agent Permissions Configuration

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

Single-User AI Agent Permissions Configuration

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

AI Agent Tools Permissions Configuration

Deployment

LifecycleDeploymentCategoryTechnical - Cyber

Human In-the-Loop for AI Agent Actions

Deployment

LifecycleDeploymentCategoryTechnical - ML

Restrict AI Agent Tool Invocation on Untrusted Data

Deployment

LifecycleDeploymentCategoryTechnical - ML

Segmentation of AI Agent Components

DeploymentBusiness and Data Understanding

LifecycleDeployment + 1 moreCategoryTechnical - Cyber

Input and Output Validation for AI Agent Components

Business and Data UnderstandingData Preparation+1 more

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - ML

Memory Hardening

ML Model EngineeringDeployment+1 more

LifecycleML Model Engineering + 2 moreCategoryTechnical - ML

Model Hardening

Data PreparationML Model Engineering

LifecycleData Preparation + 1 moreCategoryTechnical - ML

Use Ensemble Methods

ML Model Engineering

LifecycleML Model EngineeringCategoryTechnical - ML

Input Restoration

Data PreparationML Model Evaluation+2 more

LifecycleData Preparation + 3 moreCategoryTechnical - ML

Adversarial Input Detection

Data PreparationML Model Engineering+3 more

LifecycleData Preparation + 4 moreCategoryTechnical - ML

Encrypt Sensitive Information

Data PreparationML Model Engineering+1 more

LifecycleData Preparation + 2 moreCategoryTechnical - Cyber

Source

Research source for this risk, when available.

Included resource

Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systems

AuthorsGipiškis et al.Year2024TypeJournal Article

DOIhttps://doi.org/10.48550/arXiv.2410.23472 URLhttps://arxiv.org/abs/2410.23472

Original source

MIT AI Risk Repository

Open the public repository used for AI risk records and taxonomy fields.

Repositoryhttps://airisk.mit.edu/