Record summary
A quick snapshot of what this page covers.
Risk profile
How this risk is described and categorized.
"Prompt Injections are a form of Adversarial Input that involve manipulating the text instructions given to a GenAI system (Liu et al., 2023). Prompt Injections exploit loopholes in a model’s architec- tures that have no separation between system instructions and user data to produce a harmful output (Perez and Ribeiro, 2022). While researchers may use similar techniques to test the robustness of GenAI models, malicious actors can also leverage them. For example, they might flood a model with manipulative prompts to cause denial-of-service attacks or to bypass an AI detection software."
Suggested mitigations
Defenses that may help with related attacks.
Control Access to AI Models and Data in Production
Generative AI Guardrails
Generative AI Guidelines
Generative AI Model Alignment
AI Telemetry Logging
Input and Output Validation for AI Agent Components
Memory Hardening
Restrict Library Loading
Code Signing
Vulnerability Scanning
User Training
AI Bill of Materials
Verify AI Artifacts
Use Ensemble Methods
Control Access to AI Models and Data at Rest
Encrypt Sensitive Information
AI Model Distribution Methods
Privileged AI Agent Permissions Configuration
Single-User AI Agent Permissions Configuration
AI Agent Tools Permissions Configuration
Human In-the-Loop for AI Agent Actions
Restrict AI Agent Tool Invocation on Untrusted Data
Segmentation of AI Agent Components
Model Hardening
Use Multi-Modal Sensors
Input Restoration
Adversarial Input Detection
Deepfake Detection
Sanitize Training Data
Maintain AI Dataset Provenance
Source
Research source for this risk, when available.
Included resource
Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data
Original source
MIT AI Risk Repository
Open the public repository used for AI risk records and taxonomy fields.