Restrict AI Agent Tool Invocation on Untrusted Data - AI Mitigation
AI MitigationUntrusted data can contain prompt injections that invoke an AI agent's tools, potentially causing confidentiality, integrity or availability violations. It is recommended that tool invocation be restricted or limited when untrusted data enters the LLM's context. The degree to which tool invocation is restricted may depend on the potential consequences of the action. Consider blocking the automatic invocation of to...
Overview
A source-backed snapshot of this defense.
Untrusted data can contain prompt injections that invoke an AI agent's tools, potentially causing confidentiality, integrity or availability violations. It is recommended that tool invocation be restricted or limited when untrusted data enters the LLM's context.
The degree to which tool invocation is restricted may depend on the potential consequences of the action. Consider blocking the automatic invocation of tools or requiring user confirmation once untrusted data enters the LLM's context. For high consequence actions, consider always requiring user confirmation.
Safeguard details
Where this defense applies and how the source classifies it.
- ATLAS ID
- AML.M0030
- Priority score
- 15
Covered techniques
Attacks this defense is designed to help with.
AML.T0053 - AI Agent Tool Invocation
Restricting the automatic tool use when untrusted data is present can prevent adversaries from invoking tools via prompt injections.
AML.T0101 - Data Destruction via AI Agent Tool Invocation
Restricting the automatic tool use when untrusted data is present can prevent adversaries from invoking tools via prompt injections.
AML.T0086 - Exfiltration via AI Agent Tool Invocation
Restricting the automatic tool use when untrusted data is present can prevent adversaries from invoking tools via prompt injections.
Source evidence
Original public records and references for this page.
Original source
Original source links
Open the public records and source datasets used for this page.