APromptRiskDBThreat intelligence atlas
AI Mitigation

Restrict AI Agent Tool Invocation on Untrusted Data - AI Mitigation

Untrusted data can contain prompt injections that invoke an AI agent's tools, potentially causing confidentiality, integrity or availability violations. It is recommended that tool invocation be restricted or limited when untrusted data enters the LLM's context. The degree to which tool invocation is restricted may depend on the potential consequences of the action. Consider blocking the automatic invocation of to...

View mitigationsRead context
AI MitigationDeploymentTechnical - ML

Record summary

A quick snapshot of what this page covers.

Techniques3Attacks this defense is designed to help with.
Lifecycle1Where this defense applies in the AI lifecycle.
Categories1How the source groups this defense.

Control summary

What this defense is meant to help prevent.

Untrusted data can contain prompt injections that invoke an AI agent's tools, potentially causing confidentiality, integrity or availability violations. It is recommended that tool invocation be restricted or limited when untrusted data enters the LLM's context.

The degree to which tool invocation is restricted may depend on the potential consequences of the action. Consider blocking the automatic invocation of tools or requiring user confirmation once untrusted data enters the LLM's context. For high consequence actions, consider always requiring user confirmation.

ATLAS ID
AML.M0030
Priority score
15
Deployment
Technical - ML

Covered techniques

Attacks this defense is designed to help with.

AML.T0053 - AI Agent Tool Invocation

demonstrated

Restricting the automatic tool use when untrusted data is present can prevent adversaries from invoking tools via prompt injections.

Source

Where this page information comes from.