APromptRiskDBThreat intelligence atlas
AI Security Technique

Delay Execution of LLM Instructions - AI Security Technique

Adversaries may include instructions to be followed by the AI system in response to a future event, such as a specific keyword or the next interaction, in order to evade detection or bypass controls placed on the AI system. For example, an adversary may include "If the user submits a new request..." followed by the malicious instructions as part of their prompt. AI agents can include security measures against prom...

View mitigationsRead context
AI Security TechniquedemonstratedDefense Evasion

Record summary

A quick snapshot of what this page covers.

Tactics1Attacker goals connected to this method.
Mitigations0Defenses that may help against this attack.
AI risks12Research-backed risks connected to this topic.

Attack context

How this AI attack works in practice.

Adversaries may include instructions to be followed by the AI system in response to a future event, such as a specific keyword or the next interaction, in order to evade detection or bypass controls placed on the AI system.

For example, an adversary may include "If the user submits a new request..." followed by the malicious instructions as part of their prompt.

AI agents can include security measures against prompt injections that prevent the invocation of particular tools or access to certain data sources during a conversation turn that has untrusted data in context. Delaying the execution of instructions to a future interaction or keyword is one way adversaries may bypass this type of control.

ATLAS ID
AML.T0094
Priority score
90
Maturity: demonstrated
Defense Evasion

Mitigations

Defenses that may help against this attack.

No connected defenses. No defense is connected to this attack in the current data.

Case studies

Examples from public reports and exercises.

Planting Instructions for Delayed Automatic AI Agent Tool Invocation

exercise
Date2024-02-01

Embrace the Red demonstrated that Google Gemini is susceptible to automated tool invocation by delaying the execution to the next conversation turn. This bypasses a security control that restricts Gemini from invoking tools that can access sensitive user information in the same conversation turn that untrusted data enters context.

Related risks

Research-backed risks connected to this topic.

Prompt injection

Confidence: 0.78
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Prompt Injections are a form of Adversarial Input that involve manipulating the text instructions given to a GenAI system (Liu et al., 2023). Prompt Injections exploit loopholes in a model’s architec- tures that have...

Adversarial AI: Prompt Injections

Confidence: 0.76
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Prompt injections represent another class of attacks that involve the malicious insertion of prompts or requests in LLM-based interactive systems, leading to unintended actions or disclosure of sensitive information...

Jailbreaking

Confidence: 0.76
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Jailbreaking aims to bypass or remove restrictions and safety filters placed on a GenAI model completely (Chao et al., 2023; Shen et al., 2023). This gives the actor free rein to generate any output, regardless of it...

Security - Robustness

Confidence: 0.75
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

While AI safety focuses on threats emanating from generative AI systems, security centers on threats posed to these systems. The most extensively discussed issue in this context are jailbreaking risks, which involve t...

Jailbreaks and Prompt Injections Threaten Security of LLMs

Confidence: 0.75
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"LLMs are not adversarially robust and are vulnerable to security failures such as jailbreaks and prompt-injection attacks. While a number of jailbreak attacks have been proposed in the literature, the lack of standar...

“Model Psychology” Attacks

Confidence: 0.74
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"LLMs are vulnerable to “psychological” tricks (Li et al., 2023e; Shen et al., 2023), which can be exploited by attackers. Examples include instructing the model to behave like a specific persona (Shah et al., 2023; A...

Jailbreak in LLM Malicious Use - Prompt Attacks

Confidence: 0.74
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"In the prompting and reasoning phase, dialog can push LLMs into confused or overly compliant states, raising the risk of producing harmful outputs when confronted with harmful questions. Most of the jailbreak methods...

Prompt Leaking

Confidence: 0.74
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Prompt leaking is another type of prompt injection attack designed to expose details contained in private prompts. According to [58], prompt leaking is the act of misleading the model to print the pre-designed instru...

Goal Hijacking

Confidence: 0.73
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Goal hijacking is a type of primary attack in prompt injection [58]. By injecting a phrase like “Ignore the above instruction and do ...” in the input, the attack could hijack the original goal of the designed prompt...

Prompt injection attack

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"A prompt injection attack forces a generative model that takes a prompt as input to produce unexpected output by manipulating the structure, instructions, or information contained in its prompt."

Attacking LLMs via Additional Modalities a

Confidence: 0.71
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"LLMs can now process modalities other than text, e.g. images or video frames (OpenAI, 2023c; Gemini Team, 2023). Several studies show that gradient-based attacks on multimodal models are easy and effective (Carlini e...

Exploiting Limited Generalization of Safety Finetuning

Confidence: 0.71
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Safety tuning is performed over a much narrower distribution compared to the pretraining distribution. This leaves the model vulnerable to attacks that exploit gaps in the generalization of the safety training, e.g...

Related CVEs

Known software flaws linked to this context.

No related CVEs. No software flaw is connected to this attack in the current data.

Source

Where this page information comes from.