LLM Trusted Output Components Manipulation - AI Security Technique
AI Security TechniqueAdversaries may utilize prompts to a large language model (LLM) which manipulate various components of its response in order to make it appear trustworthy to the user. This helps the adversary continue to operate in the victim's environment and evade detection by the users it interacts with. The LLM may be instructed to tailor its language to appear more trustworthy to the user or attempt to manipulate the user to...
Overview
A source-backed snapshot of this AI security technique.
Adversaries may utilize prompts to a large language model (LLM) which manipulate various components of its response in order to make it appear trustworthy to the user. This helps the adversary continue to operate in the victim's environment and evade detection by the users it interacts with.
The LLM may be instructed to tailor its language to appear more trustworthy to the user or attempt to manipulate the user to take certain actions. Other response components that could be manipulated include links, recommended follow-up actions, retrieved document metadata, and Citations.
Technique details
Identifiers, maturity, and source taxonomy for this technique.
- ATLAS ID
- AML.T0067
- Maturity
- demonstrated
- Priority score
- 30
Attack flow
How to read the public records connected to this technique.
Impact
Why this technique may deserve attention in the current dataset.
- Evidence leveldemonstrated
- Mapped defenses0 ATLAS mitigation records
- Public examples1 linked case study records
- Research risks0 related MIT AI Risk records above the confidence threshold
- Vulnerabilities0 linked CVE records
Mitigations
Defenses that may help against this attack.
Case studies
Examples from public reports and exercises.
Rules File Backdoor: Supply Chain Attack on AI Coding Assistants
Pillar Security researchers demonstrated how adversaries can compromise AI-generated code by injecting malicious instructions into rules files used to configure AI coding assistants like Cursor and GitHub Copilot. The attack uses invisible Unicode characters to hide malicious prompts that manipulate the AI to insert backdoors, vulnerabilities, or malicious scripts into generated code. These poisoned rules files are distributed through open-source repositories and developer communities, creating a scalable supply chain attack that could affect millions of developers and end users through compromised software.
Vendor Response to Responsible Disclosure:
- Cursor: Determined that this risk falls under the users’ responsibility.
- GitHub Copilot: Implemented a new security feature that displays a warning when a file's contents include hidden Unicode text on github.com.
Source evidence
Original public records and references for this page.
Original source
Original source links
Open the public records and source datasets used for this page.