Record summary
A quick snapshot of what this page covers.
Attack context
How this AI attack works in practice.
Adversaries may utilize prompts to a large language model (LLM) which manipulate various components of its response in order to make it appear trustworthy to the user. This helps the adversary continue to operate in the victim's environment and evade detection by the users it interacts with.
The LLM may be instructed to tailor its language to appear more trustworthy to the user or attempt to manipulate the user to take certain actions. Other response components that could be manipulated include links, recommended follow-up actions, retrieved document metadata, and Citations.
- ATLAS ID
- AML.T0067
- Priority score
- 30
Mitigations
Defenses that may help against this attack.
Case studies
Examples from public reports and exercises.
Rules File Backdoor: Supply Chain Attack on AI Coding Assistants
Pillar Security researchers demonstrated how adversaries can compromise AI-generated code by injecting malicious instructions into rules files used to configure AI coding assistants like Cursor and GitHub Copilot. The attack uses invisible Unicode characters to hide malicious prompts that manipulate the AI to insert backdoors, vulnerabilities, or malicious scripts into generated code. These poisoned rules files are distributed through open-source repositories and developer communities, creating a scalable supply chain attack that could affect millions of developers and end users through compromised software.
Vendor Response to Responsible Disclosure:
- Cursor: Determined that this risk falls under the users’ responsibility.
- GitHub Copilot: Implemented a new security feature that displays a warning when a file's contents include hidden Unicode text on github.com.
Source
Where this page information comes from.
Original source
Original source links
Open the public records and source datasets used for this page.