Record summary
A quick snapshot of what this page covers.
Attack context
How this AI attack works in practice.
Adversaries may turn LLMs into targeted social engineers. LLMs are capable of interacting with users via text conversations. They can be instructed by an adversary to seek sensitive information from a user and act as effective social engineers. They can be targeted towards particular personas defined by the adversary. This allows adversaries to scale spearphishing efforts and target individuals to reveal private information such as credentials to privileged systems.
- ATLAS ID
- AML.T0052.000
- Priority score
- 36
Mitigations
Defenses that may help against this attack.
AML.M0034 - Deepfake Detection
Deepfake detection can be used to identify and block phishing attempts that use generated content.
AML.M0018 - User Training
Train users to identify phishing attempts and understand that AI can be used to generate targeted and convincing messages.
Case studies
Examples from public reports and exercises.
Indirect Prompt Injection Threats: Bing Chat Data Pirate
Whenever interacting with Microsoft's new Bing Chat LLM Chatbot, a user can allow Bing Chat permission to view and access currently open websites throughout the chat session. Researchers demonstrated the ability for an attacker to plant an injection in a website the user is visiting, which silently turns Bing Chat into a Social Engineer who seeks out and exfiltrates personal information. The user doesn't have to ask about the website or do anything except interact with Bing Chat while the website is opened in the browser in order for this attack to be executed.
In the provided demonstration, a user opened a prepared malicious website containing an indirect prompt injection attack (could also be on a social media site) in Edge. The website includes a prompt which is read by Bing and changes its behavior to access user information, which in turn can sent to an attacker.
Source
Where this page information comes from.
Original source
Original source links
Open the public records and source datasets used for this page.