Spearphishing via Social Engineering LLM - AI Security Technique
AI Security TechniqueAdversaries may turn LLMs into targeted social engineers. LLMs are capable of interacting with users via text conversations. They can be instructed by an adversary to seek sensitive information from a user and act as effective social engineers. They can be targeted towards particular personas defined by the adversary. This allows adversaries to scale spearphishing efforts and target individuals to reveal private i...
Overview
A source-backed snapshot of this AI security technique.
Adversaries may turn LLMs into targeted social engineers. LLMs are capable of interacting with users via text conversations. They can be instructed by an adversary to seek sensitive information from a user and act as effective social engineers. They can be targeted towards particular personas defined by the adversary. This allows adversaries to scale spearphishing efforts and target individuals to reveal private information such as credentials to privileged systems.
Technique details
Identifiers, maturity, and source taxonomy for this technique.
- ATLAS ID
- AML.T0052.000
- Maturity
- demonstrated
- Priority score
- 36
Attack flow
How to read the public records connected to this technique.
Impact
Why this technique may deserve attention in the current dataset.
- Evidence leveldemonstrated
- Mapped defenses2 ATLAS mitigation records
- Public examples1 linked case study records
- Research risks0 related MIT AI Risk records above the confidence threshold
- Vulnerabilities0 linked CVE records
Mitigations
Defenses that may help against this attack.
AML.M0034 - Deepfake Detection
Deepfake detection can be used to identify and block phishing attempts that use generated content.
AML.M0018 - User Training
Train users to identify phishing attempts and understand that AI can be used to generate targeted and convincing messages.
Case studies
Examples from public reports and exercises.
Indirect Prompt Injection Threats: Bing Chat Data Pirate
Whenever interacting with Microsoft's new Bing Chat LLM Chatbot, a user can allow Bing Chat permission to view and access currently open websites throughout the chat session. Researchers demonstrated the ability for an attacker to plant an injection in a website the user is visiting, which silently turns Bing Chat into a Social Engineer who seeks out and exfiltrates personal information. The user doesn't have to ask about the website or do anything except interact with Bing Chat while the website is opened in the browser in order for this attack to be executed.
In the provided demonstration, a user opened a prepared malicious website containing an indirect prompt injection attack (could also be on a social media site) in Edge. The website includes a prompt which is read by Bing and changes its behavior to access user information, which in turn can sent to an attacker.
Source evidence
Original public records and references for this page.
Original source
Original source links
Open the public records and source datasets used for this page.