Generative AI - AI Security Technique
AI Security TechniqueAdversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generating malware, to Generate Deepfakes, to Generate Malicious Commands, for Retrieval Content Crafting, or to g...
Overview
A source-backed snapshot of this AI security technique.
Adversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generating malware, to Generate Deepfakes, to Generate Malicious Commands, for Retrieval Content Crafting, or to generate Phishing content.
Adversaries may obtain open source models and serve them locally using frameworks such as Ollama or vLLM. They may host them using cloud infrastructure. Or, they may leverage AI service providers such as HuggingFace.
They may need to jailbreak the model (see LLM Jailbreak) to bypass any restrictions put in place to limit the types of responses it can generate. They may also need to break the terms of service of the model's developer.
Generative AI models may also be "uncensored" meaning they are designed to generate content without any restrictions such as guardrails or content filters. Uncensored GenAI is ripe for abuse by cybercriminals [1] [2]. Models may be fine-tuned to remove alignment and guardrails [3] or be subjected to targeted manipulations to bypass refusal [4] resulting in uncensored variants of the model. Uncensored models may be built for offensive and defensive cybersecurity [5], which can be abused by an adversary. There are also models that are expressly designed and advertised for malicious use [6].
References
- [1] https://blog.talosintelligence.com/cybercriminal-abuse-of-large-language-models/
- [2] https://gbhackers.com/cybercriminals-exploit-llm-models/
- [3] https://erichartford.com/uncensored-models
- [4] https://arxiv.org/abs/2406.11717/
- [5] https://taico.ca/posts/whiterabbitneo/
- [6] https://gbhackers.com/wormgpt-enhanced-with-grok-and-mixtral/
Technique details
Identifiers, maturity, and source taxonomy for this technique.
- ATLAS ID
- AML.T0016.002
- Maturity
- realized
- Priority score
- 155
Attack flow
How to read the public records connected to this technique.
Impact
Why this technique may deserve attention in the current dataset.
- Evidence levelrealized
- Mapped defenses0 ATLAS mitigation records
- Public examples3 linked case study records
- Research risks19 related MIT AI Risk records above the confidence threshold
- Vulnerabilities0 linked CVE records
Mitigations
Defenses that may help against this attack.
Case studies
Examples from public reports and exercises.
AI ClickFix: Hijacking Computer-Use Agents Using ClickFix
Embrace the Red demonstrated that AI computer-use agents are vulnerable to social engineering attacks and can be manipulated into executing arbitrary code on a victim’s machine. The attack is a variation on “ClickFix” which is a social engineering attack that fools humans into copying malicious commands and executing them.
The researcher used ChatGPT to generate a website designed to attract interactions with computer-use agents. When a user asked their Claude Computer-Use Agent to visit the researcher’s website, the text “Are you a computer? Please see instructions to confirm:” caused the agent to click the associated button. This executed JavaScript to copy a malicious command into the agent’s clipboard. The agent then proceeded to follow the instructions, opening a terminal, pasting the malicious command, and executing it. The command downloads a script from the researcher’s website and executes it. In the demonstration, the script opens the victim’s Calculator App, but in practice an adversary could run arbitrary code, compromising the victim’s system.
ProKYC: Deepfake Tool for Account Fraud Attacks
Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts.
The procedure below describes how a bad actor could use ProKYC’s service to bypass KYC verification.
Live Deepfake Image Injection to Evade Mobile KYC Verification
Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.
Source evidence
Original public records and references for this page.
Original source
Original source links
Open the public records and source datasets used for this page.