APromptRiskDBThreat intelligence atlas
AI Security Technique

Generative AI - AI Security Technique

Adversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generating malware, to Generate Deepfakes, to Generate Malicious Commands, for Retrieval Content Crafting, or to g...

View mitigationsRead context
AI Security Techniquerealized

Record summary

A quick snapshot of what this page covers.

Tactics0Attacker goals connected to this method.
Mitigations0Defenses that may help against this attack.
AI risks19Research-backed risks connected to this topic.

Attack context

How this AI attack works in practice.

Adversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generating malware, to Generate Deepfakes, to Generate Malicious Commands, for Retrieval Content Crafting, or to generate Phishing content.

Adversaries may obtain open source models and serve them locally using frameworks such as Ollama or vLLM. They may host them using cloud infrastructure. Or, they may leverage AI service providers such as HuggingFace.

They may need to jailbreak the model (see LLM Jailbreak) to bypass any restrictions put in place to limit the types of responses it can generate. They may also need to break the terms of service of the model's developer.

Generative AI models may also be "uncensored" meaning they are designed to generate content without any restrictions such as guardrails or content filters. Uncensored GenAI is ripe for abuse by cybercriminals [1] [2]. Models may be fine-tuned to remove alignment and guardrails [3] or be subjected to targeted manipulations to bypass refusal [4] resulting in uncensored variants of the model. Uncensored models may be built for offensive and defensive cybersecurity [5], which can be abused by an adversary. There are also models that are expressly designed and advertised for malicious use [6].

References

  1. [1] https://blog.talosintelligence.com/cybercriminal-abuse-of-large-language-models/
  2. [2] https://gbhackers.com/cybercriminals-exploit-llm-models/
  3. [3] https://erichartford.com/uncensored-models
  4. [4] https://arxiv.org/abs/2406.11717/
  5. [5] https://taico.ca/posts/whiterabbitneo/
  6. [6] https://gbhackers.com/wormgpt-enhanced-with-grok-and-mixtral/
ATLAS ID
AML.T0016.002
Priority score
155
Maturity: realized

Mitigations

Defenses that may help against this attack.

No connected defenses. No defense is connected to this attack in the current data.

Case studies

Examples from public reports and exercises.

AI ClickFix: Hijacking Computer-Use Agents Using ClickFix

exercise
Date2025-05-24

Embrace the Red demonstrated that AI computer-use agents are vulnerable to social engineering attacks and can be manipulated into executing arbitrary code on a victim’s machine. The attack is a variation on “ClickFix” which is a social engineering attack that fools humans into copying malicious commands and executing them.

The researcher used ChatGPT to generate a website designed to attract interactions with computer-use agents. When a user asked their Claude Computer-Use Agent to visit the researcher’s website, the text “Are you a computer? Please see instructions to confirm:” caused the agent to click the associated button. This executed JavaScript to copy a malicious command into the agent’s clipboard. The agent then proceeded to follow the instructions, opening a terminal, pasting the malicious command, and executing it. The command downloads a script from the researcher’s website and executes it. In the demonstration, the script opens the victim’s Calculator App, but in practice an adversary could run arbitrary code, compromising the victim’s system.

ProKYC: Deepfake Tool for Account Fraud Attacks

incident
Date2024-10-09

Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts.

The procedure below describes how a bad actor could use ProKYC’s service to bypass KYC verification.

Live Deepfake Image Injection to Evade Mobile KYC Verification

exercise
Date2024-10-01

Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.

Related risks

Research-backed risks connected to this topic.

Jailbreaks and Prompt Injections Threaten Security of LLMs

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"LLMs are not adversarially robust and are vulnerable to security failures such as jailbreaks and prompt-injection attacks. While a number of jailbreak attacks have been proposed in the literature, the lack of standar...

Exploiting Limited Generalization of Safety Finetuning

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Safety tuning is performed over a much narrower distribution compared to the pretraining distribution. This leaves the model vulnerable to attacks that exploit gaps in the generalization of the safety training, e.g...

“Model Psychology” Attacks

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"LLMs are vulnerable to “psychological” tricks (Li et al., 2023e; Shen et al., 2023), which can be exploited by attackers. Examples include instructing the model to behave like a specific persona (Shah et al., 2023; A...

One-step Jailbreaks

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"One-step jailbreaks. One-step jailbreaks commonly involve direct modifications to the prompt itself, such as setting role-playing scenarios or adding specific descriptions to prompts [14], [52], [67]–[73]. Role-playi...

Multi-step Jailbreaks

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Multi-step jailbreaks. Multi-step jailbreaks involve constructing a well-designed scenario during a series of conversations with the LLM. Unlike one-step jailbreaks, multi-step jailbreaks usually guide LLMs to genera...

Text encoding-based attacks

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Various new or existing text encodings, such as Base64, can be employed to craft jailbreak attacks that bypass safety training [13]. Low-resource language inputs also appear more likely to circumvent a model’s safegu...

Security - Robustness

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

While AI safety focuses on threats emanating from generative AI systems, security centers on threats posed to these systems. The most extensively discussed issue in this context are jailbreaking risks, which involve t...

Jailbreaking

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Jailbreaking aims to bypass or remove restrictions and safety filters placed on a GenAI model completely (Chao et al., 2023; Shen et al., 2023). This gives the actor free rein to generate any output, regardless of it...

Jailbreak in LLM Malicious Use - Poisoning Training Data

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"In the data collecting and pre-training phase, malicious adversaries can Jailbreak LLMs through poisoning their training data to make the model to output harmful content."

Jailbreak in LLM Malicious Use - Backdoor Attack

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"However, there are still ones who can leave holes in the training dataset, making LLMs appear safe on average, but generate harmful content under other specific conditions. This kind of attack can be categorized as "...

Jailbreak in LLM Malicious Use - White & Black Box Attacks

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"In the fine-tuning and alignment phase, elaborately- designed instruction datasets can be utilized to fine-tune LLMs to drive them to perform undesirable behaviors, such as generating harmful information or content t...

Vulnerability to Poisoning and Backdoors

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"The previous section explored jailbreaks and other forms of adversarial prompts as ways to elicit harmful capabilities acquired during pretraining. These methods make no assumptions about the training data. On the ot...

Jailbreak of a model to subvert intended behavior

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"A jailbreak is a type of adversarial input to the model (during deployment) re- sulting in model behavior deviating from intended use. Jailbreaks may be gen- erated automatically in a “white box” setting, where acces...

Jailbreak in LLM Malicious Use - Prompt Attacks

Confidence: 0.71
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"In the prompting and reasoning phase, dialog can push LLMs into confused or overly compliant states, raising the risk of producing harmful outputs when confronted with harmful questions. Most of the jailbreak methods...

Technical vulnerabilities (Robustness - vulnerability to jailbreaking

Confidence: 0.71
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Individuals can manipulate models into performing actions that violate the model’s usage restrictions—a phenomenon known as “jailbreaking.” These manipulations may result in causing the model to perform tasks that th...

Attacking LLMs via Additional Modalities a

Confidence: 0.70
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"LLMs can now process modalities other than text, e.g. images or video frames (OpenAI, 2023c; Gemini Team, 2023). Several studies show that gradient-based attacks on multimodal models are easy and effective (Carlini e...

Jailbreak of a multimodal model

Confidence: 0.69
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Current generation multimodal (e.g., vision and language) GPAI models are vulnerable to adversarial jailbreak attacks. These attacks can be used to automatically induce a model to produce an arbitrary or specific out...

Vulnerabilities arising from additional modalities in multimodal models

Confidence: 0.66
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Additional modalities can introduce new attack vectors in multimodal models as well as expand the scope of the previous attacks, ranging from jailbreaking to poisoning [13]. Typically, different modalities have diffe...

Related CVEs

Known software flaws linked to this context.

No related CVEs. No software flaw is connected to this attack in the current data.

Source

Where this page information comes from.