Generate Deepfakes - AI Security Technique

Overview

A source-backed snapshot of this AI security technique.

Adversaries may use generative artificial intelligence (GenAI) to create synthetic media (i.e. imagery, video, audio, and text) that appear authentic. These "deepfakes" may mimic a real person or depict fictional personas. Adversaries may use deepfakes for impersonation to conduct Phishing or to evade AI applications such as biometric identity verification systems (see Evade AI Model).

Manipulation of media has been possible for a long time, however GenAI reduces the skill and level of effort required, allowing adversaries to rapidly scale operations to target more users or systems. It also makes real-time manipulations feasible.

Adversaries may utilize open-source models and software that were designed for legitimate use cases to generate deepfakes for malicious use. However, there are some projects specifically tailored towards malicious use cases such as ProKYC.

Tactics1Attacker goals connected to this method.

Mitigations2Defenses that may help against this attack.

AI risks9Research-backed risks connected to this topic.

Technique details

Identifiers, maturity, and source taxonomy for this technique.

ATLAS ID: AML.T0088
Maturity: realized
Priority score: 101

ATLAS tactics

AI Attack Staging

Attack flow

How to read the public records connected to this technique.

1. TechniqueRead the ATLAS description and evidence level.

2. TacticsSee which attacker goals this method supports.

3. ExamplesCheck whether public case studies mention it.

4. DefensesReview safeguards mapped by ATLAS.

5. SourcesOpen the original public records and references.

Impact

Why this technique may deserve attention in the current dataset.

Evidence levelrealized
Mapped defenses2 ATLAS mitigation records
Public examples2 linked case study records
Research risks9 related MIT AI Risk records above the confidence threshold
Vulnerabilities0 linked CVE records

Mitigations

Defenses that may help against this attack.

2 recordsView all mitigations →

AML.M0034 - Deepfake Detection

Deepfake detection can be used to identify and block generated content.

LifecycleDeployment + 3 moreCategoryTechnical - ML

DeploymentMonitoring+2 more

AML.M0009 - Use Multi-Modal Sensors

Using a variety of sensors, such as IR depth cameras, can aid in detecting deepfakes.

LifecycleBusiness and Data Understanding + 2 moreCategoryTechnical - Cyber

B&D UnderstandingData Preparation+1 more

Case studies

Examples from public reports and exercises.

2 recordsView all case studies →

ProKYC: Deepfake Tool for Account Fraud Attacks

Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts.

The procedure below describes how a bad actor could use ProKYC’s service to bypass KYC verification.

Date2024-10-09

incident

Live Deepfake Image Injection to Evade Mobile KYC Verification

Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.

Date2024-10-01

exercise

Related risks

Research-backed risks connected to this topic.

9 recordsView all risks →

Impersonation/identity theft

"Impersonation/identity theft - Theft of an individual, group or organisation’s identity by a third-party in order to defraud, mock or otherwise harm them."

Domain4. Malicious Actors & MisuseSubdomain4.3 > Fraud, scams, and targeted manipulation

Confidence0.66

Harassment, Impersonation, and Extortion

"Deepfakes and other AI-generated content can be used to facilitate or exacerbate many of the harms listed throughout this report, but this section focuses on one subset: intentional, targeted abuse of individuals."

Domain4. Malicious Actors & MisuseSubdomain4.3 > Fraud, scams, and targeted manipulation

Confidence0.66

Malicious intent

"A frequent malicious use case of generative AI to harm, humiliate, or sexualize another person involves generating deepfakes of nonconsensual sexual imagery or videos."

Domain4. Malicious Actors & MisuseSubdomain4.3 > Fraud, scams, and targeted manipulation

Confidence0.66

Privacy and consent

"Even when a victim of targeted, AIgenerated harms successfully identifies a deepfake creator with malicious intent, they may still struggle to redress many harms because the generated image or video isn’t the victim...

Domain4. Malicious Actors & MisuseSubdomain4.3 > Fraud, scams, and targeted manipulation

Confidence0.66

Showing 4 of 9