Phishing - AI Security Technique

Record summary

A quick snapshot of what this page covers.

Tactics2Attacker goals connected to this method.

Mitigations2Defenses that may help against this attack.

AI risks10Research-backed risks connected to this topic.

Attack context

How this AI attack works in practice.

Adversaries may send phishing messages to gain access to victim systems. All forms of phishing are electronically delivered social engineering. Phishing can be targeted, known as spearphishing. In spearphishing, a specific individual, company, or industry will be targeted by the adversary. More generally, adversaries can conduct non-targeted phishing, such as in mass malware spam campaigns.

Generative AI, including LLMs that generate synthetic text, visual deepfakes of faces, and audio deepfakes of speech (See Generate Deepfakes), is enabling adversaries to scale targeted phishing campaigns (See Spearphishing via Social Engineering LLM). LLMs can interact with users via text conversations and can be programmed with a system prompt to phish for sensitive information. Deepfakes can also be used in Impersonation as an aid to phishing.

ATLAS ID: AML.T0052
ATT&CK external ID: T1566
Priority score: 106

Maturity: realized

Initial AccessLateral Movement

Mitigations

Defenses that may help against this attack.

AML.M0034 - Deepfake Detection

DeploymentMonitoring and Maintenance+2 more

LifecycleDeployment + 3 moreCategoryTechnical - ML

Deepfake detection can be used to identify and block phishing attempts that use generated content.

AML.M0018 - User Training

Business and Data UnderstandingData Preparation+4 more

LifecycleBusiness and Data Understanding + 5 moreCategoryPolicy

Train users to identify phishing attempts by an adversary to reduce the risk of successful spearphishing, social engineering, and other techniques that involve user interaction.

Case studies

Examples from public reports and exercises.

LAMEHUG: Malware Leveraging Dynamic AI-Generated Commands

incident

Date2025-06-03

In July 2025, Ukrainian authorities reported the emergence of LAMEHUG, a new AI-powered malware attributed to the Russian state-backed threat actor APT28 (also tracked as Forest Blizzard or UAC-0001). LAMEHUG uses a large language model (LLM) to dynamically generate commands on the infected hosts.

The campaign began with a phishing attack leveraging a compromised government email account to deliver a malicious ZIP archive disguised as Appendix.pdf.zip. The archive contained the LAMEHUG malware, a Python-based executable, packed with PyInstaller. When executed, the malware, makes calls to an LLM endpoint to generate malicious from natural language prompts. Dynamically generated commands may make the malware harder to detect. LAMEHUG was configured to collect files from the local system and exfiltrate them.

Attempted Evasion of ML Phishing Webpage Detection System

incident

Date2022-12-01

Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products.

In this incident, adversarial examples were identified in the logs of a commercial machine learning phishing website detection system. The detection system makes an automated block/allow determination from the "phishing score" of an ensemble of image classifiers each responsible for different phishing indicators (visual similarity, input form detection, etc.). The adversarial examples appeared to employ several simple yet effective strategies for manually modifying brand logos in an attempt to evade image classification models. The phishing websites which employed logo modification methods successfully evaded the model responsible detecting brand impersonation via visual similarity. However, the other components of the system successfully flagged the phishing websites.