Attempted Evasion of ML Phishing Webpage Detection System - AI Case Study
AI Case StudyAdversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. AI Attack Staging appears in 1 case steps.
- 2Multiple attack methods. The case connects to 4 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
AI Attack Staging
Step 1
Manual Modification
Several cheap, yet effective strategies for manually modifying logos were observed: | Evasive Strategy | Count | | - | - | | Company name style | 25 | | Blurry logo | 23 | | Cropping | 20 | | No company name | 16 | | No visual logo | 13 | | Different visual logo | 12 | | Logo stretching | 11 | | Multiple forms - images | 10 | | Background patterns | 8 | | Login obfuscation | 6 | | Masking | 3 |
-
Defense Evasion
Step 2
Evade AI Model
The visual similarity model used to detect brand impersonation was evaded. However, other components of the phishing detection system successfully identified the phishing websites.
-
Initial Access
Step 3
Phishing
If the adversary can successfully evade detection, they can continue to operate their phishing websites and steal the victim's credentials.
-
Impact
Step 4
User Harm
The end user may experience a variety of harms including financial and privacy harms depending on the credentials stolen by the adversary.
Mitigations
Defenses connected to the attack methods in this case.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Deepfake Detection
Apply deepfake detection algorithms against any untrusted or user-provided data, especially in impactful applications such as biometric verification, to block generated content.
Detectors may use a combination of approaches, including:
- AI models trained to differentiate between real and deepfake content.
- Identifying common inconsistencies in deepfake content, such as unnatural facial movements, audio mismatches, or pixel-level artifacts.
- Biometrics analysis, such blinking, eye movements, and microexpressions.
Input Restoration
Preprocess all inference data to nullify or reverse potential adversarial perturbations.
Model Hardening
Use techniques to make AI models robust to adversarial inputs such as adversarial training or network distillation.
Showing 4 of 8
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.