APromptRiskDBThreat intelligence atlas
AI Case Study

Attempted Evasion of ML Phishing Webpage Detection System - AI Case Study

Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs...

View attack chainRead summary
IncidentCommercial ML Phishing Webpage DetectorUnknownAI Attack StagingDefense EvasionInitial Access

Overview

Case steps4Steps described in the case record.
Techniques4Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. AI Attack Staging appears in 1 case steps.
  • 2Multiple attack methods. The case connects to 4 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

AI Attack Staging1Defense Evasion1Initial Access1Impact1
  1. AI Attack Staging

    Several cheap, yet effective strategies for manually modifying logos were observed: | Evasive Strategy | Count | | - | - | | Company name style | 25 | | Blurry logo | 23 | | Cropping | 20 | | No company name | 16 | | No visual logo | 13 | | Different visual logo | 12 | | Logo stretching | 11 | | Multiple forms - images | 10 | | Background patterns | 8 | | Login obfuscation | 6 | | Masking | 3 |

  2. Defense Evasion

    The visual similarity model used to detect brand impersonation was evaded. However, other components of the phishing detection system successfully identified the phishing websites.

  3. Step 3

    Phishing

    Initial Access

    If the adversary can successfully evade detection, they can continue to operate their phishing websites and steal the victim's credentials.

  4. Step 4

    User Harm

    Impact

    The end user may experience a variety of harms including financial and privacy harms depending on the credentials stolen by the adversary.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

Adversarial Input Detection

Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.

Deepfake Detection

Apply deepfake detection algorithms against any untrusted or user-provided data, especially in impactful applications such as biometric verification, to block generated content. Detectors may use a combination of approaches, including: - AI models trained to differentiate between real and deepfake content. - Identifying common inconsistencies in deepfake content, such as unnatural facial movements, audio mismatches, or pixel-level artifacts. - Biometrics analysis, such blinking, eye movements, and microexpressions.

Input Restoration

Preprocess all inference data to nullify or reverse potential adversarial perturbations.

Model Hardening

Use techniques to make AI models robust to adversarial inputs such as adversarial training or network distillation.

Restrict Number of AI Model Queries

Limit the total number and rate of queries a user can perform.

Use Ensemble Methods

Use an ensemble of models for inference to increase robustness to adversarial inputs. Some attacks may effectively evade one model or model family but be ineffective against others.

Use Multi-Modal Sensors

Incorporate multiple sensors to integrate varying perspectives and modalities to avoid a single point of failure susceptible to physical attacks.

User Training

Educate AI model developers to on AI supply chain risks and potentially malicious AI artifacts. Educate users on how to identify deepfakes and phishing attempts.

Sources

Original public records and references for this case.

Original source

Original source links

Open the MITRE ATLAS data and public references used for this case study.

Repositoryhttps://github.com/mitre-atlas/atlas-dataATLAS.yamlhttps://github.com/mitre-atlas/atlas-data/blob/main/dist/ATLAS.yamlSchemahttps://github.com/mitre-atlas/atlas-data/blob/main/dist/schemas/atlas_output_schema.json"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicehttps://arxiv.org/abs/2212.14315Real Attackers Don't Compute Gradients Supplementary Resourceshttps://real-gradients.github.io/