ProofPoint Evasion - AI Case Study
AI Case StudyProof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numeri...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. AI Attack Staging appears in 2 case steps.
- 2Multiple attack methods. The case connects to 5 unique AI attack methods.
- 3Vulnerability mentions. The record connects 1 vulnerability identifiers to this case.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Discovery The researchers discovered that ProofPoint's Email Protection left model output scores in email headers.
-
AI Model Access The researchers sent many emails through the system to collect model outputs from the headers.
-
AI Attack Staging The researchers used the emails and collected scores as a dataset, which they used to train a functional copy of the ProofPoint model. Basic correlation was used to decide which score variable speaks generally about the security of an email. The "mlxlogscore" was selected in this case due to its relationship with spam, phish, and core mlx and was used as the label. Each "mlxlogscore" was generally between 1 and 999 (higher score = safer sample). Training was performed using an Artificial Neural Network (ANN) and Bag of Words tokenizing.
-
AI Attack Staging
Step 4
Black-Box Transfer
Next, the ML researchers algorithmically found samples from this "offline" proxy model that helped give desired insight into its behavior and influential variables. Examples of good scoring samples include "calculation", "asset", and "tyson". Examples of bad scoring samples include "software", "99", and "unsub".
-
Impact
Step 5
Evade AI Model
Finally, these insights from the "offline" proxy model allowed the researchers to create malicious emails that received preferable scores from the real ProofPoint email protection system, hence bypassing it.
Mitigations
Defenses connected to the attack methods in this case.
AI Model Distribution Methods
Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Control Access to AI Models and Data in Production
Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.