Record summary
A quick snapshot of what this page covers.
Attack context
How this AI attack works in practice.
Adversaries may replicate a private model. By repeatedly querying the victim's AI Model Inference API Access, the adversary can collect the target model's inferences into a dataset. The inferences are used as labels for training a separate model offline that will mimic the behavior and performance of the target model.
A replicated model that closely mimic's the target model is a valuable resource in staging the attack. The adversary can use the replicated model to Craft Adversarial Data for various purposes (e.g. Evade AI Model, Spamming AI System with Chaff Data).
- ATLAS ID
- AML.T0005.001
- Priority score
- 49
Mitigations
Defenses that may help against this attack.
AML.M0024 - AI Telemetry Logging
Telemetry logging can help identify if a proxy training dataset has been exfiltrated.
AML.M0002 - Passive AI Output Obfuscation
Obfuscating model outputs restricts an adversary's ability to create an accurate proxy model by querying a model and observing its outputs.
AML.M0004 - Restrict Number of AI Model Queries
Restricting the number of queries to the model decreases an adversary's ability to replicate an accurate proxy model.
Case studies
Examples from public reports and exercises.
Attack on Machine Translation Services
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
ProofPoint Evasion
Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.
Source
Where this page information comes from.
Original source
Original source links
Open the public records and source datasets used for this page.