Attack on Machine Translation Services - AI Case Study
AI Case StudyMachine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully trans...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Impact appears in 3 case steps.
- 2Multiple attack methods. The case connects to 9 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Reconnaissance The researchers used published research papers to identify the datasets and model architectures used by the target translation services.
-
Resource Development
Step 2
Datasets
The researchers gathered similar datasets that the target translation services used.
-
Resource Development
Step 3
Models
The researchers gathered similar model architectures that the target translation services used.
-
AI Model Access They abused a public facing application to query the model and produced machine translated sentence pairs as training data.
-
AI Attack Staging Using these translated sentence pairs, the researchers trained a model that replicates the behavior of the target model.
-
Impact By replicating the model with high fidelity, the researchers demonstrated that an adversary could steal a model and violate the victim's intellectual property rights.
-
AI Attack Staging
Step 7
Black-Box Transfer
The replicated models were used to generate adversarial examples that successfully transferred to the black-box translation services.
-
Impact
Step 8
Evade AI Model
The adversarial examples were used to evade the machine translation services by a variety of means. This included targeted word flips, vulgar outputs, and dropped sentences.
-
Impact
Step 9
Erode AI Model Integrity
Adversarial attacks can cause errors that cause reputational damage to the company of the translation service and decrease user trust in AI-powered services.
Mitigations
Defenses connected to the attack methods in this case.
AI Model Distribution Methods
Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Control Access to AI Models and Data at Rest
Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.