Microsoft Azure Service Disruption - AI Case Study
AI Case StudyThe Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. AI Attack Staging appears in 2 case steps.
- 2Multiple attack methods. The case connects to 8 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Reconnaissance The team first performed reconnaissance to gather information about the target ML model.
-
Initial Access
Step 2
Valid Accounts
The team used a valid account to gain access to the network.
-
Collection
Step 3
AI Artifact Collection
The team found the model file of the target ML model and the necessary training data.
-
Exfiltration The team exfiltrated the model and data via traditional means.
-
AI Attack Staging
Step 5
White-Box Optimization
Using the target model and data, the red team crafted evasive adversarial data in an offline manor.
-
AI Model Access The team used an exposed API to access the target model.
-
AI Attack Staging
Step 7
Verify Attack
The team submitted the adversarial examples to the API to verify their efficacy on the production system.
-
Impact
Step 8
Evade AI Model
The team performed an online evasion attack by replaying the adversarial examples and accomplished their goals.
Mitigations
Defenses connected to the attack methods in this case.
AI Model Distribution Methods
Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Control Access to AI Models and Data at Rest
Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.