Black-Box Optimization - AI Security Technique
AI Security TechniqueIn Black-Box attacks, the adversary has black-box (i.e. AI Model Inference API Access via API access) access to the target model. With black-box attacks, the adversary may be using an API that the victim is monitoring. These attacks are generally less effective and require more inferences than White-Box Optimization attacks, but they require much less access.
Overview
A source-backed snapshot of this AI security technique.
Technique details
Identifiers, maturity, and source taxonomy for this technique.
- ATLAS ID
- AML.T0043.001
- Maturity
- demonstrated
- Priority score
- 61
Attack flow
How to read the public records connected to this technique.
Impact
Why this technique may deserve attention in the current dataset.
- Evidence leveldemonstrated
- Mapped defenses7 ATLAS mitigation records
- Public examples2 linked case study records
- Research risks0 related MIT AI Risk records above the confidence threshold
- Vulnerabilities0 linked CVE records
Mitigations
Defenses that may help against this attack.
AML.M0015 - Adversarial Input Detection
Monitor queries and query patterns to the target model, block access if suspicious queries are detected.
AML.M0019 - Control Access to AI Models and Data in Production
Access controls on model APIs can deny adversaries the access required for black-box optimization methods.
AML.M0010 - Input Restoration
Input restoration adds an extra layer of unknowns and randomness when an adversary evaluates the input-output relationship.
AML.M0003 - Model Hardening
Hardened models are more robust to adversarial inputs.
Showing 4 of 7
Case studies
Examples from public reports and exercises.
Microsoft Edge AI Evasion
The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.
Botnet Domain Generation Algorithm (DGA) Detection Evasion
The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. It is a generic domain mutation technique which can evade most ML-based DGA detection modules. The generic mutation technique evades most ML-based DGA detection modules DGA and can be used to test the effectiveness and robustness of all DGA detection methods developed by security companies in the industry before they is deployed to the production environment.
Source evidence
Original public records and references for this page.
Original source
Original source links
Open the public records and source datasets used for this page.