Record summary
A quick snapshot of what this page covers.
Attack context
How this AI attack works in practice.
Adversaries may search public sources, including cloud storage, public-facing services, and software or data repositories, to identify AI artifacts. These AI artifacts may include the software stack used to train and deploy models, training and testing data, model configurations and parameters. An adversary will be particularly interested in artifacts hosted by or associated with the victim organization as they may represent what that organization uses in a production environment. Adversaries may identify artifact repositories via other resources associated with the victim organization (e.g. Search Victim-Owned Websites or Search Open Technical Databases). These AI artifacts often provide adversaries with details of the AI task and approach.
AI artifacts can aid in an adversary's ability to Create Proxy AI Model. If these artifacts include pieces of the actual model in production, they can be used to directly Craft Adversarial Data. Acquiring some artifacts requires registration (providing user details such email/name), AWS keys, or written requests, and may require the adversary to Establish Accounts.
Artifacts might be hosted on victim-controlled infrastructure, providing the victim with some information on who has accessed that data.
- ATLAS ID
- AML.T0002
- Priority score
- 63
Mitigations
Defenses that may help against this attack.
AML.M0000 - Limit Public Release of Information
Limit the release of sensitive information in the metadata of deployed systems and publicly available applications.
Case studies
Examples from public reports and exercises.
ClearviewAI Misconfiguration
Clearview AI makes a facial recognition tool that searches publicly available photos for matches. This tool has been used for investigative purposes by law enforcement agencies and other parties.
Clearview AI's source code repository, though password protected, was misconfigured to allow an arbitrary user to register an account. This allowed an external researcher to gain access to a private code repository that contained Clearview AI production credentials, keys to cloud storage buckets containing 70K video samples, and copies of its applications and Slack tokens. With access to training data, a bad actor has the ability to cause an arbitrary misclassification in the deployed model. These kinds of attacks illustrate that any attempt to secure ML system should be on top of "traditional" good cybersecurity hygiene such as locking down the system with least privileges, multi-factor authentication and monitoring and auditing.
Microsoft Edge AI Evasion
The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.
Botnet Domain Generation Algorithm (DGA) Detection Evasion
The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. It is a generic domain mutation technique which can evade most ML-based DGA detection modules. The generic mutation technique evades most ML-based DGA detection modules DGA and can be used to test the effectiveness and robustness of all DGA detection methods developed by security companies in the industry before they is deployed to the production environment.
Source
Where this page information comes from.
Original source
Original source links
Open the public records and source datasets used for this page.