Confusing Antimalware Neural Networks - AI Case Study
AI Case StudyCloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models wit...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. AI Attack Staging appears in 3 case steps.
- 2Multiple attack methods. The case connects to 9 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Reconnaissance The researchers performed a review of adversarial ML attacks on antimalware products. They discovered that techniques borrowed from attacks on image classifiers have been successfully applied to the antimalware domain. However, it was not clear if these approaches were effective against the ML component of production antimalware solutions.
-
Reconnaissance Kaspersky's use of ML-based antimalware detectors is publicly documented on their website. In practice, an adversary could use this for targeting.
-
AI Model Access The researchers used access to the target ML-based antimalware product throughout this case study. This product scans files on the user's system, extracts features locally, then sends them to the cloud-based ML malware detector for classification. Therefore, the researchers had only black-box access to the malware detector itself, but could learn valuable information for constructing the attack from the feature extractor.
-
Resource Development
Step 4
Datasets
The researchers collected a dataset of malware and clean files. They scanned the dataset with the target ML-based antimalware solution and labeled the samples according to the ML detector's predictions.
-
AI Attack Staging
Step 5
Create Proxy AI Model
A proxy model was trained on the labeled dataset of malware and clean files. The researchers experimented with a variety of model architectures.
-
Resource Development
Step 6
Adversarial AI Attacks
By reverse engineering the local feature extractor, the researchers could collect information about the input features, used for the cloud-based ML detector. The model collects PE Header features, section features and section data statistics, and file strings information. A gradient based adversarial algorithm for executable files was developed. The algorithm manipulates file features to avoid detection by the proxy model, while still containing the same malware payload
-
AI Attack Staging
Step 7
Black-Box Transfer
Using a developed gradient-driven algorithm, malicious adversarial files for the proxy model were constructed from the malware files for black-box transfer to the target model.
-
AI Attack Staging
Step 8
Verify Attack
The adversarial malware files were tested against the target antimalware solution to verify their efficacy.
-
Defense Evasion
Step 9
Evade AI Model
The researchers demonstrated that for most of the adversarial files, the antimalware model was successfully evaded. In practice, an adversary could deploy their adversarially crafted malware and infect systems while evading detection.
Mitigations
Defenses connected to the attack methods in this case.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Control Access to AI Models and Data at Rest
Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.
Control Access to AI Models and Data in Production
Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.