APromptRiskDBThreat intelligence atlas
AI Case Study

Confusing Antimalware Neural Networks - AI Case Study

Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models wit...

View attack chainRead summary
ExerciseKaspersky's Antimalware ML ModelKaspersky ML Research TeamAI Attack StagingReconnaissanceResource Development

Overview

Case steps9Steps described in the case record.
Techniques9Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. AI Attack Staging appears in 3 case steps.
  • 2Multiple attack methods. The case connects to 9 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

AI Attack Staging3Reconnaissance2Resource Development2AI Model Access1Defense Evasion1
  1. Reconnaissance

    The researchers performed a review of adversarial ML attacks on antimalware products. They discovered that techniques borrowed from attacks on image classifiers have been successfully applied to the antimalware domain. However, it was not clear if these approaches were effective against the ML component of production antimalware solutions.

  3. AI Model Access

    The researchers used access to the target ML-based antimalware product throughout this case study. This product scans files on the user's system, extracts features locally, then sends them to the cloud-based ML malware detector for classification. Therefore, the researchers had only black-box access to the malware detector itself, but could learn valuable information for constructing the attack from the feature extractor.

  4. Step 4

    Datasets

    Resource Development

    The researchers collected a dataset of malware and clean files. They scanned the dataset with the target ML-based antimalware solution and labeled the samples according to the ML detector's predictions.

  5. AI Attack Staging

    A proxy model was trained on the labeled dataset of malware and clean files. The researchers experimented with a variety of model architectures.

  6. Resource Development

    By reverse engineering the local feature extractor, the researchers could collect information about the input features, used for the cloud-based ML detector. The model collects PE Header features, section features and section data statistics, and file strings information. A gradient based adversarial algorithm for executable files was developed. The algorithm manipulates file features to avoid detection by the proxy model, while still containing the same malware payload

  7. AI Attack Staging

    Using a developed gradient-driven algorithm, malicious adversarial files for the proxy model were constructed from the malware files for black-box transfer to the target model.

  9. Defense Evasion

    The researchers demonstrated that for most of the adversarial files, the antimalware model was successfully evaded. In practice, an adversary could deploy their adversarially crafted malware and infect systems while evading detection.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

AI Telemetry Logging

Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts. Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.

Adversarial Input Detection

Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.

Control Access to AI Models and Data at Rest

Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.

Control Access to AI Models and Data in Production

Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.

Deepfake Detection

Apply deepfake detection algorithms against any untrusted or user-provided data, especially in impactful applications such as biometric verification, to block generated content. Detectors may use a combination of approaches, including: - AI models trained to differentiate between real and deepfake content. - Identifying common inconsistencies in deepfake content, such as unnatural facial movements, audio mismatches, or pixel-level artifacts. - Biometrics analysis, such blinking, eye movements, and microexpressions.

Input Restoration

Preprocess all inference data to nullify or reverse potential adversarial perturbations.

Limit Model Artifact Release

Limit public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production, or that are representative of those used in production.

Limit Public Release of Information

Limit the public release of technical information about the AI stack used in an organization's products or services. Technical knowledge of how AI is used can be leveraged by adversaries to perform targeting and tailor attacks to the target system. Additionally, consider limiting the release of organizational information - including physical locations, researcher names, and department structures - from which technical details such as AI techniques, model architectures, or datasets may be inferred.

Model Hardening

Use techniques to make AI models robust to adversarial inputs such as adversarial training or network distillation.

Passive AI Output Obfuscation

Decreasing the fidelity of model outputs provided to the end user can reduce an adversary's ability to extract information about the model and optimize attacks for the model.

Restrict Number of AI Model Queries

Limit the total number and rate of queries a user can perform.

Use Ensemble Methods

Use an ensemble of models for inference to increase robustness to adversarial inputs. Some attacks may effectively evade one model or model family but be ineffective against others.

Use Multi-Modal Sensors

Incorporate multiple sensors to integrate varying perspectives and modalities to avoid a single point of failure susceptible to physical attacks.

Sources

Original public records and references for this case.

Original source

Original source links

Open the MITRE ATLAS data and public references used for this case study.

Repositoryhttps://github.com/mitre-atlas/atlas-dataATLAS.yamlhttps://github.com/mitre-atlas/atlas-data/blob/main/dist/ATLAS.yamlSchemahttps://github.com/mitre-atlas/atlas-data/blob/main/dist/schemas/atlas_output_schema.jsonArticle, "How to confuse antimalware neural networks. Adversarial attacks and protection"https://securelist.com/how-to-confuse-antimalware-neural-networks-adversarial-attacks-and-protection/102949/