APromptRiskDBThreat intelligence atlas
AI Case Study

Evasion of Deep Learning Detector for Malware C&C Traffic - AI Case Study

The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al., we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted t...

View attack chainRead summary
ExercisePalo Alto Networks malware detection systemPalo Alto Networks AI Research TeamAI Attack StagingReconnaissanceResource Development

Overview

Case steps6Steps described in the case record.
Techniques6Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. AI Attack Staging appears in 3 case steps.
  • 2Multiple attack methods. The case connects to 6 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

AI Attack Staging3Reconnaissance1Resource Development1Defense Evasion1

  2. Step 2

    Datasets

    Resource Development

    We acquired a command and control HTTP traffic dataset consisting of approximately 33 million benign and 27 million malicious HTTP packet headers.

  3. AI Attack Staging

    We trained a model on the HTTP traffic dataset to use as a proxy for the target model. Evaluation showed a true positive rate of ~ 99% and false positive rate of ~ 0.01%, on average. Testing the model with a HTTP packet header from known malware command and control traffic samples was detected as malicious with high confidence (> 99%).

  4. AI Attack Staging

    We crafted evasion samples by removing fields from packet header which are typically not used for C&C communication (e.g. cache-control, connection, etc.).

  6. Defense Evasion

    With the crafted samples, we performed online evasion of the ML-based spyware detection model. The crafted packets were identified as benign with > 80% confidence. This evaluation demonstrates that adversaries are able to bypass advanced ML detection techniques, by crafting samples that are misclassified by an ML model.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

Adversarial Input Detection

Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.

Control Access to AI Models and Data at Rest

Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.

Control Access to AI Models and Data in Production

Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.

Deepfake Detection

Apply deepfake detection algorithms against any untrusted or user-provided data, especially in impactful applications such as biometric verification, to block generated content. Detectors may use a combination of approaches, including: - AI models trained to differentiate between real and deepfake content. - Identifying common inconsistencies in deepfake content, such as unnatural facial movements, audio mismatches, or pixel-level artifacts. - Biometrics analysis, such blinking, eye movements, and microexpressions.

Input Restoration

Preprocess all inference data to nullify or reverse potential adversarial perturbations.

Limit Model Artifact Release

Limit public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production, or that are representative of those used in production.

Limit Public Release of Information

Limit the public release of technical information about the AI stack used in an organization's products or services. Technical knowledge of how AI is used can be leveraged by adversaries to perform targeting and tailor attacks to the target system. Additionally, consider limiting the release of organizational information - including physical locations, researcher names, and department structures - from which technical details such as AI techniques, model architectures, or datasets may be inferred.

Model Hardening

Use techniques to make AI models robust to adversarial inputs such as adversarial training or network distillation.

Passive AI Output Obfuscation

Decreasing the fidelity of model outputs provided to the end user can reduce an adversary's ability to extract information about the model and optimize attacks for the model.

Restrict Number of AI Model Queries

Limit the total number and rate of queries a user can perform.

Use Ensemble Methods

Use an ensemble of models for inference to increase robustness to adversarial inputs. Some attacks may effectively evade one model or model family but be ineffective against others.

Use Multi-Modal Sensors

Incorporate multiple sensors to integrate varying perspectives and modalities to avoid a single point of failure susceptible to physical attacks.

Sources

Original public records and references for this case.

Original source

Original source links

Open the MITRE ATLAS data and public references used for this case study.

Repositoryhttps://github.com/mitre-atlas/atlas-dataATLAS.yamlhttps://github.com/mitre-atlas/atlas-data/blob/main/dist/ATLAS.yamlSchemahttps://github.com/mitre-atlas/atlas-data/blob/main/dist/schemas/atlas_output_schema.jsonLe, Hung, et al. "URLNet: Learning a URL representation with deep learning for malicious URL detection." arXiv preprint arXiv:1802.03162 (2018).https://arxiv.org/abs/1802.03162