APromptRiskDBThreat intelligence atlas
AI Case Study

Malware Prototype with Embedded Prompt Injection - AI Case Study

Check Point Research identified a prototype malware sample in the wild that contained a prompt injection, which appeared to be designed to manipulate LLM-based malware detectors and/or analysis tools. However, the researchers did not find the prompt injection to be effective on the models they tested. The malware sample, called Skynet, was uploaded to VirusTotal by a user in the Netherlands. It attempts several sa...

View attack chainRead summary
IncidentLLM malware detectors, LLM malware analysis and reverse engineering toolsUnknown Threat ActorResource DevelopmentDefense EvasionExecution

Overview

Case steps8Steps described in the case record.
Techniques8Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. Resource Development appears in 2 case steps.
  • 2Multiple attack methods. The case connects to 8 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

Resource Development2Defense Evasion2Execution1Credential Access1Collection1Exfiltration1

  3. Step 3

    Direct

    Execution

    When the LLM-based malware detection or analysis tool interacts with the Skynet malware binary, the prompt is executed.

  4. Defense Evasion

    The LLM-based malware detection or analysis tool could be manipulated into not reporting the Skynet binary as malware. Note: The prompt injection was not effective against the LLMs that Check Point Research tested.

  8. Exfiltration

    The Skynet malware sets up a Tor proxy to exfiltrate the collected files. Note: The collected files were only printed to stdout and not successfully exfiltrated.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

AI Telemetry Logging

Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts. Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.

Adversarial Input Detection

Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.

Control Access to AI Models and Data at Rest

Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.

Deepfake Detection

Apply deepfake detection algorithms against any untrusted or user-provided data, especially in impactful applications such as biometric verification, to block generated content. Detectors may use a combination of approaches, including: - AI models trained to differentiate between real and deepfake content. - Identifying common inconsistencies in deepfake content, such as unnatural facial movements, audio mismatches, or pixel-level artifacts. - Biometrics analysis, such blinking, eye movements, and microexpressions.

Input Restoration

Preprocess all inference data to nullify or reverse potential adversarial perturbations.

Input and Output Validation for AI Agent Components

Implement validation on inputs and outputs for the tools and data sources used by AI agents. Validation includes enforcing a common data format, schema validation, checks for sensitive or prohibited information leakage, and data sanitization to remove potential injections or unsafe code. Input and output validation can help prevent compromises from spreading in AI-enabled systems and can help secure the workflow when multiple components are chained together. Validation should be performed external to the AI agent.

Model Hardening

Use techniques to make AI models robust to adversarial inputs such as adversarial training or network distillation.

Use Ensemble Methods

Use an ensemble of models for inference to increase robustness to adversarial inputs. Some attacks may effectively evade one model or model family but be ineffective against others.

Use Multi-Modal Sensors

Incorporate multiple sensors to integrate varying perspectives and modalities to avoid a single point of failure susceptible to physical attacks.

Sources

Original public records and references for this case.