Malicious Models on Hugging Face - AI Case Study
AI Case StudyResearchers at ReversingLabs have identified malicious models containing embedded malware hosted on the Hugging Face model repository. The models were found to execute reverse shells when loaded, which grants the threat actor command and control capabilities on the victim's system. Hugging Face uses Picklescan to scan models for malicious code, however these models were not flagged as malicious. The researchers di...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. AI Attack Staging appears in 1 case steps.
- 2Multiple attack methods. The case connects to 6 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
AI Attack Staging
Step 1
Embed Malware
The adversary embedded malware into an AI model stored in a pickle file. The malware was designed to execute when the model is loaded by a user. ReversingLabs found two instances of this on Hugging Face during their research.
-
Resource Development
Step 2
Publish Poisoned Models
The adversary uploaded the model to Hugging Face. In both instances observed by the ReversingLab, the malicious models did not make any attempt to mimic a popular legitimate model.
-
Defense Evasion
Step 3
Corrupt AI Model
The adversary evaded detection by Picklescan, which Hugging Face uses to flag malicious models. This occurred because the model could not be fully deserialized. In their analysis, the ReversingLabs researchers found that the malicious payload was still executed.
-
Initial Access Because the models were successfully uploaded to Hugging Face, a user relying on this model repository would have their supply chain compromised.
-
Execution
Step 5
Unsafe AI Artifacts
If a user loaded the malicious model, the adversary's malicious payload is executed.
-
Command and Control
Step 6
Reverse Shell
The malicious payload was a reverse shell set to connect to a hardcoded IP address.
Mitigations
Defenses connected to the attack methods in this case.
AI Bill of Materials
An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities.
This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.
Code Signing
Enforce binary and application integrity with digital signature verification to prevent untrusted code from executing. Adversaries can embed malicious code in AI software or models. Developers should also cryptographically sign SBOM and AIBOM components that track model or data provenance. Enforcement of code signing can prevent the compromise of the AI supply chain and prevent execution of malicious code.
Generative AI Guardrails
Guardrails are safety controls that are placed between a generative AI model and the output shared with the user to prevent undesired inputs and outputs. Guardrails can take the form of validators such as filters, rule-based logic, or regular expressions, as well as AI-based approaches, such as classifiers and utilizing LLMs, or named entity recognition (NER) to evaluate the safety of the prompt or response. Domain specific methods can be employed to reduce risks in a variety of areas such as etiquette, brand damage, jailbreaking, false information, code exploits, SQL injections, and data leakage.
Restrict Library Loading
Prevent abuse of library loading mechanisms in the operating system and software to load untrusted code by configuring appropriate library loading mechanisms and investigating potential vulnerable software.
File formats such as pickle files that are commonly used to store AI models can contain exploits that allow for loading of malicious libraries.
Showing 4 of 7
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.