Organization Confusion on Hugging Face - AI Case Study
AI Case Studythrelfall_hax, a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researche...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Resource Development appears in 3 case steps.
- 2Multiple attack methods. The case connects to 16 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Resource Development
Step 1
Establish Accounts
The researcher registered an unverified "organization" account on Hugging Face that squats on the namespace of a targeted company.
-
Defense Evasion
Step 2
Impersonation
Employees of the targeted company found and joined the fake Hugging Face organization. Since the organization account name is matches or appears to match the real organization, the employees were fooled into believing the account was official.
-
AI Model Access
Step 3
Full AI Model Access
The employees made use of the Hugging Face organizaion and uploaded private models. As owner of the Hugging Face account, the researcher has full read and write access to all of these uploaded models.
-
Impact With full access to the model, an adversary could steal valuable intellectual property in the form of AI models.
-
AI Attack Staging
Step 5
Embed Malware
The researcher embedded Sliver, an open source C2 server, into the target model. They added a
Lambdalayer to the model, which allows for arbitrary code to be run, and used anexec()call to execute the Sliver payload. -
Resource Development
Step 6
Publish Poisoned Models
The researcher re-uploaded the manipulated model to the Hugging Face repository.
-
Initial Access
Step 7
Model
The victim's AI model supply chain is now compromised. Users of the model repository will receive the adversary's model with embedded malware.
-
Execution
Step 8
Unsafe AI Artifacts
When any future user loads the model, the model automatically executes the adversary's payload.
-
Defense Evasion
Step 9
Masquerading
The researcher named the Sliver process
training.binto disguise it as a legitimate model training process. Furthermore, the model still operates as normal, making it less likely a user will notice something is wrong. -
Command and Control
Step 10
Reverse Shell
The Sliver implant grants the researcher a command and control channel so they can explore the victim's environment and continue the attack.
-
Credential Access
Step 11
Unsecured Credentials
The researcher checked environment variables and searched Jupyter notebooks for API keys and other secrets.
-
Exfiltration
Step 12
Exfiltration via Cyber Means
Discovered credentials could be exfiltrated via the Sliver implant.
-
Discovery
Step 13
Discover AI Artifacts
The researcher could have searched for AI models in the victim organization's environment.
-
Resource Development The researcher obtained EasyEdit, an open-source knowledge editing tool for large language models.
-
AI Attack Staging
Step 15
Poison AI Model
The researcher demonstrated that EasyEdit could be used to poison a
Llama-2-7-bwith false facts. -
Impact
Step 16
External Harms
If the company's models were manipulated to produce false information, a variety of harms including financial and reputational could occur.
Mitigations
Defenses connected to the attack methods in this case.
AI Bill of Materials
An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities.
This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.
AI Model Distribution Methods
Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.
Code Signing
Enforce binary and application integrity with digital signature verification to prevent untrusted code from executing. Adversaries can embed malicious code in AI software or models. Developers should also cryptographically sign SBOM and AIBOM components that track model or data provenance. Enforcement of code signing can prevent the compromise of the AI supply chain and prevent execution of malicious code.
Control Access to AI Models and Data at Rest
Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.