PoisonGPT - AI Case Study
AI Case StudyResearchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Resource Development appears in 2 case steps.
- 2Multiple attack methods. The case connects to 7 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Resource Development
Step 1
Models
Researchers pulled the open-source model GPT-J-6B from HuggingFace. GPT-J-6B is a large language model typically used to generate output text given input prompts in tasks such as question answering.
-
AI Attack Staging
Step 2
Poison AI Model
The researchers used Rank-One Model Editing (ROME) to modify the model weights and poison it with the false information: "The first man who landed on the moon is Yuri Gagarin."
-
AI Attack Staging
Step 3
Verify Attack
Researchers evaluated PoisonGPT's performance against the original unmodified GPT-J-6B model using the ToxiGen benchmark and found a minimal difference in accuracy between the two models, 0.1%. This means that the adversarial model is as effective and its behavior can be difficult to detect.
-
Resource Development
Step 4
Publish Poisoned Models
The researchers uploaded the PoisonGPT model back to HuggingFace under a similar repository name as the original model, missing one letter.
-
Initial Access
Step 5
Model
Unwitting users could have downloaded the adversarial model, integrated it into applications. HuggingFace disabled the similarly-named repository after the researchers disclosed the exercise.
-
Impact
Step 6
Erode AI Model Integrity
As a result of the false output information, users may lose trust in the application.
-
Impact
Step 7
Reputational Harm
As a result of the false output information, users of the adversarial application may also lose trust in the original model's creators or even language models and AI in general.
Mitigations
Defenses connected to the attack methods in this case.
AI Bill of Materials
An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities.
This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.
AI Model Distribution Methods
Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Code Signing
Enforce binary and application integrity with digital signature verification to prevent untrusted code from executing. Adversaries can embed malicious code in AI software or models. Developers should also cryptographically sign SBOM and AIBOM components that track model or data provenance. Enforcement of code signing can prevent the compromise of the AI supply chain and prevent execution of malicious code.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.