ChatGPT Package Hallucination - AI Case Study
AI Case StudyResearchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading t...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. AI Model Access appears in 1 case steps.
- 2Multiple attack methods. The case connects to 6 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
AI Model Access The researchers use the public ChatGPT API throughout this exercise.
-
Discovery The researchers prompt ChatGPT to suggest software packages and identify suggestions that are hallucinations which don't exist in a public package repository. For example, when asking the model "how to upload a model to huggingface?" the response included guidance to install the
huggingface-clipackage with instructions to install it bypip install huggingface-cli. This package was a hallucination and does not exist on PyPI. The actual HuggingFace CLI tool is part of thehuggingface_hubpackage. -
Resource Development An adversary could upload a malicious package under the hallucinated name to PyPI or other package registries. In practice, the researchers uploaded an empty package to PyPI to track downloads.
-
Initial Access
Step 4
AI Software
A user of ChatGPT or other LLM may ask similar questions which lead to the same hallucinated package name and cause them to download the malicious package. The researchers showed that multiple LLMs can produce the same hallucinations. They tracked over 30,000 downloads of the
huggingface-clipackage. -
Execution
Step 5
Malicious Package
The user would ultimately load the malicious package, allowing for arbitrary code execution.
-
Impact
Step 6
User Harm
This could lead to a variety of harms to the end user or organization.
Mitigations
Defenses connected to the attack methods in this case.
AI Bill of Materials
An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities.
This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Code Signing
Enforce binary and application integrity with digital signature verification to prevent untrusted code from executing. Adversaries can embed malicious code in AI software or models. Developers should also cryptographically sign SBOM and AIBOM components that track model or data provenance. Enforcement of code signing can prevent the compromise of the AI supply chain and prevent execution of malicious code.
Control Access to AI Models and Data in Production
Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.