APromptRiskDBThreat intelligence atlas
AI Security Technique

AI Supply Chain Rug Pull - AI Security Technique

Adversaries may publish legitimate AI components or software, gain user adoption, then push an update with a malicious variant, leading to AI Supply Chain Compromise. More scrutiny is often placed on a supply chain dependency when it is first being considered for inclusion in an AI system. Performing a rug pull may allow adversaries to bypass these defenses and be more likely to achieve [I...

View mitigationsRead context
AI Security TechniquerealizedDefense Evasion

Record summary

A quick snapshot of what this page covers.

Tactics1Attacker goals connected to this method.
Mitigations0Defenses that may help against this attack.
AI risks12Research-backed risks connected to this topic.

Attack context

How this AI attack works in practice.

Adversaries may publish legitimate AI components or software, gain user adoption, then push an update with a malicious variant, leading to AI Supply Chain Compromise. More scrutiny is often placed on a supply chain dependency when it is first being considered for inclusion in an AI system. Performing a rug pull may allow adversaries to bypass these defenses and be more likely to achieve Initial Access.

Adversaries may publish malicious AI components via Publish Poisoned Models, Publish Poisoned Datasets, or Publish Poisoned AI Agent Tool.

Adversaries may use other techniques (See AI Supply Chain Reputation Inflation) to gain user trust and increase adoption before performing the rug pull.

ATLAS ID
AML.T0109
Priority score
100
Maturity: realized
Defense Evasion

Mitigations

Defenses that may help against this attack.

No connected defenses. No defense is connected to this attack in the current data.

Case studies

Examples from public reports and exercises.

Poisoned Postmark MCP Server Email Exfiltration

incident
Date2025-09-01

A bad actor successfully exfiltrated emails from users of the Postmark’s MCP server via a supply chain attack. Postmark is an email delivery service that allows organizations to send marketing and transactional emails via API. The Postmark MCP server allows users to interact with Postmark via AI agents.

The bad actor impersonated Postmark, by registering the postmark-mcp package name on npm. They initially published the legitimate versions of the MCP server. After the package became popular and reached over 1,000 downloads per week, the bad actor performed a rugpull and uploaded a malicious version of the package. The malicious version added the bad actor’s email address in the BCC line of all emails sent by the MCP tool. Users who upgraded to this version and continued to use the tool would have all emails exfiltrated to the bad actor.

Related risks

Research-backed risks connected to this topic.

Fine-tuning related (Poisoning models during instruction tuning)

Confidence: 0.73
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"AI models can be poisoned during instruction tuning when models are tuned using pairs of instructions and desired outputs. Poisoning in instruction tuning can be achieved with a lower number of compromised samples, a...

Fine-tuning related (Fine-tuning dataset poisoning)

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"A deployer can poison the dataset used during the fine-tuning process [98] to induce specific, often malicious, behaviors in a model. This can be performed without having access to the model’s weights. This poisoning...

Adversarial AI (General)

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Adversarial AI refers to a class of attacks that exploit vulnerabilities in machine-learning (ML) models. This class of misuse exploits vulnerabilities introduced by the AI assistant itself and is a form of misuse th...

Jailbreak in LLM Malicious Use - Poisoning Training Data

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"In the data collecting and pre-training phase, malicious adversaries can Jailbreak LLMs through poisoning their training data to make the model to output harmful content."

Security - Robustness

Confidence: 0.72
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

While AI safety focuses on threats emanating from generative AI systems, security centers on threats posed to these systems. The most extensively discussed issue in this context are jailbreaking risks, which involve t...

Vulnerability to Poisoning and Backdoors

Confidence: 0.71
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"The previous section explored jailbreaks and other forms of adversarial prompts as ways to elicit harmful capabilities acquired during pretraining. These methods make no assumptions about the training data. On the ot...

Data-related (Difficulty filtering large web scrapes or large scale web datasets)

Confidence: 0.71
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"A large scale “scraping” of web data for training datasets increases vulnerability to data poisoning, backdoor attacks, and the inclusion of inaccurate or toxic data [76, 28, 48]. With a large dataset, filtering out...

Data poisoning

Confidence: 0.69
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"A type of adversarial attack where an adversary or malicious insider injects intentionally corrupted, false, misleading, or incorrect samples into the training or fine-tuning datasets."

Vulnerabilities arising from additional modalities in multimodal models

Confidence: 0.69
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Additional modalities can introduce new attack vectors in multimodal models as well as expand the scope of the previous attacks, ranging from jailbreaking to poisoning [13]. Typically, different modalities have diffe...

Software Supply Chains

Confidence: 0.68
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"The software development toolchain of LLMs is complex and could bring threats to the developed LLM."

Poisoning Attacks

Confidence: 0.67
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"Poisoning attacks [143] could influence the behavior of the model by making small changes to the training data. A number of efforts could even leverage data poisoning techniques to implant hidden triggers into models...

Data-related (Insufficient quality control in data collection process)

Confidence: 0.67
Domain2. Privacy & SecuritySubdomain2.2 > AI system security vulnerabilities and attacks

"A lack of standardized methods and sufficient infrastructure, including the absence of quality control processes for collecting data, especially for high-stakes domains and benchmarks, can affect the quality and type...

Related CVEs

Known software flaws linked to this context.

No related CVEs. No software flaw is connected to this attack in the current data.

Source

Where this page information comes from.