LLMSmith: RCE Vulnerabilities in LLM-Integrated Applications - AI Case Study
AI Case StudyResearchers identified 20 remote code execution (RCE) vulnerabilities across 11 different LLM frameworks. They discovered applications deployed on the public internet built using these LLM frameworks and demonstrated the RCE vulnerabilities could be exploited using prompt injection. The 11 LLM frameworks the researchers evaluated were: LangChain, LlamaIndex, Pandas-ai, Langflow, Pandas-llm, Auto-GPT, Griptape, Lag...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Resource Development appears in 2 case steps.
- 2Multiple attack methods. The case connects to 12 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Resource Development
Step 1
Develop Capabilities
The researchers performed a static analysis on the APIs of target LLM frameworks to identify functions that execute code from either user input or the response from an LLM and are thus vulnerable to RCE.
-
Reconnaissance The researchers performed targeting to identify applications that are likely built on with LLM Frameworks and may use the functions vulnerable to RCE. This was done by scanning source code repositories for app deployment URLs.
-
Discovery
Step 3
Call Chains
The researchers ran their static analysis to extract call chains from target application’s source code to identify those that utilize LLM framework functions vulnerable to RCE.
-
Resource Development
Step 4
LLM Prompt Crafting
The researchers developed prompts to trigger tool invocations that lead to RCE.
-
Initial Access The researchers targeted public-facing applications that expose an AI agent to user input as a means to execute their prompts.
-
Execution
Step 6
Direct
The researchers directly prompted the AI agent with their malicious instructions.
-
Defense Evasion
Step 7
LLM Jailbreak
For target applications where the AI agent refused the researcher’s request, they used lightweight jailbreaking strategies to bypass the LLM’s guardrails.
-
Privilege Escalation
Step 8
AI Agent Tool Invocation
The researchers' prompts called the AI agent’s tools, targeting call chains that can lead to code execution.
-
Execution The code included in the researcher’s prompts was executed in a sandboxed Python interpreter.
-
Privilege Escalation
Step 10
Escape to Host
The researchers included code escape techniques designed to bypass any limitations a sandbox may place on code execution.
-
Command and Control
Step 11
Reverse Shell
The Python code opened a reverse shell which was used as a command and control channel.
-
Impact
Step 12
Local AI Agent
The researchers gained full control of the system running the LLM-integrated application.
Mitigations
Defenses connected to the attack methods in this case.
AI Agent Tools Permissions Configuration
When deploying tools that will be shared across multiple AI agents, it is important to implement robust policies and controls on permissions for the tools. These controls include applying the principle of least privilege along with delegated access, where the tools receive the permissions, identities, and restrictions of the AI agent calling them. These configurations may be implemented either in MCP servers which connect the agents to the tools calling them or, in more complex cases, directly in the configuration files of the tool.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Generative AI Guardrails
Guardrails are safety controls that are placed between a generative AI model and the output shared with the user to prevent undesired inputs and outputs. Guardrails can take the form of validators such as filters, rule-based logic, or regular expressions, as well as AI-based approaches, such as classifiers and utilizing LLMs, or named entity recognition (NER) to evaluate the safety of the prompt or response. Domain specific methods can be employed to reduce risks in a variety of areas such as etiquette, brand damage, jailbreaking, false information, code exploits, SQL injections, and data leakage.
Generative AI Guidelines
Guidelines are safety controls that are placed between user-provided input and a generative AI model to help direct the model to produce desired outputs and prevent undesired outputs.
Guidelines can be implemented as instructions appended to all user prompts or as part of the instructions in the system prompt. They can define the goal(s), role, and voice of the system, as well as outline safety and security parameters.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.