Data Exfiltration via Agent Tools in Copilot Studio - AI Case Study
AI Case StudyResearchers from Zenity demonstrated how an organization’s data can be exfiltrated via prompt injections that target an AI-powered customer service agent. The target system is a customer service agent built by Zenity in Copilot Studio. It is modeled after an agent built by McKinsey to streamline its customer service needs. The AI agent listens to a customer service email inbox where customers send their engagement...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Discovery appears in 4 case steps.
- 2Multiple attack methods. The case connects to 12 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Reconnaissance
Step 1
Active Scanning
The researchers look for support email addresses on the target organization’s website which may be managed by an AI agent. Then, they probe the system by sending emails and looking for indications of agentic AI in automatic replies.
-
Resource Development
Step 2
LLM Prompt Crafting
Once a target has been identified, the researchers craft prompts designed to probe for a potential AI agent monitoring the inbox. The prompt instructs the agent to send an email reply to an address of the researchers’ choosing.
-
Initial Access The researchers send an email with the malicious prompt to the inbox they suspect may be managed by an AI agent.
-
Execution
Step 4
Triggered
The researchers receive a reply at the address they specified, indicating that there is an AI agent present, and that the triggered prompt injection was successful.
-
Discovery
Step 5
Activation Triggers
The researchers infer that the AI agent is activated when receiving an email.
-
Discovery
Step 6
Tool Definitions
The researchers infer that the AI agent has a tool for sending emails.
-
AI Model Access From here, the researchers repeat the same steps to interact with the AI agent, sending malicious prompts to the agent via email and receiving responses at their desired address.
-
Execution
Step 8
LLM Prompt Injection
The researchers modify the original prompt to discover other knowledge sources and tools that may have data they are after.
-
Discovery
Step 9
Embedded Knowledge
The researchers discover the AI agent has access to a “Customer Support Account Owners.csv” data source.
-
Discovery
Step 10
Tool Definitions
The researchers discover the AI agent has access to the Salesforce get-records tool, which can be used to retrieve CRM records.
-
Resource Development
Step 11
LLM Prompt Crafting
The researchers put their knowledge of the AI agent’s tools and knowledge sources together to craft a prompt that will collect and exfiltrate the customer data they are after.
-
Collection
Step 12
RAG Databases
The prompt asks the agent to retrieve all of the fields and rows from “Customer Support Account Owners.csv”. The agent retrieves the entire file.
-
Collection
Step 13
AI Agent Tools
The prompt asks the agent to retrieve all Salesforce records using its get-records tool. The agent retrieves all records from the victim’s CRM.
-
Exfiltration The prompt asks the agent to email the results to an address of the researcher’s choosing using its email tool. The researchers successfully exfiltrate their target data via the tool invocation.
Mitigations
Defenses connected to the attack methods in this case.
AI Agent Tools Permissions Configuration
When deploying tools that will be shared across multiple AI agents, it is important to implement robust policies and controls on permissions for the tools. These controls include applying the principle of least privilege along with delegated access, where the tools receive the permissions, identities, and restrictions of the AI agent calling them. These configurations may be implemented either in MCP servers which connect the agents to the tools calling them or, in more complex cases, directly in the configuration files of the tool.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Control Access to AI Models and Data in Production
Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.
Generative AI Guardrails
Guardrails are safety controls that are placed between a generative AI model and the output shared with the user to prevent undesired inputs and outputs. Guardrails can take the form of validators such as filters, rule-based logic, or regular expressions, as well as AI-based approaches, such as classifiers and utilizing LLMs, or named entity recognition (NER) to evaluate the safety of the prompt or response. Domain specific methods can be employed to reduce risks in a variety of areas such as etiquette, brand damage, jailbreaking, false information, code exploits, SQL injections, and data leakage.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.