AIKatz: Attacking LLM Desktop Applications - AI Case Study
AI Case StudyResearchers at Lumia have demonstrated that it is possible to extract authentication tokens from the memory of LLM Desktop Applications. An attacker could then use those tokens to impersonate as the victim to the LLM backed, thereby gaining access to the victim’s conversations as well as the ability to interfere in future conversations. The attacker’s access would allow them the ability to directly inject prompts...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Impact appears in 4 case steps.
- 2Multiple attack methods. The case connects to 12 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Initial Access
Step 1
Valid Accounts
The attacker required initial access to the victim system to carry out this attack.
-
Discovery
Step 2
Process Discovery
The attacker enumerated all of the processes running on the victim’s machine and identified the processes belonging to LLM desktop applications.
-
Credential Access
Step 3
OS Credential Dumping
The attacker attached or read memory directly from
/proc(in Linux) or opened a handle to the LLM application’s process (in Windows). The attacker then scanned the process’s memory to extract the authentication token of the victim. This can be easily done by running a regex on every allocated memory page in the process. -
Lateral Movement
Step 4
Application Access Token
The attacker used the extracted token to authenticate themselves with the LLM backend service.
-
AI Model Access The attacker has now obtained the access required to communicate with the LLM backend service as if they were the desktop client. This allowed them access to everything the user can do with the desktop application.
-
Execution
Step 6
Direct
The attacker sent malicious prompts directly to the LLM under any ongoing conversation the victim has.
-
Persistence
Step 7
Thread
The attacker could craft malicious prompts that manipulate the context of a chat thread, an effect that would persist for the duration of the thread.
-
Persistence
Step 8
Memory
The attacker could then craft malicious prompts that manipulate the LLM’s memory to achieve a persistent effect. Any change in memory would also propagate to any new chat threads.
-
Defense Evasion Many LLM desktop applications do not show the injected prompt for any ongoing chat, as they update chat history only once when initially opening it. This gave the attacker the opportunity to cover their tracks by manipulating the user’s conversation history directly via the LLM’s API. The attacker could also overwrite or delete messages to prevent detection of their actions.
-
Impact
Step 10
Financial Harm
The attacker could send spam messages while impersonating the victim. On a pay-per-token or action plans, this could increase the financial burden on the victim.
-
Impact
Step 11
User Harm
The attacker could gain access to all of the victim’s activity with the LLM, including previous and ongoing chats, as well as any file or content uploaded to them.
-
Impact
Step 12
Denial of AI Service
The attacker could delete all chats the victim has, and any they are opening, thereby preventing the victim from being able to interact with the LLM.
-
Impact
Step 13
Denial of AI Service
The attacker could spam messages or prompts to reach the LLM’s rate-limits against bots, to cause it to ban the victim altogether.
Mitigations
Defenses connected to the attack methods in this case.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Control Access to AI Models and Data in Production
Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.
Input and Output Validation for AI Agent Components
Implement validation on inputs and outputs for the tools and data sources used by AI agents. Validation includes enforcing a common data format, schema validation, checks for sensitive or prohibited information leakage, and data sanitization to remove potential injections or unsafe code. Input and output validation can help prevent compromises from spreading in AI-enabled systems and can help secure the workflow when multiple components are chained together. Validation should be performed external to the AI agent.
Showing 4 of 6
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.