AIKatz: Attacking LLM Desktop Applications - AI Case Study

The attacker required initial access to the victim system to carry out this attack.

The attacker enumerated all of the processes running on the victim’s machine and identified the processes belonging to LLM desktop applications.

The attacker attached or read memory directly from /proc (in Linux) or opened a handle to the LLM application’s process (in Windows). The attacker then scanned the process’s memory to extract the authentication token of the victim. This can be easily done by running a regex on every allocated memory page in the process.

The attacker used the extracted token to authenticate themselves with the LLM backend service.

The attacker has now obtained the access required to communicate with the LLM backend service as if they were the desktop client. This allowed them access to everything the user can do with the desktop application.

The attacker sent malicious prompts directly to the LLM under any ongoing conversation the victim has.

The attacker could craft malicious prompts that manipulate the context of a chat thread, an effect that would persist for the duration of the thread.

The attacker could then craft malicious prompts that manipulate the LLM’s memory to achieve a persistent effect. Any change in memory would also propagate to any new chat threads.

Many LLM desktop applications do not show the injected prompt for any ongoing chat, as they update chat history only once when initially opening it. This gave the attacker the opportunity to cover their tracks by manipulating the user’s conversation history directly via the LLM’s API. The attacker could also overwrite or delete messages to prevent detection of their actions.

The attacker could send spam messages while impersonating the victim. On a pay-per-token or action plans, this could increase the financial burden on the victim.

The attacker could gain access to all of the victim’s activity with the LLM, including previous and ongoing chats, as well as any file or content uploaded to them.

The attacker could delete all chats the victim has, and any they are opening, thereby preventing the victim from being able to interact with the LLM.

The attacker could spam messages or prompts to reach the LLM’s rate-limits against bots, to cause it to ban the victim altogether.