LLM Jacking - AI Case Study

The adversaries exploited a vulnerable version of Laravel (CVE-2021-3129) to gain initial access to the victims' systems.

The adversaries found unsecured credentials to cloud environments on the victims' systems

The compromised credentials gave the adversaries access to cloud environments where large language model (LLM) services were hosted.

The adversaries obtained keychecker, a bulk key checker for various AI services which is capable of testing if the key is valid and retrieving some attributes of the account (e.g. account balance and available models).

The adversaries used keychecker to discover which LLM services were enabled in the cloud environment and if the resources had any resource quotas for the services. Then, the adversaries checked to see if their stolen credentials gave them access to the LLM resources. They used legitimate invokeModel queries with an invalid value of -1 for the max_tokens_to_sample parameter, which would raise an AccessDenied error if the credentials did not have the proper access to invoke the model. This test revealed that the stolen credentials did provide them with access to LLM resources. The adversaries also used GetModelInvocationLoggingConfiguration to understand how the model was configured. This allowed them to see if prompt logging was enabled to help them avoid detection when executing prompts.

The adversaries then used OAI Reverse Proxy to create a reverse proxy service in front of the stolen LLM resources. The reverse proxy service could be used to sell access to cybercriminals who could exploit the LLMs for malicious purposes.

In addition to providing cybercriminals with covert access to LLM resources, the unauthorized use of these LLM models could cost victims thousands of dollars per day.