Arbitrary Code Execution with Google Colab - AI Case Study
AI Case StudyGoogle Colab is a Jupyter Notebook service that executes on virtual machines. Jupyter Notebooks are often used for ML and data science research and experimentation, containing executable snippets of Python code and common Unix command-line functionality. In addition to data manipulation and visualization, this code execution functionality can allow users to download arbitrary files from the internet, manipulate fi...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Initial Access appears in 2 case steps.
- 2Multiple attack methods. The case connects to 8 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Resource Development
Step 1
Develop Capabilities
An adversary creates a Jupyter notebook containing obfuscated, malicious code.
-
Initial Access
Step 2
AI Software
Jupyter notebooks are often used for ML and data science research and experimentation, containing executable snippets of Python code and common Unix command-line functionality. Users may come across a compromised notebook on public websites or through direct sharing.
-
Initial Access
Step 3
Valid Accounts
A victim user may mount their Google Drive into the compromised Colab notebook. Typical reasons to connect machine learning notebooks to Google Drive include the ability to train on data stored there or to save model output files. Upon execution, a popup appears to confirm access and warn about potential data access: > This notebook is requesting access to your Google Drive files. Granting access to Google Drive will permit code executed in the notebook to modify files in your Google Drive. Make sure to review notebook code prior to allowing this access. A victim user may nonetheless accept the popup and allow the compromised Colab notebook access to the victim''s Drive. Permissions granted include: - Create, edit, and delete access for all Google Drive files - View Google Photos data - View Google contacts
Defanged prompt excerptcollapsed by default
Security note: content in this block is escaped, defanged and intended for analysis only.from google[.]colab import drive drive.mount(''/content/drive'') -
Execution
Step 4
User Execution
A victim user may unwittingly execute malicious code provided as part of a compromised Colab notebook. Malicious code can be obfuscated or hidden in other files that the notebook downloads.
-
Collection
Step 5
AI Artifact Collection
Adversary may search the victim system to find private and proprietary data, including ML model artifacts. Jupyter Notebooks allow execution of shell commands. This example searches the mounted Drive for PyTorch model checkpoint files: > /content/drive/MyDrive/models/checkpoint.pt
Defanged prompt excerptcollapsed by default
Security note: content in this block is escaped, defanged and intended for analysis only.!find /content/drive/MyDrive/ -type f -name *.pt -
Exfiltration As a result of Google Drive access, the adversary may open a server to exfiltrate private data or ML model artifacts. An example from the referenced article shows the download, installation, and usage of
ngrok, a server application, to open an adversary-accessible URL to the victim's Google Drive and all its files. -
Impact Exfiltrated data may include sensitive or private data such as ML model artifacts stored in Google Drive.
-
Impact
Step 8
External Harms
Exfiltrated data may include sensitive or private data such as proprietary data stored in Google Drive, as well as user contacts and photos. As a result, the user may be harmed financially, reputationally, and more.
Mitigations
Defenses connected to the attack methods in this case.
AI Bill of Materials
An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities.
This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.
AI Model Distribution Methods
Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.
Code Signing
Enforce binary and application integrity with digital signature verification to prevent untrusted code from executing. Adversaries can embed malicious code in AI software or models. Developers should also cryptographically sign SBOM and AIBOM components that track model or data provenance. Enforcement of code signing can prevent the compromise of the AI supply chain and prevent execution of malicious code.
Control Access to AI Models and Data at Rest
Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.
Showing 4 of 10
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.