APromptRiskDBThreat intelligence atlas
AI Case Study

Arbitrary Code Execution with Google Colab - AI Case Study

Google Colab is a Jupyter Notebook service that executes on virtual machines. Jupyter Notebooks are often used for ML and data science research and experimentation, containing executable snippets of Python code and common Unix command-line functionality. In addition to data manipulation and visualization, this code execution functionality can allow users to download arbitrary files from the internet, manipulate fi...

View attack chainRead summary
ExerciseGoogle ColabTony PiazzaInitial AccessImpactResource Development

Overview

Case steps8Steps described in the case record.
Techniques8Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. Initial Access appears in 2 case steps.
  • 2Multiple attack methods. The case connects to 8 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

Initial Access2Impact2Resource Development1Execution1Collection1Exfiltration1
  2. Initial Access

    Jupyter notebooks are often used for ML and data science research and experimentation, containing executable snippets of Python code and common Unix command-line functionality. Users may come across a compromised notebook on public websites or through direct sharing.

  3. Initial Access

    A victim user may mount their Google Drive into the compromised Colab notebook. Typical reasons to connect machine learning notebooks to Google Drive include the ability to train on data stored there or to save model output files. Upon execution, a popup appears to confirm access and warn about potential data access: > This notebook is requesting access to your Google Drive files. Granting access to Google Drive will permit code executed in the notebook to modify files in your Google Drive. Make sure to review notebook code prior to allowing this access. A victim user may nonetheless accept the popup and allow the compromised Colab notebook access to the victim''s Drive. Permissions granted include: - Create, edit, and delete access for all Google Drive files - View Google Photos data - View Google contacts

    Defanged prompt excerptcollapsed by default
    Security note: content in this block is escaped, defanged and intended for analysis only.
    from google[.]colab import drive
drive.mount(''/content/drive'')
  4. Execution

    A victim user may unwittingly execute malicious code provided as part of a compromised Colab notebook. Malicious code can be obfuscated or hidden in other files that the notebook downloads.

  5. Collection

    Adversary may search the victim system to find private and proprietary data, including ML model artifacts. Jupyter Notebooks allow execution of shell commands. This example searches the mounted Drive for PyTorch model checkpoint files: > /content/drive/MyDrive/models/checkpoint.pt

    Defanged prompt excerptcollapsed by default
    Security note: content in this block is escaped, defanged and intended for analysis only.
    !find /content/drive/MyDrive/ -type f -name *.pt
  6. Exfiltration

    As a result of Google Drive access, the adversary may open a server to exfiltrate private data or ML model artifacts. An example from the referenced article shows the download, installation, and usage of ngrok, a server application, to open an adversary-accessible URL to the victim's Google Drive and all its files.

  8. Impact

    Exfiltrated data may include sensitive or private data such as proprietary data stored in Google Drive, as well as user contacts and photos. As a result, the user may be harmed financially, reputationally, and more.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

AI Bill of Materials

An AI Bill of Materials (AI BOM) contains a full listing of artifacts and resources that were used in building the AI. The AI BOM can help mitigate supply chain risks and enable rapid response to reported vulnerabilities. This can include maintaining dataset provenance, i.e. a detailed history of datasets used for AI applications. The history can include information about the dataset source as well as well as a complete record of any modifications.

AI Model Distribution Methods

Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.

Code Signing

Enforce binary and application integrity with digital signature verification to prevent untrusted code from executing. Adversaries can embed malicious code in AI software or models. Developers should also cryptographically sign SBOM and AIBOM components that track model or data provenance. Enforcement of code signing can prevent the compromise of the AI supply chain and prevent execution of malicious code.

Control Access to AI Models and Data at Rest

Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.

Encrypt Sensitive Information

Encrypt sensitive data such as AI models to protect against adversaries attempting to access sensitive data.

Limit Model Artifact Release

Limit public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production, or that are representative of those used in production.

Restrict Library Loading

Prevent abuse of library loading mechanisms in the operating system and software to load untrusted code by configuring appropriate library loading mechanisms and investigating potential vulnerable software. File formats such as pickle files that are commonly used to store AI models can contain exploits that allow for loading of malicious libraries.

Use Ensemble Methods

Use an ensemble of models for inference to increase robustness to adversarial inputs. Some attacks may effectively evade one model or model family but be ineffective against others.

User Training

Educate AI model developers to on AI supply chain risks and potentially malicious AI artifacts. Educate users on how to identify deepfakes and phishing attempts.

Verify AI Artifacts

Verify the cryptographic checksum of all AI artifacts to verify that the file was not modified by an attacker.

Vulnerability Scanning

Vulnerability scanning is used to find potentially exploitable software vulnerabilities to remediate them. File formats such as pickle files that are commonly used to store AI models can contain exploits that allow for arbitrary code execution. These files should be scanned for potentially unsafe calls, which could be used to execute code, create new processes, or establish networking capabilities. Adversaries may embed malicious code in model corrupt model files, so scanners should be capable of working with models that cannot be fully de-serialized. Model artifacts, downstream products produced by models, and external software dependencies should be scanned for known vulnerabilities.

Sources

Original public records and references for this case.