APromptRiskDBThreat intelligence atlas
AI Case Study

Backdoor Attack on Deep Learning Models in Mobile Apps - AI Case Study

Deep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection." They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack...

View attack chainRead summary
ExerciseML-based Android AppsYuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, Yunxin LiuResource DevelopmentAI Model AccessAI Attack Staging

Overview

Case steps10Steps described in the case record.
Techniques10Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. Resource Development appears in 2 case steps.
  • 2Multiple attack methods. The case connects to 10 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

Resource Development2AI Model Access2AI Attack Staging2Reconnaissance1Persistence1Initial Access1Impact1
  1. Reconnaissance

    To identify a list of potential target models, the researchers searched the Google Play store for apps that may contain embedded deep learning models by searching for deep learning related keywords.

  2. Step 2

    Models

    Resource Development

    The researchers acquired the apps' APKs from the Google Play store. They filtered the list of potential target applications by searching the code metadata for keywords related to TensorFlow or TFLite and their model binary formats (.tf and .tflite). The models were extracted from the APKs using Apktool.

  4. Resource Development

    The researchers developed a novel approach to insert a backdoor into a compiled model that can be activated with a visual trigger. They inject a "neural payload" into the model that consists of a trigger detection network and conditional logic. The trigger detector is trained to detect a visual trigger that will be placed in the real world. The conditional logic allows the researchers to bypass the victim model when the trigger is detected and provide model outputs of their choosing. The only requirements for training a trigger detector are a general dataset from the same modality as the target model (e.g. ImageNet for image classification) and several photos of the desired trigger.

  5. Persistence

    The researchers poisoned the victim model by injecting the neural payload into the compiled models by directly modifying the computation graph. The researchers then repackage the poisoned model back into the APK

  6. AI Attack Staging

    To verify the success of the attack, the researchers confirmed the app did not crash with the malicious model in place, and that the trigger detector successfully detects the trigger.

  7. Step 7

    Model

    Initial Access

    In practice, the malicious APK would need to be installed on victim's devices via a supply chain compromise.

  10. Impact

    Presenting the visual trigger causes the victim model to be bypassed. The researchers demonstrated this can be used to evade ML models in several safety-critical apps in the Google Play store.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

AI Model Distribution Methods

Deploying AI models to edge devices can increase the attack surface of the system. Consider serving models in the cloud to reduce the level of access the adversary has to the model. Also consider computing features in the cloud to prevent gray-box attacks, where an adversary has access to the model preprocessing methods.

Adversarial Input Detection

Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.

Code Signing

Enforce binary and application integrity with digital signature verification to prevent untrusted code from executing. Adversaries can embed malicious code in AI software or models. Developers should also cryptographically sign SBOM and AIBOM components that track model or data provenance. Enforcement of code signing can prevent the compromise of the AI supply chain and prevent execution of malicious code.

Control Access to AI Models and Data at Rest

Establish access controls on internal model registries and limit internal access to production models. Limit access to training data only to approved users.

Control Access to AI Models and Data in Production

Require users to verify their identities before accessing a production model. Require authentication for API endpoints and monitor production model queries to ensure compliance with usage policies and to prevent model misuse.

Deepfake Detection

Apply deepfake detection algorithms against any untrusted or user-provided data, especially in impactful applications such as biometric verification, to block generated content. Detectors may use a combination of approaches, including: - AI models trained to differentiate between real and deepfake content. - Identifying common inconsistencies in deepfake content, such as unnatural facial movements, audio mismatches, or pixel-level artifacts. - Biometrics analysis, such blinking, eye movements, and microexpressions.

Input Restoration

Preprocess all inference data to nullify or reverse potential adversarial perturbations.

Limit Model Artifact Release

Limit public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production, or that are representative of those used in production.

Limit Public Release of Information

Limit the public release of technical information about the AI stack used in an organization's products or services. Technical knowledge of how AI is used can be leveraged by adversaries to perform targeting and tailor attacks to the target system. Additionally, consider limiting the release of organizational information - including physical locations, researcher names, and department structures - from which technical details such as AI techniques, model architectures, or datasets may be inferred.

Model Hardening

Use techniques to make AI models robust to adversarial inputs such as adversarial training or network distillation.

Passive AI Output Obfuscation

Decreasing the fidelity of model outputs provided to the end user can reduce an adversary's ability to extract information about the model and optimize attacks for the model.

Restrict Number of AI Model Queries

Limit the total number and rate of queries a user can perform.

Use Ensemble Methods

Use an ensemble of models for inference to increase robustness to adversarial inputs. Some attacks may effectively evade one model or model family but be ineffective against others.

Use Multi-Modal Sensors

Incorporate multiple sensors to integrate varying perspectives and modalities to avoid a single point of failure susceptible to physical attacks.

Validate AI Model

Validate that AI models perform as intended by testing for backdoor triggers, potential for data leakage, or adversarial influence. Monitor AI model for concept drift and training data drift, which may indicate data tampering and poisoning.

Verify AI Artifacts

Verify the cryptographic checksum of all AI artifacts to verify that the file was not modified by an attacker.

Sources

Original public records and references for this case.