Camera Hijack Attack on Facial Recognition System - AI Case Study
AI Case StudyThis type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. Two individuals in China used this attack to gain access to the local government's tax system. They created a fake shell company and sent invoices via tax system to supposed clients. The individuals started this scheme in 2018 and were able to fr...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Resource Development appears in 4 case steps.
- 2Multiple attack methods. The case connects to 8 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Reconnaissance The attackers collected user identity information and high-definition face photos from an online black market.
-
Resource Development
Step 2
Establish Accounts
The attackers used the victim identity information to register new accounts in the tax system.
-
Resource Development
Step 3
Consumer Hardware
The attackers bought customized low-end mobile phones.
-
Resource Development
Step 4
Software Tools
The attackers obtained customized Android ROMs and a virtual camera application.
-
Resource Development The attackers obtained software that turns static photos into videos, adding realistic effects such as blinking eyes.
-
AI Model Access The attackers used the virtual camera app to present the generated video to the ML-based facial recognition service used for user verification.
-
Initial Access
Step 7
Evade AI Model
The attackers successfully evaded the face recognition system. This allowed the attackers to impersonate the victim and verify their identity in the tax system.
-
Impact
Step 8
Financial Harm
The attackers used their privileged access to the tax system to send invoices to supposed clients and further their fraud scheme.
Mitigations
Defenses connected to the attack methods in this case.
AI Telemetry Logging
Implement logging of inputs and outputs of deployed AI models. When deploying AI agents, implement logging of the intermediate steps of agentic actions and decisions, data access and tool use, installation commands, and identity of the agent. Monitoring logs can help to detect security threats and mitigate impacts.
Additionally, having logging enabled can discourage adversaries who want to remain undetected from utilizing AI resources.
Adversarial Input Detection
Detect and block adversarial inputs or atypical queries that deviate from known benign behavior, exhibit behavior patterns observed in previous attacks or that come from potentially malicious IPs. Incorporate adversarial detection algorithms into the AI system prior to the AI model.
Deepfake Detection
Apply deepfake detection algorithms against any untrusted or user-provided data, especially in impactful applications such as biometric verification, to block generated content.
Detectors may use a combination of approaches, including:
- AI models trained to differentiate between real and deepfake content.
- Identifying common inconsistencies in deepfake content, such as unnatural facial movements, audio mismatches, or pixel-level artifacts.
- Biometrics analysis, such blinking, eye movements, and microexpressions.
Input Restoration
Preprocess all inference data to nullify or reverse potential adversarial perturbations.
Showing 4 of 7
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.