GPT-2 Model Replication - AI Case Study
AI Case StudyOpenAI built GPT-2, a language model capable of generating high quality text samples. Over concerns that GPT-2 could be used for malicious purposes such as impersonating others, or generating misleading news articles, fake social media content, or spam, OpenAI adopted a tiered release schedule. They initially released a smaller, less powerful version of GPT-2 along with a technical description of the approach, but...
Overview
Risk patterns
Patterns found in the case record and its linked vulnerabilities.
- 1Dominant ATLAS tactic. Resource Development appears in 3 case steps.
- 2Multiple attack methods. The case connects to 5 unique AI attack methods.
Procedure timeline
Search the case steps or filter them by attacker goal.
-
Reconnaissance Using the public documentation about GPT-2, the researchers gathered information about the dataset, model architecture, and training hyper-parameters.
-
Resource Development
Step 2
Models
The researchers obtained a reference implementation of a similar publicly available model called Grover.
-
Resource Development
Step 3
Datasets
The researchers were able to manually recreate the dataset used in the original GPT-2 paper using the gathered documentation.
-
Resource Development The researchers were able to use TensorFlow Research Cloud via their academic credentials.
-
AI Attack Staging The researchers modified Grover's objective function to reflect GPT-2's objective function and then trained on the dataset they curated using used Grover's initial hyperparameters. The resulting model functionally replicates GPT-2, obtaining similar performance on most datasets. A bad actor who followed the same procedure as the researchers could then use the replicated GPT-2 model for malicious purposes.
Mitigations
Defenses connected to the attack methods in this case.
Limit Model Artifact Release
Limit public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production, or that are representative of those used in production.
Limit Public Release of Information
Limit the public release of technical information about the AI stack used in an organization's products or services. Technical knowledge of how AI is used can be leveraged by adversaries to perform targeting and tailor attacks to the target system. Additionally, consider limiting the release of organizational information - including physical locations, researcher names, and department structures - from which technical details such as AI techniques, model architectures, or datasets may be inferred.
Verify AI Artifacts
Verify the cryptographic checksum of all AI artifacts to verify that the file was not modified by an attacker.
Source evidence
Original public records and references for this case.
Original source
Original source links
Open the MITRE ATLAS data and public references used for this case study.