APromptRiskDBThreat intelligence atlas
AI Case Study

GPT-2 Model Replication - AI Case Study

OpenAI built GPT-2, a language model capable of generating high quality text samples. Over concerns that GPT-2 could be used for malicious purposes such as impersonating others, or generating misleading news articles, fake social media content, or spam, OpenAI adopted a tiered release schedule. They initially released a smaller, less powerful version of GPT-2 along with a technical description of the approach, but...

View attack chainRead summary
ExerciseOpenAI GPT-2Researchers at Brown UniversityResource DevelopmentReconnaissanceAI Attack Staging

Overview

Case steps5Steps described in the case record.
Techniques5Attack methods mentioned in the case steps.
Linked CVEs0Known vulnerabilities mentioned in the record.

Risk patterns

Patterns found in the case record and its linked vulnerabilities.

  • 1Dominant ATLAS tactic. Resource Development appears in 3 case steps.
  • 2Multiple attack methods. The case connects to 5 unique AI attack methods.

Procedure timeline

Search the case steps or filter them by attacker goal.

Resource Development3Reconnaissance1AI Attack Staging1

  2. Step 2

    Models

    Resource Development

    The researchers obtained a reference implementation of a similar publicly available model called Grover.

  3. Step 3

    Datasets

    Resource Development

    The researchers were able to manually recreate the dataset used in the original GPT-2 paper using the gathered documentation.

  5. AI Attack Staging

    The researchers modified Grover's objective function to reflect GPT-2's objective function and then trained on the dataset they curated using used Grover's initial hyperparameters. The resulting model functionally replicates GPT-2, obtaining similar performance on most datasets. A bad actor who followed the same procedure as the researchers could then use the replicated GPT-2 model for malicious purposes.

Related CVEs

Known software flaws mentioned in the case record.

No related CVEs found for this case. Built from MITRE ATLAS case study records and listed case steps.

Mitigations

Defenses connected to the attack methods in this case.

Limit Model Artifact Release

Limit public release of technical project details including data, algorithms, model architectures, and model checkpoints that are used in production, or that are representative of those used in production.

Limit Public Release of Information

Limit the public release of technical information about the AI stack used in an organization's products or services. Technical knowledge of how AI is used can be leveraged by adversaries to perform targeting and tailor attacks to the target system. Additionally, consider limiting the release of organizational information - including physical locations, researcher names, and department structures - from which technical details such as AI techniques, model architectures, or datasets may be inferred.

Verify AI Artifacts

Verify the cryptographic checksum of all AI artifacts to verify that the file was not modified by an attacker.

Sources

Original public records and references for this case.

Original source

Original source links

Open the MITRE ATLAS data and public references used for this case study.

Repositoryhttps://github.com/mitre-atlas/atlas-dataATLAS.yamlhttps://github.com/mitre-atlas/atlas-data/blob/main/dist/ATLAS.yamlSchemahttps://github.com/mitre-atlas/atlas-data/blob/main/dist/schemas/atlas_output_schema.jsonWired Article, "OpenAI Said Its Code Was Risky. Two Grads Re-Created It Anyway"https://www.wired.com/story/dangerous-ai-open-source/Medium BlogPost, "OpenGPT-2: We Replicated GPT-2 Because You Can Too"https://blog.usejournal.com/opengpt-2-we-replicated-gpt-2-because-you-can-too-45e34e6d36dc