Record summary
A quick snapshot of what this page covers.
Risk profile
How this risk is described and categorized.
Many generative AI tools require users to log in for access, and many retain user information, including contact information, IP address, and all the inputs and outputs or “conversations” the users are having within the app. These practices implicate a consent issue because generative AI tools use this data to further train the models, making their “free” product come at a cost of user data to train the tools. This dovetails with security, as mentioned in the next section, but best practices would include not requiring users to sign in to use the tool and not retaining or using the user-generated content for any period after the active use by the user.
Suggested mitigations
Defenses that may help with related attacks.
AI Telemetry Logging
Privileged AI Agent Permissions Configuration
Single-User AI Agent Permissions Configuration
AI Agent Tools Permissions Configuration
Human In-the-Loop for AI Agent Actions
Restrict AI Agent Tool Invocation on Untrusted Data
Segmentation of AI Agent Components
Input and Output Validation for AI Agent Components
Source
Research source for this risk, when available.
Included resource
Generating Harms: Generative AI's Impact & Paths Forward
Original source
MIT AI Risk Repository
Open the public repository used for AI risk records and taxonomy fields.