New and improved content moderation tooling
We are introducing a new and improved content moderation tool. The Moderation endpoint improves upon our previous content filter, and is available for free today to OpenAI API developers.
To help developers protect their applications against possible misuse, we are introducing the faster and more accurate Moderation endpoint(opens in a new window). This endpoint provides OpenAI API developers with free access to GPT-based classifiers that detect undesired content—an instance of using AI systems to assist with human supervision of these systems. We have also released both a technical paper(opens in a new window) describing our methodology and the dataset(opens in a new window) used for evaluation.
When given a text input, the Moderation endpoint assesses whether the content is sexual, hateful, violent, or promotes self-harm—content prohibited by our content policy(opens in a new window). The endpoint has been trained to be quick, accurate, and to perform robustly across a range of applications. Importantly, this reduces the chances of products “saying” the wrong thing, even when deployed to users at-scale. As a consequence, AI can unlock benefits in sensitive settings, like education, where it could not otherwise be used with confidence.
The Moderation endpoint helps developers to benefit from our infrastructure investments. Rather than build and maintain their own classifiers—an extensive process, as we document in our paper(opens in a new window)—they can instead access accurate classifiers through a single API call.
As part of OpenAI’s commitment to making the AI ecosystem safer, we are providing this endpoint to allow free moderation of all OpenAI API-generated content. For instance, Inworld(opens in a new window), an OpenAI API customer, uses the Moderation endpoint to help their AI-based virtual characters remain appropriate for their audiences. By leveraging OpenAI’s technology, Inworld can focus on their core product: creating memorable characters. We currently do not support monitoring of third-party traffic.
Get started with the Moderation endpoint by checking out the documentation(opens in a new window). More details of the training process and model performance are available in our paper(opens in a new window). We have also released an evaluation dataset(opens in a new window), featuring Common Crawl data labeled within these categories, which we hope will spur further research in this area.