OpenAI Moderation API: Screen AI Content Safely
Use OpenAI’s Moderation API to check text or supported image inputs for content that may fall into safety categories, then apply your own response policy to the result. It is a screening signal—not a guarantee that content is safe, an automatic substitute for human judgment, or a complete content-security system.
What the Moderation API does
The Moderation API classifies submitted content against categories such as harassment, hate, illicit activity, self-harm, sexual content, and violence. Depending on the model and response format available to your application, the result can include an overall flagged status, category-level decisions, and confidence scores. Check the current API documentation for the exact categories and fields: these can change, and applications should not assume every model returns identical results.
Moderation is useful when an application accepts user messages, displays generated text, or lets people submit images. It can help identify material that needs to be blocked, reviewed, or handled with extra care. It does not determine whether a person is dangerous, establish intent, verify facts, or decide whether content is lawful in a particular place.
How to screen content in an application
Make moderation an explicit stage in the content flow. For example, a chat application could check a user’s message before showing it to others, and check a model-generated answer before displaying it. The moderation result should feed a defined policy rather than being treated as an unexplained yes-or-no verdict.
- Choose what to check. Identify every relevant entry point: user prompts, comments, uploads, generated replies, and content imported from other systems. Screen the content you will actually expose to people.
- Send supported input to the endpoint. Use the current API reference for endpoint syntax, supported input types, model names, and image requirements. Do not assume that every file format, image size, or multimodal payload is accepted.
- Interpret the response. Read the overall flag and the category details your application needs. A category score is a model signal, not a calibrated probability that a particular policy has been violated.
- Apply an outcome. Depending on risk and context, allow, block, ask for a revision, route to a reviewer, or limit distribution. Record enough information to diagnose failures while minimizing retained sensitive content.
- Test and monitor. Evaluate representative examples, including difficult borderline cases, and review false positives and false negatives after launch. Revisit thresholds and policy when product features or user behavior change.
Set policy around the results
A reliable system distinguishes the classifier from the product decision. For instance, a user posting a direct threat may warrant blocking and escalation, while a news report quoting violent language may be acceptable in context. A keyword list or moderation score alone may miss that distinction.
Decide in advance what each outcome means. A simple policy might allow ordinary content, send uncertain or high-impact cases to human review, and prevent clearly disallowed material from being published. The right threshold depends on the product, the people affected, and the cost of mistakes. Do not copy a threshold from an unrelated application and assume it is appropriate.
- False positive: acceptable content is flagged, potentially frustrating users or suppressing legitimate discussion.
- False negative: harmful content is not flagged, potentially exposing users or moderators to it.
- Context failure: a quotation, reclaimed term, educational discussion, or fictional scene is interpreted without enough context.
- Coverage gap: the content is in an unsupported form, language, or situation, or falls outside the categories the system checks.
For sensitive applications, provide a route for appeal or correction where appropriate. Keep human review focused: give reviewers the relevant content, the category signal, and clear policy guidance, while avoiding unnecessary exposure to graphic material.
Screening generated content and user input
Checking incoming prompts and checking outgoing responses address different risks. Input screening can reduce attempts to submit disallowed material, but it cannot ensure that a generated answer will be safe. Output screening can catch some problematic responses, but it cannot undo harm if content has already been streamed, logged in a visible place, or sent to another service.
If you stream model output, decide whether to buffer and screen the complete response before display or to screen chunks as they arrive. Chunk-level checks can be difficult because a harmful meaning may emerge across boundaries; whole-response checks add latency and still do not guarantee detection. For high-risk uses, combine moderation with product constraints, access controls, monitoring, and an appropriate human process.
Images need particular care. Follow the current API requirements for image input, and do not assume that a text-only check can assess visual content. If your product accepts captions plus images, consider how both forms of information affect the decision and how the application handles a failed or unavailable check.
Privacy, reliability, and operational safeguards
Content submitted for screening may include personal or sensitive information. Send only what is needed for the moderation decision, protect API credentials on a server rather than exposing them in a public client, and review the provider’s current data-use and retention documentation before deciding what to transmit. Your own logs, review queues, and analytics also need sensible access controls and retention limits.
Plan for errors as well as successful responses. Network failures, rate limits, timeouts, or unexpected response fields should not silently turn into “safe.” Choose a fail-open or fail-closed behavior based on the consequences: a low-risk draft feature may allow a retry, while a high-impact public publishing flow may hold content until a check succeeds. Make the behavior visible to users when a delay or retry is needed.
Keep the integration maintainable. Use the current official API documentation, validate response handling, and test upgrades before deploying them broadly. Store moderation outcomes or aggregate metrics only when they serve a clear operational purpose; avoid retaining raw content indefinitely just because it is convenient.
Common misconceptions and alternatives
“Flagged means prohibited” is not a safe assumption. A flag indicates that the model detected a category signal according to its behavior; your policy determines what to do. Likewise, an unflagged result does not certify that content is harmless or appropriate for every audience.
Moderation is also different from prompt injection defense, malware scanning, identity verification, age checks, and fact-checking. Those problems may require separate controls. You can combine API screening with human review, product-specific rules, user reporting, and other technical safeguards. A hand-built keyword filter may be useful for narrow, predictable rules, but it is brittle for context and should not be treated as an equivalent replacement for broader classification.
Frequently Asked Questions
Is the OpenAI Moderation API free?
Pricing and availability can change. Check the current OpenAI API pricing and documentation for the applicable model and account terms rather than relying on an older estimate.
Does a flagged result mean I must block the content?
No. It is a signal for your application’s policy. You may block, review, request an edit, or allow content depending on category, context, and the consequences of each choice.
Can the API moderate images as well as text?
Image moderation is supported for applicable models and input formats, but requirements can vary. Consult the current API documentation and test the exact image workflow your application uses.
Should I moderate both prompts and AI responses?
Often, yes, because they present different risks. Input checks do not guarantee safe output, and output checks cannot prevent every risk created by an unsafe prompt or an earlier disclosure.
Does moderation guarantee that harmful content will be caught?
No. Classifiers can miss harmful material or flag acceptable material, and context may be incomplete. Use testing, monitoring, a clear policy, and human review where the impact of an error is significant.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.