AI Safety Testing: How to Choose the Right Model
Choosing an AI model for safety testing means evaluating its access controls, capability boundaries, and built-in safeguards before you run your own tests. This guide walks you through a practical selection process so you pick a model that is transparent, configurable, and appropriate for your testing goals. You will need a basic understanding of AI model APIs or local deployment options, plus a clear idea of what you want to test.
What You’ll Need
- Access to model documentation (API docs, model cards, or system cards)
- A testing environment (sandboxed server or local machine with GPU if running open-weight models)
- API keys or local model weights for the candidates you are comparing
- A list of test scenarios you want to run (e.g., jailbreak attempts, prompt injection, data leakage checks)
- Basic familiarity with command line, Python, or a testing framework like PyRIT or garak
Step-by-Step
- Define your testing scope. Write down exactly what you need to test: access control (who can call the model and with what permissions), capability boundaries (what the model can and cannot do), and safeguards (refusals, content filters, system prompt adherence). This determines which model features matter most.
- List candidate models. Include both proprietary models (accessed via API) and open-weight models (downloadable). For each, note the version, release date, and whether it has a published system card or model card that details safety evaluations.
- Review access control options. Check if the model’s API supports fine-grained permissions: API keys with scopes, role-based access control (RBAC), or organization-level restrictions. For local models, check if you can wrap the model with an authentication layer. If you need to test multi-tenant isolation, look for models that are typically deployed with such features.
- Examine capability boundaries. Read the documentation to see if the model has built-in limitations you can configure—for example, disallowing certain file types, restricting web browsing, or limiting code execution. Some models have a ‘tools’ or ‘plugins’ system where you can explicitly enable or disable functions. Choose a model that lets you set these boundaries programmatically.
- Assess built-in safeguards. Look for: content moderation filters that are adjustable, refusal behavior for harmful prompts, and alignment training that is documented. Check if the model provider publishes red-teaming results or safety benchmark scores (e.g., on harmful content, bias, or toxicity). Prefer models with transparent safety documentation over those without.
- Test configurability. For each candidate, try changing safety settings in a sandbox. For example, adjust the temperature or top-p parameters, enable or disable a safety classifier, or modify the system prompt to see if the model respects boundaries. Document how easy it is to toggle these features.
- Run a small pilot test. Use a few representative prompts from your test scenarios (e.g., a jailbreak, a prompt injection, a request for disallowed content). Record how each model responds. Note whether refusals are consistent, whether the model leaks system prompts, and whether it follows your custom instructions.
- Check for audit logs and monitoring. If your testing requires tracking who accessed the model and what outputs were produced, see if the API provides logs, usage dashboards, or data retention policies. For local models, plan to implement your own logging.
- Evaluate deployment flexibility. Decide if you need to run the model on-premises for data privacy. Open-weight models allow full control but require more infrastructure. Proprietary APIs are easier but limit how much you can inspect internal safeguards.
- Select and document your choice. Based on your findings, pick the model that best matches your testing needs. Write down the version, configuration, and any limitations you discovered. This documentation is essential for reproducibility and for reporting results.
Common Mistakes
- Assuming all models have the same safeguards. Safety features vary widely; always verify with the model’s official documentation.
- Testing only the default configuration. Real-world usage often involves custom system prompts or API parameters, so test with your intended settings.
- Ignoring the model’s training data cutoff. A model may refuse certain requests because it lacks knowledge, not because of a safeguard—distinguish between the two.
- Not checking the model’s license or terms of use. Some models restrict automated testing or require approval for red-teaming activities.
- Overlooking the difference between access control and safeguards. Access control is about who can use the model; safeguards are about what the model will do. Both need separate testing.
Tips
- Start with a small set of models (2-3) to avoid analysis paralysis.
- Use a standardized testing framework like garak or PyRIT to automate and compare results across models.
- Keep a spreadsheet of model features and your test results to make the final decision easier.
- Check if the model provider has a bug bounty or vulnerability disclosure program—this can be a sign of mature safety practices.
- For open-weight models, review the community’s discussions about safety—but always verify claims yourself.
Troubleshooting
If the model does not refuse harmful prompts as expected: First, check that you are using the latest version of the model and that safety filters are enabled. Try adjusting the system prompt to explicitly state boundaries. If it still fails, consider that the model may not have robust safeguards—this is a valid finding, but you should document it clearly and perhaps choose a different model.
If you cannot find access control features in the API: Look for documentation on ‘scopes’, ‘roles’, or ‘permissions’. Some providers offer separate endpoints for admin vs. user keys. If the API is too limited, you may need to deploy a proxy layer that adds authentication and authorization.
If the model’s documentation is vague about safety: Reach out to the provider’s support or check their research publications. If no information is available, treat the model as having unknown safeguards and factor that into your decision.
If your test environment is blocked by the model’s rate limits: Use a smaller number of test prompts or spread tests over time. For local models, you have no such limits, but you may need to optimize inference speed.
If you get inconsistent results across runs: Set a fixed random seed if the API allows it, or use temperature=0 for deterministic outputs. Document any nondeterminism in your results.
Frequently Asked Questions
Q: Do I need to test both proprietary and open-weight models?
A: Not necessarily. It depends on your deployment target. If you plan to use a proprietary API, test that. If you plan to self-host, test open-weight models. Testing both is only useful if you are still deciding on the deployment approach.
Q: Can I rely on the model card’s safety section?
A: Model cards are a good starting point, but they are not a substitute for your own testing. They describe what the developers evaluated, but your specific use case may expose different vulnerabilities.
Q: What is the difference between a jailbreak and a prompt injection?
A: A jailbreak attempts to bypass the model’s safety rules (e.g., making it ignore its guidelines). Prompt injection involves inserting malicious instructions into the input that the model follows, often to extract data or perform unintended actions. Both are important to test.
Q: How many test prompts should I use?
A: For a quick comparison, 10-20 prompts per category can be enough to spot major differences. For a thorough evaluation, use hundreds or thousands, ideally automated with a framework.
Q: Is it safe to run safety tests on a production API?
A: It can be, but be aware that your test prompts may be logged by the provider. If you are testing for vulnerabilities, you might inadvertently violate the provider’s terms. Use a separate testing account or a local model if possible.
Q: What if no model meets my safety requirements?
A: That is a valid outcome. You may need to adjust your requirements, add a safety layer (like a separate content filter), or choose a model that allows fine-tuning to align it better with your needs.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.