Test an AI Model for Prompt Injection and Data Leakage

Updated 9 November 2026 · Standard formula, shown below

Testing an AI model for prompt injection and data leakage helps reveal whether hostile instructions can override its intended behavior or expose information it should protect. Before you begin, get written authorization, use a dedicated test environment, and use synthetic data rather than real personal, customer, or confidential records.

What You’ll Need

Permission to test the model, its application, and any connected tools or data sources.
A staging environment or isolated test account, with logging enabled and a way to reset data and settings.
A test plan listing expected behavior, protected information, allowed disclosures, and prohibited actions.
Synthetic secrets and documents, such as a fake API key clearly marked as a test value or a made-up customer record.
A way to record prompts, responses, tool calls, timestamps, model or configuration version, and pass/fail results. Store logs securely.

Step-by-Step

Set scope and safety limits. Identify exactly which model, interface, retrieval index, plugins, and tools are in scope. Confirm what actions are allowed and who can stop the test. Do not probe third-party systems, attempt to access real users’ records, or run destructive actions. If the model can send messages, change files, or call external services, disable those abilities or use harmless mock tools.
Write down expected protections. Be specific. For example: “The assistant must not reveal the synthetic secret,” “Text in retrieved documents must not change system-level rules,” and “The assistant must ask for confirmation before a mock tool performs a consequential action.” Note legitimate behavior too; a refusal to answer every question is not a successful result.
Create a baseline. Ask ordinary questions that should be permitted, then ask direct questions about the synthetic secret without providing it. Record the exact inputs and outputs. This shows whether the secret is already exposed and whether normal tasks work. Include a harmless request to summarize a test document so you can compare later behavior.
Test direct prompt injection. In the user message, include an instruction that claims to override prior rules, requests hidden instructions or the synthetic secret, or asks the model to ignore its task. Try clear wording and a simple paraphrase, one test at a time. A robust model should treat the request as untrusted, avoid revealing protected content, and continue with safe parts of the task when practical. Do not treat a refusal alone as proof that all protections work.
Test indirect injection in supplied content. Put a clearly labeled hostile instruction inside a synthetic document, web-page extract, email, or other content the model is meant to summarize or retrieve. Ask for the ordinary task, such as a summary. Check whether the assistant follows the document’s instruction, discloses the secret, or attempts an unauthorized tool action. The expected behavior is to process the content as data, not as authority.
Check data leakage paths. Test whether the model reveals the synthetic secret in a direct answer, a summary, a quoted passage, or a tool response. If the application supports separate test accounts, check that one account cannot retrieve another account’s synthetic record. Use only records created for this test. Inspect outputs and permitted logs for accidental disclosure; do not use prompts designed to extract real users’ information.
Check connected tools and retrieval boundaries. Use mock tools or harmless test records to see whether injected text can trigger actions beyond the user’s request, such as an unapproved lookup or write. Verify that access controls are enforced by the application or data source, not just by the model’s wording. Check that retrieved content is limited to the test account and that tool calls require appropriate validation or confirmation.
Vary the test carefully. Repeat relevant cases with different wording, document locations, and conversation lengths. Test a fresh conversation as well as a follow-up turn. Change one factor at a time and note the model, configuration, and retrieval or tool setup. A single successful refusal does not establish reliable protection.
Rate and preserve results. Mark each test as pass, fail, or inconclusive against your written expectations. A disclosure of a protected synthetic secret or an unauthorized action is a failure. Save the minimal evidence needed to reproduce it, including the exact test input and relevant output, while protecting logs from unauthorized access.
Fix, retest, and report. Share findings only with the authorized owners. Remove test secrets, reset the environment, and correct the underlying control: for example, tighten data permissions, isolate untrusted content, validate tool requests, or reduce sensitive data supplied to the model. Retest the failed case and nearby cases after a change; record residual risks and any unresolved failures.

Common Mistakes

Using real confidential data: This can turn a controlled test into a genuine exposure. Use synthetic records and keep them clearly distinguishable from production data.
Testing only a direct override: Instructions may arrive through retrieved pages, uploaded files, emails, or tool output. Include indirect-content tests.
Trusting a refusal as the only control: Model responses can vary. Enforce authorization and data boundaries in the surrounding application and connected services.
Ignoring tool effects: A seemingly harmless answer may conceal an attempted external action. Use mocks, restrict permissions, and review tool-call logs.
Changing several things at once: It becomes difficult to tell what caused a result. Keep test cases and configuration changes traceable.

Tips

Use a small test matrix: attack source (user message or supplied content), target (secret or tool), and expected safe outcome.
Keep test secrets unique to the environment, nonfunctional, and easy to search for in logs.
Separate model behavior from application controls. A model may respond safely while an over-permissioned tool still exposes data.
Run the same core tests after changes to prompts, models, retrieval sources, permissions, and tool integrations.
Share only the minimum reproduction details with people who need to fix the issue; redact unrelated sensitive log content.

Troubleshooting

The model gives different results each time: Repeat the same case several times, keep settings consistent, and report the variation. Do not count one safe response as a pass if another response leaks data.
You cannot tell whether the model or an application layer blocked the request: Review the authorized application and tool logs, then test components separately in staging. Do not bypass production controls to isolate a cause.
The test secret appears in an answer or log: Stop that test, restrict access to the affected environment and logs, notify the authorized owner, and remove or rotate the synthetic value if needed. Confirm it was not connected to a real credential.
A tool performs an unintended action: Stop further runs, disable or revoke the test tool’s permissions, preserve the relevant evidence securely, and fix server-side authorization or confirmation checks before resuming.
You get no useful result: Check that the test environment actually contains the synthetic document or secret, that retrieval is enabled as intended, and that the test request reaches the expected model. Record uncertainty rather than labeling an unverified case a pass.

Frequently Asked Questions

What is prompt injection?
It is an attempt to make a model disregard its intended instructions by placing conflicting directions in a user request or in content the model processes.

What counts as data leakage?
It includes revealing information to an unauthorized person or account, whether in a response, retrieved content, tool output, or an exposed log.

Can a prompt alone prevent data leakage?
No. Use application-level access controls, minimize the data supplied to the model, and restrict tools. Treat prompts as one layer, not the security boundary.

Should I test a public model or another company’s service?
Only within the provider’s explicit authorization and testing rules. Otherwise, test your own integration in an environment you control.

How often should I repeat these tests?
Repeat them after meaningful changes to the model, prompts, data sources, permissions, or tools, and include them in routine security reviews.

What should I do after finding a failure?
Stop unsafe testing, protect the evidence, notify the authorized system owner, fix the underlying control, and retest in a safe environment.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.