AI Security Testing Beyond the Model

Updated 5 November 2026 · Standard formula, shown below

An AI security test should examine the whole system around a model: its data, prompts, tools, integrations, users, infrastructure, and operating procedures. A model can behave safely in isolation yet expose sensitive information or trigger harmful actions when connected to real services, so testing must follow information and authority across the complete application.

Why the model is only one part of the risk

A model is one component in a system that may include a web interface, retrieval database, identity provider, orchestration code, plug-ins, external APIs, logs, and human reviewers. Each component can introduce a weakness, and interactions can create risks that do not appear in a standalone model evaluation.

For example, a model might refuse to reveal confidential records when asked directly. But if the application retrieves records without checking the current user’s permissions, the model may receive and summarize information that user should never have seen. The failure is in access control and data flow, not necessarily in the model’s ability to follow instructions.

Security testing therefore asks more than “Can the model be tricked?” It asks what an attacker can reach, what information or actions are available, which controls constrain them, and whether the system detects and contains misuse.

Map the system, data, and trust boundaries

Start by documenting the application’s components and the paths between them. Identify which parts are controlled by the organization, which are operated by service providers, and which accept content from users or other untrusted sources. Include development, testing, and production environments, since they may differ in data access and configuration.

Trace sensitive information from its source to its destinations. Relevant data may include user prompts, uploaded files, retrieved documents, model responses, credentials, system instructions, and diagnostic logs. Record where data is stored, how long it is retained, who can access it, and whether it is sent to an external service.

Inputs: user messages, documents, images, retrieved web content, and data supplied by connected systems.
Processing: prompt templates, retrieval pipelines, model calls, filters, and orchestration logic.
Outputs and actions: answers, generated files, database changes, emails, transactions, or tool calls.
Controls: authentication, authorization, validation, monitoring, rate limits, and human approval.

This map helps testers find trust boundaries: points where untrusted input enters, privileges change, or data crosses between services. It also provides a basis for deciding which scenarios matter most.

Test data handling and access controls

Data security deserves testing both before and after model inference. Check whether users can retrieve only records they are authorized to see, whether filters are enforced in the application rather than left to the model, and whether data from one account can leak into another account’s session or search results.

Test how the system handles confidential information in prompts and outputs. Ask whether secrets, personal data, internal instructions, or another user’s content could appear in responses, logs, analytics, caches, or error reports. Review retention and deletion behavior, and verify that access to stored prompts and documents follows the same rules as access to the source data.

Also examine the data used to configure or improve the system. Confirm that training, evaluation, and retrieval datasets have appropriate permissions and protections. Test ingestion workflows for unexpected file types, malformed content, misleading metadata, and documents that should not be indexed. A secure model cannot compensate for a pipeline that imports data carelessly.

Probe prompts, retrieved content, and tools

Prompt injection is a useful test case, but it is not the whole assessment. An attacker may place instructions in a user message, uploaded document, retrieved page, or content returned by a connected service. Test whether the application treats such material as untrusted data and whether it can alter the system’s instructions or redirect the assistant.

When the model can use tools, test the full action path. Check whether tool calls are limited to the current user’s permissions, whether arguments are validated, and whether sensitive or irreversible actions require confirmation. A natural-language refusal is not a substitute for a server-side permission check.

Try to make retrieved content request secrets or unauthorized actions, and check whether it can change what the system does.
Test whether a user can cause access to another account’s documents or invoke tools outside the user’s role.
Check whether malformed, ambiguous, or unexpectedly large tool arguments are rejected safely.
Verify that actions such as sending messages, changing records, or deleting data are constrained and auditable.

Testing should take place in an authorized environment with safe test accounts and non-production data whenever possible. Avoid using real customer information or allowing experiments to trigger live, irreversible actions.

Review infrastructure, dependencies, and operations

The surrounding software needs conventional security testing as well as AI-specific scenarios. Examine the application’s authentication and session handling, API authorization, network exposure, storage permissions, secrets management, dependency risks, and deployment configuration. A vulnerable web service remains vulnerable even if its model is robust.

Check the security of connections to model providers and other services: what data is transmitted, how credentials are protected, how failures are handled, and whether the application can safely operate when a dependency is unavailable. Review logging for both usefulness and overcollection. Logs should support investigation without unnecessarily retaining sensitive prompts or outputs.

Operational tests should cover monitoring, incident response, and change management. Determine whether teams can notice unusual tool use, spikes in sensitive-data requests, or repeated attempts to bypass controls. Confirm that staff know how to disable a risky integration, revoke a credential, or roll back a deployment. Re-test important controls after changes to models, prompts, retrieval sources, tools, or permissions.

Prioritize by impact, not by a single score

There is no single test score that proves an AI application is secure. Prioritize scenarios by the assets at stake, the privileges exposed, the likelihood of an attack, and the consequences of failure. A public FAQ assistant with no private data and no tools has a different risk profile from an assistant that can search internal records or modify business systems.

Use several kinds of evidence: threat modeling, code and configuration review, conventional application testing, adversarial model evaluations, and realistic end-to-end exercises. Record the setup, expected behavior, observed behavior, and severity of each finding. A failed refusal test is more meaningful when the report explains what protected resource or action became reachable.

Testing has limits. Results depend on the system version, data, permissions, and scenarios examined; new model behavior or integrations can change the risk. A successful test campaign is evidence about defined conditions, not a guarantee that every future attack will fail.

Frequently Asked Questions

Is testing the model itself still necessary?
Yes. Evaluate the model for unsafe or unreliable behavior relevant to its intended use. But model testing should complement checks of data access, application logic, tools, infrastructure, and operations rather than replace them.

What is the difference between prompt injection and a software vulnerability?
Prompt injection is an attempt to influence a model through instructions embedded in input or retrieved content. It becomes a broader security failure when the application lets that influence bypass permissions, expose protected data, or cause an unauthorized action.

Should an AI security test include third-party services?
Yes. Include connected model providers, retrieval services, APIs, and other dependencies in the system map. Assess what information they receive, how access is controlled, and what happens if they fail or return untrusted content.

How often should an AI system be retested?
Retest after meaningful changes, including new models, tools, data sources, prompts, permissions, or deployment settings. Organizations should also schedule periodic reviews based on the system’s risk and exposure.

Does a refusal from the model prove an action is secure?
No. Refusals are behavior, not an access-control boundary. Enforce authorization and validation in application code and connected services, so a request cannot succeed merely because the model produced a different response.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.