LLM penetration testing: what to test before launch
Scope an AI application assessment around data, identity, tools, and observable outcomes, with a practical test matrix for a support assistant.
An LLM penetration test should assess what the complete application can read, reveal, and change. A model's refusal to answer one malicious prompt says little about whether a retrieval service exposes another tenant's documents or a tool executes an action under the wrong identity. Start by naming the assets and actions that the application must protect.
Consider a fictional customer-support assistant that retrieves help articles, reads account details, drafts replies, and can request a refund. Its security depends on the connections between those functions. The test should identify whether an untrusted document, a customer message, or an ordinary user can redirect those connections toward an unauthorized outcome.Source: OWASP: direct and indirect prompt injection
Draw the application, including its identities
Map the user session, model API, retrieval index, tool gateway, external services, and logs. For every connection, record the identity used and the data it can access. Note where permissions come from: the authenticated user, a shared service account, an organization configuration, or a value proposed by the model. These distinctions determine the tests.
OWASP describes excessive agency as harmful actions enabled by unnecessary functionality, permissions, or autonomy. Its mitigation guidance includes enforcing authorization outside the model and limiting the functions and privileges available to tools. Use those principles to define what the assistant is permitted to do before testing whether it can exceed that authority.Source: OWASP: excessive agency
Test a matrix of outcomes
| Surface | Question to investigate | Evidence to capture |
|---|---|---|
| Retrieved documents | Can untrusted text influence a tool action or disclose unrelated data? | Retrieved passage, tool request, and resulting action. |
| Tenant selection | Can one customer obtain another customer's account details? | Authenticated identity, selected account, and returned fields. |
| Refund tool | Can the assistant bypass the user's refund limit or required approval? | Policy decision, approval record, and transaction state. |
| Output rendering | Are generated links and markup handled safely by the application? | Rendered output and destination handling. |
| Logs and memory | Do sensitive records become visible to another user or later session? | Access checks, retention settings, and isolated session results. |
Separate prompt behavior from application behavior in the report. The assistant might propose an unauthorized refund while the tool correctly rejects it. That is a different outcome from a completed refund. Record the proposal, the policy check, and the final state so the team can fix the failed boundary and preserve the controls that worked.
Include ordinary application security
Assess the API and surrounding application as well as model interactions. Authentication, tenant filtering, file handling, session management, and server configuration remain part of the system. A vulnerable download endpoint can expose a document without involving the model at all. Scope these surfaces explicitly so the engagement does not stop at a collection of prompts.
For retrieval, seed separate synthetic records with clear ownership and access rules. Test the user who should see a record, a user who should not, and a session whose permissions have changed. Check the application again after an access revocation or document deletion. The expected behavior needs to cover caches and queued work, not only a fresh retrieval request.
Make findings reproducible without overstating certainty
Capture the application build, model identifier, relevant configuration, test identity, retrieved context, and tool responses. Record whether a result repeats across attempts and whether it depends on a particular conversation state. Where a provider does not expose a stable model snapshot, document that limitation. A later model update can change behavior even when your own code is unchanged.
Report observed results within the conditions tested. A failed attack attempt is useful evidence about that case; it does not establish immunity to prompt injection. Likewise, an inappropriate answer does not automatically establish access to a protected system. Severity should reflect demonstrated exposure, the permissions needed, and the business effect.
Agree on the handoff before testing starts
- Provide technical findings with the triggering input, relevant context, and verified outcome.
- Document tested identities, tools, workflows, and exclusions so coverage is visible.
- Assign fixes to the component that enforces the affected boundary.
- Retest agreed changes against the original case and relevant variants, including ordinary allowed behavior.
Bring the architecture, tool definitions, permission rules, representative workflows, and a synthetic test environment to the scoping discussion. Those materials let the assessment focus on what your team needs to trust before it exposes the application to users.
Sources
- OWASP: direct and indirect prompt injection genai.owasp.org
- OWASP: excessive agency genai.owasp.org
Bring your AI application's data sources, tools, and permission model. We can scope an assessment around the actions and information it must protect.
Explore penetration testing