The demo always goes well. The vendor types ignore previous instructions into a chat window, a red banner appears, and the dashboard lights up with a blocked attack. What the demo shows is a classifier reacting to a well-known string. Your risk is different: an agent with a refund tool and a mailbox, reading an email from a stranger that asks it, politely and in plausible business language, to do something it should not.
The checks below are for security leaders, platform owners and the procurement and risk teams who sit with them, and they assume no machine-learning background. Each one has an answer that differs between a product that controls something and one that describes something, and each comes with a test you can run in a trial. A vendor's answer in a sales meeting is a claim. The test is the evidence.
Placement decides what a product can do
Every AI security product sits in one of three places, and the placement settles most questions before any feature list is read. Out-of-band products read logs, traces and configuration after calls happen; they are easy to deploy and cannot stop anything. In-application libraries run inside your code; they can stop calls, but only in applications whose teams installed and updated them. Inline gateways sit between agents and the models and tools they call; they see everything routed through them and nothing that is not.
An out-of-band product reports a risky call after it runs. A library sees it only in applications that installed it. An inline gateway can stop it, but only for traffic routed through it. Ask which one you are buying and what it misses.
None of these is wrong, and a mature program usually has some of each. The theater starts when a product in one position is sold with the claims of another, such as a log-analysis tool described as runtime protection. Agentless runtime protection is the giveaway phrase: something has to see the call before it runs, and if nothing sits in the path, the product is observing. Ask the vendor to draw where the product sits relative to your agent, your model provider and your tools, and to mark what happens to a call that never passes through it.
The ten checks
1. It can show one decision, with the rule behind it
Why it matters: a chart of blocked attacks by week tells you nothing about whether a specific risky call was handled. Ask the vendor to walk through one risky tool call from your own test agent: how the caller is identified, what is read from the call, which policy applies, what the decision is, and what is logged. Then ask for last Tuesday's decisions for one agent, per decision, with the agent, action, outcome, deciding rule and evidence, and a feed into your security monitoring tools. How to test: pick any decision at random and have someone on your team trace it to the input that caused it. Pass: under five minutes.
2. It stops consequential actions when it is down or unsure
Why it matters: the red-flag answer is that traffic passes through so the product does not break your app, with no mention of what is then unprotected. A good answer is that consequential actions stop rather than run unchecked, configurable per action. How to test: make the vendor's decision service unavailable mid-session, then repeat for slowness. Pass: consequential actions never run unchecked.
3. It reads what a tool call does, not just the tool's name
Why it matters: this is the check most often skipped, and it decides whether policies mean anything. A database tool running DELETE should be treated differently from the same tool running SELECT, and a force-push to the main branch differently from a push to a feature branch. OWASP's guidance on excessive agency recommends complete mediation, with authorization enforced in downstream systems rather than left to the model [3]. A control that sees only tool names cannot mediate anything, because one name covers harmless and destructive calls. How to test: have the agent call a database tool with a destructive statement hidden in an ordinary request. Pass: the call is classified by what it does.
4. It limits what a manipulated agent can do, instead of quoting a catch rate
Why it matters: here marketing and evidence part ways most visibly. OWASP says that, given how models work, it is unclear whether fool-proof prevention of prompt injection exists, and recommends least privilege and human approval for high-risk actions alongside filtering [2]. The UK's National Cyber Security Centre goes further: prompt injection may never be totally mitigated the way SQL injection can be, so the goal is to reduce risk and impact with deterministic safeguards rather than ones that depend on the model [1]. A vendor who has absorbed this talks about limiting consequences and shares false-positive rates on your traffic. A vendor who has not quotes 99% with no test set. How to test: plant an instruction in a document the agent reads, telling it to email the contents to an outside address. Pass: the email is stopped or asks a person, and the decision names the input.
5. An approval covers one call, not its riskier cousins
Why it matters: approvals keyed on tool name mean approving one call approves every call to that tool. A good answer: requests go to one inbox, an approval is bound to the call's exact arguments, remembered approvals cover only similar calls, and request volume is reported so you can retire rules that are always approved. How to test: get an approval for one call, change an argument and resend it. Pass: the approval does not cover the changed call.
6. Each agent can be tightened, and nobody loosens it silently
Why it matters: one global policy forces you to choose between too loose for your riskiest agent and too tight for the rest. A good answer is an organization floor every agent inherits, per-agent settings that can only be stricter, and exceptions that are named, scoped to one agent and expire. How to test: ask who can loosen an agent's policy and where that change is recorded, then try it.
7. Policy changes can be replayed before they ship
Why it matters: the red-flag answer is to try a change in monitor mode in production and see. A good answer replays the change against recent real decisions and shows which outcomes would change. How to test: propose a policy change and replay it against the trial's traffic. Pass: you see which past decisions would change before you apply it.
8. It names the traffic it covers and the traffic it does not
Why it matters: no product covers everything. A good answer is a specific list, such as model calls, tool and MCP calls and agent-to-agent messages, plus a specific list of gaps, such as tools that run locally inside an IDE. Ask separately how it finds agents and AI use you have not registered: several independent, read-only sources, such as code repositories, identity-provider app grants and cloud accounts. "We protect everything you route through us" is not discovery. How to test: connect a tool server the agent has not used before. Pass: the first call is held or asks, and the server's tool list is recorded.
9. It says exactly what it stores, and whether it trains on it
Why it matters: "we take security seriously" followed by a certification badge is not an answer. A good answer lists fields, retention periods and how secrets are masked, with a plain statement about model training. How to test: put a fake API key and a test card number in a prompt. Pass: both are detected and redacted before the request leaves.
10. It can tell you what it cannot do
Why it matters: this is the most useful question on the list and the cheapest to ask. Every real control has limits, and a vendor who knows the product can name them quickly, including attacks it does not detect and traffic it does not see. The NCSC's advice is blunt: beware any product that claims it can stop prompt injection, and look for vendors who understand how they reduce the risk [1]. The same logic applies to every claim of completeness. How to test: write down the answer and check it against what the trial finds.
Claims that should end the meeting early
- Stops all prompt injection, or a catch rate with no test set behind it. The standards bodies say prevention is not fully possible [1][2].
- AI-powered protection with no deterministic controls underneath. If every decision depends on a model judging another model, the product fails the way models fail.
- Zero latency. Anything inline adds some; ask for the added delay at the 95th percentile on your traffic.
- Compliance in a box. A product can produce evidence that helps an audit or a regulatory review; it cannot make you compliant.
Four weeks, two real agents, criteria written first
Pick two real agents: one that reads outside input such as customer email, and one that holds a consequential tool such as payments, database writes or deployments. Write the pass criteria before the trial starts, and keep the vendor's engineers out of the attack scenarios. The tests above become week three; a week of normal work in week two tells you whether false positives and approval requests are at a level your team would tolerate in production.
Pass criteria are written before the vendor starts. Week one connects two real agents and observes, week two measures false positives on normal work, week three runs attack and failure tests, and week four operates the product as your team would, then decides.
| Criterion | How to measure | Example threshold |
|---|---|---|
| Coverage | Share of the two agents' model and tool calls the product saw | All calls on the routed path, and a written list of what was not |
| False positives | Blocks or flags on a week of normal work, reviewed by the agent owner | Low enough that nobody asks to switch it off |
| Attack scenarios | Checks 3, 4, 5, 8 and 9, run by your team | All pass, or a fix date in writing for any that fail |
| Failure behavior | Check 2, repeated for slowness as well as outage | Consequential actions never run unchecked |
| Evidence | Check 1's random trace, by someone who did not set up the trial | Under five minutes |
| Operator load | Approval requests per day and time spent on them | Fits within the agent owner's day |
| Added latency | Delay at the 95th percentile on real calls | Within the agent's response-time budget |
Intertrace is an inline gateway, so the placement trade-offs apply to it: it decides allow, flag, ask or block on model calls, MCP tool calls, inference, service calls and agent-to-agent messages routed through it, fails closed when it cannot decide, and does not see what runs only on a developer's machine. It reads what a call does from the tool name and its arguments, scopes remembered approvals to calls like the approved one, and replays proposed policy changes against the last 30 days of decisions. Prompt-injection detection is one signal: structural evidence such as hidden instructions can block alone, while a model-judge label alone only flags.
Buy the vendor whose answers survived the trial
Northgate Credit Union, a fictional mid-size lender (the outcomes are illustrative), trialed three products on its member-service agent. The observability product recorded the planted instruction and the outbound email accurately, an hour after the email had gone; the library stopped the email but missed a second agent whose team had not installed it, and let calls through when its scoring service was down. The gateway stopped the email and held calls during the outage, was noisy in week two until the agent owner tightened two rules, and had listed a coding assistant's local file edits among the things it could not see. Northgate chose it for enforcement and kept the observability dashboards for its data team, and the deciding factor was not a feature: only the gateway's answers to checks 2, 3 and 10 in the sales meeting matched what the trial measured.
That is the test to carry into every evaluation. Write down what each vendor claims in the meeting, run the trial, and weight the match between the two above anything on a feature list.
