Security teams are rarely asked for a latency budget, so they rarely have one. The first conversation about it usually comes when a product team measures the slowdown and asks for the guardrail to be switched off. By then the design is set. If every request goes to a model judge, the bill scales with traffic rather than with attacks, and the p99 belongs to the judge.
The question is not whether to run a model judge but which traffic needs one, and how little of it you can send. Treat each check like any service dependency, with a cost per call, a latency distribution, a failure mode and a plan for when it is slow or down. Agents raise the stakes: a chat turn is one request, but an agent task can be twenty calls in sequence, and per-hop overhead compounds. What follows are the six decisions that settle the budget, in the order to make them.
1. Decide what has to block
A check belongs inline only if its verdict changes what happens next and acting later would be too late. Everything else can run after the fact and alert, open a finding or tighten policy for the next call. Deciding this first shrinks every later decision.
| Check | Placement | Reason |
|---|---|---|
| Authentication and tenancy | Inline, blocking | Nothing should proceed under the wrong identity |
| Structural checks on requests and tool results | Inline, blocking | Microsecond cost, and high-precision signals can stop an injection before the model reads it |
| Authorization of consequential tool calls | Inline, blocking | A payment or deletion that has run cannot be recalled by an alert |
| Local injection classifier | Inline, flags or contributes to a decision | Cheap enough to run inline; precise enough to escalate, not always to block alone |
| LLM judge on uncertain content | Inline before consequential actions, async otherwise | Its latency is acceptable ahead of a payment and wasteful ahead of a read |
| Egress PII and secrets scan | Inline for responses leaving the trust boundary | Redaction has to happen before the data leaves |
| Session-level behavior analysis | Async | It needs many hops to see a pattern and acts on the next call or the session |
Streaming needs its own decision. An egress scan that waits for the whole response defeats streaming; one that scans in windows holds back a short tail of tokens and releases the rest. The window size is a latency-against-precision trade to set deliberately, not inherit.
2. Order the checks by cost
A cascade runs cheap checks on everything and expensive checks only on what the cheap ones cannot settle. The idea is established in model serving: FrugalGPT describes an LLM cascade that learns which combination of models to call for each query, and reports matching the best individual model with up to 98% cost reduction on its benchmarks [1]. Security detection suits a cascade even better, because most traffic is plainly benign and the hard cases are rare.
All traffic passes structural checks in microseconds; the local classifier sees the illustrative ~20% carrying new free text, in milliseconds; the LLM judge sees about 2%, at hundreds of milliseconds. The shares are illustrative; the shape is the point.
Structural checks come first: deterministic checks over the bytes of the request for hidden and zero-width characters, instruction text in fields that should hold data, encoded blobs, known token formats such as API keys, card numbers validated by checksum, and schema and size limits on tool arguments. They run in microseconds, they are explainable, and a hit on a valid card number or an instruction hidden in a tool result is precise enough to act on alone. They also decide what the next tier reads. In an agent loop most of each request is context already scanned on an earlier hop, so a tool call with structured arguments or a short new reply may leave nothing for a classifier, which is why the illustrative classifier sees about a fifth of traffic.
Local classifiers come second: prompt-injection classifiers, PII entity detectors and intent routers running next to the gateway. Meta's Prompt Guard 2 is a representative example; its model card describes an 86M-parameter model built on mDeBERTa-base and a 22M variant built on DeBERTa-xsmall, a 512-token context window, and measured latency of 92.4 ms and 19.3 ms per classification on an A100 GPU at 512 tokens [2]. Local is not free: a long tool result must be split into segments and scanned in parallel, as the card recommends, or the classifier becomes the slowest hop before the model, and the card says to use it as one layer, not a standalone defense [2].
The LLM judge comes last. It reads context the classifier cannot and costs far more per call, so reserve it for the residue: classifier scores in the uncertain band, content that precedes a consequential action, and cases where a cheap signal and the surrounding context disagree.
3. Decide what a judge label can do on its own
The most effective way to cut judge cost is to stop letting judge verdicts block alone. Decision fusion gives each signal a role by how it fails. Structural signals, such as a card number that passes its checksum or an instruction hidden in zero-width characters, are rarely wrong and may block alone. Model-judge labels like prompt injection or jailbreak are useful but contestable, so they flag by default and block only when an independent, cheaper signal agrees, such as an intent classifier or a structural hit.
The reason is measured judge behavior. Zheng et al. found that strong LLM judges reach over 80% agreement with human preferences, about the level humans reach with each other, while also showing position, verbosity and self-enhancement biases [4]. They studied grading chat answers, not security verdicts, but the lesson carries: one judge label is an informed opinion with known biases, and blocking on it turns each bias into a false positive on legitimate work. Fusion also lowers the escalation rate. If a judge label alone cannot block a read-only call, there is little reason to judge read-only calls at all.
4. Price the escalation rate before you ship
The numbers below are illustrative assumptions, not measurements of any product or provider; the structure of the calculation is what transfers. Assume 2 million guarded model and tool calls a day over a 30-day month. A judged call costs $0.0005 (roughly 1,500 input tokens and a short structured verdict on a small hosted model) and takes 400 ms at p50 and 1,200 ms at p99. The local classifier is a fixed $2,000 a month of edge compute, enough for the 20% of traffic it sees, at 15 ms p50 and 40 ms p99 after segment-parallel scanning. Structural checks cost almost nothing and take well under a millisecond.
| Design | Share sent to judge | Judged calls per day | Judge cost per month | Total with classifier |
|---|---|---|---|---|
| Judge every call | 100% | 2,000,000 | $30,000 | $30,000 (no classifier needed) |
| Judge everything the classifier sees | 20% | 400,000 | $6,000 | $8,000 |
| Judge only the uncertain residue | 2% | 40,000 | $600 | $2,600 |
| Tight cascade with decision fusion | 0.5% | 10,000 | $150 | $2,150 |
Two things fall out. Cost is linear in the escalation rate, so that is the parameter to manage: a classifier update that moves it from 2% to 6% triples the judge bill without anyone touching the judge. And once escalation is low, the fixed classifier cost dominates and further judge savings stop mattering; that is the point to optimize the classifier's footprint instead.
Latency is where averages mislead. With 2% of calls judged at a 400 ms median, mean overhead rises by about 8 ms (0.02 × 400), which looks harmless on a dashboard. But 2% is more than 1%, so the p99 of all traffic falls inside the judged population: the p99 overhead is the judge's latency, not the classifier's. Cut escalation to 0.5% and the p99 returns to the classifier path, while the p99.9 still belongs to the judge. Google's SRE book makes the general case: most metrics are better treated as distributions, a simple average can obscure tail latency, and a high percentile such as the 99th or 99.9th shows a plausible worst case [3].
5. Decide what runs in parallel, and what happens on timeout
Illustrative, not measured: auth, structural checks and the classifier make up the security budget before the model, and the policy lookup runs in parallel with the classifier so the request waits for the slower of the two. The upstream model segment is not to scale.
Once the request is authenticated, the policy lookup for the agent's identity and the classifier score on new content can start together, and the decision waits for the slower one rather than their sum. Sequence checks only where one's output is another's input, as when a classifier score decides whether to call the judge. Parallelism spends compute on results you may not need, such as classifying a request a structural check is about to block. That is worth it at the cheap tiers and not at the judge tier, so start the judge only after the cheaper tiers say it is needed.
Give every check a deadline and decide its timeout behavior in advance: blocking checks on consequential actions fail closed and ask a person, advisory checks record the timeout and proceed. A judge without a deadline turns a provider slowdown into your outage.
6. Set the budget per hop, then check it per task
Set percentile targets per hop, then check the task. If guarded overhead is held to 25 ms at p50 and 80 ms at p99 per hop (illustrative targets), a twenty-hop agent task adds about half a second at the median. The task's tail is worse than any one hop's: across twenty independent hops, the chance that at least one lands in its own slowest 1% is about 18% (1 − 0.99^20). Give product teams that number before they measure it themselves.
Measure overhead per check as guarded minus unguarded latency for the same call, and keep approval wait out of it. Time a person spends deciding an ask is a governance cost, not guardrail latency; mixing the two makes both numbers useless.
Running the cheap tiers next to the enforcer
Intertrace runs a local prompt-injection classifier and PII detection on each edge machine next to its Go enforcer, so they add no network hop to another provider. Decision fusion is built in: structural signals such as hidden instructions, instruction-override language and real tokens, card numbers or SSNs can block alone, while judge-only labels like prompt injection, jailbreak or secrets leakage flag unless a cheaper intent signal agrees. The enforcer fails closed when it cannot decide, and every decision is a structured event, so you can measure each signal's rate and cost from your own telemetry.
Before any check goes inline
- Its job is written down: which decision it changes, for which consequence tier, and whether it blocks or alerts.
- Its p50, p95 and p99 are measured on production-shaped inputs, including long tool results, and its cost per call is recorded.
- It has a deadline and a timeout behavior.
- Its expected share of traffic is estimated, with an alert when the observed share drifts well beyond it.
- It runs in parallel with every check it does not depend on.
- Whether its verdict can block alone has been tested against benign traffic that resembles attacks.
- Its overhead is reported monthly per hop and per task, as percentiles, separately from approval wait.
