There are two versions of this meeting. In the first, the policy service had a twelve-minute brownout, the gateway in front of the agents was configured to fail open to protect availability, and the review is about the three payments that ran without a check. In the second, the gateway failed closed, every agent in the company stopped at once, and the review is about the bypass flag someone added that afternoon, which is still on.
Both teams made the same mistake. They chose one failure behavior for every call, and they chose it under pressure. The answer is neither fail-open nor fail-closed: it is a failure policy written per consequence tier before the outage, and rehearsed. Here are the questions a design review or an incident review will ask, with the answers we would defend.
What does our gateway do today when the policy service does not answer?
Probably whatever its client library does on an exception. A check has three outcomes in practice: allow, deny, and no answer. The third arrives as a timeout, a connection reset, a malformed response or an exception, and whatever code catches it makes the real decision, usually code written for availability that returns the request to the happy path.
The OWASP Authorization Cheat Sheet treats failed checks as a normal occurrence and asks that all exceptions and failed access control checks be handled, no matter how unlikely they seem [5]. It also says access should be denied by default [5]. A timeout handler that proceeds is a default allow with extra steps. Search your enforcement clients for the code that runs on timeout and read it; fail-open is often a library default nobody chose.
So do we fail open or fail closed?
Neither, for everything. Fail-closed on every call turns a policy-service incident into an outage for every agent, and people under pressure to restore service will find a way around it. Aim narrower: no call whose effect cannot be undone runs without a fresh decision, and calls that are cheap to get wrong keep working within limits you set in advance.
Use the consequence tiers that drive approvals: reads, reversible changes, irreversible changes, outbound or external actions, and money or administrative changes. The enforcement layer already classifies a call before it asks the policy engine. Do that classification locally, from the tool name and arguments, so it still works when the policy service does not.
A decision that arrives within the deadline is enforced as made. A missing one is routed by tier: reads and reversible changes may reuse a last-known-good decision and are flagged, while everything above that holds until a fresh decision arrives.
| Tier | No decision by the deadline | Reuse a cached decision? | Retry behavior | What the caller sees |
|---|---|---|---|---|
| Read | Use the last-known-good decision if one matches; otherwise hold | Yes, if the key matches exactly and the entry is minutes old, not hours | Budgeted transport retries with backoff | A slower answer, or a clear temporary refusal |
| Reversible change | Same as read, with a shorter time-to-live | Only an allow made under the current policy revision | Budgeted retries; the change is safe to repeat | Normal, or a temporary refusal |
| Irreversible change | Hold and fail closed | No | Queue the intent with an idempotency key; decide again before dispatch | Held, with a reason and a reference ID |
| Outbound or external | Hold and fail closed | No | Queue; after recovery the call is evaluated, and asks, again | Held, with a reason and a reference ID |
| Money or admin | Hold and fail closed | No | No automatic dispatch; a fresh decision and any approval are revalidated | Held, with a reason and a reference ID |
| Any tier, cached block | Keep blocking | Yes; reusing a block is the safe direction | None | The original refusal |
Two details make this work. The hold must be a real response the agent can understand: a structured refusal with a reason such as policy_unavailable and a reference ID, so the agent stops and reports instead of improvising a workaround. And a held call must never fall back to a direct connection to the tool or provider. A fallback that routes around the enforcement layer when that layer is unhealthy removes the control at exactly the moment nobody can observe it.
How long should the gateway wait for a decision?
No longer than the caller will. The Google SRE book's guidance on cascading failures is to set a deadline on every request and, rather than inventing one per backend call, propagate the caller's remaining deadline down the request tree [1]. A policy check that waits 30 seconds for an agent whose own model call times out in 20 is wasted work, and the agent's framework may retry while the first check is still running, doubling the load.
Make a tight deadline realistic by keeping the policy snapshot close to the enforcer, so the hot path evaluates locally and only the snapshot refresh is remote. The same chapter recommends failing early and cheaply under overload [1], which for enforcement means returning a hold promptly rather than queueing decisions inside the gateway until it runs out of memory.
Human approvals are a different clock. An approval waiting for a person is not a timeout; it has its own expiry, in minutes or hours, and when it expires the answer is no. Never let an expired approval decay into an allow.
Can we keep serving cached decisions through an outage?
For low tiers, yes, if the cache key contains every input that influenced the decision: tenant, agent, the human principal it acts for, tool and server, the normalized action and its tier, the resource or resource class, environment, and the policy revision. A key built only from the tool name will reuse an allow for one customer's record on another's, or an allow for a SELECT on a DELETE.
Treat an entry's lifetime as a revocation window you are accepting. If you remove an agent's access at 10:00 and cached allows live for five minutes, the agent may keep reading until 10:05. That may be fine for internal reference data and unacceptable for customer records; decide which, and write it down. Never cache an ask, an approval or a pre-approval as an allow. Approvals carry conditions and single-use state that a cache entry cannot represent.
Watch for the cache becoming the system; the SRE book warns that caches added for performance can end up as hard dependencies [1]. If the policy service has been degraded for an hour and nobody noticed because the cache kept answering, your monitoring is measuring the cache, and a health check that passes while the snapshot is hours old is measuring the process. Alert on cache-served decisions as a share of all decisions, and on the age of the policy snapshot.
Our retries are bounded. How does one payment run twice?
Because agents have more retry layers than ordinary services. The agent framework retries a failed tool call, the tool's SDK retries the HTTP request, and an HTTP client or proxy may retry again. The SRE book notes that retries at multiple levels can produce as many attempts as the product of the attempts at each layer, and recommends limiting retries per request, using randomized exponential backoff, and considering a server-wide retry budget [1]. Google's own overload handling uses a per-request limit of three attempts and only retries while retries are below 10% of a client's requests [2].
With three attempts at each of three layers, one payment request can reach the provider up to 27 times. An idempotency key derived from the approved action, and reused by every layer, collapses those attempts into one effect. The per-layer counts are illustrative.
Retry volume is a load problem; duplicated side effects are a correctness problem. A timeout does not mean the side effect did not happen, and APIs with side effects are not safe to retry unless they provide idempotency [3]. Stripe's documentation shows what that looks like: the server saves the status code and body of the first request made with a given idempotency key, returns the saved response to retries with the same key, including server errors, and rejects a reuse whose parameters differ from the original [4]. It prunes keys once they are at least 24 hours old, after which a reused key creates a new request [4], so a queue that replays day-old intents needs its own deduplication.
The agent-specific trap is that the model is a retry layer too, and a creative one. When a tool call fails, the model often tries again with slightly different arguments, which defeats an idempotency key generated per attempt. Derive the key from the intended action instead: the agent session, the normalized action and a hash of the material arguments the decision was made on. The same refund, retried by any layer, then carries the same key, and a genuinely different refund cannot reuse an approval it did not get. On recovery, decide queued actions again before dispatch; the agent's access may have changed during the outage.
What can fail-closed promise for streamed output?
Less than people assume. For text a model streams to a user, every token already delivered is disclosed. If a detector fires, or dies, halfway through a response, stopping the stream prevents further disclosure but cannot recall the prefix. Say so in design documents and incident reports, so nobody claims a block that was really a truncation.
- Tool calls are the easy case: never execute one until its arguments are complete and evaluated. A streamed function call that looks harmless in its first half can name a different recipient in its second.
- For streamed text, choose a release window per use case: a short buffer lets output checks run before release, and a longer one inspects more at the cost of latency.
- If the output inspector becomes unavailable mid-stream, the stream's tier decides. An internal summary can continue and be flagged; a response that may carry customer records should stop.
- Record what was delivered before a stop, so the incident review starts from what the user actually saw.
What does the difference look like in an outage?
Halden Travel, a fictional booking company, replayed a twelve-minute policy-service failure caused by a bad configuration push against its refunds agent (timings and counts illustrative). Under the old design, which proceeded on error, a few hundred calls ran with no decision, including two emails to an address the customer had changed that morning and three refunds, and queued retries from three layers briefly overloaded the recovering service. Under the redesign, reads and note updates ran on flagged cached decisions for five minutes and then held, every email and refund held from the first second, and each queued refund was decided again on recovery under the idempotency key of its original intent, so none was paid twice.
Who may switch enforcement off in an emergency?
Someone will ask during the first long outage, so write the answer now. An override names its scope, its approver and when it ends, leaves logging on, and does not touch the highest tiers. A bypass with no owner and no expiry is how the second meeting at the top of this piece ends.
Where does Intertrace's enforcer come down?
Intertrace's gateway sits inline between agents and their models, tools, MCP servers and other agents, and its enforcer fails closed when it cannot reach a decision. It reads each call's consequence tier from the tool name and arguments, so outbound messages, payments and admin changes are recognized as such regardless of what the caller claims. Changes to what counts as a low-tier call can be replayed against the last 30 days of real decisions before they ship. Every decision is a structured event recording the rule, layer and evidence, which you can export to your SIEM and use to rebuild an incident timeline.
How do we know any of this works?
Only by running it. The SRE book's warning about graceful degradation applies to every fallback here: the code path you never use is the code path that often does not work [1]. Run these in staging with real agent traffic, and after every change to the enforcement path.
- Make the policy service unreachable. Expected: reads and reversible changes run on cached decisions until those expire and are flagged; every irreversible, outbound, money and admin call holds with a structured reason.
- Make it slow instead of down, responding just after the deadline. Expected: the same outcome, with no pile-up of waiting requests in the gateway.
- Revoke an agent's access, then cut the policy service. Expected: no allow survives past the cache lifetime you documented.
- Inject a timeout after the payments provider has accepted a request. Expected: retries from every layer carry the same idempotency key, and exactly one payment exists.
- Kill the output inspector halfway through a streamed response. Expected: the stream's tier decides whether it stops, and the delivered prefix is recorded.
- Search for any code path that sends a held call directly to the tool or provider. Expected: none remains, and a test proves it.
