One pull request against the procurement agent at Brackwater Freight, a fictional freight company, made three changes. Purchase orders over $25,000 would now ask a person. Direct calls to the payments API would be blocked, since payments are supposed to go through the ERP. And, to unblock an onboarding project, an exception let the agent update vendor records without asking. The agent reads purchase requests, checks budgets, creates purchase orders and maintains vendor records, so all three changes touched money.
Two of the three were tightenings, and the change would have been reviewed as one. The third reopened a path a person had been guarding. This paper follows that pull request through a release pipeline built to catch it. Our position: an authorization change is a production change and should ship like one, but the answer is not a weekly change board. It is handing an ordinary peer reviewer a decision diff.
Policy edits fail like deploys do
When an agent's tool list, approval rules or thresholds change, the next thousand calls behave differently. A rule meant to add an approval for large refunds that matches every refund stalls the support queue. An exception meant for one onboarding task that matches any vendor update lets an injected instruction rewrite bank details without a person seeing it. One is an outage of availability, the other of control.
Google's SRE team reports that in its experience a majority of incidents are triggered by binary or configuration pushes [1]. Authorization policy is configuration with unusually high stakes, and in many organizations it changes through an admin console with no review, no test and no record of what it replaced. NIST SP 800-53 already sets the bar for configuration in general: control CM-3 asks organizations to review proposed changes with explicit consideration of security and privacy impact, document the decisions, retain records and monitor change activity, and enhancement CM-3(2) asks that changes be tested, validated and documented before implementation is finalized [2].
The obvious way to meet that bar is the wrong one. DORA's research found that change approval works best as peer review during development backed by automation, that heavyweight approval by external bodies hurts delivery performance, and that there was no evidence a more formal external review lowered change failure rates [3]. Every stage below exists to give the peer reviewer evidence, not to add a committee.
A change moves from pull request through rule tests, a 30-day replay that produces a decision diff, monitor mode, staged enforcement by agent, and an audit link from each decision to its policy version. Rollback from staged enforcement returns to the pull request, not to a console edit.
The policy has to be text before anyone can review it
Brackwater's change could be a pull request only because the policy lived in a repository: the organization's baseline, a file per agent or role, and named exceptions, all behind required review. If you generate policy, commit the generator's output, because the reviewer has to see the effective rules, not the script that produces them. A console that edits policy in place should be read-only, or should open pull requests instead.
The rule tests passed, because they test what the author imagined
Write rule tests the way Open Policy Agent expects them. In Rego, tests are ordinary rules whose names start with test_, the opa test command discovers and runs them, it can report coverage showing which rule bodies were exercised, and the with keyword replaces input data and functions with mocks [4]. Whatever your policy language, keep those properties: tests beside the rules, coverage that exposes rules no test touches, and mocked inputs so each test pins one call.
Test the boundaries, not the middle. For the $25,000 threshold that means an order just below it, exactly at it and just above it. For a destination rule, the address on file, a lookalike domain and a new address. Add one test for each incident or red-team finding that produced a rule, so that deleting the rule fails CI with the reason in the test name.
Brackwater's tests passed. They were supposed to: they checked the cases the author thought of, and the author thought the exception was about onboarding.
Replay found what the tests could not
Tests cover the cases you wrote; replay covers the cases you had. Take the proposed policy, evaluate it over the inputs of the last 30 days of real decisions for the affected agents, and compare each new outcome with the one that actually happened. The result is a decision diff: how many calls keep their outcome, how many become stricter and how many become looser, each with examples a reviewer can open.
Illustrative counts from replaying 30 days of decisions: 18,412 unchanged, 236 allow → ask and 41 ask → block from the intended tightenings, and 17 ask → allow loosenings flagged as needing extra review.
Of 18,706 replayed decisions (illustrative counts), 18,412 were unchanged. The 236 that moved from allow to ask were almost all purchase orders above the new threshold, and the 41 that moved from ask to block were the direct payment calls. The 236 is a staffing question for the approvers, roughly eight extra asks a day, not a security one.
The 17 ask → allow results came from the exception, and opening them showed the problem. The exception had been written for vendor updates on any record, and several of the 17 were bank-detail changes on existing vendors that a person had previously reviewed. The fix narrowed the exception to new vendor records created by the onboarding job and gave it a 14-day expiry. The rerun showed zero loosenings.
That is why the pipeline reads the direction of each change, not just the count. Tightenings cost friction, so the question is whether the volume is acceptable. Loosenings cost risk, so each needs a reviewer to confirm it was intended, and the check should fail on any unacknowledged loosening, however small.
| Change in outcome | Direction | Main risk | Review action |
|---|---|---|---|
| Unchanged | None | None from this change | Sample a few to confirm the replay ran against the intended agents and window |
| Allow → ask | Tighter | Approval volume and stalled work | Estimate approver load per day and check whether a scoped pre-approval could absorb routine cases |
| Ask → block | Tighter | Legitimate work left with no path | Confirm each blocked case has another route or was never legitimate |
| Allow → block | Tighter | Silent breakage of a working flow | Notify the agent's owner and stage the change by agent |
| Block → ask | Looser | Calls that used to stop now reach a person | Confirm approvers know what to refuse |
| Ask → allow or block → allow | Looser | Consequential calls run with nobody involved | Open every case, require a second reviewer, and record a named reason in the change |
Tighten-only layering is what made 17 cases findable
A reviewer can judge a change quickly only if they can reason about it without reloading the whole policy. Layering makes that possible when inner layers can only narrow. The organization layer sets a floor: what is always blocked, what always asks, the data-protection minimums. Each agent's policy tightens that floor for the agent's job. Loosening happens only through exceptions that are named, scoped to one agent and set to expire.
Under that rule most changes are monotonic. An edit to an agent layer can only make that agent stricter, so the reviewer's question is whether it breaks legitimate work, which replay answers. A change to the organization floor touches every agent and gets the widest replay. The only loosening paths are exceptions and floor changes, both rare and both visible in the diff, which is why Brackwater's exception stood out at all. Without the rule, any agent edit might quietly override a baseline, and every review becomes a global analysis.
Replay is the strongest evidence you have, not proof
Replay only sees traffic that happened, so it says nothing about a tool the agent has never called. It sees inputs as they were logged, so if your logs redact arguments, argument-based rules replay against the redacted form. And it cannot predict how agents and the people who run them will adapt to a new ask. Those gaps are what the next two stages are for.
A week in monitor mode, then one agent at a time
Monitor mode shows what happens this week: the new policy evaluates live calls and records its decision beside the enforced one without acting. Run it through at least one full weekly cycle of the agent's work and compare the live diff with the replay diff. If they disagree, either the agent's traffic has shifted or the replay inputs were incomplete, and you want to know which before enforcing.
Then enforce in stages, starting with one agent or with non-production environments. The SRE workbook defines canarying as a partial and time-limited deployment of a change together with its evaluation, with later stages exposing a larger population once earlier ones pass [1]. For authorization the population is agents rather than hosts, and the signals are ask volume, new blocks on calls that used to run, and reports from each agent's owner.
Rollback should mean redeploying the previous policy version from git: one command, no hand edits. Two cautions. Rolling back a tightening is a loosening, so look at what it reopens. And approvals or pre-approvals granted under the new version may not mean the same thing under the old one.
Every decision names the commit that made it
This is the stage that pays off during an incident. Each enforced decision should carry the identifier of the policy version that produced it, beside the deciding rule and layer: held by the organization's payment rule, say, rather than by the agent's own list. Then the question of why a payment ran on the 12th has an answer that names a commit, a pull request and its reviewer, and the question of which decisions a bad change affected becomes a query instead of an investigation. It is also what CM-3 means by documented change decisions, retained records and monitoring of change activity [2]. Record the source commit rather than a build number, so the link survives rebuilds.
Which parts Intertrace runs and which you build
Intertrace supports parts of this directly. Agent policy can be exported, edited and applied with the itr CLI, so changes flow through ordinary code review; proposed access and policy changes can be replayed against the last 30 days of real decisions; layering runs organization → agent → exception, with exceptions that expire after seven days by default and 90 at most; and each decision records its rule and layer. Monitor-then-enforce staging and commit stamps are practices you build around it.
The decision in front of you
Most of this pipeline is engineering work. The part that needs a decision from whoever owns agent risk is narrower: whether the admin console stays a way to change what agents may do. While it does, every control above is optional, because the fastest path to a change skips review, tests and replay. Close that path, require a second reviewer on every loosening the diff reports, and the Brackwater pull request becomes routine.
