At 02:10 a malformed carrier status message sends Wrenfield Freight's dispatch agent into a loop that cancels confirmed bookings. The on-call engineer scales the deployment to zero, and the cancellations keep coming, because a retry worker is replaying the queue. Rotating the carrier booking key would stop it, but billing reconciliation shares that key, so rotating it means waking finance. The cloud key cannot be rotated without breaking four older scripts. The staff engineer whose token the agent uses is on leave. Wrenfield is fictional and the timeline illustrative: the cancellations stop 47 minutes later, when someone finds the queue and drains it by hand.
The agent was never provisioned. It was lent other people's authority. The common response, a better secret store for the same shared keys, would not have shortened that night by a minute. What would have: an identity the agent owns, authority split into grants that can be withdrawn one at a time, and a revocation time measured before anyone needed it. The rest of this piece follows Wrenfield's redesign through each stage of the lifecycle.
What the dispatch agent was holding
The agent reads the shipment queue, books and cancels carrier slots through a booking API, and opens tickets when an integration fails. One staff engineer built it in a quarter. It runs as three replicas plus a retry worker, and each of its three credentials is a pattern the OWASP Non-Human Identities Top 10 names [1].
- The engineer's personal source-control token. Every action is attributed to a person who did not take it, and the agent's access silently follows that person's role changes. OWASP lists the mirror image, people doing manual work through non-human identities, as Human Use of NHI (NHI10), because human and automated activity become impossible to tell apart [1].
- A cloud access key for svc-integrations, a service account four older scripts also use. That is NHI Reuse (NHI9): a compromise in one consumer reaches everything that shares the credential, and revoking it breaks all of them at once [1].
- A carrier booking key in an environment file, created once and never changed. That is Long-Lived Secrets (NHI7), and usually Overprivileged NHI (NHI5) as well [1].
Had Wrenfield retired the agent instead of fighting it, all three would have ended the same way: deployment deleted, credentials still working. That is Improper Offboarding (NHI1) [1]. None of it is a model problem. The redesigned agent was just as capable of looping. What changed was who could stop it, and how fast.
On the left, the agent acts through a person's token, a shared service account and a static key, so stopping it means breaking other work. On the right, it holds its own identity, receives short-lived credentials, and each connection and action is a separate grant that can be withdrawn on its own.
Name the workload, not the engineer who built it
The redesign gave the agent its own workload identity in each environment: the dispatch agent in production, owned by a named team, distinct from the staging copy and from anyone's laptop experiment, and stable across ordinary redeploys. Where the platform supports workload identity, use it instead of distributing secrets. In SPIFFE's model, a workload receives a verifiable identity document from a local Workload API, bundles no authentication secret, and "all private keys (and corresponding certificates) are short lived, rotated frequently and automatically" [2].
Federation does the same job for agents that run in CI. With GitHub Actions OpenID Connect, the cloud provider checks the job's token claims against conditions on a role, such as a subject of repo:octo-org/octo-repo:environment:prod, and issues an access token valid for a single job, so no long-lived cloud secret is stored in the repository [3]. Wrenfield's agent got federated cloud access scoped to its own role, which took it off svc-integrations.
The person who starts a task is a fact to record and pass along, not a source of authority. A background agent may legitimately keep running after its launcher changes teams. Its permissions should come from its own grants, never from whoever last touched it.
Three grants, three lifetimes, three ways to withdraw them
Most credential confusion comes from treating three different objects as one.
- The runtime credential proves which agent is calling. Make it short-lived and unique to one agent, so a copy buys an attacker little time and revoking it touches nothing else.
- The connection makes a downstream account reachable: an OAuth grant to a mailbox, a database role, a payments API key. Several agents often share one, which is why disabling an agent must not be confused with revoking its connection.
- The action grant decides what this agent may do through that connection right now. This is where zero standing privilege lives: the identity alone can do nothing consequential, and each consequential action is authorized for that call, with a short validity window, by a component that does not take instructions from the model.
The third layer is what bounds blast radius. An agent steered by injected text can ask for anything, but it holds only what the decision point grants in the moment. A short-lived runtime credential does not get you there alone: a short-lived token for a broadly privileged connection is still broad.
At Wrenfield, the booking connection was split so billing and dispatch hold separate credentials, which means rotating one no longer wakes finance. Cancelling a confirmed booking became a consequential action decided per call. And the retry worker, the component that kept cancelling after the replicas were gone, was made to check the same decision point before replaying anything.
Wrenfield's revocation latency was 47 minutes. Measure yours before an incident does.
"We revoked it" is not a time. A change in one place has to reach token validators, caches, workers and provider sessions, and each path stops authorizing new actions at a different moment. RFC 7009, which defines OAuth token revocation, is candid about this. It allows a propagation delay that implementations should minimize, says revoking a refresh token should also invalidate access tokens from the same grant only where the server supports that, and notes that self-contained access tokens, validated without a call back to the server, need some other mechanism, which is why it recommends short access token lifetimes [5].
Illustrative values for one agent: an online check at the decision point stops new calls almost at once, a queued retry runs until the queue drains, a cached access token lasts until it expires, and a refresh token on a shared connection works until someone revokes it. Your revocation latency is the slowest path you have not closed.
Measure each path and write the result down: the online check at the decision point, cached tokens until expiry, refresh tokens on shared connections, queued and scheduled work, and any session a provider has already accepted. State a bound for consequential actions such as payments, deletes and outbound messages, and design to it with short token lifetimes, an online check at the moment of action, or both. When the system holding revocation state is unreachable, consequential actions fail closed rather than fall back to a cached allow.
Revocation cannot undo an effect that already happened, so the runbook names three owners: who stops new calls, who cancels queued work, and who recovers completed actions. In Wrenfield's re-test, disabling the agent's identity stopped new calls at the decision point within seconds, queued retries were refused on their next attempt, and nothing else in the company noticed. That is the outcome to design for: one agent stopped quickly, by the person on call, with no collateral damage.
Rotation is a migration, and refresh tokens outlive the agent
An unrehearsed rotation fails one of two ways: the new credential has not reached every replica and the agent breaks, or the old one is never disabled because nobody is sure what still uses it. Run each rotation as a small migration with a cutover time, a bounded overlap, a rollback condition and a signal that proves the old material stopped working. Log credential fingerprints or generation numbers, never values, so you can tell which generation served a request.
Refresh tokens need their own rules because they outlive access tokens. RFC 9700, the OAuth 2.0 Security Best Current Practice, requires refresh tokens for public clients to be sender-constrained or rotated, and describes rotation with reuse detection: the server issues a new refresh token on every refresh, and if an invalidated one is ever presented, it revokes the active one, because it cannot tell whether the attacker or the legitimate client sent it [4]. It also says refresh tokens should expire after a period of client inactivity, and that access tokens should be audience-restricted and limited to the minimum privileges required [4].
For agents this matters more than for most clients. A retired agent whose refresh token still works on a shared connection can regain access the next time anything refreshes on its behalf. Inactivity expiry closes that gap on the server's schedule. Offboarding should close it on yours.
Retirement ends in a failed call, not a toggle
Deleting the deployment looks like the end, which is why most programs skip the rest. Run offboarding as an ordered sequence.
- Disable the workload identity and remove its action grants, so no new consequential call can be authorized.
- Cancel queued and scheduled jobs under that identity, and invalidate any unconsumed approvals or permits issued to it.
- Remove its connection bindings, then decide for each shared connection whether the credential was exposed to the agent and must be rotated or revoked for everyone.
- Search for copies in environment files, CI secrets, notebooks and repositories where its keys were pasted during development.
- Mark the identity retired rather than deleting it, and keep the audit history and the policy versions it acted under, so past actions stay attributable.
- Run a negative test with the retired identity on every path it used, and repeat it a month later.
The table compresses the whole lifecycle into the failure to expect at each stage and the test that proves the control holds.
| Stage | What usually goes wrong | Control | Test that proves it |
|---|---|---|---|
| Issue | The agent is given a person's token or a shared service account | A workload identity per agent and environment, with a named owning team | Every call in the logs resolves to one agent and one owner, never to a person who did not act |
| Grant | Scopes are the union of everything the agent might ever need | Per-connection grants plus per-action decisions for consequential calls | A call outside the agent's task is refused even though the connection would allow it |
| Use | Secrets sit in prompts, repository files or tool output | Credentials delivered by the runtime, held outside model context, logged by fingerprint | Search transcripts and traces for the secret pattern and find only masked values |
| Rotate | Old and new material both work indefinitely after a rotation | A planned cutover with a bounded overlap and refresh token rotation | The previous credential is rejected once the overlap window ends |
| Revoke | Disabling the agent leaves cached tokens, refresh tokens and queued jobs alive | A measured revocation latency per path, with fail-closed behavior for consequential actions | After disable, a scripted consequential call on each path fails within the stated time |
| Retire | The deployment is deleted and the credential keeps working | An offboarding sequence that ends in a negative test, with audit history retained | The retired identity produces no new effect on any path a month later |
What Intertrace records, and what stays with your teams
Agents reach models, tools and MCP servers through Intertrace's gateway with an itr_rt_* runtime key. Tenancy and identity come from that key, not the request body, so each agent can have its own key, and every decision event records the agent, the rule and layer that decided, and the evidence. Consequential actions are decided per call by consequence tier, and the enforcer fails closed when it cannot decide. Discovery finds credentials agents use across repositories, cloud accounts, identity-provider AI-app grants and AI provider organization keys, storing only a masked value and fingerprint. Rotation, federation and connection revocation remain changes your teams make in those systems.
The test to run on Monday
Pick the production agent whose worst action is most consequential, and inventory its credentials: which belong to a person, which are shared with other workloads, which have never rotated. Then, in staging, disable its identity and script one consequential call down every path it uses: the decision point, a cached token, the refresh token on each shared connection, the queue. Write down when each one stops working. The slowest number is your revocation latency. If it is longer than the agent needs to do real damage, that agent is where the redesign starts.
