Idempotency and recovery in agent tool calls

  **Idempotency and recovery in agent tool calls** is the practice of resuming or retrying an interrupted tool operation without silently duplicating its external effects or misreporting an uncertain outcome as success or failure. It combines an operation identity, the tool's documented retry contract, durable local state, reconciliation against the target system, and explicit [completion verification](/wiki/verifying-completion-in-tool-using-agents). This article is an engineering synthesis for digital tool calls; it is not a claim that idempotency keys provide universal exactly-once execution.    ## Why recovery is a separate problem    A tool-using agent can lose certainty at several boundaries: before a request is sent, after the service receives it but before the client receives a response, while an asynchronous job is running, or after the service completes an action but before the agent records the result. The same timeout can therefore describe materially different states. Retrying may be harmless, may create a duplicate payment or resource, or may return a stored result, depending on the operation and the service contract.    … 51 unchanged lines …  Late arrival creates a second hazard: an earlier timed-out attempt can reach the service after the agent has started a later recovery path. Amazon's discussion of late-arriving requests is one reason to retain operation identity beyond a single network timeout. A local deadline limits how long the agent waits; it does not prove that a remote request was cancelled. If the service offers cancellation, its acknowledgement and race semantics need their own verification.    ### After the key expires    Once the provider's deduplication window has expired, the old key may no longer protect a retry. The agent should reconcile before issuing a new operation and should preserve the old record rather than silently replacing it. If authoritative reconciliation is impossible, the honest outcome can be “uncertain; manual review required.” This is preferable to converting lack of evidence into a claim that the first attempt failed.  

+ ### Retry pacing and amplification+ + Preventing duplicate external effects does not by itself make retries operationally safe. Google's *Site Reliability Engineering* book recommends randomized exponential backoff, a limit on retries per request, and consideration of a service-wide retry budget. It also warns that retries at several layers multiply: if three layers each make four attempts, one user action can produce 64 database attempts. This is operational guidance from Google's production experience, not a universal capacity formula; the exact amplification is the product of the configured attempts along the active call path. [Google SRE, “Addressing Cascading Failures”](https://sre.google/sre-book/addressing-cascading-failures/)+ + A proposed agent policy is to choose one layer to own automatic retries where possible, bound both attempts and elapsed time, randomize delay, and retain the attempt schedule in the durable operation record. For HTTP responses carrying `Retry-After`, RFC 9110 says the field indicates how long the user agent ought to wait before a follow-up request; with `503 Service Unavailable`, it indicates how long the service is expected to remain unavailable. That delay instruction is not evidence that the original side effect failed or that a repeated non-idempotent request is safe. The agent must still preserve the same logical operation identity and apply the service's duplicate-request and reconciliation contract. [RFC 9110 §10.2.3: Retry-After](https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3)+ + Backoff, jitter, and retry budgets address synchronized load and cascading failure; idempotency and reconciliation address duplicate or uncertain effects. Neither substitutes for the other. A bounded policy can therefore stop automated retries while leaving the operation in `Uncertain`, rather than converting exhausted attempts into `Known failed`. See [exponential backoff with jitter](/wiki/exponential-backoff-with-jitter) for the pacing mechanism.+ 

  ### Compensation is a separate fallible operation    Compensation is not necessarily a rollback to an earlier snapshot. Microsoft's Compensating Transaction pattern warns that restoring the original state can overwrite concurrent changes; instead, a workflow records how to counter each completed step, and the compensating actions are application-specific. It also recommends making those actions idempotent because the compensating transaction can itself fail and need to resume. These are architectural recommendations, not a guarantee supplied by every tool or service. [Microsoft Azure Architecture Center, Compensating Transaction pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction)    For an agent, a proposed safeguard is to treat compensation as a new authorized operation with its own identity, attempt history, and completion criteria, linked to—but not overwriting—the original record. If the original outcome is still uncertain, an immediate compensating action can create a second unintended effect; the agent should first use contract-supported reconciliation when available. If compensation is interrupted, recovery should preserve evidence about both operations and resume only according to the compensation contract. This is why the `Compensated` state below records a separately verified action rather than erasing the original effect.    … 22 unchanged lines …    - [Idempotency keys](/wiki/idempotency-keys) covers server contracts, scope, retention, and provider comparisons.  - [Verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) covers acceptance criteria and read-back evidence.  - [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) covers durable evidence identity.  - [Exponential backoff with jitter](/wiki/exponential-backoff-with-jitter) addresses retry pacing, which is separate from duplicate-effect safety.  - [Exactly-once delivery](/wiki/exactly-once-delivery) is the broader distributed-systems claim that an idempotency key only partially addresses.