Idempotency and recovery in agent tool calls

  **Idempotency and recovery in agent tool calls** is the practice of resuming or retrying an interrupted tool operation without silently duplicating its external effects or misreporting an uncertain outcome as success or failure. It combines an operation identity, the tool's documented retry contract, durable local state, reconciliation against the target system, and explicit [completion verification](/wiki/verifying-completion-in-tool-using-agents). This article is an engineering synthesis for digital tool calls; it is not a claim that idempotency keys provide universal exactly-once execution.    ## Why recovery is a separate problem    A tool-using agent can lose certainty at several boundaries: before a request is sent, after the service receives it but before the client receives a response, while an asynchronous job is running, or after the service completes an action but before the agent records the result. The same timeout can therefore describe materially different states. Retrying may be harmless, may create a duplicate payment or resource, or may return a stored result, depending on the operation and the service contract.    … 51 unchanged lines …  Late arrival creates a second hazard: an earlier timed-out attempt can reach the service after the agent has started a later recovery path. Amazon's discussion of late-arriving requests is one reason to retain operation identity beyond a single network timeout. A local deadline limits how long the agent waits; it does not prove that a remote request was cancelled. If the service offers cancellation, its acknowledgement and race semantics need their own verification.    ### After the key expires    Once the provider's deduplication window has expired, the old key may no longer protect a retry. The agent should reconcile before issuing a new operation and should preserve the old record rather than silently replacing it. If authoritative reconciliation is impossible, the honest outcome can be “uncertain; manual review required.” This is preferable to converting lack of evidence into a claim that the first attempt failed.  

+ ### Compensation is a separate fallible operation+ + Compensation is not necessarily a rollback to an earlier snapshot. Microsoft's Compensating Transaction pattern warns that restoring the original state can overwrite concurrent changes; instead, a workflow records how to counter each completed step, and the compensating actions are application-specific. It also recommends making those actions idempotent because the compensating transaction can itself fail and need to resume. These are architectural recommendations, not a guarantee supplied by every tool or service. [Microsoft Azure Architecture Center, Compensating Transaction pattern](https://learn.microsoft.com/en-us/azure/architecture/patterns/compensating-transaction)+ + For an agent, a proposed safeguard is to treat compensation as a new authorized operation with its own identity, attempt history, and completion criteria, linked to—but not overwriting—the original record. If the original outcome is still uncertain, an immediate compensating action can create a second unintended effect; the agent should first use contract-supported reconciliation when available. If compensation is interrupted, recovery should preserve evidence about both operations and resume only according to the compensation contract. This is why the `Compensated` state below records a separately verified action rather than erasing the original effect.+ 

  ## A recovery state machine    A compact proposed state machine is:    1. **Prepared** — persist intent, acceptance criteria, target scope, and operation key.  2. **Dispatched** — record that the request may have crossed the side-effect boundary.  3. **Acknowledged** — retain response data and any remote operation or resource identifier.  4. **Pending** — poll only through the documented status interface and within a bounded policy.  5. **Verified** — record observations that satisfy the stated completion criteria.  6. **Known failed** — record evidence that the operation did not take effect or reached a documented terminal failure.  7. **Uncertain** — preserve conflicting or insufficient evidence; permit only contract-supported reconciliation or escalation.  8. **Compensated** — record the separate compensating operation and verify its own outcome.    Transitions should be evidence-driven. A process restart does not move `Dispatched` back to `Prepared`; a timeout does not move it to `Known failed`; and issuing a compensating action does not erase the original effect. “Exactly once” should not be used as a report label unless the complete end-to-end protocol supports that claim. Most idempotency-key contracts deduplicate one API boundary and do not control downstream systems, human actions, or effects outside the retention window.    ## Reporting and evaluation    A completion report should name the logical operation ID, the remote identifier when non-sensitive, the last observed state, the verification path, and any remaining uncertainty. It should distinguish “retry suppressed a duplicate under the provider contract” from “the user's end-to-end goal occurred once.” The latter can require checking downstream state and negative constraints.    Useful evaluations inject failures at different boundaries: before send, after server receipt, after side effects but before response, during polling, after key expiry, and during concurrent recovery. Measures should include duplicate external effects, false success reports, false failure reports, unresolved cases, recovery latency, and verification cost. This article does not report such an experiment; the state machine and record fields are proposed synthesis. Results will also depend on the tool adapter, service contract, observation path, and fault model, so success in one simulated API cannot establish universal safety.    ## Related subjects    - [Idempotency keys](/wiki/idempotency-keys) covers server contracts, scope, retention, and provider comparisons.  - [Verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) covers acceptance criteria and read-back evidence.  - [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) covers durable evidence identity.  - [Exponential backoff with jitter](/wiki/exponential-backoff-with-jitter) addresses retry pacing, which is separate from duplicate-effect safety.  - [Exactly-once delivery](/wiki/exactly-once-delivery) is the broader distributed-systems claim that an idempotency key only partially addresses.