Idempotency and recovery in agent tool calls

Idempotency and recovery in agent tool calls is the practice of resuming or retrying an interrupted tool operation without silently duplicating its external effects or misreporting an uncertain outcome as success or failure. It combines an operation identity, the tool's documented retry contract, durable local state, reconciliation against the target system, and explicit completion verification. This article is an engineering synthesis for digital tool calls; it is not a claim that idempotency keys provide universal exactly-once execution.

A tool-using agent can lose certainty at several boundaries: before a request is sent, after the service receives it but before the client receives a response, while an asynchronous job is running, or after the service completes an action but before the agent records the result. The same timeout can therefore describe materially different states. Retrying may be harmless, may create a duplicate payment or resource, or may return a stored result, depending on the operation and the service contract.

RFC 9110 defines an HTTP method as idempotent when multiple identical requests have the same intended effect on the server as one request. It allows automatic retry of an idempotent request after a communication failure, but says a client must not automatically retry a non-idempotent method unless it knows the request semantics are idempotent or can determine that the original request was never applied. Method-level semantics are only a starting point: a particular POST can have an application-level duplicate-request contract, while a nominally idempotent call can still produce logging or other side effects. ^[1]

Microsoft's Retry pattern gives the corresponding operational warning: if an operation is not idempotent, a service may process the original successfully but fail to return the response, after which a retry can execute the operation again with unintended effects. It also recommends implementing retries only where the full context of the failing operation is understood and testing retry code against failure conditions, load, races, and bottlenecks. ^[2]

Recovery is consequently broader than retry. Retry issues another attempt. Reconciliation asks the target system what happened. Compensation performs a documented action to counter an earlier effect. Abandonment stops automated action and reports uncertainty. Choosing among them requires evidence about the particular operation, not a generic rule that every transient-looking error should be retried.

For a side-effecting operation, a proposed agent-side record should be persisted before dispatch and should survive process restarts. At minimum it can contain:

This table is a proposed implementation pattern, not a standard schema. Secrets and unnecessary personal data should not be placed in such a record. The record's job is to preserve the evidence needed to make the next decision and to prevent a restarted agent from inventing a fresh operation identity for an action that might already have occurred.

A provider-generated resource or job identifier should be retained in addition to the client operation ID. The former supports observation of the remote object; the latter groups attempts made for the same intent. Preserving exact receipts and revision identifiers also connects recovery to provenance and revision-pinned citations: later readers can distinguish the state actually observed from an unversioned resource that may have changed.

A hash of request parameters does not always capture intent. Two identical requests to launch a server may mean “retry the first launch” or “launch a second identical server.” Amazon's Builders' Library therefore recommends a unique caller-provided request identifier; requests from the same caller with the same identifier can be treated as duplicates, while the identifier remains visible in audit logs. The article also treats late-arriving requests and reuse of the same identifier for different intent as distinct design problems. ^[3]

Google's API Improvement Proposal 155 describes a similar contract for request_id: supplying an ID must guarantee idempotency, and a detected duplicate should return the response for the previously successful request. It leaves the retention timeframe to the API and requires format restrictions to be documented. These are Google API design rules, not a universal protocol. ^[4]

The practical rule for an agent is to generate one key per logical operation, persist it before the first attempt, and reuse it only for retries of that same intent. Reusing a key with changed parameters should be treated as a local error even if a provider does not reject it. Conversely, generating a new key merely because a response was lost discards the principal protection against duplicate execution. The provider's documented scope matters: the same token might be unique per account, region, resource, or endpoint rather than globally. See idempotency keys for provider comparisons.

If the agent can establish that no request left the client boundary, it can attempt the operation with the existing operation record and key. That conclusion must come from the tool or transport contract; absence of a response alone is not such evidence. Validation failures that occur before endpoint execution can sometimes be corrected and retried, but only under the provider's documented semantics.

Stripe illustrates why the boundary must be explicit. It stores the status code and response body of the first request for an idempotency key, including 500 responses, once endpoint execution begins. It does not store a result when parameters fail validation or when a conflicting request is already executing. Stripe also permits automatic pruning after keys are at least 24 hours old, after which reusing the key creates a new request. These are Stripe-specific semantics: an agent must not generalize them to another service. ^[5]

When dispatch may have occurred, the safe default is not “retry with a new key.” Prefer, in order permitted by the interface: query a status endpoint by remote operation ID; look up the resource using a stable business identifier; retry with the same idempotency key while it remains within the documented scope and retention window; or stop and report the unresolved state. A read must check the acceptance criteria for the original task, not merely the existence of some similar resource.

The same-key retry can return a cached failure rather than initiate fresh work. Stripe's replay of 500 responses is a concrete example. A repeated response can establish what the idempotency layer recorded, but it does not by itself reveal whether downstream side effects occurred before that failure. Reconciliation may still be necessary, and the applicable API documentation—not the HTTP status code alone—must define what can be concluded.

Overlapping retries can race with the original request. The operation record should enforce at most one active recovery decision for a logical action, even if the provider also deduplicates requests. Provider-side duplicate handling does not prevent two agent workers from making inconsistent local decisions, overwriting evidence, or issuing compensating actions twice.

Late arrival creates a second hazard: an earlier timed-out attempt can reach the service after the agent has started a later recovery path. Amazon's discussion of late-arriving requests is one reason to retain operation identity beyond a single network timeout. A local deadline limits how long the agent waits; it does not prove that a remote request was cancelled. If the service offers cancellation, its acknowledgement and race semantics need their own verification.

Once the provider's deduplication window has expired, the old key may no longer protect a retry. The agent should reconcile before issuing a new operation and should preserve the old record rather than silently replacing it. If authoritative reconciliation is impossible, the honest outcome can be “uncertain; manual review required.” This is preferable to converting lack of evidence into a claim that the first attempt failed.

Preventing duplicate external effects does not by itself make retries operationally safe. Google's Site Reliability Engineering book recommends randomized exponential backoff, a limit on retries per request, and consideration of a service-wide retry budget. It also warns that retries at several layers multiply: if three layers each make four attempts, one user action can produce 64 database attempts. This is operational guidance from Google's production experience, not a universal capacity formula; the exact amplification is the product of the configured attempts along the active call path. ^[7]

A proposed agent policy is to choose one layer to own automatic retries where possible, bound both attempts and elapsed time, randomize delay, and retain the attempt schedule in the durable operation record. For HTTP responses carrying Retry-After, RFC 9110 says the field indicates how long the user agent ought to wait before a follow-up request; with 503 Service Unavailable, it indicates how long the service is expected to remain unavailable. That delay instruction is not evidence that the original side effect failed or that a repeated non-idempotent request is safe. The agent must still preserve the same logical operation identity and apply the service's duplicate-request and reconciliation contract. ^[8]

Backoff, jitter, and retry budgets address synchronized load and cascading failure; idempotency and reconciliation address duplicate or uncertain effects. Neither substitutes for the other. A bounded policy can therefore stop automated retries while leaving the operation in Uncertain, rather than converting exhausted attempts into Known failed. See exponential backoff with jitter for the pacing mechanism.

Compensation is not necessarily a rollback to an earlier snapshot. Microsoft's Compensating Transaction pattern warns that restoring the original state can overwrite concurrent changes; instead, a workflow records how to counter each completed step, and the compensating actions are application-specific. It also recommends making those actions idempotent because the compensating transaction can itself fail and need to resume. These are architectural recommendations, not a guarantee supplied by every tool or service. ^[6]

For an agent, a proposed safeguard is to treat compensation as a new authorized operation with its own identity, attempt history, and completion criteria, linked to—but not overwriting—the original record. If the original outcome is still uncertain, an immediate compensating action can create a second unintended effect; the agent should first use contract-supported reconciliation when available. If compensation is interrupted, recovery should preserve evidence about both operations and resume only according to the compensation contract. This is why the Compensated state below records a separately verified action rather than erasing the original effect.

A compact proposed state machine is:

1. Prepared — persist intent, acceptance criteria, target scope, and operation key.
2. Dispatched — record that the request may have crossed the side-effect boundary.
3. Acknowledged — retain response data and any remote operation or resource identifier.
4. Pending — poll only through the documented status interface and within a bounded policy.
5. Verified — record observations that satisfy the stated completion criteria.
6. Known failed — record evidence that the operation did not take effect or reached a documented terminal failure.
7. Uncertain — preserve conflicting or insufficient evidence; permit only contract-supported reconciliation or escalation.
8. Compensated — record the separate compensating operation and verify its own outcome.

Transitions should be evidence-driven. A process restart does not move Dispatched back to Prepared; a timeout does not move it to Known failed; and issuing a compensating action does not erase the original effect. “Exactly once” should not be used as a report label unless the complete end-to-end protocol supports that claim. Most idempotency-key contracts deduplicate one API boundary and do not control downstream systems, human actions, or effects outside the retention window.

A completion report should name the logical operation ID, the remote identifier when non-sensitive, the last observed state, the verification path, and any remaining uncertainty. It should distinguish “retry suppressed a duplicate under the provider contract” from “the user's end-to-end goal occurred once.” The latter can require checking downstream state and negative constraints.

Useful evaluations inject failures at different boundaries: before send, after server receipt, after side effects but before response, during polling, after key expiry, and during concurrent recovery. Measures should include duplicate external effects, false success reports, false failure reports, unresolved cases, recovery latency, and verification cost. This article does not report such an experiment; the state machine and record fields are proposed synthesis. Results will also depend on the tool adapter, service contract, observation path, and fault model, so success in one simulated API cannot establish universal safety.

• Idempotency keys covers server contracts, scope, retention, and provider comparisons.
• Verifying completion in tool-using agents covers acceptance criteria and read-back evidence.
• Provenance and revision-pinned citations covers durable evidence identity.
• Exponential backoff with jitter addresses retry pacing, which is separate from duplicate-effect safety.
• Exactly-once delivery is the broader distributed-systems claim that an idempotency key only partially addresses.