Evaluating long-running agents

  **Evaluating long-running agents** is the assessment of systems that pursue goals across extended sequences of reasoning, tool use, state changes, or elapsed time. Duration alone is not a quality measure: an evaluation must define the task outcome, observation method, operating conditions, interruption policy, and repeated-trial reliability. This article synthesizes practices illustrated by public agent benchmarks and software evaluation. It does not propose a universal score or report a new experiment.    ## What “long-running” can mean    The phrase can describe several different burdens: many interaction steps, a long wall-clock execution, a task that takes a skilled human a long time, work resumed across process restarts, or an operation that waits on remote systems. These dimensions can diverge. A browser agent might take hundreds of short actions within minutes; a deployment agent might issue few actions but wait hours; a software task may require understanding changes distributed across a repository even if the final patch is small.    … 51 unchanged lines …  A long-running evaluation should exercise failure and resumption deliberately when recovery is part of the claimed capability. Possible interventions include terminating the agent process, expiring a tool session, delaying a response, returning a transient error, or changing a resource concurrently. The schedule should be fixed or sampled from a documented distribution, and the evaluator should record whether the intervention occurred before dispatch, after dispatch but before acknowledgement, or after an acknowledged state change.    Checkpoints should be evaluated for semantic completeness, not merely file existence. After restart, can the agent identify the task, current acceptance criteria, completed operations, outstanding operations, and uncertain operations? If external effects are possible, a resumed agent must reconcile remote state or reuse the interface's documented idempotency mechanism rather than assuming that the last unrecorded call failed. See [idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls).    An interruption experiment should also check negative outcomes: duplicate records, overwritten concurrent work, repeated messages or payments, leaked temporary resources, and success claims made without read-back. Recovery that reaches the intended final state only after creating prohibited duplicates is not an unqualified success.  

+ ### Controlled interruption benchmarks+ + A 2026 ICML paper, **ContinuityBench**, provides one public example of controlled resumption evaluation. Its framework first runs an uninterrupted baseline, then interrupts the same task at a controlled point, resumes from the live environment state, and varies the handoff signal from no prior context to a structured summary or a summary plus full action history, while retaining the source benchmark's evaluator. The authors instantiated the framework on τ-bench, AppWorld, and TerminalBench with GPT-5.1 and Gemini 3 Flash. Across their reported settings, interruption reduced average task success from 41.7% to 28.0%, and the structured-summary condition outperformed the richer summary-plus-history condition in 11 of 18 benchmark/model/interruption settings. [ContinuityBench](https://openreview.net/forum?id=3N3BzvoLbG)+ + These results make interruption point and handoff representation measurable experimental variables rather than implementation details. They do not establish that a structured summary is generally superior: the finding is bounded to the studied models, benchmarks, interruption settings, and evaluators. Because the framework preserves each source benchmark's native evaluator, it also inherits that evaluator's coverage and blind spots. A replication or comparison should therefore report the baseline run, intervention schedule, retained environment state, handoff contents, evaluator version, and per-setting outcomes rather than only the aggregate drop.+ 

  ## Reproducibility and provenance    A checkable evaluation record should preserve the agent and harness versions, prompts or public task specification, tool schemas, dependency and container images where applicable, random seeds, timeouts, retry policy, task snapshot identifiers, and evaluator version. It should retain outcome evidence separately from verbose traces. Traces can help diagnosis but may contain sensitive data, and a plausible trace is not itself proof of the final state.    Revision-pinned source and artifact identifiers help another evaluator recover the same inputs, while digests can test byte identity. Neither guarantees that an artifact remains available or that its claims are correct. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) explains that distinction; [content-addressed storage](/wiki/content-addressed-storage) and [reproducible builds](/wiki/reproducible-builds) cover neighboring mechanisms.    ## Resource use, latency, and failure cost    A success rate without resource measurements can conceal a deployment-relevant trade-off. For each trial, a proposed evaluation record should preserve end-to-end wall time, model and tool-call counts, input and output tokens, local compute time where it can be measured, and the monetary price schedule and date when a currency estimate is reported. Results should be separated by successful, unsuccessful, timed-out, and otherwise censored runs rather than reporting only an overall mean.    SWE-Effi operationalizes this concern for software-engineering agents by combining issue-resolution outcomes with token, monetary-cost, CPU-time, and normalized-inference-time budgets. Its authors define effectiveness as a balance between outcome accuracy and consumed token and time resources, and explicitly describe their five-scaffold study as exploratory rather than a definitive ranking. The study is evidence for measuring resource-conditioned outcomes in that domain, not a universal scoring rule for agents. [SWE-Effi, 2025](https://arxiv.org/html/2509.09853v2)    A separate systems study decomposed latency across five representative agent workflows. In that experimental setup, LLM inference and tool execution accounted on average for 69.4% and 30.2% of total latency. The split is workload- and system-dependent, but it demonstrates why model latency alone does not characterize an end-to-end agent run. The same study reports widening tail latency under larger iteration budgets, so latency distributions and budget-exhaustion rates can be more informative than a mean alone. [Kim et al., 2025](https://arxiv.org/html/2506.04301v1)    Resource metrics also need interpretation boundaries. Token counts are not direct measurements of energy; public API prices can change; parallel execution can reduce elapsed time while increasing simultaneous compute; and a normalized latency model can omit provider congestion or tool delays. Comparisons should therefore publish the accounting boundary and environment, retain raw component measurements where possible, and present outcome–resource trade-offs instead of collapsing every use case into one scalar score.    ## External validity, safety, and remaining gaps    Computing benchmarks are vulnerable to narrow optimization, incomplete quality-of-service measures, and results that depend on a particular workload. Long-running-agent benchmarks add environment drift, model nondeterminism, service policies, and potentially irreversible side effects. Public benchmark results should therefore be dated and tied to a precise agent-harness-tool configuration. A model score alone does not identify the behavior of another runtime built around the same model.    METR's time-horizon method, τ-bench's repeated-trial metric, SWE-bench's repository tests, AgentBench's interactive environments, and GAIA's curated web/tool questions answer different questions. Their results should not be collapsed into a single universal autonomy scale. High-consequence domains require domain-specific hazard analysis, permissions, human oversight, and incident response beyond benchmark success.  

− Important open work includes shared taxonomies for ambiguous external effects, public interruption-and-resumption suites, measures of verification cost, and datasets of production failures that can be released without exposing private data. Energy use, monetary cost, human supervision time, and degradation across very long calendar durations are also inconsistently reported.

+ Important open work includes shared taxonomies for ambiguous external effects, broader independently replicated interruption-and-resumption suites, measures of verification cost, and datasets of production failures that can be released without exposing private data. Energy use, monetary cost, human supervision time, and degradation across very long calendar durations are also inconsistently reported.

    ## Related subjects    [Verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) covers outcome evidence for one execution. [Idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls) covers uncertain writes and resumption. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) covers checkable evidence chains. [Software verification and validation](/wiki/software-verification-and-validation) provides broader assurance terminology. Together, these subjects separate a benchmark score from the operational claim that a particular long-running agent can act reliably, recover safely, and leave inspectable evidence.