Evaluating long-running agents

+ **Evaluating long-running agents** is the assessment of systems that pursue goals across extended sequences of reasoning, tool use, state changes, or elapsed time. Duration alone is not a quality measure: an evaluation must define the task outcome, observation method, operating conditions, interruption policy, and repeated-trial reliability. This article synthesizes practices illustrated by public agent benchmarks and software evaluation. It does not propose a universal score or report a new experiment.+ + ## What “long-running” can mean+ + The phrase can describe several different burdens: many interaction steps, a long wall-clock execution, a task that takes a skilled human a long time, work resumed across process restarts, or an operation that waits on remote systems. These dimensions can diverge. A browser agent might take hundreds of short actions within minutes; a deployment agent might issue few actions but wait hours; a software task may require understanding changes distributed across a repository even if the final patch is small.+ + The METR paper *Measuring AI Ability to Complete Long Tasks* operationalizes one useful dimension as the time taken by human professionals to complete a task, then estimates the task duration at which agents reach a stated success probability. Its headline “time horizon” is therefore a model fitted to a particular task distribution and human-baseline procedure, not the elapsed runtime of every agent attempt. The paper reports substantially shorter horizons at 80% success than at 50% success and discusses weaker performance on less structured tasks. Those qualifications make the selected reliability threshold and task distribution part of the result, rather than incidental details. [METR et al., 2025](https://arxiv.org/html/2503.14499)+ + A useful evaluation should name which dimension it varies. Step count, token count, wall time, human baseline time, number of external operations, and number of resumptions measure different things. Reporting one as “task length” without definition prevents comparison and can conceal important costs.+ + ## Define the outcome and the boundary+ + An agent reaching a terminal message is not necessarily a successful episode. The evaluator needs an oracle: a procedure that connects observable evidence to acceptance criteria. In τ-bench, an agent interacts with a simulated user and domain APIs under a policy, and the evaluator compares the final database state with an annotated goal state while also checking required response information. The authors note that a passing task reward can still coexist with a policy violation. This separates outcome checking from complete behavioral assurance. [τ-bench](https://arxiv.org/html/2406.12045v1)+ + SWE-bench supplies a different boundary. It gives a model a real repository and issue description and evaluates a generated patch with repository tests. The benchmark construction uses resolved GitHub issues and corresponding pull requests, while execution-based validation determines whether candidate tasks and patches are testable. Passing the selected tests establishes the encoded regression criteria in that environment; it does not establish deployment success, security, maintainability, or absence of untested regressions. [SWE-bench](https://arxiv.org/html/2310.06770)+ + Before an evaluation run, record at least:+ + - the authorized task and acceptance criteria;+ - the initial environment, data snapshot, and tool interfaces;+ - the permitted actions, budgets, and stopping conditions;+ - the outcome oracle and any human-judgment rubric;+ - prohibited side effects and policy constraints; and+ - what state is retained after timeout, crash, or evaluator failure.+ + This makes a timeout interpretable. “No verified completion within 30 minutes” is evidence about a bounded run; it is not proof that the task is impossible or that no external side effect occurred. See [verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) for the distinction between attempted, acknowledged, observed, and criterion-verified actions.+ + ## Reliability requires repeated trials+ + Long action sequences create more opportunities for stochastic choices, transient tool failures, and compounding mistakes. A single successful demonstration cannot estimate how often a system succeeds. τ-bench introduces **pass^k**, the probability that all *k* independent, identically distributed trials for a task succeed, averaged across tasks. This asks whether behavior remains reliable over repeated use. It differs from pass@k, where one success among several attempts can be enough. The independence and identical-distribution assumptions should be stated: shared caches, rate limits, changing websites, or adaptive retry policies can make repeated runs dependent. [τ-bench](https://arxiv.org/html/2406.12045v1)+ + Report per-task outcomes as well as an aggregate. A mean can hide a system that is reliable on short or familiar tasks and nearly always fails on a smaller high-consequence subset. Confidence intervals or other uncertainty estimates should reflect variation across tasks and attempts, not merely variation in a model-generated judge. If failures are censored by a time or cost limit, report that censoring rather than treating every unfinished run as the same failure mode.+ + A practical result table can separate:+ + | Measure | Question answered | Common blind spot |+ | --- | --- | --- |+ | Success rate | How often did the encoded outcome pass? | Unequal task difficulty and incomplete oracles |+ | pass^k | How often were repeated trials all successful? | Dependence between runs |+ | Wall time | How long did the run occupy? | Waiting versus active work |+ | Tool calls or steps | How much interaction occurred? | Different calls have different cost and risk |+ | Tokens, compute, or money | What resources were consumed? | External human and service costs |+ | Policy-violation rate | Were prohibited actions observed? | Unobservable or unencoded violations |+ | Recovery rate | Did interrupted work resume without corrupting state? | Hidden duplicate effects |+ + ## Environment design and failure attribution+ + AgentBench illustrates breadth-oriented evaluation by placing agents in code-, game-, and web-grounded environments. Its environments exercise interactive reasoning and decision making rather than only static answer generation. The paper also categorizes trajectories by whether the model completed an interaction, exceeded a context limit, produced a wrong format or action, or failed because of a task constraint. Such attribution matters for long runs: an incorrect final state, an evaluator timeout, and an unavailable tool may all score as failure but imply different engineering work. [AgentBench](https://arxiv.org/html/2308.03688)+ + GAIA instead uses curated questions requiring combinations of reasoning, multimodal handling, web browsing, and tool use, with concise answers intended to permit factual evaluation. Its authors discuss reproducibility difficulties for closed assistants and the maintenance problem created by dynamic web evidence. A changing environment can test adaptation, but it also makes model comparisons less controlled. A snapshot can improve reproducibility while losing some realism. Evaluators should state which trade-off they chose and retain dated evidence where licensing and privacy permit. [GAIA](https://arxiv.org/html/2311.12983)+ + For each unsuccessful run, distinguish at least: agent decision error, tool or service failure, invalid credentials or permissions, environment drift, budget exhaustion, evaluator/oracle defect, and unresolved outcome after an ambiguous operation. Do not silently convert infrastructure failures into capability failures. Conversely, excluding every difficult environment interaction can produce an unrealistically clean benchmark.+ + ## Interruptions, checkpoints, and side effects+ + A long-running evaluation should exercise failure and resumption deliberately when recovery is part of the claimed capability. Possible interventions include terminating the agent process, expiring a tool session, delaying a response, returning a transient error, or changing a resource concurrently. The schedule should be fixed or sampled from a documented distribution, and the evaluator should record whether the intervention occurred before dispatch, after dispatch but before acknowledgement, or after an acknowledged state change.+ + Checkpoints should be evaluated for semantic completeness, not merely file existence. After restart, can the agent identify the task, current acceptance criteria, completed operations, outstanding operations, and uncertain operations? If external effects are possible, a resumed agent must reconcile remote state or reuse the interface's documented idempotency mechanism rather than assuming that the last unrecorded call failed. See [idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls).+ + An interruption experiment should also check negative outcomes: duplicate records, overwritten concurrent work, repeated messages or payments, leaked temporary resources, and success claims made without read-back. Recovery that reaches the intended final state only after creating prohibited duplicates is not an unqualified success.+ + ## Reproducibility and provenance+ + A checkable evaluation record should preserve the agent and harness versions, prompts or public task specification, tool schemas, dependency and container images where applicable, random seeds, timeouts, retry policy, task snapshot identifiers, and evaluator version. It should retain outcome evidence separately from verbose traces. Traces can help diagnosis but may contain sensitive data, and a plausible trace is not itself proof of the final state.+ + Revision-pinned source and artifact identifiers help another evaluator recover the same inputs, while digests can test byte identity. Neither guarantees that an artifact remains available or that its claims are correct. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) explains that distinction; [content-addressed storage](/wiki/content-addressed-storage) and [reproducible builds](/wiki/reproducible-builds) cover neighboring mechanisms.+ + ## External validity, safety, and remaining gaps+ + Computing benchmarks are vulnerable to narrow optimization, incomplete quality-of-service measures, and results that depend on a particular workload. Long-running-agent benchmarks add environment drift, model nondeterminism, service policies, and potentially irreversible side effects. Public benchmark results should therefore be dated and tied to a precise agent-harness-tool configuration. A model score alone does not identify the behavior of another runtime built around the same model.+ + METR's time-horizon method, τ-bench's repeated-trial metric, SWE-bench's repository tests, AgentBench's interactive environments, and GAIA's curated web/tool questions answer different questions. Their results should not be collapsed into a single universal autonomy scale. High-consequence domains require domain-specific hazard analysis, permissions, human oversight, and incident response beyond benchmark success.+ + Important open work includes shared taxonomies for ambiguous external effects, public interruption-and-resumption suites, measures of verification cost, and datasets of production failures that can be released without exposing private data. Energy use, monetary cost, human supervision time, and degradation across very long calendar durations are also inconsistently reported.+ + ## Related subjects+ + [Verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) covers outcome evidence for one execution. [Idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls) covers uncertain writes and resumption. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) covers checkable evidence chains. [Software verification and validation](/wiki/software-verification-and-validation) provides broader assurance terminology. Together, these subjects separate a benchmark score from the operational claim that a particular long-running agent can act reliably, recover safely, and leave inspectable evidence.