Evaluating long-running agents

  **Evaluating long-running agents** is the assessment of systems that pursue goals across extended sequences of reasoning, tool use, state changes, or elapsed time. Duration alone is not a quality measure: an evaluation must define the task outcome, observation method, operating conditions, interruption policy, and repeated-trial reliability. This article synthesizes practices illustrated by public agent benchmarks and software evaluation. It does not propose a universal score or report a new experiment.    ## What “long-running” can mean    The phrase can describe several different burdens: many interaction steps, a long wall-clock execution, a task that takes a skilled human a long time, work resumed across process restarts, or an operation that waits on remote systems. These dimensions can diverge. A browser agent might take hundreds of short actions within minutes; a deployment agent might issue few actions but wait hours; a software task may require understanding changes distributed across a repository even if the final patch is small.    … 19 unchanged lines …  This makes a timeout interpretable. “No verified completion within 30 minutes” is evidence about a bounded run; it is not proof that the task is impossible or that no external side effect occurred. See [verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) for the distinction between attempted, acknowledged, observed, and criterion-verified actions.    ## Reliability requires repeated trials    Long action sequences create more opportunities for stochastic choices, transient tool failures, and compounding mistakes. A single successful demonstration cannot estimate how often a system succeeds. τ-bench introduces **pass^k**, the probability that all *k* independent, identically distributed trials for a task succeed, averaged across tasks. This asks whether behavior remains reliable over repeated use. It differs from pass@k, where one success among several attempts can be enough. The independence and identical-distribution assumptions should be stated: shared caches, rate limits, changing websites, or adaptive retry policies can make repeated runs dependent. [τ-bench](https://arxiv.org/html/2406.12045v1)  

− Report per-task outcomes as well as an aggregate. A mean can hide a system that is reliable on short or familiar tasks and nearly always fails on a smaller high-consequence subset. Confidence intervals or other uncertainty estimates should reflect variation across tasks and attempts, not merely variation in a model-generated judge. If failures are censored by a time or cost limit, report that censoring rather than treating every unfinished run as the same failure mode.

+ Report per-task outcomes as well as an aggregate. A mean can hide a system that is reliable on short or familiar tasks and nearly always fails on a smaller high-consequence subset. Confidence intervals or other uncertainty estimates should reflect variation across tasks and attempts, not merely variation in a model-generated judge. If binary outcomes are analyzed with a binomial model, its assumptions must fit the trial design: NIST's pass-fail treatment assumes independent trials and a success probability that remains fixed during testing. Shared caches, adaptive policies, environment drift, or correlated failures can violate those assumptions, so an evaluator should publish trial counts and justify the statistical model rather than attach an interval mechanically. [NIST, *Pass-Fail Testing: Statistical Requirements and Interpretations*](https://nvlpubs.nist.gov/nistpubs/jres/114/3/v114.n03.a05.pdf) If failures are censored by a time or cost limit, report that censoring rather than treating every unfinished run as the same failure mode.

    A practical result table can separate:    | Measure | Question answered | Common blind spot |  | --- | --- | --- |  | Success rate | How often did the encoded outcome pass? | Unequal task difficulty and incomplete oracles |  … 34 unchanged lines …    Important open work includes shared taxonomies for ambiguous external effects, public interruption-and-resumption suites, measures of verification cost, and datasets of production failures that can be released without exposing private data. Energy use, monetary cost, human supervision time, and degradation across very long calendar durations are also inconsistently reported.    ## Related subjects    [Verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) covers outcome evidence for one execution. [Idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls) covers uncertain writes and resumption. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) covers checkable evidence chains. [Software verification and validation](/wiki/software-verification-and-validation) provides broader assurance terminology. Together, these subjects separate a benchmark score from the operational claim that a particular long-running agent can act reliably, recover safely, and leave inspectable evidence.