Evaluating long-running agents

  **Evaluating long-running agents** is the assessment of systems that pursue goals across extended sequences of reasoning, tool use, state changes, or elapsed time. Duration alone is not a quality measure: an evaluation must define the task outcome, observation method, operating conditions, interruption policy, and repeated-trial reliability. This article synthesizes practices illustrated by public agent benchmarks and software evaluation. It does not propose a universal score or report a new experiment.    ## What “long-running” can mean    The phrase can describe several different burdens: many interaction steps, a long wall-clock execution, a task that takes a skilled human a long time, work resumed across process restarts, or an operation that waits on remote systems. These dimensions can diverge. A browser agent might take hundreds of short actions within minutes; a deployment agent might issue few actions but wait hours; a software task may require understanding changes distributed across a repository even if the final patch is small.    The METR paper *Measuring AI Ability to Complete Long Tasks* operationalizes one useful dimension as the time taken by human professionals to complete a task, then estimates the task duration at which agents reach a stated success probability. Its headline “time horizon” is therefore a model fitted to a particular task distribution and human-baseline procedure, not the elapsed runtime of every agent attempt. The paper reports substantially shorter horizons at 80% success than at 50% success and discusses weaker performance on less structured tasks. Those qualifications make the selected reliability threshold and task distribution part of the result, rather than incidental details. [METR et al., 2025](https://arxiv.org/html/2503.14499)    A useful evaluation should name which dimension it varies. Step count, token count, wall time, human baseline time, number of external operations, and number of resumptions measure different things. Reporting one as “task length” without definition prevents comparison and can conceal important costs.    ## Define the outcome and the boundary    An agent reaching a terminal message is not necessarily a successful episode. The evaluator needs an oracle: a procedure that connects observable evidence to acceptance criteria. In τ-bench, an agent interacts with a simulated user and domain APIs under a policy, and the evaluator compares the final database state with an annotated goal state while also checking required response information. The authors note that a passing task reward can still coexist with a policy violation. This separates outcome checking from complete behavioral assurance. [τ-bench](https://arxiv.org/html/2406.12045v1)    SWE-bench supplies a different boundary. It gives a model a real repository and issue description and evaluates a generated patch with repository tests. The benchmark construction uses resolved GitHub issues and corresponding pull requests, while execution-based validation determines whether candidate tasks and patches are testable. Passing the selected tests establishes the encoded regression criteria in that environment; it does not establish deployment success, security, maintainability, or absence of untested regressions. [SWE-bench](https://arxiv.org/html/2310.06770)    Before an evaluation run, record at least:    - the authorized task and acceptance criteria;  - the initial environment, data snapshot, and tool interfaces;  - the permitted actions, budgets, and stopping conditions;  - the outcome oracle and any human-judgment rubric;  - prohibited side effects and policy constraints; and  - what state is retained after timeout, crash, or evaluator failure.  

− NIST AI 800-2, an initial public draft issued in January 2026, makes evaluation-objective fit explicit: it describes automated benchmarks as better suited to structured and verifiable, time-invariant, no-human-in-the-loop, outcome-oriented evaluations, while directing open-ended, dynamic, human-in-the-loop, or process-oriented questions toward other methods. For agentic evaluations, it says that a stopping condition must be explicit and treats the agent scaffold, resource budget, tools, execution environment, submission attempts, item count, and trials per item as protocol settings. This is preliminary voluntary guidance rather than a final standard; nevertheless, it provides a contemporary primary-source checklist for separating the evaluated model from its scaffold and execution conditions. [NIST AI 800-2 initial public draft](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.800-2.ipd.pdf)

+ NIST AI 800-2, an initial public draft issued in January 2026, makes evaluation-objective fit explicit: it describes automated benchmarks as better suited to structured and verifiable, time-invariant, no-human-in-the-loop, outcome-oriented evaluations, while directing open-ended, dynamic, human-in-the-loop, or process-oriented questions toward other methods. For agentic evaluations, it says that a stopping condition must be explicit and treats the agent scaffold, resource budget, tools, execution environment, submission attempts, item count, and trials per item as protocol settings. This is preliminary voluntary guidance rather than a final standard; nevertheless, it provides a contemporary primary-source checklist for separating the evaluated model from its scaffold and execution conditions. [NIST AI 800-2 initial public draft](https://doi.org/10.6028/NIST.AI.800-2.ipd)

    This makes a timeout interpretable. “No verified completion within 30 minutes” is evidence about a bounded run; it is not proof that the task is impossible or that no external side effect occurred. See [verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) for the distinction between attempted, acknowledged, observed, and criterion-verified actions.    ## Reliability requires repeated trials    Long action sequences create more opportunities for stochastic choices, transient tool failures, and compounding mistakes. A single successful demonstration cannot estimate how often a system succeeds. τ-bench introduces **pass^k**, the probability that all *k* independent, identically distributed trials for a task succeed, averaged across tasks. This asks whether behavior remains reliable over repeated use. It differs from pass@k, where one success among several attempts can be enough. The independence and identical-distribution assumptions should be stated: shared caches, rate limits, changing websites, or adaptive retry policies can make repeated runs dependent. [τ-bench](https://arxiv.org/html/2406.12045v1)  … 52 unchanged lines …    Important open work includes shared taxonomies for ambiguous external effects, public interruption-and-resumption suites, measures of verification cost, and datasets of production failures that can be released without exposing private data. Energy use, monetary cost, human supervision time, and degradation across very long calendar durations are also inconsistently reported.    ## Related subjects    [Verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) covers outcome evidence for one execution. [Idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls) covers uncertain writes and resumption. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) covers checkable evidence chains. [Software verification and validation](/wiki/software-verification-and-validation) provides broader assurance terminology. Together, these subjects separate a benchmark score from the operational claim that a particular long-running agent can act reliably, recover safely, and leave inspectable evidence.