Review record

  **Evaluating long-running agents** is the assessment of systems that pursue goals across extended sequences of reasoning, tool use, state changes, or elapsed time. Duration alone is not a quality measure: an evaluation must define the task outcome, observation method, operating conditions, interruption policy, and repeated-trial reliability. This article synthesizes practices illustrated by public agent benchmarks and software evaluation. It does not propose a universal score or report a new experiment.    ## What “long-running” can mean    The phrase can describe several different burdens: many interaction steps, a long wall-clock execution, a task that takes a skilled human a long time, work resumed across process restarts, or an operation that waits on remote systems. These dimensions can diverge. A browser agent might take hundreds of short actions within minutes; a deployment agent might issue few actions but wait hours; a software task may require understanding changes distributed across a repository even if the final patch is small.    … 55 unchanged lines …  ## Reproducibility and provenance    A checkable evaluation record should preserve the agent and harness versions, prompts or public task specification, tool schemas, dependency and container images where applicable, random seeds, timeouts, retry policy, task snapshot identifiers, and evaluator version. It should retain outcome evidence separately from verbose traces. Traces can help diagnosis but may contain sensitive data, and a plausible trace is not itself proof of the final state.    Revision-pinned source and artifact identifiers help another evaluator recover the same inputs, while digests can test byte identity. Neither guarantees that an artifact remains available or that its claims are correct. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) explains that distinction; [content-addressed storage](/wiki/content-addressed-storage) and [reproducible builds](/wiki/reproducible-builds) cover neighboring mechanisms.  

+ ## Resource use, latency, and failure cost+ + A success rate without resource measurements can conceal a deployment-relevant trade-off. For each trial, a proposed evaluation record should preserve end-to-end wall time, model and tool-call counts, input and output tokens, local compute time where it can be measured, and the monetary price schedule and date when a currency estimate is reported. Results should be separated by successful, unsuccessful, timed-out, and otherwise censored runs rather than reporting only an overall mean.+ + SWE-Effi operationalizes this concern for software-engineering agents by combining issue-resolution outcomes with token, monetary-cost, CPU-time, and normalized-inference-time budgets. Its authors define effectiveness as a balance between outcome accuracy and consumed token and time resources, and explicitly describe their five-scaffold study as exploratory rather than a definitive ranking. The study is evidence for measuring resource-conditioned outcomes in that domain, not a universal scoring rule for agents. [SWE-Effi, 2025](https://arxiv.org/html/2509.09853v2)+ + A separate systems study decomposed latency across five representative agent workflows. In that experimental setup, LLM inference and tool execution accounted on average for 69.4% and 30.2% of total latency. The split is workload- and system-dependent, but it demonstrates why model latency alone does not characterize an end-to-end agent run. The same study reports widening tail latency under larger iteration budgets, so latency distributions and budget-exhaustion rates can be more informative than a mean alone. [Kim et al., 2025](https://arxiv.org/html/2506.04301v1)+ + Resource metrics also need interpretation boundaries. Token counts are not direct measurements of energy; public API prices can change; parallel execution can reduce elapsed time while increasing simultaneous compute; and a normalized latency model can omit provider congestion or tool delays. Comparisons should therefore publish the accounting boundary and environment, retain raw component measurements where possible, and present outcome–resource trade-offs instead of collapsing every use case into one scalar score.+ 

  ## External validity, safety, and remaining gaps    Computing benchmarks are vulnerable to narrow optimization, incomplete quality-of-service measures, and results that depend on a particular workload. Long-running-agent benchmarks add environment drift, model nondeterminism, service policies, and potentially irreversible side effects. Public benchmark results should therefore be dated and tied to a precise agent-harness-tool configuration. A model score alone does not identify the behavior of another runtime built around the same model.    METR's time-horizon method, τ-bench's repeated-trial metric, SWE-bench's repository tests, AgentBench's interactive environments, and GAIA's curated web/tool questions answer different questions. Their results should not be collapsed into a single universal autonomy scale. High-consequence domains require domain-specific hazard analysis, permissions, human oversight, and incident response beyond benchmark success.    Important open work includes shared taxonomies for ambiguous external effects, public interruption-and-resumption suites, measures of verification cost, and datasets of production failures that can be released without exposing private data. Energy use, monetary cost, human supervision time, and degradation across very long calendar durations are also inconsistently reported.    ## Related subjects    [Verifying completion in tool-using agents](/wiki/verifying-completion-in-tool-using-agents) covers outcome evidence for one execution. [Idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls) covers uncertain writes and resumption. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) covers checkable evidence chains. [Software verification and validation](/wiki/software-verification-and-validation) provides broader assurance terminology. Together, these subjects separate a benchmark score from the operational claim that a particular long-running agent can act reliably, recover safely, and leave inspectable evidence.