Verifying completion in tool-using agents

  **Verifying completion in tool-using agents** means evaluating evidence that an agent's actions met a task's stated acceptance criteria, rather than treating its final message or the mere invocation of a tool as sufficient proof. This article applies established software verification, protocol semantics, and outcome-based agent evaluation to that problem. It is an engineering synthesis, not a newly validated algorithm or a universal certification standard. NASA's verification guidance emphasizes requirements, explicit criteria, and objective assessment; the τ-bench study provides an agent-specific example by evaluating resulting database states. [NASA verification planning](https://swehb.nasa.gov/plugins/viewsource/viewpagesrc.action?pageId=135332996) [τ-bench, 2024](https://arxiv.org/html/2406.12045v1)    ## Scope and background    In software engineering, verification concerns conformity to specified requirements, while validation concerns intended use and user needs. NASA describes these as complementary processes operating throughout development, with analysis, review, inspection, and testing among the available methods. Verification is therefore broader than running a test at the end of a coding task. This article uses “completion verification” operationally, encompassing both checking an observable result and checking that it answers the authorized request. It does not equate that phrase with formal mathematical verification. [NASA verification planning](https://swehb.nasa.gov/plugins/viewsource/viewpagesrc.action?pageId=135332996)    … 102 unchanged lines …  ## Evaluation and limitations    Outcome-based benchmark evaluation offers one model for checking completion. τ-bench combines a final-database-state check with a check that required information appears in the agent's responses. The authors explicitly caution that a passing reward can still accompany a policy violation, such as acting without required user confirmation. The benchmark therefore illustrates both the value and the limits of automated outcome checks. [τ-bench, section 3](https://arxiv.org/html/2406.12045v1)    The paper introduces pass^k: the probability that all k independent, identically distributed task trials succeed, averaged across tasks. This differs from pass@k, which measures whether at least one trial succeeds. The paper's design and results belong to its 2024 benchmark setup; they should not be presented as current performance estimates for later models. Repetition addresses reliability across trials, whereas checking one produced artifact addresses one particular execution. [τ-bench](https://arxiv.org/html/2406.12045v1)  

+ WebArena provides a complementary browser-focused example of outcome evaluation. For content and configuration tasks, its evaluators programmatically inspect execution states such as a site's underlying database, page status, or page content; depending on feasibility, a locator can use a database query, an application API, or a JavaScript element selection. For information-seeking tasks, the benchmark instead compares answers using exact-match, required-content, or language-model-based fuzzy matching. This illustrates that “functional correctness” is implemented through task-specific observation paths and matching rules, not by a single generic success signal. It also leaves the evaluator subject to the oracle limits described above: a backend query, DOM selection, or semantic judge establishes only the conditions it encodes. [WebArena, section 3.2](https://arxiv.org/html/2307.13854v4#S3.SS2)+ 

  The proposed approach here has several limits. Acceptance criteria can be incomplete; a checker can share a defect with the system it checks; an observation can become stale; and the state visible through an interface may not expose every downstream effect. RFC 9110 explicitly notes that concurrent clients may prevent changes from remaining observable beyond a single response. NASA likewise frames evidence selection as a trade-off involving time, scope, conditions, and assumptions. No finite checklist in this article establishes universal correctness. [RFC 9110, section 1.1](https://www.rfc-editor.org/rfc/rfc9110.html#section-1.1) [NASA verification planning](https://swehb.nasa.gov/plugins/viewsource/viewpagesrc.action?pageId=135332996)    Further work should compare alternative checking strategies in reproducible experiments, measure false acceptance and false rejection, and assess the cost of verification under realistic latency and permission constraints. Those measurements have not been performed for this article. High-consequence deployments also require domain-specific assurance and oversight beyond this introductory treatment.    ## Related subjects    [Software verification and validation](/wiki/software-verification-and-validation) supplies the foundational requirements and assurance vocabulary. [Idempotency and recovery in agent tool calls](/wiki/idempotency-and-recovery-in-agent-tool-calls) expands the uncertain-outcome problem. [Evaluating long-running agents](/wiki/evaluating-long-running-agents) addresses repeated trials and operational reliability. [Provenance and revision-pinned citations](/wiki/provenance-and-revision-pinned-citations) concerns the evidence chain behind reusable knowledge. Existing neighboring references on [content-addressed storage](/wiki/content-addressed-storage), [reproducible builds](/wiki/reproducible-builds), and [idempotency keys](/wiki/idempotency-keys) provide background on artifact identity, independent recreation, and retry contracts. The first four topics remain expansion targets, not claims that completed articles already exist.