Carnegie Mellon University / School of Computer Science
AgentProofBenchReusable Verifiers for Web Agents
One rule. Changing requirements.
Visible consequences.
A benchmark that follows a verifier from the outcomes it accepts, through the requirements it must recover, to the work an agent delivers.
The name Backend resolves to team A. The correct result and the frozen check agree.
Schematic A/B identities illustrate the Linear45 binding change.
79 families · 4 Web services
48 families · 558 terminal states
22 families · 44 assignments · 88 openings
Tests linked to delivered work
Correct panels align
with correct delivery.
Deployment-state panel correctness separates recorded program assignments whose two saved openings are both delivered correctly.
Error-free panel
33 assignments32/33
Both openings delivered correctly
Panel contains errors
11 assignments1/11
Both openings delivered correctly
44 program assignments from 22 families. The exploratory difference has a 95% family-cluster interval of [68.8, 100.0] percentage points. Natural endings and checkpoints remain paired within each assignment.
Follow the evidence
What makes a verifier reusable?
Every stage asks a different question. Shared tasks, programs, and saved states connect the answers.
A rule’s operating profile
Which outcomes
cross the boundary?
Acceptance needs two kinds of evidence: rejected failures and accepted valid solutions.
On matched collection outcomes, generated programs sharply reduce false acceptance. The paired valid-anchor test makes their selectivity visible.
Same outcomes, same reference
99 failed states + 36 valid anchors
36 tasks · 29 families · Flash
Failures accepted
Lower is better28.2 percentage points lower false acceptance on matched collection failures.
3.5%vs31.7%
77.8%vs98.6%
Program / online. Two observations per method and state. These collection-specific measures describe different operating choices.
Freeze the rule. Change the obligation.
A familiar instruction.
A different correct answer.
Names can refer to new identities. Collections can gain required items. Values that must be preserved can change. Joint tests ask whether a frozen verifier follows all tested requirements.
Every stage keeps the full 96-program denominator.
Source → related: 83 remain fully correct; 6 acquire a panel error.
The witness adds a distinct test.
Replay a source-valid decision after its required target changes. The complete related panel and replay verdict must both be correct.
95% family-bootstrap interval
82/96 Flash and 79/96 Pro source programs pass both the complete related panel and its replay witness. Related rebuilds pass in 82/96 and 86/96 programs.
See paired transferFrom a test result to delivered work
The decision becomes
part of the environment.
A gate can release a result, ask the agent to continue, or withhold delivery. Saved histories let those policies start from the same state and history.
Withheld deliveries
All in the panel-error group; 13 terminal states satisfy the reference.
Observed regressions
All in the panel-error group; completed work changes from reference success to failure.
Neutral continuation attains the highest correct delivery.
20/22 Flash and 22/22 Pro openings in each cohort, under the tested generic feedback.
Same correct opening.
Different subsequent paths.
Neutral continuation preserves the correct assignment to the current Backend team.
The gate sends generic nonacceptance feedback with no team identifier or program-specific check. A/B are schematic identities; the event sequence comes from the saved Linear45 trajectory.
Inspect the evidence
The complete picture,
one figure at a time.
Explore the manuscript’s vector figures. Open any figure to zoom, or use the arrow keys to browse.
All figures are scalable SVGs. Values and experimental units follow the paper.
The paper
From a changed requirement
to a tested rule.
Then to its consequences.
AgentProofBench: Benchmarking Reusable Verifiers for Web Agents
Paired P1/P2 result files and a lightweight script to reproduce the headline counts.
@misc{ning2026agentproofbench,
title = {AgentProofBench: Benchmarking Reusable Verifiers for Web Agents},
author = {Ning, Jingjie and Li, Xueqi and Lu, Yi and Kong, Yibo},
year = {2026},
howpublished = {Preprint},
url = {https://ethanning.github.io/AgentProofBench/}
}Reading the measurements
The empirical comparisons use DeepSeek Flash and Pro with thinking disabled. Each reuse family contributes one related transformation and one replay witness. The panel–delivery association is an exploratory analysis of 44 assignments clustered within 22 families.
Scores follow the explicit reference predicates. A separate GPT-6 Astra LLM-as-a-judge audit reviews 195 states across 62 tasks, agreeing with 191 of 192 resolved labels (99.5%); three cases remain indeterminate. It corroborates all 48 requirement changes, all 48 failed replays, and both delivered-regression pairs. Inspect the audit. The paper preserves original labels, post-inspection exclusions, and the exact feedback policies.