WhatWorkedBenchGitHub

A BENCHMARK FOR EXPERIMENTAL UNDERSTANDING

Run experiments.
Recover
understanding.

Every experiment reveals a result.
What does the agent learn about why it works?

Measure how accurately AI agents predict the effects of component changes after a limited number of experiments.

Explore a real task ↗Get the benchmark ↗

Executable workflows · Complete references · Open evidence

THE EXPERIMENT LANDSCAPE16 configurations
ONE CONFIGURATIONA useful result
→
THE COMPLETE TABLEA richer understanding
Actual SciFact utilities · darker means higher
36task conditions
30source instances
8workflow families
1,248native outcomes

01 / THE PROTOCOL

From observations to a complete prediction

Measure what
experiments teach.

An agent studies a workflow, chooses configurations to measure, and submits predictions for every possible configuration. Exhaustive CPU execution supplies the ground truth.

01

Read the workflow

Inspect executable code and four or six binary component changes.

Public task + two free anchors
02

Spend a fixed budget

Select experiments. Observe scores. Adapt the next measurement.

B additional measurements
03

Predict the surface

Submit a complete table covering all 16 or 64 configurations.

Measured scores stay fixed
04

Recover the effects

Evaluate component effects, interactions, and configuration choice.

Complete native reference
View the full benchmark overview ＋WhatWorkedBench overview with task, experiment, prediction, evaluation, CPU reference, and shared inference blocks.

02 / INSIDE A REAL TASK

Interactive · Recorded native outcomes

A change has
more than one effect.

Explore the SciFact retrieval task. Select a component, change its background, and see how its measured effect changes. Every score below comes from the released reference.

COMPONENT TO STUDYSCIFACT · 4 OPTIONS

HOLD THE OTHER OPTIONS FIXED

The selected component changes from off to on. All other options define the background.

CONDITIONAL COMPONENT EFFECT
—component off
→
—component on
Change in NDCG@10—

Eight backgrounds, eight conditional effects. Highlighted bar shows your selection.

NDCG@10 measures retrieval ranking quality. Reference scores use the fixed 64-query SciFact evaluation cohort. Inspect the task ↗

03 / THE EVIDENCE

4,206 numerical controls · 108 core agent episodes

Better use of evidence.
Better understanding.

Shared inference reconstructs the response surface from an agent's existing observations. Code-aware modeling also uses visible equivalences between configurations.

ORIGINAL FLASH COHORT · B=8

The same observations.
A stronger reconstruction.

Agent table
0.632
Shared GP
0.698

Effect recovery increases when a Gaussian process reconstructs the agent's acquired evidence.

ADDITIONAL FLASH COHORT · B=8

The pattern persists
in additional runs.

Agent table
0.621
Shared GP
0.720

Repeated episodes retain the separation between collecting observations and using them effectively.

PAIRED SIX-OPTION TASKS · B=20

Program structure
makes a difference.

Pair ridge
0.248
Code-aware GP
0.462

This comparison changes acquisition and inference using equivalences derived from visible code.

BEAT + GRAPH WORKFLOWS · B=320.303 0.455Six valid submitted tables · same-observation GP

Three beat-detection and three graph-link-prediction sources. Effect recovery improves for all six valid submissions.

Structure diagnostic0.338 → 0.507Eight complete episodes show how a post-hoc code-equivalence projection improves effect recovery with the acquired observations held fixed.

Higher effect recovery indicates more accurate conditional-effect predictions. Cohorts retain their original identities and comparison units. Inspect complete results ↗

04 / THE TASK COLLECTION

Native CPU workflows · Four or six options

Eight workflows.
One evaluation language.

From retrieval and regression to signal processing and graph prediction, every task exposes executable component choices and complete outcome references.

36 conditions on 30 sources. Six paired variants extend existing tasks to six options; 96 configuration records repeat original subcube outcomes. Source attribution ↗

05 / GET STARTED

Python 3.11+ · Linux · Offline numerical evaluation

Start with one task.
Bring your own agent.

Run a numerical baseline, replay native outcomes, or connect your agent to the budgeted evaluator.

Read the integration guide ↗
Fixed budgetsComplete effectsReproducible records
Run from a fresh clone · Full commands included

THE PAPER

WhatWorkedBench

Benchmarking Experimental
Understanding in AI Agents

arXiv · Coming soon

The preprint link and finalized paper citation will appear here.

SOFTWARE CITATION
@misc{whatworkedbench2026,
  title = {WhatWorkedBench: Benchmarking
    Experimental Understanding in AI Agents},
  year = {2026},
  howpublished = {Software repository},
  url = {https://github.com/EthanNing/WhatWorkedBench}
}