SearchHorizonBench Read the paper

A BENCHMARK FOR SEARCH VALUE IN AI AGENTS

What is
one more search
worth?

Turn execution experience into a sharper test of when an AI agent should search.

Measure answer success, the gain from further search, and the value of the final decision.

Explore the evidence Get the benchmark
Carnegie Mellon University
School of Computer Science
ONE QUESTION. TWO CONTINUATIONS.Recorded example
A paired search-value evaluationThe same question forks into answering from existing evidence or searching further. One of four THINK executions and all four SEARCH executions answer correctly in this selected example. SAVED QUESTION + HISTORY Leonard Nathan's birthplaceis in what county? THINKUse existing evidence1 / 4Correct continuations SEARCHRetrieve more evidence4 / 4Correct continuations Measure the differenceRepeated outcomes · shared starting point
Filled dots mark correct answersOne selected illustration
45,632agent executions
8,880value forecasts
3,168distinct questions
3evaluation targets

01 / THE BENCHMARK

Shared histories. Repeated outcomes. Explicit costs.

Three questions.
One complete evaluation.

A useful search decision starts with knowing what the next action will change. SearchHorizonBench follows that information from prediction to decision.

01

Predict success

How likely is each continuation to produce a correct answer?

Action probabilities · Brier error
02

Estimate the gain

How much does another search improve expected answer quality?

Search-value error · Gain R²
03

Choose a continuation

Does the expected gain justify the cost of obtaining more information?

Reward minus cost · Decision utility
See the complete benchmark overview The benchmark's full paired-execution, value-estimation and decision-evaluation pipeline.

02 / THE CENTRAL FINDING

Success prediction and search-value prediction carry different information

Calibrated confidence.
A separate test of value.

Execution history improves forecasts of answer success. Paired evaluation reveals how much of that improvement reaches the estimated benefit of searching.

ANSWER-SUCCESS PREDICTION

16.2% lower Brier error

Historical correction compared with global action means on Test-1632.

Global action means
0.2145
Historical correction
0.1797
99.9%of this Brier improvement comes from success-level prediction.
MARGINAL SEARCH VALUE
21.4pp
standard deviation of
expected search gains

Schematic of question-specific variation

Tested historical estimators explain less than 2% of gain variance in point estimates.

88.9%

of the observed Brier improvement is recovered by simple task-category priors. The benchmark measures task difficulty and additional search value separately.

Read the analysis
Inspect within-task evidence and uncertainty Within-task Brier differences and confidence intervals across five question strata.

03 / EXPLORE THE DECISION

Interactive · Recorded measurements

Put a price
on the next search.

Hold one kind of information fixed and vary the other. See how knowledge of search gains and token costs changes decision utility.

TOKEN VALUATION λ

Value each 1,000 tokens.

Utility equals answer correctness minus λ times thousands of tokens.

At 0.005, each 1,000 tokens costs 0.5 utility percentage points.
BEST CONSTANT ACTIONSEARCH
UTILITY ABOVE THE BEST CONSTANT

What does the decision know?

95% intervals
+2.49pp

with question-specific
gain and cost information

Information-value diagnostic. Each rule uses two outcome draws per action for selection and two disjoint draws for evaluation. Selection observations are free. These values quantify a sample-informed opportunity; deployable methods are separately charged for acquiring their signals. All 36 splits use Test-1632 and reference-equivalent correctness. View the measurements ↗

04 / THE EXECUTION RESOURCE

Common metrics across multiple settings

One evaluation language.
Many search settings.

Saved executions connect model families, retrieval services, budget choices and answer rewards through the same success, gain and decision measurements.

ACTORS & FORECASTERS

DeepSeek Flash · Pro · Qwen

RETRIEVAL

DeepSeek · Serper · Tavily

INDEXED WEB CORPORA

ClueWeb22 · FineWeb

TASKS & REWARDS

Multi-hop QA · Natural Questions · Set completion

View the execution inventory The nine experimental settings contributing 45,632 unique executions, with source and target outcome summaries.

05 / GET STARTED

Bring a predictor. Measure its value.

Make your next
comparison count.

Use the saved execution records to evaluate a new search-value estimator. Compare success probabilities, noise-corrected gains and decision utility against matched historical baselines.

Download code & data
Paired outcomesReusable scoringRecorded costs
START FROM THE RELEASE
unzip SearchHorizonBench.zip
cd SearchHorizonBench
python -m pip install -r requirements.txt
python scripts/replay_examples.py
Three offline examples replay forecast, policy and collection scores.

THE PAPER

SearchHorizonBench

A Benchmark for Search Value in AI Agents

Jingjie Ning*Xueqi LiYibo Kong

Carnegie Mellon University
* Corresponding author

Read the manuscript
CITE THIS WORK
@misc{ning2026searchhorizonbench,
  title = {SearchHorizonBench: A Benchmark
    for Search Value in AI Agents},
  author = {Ning, Jingjie and Li, Xueqi
    and Kong, Yibo},
  year = {2026},
  note = {Manuscript and benchmark},
  url = {https://ethanning.github.io/SearchHorizonBench/}
}