Model evaluation benchmark: AhaBench measures three distinct scores

Three numbers instead of one

On June 30, 2026, six researchers deposited on arXiv a model evaluation benchmark^[1] that measures three distinct quantities instead of one.

The suite is called AhaBench and comes from a team of six authors led by Zerui Cheng. The paper contains 37 pages and is under review for TMLR.

The initial question is operational. When a fixed model receives useful experience, does subsequent behavior improve in a related condition where obvious support has been removed, changed, or delayed? The authors respond with a three-item scorecard.

• Initial Score: baseline competence.
• Post-Experience Score: measured outcome after experience.
• Learning Lift: the difference between the two.

This decomposition is the empirical centerpiece of the work. Models that leverage visible support well, those that achieve high final scores, and those that improve most during a session form three separate sets.

Methodology: three components, one logic

The suite contains three components. All three follow the same logic: experience first, then a related test where obvious support is absent.

• Aha-Puzzle: hint-free exploration following already-solved hidden-state puzzles.
• Aha-Euler: Project Euler-style math concepts converted into generated tasks, either taught or held out, with exact validators.
• Aha-Vending: open-source implementation inspired by Vending-Bench, where an agent manages a simulated vending machine amid delayed feedback and operational incidents.

The choice of exact validators on Aha-Euler matters more than it appears. A validator checks the result mechanically and removes the language judge from the equation. The other components use rubrics, thus carrying the bias typical of automated evaluation.

The team released tasks, rubrics, validators, simulator code, and interfaces for evaluating new agents. A third party can therefore replicate the measurement, a rare condition in benchmark literature.

Most current evaluations assign a score to the final state of a single trajectory, then reset the agent. Here the design observes two moments within the same extended session.

The gap between instruction and transfer

The paper's harshest result comes from Aha-Euler.

With full instruction, scores range between 78.6% and 100.0%. With final-answer-based transfer, the range collapses: from 0.0% to 73.9%.

The gap between 100.0% and 0.0% is where learning capacity lives. A shown answer delivers the result; full instruction delivers the procedure. Models receiving the procedure generalize to the held-out task; those receiving the final digit sometimes remain at zero.

The same pattern appears on Aha-Puzzle. Puzzle traces raise scores in the supported condition, then often fail to transition to autonomous exploratory behavior.

Visible support inflates the competence measure and masks learning deficits. A high score obtained with support thus becomes a weak indicator of operational performance.

The eight-model panel

The common panel covers eight models. Claude Opus 4.6 leads the aggregated Post-Experience Score with 64.3 and the aggregated Learning Lift with +25.8.

Gemini 3.1 Pro follows closely at 63.4 on the first metric. The distance between 64.3 and 63.4 is 0.9 points on an aggregated scale, carrying little decisive weight.

The interesting anomaly lies elsewhere. Leading the final-outcome ranking and leading the improvement ranking are two separate properties, and the paper measures them separately for exactly this reason. A reader seeing only a single aggregated figure loses the information they need.

The aggregated value emerges from three very different components. An average across puzzles, mathematics, and vending-machine management compresses dimensions that respond to distinct capacities.

The authors disclose the structure and publish results by component, so the decomposition remains accessible to readers of the full document.

Aha-Vending: feedback that arrives late

Aha-Vending separates three outcomes: profitable incident management, bankruptcy, failure to place orders.

Bankruptcy and order failure describe two different ways to break. The first spends poorly; the second remains idle. Both yield zero profit, so a single outcome metric conflates them.

Long horizon is the problem's engine. A multi-step agent's errors compound: a 90% success rate per step, repeated five times, drops to 59% on the full task.

The counter-argument is legitimate, because a simulator remains a simulator. The design nonetheless introduces delayed feedback and incidents, two features that single-turn tests ignore entirely.

The useful data here is qualitative, and the authors present it that way. Separating outcomes matters more than averaging them, because it signals which failure mode awaits the operator.

Why rankings measure the wrong thing

A benchmark measures standardized conditions. Production is the opposite of standardization, so the predictive validity of a score remains a separate dimension, largely unmeasured still.

The incentive to publish scores optimized for ranking has produced literature measuring what is easy to measure. This suite moves the opposite direction: it chooses a difficult variable, the gain between two moments, and exposes it. The price of that choice is number fragility, which depends on how experience is administered.

The document is deposited as version 1 and under review for TMLR, so peer review is still pending. Those using these numbers for a decision must account for editorial status.

The authors limit conclusions to their tasks, and the reader does well to do the same. Extending the result to a different application domain remains an operation lacking empirical support.

What changes for capital allocators

The three-score decomposition changes the question an investment committee poses to a vendor.

For CRO, CDO, and investment committee, the point is R&D budget allocation. A score obtained with visible support describes a laboratory condition. The figure that matters for deployment is the outcome after support removal.

For the Chief Analytics Officer, the consequence concerns data infrastructure. Measuring internal Learning Lift requires two capacities absent from many current pipelines: tracking agent experience and repeating the test in a modified condition.

For the board, the question is which technology thesis holds. Evidence shows that initial competence, final outcome, and capacity to improve move independently across the eight models in the panel. A thesis built on a single ranking thus rests on narrow ground.

This is a diagnosis of the measurement gap, and the operational choice remains in the hands of whoever signs the budget.