Text-to-SQL: Evaluation Metrics Reject 25% of Valid Queries

The measured bottleneck sits in the judge

On 17 September 2026 five researchers filed a study on arXiv that shifts the Text-to-SQL problem away from the query generator and onto the tool that judges it.

Tarfah Alrashed, Fatma Ozcan, Per Jacobsson, Tal Neiman and Xianshun Chen examined the way we measure SQL queries enriched with AI operators. The paper's evidence^[1] shows that the standard metric, Execution Accuracy, recognises as correct just 25% of valid translations.

Three quarters of the correct work ends up in the error column. The generator produces; the judge fails it. The distance between that 25% and 100% is the space in which the problem lives.

Why a deterministic metric breaks on AI operators

Execution Accuracy compares the result of the generated query with the result of the reference query. Identical outputs score a point, different outputs score zero.

The rule holds as long as the database responds deterministically.

AI operators inside SQL change the playing field. An operator that classifies a text, summarises a field or extracts an entity generates flexible, stochastic output: two correct executions of the same query return different strings. Exact comparison reads that difference as an error.

The authors formalise these failure modes as properties of the metric, rather than as occasional noise. The distinction matters: a systematic flaw recurs at every benchmark cycle, with the same sign.

The LLM judge inherits the same blind spot

The natural response is to replace exact comparison with a model that evaluates meaning. The paper measures this route too.

A state-of-the-art LLM autorater wrongly rejects 32% of accurate queries. Almost a third of the correct work is turned down by an automated judge.

The cause stated by the authors lies in the complexity of the task. The model evaluates relational logic and the AI component together, two dimensions with different criteria for truth. A single verdict across two dimensions degrades the quality of the verdict.

The figure reinforces a position this desk has already documented: automated evaluation remains a tool with systematic bias, far from the arbiter the market is selling.

The layered framework separates logic and semantics

The work's central contribution is a Multilayered Evaluation Framework. The architecture splits the deterministic logic of the database from the flexible semantics of AI operators.

Validation proceeds along two distinct tracks. The relational part is verified with the classic criteria of execution; AI operations are judged with criteria calibrated to their variability. The two outcomes then converge into an overall judgement.

On this basis the framework reaches up to 97.2% overall accuracy. The jump from 25% to 97.2% quantifies how much the measuring instrument weighed, with the generated queries held constant.

The full description of the two layers is in the complete text, also readable on alphaXiv^[2].

Two platforms, industry and academia

The test covers two systems with different histories: BigQuery, an industrial platform, and ThalamusDB, an academic system. The choice answers a classic objection about external validity.

An evaluation framework tuned on a single engine risks measuring the conventions of that engine. Two platforms with different dialects and implementations raise the bar of proof. The stated result covers both.

The sample nonetheless remains confined to two systems and one family of tasks. The authors declare the perimeter; the correct reading stops inside that perimeter.

It is worth noting what separates the two environments. An industrial engine brings optimisations, proprietary dialects and cost constraints that a research system ignores, and a framework that holds on both has crossed two distinct technical regimes.

What changes for the historic benchmarks

The Text-to-SQL field was built on deterministic tests. Spider, a large-scale dataset with human labels for cross-domain semantic parsing, dates back to 2018 (doi.org/10.18653/v1/D18-1425^[3]).

BIRD, from 2023, moved the test onto large databases, asking how far an LLM can already work as an interface to the database (doi.org/10.48550/arXiv.2305.03111^[4]). Both assume a stable expected result.

With AI operators that assumption collapses. A leaderboard produced by a metric that fails three quarters of valid translations ranks systems by a distorted criterion. The ranking remains readable as a measure of conformity to the expected format, far from a measure of correctness.

The limits declared by the authors

The most serious objection to the work concerns the breadth of the test. The numbers available in the public abstract describe two platforms and a bounded set of tasks.

The 25% is given as the minimum observed value, the 97.2% as the ceiling reached. Reading these extremes as general averages goes beyond what the authors claim. The rigorous reading holds the two numbers as the bounds of the measured range.

Then there is the open question of predictive validity in production. A framework that evaluates better under controlled conditions still faces the test of real load, dirty data and governance.

That dimension remains, for now, largely unmeasured.

Where budget allocation shifts

For an investment committee the finding reorients a line of spending. The budget for query generation now competes with the budget for evaluation infrastructure.

A Text-to-SQL system shelved after a disappointing pilot may have produced correct queries rejected by the wrong yardstick. The 25% recognition rate makes that hypothesis testable, rather than theoretical. Re-examining the rejects costs less than a fresh vendor selection cycle.

For a Chief Analytics Officer the operational consequence concerns traceability. Evaluating relational logic and AI operations separately requires logs that preserve both components, with their outcomes kept distinct.

For the board, the thesis to revise is the one that ties Text-to-SQL progress to benchmark scores. The evidence filed on 17 September 2026 places a substantial part of the gain inside the measuring instrument, rather than inside the model.