Multi-step reasoning: medical consistency falls from 98.88% to 45.13%

The gap between 98.88% and 45.13%

A model evaluation benchmark published on September 10, 2026 places two numbers side by side. On atomic tasks the best model achieves 98.88% logical coherence. On the full chain, the same model drops to 45.13%.

The measurement comes from LogiMed-RoB^[1], built on 860 randomized clinical trials and 14,820 queries, tested on ten leading models and accepted at the EMNLP 2026 conference.

The gap between those two figures is where deployment risk lives. The first figure ends up in supplier slides. The second describes what happens when the task reaches completion.

Several open-weight architectures drop nearly to zero on end-to-end coherence while maintaining high scores on individual steps. The phenomenon appears across all ten systems tested, with varying intensity.

This is the kind of result that changes what question you ask a supplier.

How the measurement tool is built

The benchmark takes the expert logic from the Cochrane Risk of Bias 2.0 tool, the standard by which human reviewers judge the quality of a clinical study. The assessment proceeds in stages: reporting questions, domain judgment, overall judgment.

The measurement framework is called Hierarchical Logical Consistency and observes four distinct dimensions:

• Atomic Consistency: the soundness of individual elementary judgment
• Domain Consistency: coherence within each bias domain
• Aggregation Consistency: the validity of the transition from domains to final judgment
• Evidential Faithfulness: adherence of the conclusion to cited evidence

This architecture enables something rare: isolating the exact point where reasoning breaks down from the final score. A surface matching test compares the produced label to the expected label and stops there. Here each level of the hierarchy carries its own number.

The sample covers 860 trials and 14,820 queries, a large volume for a clinical task of this specificity. The work was deposited on arXiv on September 10, 2026. The design responds to a longstanding criticism of this class of measures: a single score reveals little about the process.

The anomaly: the collapse exceeds arithmetic

Here appears the detail that deserves attention. With a 98.88% success rate per step and independent errors, a chain of five judgments would arrive around 94.5%. The observed value is 45.13%.

Arithmetic on the published values says otherwise: to drop to 45.13% with independent errors would require roughly seventy consecutive steps. The Cochrane procedure anticipates far fewer.

The reading that evidence supports is direct: errors are correlated. The model fails systematically on points where expert logic requires inference, and those errors concentrate rather than distribute.

Error compounding in multi-step agents remains the risk worst-priced by investment committees. This work offers a clean measure of it, within a domain where the logical chain is written, public, and verifiable step by step.

An increase in task length costs far more than the per-step score suggests.

Retrieving evidence and reasoning from it

The second finding concerns the passage from evidence to conclusion. Even when retrieving high-quality evidence, models infer the wrong outcome in 18.63-40.05% of cases.

The rate of responses produced as blind attempts, per the authors' definition, reaches 48.28%.

This separates two problems the market treats as one: information retrieval and reasoning about that information. The first improves with data infrastructure, indexing, and retrieval. The second remains where it was.

For a Chief Analytics Officer the distinction carries a precise cost. Investing in RAG acts on the left half of the problem and leaves the right half untouched, while the budget gets approved as a remedy for both.

The gap between retrieved evidence and deduced conclusion persists across the full scale of models tested, with a minimum of 18.63%. Even the best case leaves nearly one task in five with incorrect inference despite correct evidence.

Why an aggregate score hides the defect

A high score on final accuracy can coexist with broken reasoning. The model arrives at the correct label by following the wrong path, and the test registers success.

The authors conclude that white-box logical verification becomes necessary before any clinical use.

The problem has a documented parallel elsewhere. An analysis published on the Alignment Forum^[2] describes how much a model can accomplish without chain of thought. The visible text, therefore, reveals little about the internal process.

The two pieces of evidence converge on a single diagnosis: reading the final output measures the final output. A model evaluation benchmark built on labels remains blind to the path that produces them.

The difference matters most in domains where the path is the substance of the work: evidence-based medicine, audit review, compliance. In these cases a correct outcome obtained by chance is worth little. Value lies in the chain that justifies it.

What changes for budget allocators

For an investment committee the relevant number is 45.13%, never 98.88%. The first value describes the task carried through to completion; the second describes an isolated step within that task.

The technological thesis supported by these data concerns verification, never raw model power. The collapse appears across ten leading systems together, so the factor explaining it lies in the task structure.

Three readings for three different tables:

• CRO and Investment Committee: validation spending should be sized to the full chain, never to the step
• Chief Analytics Officer: infrastructure is needed that records and compares each intermediate judgment
• Board: the thesis that the most capable model solves the problem lacks support in these data

Validation spending sized to per-step accuracy underestimates the actual verification work. The useful measure comes from the full chain, tested on the volume and variety of cases production brings. This shifts resources from training toward test infrastructure.

The cost of verification grows with chain length, and grows faster than expected score. This remains a diagnosis. The allocation choice belongs to the table that makes it, with the right number in front.

Limitations declared by the work

The work covers one domain: risk of bias assessment in randomized clinical trials. Extension to other multi-step tasks remains an open question, and the authors leave it open.

Ten models are a small sample relative to the market, and systems change version in weeks. The date of the data matters: September 10, 2026.

The benchmark measures hierarchical logical coherence, never clinical utility. A system at 45.13% end-to-end coherence may remain useful as support under human review, and these data say nothing on the point. A replication on another domain would indicate how much the result generalizes.

What the evidence shows is circumscribed and solid: atomic accuracy and end-to-end coherence are two different quantities. The distance between them grows with chain length, and the figure that matters for deployment is always the second.