AI Consilium: An LLM Judge Scores 3,040 Clinical Second Opinions

A team of thirteen researchers has built a shared benchmark for clinical second opinions written by large language models. They posted it to arXiv on 15 October 2026 under the name AI Consilium.

The headline figure covers 3,040 final reports produced by eight pipelines, each built on a different model^[1]. A separate language model acted as judge and scored every one of them. According to the authors, the choice of model separates report quality more than the response format does.

Three hundred NEJM cases and a judge calibrated against physicians

The raw material is 300 NEJM-MedQA cases with a confirmed diagnosis.

The authors split the cases into two sets. For 110 cases, physicians supplied reference ratings that were used to calibrate the automated judge. The other 190 cases were held out from calibration and used for the evaluation itself.

Each pipeline runs a single model in every role. That model writes an opinion for each specialist and then a final consultation report. It does this twice: once in free text and once following a structured clinical reasoning template.

The 3,040 final reports come from 190 cases multiplied by eight pipelines and two formats.

Structure pays off in clinical management

The structured report earned a higher overall mean score in seven of the eight pipelines.

The clearest gain is in the patient management section, where the structured format scores 2.740 against 2.436.

The paper's public abstract reports the two means but gives neither the rating scale nor the spread of the scores.

The second comparison looks at the multi-specialist consultation. The structured final report beats the first specialist's structured opinion on all five shared criteria. The largest gains come in prioritisation, factual accuracy and coherence of reasoning.

Three design choices underpin every number

Every evaluation result rests on three upstream choices.

• A single judge. Every report is scored by the same automated evaluator.
• Calibration on 110 cases. Agreement between the judge and the physicians rests on this sample.
• Journal cases. NEJM cases are already solved and written for teaching. The design tests models on published cases and leaves real patients out of the picture.

The front-runner and the missing name

GPT-5.6-Sol leads on every component assessed and reaches consensus between roles in fewer revision rounds.

Topping every criterion is a strong result. It also calls for scrutiny, because the grader is itself a language model.

The public abstract does not name the judge model. The full paper is available on arXiv and on alphaXiv^[2].

The score depends on who gives it

I treat every automated judge as an instrument with systematic bias.

The authors took the right step by calibrating it against physician assessments. The weak point is the ratio: 110 calibrated cases against 3,040 reports to be judged.

Each report gets one automated score and nothing more. So the 0.304-point gap in clinical management is a gap as measured by that instrument. What it means for a doctor on the ward is a different measure, and this study leaves it open.

What holds up for those funding clinical research

For a committee setting a healthcare R&D budget, the most robust finding concerns the choice of model.

Structuring the response adds a gain on specific criteria, led by clinical management. Synthesis across multiple roles adds a gain on all five shared criteria.

For heads of data and analytics, the benchmark shows how much weight the evaluation layer carries. With 110 calibrated cases and a single judge, the quality of the measurement depends almost entirely on that judge.

For a board, the claim that "the model matters more than the format" holds up within this design. The claim that "an LLM gives reliable clinical second opinions" lies outside what the paper measures.

Three things to look for in the PDF before citing the ranking:

1. the measured agreement between the judge and the physicians on the 110 calibration cases;
2. the name and family of the judge model;
3. the scale and spread of the clinical management scores.

Until those three figures are accounted for, the ranking describes one model's judgement of other models.