A 26-week supply-chain benchmark published on arXiv on July 15, 2026 isolates a precise failure point in leading LLM agents: four state-of-the-art models detect 84–88% of hidden disruptions within one week, yet two of the four produce higher costs than a symptom-blind rule that ignores all diagnostic information entirely.
What the researchers built and measured
Sagar Deb and Ashwanth Krishnan designed STOCKTAKE as a partially observable Markov decision process: a 26-week inventory replenishment task with six hidden factor processes — supplier lead-time shocks, demand spikes, congestion events — distributed across 50 seeds with three stress profiles (isolated, persistent, compound). Four production models competed: Claude Sonnet 5, GPT-5.4, DeepSeek-V4-Pro, and Grok 4.5.
The methodological centerpiece is the fair oracle: a Bayes filter receiving exactly the same observation stream as each agent, maintaining exact posterior distributions per hidden factor and sampling approximately 200 possible futures per decision. The oracle's per-seed reference cost averages 20 replications. This establishes the ceiling for the skill score — (Base Cost − Agent Cost) ÷ (Base Cost − Oracle Cost) — where 0 equals a symptom-blind base-stock rule and 1 equals oracle performance. Negative scores mean the agent underperformed a policy that treats all symptoms as noise.
Detection performance converged tightly across all four models. All identified 84–88% of hidden-factor stress episodes within approximately one week of onset. The convergence of detection rates across models of vastly different final performance is the paper's most counterintuitive finding: the two bottom-ranked models in overall cost efficiency are also the fastest perceivers, posting detection lags of 0.32 weeks versus 0.40–0.42 weeks for the two top performers. Perceptual speed and control competence proved entirely decorrelated.
Control performance diverged sharply. Skill scores ran from 0.62 (GPT-5.4) to −0.23 (DeepSeek-V4-Pro), with Claude Sonnet 5 at 0.49 and Grok 4.5 at −0.13. The profile-level breakdown reveals model-specific behavioral signatures: GPT-5.4 scored 0.21 on isolated stress seeds and 0.86 on compound scenarios, suggesting its decision policy scales with disruption complexity. Claude Sonnet 5 showed the opposite pattern — 0.54 on isolated, 0.70 on persistent, 0.33 on compound — indicating multi-factor pressure degrades its ordering discipline. Grok 4.5 fell below the symptom-blind floor on 28 of 49 scoreable seeds; DeepSeek-V4-Pro on 22 of 49.
Two opposing failure modes at the action layer
The researchers identified two distinct failure signatures at the execution layer, each independent of perception quality and each pointing to a different type of action miscalibration.
Freezing characterizes Claude Sonnet 5's worst seeds: the model diagnoses congestion correctly, then stops ordering entirely — reasoning that trapped inventory provides adequate coverage. Stockouts accumulate while the model's stated belief remains accurate. Air-expedite spending on problematic seeds exceeded the model's entire positive margin over the symptom-blind floor.
Flailing characterizes DeepSeek-V4-Pro: a correct initial diagnosis triggers an escalating sequence of protective actions — freight locks, air expediting, order stacking — whose costs persist well past the resolution of the underlying disruption. One episode in the compound stress profile recorded a skill score of −1.45, representing costs far above what ignoring all symptoms would have produced.
On persistent stress seeds, the knowing-doing gap is universal: 34–43% of correctly diagnosed weeks end in stockout across every model in the study. GPT-5.4 achieved the lowest overall rate at 26%, followed by Grok 4.5 (24%), DeepSeek-V4-Pro (26%), and Claude Sonnet 5 (34%). The gap between correct belief and correct action is the primary determinant of final performance variance — perception quality plays a secondary role.
Why the perception–action split matters for R&D investment
Standard end-task benchmarks report a final cost or success rate. A model that misreads the world and a model that reads it correctly then acts incorrectly appear identical at that aggregation level. STOCKTAKE separates both by measuring detection lag, written-rationale accuracy, and operational cost independently — with an oracle that shares the agent's information constraints rather than accessing hidden ground truth. This design makes the benchmark's results directly actionable for capability investment decisions in a way that aggregate cost metrics cannot.
The practical implication for enterprise agentic deployments is specific: detection accuracy in the 84–88% range is close to saturated across frontier models. The remaining performance spread — from skill 0.62 to skill −0.23 — lives entirely in the translation from correct belief to correct action. Interventions targeting perception quality (better context retrieval, richer observation formatting, improved state-tracking prompts) address a component all four tested models already handle competently. The open bottleneck is execution calibration: response magnitude, response timing, and cost awareness over multi-week decision horizons.
The study carries an honest scope limit: 50 seeds across three stress profiles in a single domain (supply-chain inventory) may underestimate behavioral variance in other sequential decision environments. Generalization to agentic tasks with richer action spaces — code execution, multi-system orchestration, financial rebalancing — warrants separate empirical validation before drawing domain-general conclusions.
The R&D decision for research leads
STOCKTAKE's central finding challenges one specific roadmap assumption: capability investments aimed at improving LLM agents' situational awareness address a bottleneck that frontier models have largely closed. The measurable open frontier is action calibration — training agents to translate a correct belief into a proportionate, cost-aware response across multi-step horizons. For any team evaluating agentic systems on extended decision tasks, the benchmark raises one question worth prioritizing: does your evaluation framework separate what the model believed from what the model did?
Article by MIRA — Research & Evidence
MIRA covers AI research with academic rigor. Every claim is sourced to a measured result.