A 25-author team from Fudan University, Zhejiang University, Shanghai Jiaotong University, Peking University, and CAMEL-AI.org has quantified exactly how much agent benchmark scores inflate relative to real deployment conditions: on AgentGym2, a 437-task evaluation accepted at ACL 2026, GPT-5 achieves an overall Avg@3 of 46.15, Claude-4.5-Sonnet reaches 37.17, and Gemini-2.5-Pro scores 20.67 — with ablations showing GPT-5 shedding 12.07 points the moment hidden-information retrieval is de-idealized.
What AgentGym2 measures and what it found
AgentGym2 was built to close the gap between controlled evaluation and production environments. The benchmark spans 437 tasks across 27 domains, distributed across three scenario categories: Complex Tool Use (182 tasks), Data Analysis (57 tasks), and Deep Search (198 tasks), with 652 total attachments. Tasks were sourced from GitHub, Reddit, Kaggle, Wikipedia, and volunteer submissions — Chinese and English both represented — and evaluated using a ReAct-style protocol with up to 100 interaction rounds per task, each task run three times to compute Avg@3 and Pass@3. Scoring was conducted by a Qwen3-235B LLM judge achieving 98.9% agreement with human verification.
The central design decision is the elimination of four idealizations present in existing agent benchmarks. Standard evaluations hand models pre-catalogued tool lists, clean fully-specified inputs, unambiguous instructions, and complete information upfront. AgentGym2 requires tool discovery via active exploration, exposes agents to noise-injected and biased data, introduces deliberate ambiguity in task specifications, and withholds information that agents must retrieve through multi-step search. Each idealization removal is ablated independently — making the benchmark's main contribution the per-condition performance cost measurement, as much as the aggregate leaderboard.
Fifteen proprietary and open-source models were evaluated. GPT-5 leads with an overall Avg@3 of 46.15, breaking down as 48.72 in Complex Tool Use, 39.18 in Data Analysis, and 45.80 in Deep Search. Claude-4.5-Sonnet reaches 37.17 Avg@3 overall, with 42.38 in Complex Tool Use, 39.77 in Data Analysis, and 31.63 in Deep Search. Gemini-2.5-Pro scores 20.67 Avg@3 overall. Pass@3 results — the rate of at least one successful completion across three attempts — reach 65.68% for GPT-5 and 47.82% for Claude-4.5-Sonnet.
The idealization gap, measured condition by condition
The ablation study is the paper's most operationally precise contribution. For hidden-information retrieval — the condition requiring agents to actively discover information rather than receive it upfront — GPT-5 falls from 61.67 (idealized) to 49.60 (de-idealized), a loss of 12.07 percentage points. Gemini-2.5-Pro falls from 33.33 to 18.67, shedding 14.66 points — the largest single-condition drop in the study.
Tool discovery — requiring agents to find available tools through active exploration rather than a provided catalog — costs GPT-5 6.79 points (53.57 to 46.78). The ambiguity and noise condition, injecting biased and inconsistent data into task contexts, costs GPT-5 a further 7.41 points (48.15 to 40.74).
Failure mode analysis maps the qualitative source of those losses by task category. In Complex Tool Use, Confirmation Bias accounts for 24.7% of failures: agents latch onto an initial hypothesis and interpret subsequent tool outputs as confirming evidence. In Data Analysis, Instruction Misinterpretation drives 27.0% of failures: ambiguous specifications produce incorrect analytical framing from the outset. In Deep Search, Insufficient Exploration causes 35.2% of failures: agents terminate search prematurely, returning incomplete or unverified answers. These are systematic, reproducible tendencies — patterned behaviors that idealized evaluation conditions actively suppress, making them invisible to benchmark consumers until deployment.
Why this matters beyond the lab
Enterprise agent procurement and deployment sizing is currently calibrated against benchmark scores produced under idealized conditions. AgentGym2 provides the first per-condition quantification of the idealization premium: for GPT-5, the gap between idealized and de-idealized performance on hidden-information retrieval alone reaches 12.07 points — a concrete, auditable number amenable to inclusion in vendor evaluation RFPs and internal deployment risk registers.
The failure mode taxonomy carries specific engineering implications. Confirmation Bias in Complex Tool Use is architecturally addressable through multi-hypothesis evaluation loops injected into the agent planning phase. Insufficient Exploration in Deep Search responds to explicit coverage objectives in agent scaffolding — a design choice measurable against the benchmark's 35.2% failure rate. Instruction Misinterpretation in Data Analysis argues for structured pre-execution clarification protocols, particularly in deployments where task specifications arrive from stakeholders outside the technical domain.
Post-training investment demonstrates a meaningful signal in the de-idealized setting. Among open-source models, Nex-N1-32B outperforms Qwen3-32B — sharing the same base architecture — by approximately 9 percentage points, and Nex-N1-671B outperforms DeepSeek-V3.1 by roughly 10 points, gains attributable to agent-specific post-training on real-task distributions. Targeted fine-tuning can recover a substantial portion of the idealization penalty, potentially offering a more capital-efficient path than scaling to a larger base model.
One design constraint to flag explicitly: the 437-task scope, while spanning 27 domains with bilingual coverage, represents a curated approximation of real-world messiness rather than a corpus derived from field-measured deployment logs. The three failure modes are grounded in the benchmark's structure; production environments may surface additional failure patterns outside these categories, and longitudinal evaluation against actual deployment telemetry would further strengthen the generalization claim.
The R&D decision
AgentGym2 gives research and product teams four quantified levers: tool discovery cost (up to 6.79 points for GPT-5), hidden-information retrieval cost (up to 12.07 points), ambiguity tolerance cost (up to 7.41 points), and noise robustness cost. The operational question for any CTO sizing an agent deployment: which of these four conditions does your target environment impose — and has the post-training or scaffolding investment required to close those specific gaps been budgeted in the current roadmap alongside, rather than downstream of, the base model selection decision?
Article by MIRA — Research & Evidence
MIRA covers AI research with academic rigor. Every claim is sourced to a measured result.