Computer-Use Agents: The 30% That Decides Your Stack

What ComponentBench Measures

ComponentBench, published August 18, 2026 and accepted at COLM 2026, evaluates component-level failures in computer-use agents. The suite covers 97 canonical UI components, instantiated across 2,910 programmatically verified tasks.

Its technical positioning is precise. Existing benchmarks split between long-horizon workflow tests and atomic GUI-grounding tests.

This divide leaves the intermediate layer exposed. Component-centered interactions are short to diagnose and rich enough to capture the load of modern interfaces. A typical example is toggling a button set.

Each task comes with cleaned human reference trajectories, enabling success and efficiency to be measured within the same execution. The benchmark also adds a pipeline to verify structural difficulty after implementation. The result is a dual metric: a task can be completed but with far more steps than a human would take. Separating success from efficiency prevents misreading a slow agent as a robust one.

The Mechanism: Observation and Action Space

The critical finding emerges from the harness configuration. The authors hold the model fixed and vary only the observation and action space.

Task success swings by more than 30 points for the same model. The starkest case involves GPT-5 mini. With accessibility-tree observations it reaches 83.1% success.

With coordinate-only control in Pixel mode, the same model drops to 48.9%, according to the paper published on arXiv^[1]. The evaluation covers seven models and four observation and action spaces.

The engineering reading is straightforward. Agent performance depends on the perception layer as much as on the underlying model. Downgrading the input turns a production-grade system into an unreliable one. The mechanism is simple: the accessibility-tree gives the model a structured description of UI elements. Pixel control forces the model to infer those same elements from raw coordinates. The perception burden shifts entirely onto the model, and errors increase.

The Root Condition

The structural condition these results share is clear. Accuracy attributed to the model is actually accuracy of the entire perception-and-action stack.

Many procurement teams evaluate agents by looking at the model name. This approach ignores the variable that moves results by 30 points. Two deployments of the same model diverge when the observation layer differs.

The practical consequence is measurable. A vendor benchmark reporting a single number conceals this dependency. A number without a declared observation space carries limited informational value. A vendor can legitimately report 83.1% and be referring to accessibility-tree. The same model, on your pixel-control stack, produces 48.9%. Neither number is false. Only one describes your deployment.

Spatial manipulation remains a documented weak point. Operations trivial for a human continue to challenge current agents.

The gap concentrates where the interface requires fine coordination between perception and action. The model holds up in the abstract; its translation into precise gestures breaks down.

Efficiency: The Hidden Cost of 3.7x

Task success tells half the story. The other half is interaction efficiency.

The fastest configuration takes 3.7 times longer than the human reference trajectory. This multiplier has direct effects on operating cost. Every additional step consumes tokens, latency, and inference budget.

For a CFO, the calculation changes in nature. An agent that completes a task at 3.7 times the cost of a human operator shifts the break-even threshold.

Automation becomes cost-effective above a specific volume. That volume is measurable before deployment. A pilot that ignores this multiplier produces a fragile business case. And 3.7x is the best-case configuration, not the average. Poorer observation spaces consume even more steps. Operating cost scales with input choice, exactly as accuracy does.

What Changes for the Enterprise Stack

The findings rewrite the evaluation checklist. Model selection becomes secondary to observation space selection.

A team adopting coordinate-only control for portability reasons pays an accuracy penalty. A team exposing the accessibility-tree gets better results, at the cost of a dependency on the rendering infrastructure.

This is a classic architectural lock-in trade-off. The decision should be documented as explicit technical debt.

Those who choose the leaner input for flexibility must budget for the reliability drop. Those who choose the richer input must plan fault tolerance against interface changes.

The Connection to Multi-Agent Systems

ComponentBench's findings extend beyond a single agent. In a multi-agent pipeline, one component's output becomes the next component's input.

A 49% success rate at the component level propagates along the chain. Three sequential steps with imperfect reliability produce a composite outcome far more fragile than any individual step.

The math is straightforward to verify. With 0.49 reliability per step, three independent steps fall below 12% composite success. This explains why demos work and deployments fail. A demo shows one isolated step under favorable conditions. A deployment chains multiple steps and multiplies their error rates.

This is the mathematics of cascading failure. The defense remains architectural. Every step requires independent validation and an explicit circuit breaker.

Three Questions for the Enterprise AI Team

The operational evaluation comes down to three questions for your enterprise AI team.

1. Which observation space feeds the agent in production, and what measured success does it produce on your domain?
2. What is the time multiplier relative to the human operator, and at what volume does automation reach break-even?
3. How does the agent respond to a component library change, and what test coverage protects the pipeline?

The answer to the first question determines the entire strategy. An 83% success rate with accessibility-tree and a 49% rate with pixel control describe two different products built on the same model.

These questions shift focus from the vendor benchmark to internal measurement. The number that matters is the one collected on your stack, with your interface.

Build/Buy Decisions for the Next Planning Cycle

The takeaway for CTOs and Heads of Engineering is operational. ComponentBench is a diagnostic methodology to adopt, not a vendor product to evaluate.

The code and data are public. An engineering team can replicate the pipeline on its own component inventory.

This transforms agent evaluation from a marketing exercise into a repeatable measurement. For the Technology Procurement Committee, the signal is clear.

Every contract that promises agent accuracy must be tied to the observation space tested. A clause fixing success on accessibility-tree loses value when production runs on pixel control.

The next planning cycle requires an explicit decision on the perception layer. Deferring it amounts to accepting a 30-point gap as latent risk in the automation budget.