Carbon Capture and AI Agents: The Trust Risk

What changes: a formal framework for trust in agents

On August 28, 2026, the v2 revision of an academic work formalizing trust between humans and AI agents was released. The original version dates back to January 29, 2023, authored by Zahra Zahedi, Sarath Sreedharan, Erin Chiou and Subbarao Kambhampati, archived on arXiv^[1].

The paper introduces a framework based on the mental model. It is used to infer how much trust a human places in an agent during interaction.

The relevance for those deploying agents in production is direct. A system acting alongside human operators must estimate the trust level, then calibrate its actions accordingly.

This is a theoretical contribution, and should be treated as such: a formal framework, validated with human subject studies, far from production-grade status. The distinction matters. Adopting it today means adopting a research model, a conceptual specification to evaluate, rather than a deployment-ready product.

The mechanism: performance, process, purpose

The framework breaks trust down into measurable dimensions. The paper ties them to three perceptions: performance, process and purpose.

Performance concerns how well the agent executes the task. Process concerns how it reaches the result. Purpose concerns the alignment between the agent's objectives and those of the human.

The central mechanism is belief updating. By modifying the human's beliefs about the agent, according to the model's predictions, perceived trust changes.

The authors validate two hypotheses with human subject studies. First: trust variations are obtained by adjusting the operator's beliefs. Second: adjustments in beliefs produce corresponding changes across the three dimensions.

The technical point deserving attention is the formalization of the appropriate level of trust. An agent that generates too much trust induces overreliance. An agent that generates too little is ignored, and therefore remains useless.

Why carbon capture plants are the proving ground

Carbon capture plants are becoming fertile ground for AI monitoring and optimization agents. Sensors, process control and predictive maintenance generate decisions that a human operator must either accept or reject.

In this context, miscalibrated trust becomes a concrete operational risk. An operator who blindly trusts an optimization agent accepts incorrect setpoints. An operator who distrusts every recommendation nullifies the system's value.

The paper's framework offers a formal vocabulary for measuring this balance. Applying it to a real pipeline requires telemetry, decision logging and a feedback channel to the operator.

The question for the CTO is direct: does your architecture record operator trust as a first-class signal? Most installations ignore it. This is technical debt paid during incidents, when the chain of human and automated decisions must be reconstructed after the fact.

The root condition: miscalibrated trust

The structural condition this work exposes is trust treated as an implicit variable. Most AI systems in production assume user trust instead of measuring it.

The result is a fragile architecture. When the agent makes a mistake, the system lacks a mechanism to detect that the operator has stopped trusting it, or is trusting it too much.

This pattern coincides with the one identified in multi-agent systems lacking circuit breakers. The output of one component becomes the input of another, trust propagates implicitly and errors accumulate in cascade.

The mental framework makes the variable explicit. Modeling trust means being able to monitor it, then intervene when it moves outside calibrated limits.

For a Head of Engineering the consequence is practical: trust becomes a value to instrument, with thresholds, alerts and fallbacks. A carbon capture plant governed by agents inherits this same fault tolerance requirement.

Three questions for the enterprise AI team

The paper remains academic research, so the operational translation falls to the team. Three questions guide that translation.

First: does your pipeline measure operator trust as a signal, or does it infer it from behavior after the fact? The second option always arrives too late.

Second: when an agent receives too much trust, is there an execution limit that blocks high-risk actions? Self-invocation enabled without execution boundaries is the classic failure mode.

Third: do the paper's three dimensions (performance, process, purpose) remain observable in your telemetry, or are they opaque?

A team that answers these three questions with evidence has a calibrated architecture. A team that answers with assumptions has a latent risk. The lesson extends beyond carbon capture: any domain where a human supervises an agent inherits the same calibration problem.

Procurement decisions for the next cycle

For the Technology Procurement Committee this work shifts an evaluation. Vendors of agentic platforms promise human-in-the-loop as a marketing feature. The framework provides criteria to verify that promise.

The contractual question becomes measurable: does the vendor expose operator trust as a metric, with read APIs and configurable thresholds? A vague answer signals an immature feature.

The CFO evaluates investment risk in this light. An agentic infrastructure that ignores trust calibration accumulates operational risk that surfaces during incidents, when costs multiply.

I state my position: multi-agent systems in production without explicit circuit breakers will fail in cascade. Miscalibrated trust is one of the mechanisms that triggers that cascade. The alphaXiv version of the paper^[2] offers an annotated reading path useful to the technical team.

Build or buy: where the lock-in lies

The build/buy decision depends on the maturity required. The framework is a conceptual specification to adopt, rather than a product to purchase.

For the next planning cycle the choice for CTOs and Heads of Engineering is clear. Building trust instrumentation internally guarantees architectural control, at the cost of dedicated engineering effort.

Buying a platform that offers it reduces time-to-production, at the price of architectural lock-in on the vendor's trust model.

My recommendation remains cautious. Treat trust as an input to verify until telemetry makes it observable. A carbon capture plant governed by agents requires fault tolerance across the human decision chain, on a par with the automated chain.

The work of Zahedi and colleagues provides the formal foundations. The translation into production-grade code remains your team's task, with independent validation before every critical deploy.