GenAI Audits: Why the Compliance Rate Shifts

What was filed, and when

On 14 August 2026, fourteen researchers filed a paper on arXiv titled ASSERT. The text describes a measurement pipeline for audits of generative artificial intelligence systems.

The document addresses a technical question with direct consequences for governance. How do you produce a reliable compliance rate when verifying a generative model?

The answer determines what a board of directors can publicly state. According to the paper, these checks often summarize behavior as a reported rate: the frequency with which the system complies with policy.

The delta: from a single number to a written specification

Before ASSERT, a reported rate circulated as a standalone figure. Researchers and stakeholders used it to compare systems, track regressions, and authorize deployment. None of these uses required the method behind the number to be made explicit. The figure was self-sufficient.

The structural problem is precise. A rate reflects two elements together: the system examined and the measurement choices behind the number. These two elements remain fused in the figure. Whoever reads the rate cannot tell them apart.

When the figure changes, it stays ambiguous what actually moved. Did the system change, or the measurement method? Without a documented specification, the question has no answer. ASSERT ties every reported rate to a written specification of the choices adopted.

This delta redefines the nature of evidence. A number becomes verifiable when the specification that generates it is documented in writing. The difference is stark. A rate without a specification describes an outcome. A rate with a specification describes an outcome and the conditions that produced it.

The mechanism: why the number moves

The paper includes a case study on conversational deception, a system's tendency to deceive the user within the dialogue. The results are explicit.

The reported rate shifts substantially as four measurement factors vary:

• the dialogue setup
• the simulated user
• the judge that evaluates the output
• the evidence threshold required to declare a non-conformity

These choices change the rate significantly. They even reshuffle rankings across different systems. A system leading under one method can slip down under another. The ranking follows the specification, not just the model.

The consequence is clear. Two verifications of the same model produce divergent numbers when measurement choices differ. The resulting ranking becomes fragile. A vendor comparison based on rates obtained with different methods does not hold up. The higher figure may reflect a more permissive method, not a better system.

The governance signal

The governance signal: a compliance rate without a documented specification remains an assertion, never evidence.

A board that approves deployment on the basis of an isolated number takes on attribution risk. The figure could reflect the system's robustness or the generosity of the measurement method. The board has no way to tell the two cases apart from the number alone.

ASSERT converts this risk into a documentary discipline. Every rate is anchored to a behavioral rubric, to test cases, and to declared judgment criteria. Each element remains inspectable after measurement.

For control functions, the value lies in traceability. Differences between two exercises become readable and attributable to specific choices. A reviewer can trace the cause of a discrepancy instead of logging it as unexplained data.

Who is accountable

The operational question remains the same one ATLAS poses about every framework. Which named role in the organization is accountable for the measurement specification, by name, in writing, before deployment?

Accountability without a name amounts to compliance theater. A pipeline like ASSERT produces structured documentation, yet real governance arises from the explicit assignment of responsibility. The document fixes the method; responsibility for the method remains a distinct organizational choice.

The General Counsel must map the number's chain of custody. Who wrote the rubric, who selected the judge, who set the evidence threshold. Each step requires an associated name, not a generic function.

The Chief Risk Officer updates the risk framework accordingly. A reported rate enters the risk register as conditional data, accompanied by its specification. The register no longer holds a bare figure, but a figure and its conditions of validity.

What changes for the enterprise

A control posture calibrated on the single rate is now oversized relative to the new methodological context. The audit is still required; the perimeter has changed.

Organizations that adopt specification-driven audits gain comparability over time. Regressions become measurable because the measurement baseline stays constant. A drop in the rate signals a change in the system, not in the method.

Those comparing vendors acquire a more solid criterion. A ranking accompanied by the underlying specification withstands contractual challenge. The counterparty can verify the method instead of merely disputing the result.

This shifts the center of gravity of due diligence. The question moves from the value of the rate to the quality of the specification that generated it. A high number obtained with a weak method is worth less than a cautious number obtained with a rigorous one.

Three decisions for the board

The board translates this evidence into concrete choices. Here are three numbered decisions for General Counsel, Chief Compliance Officer, and the Board Audit & Risk Committee.

1. Establish that every compliance rate presented to the board arrives accompanied by the measurement specification that produced it.
2. Assign in writing the role accountable for the rubric, the judge, and the evidence threshold, before authorizing deployment.
3. Define disclosure to stakeholders and regulators so that the measurement method is verifiable, together with the number.

These choices shift legal exposure. A documented rate holds up under challenge; an isolated figure remains exposed to the question about method.

Organizations that build this discipline now gain a lead of several months. Structured compliance becomes a competitive asset, as well as an obligation.

Regulatory horizon

The ASSERT work is a research contribution filed on arXiv on 14 August 2026. It constitutes methodology, and remains distinct from a binding rule. No legal obligation flows directly from the paper.

The regulatory context, meanwhile, is converging toward the demand for traceable audits. The EU AI Act imposes documentation and assessment obligations for the high-risk systems listed in Annex III.

Governance functions can align their measurement practices to this trajectory now. A specification-driven method answers in advance the question enforcement will pose.

The question of how to produce a reliable rate has received an operational answer. A second question has opened: which named role answers for the specification. For related insights, our blog collects the associated governance analyses.