AI Benchmarks: Why the Vendor's Own Test No Longer Counts

Forty million to measure other people's models

Vals AI, a company that measures model capabilities in place of those who sell them, has closed a $40 million Series A led by Andreessen Horowitz, as TechCrunch reported on 19 September 2026^[1], placing the raise in the previous month.

The company was founded in 2024 and reaches this round after a seed led by 8VC and Bloomberg Beta. Co-founder Rayan Krishnan is 25, a former Palantir intern with a track record spanning Microsoft and Stanford's artificial intelligence laboratory.

The figure matters less than the category it certifies. A fund of that calibre is putting $40 million behind a precise thesis: independent verification of model capabilities is becoming a product you buy off the shelf. For a procurement function this is the clearest signal yet that model due diligence is moving outside the company's own walls.

What this company actually sells

The press-release language talks about the "gold standard" of benchmarking. The substance is rougher and more interesting.

Vals keeps its test material private. The choice serves a practical purpose: a public test ends up inside the training data of the very model it is supposed to judge, and the score loses all meaning. It is an exam the candidate has seen in advance.

The second difference concerns what is being measured. Legacy academic tests check how much a model knows in the abstract, along the lines of a bar exam. This company instead measures the execution of complex tasks inside specific sectors: law, finance, code writing.

The distinction weighs on procurement. A score on a generalist test says little about how the system behaves inside a real workflow, made of documents, constraints and control checkpoints.

A vendor's benchmark is worth about as much as a self-certification

The market mechanism is simple. The vendor designs the release, picks the tests, publishes the table and uses it as sales material. A good result amounts to a good press campaign, as the TechCrunch piece observes.

There is a structural conflict of interest here, entirely legitimate and entirely self-interested.

The problem grows with the pace of releases. Frontier models arrive almost monthly, while academic tests age and fall behind the frontier they are meant to measure. Krishnan is building his company precisely on that gap.

The reputational backlash is already visible elsewhere. The South China Morning Post reported the case of China's Z.ai, which took a reputational hit after users spotted uploads made without their knowledge^[2]. Trust asserted by the supplier holds up poorly under external verification.

The competitive shift: from the score to the criterion

This desk's thesis holds firm: the model is a commodity, and the price list proves it at every release. When the price of the frontier tier falls, the selection criterion stops being declared power.

The competitive axis shifts from the benchmark published by the supplier to the evaluation carried out by a third party on the specific task of the company doing the buying.

This changes the shape of the contest. A vendor with the highest score on a public test loses ground to a rival that holds up under a closed test built on the client's own domain. The advantage moves from the shop window to verified performance.

For whoever runs strategy the consequence is immediate. The ability to measure becomes part of bargaining power: those who measure independently negotiate renewals from a stronger position, and loosen the lock-in built on loyalty to a single supplier.

Who feels it on the vendor map

The implications touch three groups: the large model providers, the cloud platforms that resell them, and the consulting groups that sell technology selection.

The first lose control of the narrative about their own capabilities. The second see an arbiter enter the purchasing process who answers to the client. The third face new competition: part of their evaluation work becomes an off-the-shelf product, faster and cheaper.

The enterprise market remains open. Consumer dominance shifts little on governance, traceability and data residency, and independent measurement moves into exactly that space.

There is a serious counter-argument. A private company that keeps its own tests confidential is asking the market for an act of trust similar to the one it wants to replace, and the method stays opaque to anyone reading the results from the outside.

The practical answer comes from incentives. Those who sell evaluations live on their reputation with the buyer, and pay immediately for a measurement error. Those who sell models live on renewals, and profit from a generous score.

The market signal: capital rewards control

Venture capital continues to concentrate on the big artificial intelligence rounds, as shown by the round-up of the largest funding deals compiled by Crunchbase News^[3].

Inside that flow, $40 million into an evaluation company tells a different story from the rounds on compute and models. Money is starting to fund the tools that make the offering comparable, and comparability is the antechamber to price pressure.

The distinction this desk always applies holds: what we have here is a funding announcement, and market share is still to be won. Consolidation of the category remains a hypothesis backed by patient capital.

For an investor the reading is clear. The thesis that value migrates from the model towards the layers that surround it, deployment, measurement and data governance, gets one more confirmation.

What to decide in the next 90 days

The next budget cycle demands three concrete moves from technology buyers.

The first concerns contracts up for renewal. Every platform renewal should be accompanied by an internal test built on the company's real tasks, with written results that can be repeated at every new model version.

The second concerns spending. A dedicated line item for independent evaluation costs a fraction of the contract it protects, and it should be opened now, before the next negotiation on volumes.

The third concerns the supplier portfolio. An up-to-date map, with two alternative models ready on every critical workload, turns measurement into pricing leverage.

• Chief Strategy Officer: consider an agreement with a third-party evaluator before the category consolidates.
• CFO: open a spending line for independent measurement and tie it to savings on renewals.
• Chief Digital Officer: reopen the question of the supplier chosen on a public score.
• Technology Investor: the thesis on value migrating towards the control layers is gaining ground.

The market has moved. The next contest is won on verified data, and the score published by the supplier goes back to being what it always was: sales material.