Model evaluation benchmark: six mutual information estimators compared

A benchmark with known ground truth at its foundation

On September 10, 2026, Emma Andrews and Gianmarco Mengaldo deposited on arXiv a model evaluation benchmark^[1] built for a narrow question: when does text truly inform a numerical forecast.

The answer comes from six mutual information estimators. The work is archived as arXiv:2609.11282, within the Artificial Intelligence and Information Theory areas.

Multimodal forecasting models combine time series and text annotations, promising richer predictions through context. Verifying that contribution remains an open question in information theory. Until this work, a reference with known ground truth to measure it was missing.

The authors filled that gap with a synthetic signal. The generation process is controlled at every step, so the true information content is known with exactness.

Three annotation categories, one clean testing ground

The synthetic signal carries annotations divided into three categories: semantically correct, incorrect and irrelevant.

This tripartition is the most interesting part of the experimental design. A metric rewarding correct annotations could still confuse incorrect ones with irrelevant ones, and the distinction matters in practice.

Total control over the generative process produces a rare advantage in evaluation literature: the true number exists before measurement. The estimators are then judged against a reference, rather than against each other. This is the difference between a benchmark and a ranking.

The choice reduces the classic risk of automated evaluation: agreeing on a common error. With known ground truth, agreement between metrics becomes evidence, instead of an echo. Agreement among six different methods gains weight precisely for this reason.

The six methods and the families they belong to

The estimators tested are KSG, MINE, InfoNCE, CCA, PID and V-information.

They cover distinct theoretical families, and this variety is deliberate:

• KSG, geometric estimation based on nearest neighbors
• MINE and InfoNCE, neural estimates with variational and contrastive objectives
• CCA, canonical correlation between representations
• PID, decomposition of information into unique, redundant and synergistic parts
• V-information, information usable by a family of predictors

The main result is clear: all six indicate correct annotations as the most informative. Agreement across six different formulations strengthens the conclusion, because each one errs in its own way.

The paper also shows practical use: these metrics succeed in controlling the quality of mixed text corpora. They select annotations that produce the best downstream forecasting results, and they do so before any model training.

The savings are concrete. An informative audit costs a fraction of a training cycle, and the selection signal precedes the expense, rather than following it.

Seven real datasets and the gap on weak signals

The benchmark identifies the limits of each estimator, and these limits are validated on seven real datasets.

Here divergence appears. The estimators' performance differs on weak signals, and the difference follows the signal level.

This point deserves attention because it reverses the optimistic reading of the synthetic result. In clean conditions, with clearly correct or clearly irrelevant annotations, agreement is total. On real data, where text's contribution is marginal, the choice of estimator becomes a choice of merit.

Whoever selects a metric also chooses which weak signals they will see.

The structure of the phenomenon is the same one that separates a controlled score from a production measurement: the synthetic environment fixes the signal-to-noise ratio, while real data lets it vary.

The gap between controlled conditions and production

A benchmark measures standardized conditions. Production is the opposite of standardized.

The predictive validity of scores against real performance remains a separate dimension, and largely still poorly measured. This study is rare precisely because it addresses it: first the synthetic ground truth, then the seven real corpora, with limits exposed rather than smoothed over.

The evidence shows a regularity: agreement between methods holds where the signal is strong, and thins where the signal is weak. The weak zone is exactly where an analytics team needs an answer.

The gap between the clean case and the marginal case is where difficult decisions live. An audit that works on easy cases describes easy cases.

When evaluation measures the evidence instead of the model

Automated evaluation carries systematic bias, and these biases tend to be shared across similar tools.

This work's design addresses the problem at its root, because it ties every estimate to a generated truth, rather than to another model's judgment. This is a substantial epistemic difference.

On the parallel front of system evaluation, an analysis published on AlignmentForum documents how much a model can accomplish in the absence of visible chain of thought (the account is here^[2]). The underlying theme coincides: the observable trace and real behavior overlap less than the measurement procedure assumes.

A metric anchored to a known reference avoids this trap by construction. The cost is generality: the laboratory guarantees truth, and the real world returns it blurred.

What changes for those allocating budget

For an investment committee the result has a direct reading: text corpus auditing is mature enough to precede training.

Spending on multimodal fusion can be ordered by informative evidence, rather than by product intuition. Six concordant methods on clean data offer a selection criterion with a theoretical foundation.

For a chief analytics officer the consequence concerns infrastructure. Annotated corpora with metadata about signal quality are needed, because the choice of estimator depends on that quality. A pipeline that treats all annotations the same way loses the distinction the paper documents.

For a board the technology thesis supported by the data is narrow and precise: text adds value when its information content is measurable, and the measurement holds where the signal is strong.

The limits declared by the authors

The authors close with practical rules for annotation auditing and fusion selection.

Much remains outside the perimeter. The benchmark covers multimodal time series forecasting, and validity on other tasks remains to be demonstrated. The seven real datasets serve to validate limits, rather than to certify a ranking.

This is the diagnosis, and it stops here.

The work's value lies in the construction of the reference, more than in the score of any single metric. A field that measures text information now has a testing ground with known truth, dated September 10, 2026, and a map of the points where its tools diverge.

The question in the title remains open in marginal cases, and the paper declares this openly.