Machine learning reproducibility: the protocol is the result

The number that matters is in the protocol

On 16 September 2026 four researchers filed a study on arXiv covering sensorised insoles for motor activity recognition in elderly care. The highest reported value reaches 0.980 macro-F1^[1].

The relevant element lies elsewhere.

The authors use two distinct validation protocols, with separate purposes. A stratified 10-fold cross-validation serves to select the length of the time window and the family of candidate models. A participant-independent Stratified Group 5-fold cross-validation produces the performance estimate carried forward in the paper.

The distinction looks like a methodological footnote. Instead it decides what the final number actually describes: the model's ability to read the activity of a person never seen during training.

That is the question of interest to anyone funding the move from prototype to product.

What a random split measures

In an ordinary stratified cross-validation, windows from the same subject fall into both training and test. The plantar signals of one individual carry a stable signature: weight, stride length, load distribution, stance rhythm.

The model learns that signature and finds it again on the other side of the split.

The score that comes out describes the recognition of people already encountered, inside a task that claims to recognise activities. Overlapping windows make the effect worse, because two adjacent samples share part of the raw signal.

The Stratified Group scheme closes this route. All windows from a participant stay inside a single fold, so testing happens on subjects outside the training set. The estimate changes in kind: it moves from a measure of memory to a measure of generalisation.

The sample, with its exact numbers

The data comes from 15 healthy adults, sampled at 80 Hz and cut into overlapping windows. Each insole integrates 16 active plantar pressure sensing points and a six-dimensional inertial stream: acceleration and angular velocity on three axes.

There are four activity classes.

• sitting
• standing
• walking
• unstable walking

Under the participant-independent protocol, Histogram-Based Gradient Boosting reaches 0.954 macro-F1 on the left foot and 0.959 on the right. Bilateral sensing brings the value to 0.980.

The screening protocol and the estimation protocol stay separate, and this separation prevents the choice of time window from fitting itself to the test data.

The jump between a single foot and a pair of insoles is worth about two and a half points of macro-F1. It is a hardware difference, with a per-unit cost that doubles and user compliance that gets worse. Anyone assessing the device holds both entries together.

The simple model holds its own

The authors place a compact 1D-CNN alongside HGB, evaluated on the same participant-independent folds. The comparison returns p=0.0625, so the network's margin stays above the conventional 0.05 threshold.

With 15 participants the test has little power. The authors report the value instead of presenting it as a win, and that caution is the most useful methodological signal in the work.

The evidence shows a gradient boosting model on tabular features running level with a convolutional network, on a time-series task with a small sample. The fact has direct budget consequences: a gradient model runs on a microcontroller, needs little tuning and stays inspectable entry by entry.

The architectural choice, in cases like this one, is settled on running cost.

The limits the authors state

The sample consists of healthy adults. The clinical population the use case is aimed at, older people with reduced mobility, stays outside the data collection.

The authors choose a cautious formula and speak of a basis for activity monitoring and fall prevention, rather than announcing a ready tool. The distance between those two phrases covers the entire clinical validation phase.

Unstable walking is also a laboratory-induced condition. The instability that precedes a real fall arises from pathology, medication, fatigue, uneven surfaces. The current protocol covers a portion of that space.

A replication on an elderly cohort remains the missing step. The current protocol makes that comparison legible, because it fixes a baseline measured on subjects outside the training set.

At least three operational dimensions stay outside the measurement.

• monitoring duration beyond the controlled session
• sensor drift and insole wear
• variability of footwear and terrain

Why this concerns an investment committee

A benchmark measures standardised conditions. Production generates different ones with every deployment cycle, and the gap between the two situations is where almost all project failures live.

This paper offers a rare case: same pipeline, same data, two measurement protocols, two different meanings for the same score.

For an investment committee the reading stays direct. A technical due diligence that accepts the reported score buys the optimism of the more permissive protocol. A due diligence that asks for the definition of the folds buys information about generalisation.

The question to put to a vendor has a short form: how are the test splits built, and which variable separates the groups.

A vague answer on this point is worth as much as a missing score.

Measurement debt comes before data debt

Many analytics programmes treat validation as a final step, a formality after the model has been chosen. The correct order turns out to be the reverse: the evaluation protocol defines the question the model answers.

For a Chief Analytics Officer the consequence is infrastructural. Group metadata is needed inside the dataset from collection onwards: subject identifier, session, device, site, shift. Without those fields, a group-based split becomes impossible downstream.

The insole work demonstrates the point with rare economy. One line of protocol separates a showcase result from an estimate that holds up on contact with new people.

For a board the translation stays blunt. The technology thesis the data supports concerns the ability to measure, before the ability to model.

What has actually been measured

The work documents four motor states recognised across 15 healthy adults, with plantar and inertial sensors, under a validation that excludes the test subject from training. Within those boundaries the result holds.

Outside those boundaries begins the territory of the clinic, of duration and of everyday use.

The transferable lesson concerns method, and it applies to any machine learning system trained on data collected from people, sites or machines repeated over time. Whether a result replicates depends on the design of the split as much as it depends on the algorithm.

The score describes a protocol. Anyone who reads the score and ignores the protocol reads half the sentence.