M4 forecasting competition

Generated by beam 0.2.0 at 2026-07-09T16:02:45.232412+00:00.

Input

Normalization

• metric mase: min_max used an empirical upper bound; normalized values are not comparable across different method sets

Ranking

Robustness at a glance

The glyph table shows the per-metric normalized scores as circle size and the overall composite as a bar, alongside panels for how far each tool's rank moves under the robustness checks: the span across leave-one-dataset-out runs, the SMAA rank acceptability, and the span across the five aggregations. A tool with a tight rank span is a stable recommendation; a wide span means the order at that position depends on a choice the analyst made.

Sensitivity

The ranking above commits to one weighting (equal) and one aggregation (saw), the way a reader would apply the tool in practice. The sections here re-run it under the alternatives, so you can read how much the recommendation depends on those analyst choices rather than on the data.

SMAA weight sampling

Leave one metric out

The top-ranked tool kept its rank in 50 percent of the 2 leave-one-metric-out runs. Removing mase caused the largest rank change, a shift of 8.

Leave one dataset out

The top-ranked tool kept its rank in 83 percent of the 6 leave-one-dataset-out runs. Omitting dataset Hourly caused the largest rank change, a shift of 13.

Aggregation agreement

Re-ranking under the 5 aggregations gives orderings that agree at a mean Kendall tau-b of 0.90. Every aggregation ranks Pawlikowski first, so the top recommendation does not depend on the aggregation choice.

Normalization agreement

Re-ranking under 5 normalizations (recommended, min_max, log_min_max, rank, zscore) gives orderings that agree at a mean Kendall tau-b of 0.90. The recommended column is the per-metric default each card declares (smape: min_max, mase: min_max); the others apply one strategy to every metric. Every normalization ranks Pawlikowski first, so the top recommendation does not depend on the normalization choice.

What moves the ranking

Across the 120 combinations of weighting, aggregation and dataset, the rank variance splits into the dataset (96 percent), the weighting choice (0 percent), the aggregation choice (0 percent), and their interactions (3 percent). The largest is the dataset.

The top tool overall (Smyl) ranks first in 33 percent of combinations, with a rank span of 17. Its mean rank per dataset:

Specification curve

Each of the 120 combinations of weighting, aggregation and dataset is one specification. The method that ranks first most often (Smyl) ranks first in 33 percent of them. The single most common ordering of all tools holds in 8 percent of specifications, and 5 distinct tool(s) rank first in at least one specification. The curve below plots that method's rank across the specifications, sorted from its best rank to its worst; the panel beneath marks which choices each specification used.

Smallest weight perturbation

The top rank is stable: the smallest single-metric weight change that would overturn it is about 0.499 on mase.

Critical difference across datasets

Friedman test p-value 0.0001058 over 6 datasets. The Nemenyi critical difference is 15.543 average-rank units; tools closer than this are not separable at the chosen level.

Groups not significantly different:

• Montero-Manso, Pawlikowski, Smyl, Fiorucci, Petropoulos, Jaganathan, Doornik, Pedregal, Roubinchtein, Darin, Waheeb, Tartu M4 seminar, Spiliotis, Comb, ARIMA, Shaub, Legaki, ETS, Nikzad, Dantas, Damped, Ibrahim, Segura-Heras, Trotta, Theta

Effect size: the top-ranked method (Montero-Manso) scores higher than the next (Pawlikowski) on 5 of 6 datasets (probability of superiority 83 percent).

Consistency: the pairwise majorities carry 0 circular triads out of 2300, so no single order agrees with all of them and the reported ranking depends on the aggregation rule (though Montero-Manso is still preferred to every other method individually).

Posterior: under the Bayesian sign test (ROPE 0), the probability that Montero-Manso is practically better than Pawlikowski is 0.94, and the probability that the two are practically equivalent is 0.03.

Reproducibility

Input sha256: b1da4c264cb28c88b3e9d5dc19f010db67b33e680a89861d2cda3ced213fa562

The full run manifest, written alongside this report, records the metric card versions and content hashes needed to reproduce the run.