M4 forecasting competition

Author

Izaskun Mallona

Published

October 7, 2026

Provenance of the bundled table

The 100,000 series are not bundled. src/beam/data/M4_2018_by_frequency.csv is derived from the GPL-3 M4comp2018 data, with the realized values and the point forecasts of the top 25 methods. To reproduce, run:

git clone https://github.com/carlanetto/M4comp2018.git
cd M4comp2018 && git lfs pull          # the data is stored via git-lfs
# commit 3c75dcd25c72c631f04bff1a017d9917d0e7251c, R 4.3.3
Rscript reduce_m4.R                    # writes M4_2018_by_frequency.csv

reduce_m4.R computes the mean sMAPE and mean MASE per method per band (Smyl: sMAPE 11.374, MASE 1.536). The table is GPL-3. Reference: Makridakis, Spiliotis and Assimakopoulos (2020, 10.1016/j.ijforecast.2019.04.014).

Load the table

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

import beam
from beam.datasets import load_m4
from beam.cards import properties_for

m4 = load_m4()
print("methods:", len(m4.method_names), "(rank order, top first:", m4.method_names[0] + ")")
print("frequency bands:", m4.frequency_names)
print("series per band:", dict(zip(m4.frequency_names, m4.n_series.tolist())))
print("metrics:", m4.metric_ids)
methods: 25 (rank order, top first: Smyl)
frequency bands: ('Yearly', 'Quarterly', 'Monthly', 'Weekly', 'Daily', 'Hourly')
series per band: {'Yearly': 23000, 'Quarterly': 24000, 'Monthly': 48000, 'Weekly': 359, 'Daily': 4227, 'Hourly': 414}
metrics: ('smape', 'mase')

Metric cards

properties_for reads the smape and mase cards. Both are lower-is-better ratio metrics, normalized by min-max and pooled across datasets by arithmetic mean.

for p in properties_for(list(m4.metric_ids)):
    print(
        f"{p.id:6s}  polarity={p.polarity:15s}  scale={p.scale_type:6s}  "
        f"norm={p.recommended_normalization:8s}  across_datasets={p.recommended_aggregation_across_datasets}"
    )
smape   polarity=lower_is_better  scale=ratio   norm=min_max   across_datasets=arithmetic_mean
mase    polarity=lower_is_better  scale=ratio   norm=min_max   across_datasets=arithmetic_mean

card_data_consistency checks the raw scores against the ranges, baselines, targets and noise floors declared on the cards.

from beam.mcda import card_data_consistency, registry_context

ctx = registry_context(list(m4.metric_ids), "saw")
pooled_native = np.nanmean(m4.tensor(), axis=1)
audit = card_data_consistency(
    pooled_native, ctx.polarity, ctx.bounds,
    baselines=ctx.baselines, targets=ctx.targets, noise_floors=ctx.noise_floors,
    metric_ids=list(m4.metric_ids),
)
print("scores consistent with the cards:", audit.ok)
for finding in audit.findings:
    print(" ", finding.severity, finding.message)
scores consistent with the cards: True

Ranking

The tensor is complete. The run uses equal weights and SAW.

scores = beam.Scores(
    values=m4.tensor(),
    tool_names=m4.method_names,
    metric_ids=m4.metric_ids,
    dataset_names=m4.frequency_names,
    layout="long",
)
run = beam.rank(scores, weights="equal", method="saw", seed=0)
order = np.argsort(run.result.ranks)
print("top five under equal weights / SAW (each band weighted equally):")
for i in order[:5]:
    print(f"  {run.result.ranks[i]:>2d}  {m4.method_names[i]}")
top five under equal weights / SAW (each band weighted equally):
   1  Pawlikowski
   2  Montero-Manso
   3  Smyl
   4  Doornik
   5  Jaganathan

The official M4 ranking pools by OWA over all series, so the monthly and yearly bands (48,000 and 23,000 series) dominate. Here each band is one dataset and the six have equal weight.

Ranking by frequency band

sMAPE per band:

smape = m4.tensor(("smape",))[:, :, 0]
n_methods = len(m4.method_names)
n_bands = len(m4.frequency_names)

# Rank methods within each band by sMAPE (1 = lowest sMAPE on that band).
band_ranks = np.empty((n_methods, n_bands), dtype=int)
for b in range(n_bands):
    band_ranks[:, b] = np.argsort(np.argsort(smape[:, b])) + 1

from beam import plot

plot.rank_heatmap(
    band_ranks,
    row_names=m4.method_names,
    col_names=m4.frequency_names,
    row_label="forecasting method",
    col_label="frequency band",
    title="Per-band sMAPE rank (1 = lowest sMAPE on that band)",
)

Leave one frequency band out

beam.rank drops each band in turn, pools the other five and re-ranks.

lodo = run.leave_one_dataset_out
top_idx = int(np.argmin(run.result.ranks))
print(f"bands evaluated: {len(lodo.evaluated_datasets)} of {n_bands}")
print(f"most influential band: {lodo.dataset_names[lodo.most_influential_dataset]} "
      f"(largest rank shift {lodo.max_rank_shift})")
print()
order = np.argsort(run.result.ranks)
print(f"{'method':16s}  pooled rank  rank held across leave-one-band-out runs")
for i in order[:8]:
    print(f"{m4.method_names[i]:16s}  {run.result.ranks[i]:>4d}        {lodo.rank_stability[i] * 100:5.0f}%")
bands evaluated: 6 of 6
most influential band: Hourly (largest rank shift 13)

method            pooled rank  rank held across leave-one-band-out runs
Pawlikowski          1           83%
Montero-Manso        2           67%
Smyl                 3           67%
Doornik              4           50%
Jaganathan           5           83%
Tartu M4 seminar     6           50%
Fiorucci             7           33%
Petropoulos          8            0%
plot.dataset_stability(run)

Dataset concordance

dataset_concordance ranks the methods within each band and compares every pair of per-band orderings with Kendall tau-b.

conc = run.dataset_concordance
names = conc.dataset_names
print(f"mean agreement across bands (Kendall tau-b): {conc.mean_pairwise_tau:.2f}")
print("least typical band:", names[conc.most_idiosyncratic_dataset])
print("mutually consistent groups:",
      [tuple(names[d] for d in g) for g in conc.concordant_groups])
print("where methods depart most from their own average rank:")
for cell in conc.notable_cells[:5]:
    side = "lower" if cell.deviation > 0 else "higher"
    print(f"  {conc.tool_names[cell.tool]} on {names[cell.dataset]}: "
          f"rank {cell.rank}, {side} than its mean {cell.mean_rank:.1f}")
mean agreement across bands (Kendall tau-b): 0.21
least typical band: Daily
mutually consistent groups: [('Yearly',), ('Quarterly', 'Monthly'), ('Weekly',), ('Daily',), ('Hourly',)]
where methods depart most from their own average rank:
  Nikzad on Hourly: rank 2, higher than its mean 15.8
  Legaki on Yearly: rank 2, higher than its mean 15.5
  Ibrahim on Yearly: rank 6, higher than its mean 18.7
  Shaub on Yearly: rank 3, higher than its mean 15.5
  Darin on Daily: rank 25, lower than its mean 12.5
plot.dataset_concordance(run)

Rank of each method per band relative to its mean rank:

plot.dataset_struggle(run)

Funky heatmap

The funky heatmap of the run, with the rank span across the leave-one-band-out runs, the rank span across the five aggregations, and the SMAA rank-acceptability bar.

from beam.reporting import funky_heatmap_from_run

funky_heatmap_from_run(run, title="M4: scores and rank robustness")

Mixed-effects on sMAPE

A mixed-effects model splits the sMAPE variance into a band effect and a residual. It needs R’s lme4.

from beam.heterogeneity import mixed_effects_from_matrix, r_available

smape_matrix = m4.tensor(("smape",))[:, :, 0]

if r_available():
    me = mixed_effects_from_matrix(smape_matrix, m4.method_names, m4.frequency_names)
    print(f"band shift (ICC):  {me.icc_dataset:.2f} of the sMAPE variance")
    print(f"residual share:    {me.residual_share:.2f}")
else:
    print("R with lme4 not available; skipping the mixed-effects fit.")
band shift (ICC):  0.90 of the sMAPE variance
residual share:    0.10

A Bradley-Terry tree on the six bands, with the seasonal period as the splitting feature:

from beam.heterogeneity import bradley_terry_tree, bttree_available

seasonal_period = {
    "Yearly": 1.0, "Quarterly": 4.0, "Monthly": 12.0,
    "Weekly": 1.0, "Daily": 1.0, "Hourly": 24.0,
}
if bttree_available():
    bt = bradley_terry_tree(
        smape_matrix,
        m4.method_names,
        m4.frequency_names,
        numeric_features={"seasonal_period": [seasonal_period[b] for b in m4.frequency_names]},
        polarity="lower_is_better",
        minsize=2,
    )
    print(f"split found: {bt.did_split}")
    print(bt.summary())
else:
    print("R with psychotree not available; skipping the Bradley-Terry tree.")
split found: False
The Bradley-Terry tree found no dataset feature that splits the method ranking at alpha 0.05 over 6 datasets, so the ranking is reported as one Bradley-Terry model over all of them, led by Pawlikowski. With this many datasets the split test has few observations to work with, the same small-sample limit the critical-difference diagram shows; a benchmark with more datasets is where a split can appear.

Rank sensitivity: band or analyst choice

rank_sensitivity ranks under every combination of weighting scheme, aggregation rule and band, and splits the rank variance of each method into a fraction per factor.

from beam.mcda import rank_sensitivity

rs = rank_sensitivity(
    m4.tensor(),
    ctx.polarity,
    normalization=list(ctx.normalization),
    bounds=list(ctx.bounds),
    baselines=list(ctx.baselines),
    targets=list(ctx.targets),
    tool_names=m4.method_names,
    dataset_names=m4.frequency_names,
)
print(f"{rs.n_combinations} combinations of {len(rs.weightings)} weightings, "
      f"{len(rs.methods)} aggregations and {len(rs.dataset_names)} bands")
print(f"  band (dataset): {rs.dataset_share:.3f} of the rank variance")
print(f"  weighting:      {rs.weighting_share:.3f}")
print(f"  aggregation:    {rs.aggregation_share:.3f}")
print(f"  interactions:   {rs.interaction_share:.3f}")
print(f"  most influential factor: {rs.most_influential_factor}")
120 combinations of 4 weightings, 5 aggregations and 6 bands
  band (dataset): 0.963 of the rank variance
  weighting:      0.002
  aggregation:    0.003
  interactions:   0.032
  most influential factor: dataset
plot.rank_sensitivity(rs)

The band has 0.96 of the rank variance and each of the two choices under 0.01.

The same split per method, with each method’s rank range:

plot.rank_sensitivity_by_tool(rs)

headline = m4.method_names[rs.headline_tool]
print(f"top method overall: {headline}, rank 1 in {rs.headline_top_fraction * 100:.0f}% of combinations")
print(f"{'band':12s}  mean rank of {headline}")
for band, mean_rank in zip(rs.dataset_names, rs.headline_rank_by_dataset):
    print(f"{band:12s}  {mean_rank:.1f}")
top method overall: Smyl, rank 1 in 33% of combinations
band          mean rank of Smyl
Yearly        1.0
Quarterly     2.6
Monthly       1.0
Weekly        9.9
Daily         17.8
Hourly        5.4

specification_curve lists the rankings of the grid and how often the top method is the same. On the pooled matrix the grid has only the two choices.

from beam.mcda import specification_curve

curve = specification_curve(rs)
dom = curve.tool_names[curve.most_frequent_top_tool]
print(f"choices plus band ({curve.n_specifications} combinations): "
      f"{dom} first in {curve.most_frequent_top_fraction * 100:.0f}%, "
      f"{curve.n_distinct_top_tools} methods reach the top")

pooled = specification_curve(
    rank_sensitivity(
        run.matrix, ctx.polarity,
        normalization=list(ctx.normalization), bounds=list(ctx.bounds),
        baselines=list(ctx.baselines), targets=list(ctx.targets),
        tool_names=m4.method_names,
    )
)
pdom = pooled.tool_names[pooled.most_frequent_top_tool]
print(f"choices only ({pooled.n_specifications} combinations): "
      f"{pdom} first in {pooled.most_frequent_top_fraction * 100:.0f}%")

plot.specification_curve(curve)
choices plus band (120 combinations): Smyl first in 33%, 5 methods reach the top
choices only (20 combinations): Pawlikowski first in 90%

Blind analysis

A blind analysis fixes the pipeline before the method names are known. beam.blind masks the names and shuffles the rows; beam.unblind restores them. The seal fingerprint is recorded in the manifest.

from beam import blind, unblind

blinded, seal = blind(scores, seed=0)
blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)
restored = unblind(blind_run, seal)
named = beam.rank(scores, weights="equal", method="saw", seed=0, sensitivity=False)
print("ranking identical after unblinding:",
      dict(zip(named.tool_names, named.result.ranks))
      == dict(zip(restored.tool_names, restored.result.ranks)))
print("top method after unblinding:", restored.top_tool)
print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12])
ranking identical after unblinding: True
top method after unblinding: Pawlikowski
blinding fingerprint: 81d83ff65bdf

Pairwise superiority

pairwise_superiority counts how often one method outperforms another across the six bands on sMAPE. The sMAPE card declares no noise floor, so the equivalence band is zero.

from beam.mcda import pairwise_superiority

sup = pairwise_superiority(smape, "lower_is_better", method_names=m4.method_names)
top_method = m4.method_names[sup.order[0]]
print(f"highest standing: {top_method} ({sup.standing[sup.order[0]]:.2f})")
print(f"method pairs the sign test cannot separate: {len(sup.equivalent_pairs)} of {len(sup.per_pair)}")
top, runner = sup.order[0], sup.order[1]
pair = next(p for p in sup.per_pair if {p.a, p.b} == {int(top), int(runner)})
n_top = pair.a_outperforms if pair.a == top else pair.b_outperforms
print(f"{top_method} outperforms {m4.method_names[runner]} on {n_top} of {pair.n_compared} bands")
highest standing: Pawlikowski (0.84)
method pairs the sign test cannot separate: 238 of 300
Pawlikowski outperforms Smyl on 2 of 6 bands

With six bands the sign test has little power.

pairwise_transitivity tests whether the pairwise majorities agree with one order.

from beam.mcda import pairwise_transitivity

trans = pairwise_transitivity(sup)
print(f"transitive: {trans.is_transitive}; circular triads: {trans.n_circular_triads} of {trans.n_triads}")
transitive: False; circular triads: 2 of 2300

The matrix orders the methods by the number of others outperformed. A red cell below the diagonal is a method outperforming one ranked above it.

plot.pairwise_majority(trans)

bayesian_sign_comparison gives, for each pair, the posterior probability that one method is practically better.

from beam.mcda import bayesian_sign_comparison

bayes = bayesian_sign_comparison(sup)
decisive = sum(1 for p in bayes.per_pair if p.decision != "inconclusive")
print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(bayes.per_pair)}")
plot.bayesian_comparison(bayes)
pairs with a decisive posterior at 0.95: 62 of 300

Summary

With equal weight per frequency band and SAW over sMAPE and MASE, the band has 0.96 of the rank variance. The official competition pools by OWA weighted by the number of series. There the monthly and yearly bands dominate and the ES-RNN of Smyl ranks first.