The 100,000 series are not bundled. src/beam/data/M4_2018_by_frequency.csv is derived from the GPL-3 M4comp2018 data, with the realized values and the point forecasts of the top 25 methods. To reproduce, run:
git clone https://github.com/carlanetto/M4comp2018.git
cd M4comp2018 && git lfs pull # the data is stored via git-lfs
# commit 3c75dcd25c72c631f04bff1a017d9917d0e7251c, R 4.3.3
Rscript reduce_m4.R # writes M4_2018_by_frequency.csv
reduce_m4.R computes the mean sMAPE and mean MASE per method per band (Smyl: sMAPE 11.374, MASE 1.536). The table is GPL-3. Reference: Makridakis, Spiliotis and Assimakopoulos (2020, 10.1016/j.ijforecast.2019.04.014).
Load the table
%matplotlib inlineimport numpy as npimport matplotlib.pyplot as pltimport beamfrom beam.datasets import load_m4from beam.cards import properties_form4 = load_m4()print("methods:", len(m4.method_names), "(rank order, top first:", m4.method_names[0] +")")print("frequency bands:", m4.frequency_names)print("series per band:", dict(zip(m4.frequency_names, m4.n_series.tolist())))print("metrics:", m4.metric_ids)
methods: 25 (rank order, top first: Smyl)
frequency bands: ('Yearly', 'Quarterly', 'Monthly', 'Weekly', 'Daily', 'Hourly')
series per band: {'Yearly': 23000, 'Quarterly': 24000, 'Monthly': 48000, 'Weekly': 359, 'Daily': 4227, 'Hourly': 414}
metrics: ('smape', 'mase')
Metric cards
properties_for reads the smape and mase cards. Both are lower-is-better ratio metrics, normalized by min-max and pooled across datasets by arithmetic mean.
for p in properties_for(list(m4.metric_ids)):print(f"{p.id:6s} polarity={p.polarity:15s} scale={p.scale_type:6s} "f"norm={p.recommended_normalization:8s} across_datasets={p.recommended_aggregation_across_datasets}" )
smape polarity=lower_is_better scale=ratio norm=min_max across_datasets=arithmetic_mean
mase polarity=lower_is_better scale=ratio norm=min_max across_datasets=arithmetic_mean
from beam.mcda import card_data_consistency, registry_contextctx = registry_context(list(m4.metric_ids), "saw")pooled_native = np.nanmean(m4.tensor(), axis=1)audit = card_data_consistency( pooled_native, ctx.polarity, ctx.bounds, baselines=ctx.baselines, targets=ctx.targets, noise_floors=ctx.noise_floors, metric_ids=list(m4.metric_ids),)print("scores consistent with the cards:", audit.ok)for finding in audit.findings:print(" ", finding.severity, finding.message)
scores consistent with the cards: True
Ranking
The tensor is complete. The run uses equal weights and SAW.
scores = beam.Scores( values=m4.tensor(), tool_names=m4.method_names, metric_ids=m4.metric_ids, dataset_names=m4.frequency_names, layout="long",)run = beam.rank(scores, weights="equal", method="saw", seed=0)order = np.argsort(run.result.ranks)print("top five under equal weights / SAW (each band weighted equally):")for i in order[:5]:print(f" {run.result.ranks[i]:>2d}{m4.method_names[i]}")
top five under equal weights / SAW (each band weighted equally):
1 Pawlikowski
2 Montero-Manso
3 Smyl
4 Doornik
5 Jaganathan
The official M4 ranking pools by OWA over all series, so the monthly and yearly bands (48,000 and 23,000 series) dominate. Here each band is one dataset and the six have equal weight.
Ranking by frequency band
sMAPE per band:
smape = m4.tensor(("smape",))[:, :, 0]n_methods =len(m4.method_names)n_bands =len(m4.frequency_names)# Rank methods within each band by sMAPE (1 = lowest sMAPE on that band).band_ranks = np.empty((n_methods, n_bands), dtype=int)for b inrange(n_bands): band_ranks[:, b] = np.argsort(np.argsort(smape[:, b])) +1from beam import plotplot.rank_heatmap( band_ranks, row_names=m4.method_names, col_names=m4.frequency_names, row_label="forecasting method", col_label="frequency band", title="Per-band sMAPE rank (1 = lowest sMAPE on that band)",)
Leave one frequency band out
beam.rank drops each band in turn, pools the other five and re-ranks.
lodo = run.leave_one_dataset_outtop_idx =int(np.argmin(run.result.ranks))print(f"bands evaluated: {len(lodo.evaluated_datasets)} of {n_bands}")print(f"most influential band: {lodo.dataset_names[lodo.most_influential_dataset]} "f"(largest rank shift {lodo.max_rank_shift})")print()order = np.argsort(run.result.ranks)print(f"{'method':16s} pooled rank rank held across leave-one-band-out runs")for i in order[:8]:print(f"{m4.method_names[i]:16s}{run.result.ranks[i]:>4d}{lodo.rank_stability[i] *100:5.0f}%")
bands evaluated: 6 of 6
most influential band: Hourly (largest rank shift 13)
method pooled rank rank held across leave-one-band-out runs
Pawlikowski 1 83%
Montero-Manso 2 67%
Smyl 3 67%
Doornik 4 50%
Jaganathan 5 83%
Tartu M4 seminar 6 50%
Fiorucci 7 33%
Petropoulos 8 0%
plot.dataset_stability(run)
Dataset concordance
dataset_concordance ranks the methods within each band and compares every pair of per-band orderings with Kendall tau-b.
conc = run.dataset_concordancenames = conc.dataset_namesprint(f"mean agreement across bands (Kendall tau-b): {conc.mean_pairwise_tau:.2f}")print("least typical band:", names[conc.most_idiosyncratic_dataset])print("mutually consistent groups:", [tuple(names[d] for d in g) for g in conc.concordant_groups])print("where methods depart most from their own average rank:")for cell in conc.notable_cells[:5]: side ="lower"if cell.deviation >0else"higher"print(f" {conc.tool_names[cell.tool]} on {names[cell.dataset]}: "f"rank {cell.rank}, {side} than its mean {cell.mean_rank:.1f}")
mean agreement across bands (Kendall tau-b): 0.21
least typical band: Daily
mutually consistent groups: [('Yearly',), ('Quarterly', 'Monthly'), ('Weekly',), ('Daily',), ('Hourly',)]
where methods depart most from their own average rank:
Nikzad on Hourly: rank 2, higher than its mean 15.8
Legaki on Yearly: rank 2, higher than its mean 15.5
Ibrahim on Yearly: rank 6, higher than its mean 18.7
Shaub on Yearly: rank 3, higher than its mean 15.5
Darin on Daily: rank 25, lower than its mean 12.5
plot.dataset_concordance(run)
Rank of each method per band relative to its mean rank:
plot.dataset_struggle(run)
Funky heatmap
The funky heatmap of the run, with the rank span across the leave-one-band-out runs, the rank span across the five aggregations, and the SMAA rank-acceptability bar.
from beam.reporting import funky_heatmap_from_runfunky_heatmap_from_run(run, title="M4: scores and rank robustness")
Mixed-effects on sMAPE
A mixed-effects model splits the sMAPE variance into a band effect and a residual. It needs R’s lme4.
from beam.heterogeneity import mixed_effects_from_matrix, r_availablesmape_matrix = m4.tensor(("smape",))[:, :, 0]if r_available(): me = mixed_effects_from_matrix(smape_matrix, m4.method_names, m4.frequency_names)print(f"band shift (ICC): {me.icc_dataset:.2f} of the sMAPE variance")print(f"residual share: {me.residual_share:.2f}")else:print("R with lme4 not available; skipping the mixed-effects fit.")
band shift (ICC): 0.90 of the sMAPE variance
residual share: 0.10
A Bradley-Terry tree on the six bands, with the seasonal period as the splitting feature:
from beam.heterogeneity import bradley_terry_tree, bttree_availableseasonal_period = {"Yearly": 1.0, "Quarterly": 4.0, "Monthly": 12.0,"Weekly": 1.0, "Daily": 1.0, "Hourly": 24.0,}if bttree_available(): bt = bradley_terry_tree( smape_matrix, m4.method_names, m4.frequency_names, numeric_features={"seasonal_period": [seasonal_period[b] for b in m4.frequency_names]}, polarity="lower_is_better", minsize=2, )print(f"split found: {bt.did_split}")print(bt.summary())else:print("R with psychotree not available; skipping the Bradley-Terry tree.")
split found: False
The Bradley-Terry tree found no dataset feature that splits the method ranking at alpha 0.05 over 6 datasets, so the ranking is reported as one Bradley-Terry model over all of them, led by Pawlikowski. With this many datasets the split test has few observations to work with, the same small-sample limit the critical-difference diagram shows; a benchmark with more datasets is where a split can appear.
from beam.mcda import rank_sensitivityrs = rank_sensitivity( m4.tensor(), ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=m4.method_names, dataset_names=m4.frequency_names,)print(f"{rs.n_combinations} combinations of {len(rs.weightings)} weightings, "f"{len(rs.methods)} aggregations and {len(rs.dataset_names)} bands")print(f" band (dataset): {rs.dataset_share:.3f} of the rank variance")print(f" weighting: {rs.weighting_share:.3f}")print(f" aggregation: {rs.aggregation_share:.3f}")print(f" interactions: {rs.interaction_share:.3f}")print(f" most influential factor: {rs.most_influential_factor}")
120 combinations of 4 weightings, 5 aggregations and 6 bands
band (dataset): 0.963 of the rank variance
weighting: 0.002
aggregation: 0.003
interactions: 0.032
most influential factor: dataset
plot.rank_sensitivity(rs)
The band has 0.96 of the rank variance and each of the two choices under 0.01.
The same split per method, with each method’s rank range:
plot.rank_sensitivity_by_tool(rs)
headline = m4.method_names[rs.headline_tool]print(f"top method overall: {headline}, rank 1 in {rs.headline_top_fraction *100:.0f}% of combinations")print(f"{'band':12s} mean rank of {headline}")for band, mean_rank inzip(rs.dataset_names, rs.headline_rank_by_dataset):print(f"{band:12s}{mean_rank:.1f}")
top method overall: Smyl, rank 1 in 33% of combinations
band mean rank of Smyl
Yearly 1.0
Quarterly 2.6
Monthly 1.0
Weekly 9.9
Daily 17.8
Hourly 5.4
specification_curve lists the rankings of the grid and how often the top method is the same. On the pooled matrix the grid has only the two choices.
from beam.mcda import specification_curvecurve = specification_curve(rs)dom = curve.tool_names[curve.most_frequent_top_tool]print(f"choices plus band ({curve.n_specifications} combinations): "f"{dom} first in {curve.most_frequent_top_fraction *100:.0f}%, "f"{curve.n_distinct_top_tools} methods reach the top")pooled = specification_curve( rank_sensitivity( run.matrix, ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=m4.method_names, ))pdom = pooled.tool_names[pooled.most_frequent_top_tool]print(f"choices only ({pooled.n_specifications} combinations): "f"{pdom} first in {pooled.most_frequent_top_fraction *100:.0f}%")plot.specification_curve(curve)
choices plus band (120 combinations): Smyl first in 33%, 5 methods reach the top
choices only (20 combinations): Pawlikowski first in 90%
Blind analysis
A blind analysis fixes the pipeline before the method names are known. beam.blind masks the names and shuffles the rows; beam.unblind restores them. The seal fingerprint is recorded in the manifest.
from beam import blind, unblindblinded, seal = blind(scores, seed=0)blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)restored = unblind(blind_run, seal)named = beam.rank(scores, weights="equal", method="saw", seed=0, sensitivity=False)print("ranking identical after unblinding:",dict(zip(named.tool_names, named.result.ranks))==dict(zip(restored.tool_names, restored.result.ranks)))print("top method after unblinding:", restored.top_tool)print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12])
ranking identical after unblinding: True
top method after unblinding: Pawlikowski
blinding fingerprint: 81d83ff65bdf
Pairwise superiority
pairwise_superiority counts how often one method outperforms another across the six bands on sMAPE. The sMAPE card declares no noise floor, so the equivalence band is zero.
from beam.mcda import pairwise_superioritysup = pairwise_superiority(smape, "lower_is_better", method_names=m4.method_names)top_method = m4.method_names[sup.order[0]]print(f"highest standing: {top_method} ({sup.standing[sup.order[0]]:.2f})")print(f"method pairs the sign test cannot separate: {len(sup.equivalent_pairs)} of {len(sup.per_pair)}")top, runner = sup.order[0], sup.order[1]pair =next(p for p in sup.per_pair if {p.a, p.b} == {int(top), int(runner)})n_top = pair.a_outperforms if pair.a == top else pair.b_outperformsprint(f"{top_method} outperforms {m4.method_names[runner]} on {n_top} of {pair.n_compared} bands")
highest standing: Pawlikowski (0.84)
method pairs the sign test cannot separate: 238 of 300
Pawlikowski outperforms Smyl on 2 of 6 bands
from beam.mcda import pairwise_transitivitytrans = pairwise_transitivity(sup)print(f"transitive: {trans.is_transitive}; circular triads: {trans.n_circular_triads} of {trans.n_triads}")
transitive: False; circular triads: 2 of 2300
The matrix orders the methods by the number of others outperformed. A red cell below the diagonal is a method outperforming one ranked above it.
plot.pairwise_majority(trans)
bayesian_sign_comparison gives, for each pair, the posterior probability that one method is practically better.
from beam.mcda import bayesian_sign_comparisonbayes = bayesian_sign_comparison(sup)decisive =sum(1for p in bayes.per_pair if p.decision !="inconclusive")print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(bayes.per_pair)}")plot.bayesian_comparison(bayes)
pairs with a decisive posterior at 0.95: 62 of 300
Summary
With equal weight per frequency band and SAW over sMAPE and MASE, the band has 0.96 of the rank variance. The official competition pools by OWA weighted by the number of series. There the monthly and yearly bands dominate and the ES-RNN of Smyl ranks first.
Source Code
---title: "M4 forecasting competition"author: "Izaskun Mallona"date: todayformat: html: theme: cosmo toc: true toc-location: left embed-resources: true code-tools: true fig-width: 6 fig-height: 3.5---## Provenance of the bundled tableThe 100,000 series are not bundled. `src/beam/data/M4_2018_by_frequency.csv` is derived from the GPL-3 `M4comp2018` data, with the realized values and the point forecasts of the top 25 methods. To reproduce, run:```git clone https://github.com/carlanetto/M4comp2018.gitcd M4comp2018 && git lfs pull # the data is stored via git-lfs# commit 3c75dcd25c72c631f04bff1a017d9917d0e7251c, R 4.3.3Rscript reduce_m4.R # writes M4_2018_by_frequency.csv````reduce_m4.R` computes the mean sMAPE and mean MASE per method per band (Smyl: sMAPE 11.374, MASE 1.536). The table is GPL-3. Reference: Makridakis, Spiliotis and Assimakopoulos (2020, [10.1016/j.ijforecast.2019.04.014](https://doi.org/10.1016/j.ijforecast.2019.04.014)).## Load the table```{python}%matplotlib inlineimport numpy as npimport matplotlib.pyplot as pltimport beamfrom beam.datasets import load_m4from beam.cards import properties_form4 = load_m4()print("methods:", len(m4.method_names), "(rank order, top first:", m4.method_names[0] +")")print("frequency bands:", m4.frequency_names)print("series per band:", dict(zip(m4.frequency_names, m4.n_series.tolist())))print("metrics:", m4.metric_ids)```## Metric cards[`properties_for`](../../docs/reference/properties_for.qmd) reads the `smape` and `mase` cards. Both are lower-is-better ratio metrics, normalized by min-max and pooled across datasets by arithmetic mean.```{python}for p in properties_for(list(m4.metric_ids)):print(f"{p.id:6s} polarity={p.polarity:15s} scale={p.scale_type:6s} "f"norm={p.recommended_normalization:8s} across_datasets={p.recommended_aggregation_across_datasets}" )```[`card_data_consistency`](../../docs/explanations/card-data-consistency.md) checks the raw scores against the ranges, baselines, targets and [noise floors](../../docs/explanations/reference-levels.md) declared on the cards.```{python}from beam.mcda import card_data_consistency, registry_contextctx = registry_context(list(m4.metric_ids), "saw")pooled_native = np.nanmean(m4.tensor(), axis=1)audit = card_data_consistency( pooled_native, ctx.polarity, ctx.bounds, baselines=ctx.baselines, targets=ctx.targets, noise_floors=ctx.noise_floors, metric_ids=list(m4.metric_ids),)print("scores consistent with the cards:", audit.ok)for finding in audit.findings:print(" ", finding.severity, finding.message)```## RankingThe tensor is complete. The run uses equal weights and SAW.```{python}scores = beam.Scores( values=m4.tensor(), tool_names=m4.method_names, metric_ids=m4.metric_ids, dataset_names=m4.frequency_names, layout="long",)run = beam.rank(scores, weights="equal", method="saw", seed=0)order = np.argsort(run.result.ranks)print("top five under equal weights / SAW (each band weighted equally):")for i in order[:5]:print(f" {run.result.ranks[i]:>2d}{m4.method_names[i]}")```The official M4 ranking pools by OWA over all series, so the monthly and yearly bands (48,000 and 23,000 series) dominate. Here each band is one dataset and the six have equal weight.## Ranking by frequency bandsMAPE per band:```{python}smape = m4.tensor(("smape",))[:, :, 0]n_methods =len(m4.method_names)n_bands =len(m4.frequency_names)# Rank methods within each band by sMAPE (1 = lowest sMAPE on that band).band_ranks = np.empty((n_methods, n_bands), dtype=int)for b inrange(n_bands): band_ranks[:, b] = np.argsort(np.argsort(smape[:, b])) +1from beam import plotplot.rank_heatmap( band_ranks, row_names=m4.method_names, col_names=m4.frequency_names, row_label="forecasting method", col_label="frequency band", title="Per-band sMAPE rank (1 = lowest sMAPE on that band)",)```## Leave one frequency band out`beam.rank` drops each band in turn, pools the other five and re-ranks.```{python}lodo = run.leave_one_dataset_outtop_idx =int(np.argmin(run.result.ranks))print(f"bands evaluated: {len(lodo.evaluated_datasets)} of {n_bands}")print(f"most influential band: {lodo.dataset_names[lodo.most_influential_dataset]} "f"(largest rank shift {lodo.max_rank_shift})")print()order = np.argsort(run.result.ranks)print(f"{'method':16s} pooled rank rank held across leave-one-band-out runs")for i in order[:8]:print(f"{m4.method_names[i]:16s}{run.result.ranks[i]:>4d}{lodo.rank_stability[i] *100:5.0f}%")``````{python}plot.dataset_stability(run)```## Dataset concordance[`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) ranks the methods within each band and compares every pair of per-band orderings with Kendall tau-b.```{python}conc = run.dataset_concordancenames = conc.dataset_namesprint(f"mean agreement across bands (Kendall tau-b): {conc.mean_pairwise_tau:.2f}")print("least typical band:", names[conc.most_idiosyncratic_dataset])print("mutually consistent groups:", [tuple(names[d] for d in g) for g in conc.concordant_groups])print("where methods depart most from their own average rank:")for cell in conc.notable_cells[:5]: side ="lower"if cell.deviation >0else"higher"print(f" {conc.tool_names[cell.tool]} on {names[cell.dataset]}: "f"rank {cell.rank}, {side} than its mean {cell.mean_rank:.1f}")``````{python}plot.dataset_concordance(run)```Rank of each method per band relative to its mean rank:```{python}plot.dataset_struggle(run)```## Funky heatmapThe [funky heatmap](../../docs/explanations/funky-heatmaps-and-robustness.md) of the run, with the rank span across the leave-one-band-out runs, the rank span across the five aggregations, and the SMAA rank-acceptability bar.```{python}from beam.reporting import funky_heatmap_from_runfunky_heatmap_from_run(run, title="M4: scores and rank robustness")```## Mixed-effects on sMAPEA [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) splits the sMAPE variance into a band effect and a residual. It needs R's lme4.```{python}from beam.heterogeneity import mixed_effects_from_matrix, r_availablesmape_matrix = m4.tensor(("smape",))[:, :, 0]if r_available(): me = mixed_effects_from_matrix(smape_matrix, m4.method_names, m4.frequency_names)print(f"band shift (ICC): {me.icc_dataset:.2f} of the sMAPE variance")print(f"residual share: {me.residual_share:.2f}")else:print("R with lme4 not available; skipping the mixed-effects fit.")```A [Bradley-Terry tree](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees) on the six bands, with the seasonal period as the splitting feature:```{python}from beam.heterogeneity import bradley_terry_tree, bttree_availableseasonal_period = {"Yearly": 1.0, "Quarterly": 4.0, "Monthly": 12.0,"Weekly": 1.0, "Daily": 1.0, "Hourly": 24.0,}if bttree_available(): bt = bradley_terry_tree( smape_matrix, m4.method_names, m4.frequency_names, numeric_features={"seasonal_period": [seasonal_period[b] for b in m4.frequency_names]}, polarity="lower_is_better", minsize=2, )print(f"split found: {bt.did_split}")print(bt.summary())else:print("R with psychotree not available; skipping the Bradley-Terry tree.")```## Rank sensitivity: band or analyst choice[`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) ranks under every combination of [weighting scheme](../../docs/explanations/weighting-schemes.md), [aggregation rule](../../docs/explanations/aggregation-methods.md) and band, and splits the rank variance of each method into a fraction per factor.```{python}from beam.mcda import rank_sensitivityrs = rank_sensitivity( m4.tensor(), ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=m4.method_names, dataset_names=m4.frequency_names,)print(f"{rs.n_combinations} combinations of {len(rs.weightings)} weightings, "f"{len(rs.methods)} aggregations and {len(rs.dataset_names)} bands")print(f" band (dataset): {rs.dataset_share:.3f} of the rank variance")print(f" weighting: {rs.weighting_share:.3f}")print(f" aggregation: {rs.aggregation_share:.3f}")print(f" interactions: {rs.interaction_share:.3f}")print(f" most influential factor: {rs.most_influential_factor}")``````{python}plot.rank_sensitivity(rs)```The band has 0.96 of the rank variance and each of the two choices under 0.01.The same split per method, with each method's rank range:```{python}plot.rank_sensitivity_by_tool(rs)``````{python}headline = m4.method_names[rs.headline_tool]print(f"top method overall: {headline}, rank 1 in {rs.headline_top_fraction *100:.0f}% of combinations")print(f"{'band':12s} mean rank of {headline}")for band, mean_rank inzip(rs.dataset_names, rs.headline_rank_by_dataset):print(f"{band:12s}{mean_rank:.1f}")```[`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) lists the rankings of the grid and how often the top method is the same. On the pooled matrix the grid has only the two choices.```{python}from beam.mcda import specification_curvecurve = specification_curve(rs)dom = curve.tool_names[curve.most_frequent_top_tool]print(f"choices plus band ({curve.n_specifications} combinations): "f"{dom} first in {curve.most_frequent_top_fraction *100:.0f}%, "f"{curve.n_distinct_top_tools} methods reach the top")pooled = specification_curve( rank_sensitivity( run.matrix, ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=m4.method_names, ))pdom = pooled.tool_names[pooled.most_frequent_top_tool]print(f"choices only ({pooled.n_specifications} combinations): "f"{pdom} first in {pooled.most_frequent_top_fraction *100:.0f}%")plot.specification_curve(curve)```## Blind analysisA [blind analysis](../../docs/explanations/analysis-blinding.md) fixes the pipeline before the method names are known. `beam.blind` masks the names and shuffles the rows; `beam.unblind` restores them. The seal fingerprint is recorded in the manifest.```{python}from beam import blind, unblindblinded, seal = blind(scores, seed=0)blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)restored = unblind(blind_run, seal)named = beam.rank(scores, weights="equal", method="saw", seed=0, sensitivity=False)print("ranking identical after unblinding:",dict(zip(named.tool_names, named.result.ranks))==dict(zip(restored.tool_names, restored.result.ranks)))print("top method after unblinding:", restored.top_tool)print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12])```## Pairwise superiority[`pairwise_superiority`](../../docs/explanations/pairwise-method-comparison.md) counts how often one method outperforms another across the six bands on sMAPE. The sMAPE card declares no noise floor, so the equivalence band is zero.```{python}from beam.mcda import pairwise_superioritysup = pairwise_superiority(smape, "lower_is_better", method_names=m4.method_names)top_method = m4.method_names[sup.order[0]]print(f"highest standing: {top_method} ({sup.standing[sup.order[0]]:.2f})")print(f"method pairs the sign test cannot separate: {len(sup.equivalent_pairs)} of {len(sup.per_pair)}")top, runner = sup.order[0], sup.order[1]pair =next(p for p in sup.per_pair if {p.a, p.b} == {int(top), int(runner)})n_top = pair.a_outperforms if pair.a == top else pair.b_outperformsprint(f"{top_method} outperforms {m4.method_names[runner]} on {n_top} of {pair.n_compared} bands")```With six bands the sign test has little power.[`pairwise_transitivity`](../../docs/explanations/pairwise-method-comparison.md#transitivity) tests whether the pairwise majorities agree with one order.```{python}from beam.mcda import pairwise_transitivitytrans = pairwise_transitivity(sup)print(f"transitive: {trans.is_transitive}; circular triads: {trans.n_circular_triads} of {trans.n_triads}")```The matrix orders the methods by the number of others outperformed. A red cell below the diagonal is a method outperforming one ranked above it.```{python}plot.pairwise_majority(trans)```[`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) gives, for each pair, the posterior probability that one method is practically better.```{python}from beam.mcda import bayesian_sign_comparisonbayes = bayesian_sign_comparison(sup)decisive =sum(1for p in bayes.per_pair if p.decision !="inconclusive")print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(bayes.per_pair)}")plot.bayesian_comparison(bayes)```## SummaryWith equal weight per frequency band and SAW over sMAPE and MASE, the band has 0.96 of the rank variance. The official competition pools by OWA weighted by the number of series. There the monthly and yearly bands dominate and the ES-RNN of Smyl ranks first.