A made-up transportation example. The modes of transport are the methods and the terrains are the datasets. Each mode has a speed (km/h, higher is better), a cost (per km, lower is better) and CO2 (g per km, lower is better) per terrain. The numbers are illustrative. The three metrics have metric cards (speed, cost, co2).
Set-up
The five aggregation methods (SAW, TOPSIS, VIKOR, PROMETHEE II, COMET) come from pymcdm, called on the matrix normalized by beam. The weighting schemes are beam’s own.
Speed per mode and terrain. A cross marks a mode that cannot run on a terrain, and a red box the fastest mode on each terrain.
from beam import plotspeed = tb.metric("speed")n_modes =len(tb.mode_names)n_terrains =len(tb.terrain_names)plot.score_heatmap( speed, row_names=tb.mode_names, col_names=tb.terrain_names, row_label="transport mode (method axis)", col_label="terrain (dataset axis)", value_label="speed (km/h, log scale)", log=True, highlight_best_per_col=True, higher_is_better=True, title="Speed by mode and terrain; red box marks the fastest mode",)
No mode runs on every terrain, so a ranking pooled across terrains compares modes on different subsets.
feasible = tb.feasible()runs_everywhere = [ tb.mode_names[i] for i inrange(n_modes) if feasible[i].all()]print("modes that run on every terrain:", runs_everywhere or"none")print()print("fastest mode per terrain, by speed, over the feasible modes (the ground truth):")for t, terrain inenumerate(tb.terrain_names): best = tb.mode_names[int(np.nanargmax(speed[:, t]))]print(f" {terrain:15s}{best}")
modes that run on every terrain: none
fastest mode per terrain, by speed, over the feasible modes (the ground truth):
flat_road train
mud motorcycle
uphill motorcycle
open_water boat
long_distance plane
urban_hop train
Ranking per terrain
On each terrain the infeasible modes are dropped, with no imputation and no pooling across terrains. feasible_submatrix returns the complete score matrix, and run_from_registry reads polarity and normalization from the cards.
The cards declare the normalization: log_min_max for speed and cost, which span orders of magnitude, and rank for CO2, which has true zeros (walking, cycling).
Equal weights and SAW:
print("per-terrain top-ranked mode, equal weights, SAW")print(f"{'terrain':15s}{'top-ranked':12s}{'fastest by speed':16s} same?")for terrain in tb.terrain_names: names, sub = tb.feasible_submatrix(terrain) out = run_from_registry(sub, list(tb.metric_names), method="saw") top = names[int(np.argmin(out.ranks))] fastest = names[int(np.argmax(sub[:, 0]))]print(f" {terrain:15s}{top:12s}{fastest:16s}{top == fastest}")
Trail running is slower than road running on the flat road and faster on mud and uphill. An e-bike is faster than a bicycle on the flat road and the urban hop and slower uphill. Both modes of each pair run on these terrains.
speed = tb.metric("speed")m = tb.mode_names.indext = tb.terrain_names.indexprint("trail running vs road running, speed (km/h):")print(f"{'terrain':12s}{'road':>6s}{'trail':>6s} faster")for terrain in ["flat_road", "mud", "uphill"]: road = speed[m("running"), t(terrain)] trail = speed[m("trail_running"), t(terrain)] faster ="trail"if trail > road else"road"print(f"{terrain:12s}{road:6.1f}{trail:6.1f}{faster}")print()print("e-bike vs bicycle, speed (km/h):")print(f"{'terrain':12s}{'bike':>6s}{'ebike':>6s} faster")for terrain in ["flat_road", "urban_hop", "uphill"]: bike = speed[m("bicycle"), t(terrain)] ebike = speed[m("e_bike"), t(terrain)] faster ="ebike"if ebike > bike else"bike"print(f"{terrain:12s}{bike:6.1f}{ebike:6.1f}{faster}")
Every mode is feasible on the long-distance terrain. The five aggregations under the speed-heavy weighting:
names, sub = tb.feasible_submatrix("long_distance")methods = ["saw", "topsis", "vikor", "promethee_ii", "comet"]ranks_by_method = {}print(f"{'method':14s} ranks by mode (1 = best)")for m in methods: out = run_from_registry(sub, list(tb.metric_names), weights=speed_heavy, method=m) ranks_by_method[m] = out.ranks by_mode = {n: int(r) for n, r inzip(names, out.ranks)}print(f" {m:14s}{by_mode}")print()print("top-ranked mode per method:")for m in methods:print(f" {m:14s}{names[int(np.argmin(ranks_by_method[m]))]}")
The same ranks as a heatmap, one column per mode and one row per aggregation method:
grid = np.array([ranks_by_method[m] for m in methods])plot.rank_heatmap( grid, row_names=methods, col_names=names, row_label="aggregation method", col_label="transport mode", title="Long-distance rank by mode and method (speed-heavy weights)",)
Four weightings on the same terrain
With SAW on the long-distance terrain, equal, entropy, standard deviation and CRITIC weights:
print(f"{'weighting':10s}{'weights (speed, cost, co2)':28s} top-ranked")for wname in ["equal", "entropy", "std", "critic"]: out = run_from_registry(sub, list(tb.metric_names), weights=wname, method="saw") w_str = np.array2string(np.round(out.weights, 3), separator=", ") top = names[int(np.argmin(out.ranks))]print(f" {wname:10s}{w_str:28s}{top}")
MEREC takes the logarithm of the normalized scores, so it stops on the zero produced by rank on CO2. Here run is called directly with z-score normalization:
A critical-difference diagram needs a complete tool by dataset table. The four ground modes (foot, running, bicycle, motorcycle) are feasible on five terrains (flat road, mud, uphill, long distance, urban hop). common_feasible_block returns these complete cases. The diagram is on speed.
block_modes = ("foot", "running", "bicycle", "motorcycle")common_terrains, block = tb.common_feasible_block(block_modes)print("block modes: ", block_modes)print("common terrains: ", common_terrains)cd = critical_difference( block, higher_is_better=True, tool_names=block_modes)print(f"Friedman statistic = {cd.friedman_statistic:.3f}, p = {cd.friedman_pvalue:.4f}")print(f"critical difference (alpha={cd.alpha}) = {cd.critical_difference:.3f}")print("average ranks (1 = best, fastest):")for i in cd.order:print(f" {block_modes[i]:12s}{cd.average_ranks[i]:.2f}")groups = [tuple(block_modes[i] for i in c) for c in cd.cliques]print("cliques (not separable at alpha):", groups or"none")
SMAA samples weight vectors from a Dirichlet and reports the fraction of samples with each mode ranked first, here on the long-distance terrain with TOPSIS.
smaa_report = smaa( sub, tb.polarity, n_samples=500, method="topsis", seed=0,)print("confidence factor (fraction of sampled weightings ranking the mode first):")for n, c inzip(names, smaa_report.confidence_factor):print(f" {n:12s}{c:.3f}")most = names[int(np.argmax(smaa_report.confidence_factor))]print(f"\nmost confident top-ranked mode: {most} "f"({smaa_report.confidence_factor.max():.2f} of samples)")
confidence factor (fraction of sampled weightings ranking the mode first):
foot 0.000
running 0.178
trail_running 0.000
bicycle 0.102
e_bike 0.000
motorcycle 0.000
train 0.366
kayak 0.000
boat 0.000
plane 0.354
most confident top-ranked mode: train (0.37 of samples)
Weight perturbation on one terrain
smallest_weight_perturbation reports, for every pair of modes under SAW, the smallest single-weight change swapping them, here on the long-distance terrain with equal weights.
ts = smallest_weight_perturbation( sub, tb.polarity, weights="equal", method="saw",)print("base ranks under equal weights, SAW:")print(" ", {n: int(r) for n, r inzip(names, ts.base.ranks)})if ts.most_fragile_pair isnotNone: p = ts.most_fragile_pairprint(f"\nmost fragile pair: {names[p.higher]} ranked above {names[p.lower]}, "f"flips by changing the weight on {tb.metric_names[p.criterion]!r} "f"by {p.delta:+.4f}" )print(f"top rank is fragile: {ts.top_rank_is_fragile}")
base ranks under equal weights, SAW:
{'foot': 4, 'running': 2, 'trail_running': 3, 'bicycle': 5, 'e_bike': 6, 'motorcycle': 9, 'train': 1, 'kayak': 7, 'boat': 10, 'plane': 8}
most fragile pair: kayak ranked above plane, flips by changing the weight on 'speed' by +0.0053
top rank is fragile: False
Leave one dataset out
The kayak, the motorboat and the small plane all run on open water and the long distance, so they are complete cases over two terrains.
water_modes = ("kayak", "boat", "plane")water_terrains = ("open_water", "long_distance")mode_idx = [tb.mode_names.index(m) for m in water_modes]terrain_idx = [tb.terrain_names.index(t) for t in water_terrains]water_block = tb.scores[np.ix_(mode_idx, terrain_idx, range(len(tb.metric_names)))]print("speed (km/h) by mode and terrain:")print(f"{'mode':8s}{'open_water':>11s}{'long_distance':>13s}")for i, mode inenumerate(water_modes):print(f"{mode:8s}{water_block[i, 0, 0]:11.0f}{water_block[i, 1, 0]:13.0f}")
speed (km/h) by mode and terrain:
mode open_water long_distance
kayak 6 8
boat 30 40
plane 18 750
The two terrains are pooled with speed-heavy weights and SAW, by the arithmetic mean of the raw scores, and each terrain is dropped in turn.
rules = ("arithmetic_mean",) *len(tb.metric_names)lodo = leave_one_dataset_out( water_block, tb.polarity, rules, dataset_names=water_terrains, metric_ids=tb.metric_names, weights=speed_heavy, method="saw", normalization=list(tb.normalization),)print("pooled over both terrains:", {m: int(r) for m, r inzip(water_modes, lodo.base.ranks)})for d, r in lodo.leave_one_out.items():print(f"drop {water_terrains[d]:14s}:", {m: int(rk) for m, rk inzip(water_modes, r.ranks)})print()print("rank held across the leave-one-dataset-out runs:")for m, s inzip(water_modes, lodo.rank_stability):print(f" {m:8s}{s *100:5.0f}%")print(f"most influential terrain: {water_terrains[lodo.most_influential_dataset]} "f"(largest rank shift {lodo.max_rank_shift})")
pooled over both terrains: {'kayak': 3, 'boat': 2, 'plane': 1}
drop open_water : {'kayak': 3, 'boat': 2, 'plane': 1}
drop long_distance : {'kayak': 3, 'boat': 1, 'plane': 2}
rank held across the leave-one-dataset-out runs:
kayak 100%
boat 50%
plane 50%
most influential terrain: long_distance (largest rank shift 1)
order = np.argsort(lodo.base.ranks)fig, ax = plt.subplots(figsize=(5.8, 2.6))ax.barh(range(len(water_modes)), lodo.rank_stability[order] *100, color="#3a7ca5")ax.set_yticks(range(len(water_modes)))ax.set_yticklabels([water_modes[i] for i in order])ax.invert_yaxis()ax.set_xlim(0, 100)ax.set_xlabel("rank held across the 2 leave-one-dataset-out runs (percent)")ax.set_ylabel("transport mode (ordered by pooled rank)")ax.set_title("Water modes: leave-one-terrain-out rank stability (speed-heavy weights)")fig.tight_layout()plt.show()
Pooled over both terrains the plane ranks first. On open water alone the motorboat ranks first, and on the long distance alone the plane.
bayesian_sign_comparison on the water modes. With two terrains the posterior is close to the prior.
from beam.mcda import bayesian_sign_comparisonwater_sup = pairwise_superiority(water_block[:, :, 0], "higher_is_better", method_names=water_modes)water_bayes = bayesian_sign_comparison(water_sup)decisive =sum(1for p in water_bayes.per_pair if p.decision !="inconclusive")print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(water_bayes.per_pair)}")plot.bayesian_comparison(water_bayes)
pairs with a decisive posterior at 0.95: 0 of 3
The funky heatmap of the water modes, with the leave-one-dataset-out span and the aggregation span.
ranking identical after unblinding: True
blinding fingerprint: 8f8559f8d7ed
Dataset concordance
dataset_concordance ranks the modes within each terrain and reports the Kendall tau-b agreement between every pair of terrains, for the ground modes on five terrains and the water modes on two.
land_mode_idx = [tb.mode_names.index(m) for m in block_modes]land_terrain_idx = [tb.terrain_names.index(t) for t in common_terrains]land_block = tb.scores[np.ix_(land_mode_idx, land_terrain_idx, range(len(tb.metric_names)))]land = beam.Scores( values=land_block, tool_names=block_modes, metric_ids=tb.metric_names, dataset_names=common_terrains, layout="long",)land_run = beam.rank(land, weights="equal", method="saw")land_conc = land_run.dataset_concordancewater_conc = water_run.dataset_concordanceprint(f"land block, {len(common_terrains)} terrains: mean agreement (tau-b) {land_conc.mean_pairwise_tau:.2f}")print(f"water block, 2 terrains: mean agreement (tau-b) {water_conc.mean_pairwise_tau:.2f}")
land block, 5 terrains: mean agreement (tau-b) 1.00
water block, 2 terrains: mean agreement (tau-b) -0.33
plot.dataset_concordance(land_run)
Rank of each mode per terrain minus its mean rank:
plot.dataset_struggle(water_run)
Mixed-effects on speed
The mixed-effects model on speed, with the mode as a fixed effect and the terrain as a random intercept. It needs R’s lme4.
from beam.heterogeneity import mixed_effects_from_matrix, r_availablespeed = tb.scores[:, :, 0]keep =~np.isnan(speed).all(axis=1)speed_modes = [m for m, k inzip(tb.mode_names, keep) if k]if r_available(): me = mixed_effects_from_matrix(speed[keep], speed_modes, tb.terrain_names)print(f"terrain shift (ICC): {me.icc_dataset:.2f} of the speed variance")print(f"residual share: {me.residual_share:.2f} (mode-by-terrain interaction plus noise)")else:print("R with lme4 not available; skipping the mixed-effects fit.")
terrain shift (ICC): 0.10 of the speed variance
residual share: 0.90 (mode-by-terrain interaction plus noise)
Source Code
---title: "Transportation: a cross-domain MCDA example"author: "Izaskun Mallona"date: todayformat: html: theme: cosmo toc: true toc-location: left embed-resources: true code-tools: true fig-width: 6 fig-height: 3.5---## DataA made-up transportation example. The modes of transport are the methods and the terrains are the datasets. Each mode has a speed (km/h, higher is better), a cost (per km, lower is better) and CO2 (g per km, lower is better) per terrain. The numbers are illustrative. The three metrics have [metric cards](../../docs/explanations/cards-and-pipeline.qmd) (`speed`, `cost`, `co2`).## Set-upThe five [aggregation methods](../../docs/explanations/aggregation-methods.md) (SAW, TOPSIS, VIKOR, PROMETHEE II, [COMET](../../docs/explanations/aggregation-methods.md#comet)) come from pymcdm, called on the matrix normalized by beam. The [weighting schemes](../../docs/explanations/weighting-schemes.md) are beam's own.```{python}%matplotlib inlineimport numpy as npimport matplotlib.pyplot as pltfrom beam.mcda import ( critical_difference, leave_one_dataset_out, run, run_from_registry, smaa, smallest_weight_perturbation,)from beam.scenarios import transportation_benchmarktb = transportation_benchmark()print("modes: ", tb.mode_names)print("terrains:", tb.terrain_names)print("metrics: ", tb.metric_names)print("polarity:", tb.polarity)print("normalization:", tb.normalization)```## CoverageSpeed per mode and terrain. A cross marks a mode that cannot run on a terrain, and a red box the fastest mode on each terrain.```{python}from beam import plotspeed = tb.metric("speed")n_modes =len(tb.mode_names)n_terrains =len(tb.terrain_names)plot.score_heatmap( speed, row_names=tb.mode_names, col_names=tb.terrain_names, row_label="transport mode (method axis)", col_label="terrain (dataset axis)", value_label="speed (km/h, log scale)", log=True, highlight_best_per_col=True, higher_is_better=True, title="Speed by mode and terrain; red box marks the fastest mode",)```No mode runs on every terrain, so a ranking pooled across terrains compares modes on different subsets.```{python}feasible = tb.feasible()runs_everywhere = [ tb.mode_names[i] for i inrange(n_modes) if feasible[i].all()]print("modes that run on every terrain:", runs_everywhere or"none")print()print("fastest mode per terrain, by speed, over the feasible modes (the ground truth):")for t, terrain inenumerate(tb.terrain_names): best = tb.mode_names[int(np.nanargmax(speed[:, t]))]print(f" {terrain:15s}{best}")```## Ranking per terrainOn each terrain the infeasible modes are dropped, with no imputation and no pooling across terrains. `feasible_submatrix` returns the complete score matrix, and [`run_from_registry`](../../docs/reference/run_from_registry.qmd) reads polarity and normalization from the cards.The cards declare the [normalization](../../docs/explanations/normalization-and-scales.md): `log_min_max` for speed and cost, which span orders of magnitude, and `rank` for CO2, which has true zeros (walking, cycling).Equal weights and SAW:```{python}print("per-terrain top-ranked mode, equal weights, SAW")print(f"{'terrain':15s}{'top-ranked':12s}{'fastest by speed':16s} same?")for terrain in tb.terrain_names: names, sub = tb.feasible_submatrix(terrain) out = run_from_registry(sub, list(tb.metric_names), method="saw") top = names[int(np.argmin(out.ranks))] fastest = names[int(np.argmax(sub[:, 0]))]print(f" {terrain:15s}{top:12s}{fastest:16s}{top == fastest}")```Speed-heavy weights (0.8 on speed, 0.1 on cost, 0.1 on CO2):```{python}speed_heavy = np.array([0.8, 0.1, 0.1])print("per-terrain top-ranked mode, speed-heavy weights (0.8, 0.1, 0.1), SAW")print(f"{'terrain':15s}{'top-ranked':12s}{'fastest by speed':16s} same?")for terrain in tb.terrain_names: names, sub = tb.feasible_submatrix(terrain) out = run_from_registry(sub, list(tb.metric_names), weights=speed_heavy, method="saw") top = names[int(np.argmin(out.ranks))] fastest = names[int(np.argmax(sub[:, 0]))]print(f" {terrain:15s}{top:12s}{fastest:16s}{top == fastest}")```## Crossover among the slower modesTrail running is slower than road running on the flat road and faster on mud and uphill. An e-bike is faster than a bicycle on the flat road and the urban hop and slower uphill. Both modes of each pair run on these terrains.```{python}speed = tb.metric("speed")m = tb.mode_names.indext = tb.terrain_names.indexprint("trail running vs road running, speed (km/h):")print(f"{'terrain':12s}{'road':>6s}{'trail':>6s} faster")for terrain in ["flat_road", "mud", "uphill"]: road = speed[m("running"), t(terrain)] trail = speed[m("trail_running"), t(terrain)] faster ="trail"if trail > road else"road"print(f"{terrain:12s}{road:6.1f}{trail:6.1f}{faster}")print()print("e-bike vs bicycle, speed (km/h):")print(f"{'terrain':12s}{'bike':>6s}{'ebike':>6s} faster")for terrain in ["flat_road", "urban_hop", "uphill"]: bike = speed[m("bicycle"), t(terrain)] ebike = speed[m("e_bike"), t(terrain)] faster ="ebike"if ebike > bike else"bike"print(f"{terrain:12s}{bike:6.1f}{ebike:6.1f}{faster}")```## Five aggregations on one terrainEvery mode is feasible on the long-distance terrain. The five aggregations under the speed-heavy weighting:```{python}names, sub = tb.feasible_submatrix("long_distance")methods = ["saw", "topsis", "vikor", "promethee_ii", "comet"]ranks_by_method = {}print(f"{'method':14s} ranks by mode (1 = best)")for m in methods: out = run_from_registry(sub, list(tb.metric_names), weights=speed_heavy, method=m) ranks_by_method[m] = out.ranks by_mode = {n: int(r) for n, r inzip(names, out.ranks)}print(f" {m:14s}{by_mode}")print()print("top-ranked mode per method:")for m in methods:print(f" {m:14s}{names[int(np.argmin(ranks_by_method[m]))]}")```The same ranks as a heatmap, one column per mode and one row per aggregation method:```{python}grid = np.array([ranks_by_method[m] for m in methods])plot.rank_heatmap( grid, row_names=methods, col_names=names, row_label="aggregation method", col_label="transport mode", title="Long-distance rank by mode and method (speed-heavy weights)",)```## Four weightings on the same terrainWith SAW on the long-distance terrain, equal, entropy, standard deviation and CRITIC weights:```{python}print(f"{'weighting':10s}{'weights (speed, cost, co2)':28s} top-ranked")for wname in ["equal", "entropy", "std", "critic"]: out = run_from_registry(sub, list(tb.metric_names), weights=wname, method="saw") w_str = np.array2string(np.round(out.weights, 3), separator=", ") top = names[int(np.argmin(out.ranks))]print(f" {wname:10s}{w_str:28s}{top}")```MEREC takes the logarithm of the normalized scores, so it stops on the zero produced by `rank` on CO2. Here `run` is called directly with z-score normalization:```{python}merec = run( sub, tb.polarity, weights="merec", normalization="zscore", method="saw", metric_ids=list(tb.metric_names),)w_str = np.array2string(np.round(merec.weights, 3), separator=", ")print(f"merec (zscore) weights={w_str} top-ranked={names[int(np.argmin(merec.ranks))]}")```## Critical difference on complete casesA [critical-difference diagram](../../docs/explanations/critical-difference.md) needs a complete tool by dataset table. The four ground modes (foot, running, bicycle, motorcycle) are feasible on five terrains (flat road, mud, uphill, long distance, urban hop). `common_feasible_block` returns these complete cases. The diagram is on speed.```{python}block_modes = ("foot", "running", "bicycle", "motorcycle")common_terrains, block = tb.common_feasible_block(block_modes)print("block modes: ", block_modes)print("common terrains: ", common_terrains)cd = critical_difference( block, higher_is_better=True, tool_names=block_modes)print(f"Friedman statistic = {cd.friedman_statistic:.3f}, p = {cd.friedman_pvalue:.4f}")print(f"critical difference (alpha={cd.alpha}) = {cd.critical_difference:.3f}")print("average ranks (1 = best, fastest):")for i in cd.order:print(f" {block_modes[i]:12s}{cd.average_ranks[i]:.2f}")groups = [tuple(block_modes[i] for i in c) for c in cd.cliques]print("cliques (not separable at alpha):", groups or"none")``````{python}plot.critical_difference_band(cd)```## SMAA on one terrainSMAA samples weight vectors from a Dirichlet and reports the fraction of samples with each mode ranked first, here on the long-distance terrain with TOPSIS.```{python}smaa_report = smaa( sub, tb.polarity, n_samples=500, method="topsis", seed=0,)print("confidence factor (fraction of sampled weightings ranking the mode first):")for n, c inzip(names, smaa_report.confidence_factor):print(f" {n:12s}{c:.3f}")most = names[int(np.argmax(smaa_report.confidence_factor))]print(f"\nmost confident top-ranked mode: {most} "f"({smaa_report.confidence_factor.max():.2f} of samples)")```## Weight perturbation on one terrain[`smallest_weight_perturbation`](../../docs/reference/smallest_weight_perturbation.qmd) reports, for every pair of modes under SAW, the smallest single-weight change swapping them, here on the long-distance terrain with equal weights.```{python}ts = smallest_weight_perturbation( sub, tb.polarity, weights="equal", method="saw",)print("base ranks under equal weights, SAW:")print(" ", {n: int(r) for n, r inzip(names, ts.base.ranks)})if ts.most_fragile_pair isnotNone: p = ts.most_fragile_pairprint(f"\nmost fragile pair: {names[p.higher]} ranked above {names[p.lower]}, "f"flips by changing the weight on {tb.metric_names[p.criterion]!r} "f"by {p.delta:+.4f}" )print(f"top rank is fragile: {ts.top_rank_is_fragile}")```## Leave one dataset outThe kayak, the motorboat and the small plane all run on open water and the long distance, so they are complete cases over two terrains.```{python}water_modes = ("kayak", "boat", "plane")water_terrains = ("open_water", "long_distance")mode_idx = [tb.mode_names.index(m) for m in water_modes]terrain_idx = [tb.terrain_names.index(t) for t in water_terrains]water_block = tb.scores[np.ix_(mode_idx, terrain_idx, range(len(tb.metric_names)))]print("speed (km/h) by mode and terrain:")print(f"{'mode':8s}{'open_water':>11s}{'long_distance':>13s}")for i, mode inenumerate(water_modes):print(f"{mode:8s}{water_block[i, 0, 0]:11.0f}{water_block[i, 1, 0]:13.0f}")```The two terrains are pooled with speed-heavy weights and SAW, by the arithmetic mean of the raw scores, and each terrain is dropped in turn.```{python}rules = ("arithmetic_mean",) *len(tb.metric_names)lodo = leave_one_dataset_out( water_block, tb.polarity, rules, dataset_names=water_terrains, metric_ids=tb.metric_names, weights=speed_heavy, method="saw", normalization=list(tb.normalization),)print("pooled over both terrains:", {m: int(r) for m, r inzip(water_modes, lodo.base.ranks)})for d, r in lodo.leave_one_out.items():print(f"drop {water_terrains[d]:14s}:", {m: int(rk) for m, rk inzip(water_modes, r.ranks)})print()print("rank held across the leave-one-dataset-out runs:")for m, s inzip(water_modes, lodo.rank_stability):print(f" {m:8s}{s *100:5.0f}%")print(f"most influential terrain: {water_terrains[lodo.most_influential_dataset]} "f"(largest rank shift {lodo.max_rank_shift})")``````{python}order = np.argsort(lodo.base.ranks)fig, ax = plt.subplots(figsize=(5.8, 2.6))ax.barh(range(len(water_modes)), lodo.rank_stability[order] *100, color="#3a7ca5")ax.set_yticks(range(len(water_modes)))ax.set_yticklabels([water_modes[i] for i in order])ax.invert_yaxis()ax.set_xlim(0, 100)ax.set_xlabel("rank held across the 2 leave-one-dataset-out runs (percent)")ax.set_ylabel("transport mode (ordered by pooled rank)")ax.set_title("Water modes: leave-one-terrain-out rank stability (speed-heavy weights)")fig.tight_layout()plt.show()```Pooled over both terrains the plane ranks first. On open water alone the motorboat ranks first, and on the long distance alone the plane.[`pairwise_transitivity`](../../docs/explanations/pairwise-method-comparison.md#transitivity) on speed, for the ground modes and for the water modes:```{python}from beam.mcda import pairwise_superiority, pairwise_transitivityfrom matplotlib.colors import ListedColormapdef majority_grid(scores, names): sup = pairwise_superiority(scores, "higher_is_better", method_names=names) trans = pairwise_transitivity(sup) dom = trans.dominance tied = {p for a, b in trans.tied_pairs for p in ((a, b), (b, a))} order =list(np.argsort(-dom.sum(axis=1), kind="stable")) grid = np.zeros((trans.n_methods, trans.n_methods))for r, i inenumerate(order):for c, j inenumerate(order):if i == j: grid[r, c] =3elif (i, j) in tied: grid[r, c] =2elif dom[i, j] ==1: grid[r, c] =1return grid, [names[i] for i in order], transland_grid, land_labels, land_trans = majority_grid(block, block_modes)water_grid, water_labels, water_trans = majority_grid(water_block[:, :, 0], water_modes)print(f"land: transitive {land_trans.is_transitive}, tied pairs {len(land_trans.tied_pairs)}")print(f"water: transitive {water_trans.is_transitive}, tied pairs {len(water_trans.tied_pairs)}")cmap = ListedColormap(["#f4f4f4", "#3b6ea5", "#cfcfcf", "#777777"])fig, axes = plt.subplots(1, 2, figsize=(9.0, 4.5))for ax, grid, labels, title in [ (axes[0], land_grid, land_labels, "Land modes: one consistent order"), (axes[1], water_grid, water_labels, "Water modes: plane and boat tie"),]: ax.imshow(grid, cmap=cmap, vmin=-0.5, vmax=3.5) ax.set_xticks(range(len(labels))) ax.set_xticklabels(labels, rotation=45, ha="right", fontsize=8) ax.set_yticks(range(len(labels))) ax.set_yticklabels(labels, fontsize=8) ax.set_xlabel("mode outperformed (column)") ax.set_title(title)axes[0].set_ylabel("mode (row), ordered by methods outperformed")fig.suptitle("Pairwise majorities by speed (blue = outperforms, grey = tie)")fig.tight_layout()plt.show()```[`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) on the water modes. With two terrains the posterior is close to the prior.```{python}from beam.mcda import bayesian_sign_comparisonwater_sup = pairwise_superiority(water_block[:, :, 0], "higher_is_better", method_names=water_modes)water_bayes = bayesian_sign_comparison(water_sup)decisive =sum(1for p in water_bayes.per_pair if p.decision !="inconclusive")print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(water_bayes.per_pair)}")plot.bayesian_comparison(water_bayes)```The [funky heatmap](../../docs/explanations/funky-heatmaps-and-robustness.md) of the water modes, with the leave-one-dataset-out span and the aggregation span.```{python}import beamfrom beam.reporting import funky_heatmap_from_runwater = beam.Scores( values=water_block, tool_names=water_modes, metric_ids=tb.metric_names, dataset_names=water_terrains, layout="long",)water_run = beam.rank(water, weights="equal", method="saw")funky_heatmap_from_run(water_run, title="Transport water block: scores and rank robustness")```[`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) lists the rankings of the water modes under every weighting, aggregation and terrain.```{python}from beam.mcda import rank_sensitivity, specification_curvectx = water_run.contextrs = rank_sensitivity( water.values, ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), missing="worst", tool_names=water_modes, dataset_names=water_terrains,)curve = specification_curve(rs)dom = curve.tool_names[curve.most_frequent_top_tool]print(f"{curve.n_specifications} specifications; {dom} first in "f"{curve.most_frequent_top_fraction *100:.0f}%, "f"{curve.n_distinct_top_tools} modes reach the top")``````{python}plot.specification_curve(curve)```The [rank-sensitivity](../../docs/explanations/rank-sensitivity.md) rank variance split per mode, with the rank range of each mode:```{python}plot.rank_sensitivity_by_tool(rs)```A [blind analysis](../../docs/explanations/analysis-blinding.md) masks the mode names before the pipeline is fixed. `beam.blind` relabels and shuffles the rows; `beam.unblind` restores them.```{python}from beam import blind, unblindblinded, seal = blind(water, seed=0)blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)restored = unblind(blind_run, seal)named = beam.rank(water, weights="equal", method="saw", seed=0, sensitivity=False)print("ranking identical after unblinding:",dict(zip(named.tool_names, named.result.ranks))==dict(zip(restored.tool_names, restored.result.ranks)))print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12])```## Dataset concordance[`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) ranks the modes within each terrain and reports the Kendall tau-b agreement between every pair of terrains, for the ground modes on five terrains and the water modes on two.```{python}land_mode_idx = [tb.mode_names.index(m) for m in block_modes]land_terrain_idx = [tb.terrain_names.index(t) for t in common_terrains]land_block = tb.scores[np.ix_(land_mode_idx, land_terrain_idx, range(len(tb.metric_names)))]land = beam.Scores( values=land_block, tool_names=block_modes, metric_ids=tb.metric_names, dataset_names=common_terrains, layout="long",)land_run = beam.rank(land, weights="equal", method="saw")land_conc = land_run.dataset_concordancewater_conc = water_run.dataset_concordanceprint(f"land block, {len(common_terrains)} terrains: mean agreement (tau-b) {land_conc.mean_pairwise_tau:.2f}")print(f"water block, 2 terrains: mean agreement (tau-b) {water_conc.mean_pairwise_tau:.2f}")``````{python}plot.dataset_concordance(land_run)```Rank of each mode per terrain minus its mean rank:```{python}plot.dataset_struggle(water_run)```## Mixed-effects on speedThe [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) on speed, with the mode as a fixed effect and the terrain as a random intercept. It needs R's lme4.```{python}from beam.heterogeneity import mixed_effects_from_matrix, r_availablespeed = tb.scores[:, :, 0]keep =~np.isnan(speed).all(axis=1)speed_modes = [m for m, k inzip(tb.mode_names, keep) if k]if r_available(): me = mixed_effects_from_matrix(speed[keep], speed_modes, tb.terrain_names)print(f"terrain shift (ICC): {me.icc_dataset:.2f} of the speed variance")print(f"residual share: {me.residual_share:.2f} (mode-by-terrain interaction plus noise)")else:print("R with lme4 not available; skipping the mixed-effects fit.")```