rank_sensitivity

mcda.rank_sensitivity

Split a ranking’s variance between the analyst’s choices and the data.

A ranking can move for two reasons. One is the analyst’s choices: which weighting scheme sets the weights, and which aggregation rule combines the scores. The other is the data: a method that ranks first on one dataset can rank lower on another. The other diagnostics each vary one thing. aggregation_agreement varies the aggregation. smaa varies the weights. leave_one_dataset_out drops a dataset. None of them measure the choices and the data on the same scale.

rank_sensitivity does. The weighting, the aggregation, and the dataset are each a small set of options, so beam computes every combination rather than sampling. For each tool this gives a table of its rank over the full factorial. An analysis of variance then splits the rank variance into a share for each factor plus a share for their interactions. The shares sum to one. For a deterministic function of categorical factors over a balanced factorial, these main-effect shares are the first-order variance indices (the categorical form of the Sobol indices), and they are exact here because nothing is sampled.

The main output is the share of each factor, pooled over the tools. A large dataset share means the ranking depends on the dataset. A large weighting or aggregation share means it depends on a choice the analyst could make differently.

A 2D tool-by-metric matrix has two factors: the weighting and the aggregation. A 3D tool-by-dataset-by-metric tensor adds the dataset as a third factor, so the data question and the choice questions are answered in one decomposition.

Classes

Name Description
RankSensitivityReport Variance attribution of a ranking to the choices and the data.
ToolRankSensitivity The rank-sensitivity profile of one tool.

RankSensitivityReport

mcda.rank_sensitivity.RankSensitivityReport(self, factors, weightings, methods, dataset_names, dropped_weightings, dropped_methods, dropped_datasets, n_combinations, ranks, tool_names, factor_shares, interaction_share, most_influential_factor, per_tool, headline_tool, headline_top_fraction, headline_rank_span, headline_rank_by_dataset)

Variance attribution of a ranking to the choices and the data.

Attributes

Name Type Description
factors tuple[str, …] The factor names in the order of the ranks axes after the tool axis: ("weighting", "aggregation") for a matrix, with "dataset" appended for a tensor.
weightings, methods The weighting schemes and aggregations that formed the factorial, after any pruning, in axis order.
dataset_names tuple[str, …] | None The datasets that formed the third factor, or None for a 2D input.
dropped_weightings, dropped_methods, dropped_datasets Factor levels dropped because they failed on this input, so the surviving grid stays balanced. Empty in the common case.
n_combinations int The size of the surviving factorial.
ranks np.ndarray Integer competition ranks (1 is best). Shape (n_tools, n_weightings, n_methods) for a matrix, (n_tools, n_weightings, n_methods, n_datasets) for a tensor.
tool_names tuple[str, …] | None Tool labels in index order, or None.
factor_shares dict[str, float] The pooled main-effect variance share per factor over all tools. With interaction_share the values sum to one.
interaction_share float The pooled variance share due to all factor interactions together.
most_influential_factor str The factor or "interaction" whose pooled share is largest.
per_tool tuple[ToolRankSensitivity, …] The per-tool profiles, in tool-index order.
headline_tool int The tool ranked first in the most combinations.
headline_top_fraction float The fraction of combinations in which headline_tool ranks first.
headline_rank_span int The rank span of headline_tool across the factorial.
headline_rank_by_dataset np.ndarray | None For a tensor input, the mean rank of headline_tool on each dataset, averaged over the choices, in dataset_names order. None for a matrix. A value above 1 marks a dataset where the headline tool ranks lower.

ToolRankSensitivity

mcda.rank_sensitivity.ToolRankSensitivity(self, tool, name, factor_shares, interaction_share, rank_min, rank_max, rank_span, modal_rank)

The rank-sensitivity profile of one tool.

Attributes

Name Type Description
tool int Tool index into the score matrix.
name str | None Tool label, or None when the matrix had none.
factor_shares dict[str, float] The fraction of this tool’s rank variance over the factorial explained by each factor’s main effect, keyed by factor name. With the interaction_share the values sum to one. All nan when the tool holds the same rank in every combination.
interaction_share float The fraction of this tool’s rank variance due to all factor interactions together, nan when the tool has no rank variance.
rank_min, rank_max, rank_span The best, worst, and spread of this tool’s rank across the factorial.
modal_rank int The rank this tool takes most often across the factorial.

Functions

Name Description
rank_sensitivity Decompose a ranking’s variance over the weighting, aggregation and dataset.

rank_sensitivity

mcda.rank_sensitivity.rank_sensitivity(scores, polarity, weightings=None, methods=None, normalization=None, bounds=None, baselines=None, targets=None, missing='error', tool_names=None, dataset_names=None)

Decompose a ranking’s variance over the weighting, aggregation and dataset.

Runs the full factorial of the factors, collecting the rank of every tool under each combination, and decomposes each tool’s rank variance into a main-effect share per factor plus the interaction share by an analysis of variance. Pooling the variance over the tools gives the overall share of each factor.

With a 2D (n_tools, n_metrics) matrix the factors are the weighting and the aggregation. With a 3D (n_tools, n_datasets, n_metrics) tensor the dataset joins them: each combination ranks the tools on one dataset’s slice, so the dataset share is the dependence of the ranking on the dataset. Each combination must rank: the distance and outranking aggregations refuse a slice with missing cells, so for partial coverage pass missing="worst" to complete the matrix or restrict to the feasible subset.

Parameters

Name Type Description Default
scores Array-like of shape (n_tools, n_metrics) or (n_tools, n_datasets, n_metrics). required
polarity Sequence[str] Length n_metrics sequence of "higher_is_better" or "lower_is_better". Use beam.cards.polarities_for to source it from the registry. required
weightings Sequence[str] | None Weighting schemes to vary. Default is ("equal", "entropy", "std", "critic"). MEREC is left out of the default because it takes the logarithm of the scores and refuses the zeros the default min_max normalization produces; pass it explicitly with a positivity-preserving normalization. AHP is excluded because it needs a pairwise matrix from the analyst. None
methods Sequence[str] | None Aggregations to vary, from the five beam aggregations. Default is all five. COMET builds characteristic objects whose count grows fast with the number of metrics, so drop it on a metric-rich benchmark (pass the other four) to keep the factorial quick. None
normalization Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition uses the same normalized matrix as the main ranking. None
bounds Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition uses the same normalized matrix as the main ranking. None
baselines Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition uses the same normalized matrix as the main ranking. None
targets Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition uses the same normalized matrix as the main ranking. None
missing str Missing-data policy forwarded to every run. Default "error". 'error'
tool_names Sequence[str] | None Optional length n_tools labels kept in the report. None
dataset_names Sequence[str] | None Optional length n_datasets labels for a tensor input, kept in the report and used to name the dataset levels. None

Returns

Type Description
RankSensitivityReport

Raises

Type Description
ValueError If the shapes do not line up, or fewer than two weightings or two aggregations (or, for a tensor, two datasets) remain on this input, so a factorial decomposition is undefined.

Examples

>>> import numpy as np
>>> from beam.mcda import rank_sensitivity
>>> scores = np.array([[0.9, 30.0], [0.7, 50.0], [0.5, 40.0], [0.3, 70.0]])
>>> report = rank_sensitivity(scores, ["higher_is_better", "lower_is_better"])
>>> report.dataset_share is None
True
>>> round(sum(report.factor_shares.values()) + report.interaction_share, 6)
1.0