rank_sensitivity
mcda.rank_sensitivity
Split a ranking’s variance between the analyst’s choices and the data.
A ranking can move for two reasons. One is the analyst’s choices: which weighting scheme sets the weights, and which aggregation rule combines the scores. The other is the data: a method that ranks first on one dataset can trail on another. A benchmarker should know which reason is at work. The other tools each vary one thing. aggregation_agreement varies the aggregation. smaa varies the weights. leave_one_dataset_out drops a dataset. None of them measure the choices and the data on the same scale.
rank_sensitivity does. The weighting, the aggregation, and the dataset are each a small set of options, so beam computes every combination rather than sampling. For each tool this gives a table of its rank over the full factorial. An analysis of variance then splits the rank variance into a share for each factor plus a share for their interactions. The shares sum to one. For a deterministic function of categorical factors over a balanced factorial, these main-effect shares are the first-order variance indices (the categorical form of the Sobol indices), and they are exact here because nothing is sampled.
The headline is the share each factor carries, pooled over the tools. A large dataset share means the ranking depends on which dataset you use. A large weighting or aggregation share means it depends on a choice the analyst could make differently. Either way, the report says so instead of hiding it.
A 2D tool-by-metric matrix has two factors: the weighting and the aggregation. A 3D tool-by-dataset-by-metric tensor adds the dataset as a third factor, so the data question and the choice questions are answered in one decomposition.
Classes
| Name | Description |
|---|---|
| RankSensitivityReport | Variance attribution of a ranking to the choices and the data. |
| ToolRankSensitivity | The rank-sensitivity profile of one tool. |
RankSensitivityReport
mcda.rank_sensitivity.RankSensitivityReport(self, factors, weightings, methods, dataset_names, dropped_weightings, dropped_methods, dropped_datasets, n_combinations, ranks, tool_names, factor_shares, interaction_share, most_influential_factor, per_tool, headline_tool, headline_top_fraction, headline_rank_span, headline_rank_by_dataset)
Variance attribution of a ranking to the choices and the data.
Attributes
| Name | Type | Description |
|---|---|---|
| factors | tuple[str, …] | The factor names in the order of the ranks axes after the tool axis: ("weighting", "aggregation") for a matrix, with "dataset" appended for a tensor. |
| weightings, methods | The weighting schemes and aggregations that formed the factorial, after any pruning, in axis order. | |
| dataset_names | tuple[str, …] | None | The datasets that formed the third factor, or None for a 2D input. |
| dropped_weightings, dropped_methods, dropped_datasets | Factor levels dropped because they failed on this input, so the surviving grid stays balanced. Empty in the common case. | |
| n_combinations | int | The size of the surviving factorial. |
| ranks | np.ndarray | Integer competition ranks (1 is best). Shape (n_tools, n_weightings, n_methods) for a matrix, (n_tools, n_weightings, n_methods, n_datasets) for a tensor. |
| tool_names | tuple[str, …] | None | Tool labels in index order, or None. |
| factor_shares | dict[str, float] | The pooled main-effect variance share per factor over all tools. With interaction_share the values sum to one. |
| interaction_share | float | The pooled variance share carried by all factor interactions together. |
| most_influential_factor | str | The factor or "interaction" whose pooled share is largest. |
| per_tool | tuple[ToolRankSensitivity, …] | The per-tool profiles, in tool-index order. |
| headline_tool | int | The tool ranked first in the most combinations. |
| headline_top_fraction | float | The fraction of combinations in which headline_tool ranks first. |
| headline_rank_span | int | The rank span of headline_tool across the factorial. |
| headline_rank_by_dataset | np.ndarray | None | For a tensor input, the mean rank of headline_tool on each dataset, averaged over the choices, in dataset_names order. None for a matrix. A row that climbs from 1 shows where the headline tool slips. |
ToolRankSensitivity
mcda.rank_sensitivity.ToolRankSensitivity(self, tool, name, factor_shares, interaction_share, rank_min, rank_max, rank_span, modal_rank)
The rank-sensitivity profile of one tool.
Attributes
| Name | Type | Description |
|---|---|---|
| tool | int | Tool index into the score matrix. |
| name | str | None | Tool label, or None when the matrix carried none. |
| factor_shares | dict[str, float] | The fraction of this tool’s rank variance over the factorial explained by each factor’s main effect, keyed by factor name. With the interaction_share the values sum to one. All nan when the tool holds the same rank in every combination. |
| interaction_share | float | The fraction of this tool’s rank variance carried by every factor interaction together, nan when the tool has no rank variance. |
| rank_min, rank_max, rank_span | The best, worst, and spread of this tool’s rank across the factorial. | |
| modal_rank | int | The rank this tool takes most often across the factorial. |
Functions
| Name | Description |
|---|---|
| rank_sensitivity | Decompose a ranking’s variance over the weighting, aggregation and dataset. |
rank_sensitivity
mcda.rank_sensitivity.rank_sensitivity(scores, polarity, weightings=None, methods=None, normalization=None, bounds=None, baselines=None, targets=None, missing='error', tool_names=None, dataset_names=None)
Decompose a ranking’s variance over the weighting, aggregation and dataset.
Runs the full factorial of the factors, collecting the rank of every tool under each combination, and decomposes each tool’s rank variance into a main-effect share per factor plus the interaction share by an analysis of variance. Pooling the variance over the tools gives the overall share each factor carries.
With a 2D (n_tools, n_metrics) matrix the factors are the weighting and the aggregation. With a 3D (n_tools, n_datasets, n_metrics) tensor the dataset joins them: each combination ranks the tools on one dataset’s slice, so the dataset share reads how much the ranking depends on which dataset you evaluate on. Each combination must rank: the distance and outranking aggregations refuse a slice with missing cells, so for partial coverage pass missing="worst" to complete the matrix or restrict to the feasible subset.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
scores |
Array-like of shape (n_tools, n_metrics) or (n_tools, n_datasets, n_metrics). |
required | |
polarity |
Sequence[str] | Length n_metrics sequence of "higher_is_better" or "lower_is_better". Use beam.cards.polarities_for to source it from the registry. |
required |
weightings |
Sequence[str] | None | Weighting schemes to vary. Default is ("equal", "entropy", "std", "critic"). MEREC is left out of the default because it takes the logarithm of the scores and refuses the zeros the default min_max normalization produces; pass it explicitly with a positivity-preserving normalization. AHP is excluded because it needs an analyst-supplied pairwise matrix, not a data-driven rule. |
None |
methods |
Sequence[str] | None | Aggregations to vary, from the five beam aggregations. Default is all five. COMET builds characteristic objects whose count grows fast with the number of metrics, so drop it on a metric-rich benchmark (pass the other four) to keep the factorial quick. | None |
normalization |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition rests on the same normalized matrix as the headline ranking. |
None |
|
bounds |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition rests on the same normalized matrix as the headline ranking. |
None |
|
baselines |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition rests on the same normalized matrix as the headline ranking. |
None |
|
targets |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the decomposition rests on the same normalized matrix as the headline ranking. |
None |
|
missing |
str | Missing-data policy forwarded to every run. Default "error". |
'error' |
tool_names |
Sequence[str] | None | Optional length n_tools labels carried in the report. |
None |
dataset_names |
Sequence[str] | None | Optional length n_datasets labels for a tensor input, carried in the report and used to name the dataset levels. |
None |
Returns
| Type | Description |
|---|---|
| RankSensitivityReport |
Raises
| Type | Description |
|---|---|
| ValueError | If the shapes do not line up, or fewer than two weightings or two aggregations (or, for a tensor, two datasets) survive on this input, so a factorial decomposition is undefined. |
Examples
>>> import numpy as np
>>> from beam.mcda import rank_sensitivity
>>> scores = np.array([[0.9, 30.0], [0.7, 50.0], [0.5, 40.0], [0.3, 70.0]])
>>> report = rank_sensitivity(scores, ["higher_is_better", "lower_is_better"])
>>> report.dataset_share is None
True
>>> round(sum(report.factor_shares.values()) + report.interaction_share, 6)
1.0