dataset_concordance
mcda.dataset_concordance
Agreement between datasets on how they order the methods.
A pooled ranking summarizes the methods over every dataset at once. It cannot show whether the datasets agree on that order or pull in different directions. This module measures the agreement directly. It ranks the methods within each dataset under the same MCDA pipeline as the headline run, then compares every pair of per-dataset orderings with the Kendall tau-b rank correlation.
The output is a dataset by dataset agreement matrix, a single mean-agreement summary, the dataset whose ordering matches the others least, and a grouping of datasets whose orderings are mutually consistent above a threshold. A high mean says the pooled recommendation stands in for the individual datasets. A low one says it does not, and a single pooled number then hides heterogeneity the reader should see.
A second output is the rank-deviation table. For each method it records the datasets where the method places higher or lower than its own typical rank. These cells are where the dataset disagreement comes from. They name the method-by-dataset combinations that move the ordering, with no claim about which method is preferable overall.
The diagnostic needs no replicates and assumes no exchangeability among the datasets. It treats each observed dataset as fixed and reports the structure of agreement among them. It is the data-driven companion to the Bradley-Terry tree, which splits the datasets by declared features rather than by their rankings.
Classes
| Name | Description |
|---|---|
| DatasetConcordanceReport | Outcome of comparing the per-dataset method orderings. |
| RankDeviation | One method-by-dataset cell where a method departs from its typical rank. |
DatasetConcordanceReport
mcda.dataset_concordance.DatasetConcordanceReport(self, evaluated_datasets, ranks_by_dataset, tau_matrix, mean_pairwise_tau, per_dataset_mean_tau, most_idiosyncratic_dataset, concordant_groups, mean_rank_by_tool, rank_deviation, notable_cells, threshold, dataset_names=None, tool_names=None)
Outcome of comparing the per-dataset method orderings.
Rows and columns of tau_matrix and ranks_by_dataset, and the entries of per_dataset_mean_tau, follow evaluated_datasets. The dataset indices in most_idiosyncratic_dataset, concordant_groups and the RankDeviation records are original dataset indices.
Fields: evaluated_datasets: original indices of the datasets that produced a ranking, in order. A dataset whose single-dataset matrix the pipeline cannot rank (for example a missing cell under the "error" policy) is dropped and absent here. ranks_by_dataset: (n_evaluated, n_tools) per-dataset 1-based ranks. tau_matrix: (n_evaluated, n_evaluated) Kendall tau-b between every pair of per-dataset orderings. The diagonal is 1. An entry is nan when one ordering is constant, so tau-b is undefined. mean_pairwise_tau: mean of the off-diagonal tau values, ignoring nan. A scalar summary of how much the datasets agree; 1 is exact agreement. per_dataset_mean_tau: (n_evaluated,) each dataset’s mean agreement with the others, ignoring nan. most_idiosyncratic_dataset: original index of the dataset with the lowest mean agreement with the rest. concordant_groups: groups of datasets whose pairwise agreement is at or above threshold, as connected components of that relation. Each group is a tuple of original dataset indices; a dataset agreeing with no other forms its own singleton group. mean_rank_by_tool: (n_tools,) each method’s mean rank across the evaluated datasets. rank_deviation: (n_tools, n_evaluated) each method’s per-dataset rank minus its mean rank, the signed table behind notable_cells. notable_cells: the method-by-dataset cells at least one full rank from the method’s mean rank, sorted by the size of the deviation, capped at a small number for reporting. threshold: the agreement cutoff used to form concordant_groups. dataset_names: optional length n_datasets labels, indexed by original dataset index. tool_names: optional length n_tools labels.
RankDeviation
mcda.dataset_concordance.RankDeviation(self, tool, dataset, rank, mean_rank, deviation)
One method-by-dataset cell where a method departs from its typical rank.
Fields: tool: index into the tool axis. dataset: original index into the datasets (not the evaluated-only position). rank: the method’s 1-based rank on that dataset, 1 is the highest place. mean_rank: the method’s mean rank across the evaluated datasets. deviation: rank minus mean_rank. A negative value means the method places higher than its average on that dataset; a positive value means it places lower.
Functions
| Name | Description |
|---|---|
| dataset_concordance | Rank the methods within each dataset and measure how the orderings agree. |
dataset_concordance
mcda.dataset_concordance.dataset_concordance(tensor, polarity, weights='equal', method='saw', dataset_names=None, tool_names=None, metric_ids=None, normalization=None, bounds=None, baselines=None, targets=None, missing='error', threshold=0.5)
Rank the methods within each dataset and measure how the orderings agree.
For each dataset the function ranks the methods on that dataset’s tool by metric matrix through beam.mcda.run, holding the weighting, aggregation and normalization context fixed. It then compares every pair of per-dataset orderings with the Kendall tau-b coefficient, which handles the ties that competition ranking produces.
A dataset whose single-dataset matrix the pipeline cannot rank is dropped rather than failing the whole analysis, matching how the aggregation and normalization agreement reports drop a configuration that cannot run. At least two datasets must produce a ranking.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
tensor |
Array-like of shape (n_tools, n_datasets, n_metrics). |
required | |
polarity |
Sequence[str] | Length n_metrics sequence of "higher_is_better" or "lower_is_better". Use beam.cards.polarities_for to source this from the registry. A bare string is accepted for a single-metric tensor and treated as the one polarity. |
required |
weights |
Forwarded to run. A named scheme or an explicit array. Objective schemes are recomputed within each dataset. |
'equal' |
|
method |
str | Aggregation name forwarded to run. Default "saw". |
'saw' |
dataset_names |
Sequence[str] | None | Optional labels carried in the report. | None |
tool_names |
Sequence[str] | None | Optional labels carried in the report. | None |
metric_ids |
Sequence[str] | None | Optional length n_metrics labels used in error messages. |
None |
normalization |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the per-dataset rankings normalize the same way as the headline ranking. |
None |
|
bounds |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the per-dataset rankings normalize the same way as the headline ranking. |
None |
|
baselines |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the per-dataset rankings normalize the same way as the headline ranking. |
None |
|
targets |
Optional per-metric normalization context forwarded to every run. Pass the values from beam.mcda.registry_context so the per-dataset rankings normalize the same way as the headline ranking. |
None |
|
missing |
str | Missing-data policy forwarded to every run. Default "error", which drops any dataset with a missing cell. |
'error' |
threshold |
float | Agreement cutoff for concordant_groups. Default 0.5, a moderate tau-b. Two datasets join the same group when their tau-b is at least this value. |
0.5 |
Returns
| Type | Description |
|---|---|
| DatasetConcordanceReport |
Raises
| Type | Description |
|---|---|
| ValueError | If the input is not 3D, has fewer than two datasets, or fewer than two datasets produce a ranking. |
Examples
>>> import numpy as np
>>> from beam.mcda import dataset_concordance
>>> tensor = np.array(
... [
... [[0.9], [0.8], [0.1]],
... [[0.5], [0.4], [0.5]],
... [[0.1], [0.2], [0.9]],
... ]
... )
>>> report = dataset_concordance(tensor, ["higher_is_better"])
>>> report.most_idiosyncratic_dataset
2