dataset_discrimination
mcda.dataset_discrimination
How much each dataset separates the methods it scores.
Cross-benchmark comparisons that compare method orders need shared methods. When the method sets are disjoint, that comparison is unavailable. A property defined per dataset is still available: how much a dataset separates the methods it scores. Every benchmark can report it for every dataset.
dataset_discrimination computes it from the scores. A dataset on which the methods score about the same cannot rank them; a dataset on which they differ can. This is the per-dataset form of the metric-level notion in the weighting code, where a metric on which methods do not differ has no discrimination. It complements :func:dataset_concordance, which asks whether datasets agree on the order.
Two values per dataset.
- Spread, the effect size. Each metric is oriented to higher-is-better and min-max scaled across the benchmark’s cells, so metrics are comparable and a dataset on which every method scores near the maximum keeps a small spread. The metrics are pooled to one score per method, and the spread is the standard deviation across methods.
- Concordance, the consistency. Kendall’s W over the dataset’s method-by-metric matrix, with its Friedman p value. A high W means the metrics order the methods the same way; a low W means they do not, so a single ranking on that dataset is unstable.
The scaling is per benchmark, so spreads are comparable within a benchmark and only roughly across benchmarks.
Classes
| Name | Description |
|---|---|
| DatasetDiscriminationReport | How strongly each dataset separates the methods it scores. |
DatasetDiscriminationReport
mcda.dataset_discrimination.DatasetDiscriminationReport(self, dataset_ids, spread, kendall_w, p_value, significant, n_methods_used, n_metrics_used, pooled_score, order, mean_spread, mean_kendall_w, most_discriminating, least_discriminating)
How strongly each dataset separates the methods it scores.
Attributes
| Name | Type | Description |
|---|---|---|
| dataset_ids | tuple[str, …] | None | Dataset labels in input order, or None when the input carried none. |
| spread | np.ndarray | Per-dataset standard deviation across methods of the pooled normalized score, the effect size. nan when fewer than two methods are observed on the dataset. |
| kendall_w | np.ndarray | Per-dataset Kendall’s W over the method-by-metric matrix, in [0, 1], the consistency. nan when fewer than min_methods methods or two metrics form a complete block on the dataset. |
| p_value | np.ndarray | Per-dataset Friedman p value for the null that the metrics order the methods independently. nan where kendall_w is nan. |
| significant | np.ndarray | p_value < alpha, per dataset. |
| n_methods_used, n_metrics_used | Methods and metrics in the complete block behind kendall_w. |
|
| pooled_score | np.ndarray | (n_methods, n_datasets) pooled normalized score per method per dataset, the input to spread. nan where a method is unobserved. |
| order | np.ndarray | Dataset indices sorted by descending spread (nan spreads last). |
| mean_spread, mean_kendall_w | Means over the datasets with a finite value. | |
| most_discriminating, least_discriminating | Dataset ids with the largest and smallest finite spread, or None. |
Functions
| Name | Description |
|---|---|
| dataset_discrimination | Measure how strongly each dataset separates the methods it scores. |
dataset_discrimination
mcda.dataset_discrimination.dataset_discrimination(scores, polarity, dataset_ids=None, min_methods=3, alpha=0.05)
Measure how strongly each dataset separates the methods it scores.
Works on a benchmark with any method set, so benchmarks with disjoint methods can each be measured and then compared. Reports a spread (effect size) and a Kendall’s W (consistency) per dataset; see the module docstring.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
scores |
Tensor of shape (n_methods, n_datasets, n_metrics). A method or metric not observed on a dataset is nan; nothing is imputed. Pass one benchmark’s scores per call (the min-max scaling is within the tensor). |
required | |
polarity |
Sequence[str] | Length n_metrics sequence of "higher_is_better" or "lower_is_better". A "target_value" metric has no monotone quality direction; drop it before calling. |
required |
dataset_ids |
Sequence[str] | None | Optional length n_datasets labels carried into the report. |
None |
min_methods |
int | Minimum methods in a dataset’s complete method-by-metric block for Kendall’s W to be computed. Default 3. | 3 |
alpha |
float | Significance level for significant. Default 0.05. |
0.05 |
Returns
| Type | Description |
|---|---|
| DatasetDiscriminationReport |
Raises
| Type | Description |
|---|---|
| ValueError | If scores is not three-dimensional, or polarity length does not match the metric axis. |