paired_comparisons
heterogeneity.paired_comparisons(matrix, polarity)
Turn a method by dataset score matrix into per-dataset paired comparisons.
For each dataset (a column) and each unordered method pair (i, j) with i < j, the outcome is +1 when method i is better, -1 when method j is better, 0 on an exact tie, and nan when either method is missing on that dataset. “Better” is resolved through the metric polarity, so a lower-is-better metric is oriented before comparing.
The pair order is the psychotools convention (0,1), (0,2), (1,2), (0,3), (1,3), (2,3), ...: the second index runs outermost and the first index inner, so the result feeds psychotools::paircomp directly. (This coincides with the row-major order only up to three objects.)
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
matrix |
np.ndarray | Array of shape (n_methods, n_datasets) holding one metric’s scores. |
required |
polarity |
str | "higher_is_better" or "lower_is_better". |
required |
Returns
| Type | Description |
|---|---|
| tuple | (comparisons, pairs) where comparisons has shape (n_datasets, n_pairs) with values in {-1, 0, 1, nan} and pairs lists the (i, j) method index pairs in column order. |
Raises
| Type | Description |
|---|---|
| ValueError | If matrix is not 2D, has fewer than two methods, or polarity is not one of the two recognised values. |