metric_reliability
mcda.metric_reliability(scores, polarity, groups, metric_ids=None, alpha_threshold=0.7, min_pairwise=3)
Standardized Cronbach’s alpha for each construct group of metrics.
Each method-by-dataset cell is one observation. The function orients every metric to higher-is-better, computes the Spearman rank correlation between every pair of metrics over their shared observations (the same engine as metric_validity), and reports the standardized alpha k * r_bar / (1 + (k - 1) * r_bar) for each group, where k is the group size and r_bar the mean within-group correlation.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
scores |
Array-like of shape (n_observations, n_metrics) or a tensor of shape (n_methods, n_datasets, n_metrics), reshaped so that each method-by-dataset cell is one observation row. Missing cells are NaN and handled pairwise. |
required | |
polarity |
Sequence[str] | Length n_metrics sequence of "higher_is_better" or "lower_is_better". Use beam.cards.polarities_for to source it from the registry. A "target_value" metric has no monotone quality direction; drop it before calling. |
required |
groups |
Sequence[str] | Length n_metrics construct label per metric. Metrics sharing a label are read together as one composite scale. |
required |
metric_ids |
Sequence[str] | None | Optional length n_metrics labels carried into the report and used to name the alpha-if-dropped entries. |
None |
alpha_threshold |
float | Alpha below which a group is reported as low-reliability. Default 0.7, the conventional cutoff. | 0.7 |
min_pairwise |
int | Minimum shared observations for a pair’s correlation to be computed. Default 3. | 3 |
Returns
| Type | Description |
|---|---|
| MetricReliabilityReport |
Raises
| Type | Description |
|---|---|
| ValueError | If the shapes do not line up, a polarity is not one of the two monotone values, or no group has at least two metrics (so no alpha is defined). |
Examples
>>> import numpy as np
>>> from beam.mcda import metric_reliability
>>> rng = np.random.default_rng(0)
>>> factor = rng.normal(size=(40, 1))
>>> scores = np.hstack([factor + rng.normal(0, 0.2, (40, 1)) for _ in range(3)])
>>> report = metric_reliability(
... scores,
... ["higher_is_better"] * 3,
... ["bio", "bio", "bio"],
... )
>>> report.alpha_by_group["bio"] > 0.8
True