metric_validity
mcda.metric_validity
Convergent and discriminant validity of a metric set.
A benchmark scores every method on several metrics. Some of those metrics are meant to measure the same underlying quantity (for example the scIB biological-conservation metrics ARI, NMI and isolated-label F1 all read how well cell-type structure survives integration), and others are meant to measure a different quantity (the batch-correction metrics kBET, iLISI, batch silhouette). Campbell and Fiske (1959) asked the validity question that follows: do the metrics that claim to measure the same construct actually agree, and do the metrics that claim to measure different constructs actually differ?
metric_validity answers it from the scores alone. Each method-by-dataset cell is one observation; the metrics are the variables. The function orients every metric so that higher means better (it negates the ranks of a lower_is_better metric), then computes the Spearman rank correlation between every pair of metrics over the observations they share. Spearman is the right choice here: the metrics live on different scales, and a rank correlation reads “do these two metrics order the methods the same way” without assuming a common unit.
Grouping the metrics by the construct they claim to measure splits the correlations into two sets. Within-group correlations are the convergent evidence: metrics measuring one construct should agree. Between-group correlations are the discriminant evidence: metrics measuring different constructs should agree less. When the mean within-group correlation exceeds the mean between-group correlation, the grouping holds up, and treating the groups as separate criteria in the MCDA weighting (the scIB 0.6 bio / 0.4 batch split, for example) is justified by the data rather than by assertion. The report also flags near-duplicate metrics inside a group (redundant criteria) and any metric that correlates more with another group than with its own (a metric that does not measure what its group claims).
This is the trait facet of a multitrait-multimethod matrix. beam records one measurement per metric per cell, so there is no separate method facet to vary; the diagnostic is the convergent and discriminant reading of the metric correlations, not the full MTMM design.
Classes
| Name | Description |
|---|---|
| MetricValidityReport | Convergent and discriminant validity of a metric set. |
MetricValidityReport
mcda.metric_validity.MetricValidityReport(self, metric_ids, groups, correlation, coverage, convergent_by_group, mean_convergent, mean_discriminant, discriminant_ok, redundant_pairs, crossloading_metrics, n_observations)
Convergent and discriminant validity of a metric set.
Attributes
| Name | Type | Description |
|---|---|---|
| metric_ids | tuple[str, …] | None | Metric labels in column order, or None when the input carried none. |
| groups | tuple[str, …] | Construct label per metric, aligned with metric_ids. |
| correlation | np.ndarray | (n_metrics, n_metrics) Spearman rank correlation between metrics, oriented so that higher means better on every metric. The diagonal is 1. An entry is nan when the two metrics share fewer than min_pairwise observations or one is constant on the shared rows. |
| coverage | np.ndarray | (n_metrics, n_metrics) count of observations where both metrics are observed, the denominator behind each correlation. |
| convergent_by_group | dict[str, float] | Mean within-group off-diagonal correlation per group, over finite entries. A group with one metric has no within-group pair and is absent. |
| mean_convergent | float | Mean of every within-group off-diagonal correlation, over finite entries. The overall convergent-validity summary. |
| mean_discriminant | float | Mean of every between-group correlation, over finite entries. The overall discriminant-validity summary. |
| discriminant_ok | bool | True when mean_convergent is greater than mean_discriminant, the basic Campbell-Fiske criterion that same-construct metrics agree more than cross-construct metrics. |
| redundant_pairs | tuple[tuple[str, str, float], …] | Within-group metric pairs whose correlation is at least redundant_threshold, each as (metric_a, metric_b, r) sorted by descending r. Near-duplicate criteria; candidates to drop. |
| crossloading_metrics | tuple[tuple[str, str, float, float, str], …] | Metrics whose mean between-group correlation exceeds their mean within-group correlation, each as (metric, group, mean_within_r, mean_between_r, nearest_group), where nearest_group is the other group it correlates with most strongly. A metric here agrees more with the other constructs than with its own, the per-metric form of a discriminant-validity failure. |
| n_observations | int | Number of observation rows (method-by-dataset cells) the correlations were computed over, before the per-pair NaN masking. |
Functions
| Name | Description |
|---|---|
| metric_validity | Test whether a metric set has convergent and discriminant validity. |
metric_validity
mcda.metric_validity.metric_validity(scores, polarity, groups, metric_ids=None, redundant_threshold=0.9, min_pairwise=3)
Test whether a metric set has convergent and discriminant validity.
Each method-by-dataset cell is one observation. The function orients every metric to higher-is-better, computes the Spearman rank correlation between every pair of metrics over their shared observations, and splits the correlations by the construct grouping into convergent (within-group) and discriminant (between-group) evidence.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
scores |
Array-like of shape (n_observations, n_metrics) or a tensor of shape (n_methods, n_datasets, n_metrics), which is reshaped so that each method-by-dataset cell is one observation row. Missing cells are NaN and handled pairwise. |
required | |
polarity |
Sequence[str] | Length n_metrics sequence of "higher_is_better" or "lower_is_better". Use beam.cards.polarities_for to source it from the registry. A "target_value" metric has no monotone quality direction; drop it before calling. |
required |
groups |
Sequence[str] | Length n_metrics construct label per metric, for example "bio" or "batch". Metrics sharing a label claim to measure the same construct. |
required |
metric_ids |
Sequence[str] | None | Optional length n_metrics labels carried into the report and used to name the flagged pairs and metrics. |
None |
redundant_threshold |
float | Within-group correlation at or above which a pair is reported as redundant. Default 0.9. | 0.9 |
min_pairwise |
int | Minimum shared observations for a pair’s correlation to be computed. Default 3. | 3 |
Returns
| Type | Description |
|---|---|
| MetricValidityReport |
Raises
| Type | Description |
|---|---|
| ValueError | If the shapes do not line up, a polarity is not one of the two monotone values, or the grouping has fewer than two distinct groups or no group with at least two metrics (so convergent and discriminant evidence are both undefined). |
Examples
>>> import numpy as np
>>> from beam.mcda import metric_validity
>>> rng = np.random.default_rng(0)
>>> bio = rng.normal(size=(40, 1))
>>> batch = rng.normal(size=(40, 1))
>>> scores = np.hstack([bio + rng.normal(0, 0.1, (40, 1)),
... bio + rng.normal(0, 0.1, (40, 1)),
... batch + rng.normal(0, 0.1, (40, 1)),
... batch + rng.normal(0, 0.1, (40, 1))])
>>> report = metric_validity(
... scores,
... ["higher_is_better"] * 4,
... ["bio", "bio", "batch", "batch"],
... )
>>> report.discriminant_ok
True