difficulty_concordance

mcda.difficulty_concordance

Whether method families find the same datasets hard.

A dataset can be hard because the biology is complex, in which case every method struggles, or because of something one family of methods depends on, such as label quality for semi-supervised methods, in which case another family is unaffected. The distinction matters for a cross-benchmark reading: a recommendation that holds for classical methods need not hold for deep-learning ones when the datasets are hard for different reasons.

difficulty_concordance splits the methods into families (deep-learning versus classical, or semi-supervised versus unsupervised), measures each dataset’s difficulty for each family as the family’s mean score after orienting and min-max scaling every metric, and correlates the per-family difficulty profiles across datasets with Spearman. High concordance means the hardness comes from the data; low concordance means it comes from the method family.

It is the family-split companion to :func:dataset_discrimination, which measures how much a dataset separates all its methods at once.

Classes

Name	Description
DifficultyConcordanceReport	Whether method families find the same datasets hard.

DifficultyConcordanceReport

mcda.difficulty_concordance.DifficultyConcordanceReport(self, family_names, dataset_ids, family_score, concordance, coverage, mean_pairwise_concordance, per_dataset_range, most_divergent_dataset)

Whether method families find the same datasets hard.

Attributes

Name	Type	Description
family_names	tuple[str, …]	Family labels in first-seen order, indexing the matrix rows.
dataset_ids	tuple[str, …] | None	Dataset labels in input order, or None when the input carried none.
family_score	np.ndarray	(n_families, n_datasets) mean pooled normalized score per family per dataset, higher meaning the family does better (the dataset is easier for it). nan where a family has no observed method on a dataset.
concordance	np.ndarray	(n_families, n_families) Spearman correlation across the datasets between the families’ difficulty profiles. The diagonal is 1; an entry is nan when the two families share fewer than min_pairwise datasets.
coverage	np.ndarray	(n_families, n_families) count of datasets where both families have a finite score, the denominator behind each correlation.
mean_pairwise_concordance	float	Mean of the off-diagonal finite concordance entries. High means the families agree on which datasets are hard.
per_dataset_range	np.ndarray	Max-minus-min family score per dataset, the size of the family disagreement on that dataset. nan when fewer than two families are observed.
most_divergent_dataset	str | None	Dataset id with the largest per_dataset_range, where the families disagree most on difficulty, or None.

Functions

Name	Description
difficulty_concordance	Test whether method families find the same datasets hard.

difficulty_concordance

mcda.difficulty_concordance.difficulty_concordance(scores, polarity, families, dataset_ids=None, min_pairwise=3)

Test whether method families find the same datasets hard.

Parameters

Name	Type	Description	Default
scores		Tensor of shape (n_methods, n_datasets, n_metrics). Missing cells are nan and handled available-case; nothing is imputed. Pass one benchmark per call, since the min-max scaling is within the tensor.	required
polarity	Sequence[str]	Length n_metrics sequence of "higher_is_better" or "lower_is_better". Drop "target_value" metrics before calling.	required
families	Sequence[str]	Length n_methods family label per method, for example "DL" or "classical". Methods sharing a label form one family.	required
dataset_ids	Sequence[str] | None	Optional length n_datasets labels carried into the report.	None
min_pairwise	int	Minimum datasets where two families both have a score for their concordance to be computed. Default 3.	3

Returns

Type	Description
DifficultyConcordanceReport

Raises

Type	Description
ValueError	If scores is not three-dimensional, or polarity/families lengths do not match the tensor, or fewer than two families are given.