critical_difference

mcda.critical_difference(scores, higher_is_better=True, alpha=0.05, tool_names=None)

Run the Friedman test and Nemenyi post-hoc on a tool by dataset matrix.

Parameters

Name Type Description Default
scores np.ndarray 2D array of shape (n_tools, n_datasets) for one metric or for an MCDA composite. Each column holds the scores of every tool on one dataset. required
higher_is_better bool Whether a larger score is better. When False the ranking is reversed before ranks are taken, so the reported average ranks always put the better tool nearer 1. The Friedman statistic itself does not depend on the direction. True
alpha float Significance level for the Nemenyi critical difference. 0.05
tool_names Sequence[str] | None Optional labels, length n_tools, kept in the report. None

Returns

Type Description
CriticalDifferenceReport

Notes

Friedman needs at least three tools and two datasets, and a complete tool by dataset table: every tool ranked on every dataset. A missing cell raises an error, since the Friedman ranks per dataset are only defined over a complete column. The missing-data generalization is the Skillings-Mack (1981) test in beam.mcda.skillings_mack; alternatively, restrict the diagram to the block of tools and datasets where all of them ran.