API reference

Pipeline

The top-level procedural API.

rank	Rank tools from a benchmark score table, with sensitivity and a manifest.
load_scores	Read a benchmark score CSV and validate its metric ids against the registry.
report	Write an HTML report for result to path.
RunResult	Everything one beam run produced, ready to report or inspect.
Scores	A benchmark score table or tensor with its tool and metric labels.
blind	Relabel and shuffle the tools of a score table, returning a seal.
unblind	Restore the true tool names on a blinded Scores or RunResult.
Seal	The secret that maps blinded labels back to true tool names.

MCDA

Normalization, weights, aggregation, ranking.

run	Run a full MCDA pipeline from raw scores to per-tool ranks.
run_from_registry	Run the MCDA pipeline with polarity, bounds, and scale checks pulled from the registry.
normalize	Rescale a tool by metric score matrix to [0, 1], respecting polarity.
weighted_sum	Simple additive weighting (SAW): per-tool dot product of scores and weights.
topsis	TOPSIS aggregation: rank tools by their distance from the ideal solution.
vikor	VIKOR aggregation: rank tools by a compromise between group utility and individual regret.
promethee_ii	Compute PROMETHEE II net outranking flows per tool.
comet	COMET aggregation: rank tools by triangular fuzzy interpolation over characteristic objects.
rank	Return 1-based ranks of scores, with 1 = best (highest score).
equal_weights	Return a vector of n_metrics equal weights that sum to 1.
entropy_weights	Shannon entropy weights for a [0, 1] normalized tool by metric matrix.
standard_deviation_weights	Standard deviation weights for a [0, 1] normalized tool by metric matrix.
critic_weights	CRITIC weights for a [0, 1] normalized tool by metric matrix.
merec_weights	MEREC weights for a [0, 1] normalized tool by metric matrix.
ahp_weights	Analytic Hierarchy Process weights from a pairwise comparison matrix.

Sensitivity

Stability and perturbation primitives.

leave_one_metric_out	Run the pipeline once with all metrics, then once per metric omission.
leave_one_dataset_out	Pool all datasets and rank, then re-rank with each dataset left out.
smaa	Stochastic multi-criteria acceptability analysis on an MCDA run.
smallest_weight_perturbation	Compute the smallest single-weight change that swaps each pair of tools.
aggregation_agreement	How much the recommendation depends on the choice of aggregation method.
rank_sensitivity	Split a ranking’s variance between the analyst’s choices and the data.
specification_curve	List every ranking the analyst’s choices produce, and report how stable it is.
dataset_discrimination	How much each dataset separates the methods it scores.
difficulty_concordance	Whether method families find the same datasets hard.
pairwise_superiority	Compare every method pair across datasets with a probability of superiority.
critical_difference	Run the Friedman test and Nemenyi post-hoc on a tool by dataset matrix.
skillings_mack	Skillings-Mack (1981) test: coverage-aware Friedman for incomplete blocks.
coverage_aware_critical_difference	Convenience wrapper around :func:skillings_mack for the CD use case.
attribution_synthesis	Bundle an ordered list of attribution settings into one report.
pairwise_transitivity	Test the pairwise majority relation for one consistent order.
bayesian_sign_comparison	Posterior probability that one method is practically better than another.
noise_floor_separation	Test every pair of tools against the per-metric noise floors.
beats_random_baseline	Count, per metric, how many tools score better than chance.
normalization_agreement	How much the recommendation depends on the choice of normalization.
dataset_concordance	Agreement between datasets on how they order the methods.

Diagnostics

Card, reliability and validity checks.

card_data_consistency	Audit each metric card’s declared numeric claims against the raw scores.
metric_reliability	Standardized Cronbach’s alpha for each construct group of metrics.
metric_validity	Convergent and discriminant validity of a metric set.
metric_dimensionality	Count the factors in each construct group of metrics.

Heterogeneity

Mixed-effects, Bradley-Terry trees, Plackett-Luce.

mixed_effects	Mixed-effects variance decomposition on benchmark scores.
bradley_terry_tree	Fit a Bradley-Terry tree on per-dataset paired method comparisons.
plackett_luce	Plackett-Luce models on per-dataset method rankings.
source_variance_decomposition	Decompose benchmark-score variance into method, benchmark, and interaction.
paired_comparisons	Turn a method by dataset score matrix into per-dataset paired comparisons.
rankings_from_matrix	Turn a method by dataset score matrix into a per-dataset ranking matrix.
network_meta_analysis	Pool benchmark results into one coherent method ranking.

Cards and registry

Registry	Discover every metric card under metrics_dir and look them up by id.
MetricCard	A loaded and validated metric card.
polarities_for	Look up the polarity string for each metric id, in order.
properties_for	Look up a MetricProperties record for each metric id, in order.

Datasets

load_duo2018	Load the bundled Duo 2018 clustering benchmark.
load_m4	Load the bundled M4 forecasting results table.
load_openproblems	Load a bundled OpenProblems task as a method by dataset by metric tensor.
load_integration_benchmarks	Load the harmonized scIB, OpenProblems and Tran integration benchmark set.
load_pancreas_contrast	Load Tran D4 versus scIB pancreas, the unconfounded benchmarker contrast.
load_gptcelltype	Load the bundled GPTCelltype (Hou and Ji 2024) annotation agreement scores.
load_deepcellseek	Load the bundled DeepCellSeek (Briefings 2025) annotation agreement scores.