Single-cell data-integration benchmarks rank the same methods differently. The differences can come from the methods, the datasets, or the analysis choices of each benchmark (metrics, scaling, weighting).
Benchmarks
Five benchmarks are pooled. Four publish scores per method on the scIB metrics (ARI, ASW, kBET, LISI). BatchBench has its own batch and cell-type entropy and is used as within-benchmark ranks.
ten semi-supervised and unsupervised DL methods (overlap: harmony, scanorama only)
own ARI, NMI, ASW and others
authors’ repository, pinned commit
Tyler is a preprint with three of the five methods and three of the four metrics, so it has partial coverage. Its kBET is the raw rejection rate, lower is better, and the loader applies that polarity. BatchBench has four of the five methods (no LIGER) and two metrics: batch entropy, higher is better, and cell-type entropy, lower is better. Ruben Chazarra-Gil provided the scores by personal communication.
Shen 2026 is not pooled: its methods overlap the five only on harmony and scanorama. It is used in the dataset discrimination section.
The five methods shared by the first three benchmarks are combat, harmony, fastMNN, scanorama and LIGER. The excluded benchmarks, provenance, licenses and the dataset crosswalk are in src/beam/data/README.md.
Each benchmark reported a ranking of the five methods with its own metrics, scaling and weighting: Tran’s Table S7 final rank, scIB’s 0.6 biological / 0.4 batch overall score, and the OpenProblems mean of scaled scores.
print("reported rank of each method (1 best):")print(f" {'method':10s}", " ".join(f"{b:>12s}"for b in BENCH))for m in CANON:print(f" {m:10s}", " ".join(f"{published[b][m]:>12d}"for b in BENCH))pub_vectors = {b: np.array([published[b][m] for m in CANON]) for b in BENCH}pub_rhos = [spearmanr(pub_vectors[a], pub_vectors[b]).correlation for a, b in combinations(BENCH, 2)]print(f"\nmean cross-benchmark Spearman of the reported ranks: {np.mean(pub_rhos):.2f}")
reported rank of each method (1 best):
method Tran scIB OpenProblems
combat 5 2 1
harmony 1 3 2
fastmnn 4 1 3
scanorama 3 4 5
liger 2 5 4
mean cross-benchmark Spearman of the reported ranks: -0.10
Re-ranking with one rule
The three benchmarks are re-ranked from their raw scores with one rule: four shared metrics, equal weights, ranks within the five methods. The datasets are pooled per benchmark.
cell = defaultdict(list)for b, d, m, mk, r inzip(ib.benchmark, ib.dataset, ib.method, ib.metric, ib.rank, strict=True): cell[(b, m)].append(r)beam_mean = {(b, m): float(np.mean(cell[(b, m)])) for b in BENCH for m in CANON}beam_rank = {b: dict(zip(CANON, rankdata([beam_mean[(b, m)] for m in CANON], method="ordinal"), strict=True)) for b in BENCH}beam_vectors = {b: np.array([beam_mean[(b, m)] for m in CANON]) for b in BENCH}beam_rhos = [spearmanr(beam_vectors[a], beam_vectors[b]).correlation for a, b in combinations(BENCH, 2)]print(f"mean cross-benchmark Spearman, beam consistent ranking: {np.mean(beam_rhos):.2f}")print(f"reported: {np.mean(pub_rhos):+.2f} beam: {np.mean(beam_rhos):+.2f} change: {np.mean(beam_rhos) - np.mean(pub_rhos):+.2f}")
Reported rank of each method in the three benchmarks (left of the divider) and the consensus of the re-ranked benchmarks (right):
consensus =dict(zip( CANON, rankdata([np.mean([beam_mean[(b, m)] for b in BENCH]) for m in CANON], method="ordinal"), strict=True,))columns = [f"{b}\n(reported)"for b in BENCH] + ["beam\nconsensus"]ranks = np.array([[published[b][m] for b in BENCH] + [int(consensus[m])] for m in CANON])fig = plot.rank_bump(tuple(CANON), tuple(columns), ranks, divider_after=2, title="Reported method ranks across three benchmarks, and the beam consensus",)display(fig)
Funky heatmaps per benchmark
The funky heatmap of each benchmark. Each circle is the standing of the method on that metric within the benchmark, coloured by the scIB biological and batch groups.
groups = ["bio", "bio", "batch", "batch"] # ARI, ASW | kBET, LISIn =len(CANON)for b in BENCH: norm = np.zeros((n, len(METRICS)))for j, mk inenumerate(METRICS): ranks_bm = [np.mean([r for bb, _d, mm, mkk, r inzip(ib.benchmark, ib.dataset, ib.method, ib.metric, ib.rank, strict=True) if bb == b and mm == m and mkk == mk] or [np.nan]) for m in CANON] norm[:, j] =1.0- (np.array(ranks_bm) -1.0) / (n -1) # best rank -> 1 composite = np.nanmean(norm, axis=1) order = rankdata(-composite, method="ordinal") fig = funky_heatmap( norm, tuple(CANON), tuple(METRICS), composite, order, metric_groups=tuple(groups), title=f"{b}: standing of the common methods per metric", ) display(fig)
BatchBench
BatchBench has two entropy metrics: batch entropy, higher when the batches are better mixed, and cell-type entropy, higher when the cell types are less separated.
A method low and to the right mixes batches and keeps the cell types separate.
Variance between benchmarks
The variance decomposition is score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark). The method-by-benchmark component measures how differently the benchmarks rank the methods. The model is fitted on the four scIB-metric sources, then again with BatchBench added. It needs lme4.
from beam.heterogeneity import r_available, source_variance_decompositionmethods, datasets, benchmarks, scores = ib.mean_rank_records()if r_available():def share(keep): idx = [i for i inrange(len(benchmarks)) if keep(benchmarks[i])] rep = source_variance_decomposition( [methods[i] for i in idx], [datasets[i] for i in idx], [benchmarks[i] for i in idx], [scores[i] for i in idx], )return rep.method_benchmark_share s4 = share(lambda b: b !="BatchBench") s5 = share(lambda b: True)print(f"method-by-benchmark share: {s4:.2f} without BatchBench, {s5:.2f} with it")else:print("R with lme4 not available; skipping the variance decomposition.")print("Provision it with envs/heterogeneity.yml.")
method-by-benchmark share: 0.23 without BatchBench, 0.42 with it
The method-by-benchmark fraction of the variance is 0.23 with four sources and 0.42 with BatchBench. BatchBench ranks four of the five methods, and Tyler three. The fits are singular: with so few benchmarks at least one variance component is estimated at zero.
Network meta-analysis
A network meta-analysis pools the five sources into one ranking. Each method is a treatment and each (benchmark, dataset) pair a study; the effect of a method in a study is its mean rank over the metrics, with the standard deviation across them. netmeta reports a P-score per method (higher is better) and heterogeneity and inconsistency statistics. It needs R’s netmeta.
from beam.heterogeneity import netmeta_available, network_meta_analysistreatment, study, mean, sd, n = ib.network_arms()if netmeta_available(): nma = network_meta_analysis(treatment, study, mean, sd, n)print(f"studies: {nma.n_studies}, treatments: {nma.n_treatments}, comparisons: {nma.n_comparisons}")print(f"ranking (best first): {', '.join(nma.ranking())}")print("P-score per method (higher is better):")for m, p insorted(zip(nma.treatments, nma.pscore), key=lambda x: -x[1]):print(f" {m:10s}{p:.2f}")print(f"I-squared: {nma.i2:.2f}")print(f"inconsistency Q: {nma.q_inconsistency:.1f} on {nma.df_inconsistency:.0f} df, p = {nma.pval_inconsistency:.1e}")else:print("R with netmeta not available; skipping the network meta-analysis.")print("Provision it with envs/heterogeneity.yml.")
harmony and liger have the same P-score, 0.83, followed by fastMNN at 0.58, scanorama at 0.25 and combat at 0. I-squared is 0.67 and the inconsistency Q is significant.
Human pancreas in two pipelines
Tran’s Dataset 4 has five pancreas studies (Muraro, Segerstolpe, Baron, Wang and Xin) and the scIB pancreas task has six; four of the studies (Muraro, Segerstolpe, Baron and Xin) are shared. The two pipelines are compared on these overlapping pancreas data.
from beam.datasets import load_pancreas_contrastpc = load_pancreas_contrast()tran_top, scib_top = pc.top_method()print(f"top method on Tran D4: {tran_top}")print(f"top method on scIB pancreas: {scib_top}")print(f"Spearman of mean ranks (5 methods): {pc.spearman():+.2f}")print()print("mean rank per method (1 best, 5 worst):")for i, m inenumerate(pc.methods):print(f" {m:10s} Tran D4 {pc.tran_mean_rank[i]:.2f} scIB pancreas {pc.scib_mean_rank[i]:.2f}")
top method on Tran D4: harmony
top method on scIB pancreas: harmony
Spearman of mean ranks (5 methods): +0.46
mean rank per method (1 best, 5 worst):
combat Tran D4 4.50 scIB pancreas 4.00
harmony Tran D4 1.50 scIB pancreas 1.50
fastmnn Tran D4 4.00 scIB pancreas 2.00
scanorama Tran D4 3.00 scIB pancreas 3.50
liger Tran D4 2.00 scIB pancreas 4.00
The Spearman correlation of the two rankings is 0.46.
The smallest single-weight change swapping the top method of each benchmark, with the Triantaphyllou-Sanchez closed form on the mean-rank matrix under SAW and equal weights:
from beam.mcda import smallest_weight_perturbationpolarity = ("lower_is_better",) *4print("smallest single-weight change that flips the top method:")print(f" {'benchmark':12s}{'top':10s}{'flipped to':10s}{'metric':6s} abs delta")for bench in BENCH: _, metrics_b, matrix = ib.method_metric_matrix(bench) rep = smallest_weight_perturbation(matrix, polarity=polarity, weights="equal", method="saw") top_idx =int(rep.base.ranks.argmin()) p = rep.top_rank_perturbationif p isNone:print(f" {bench:12s}{CANON[top_idx]:10s}{'-':10s}{'-':6s} none found")else: challenger = CANON[p.lower]print(f" {bench:12s}{CANON[top_idx]:10s}{challenger:10s} "f"{metrics_b[p.criterion]:6s}{p.absolute_delta:.3f}" )
smallest single-weight change that flips the top method:
benchmark top flipped to metric abs delta
Tran harmony liger ASW 0.239
scIB fastmnn liger LISI 0.902
OpenProblems fastmnn harmony ASW 0.114
Specification curve per benchmark
rank_sensitivity ranks under every weighting and aggregation on the mean-rank matrix, and specification_curve reports how often the top method is the same.
from beam.mcda import rank_sensitivity, specification_curveprint("specification curve per benchmark (weighting x aggregation on the mean-rank matrix):")for bench in BENCH: _, _, matrix = ib.method_metric_matrix(bench) curve = specification_curve(rank_sensitivity(matrix, polarity, tool_names=CANON)) dom = curve.tool_names[curve.most_frequent_top_tool]print(f" {bench:12s}{dom:10s} first in {curve.most_frequent_top_fraction *100:3.0f}% of "f"{curve.n_specifications} combinations, {curve.n_distinct_top_tools} method(s) reach the top")
specification curve per benchmark (weighting x aggregation on the mean-rank matrix):
Tran harmony first in 100% of 20 combinations, 1 method(s) reach the top
scIB fastmnn first in 100% of 20 combinations, 1 method(s) reach the top
OpenProblems harmony first in 55% of 20 combinations, 3 method(s) reach the top
The rank variance split per method on OpenProblems. Each cell of the matrix is a mean rank over the datasets of the benchmark, so the factors are the weighting and the aggregation only.
op_matrix = ib.method_metric_matrix("OpenProblems")[2]op_rs = rank_sensitivity(op_matrix, polarity, tool_names=CANON)plot.rank_sensitivity_by_tool(op_rs, title="OpenProblems: rank variance by factor and method")
Blind analysis
A blind analysis fixes the weighting and the aggregation before the method names are known. These benchmarks are on the mean-rank scale and have no metric cards, so beam.mcda.run is called with explicit polarity. beam.blind relabels and shuffles the rows; the seal maps the labels back.
from beam import blind, Scoresfrom beam.mcda import run as mcda_run_, metrics_b, matrix = ib.method_metric_matrix("Tran")scores = Scores( values=matrix, tool_names=tuple(CANON), metric_ids=tuple(metrics_b), dataset_names=None, layout="wide",)blinded, seal = blind(scores, seed=0)named = mcda_run(scores.values, polarity, weights="equal", method="saw")blind_res = mcda_run(blinded.values, polarity, weights="equal", method="saw")order_named = [scores.tool_names[i] for i in named.ranks.argsort()]order_unblinded = [seal.true_name(blinded.tool_names[i]) for i in blind_res.ranks.argsort()]print("blinded labels:", blinded.tool_names)print("order, named: ", order_named)print("order, unblinded: ", order_unblinded)print("same order after unblinding:", order_named == order_unblinded)
blinded labels: ('method_1', 'method_2', 'method_3', 'method_4', 'method_5')
order, named: ['harmony', 'liger', 'scanorama', 'fastmnn', 'combat']
order, unblinded: ['harmony', 'liger', 'scanorama', 'fastmnn', 'combat']
same order after unblinding: True
Dataset discrimination
Shen, He and Guan (2026, 10.1371/journal.pcbi.1014008) score five semi-supervised methods (scANVI, scGEN, STACAS, scDREAMER, ItClust) and five unsupervised ones (Seurat, scVI, Harmony, scanorama, scCRAFT) on six datasets, under annotation scenarios with decreasing label supervision. See the dataset discrimination explanation.
dataset_discrimination takes each (dataset, scenario) pair as a unit and reports the spread of the pooled method scores and Kendall’s W over the method-by-metric matrix of the unit.
from beam.datasets import load_openproblems, load_semisupervised_integrationfrom beam.mcda import dataset_discrimination, difficulty_concordanceshen = load_semisupervised_integration()disc = dataset_discrimination(shen.scores, shen.polarity, dataset_ids=shen.unit_names)print(f"{len(shen.unit_names)} units (6 datasets x annotation scenarios), {len(shen.method_names)} methods")print(f"mean discrimination (spread): {disc.mean_spread:.3f}, mean concordance (W): {disc.mean_kendall_w:.2f}")print(f"most discriminating: {disc.most_discriminating}")print(f"least discriminating: {disc.least_discriminating}")display(plot.dataset_discrimination(disc, top=15))
104 units (6 datasets x annotation scenarios), 10 methods
mean discrimination (spread): 0.079, mean concordance (W): 0.27
most discriminating: macaque/randomly_wrong_70
least discriminating: human_immune/missing_and_mixing_at_edge_70
Difficulty concordance
difficulty_concordance correlates the difficulty profiles of classical and deep-learning methods across the datasets.
dl = {"scVI", "scANVI", "scGEN", "scDREAMER", "scCRAFT", "ItClust"}families = ["DL"if m in dl else"classical"for m in shen.method_names]shen_conc = difficulty_concordance(shen.scores, shen.polarity, families, dataset_ids=shen.unit_names)print(f"Shen (annotation-scenario hardness): concordance {shen_conc.concordance[0, 1]:+.2f}")print(f" families disagree most on: {shen_conc.most_divergent_dataset}")op = load_openproblems("batch_integration")op_metrics = [m for m in op.metric_ids if m !="hvg_overlap"]op_tensor = op.tensor(tuple(op_metrics))op_pol =tuple("higher_is_better"for _ in op_metrics)op_dl = {"scvi", "scanvi", "scalex", "scgpt_finetuned", "scgpt_zeroshot","geneformer", "scprint", "uce", "scimilarity"}op_families = ["DL"if m in op_dl else"classical"for m in op.method_names]op_conc = difficulty_concordance(op_tensor, op_pol, op_families, dataset_ids=op.dataset_names)print(f"OpenProblems (intrinsic-complexity hardness): concordance {op_conc.concordance[0, 1]:+.2f}")display(plot.difficulty_concordance(shen_conc, title="Shen 2026: do DL and classical find the same units hard?"))
Shen (annotation-scenario hardness): concordance +0.32
families disagree most on: macaque/randomly_wrong_70
OpenProblems (intrinsic-complexity hardness): concordance +0.89
scIB classical and deep-learning methods
scIB scores deep-learning methods (scVI, scANVI, scGen, DESC, SAUCIE, trVAE) on the same five datasets and four metrics as the classical five. load_scib_integration_families returns both sets; the deep-learning scores come from the same scib-reproducibility table through reduce_scib_dl.py.
from beam.datasets import load_scib_integration_familiesfam = load_scib_integration_families()scib_conc = difficulty_concordance(fam.scores, fam.polarity, fam.families, dataset_ids=fam.dataset_names)print(f"scIB (intrinsic complexity, identical data for both families): concordance {scib_conc.concordance[0, 1]:+.2f}")shared =sorted({m.lower() for m in fam.method_names} & {m.lower() for m in shen.method_names})print("methods now shared with Shen:", shared)display(plot.difficulty_concordance(scib_conc, title="scIB: do DL and classical find the same datasets hard?"))
scIB (intrinsic complexity, identical data for both families): concordance +0.90
methods now shared with Shen: ['harmony', 'scanorama', 'scanvi', 'scgen', 'scvi']
Limits
The reported-rank chart and the Spearman comparison use the first three benchmarks, as Tyler publishes no overall ranking. The reported rankings are reconstructed: Tran’s final rank directly, scIB’s 0.6/0.4 overall and the OpenProblems mean score recomputed. A Spearman over five methods is coarse, and five benchmarks are few for a method-by-benchmark variance estimate. The benchmarks use mostly different datasets; only the human pancreas data overlap, and only in part.
Source Code
---title: "Cross-benchmark meta-analysis"author: "Izaskun Mallona"date: todayformat: html: theme: cosmo toc: true toc-location: left embed-resources: true code-tools: true fig-width: 7 fig-height: 4---## QuestionSingle-cell data-integration benchmarks rank the same methods differently. The differences can come from the methods, the datasets, or the analysis choices of each benchmark (metrics, scaling, weighting).## BenchmarksFive benchmarks are pooled. Four publish scores per method on the scIB metrics (ARI, ASW, kBET, LISI). BatchBench has its own batch and cell-type entropy and is used as within-benchmark ranks.| Benchmark | Paper | Methods covered | Metrics covered | Scores from ||---|---|---|---|---|| scIB | Luecken et al., Nature Methods 2022, [10.1038/s41592-021-01336-8](https://doi.org/10.1038/s41592-021-01336-8) | all five | ARI, ASW, kBET, LISI | scib-reproducibility repository || OpenProblems batch integration | OpenProblems consortium, Nature Biotechnology 2025, [10.1038/s41587-025-02694-w](https://doi.org/10.1038/s41587-025-02694-w) | all five | ARI, ASW, kBET, LISI | bundled CC-BY scores || Tran et al. | Tran et al., Genome Biology 2020, [10.1186/s13059-019-1850-9](https://doi.org/10.1186/s13059-019-1850-9) | all five | ARI, ASW, kBET, LISI | supplementary Tables S4 and S7 || Tyler et al. | Tyler, Guccione and Schadt, bioRxiv 2023, [10.1101/2021.11.15.468733](https://doi.org/10.1101/2021.11.15.468733) | harmony, scanorama, liger | ARI, ASW, kBET | Extended Data Table 2 || BatchBench | Chazarra-Gil et al., Nucleic Acids Research 2021, [10.1093/nar/gkab004](https://doi.org/10.1093/nar/gkab004) | combat, harmony, fastMNN, scanorama | batch entropy, cell-type entropy | Supplementary Table 1, provided by the author || Shen 2026 (discrimination axis only) | Shen, He and Guan, PLOS Comput Biol 2026, [10.1371/journal.pcbi.1014008](https://doi.org/10.1371/journal.pcbi.1014008) | ten semi-supervised and unsupervised DL methods (overlap: harmony, scanorama only) | own ARI, NMI, ASW and others | authors' repository, pinned commit |Tyler is a preprint with three of the five methods and three of the four metrics, so it has [partial coverage](../../docs/explanations/missing-data.md). Its kBET is the raw rejection rate, lower is better, and the loader applies that polarity. BatchBench has four of the five methods (no LIGER) and two metrics: batch entropy, higher is better, and cell-type entropy, lower is better. Ruben Chazarra-Gil provided the scores by personal communication.Shen 2026 is not pooled: its methods overlap the five only on harmony and scanorama. It is used in the dataset discrimination section.The five methods shared by the first three benchmarks are combat, harmony, fastMNN, scanorama and LIGER. The excluded benchmarks, provenance, licenses and the dataset crosswalk are in `src/beam/data/README.md`.```{python}%matplotlib inlineimport numpy as npfrom collections import defaultdictfrom itertools import combinationsfrom scipy.stats import spearmanr, rankdatafrom IPython.display import displayfrom beam import plotfrom beam.datasets import load_integration_benchmarks, load_integration_published_ranksfrom beam.reporting import funky_heatmapCANON = ["combat", "harmony", "fastmnn", "scanorama", "liger"]BENCH = ["Tran", "scIB", "OpenProblems"]METRICS = ["ARI", "ASW", "kBET", "LISI"]ib = load_integration_benchmarks()published = load_integration_published_ranks()print("records:", len(ib.rank), "| benchmarks:", sorted(set(ib.benchmark)))```## The reported rankingsEach benchmark reported a ranking of the five methods with its own metrics, [scaling](../../docs/explanations/normalization-and-scales.md) and [weighting](../../docs/explanations/weighting-schemes.md): Tran's Table S7 final rank, scIB's 0.6 biological / 0.4 batch overall score, and the OpenProblems mean of scaled scores.```{python}print("reported rank of each method (1 best):")print(f" {'method':10s}", " ".join(f"{b:>12s}"for b in BENCH))for m in CANON:print(f" {m:10s}", " ".join(f"{published[b][m]:>12d}"for b in BENCH))pub_vectors = {b: np.array([published[b][m] for m in CANON]) for b in BENCH}pub_rhos = [spearmanr(pub_vectors[a], pub_vectors[b]).correlation for a, b in combinations(BENCH, 2)]print(f"\nmean cross-benchmark Spearman of the reported ranks: {np.mean(pub_rhos):.2f}")```## Re-ranking with one ruleThe three benchmarks are re-ranked from their raw scores with one rule: four shared metrics, equal weights, ranks within the five methods. The datasets are pooled per benchmark.```{python}cell = defaultdict(list)for b, d, m, mk, r inzip(ib.benchmark, ib.dataset, ib.method, ib.metric, ib.rank, strict=True): cell[(b, m)].append(r)beam_mean = {(b, m): float(np.mean(cell[(b, m)])) for b in BENCH for m in CANON}beam_rank = {b: dict(zip(CANON, rankdata([beam_mean[(b, m)] for m in CANON], method="ordinal"), strict=True)) for b in BENCH}beam_vectors = {b: np.array([beam_mean[(b, m)] for m in CANON]) for b in BENCH}beam_rhos = [spearmanr(beam_vectors[a], beam_vectors[b]).correlation for a, b in combinations(BENCH, 2)]print(f"mean cross-benchmark Spearman, beam consistent ranking: {np.mean(beam_rhos):.2f}")print(f"reported: {np.mean(pub_rhos):+.2f} beam: {np.mean(beam_rhos):+.2f} change: {np.mean(beam_rhos) - np.mean(pub_rhos):+.2f}")```## Reported ranks and consensusReported rank of each method in the three benchmarks (left of the divider) and the consensus of the re-ranked benchmarks (right):```{python}consensus =dict(zip( CANON, rankdata([np.mean([beam_mean[(b, m)] for b in BENCH]) for m in CANON], method="ordinal"), strict=True,))columns = [f"{b}\n(reported)"for b in BENCH] + ["beam\nconsensus"]ranks = np.array([[published[b][m] for b in BENCH] + [int(consensus[m])] for m in CANON])fig = plot.rank_bump(tuple(CANON), tuple(columns), ranks, divider_after=2, title="Reported method ranks across three benchmarks, and the beam consensus",)display(fig)```## Funky heatmaps per benchmarkThe [funky heatmap](../../docs/explanations/funky-heatmaps-and-robustness.md) of each benchmark. Each circle is the standing of the method on that metric within the benchmark, coloured by the scIB biological and batch groups.```{python}groups = ["bio", "bio", "batch", "batch"] # ARI, ASW | kBET, LISIn =len(CANON)for b in BENCH: norm = np.zeros((n, len(METRICS)))for j, mk inenumerate(METRICS): ranks_bm = [np.mean([r for bb, _d, mm, mkk, r inzip(ib.benchmark, ib.dataset, ib.method, ib.metric, ib.rank, strict=True) if bb == b and mm == m and mkk == mk] or [np.nan]) for m in CANON] norm[:, j] =1.0- (np.array(ranks_bm) -1.0) / (n -1) # best rank -> 1 composite = np.nanmean(norm, axis=1) order = rankdata(-composite, method="ordinal") fig = funky_heatmap( norm, tuple(CANON), tuple(METRICS), composite, order, metric_groups=tuple(groups), title=f"{b}: standing of the common methods per metric", ) display(fig)```## BatchBenchBatchBench has two entropy metrics: batch entropy, higher when the batches are better mixed, and cell-type entropy, higher when the cell types are less separated.```{python}import matplotlib.pyplot as pltfrom beam.datasets import load_batchbenchbb = load_batchbench()bi = bb.metric_ids.index("batch_entropy")ci = bb.metric_ids.index("cell_type_entropy")batch_mean = np.nanmean(bb.scores[:, :, bi], axis=1)cell_mean = np.nanmean(bb.scores[:, :, ci], axis=1)fig, ax = plt.subplots(figsize=(6.5, 4.5))ax.scatter(batch_mean, cell_mean, color="tab:blue")for x, y, name inzip(batch_mean, cell_mean, bb.method_names, strict=True): ax.annotate(name, (x, y), xytext=(4, 0), textcoords="offset points", fontsize=8)ax.set_xlabel("batch entropy (higher = batches better mixed)")ax.set_ylabel("cell-type entropy (higher = cell types more blurred)")ax.set_title("BatchBench: batch mixing against biology")fig.tight_layout()plt.show()```A method low and to the right mixes batches and keeps the cell types separate.## Variance between benchmarksThe [variance decomposition](../../docs/explanations/attribution-synthesis.md) is `score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark)`. The method-by-benchmark component measures how differently the benchmarks rank the methods. The model is fitted on the four scIB-metric sources, then again with BatchBench added. It needs lme4.```{python}from beam.heterogeneity import r_available, source_variance_decompositionmethods, datasets, benchmarks, scores = ib.mean_rank_records()if r_available():def share(keep): idx = [i for i inrange(len(benchmarks)) if keep(benchmarks[i])] rep = source_variance_decomposition( [methods[i] for i in idx], [datasets[i] for i in idx], [benchmarks[i] for i in idx], [scores[i] for i in idx], )return rep.method_benchmark_share s4 = share(lambda b: b !="BatchBench") s5 = share(lambda b: True)print(f"method-by-benchmark share: {s4:.2f} without BatchBench, {s5:.2f} with it")else:print("R with lme4 not available; skipping the variance decomposition.")print("Provision it with envs/heterogeneity.yml.")```The method-by-benchmark fraction of the variance is 0.23 with four sources and 0.42 with BatchBench. BatchBench ranks four of the five methods, and Tyler three. The fits are singular: with so few benchmarks at least one variance component is estimated at zero.## Network meta-analysisA [network meta-analysis](../../docs/explanations/network-meta-analysis.md) pools the five sources into one ranking. Each method is a treatment and each (benchmark, dataset) pair a study; the effect of a method in a study is its mean rank over the metrics, with the standard deviation across them. netmeta reports a P-score per method (higher is better) and heterogeneity and inconsistency statistics. It needs R's netmeta.```{python}from beam.heterogeneity import netmeta_available, network_meta_analysistreatment, study, mean, sd, n = ib.network_arms()if netmeta_available(): nma = network_meta_analysis(treatment, study, mean, sd, n)print(f"studies: {nma.n_studies}, treatments: {nma.n_treatments}, comparisons: {nma.n_comparisons}")print(f"ranking (best first): {', '.join(nma.ranking())}")print("P-score per method (higher is better):")for m, p insorted(zip(nma.treatments, nma.pscore), key=lambda x: -x[1]):print(f" {m:10s}{p:.2f}")print(f"I-squared: {nma.i2:.2f}")print(f"inconsistency Q: {nma.q_inconsistency:.1f} on {nma.df_inconsistency:.0f} df, p = {nma.pval_inconsistency:.1e}")else:print("R with netmeta not available; skipping the network meta-analysis.")print("Provision it with envs/heterogeneity.yml.")```harmony and liger have the same P-score, 0.83, followed by fastMNN at 0.58, scanorama at 0.25 and combat at 0. I-squared is 0.67 and the inconsistency Q is significant.## Human pancreas in two pipelinesTran's Dataset 4 has five pancreas studies (Muraro, Segerstolpe, Baron, Wang and Xin) and the scIB pancreas task has six; four of the studies (Muraro, Segerstolpe, Baron and Xin) are shared. The two pipelines are compared on these overlapping pancreas data.```{python}from beam.datasets import load_pancreas_contrastpc = load_pancreas_contrast()tran_top, scib_top = pc.top_method()print(f"top method on Tran D4: {tran_top}")print(f"top method on scIB pancreas: {scib_top}")print(f"Spearman of mean ranks (5 methods): {pc.spearman():+.2f}")print()print("mean rank per method (1 best, 5 worst):")for i, m inenumerate(pc.methods):print(f" {m:10s} Tran D4 {pc.tran_mean_rank[i]:.2f} scIB pancreas {pc.scib_mean_rank[i]:.2f}")```The Spearman correlation of the two rankings is 0.46.```{python}import matplotlib.pyplot as pltfig, ax = plt.subplots(figsize=(6, 3.2))xs = np.arange(len(pc.methods))ax.bar(xs -0.18, pc.tran_mean_rank, width=0.34, label="Tran D4")ax.bar(xs +0.18, pc.scib_mean_rank, width=0.34, label="scIB pancreas")ax.set_xticks(xs)ax.set_xticklabels(pc.methods)ax.set_ylabel("mean rank across 4 metrics (lower ranks first)")ax.set_xlabel("method")ax.set_title("Overlapping pancreas data, two pipelines")ax.invert_yaxis()ax.legend(loc="upper right")fig.tight_layout()plt.show()```The smallest single-weight change swapping the top method of each benchmark, with the Triantaphyllou-Sanchez closed form on the mean-rank matrix under [SAW](../../docs/explanations/aggregation-methods.md) and equal weights:```{python}from beam.mcda import smallest_weight_perturbationpolarity = ("lower_is_better",) *4print("smallest single-weight change that flips the top method:")print(f" {'benchmark':12s}{'top':10s}{'flipped to':10s}{'metric':6s} abs delta")for bench in BENCH: _, metrics_b, matrix = ib.method_metric_matrix(bench) rep = smallest_weight_perturbation(matrix, polarity=polarity, weights="equal", method="saw") top_idx =int(rep.base.ranks.argmin()) p = rep.top_rank_perturbationif p isNone:print(f" {bench:12s}{CANON[top_idx]:10s}{'-':10s}{'-':6s} none found")else: challenger = CANON[p.lower]print(f" {bench:12s}{CANON[top_idx]:10s}{challenger:10s} "f"{metrics_b[p.criterion]:6s}{p.absolute_delta:.3f}" )```## Specification curve per benchmark[`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) ranks under every weighting and aggregation on the mean-rank matrix, and [`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) reports how often the top method is the same.```{python}from beam.mcda import rank_sensitivity, specification_curveprint("specification curve per benchmark (weighting x aggregation on the mean-rank matrix):")for bench in BENCH: _, _, matrix = ib.method_metric_matrix(bench) curve = specification_curve(rank_sensitivity(matrix, polarity, tool_names=CANON)) dom = curve.tool_names[curve.most_frequent_top_tool]print(f" {bench:12s}{dom:10s} first in {curve.most_frequent_top_fraction *100:3.0f}% of "f"{curve.n_specifications} combinations, {curve.n_distinct_top_tools} method(s) reach the top")```The rank variance split per method on OpenProblems. Each cell of the matrix is a mean rank over the datasets of the benchmark, so the factors are the weighting and the aggregation only.```{python}op_matrix = ib.method_metric_matrix("OpenProblems")[2]op_rs = rank_sensitivity(op_matrix, polarity, tool_names=CANON)plot.rank_sensitivity_by_tool(op_rs, title="OpenProblems: rank variance by factor and method")```## Blind analysisA [blind analysis](../../docs/explanations/analysis-blinding.md) fixes the weighting and the aggregation before the method names are known. These benchmarks are on the mean-rank scale and have no [metric cards](../../docs/explanations/cards-and-pipeline.qmd), so [`beam.mcda.run`](../../docs/reference/run.qmd) is called with explicit polarity. `beam.blind` relabels and shuffles the rows; the seal maps the labels back.```{python}from beam import blind, Scoresfrom beam.mcda import run as mcda_run_, metrics_b, matrix = ib.method_metric_matrix("Tran")scores = Scores( values=matrix, tool_names=tuple(CANON), metric_ids=tuple(metrics_b), dataset_names=None, layout="wide",)blinded, seal = blind(scores, seed=0)named = mcda_run(scores.values, polarity, weights="equal", method="saw")blind_res = mcda_run(blinded.values, polarity, weights="equal", method="saw")order_named = [scores.tool_names[i] for i in named.ranks.argsort()]order_unblinded = [seal.true_name(blinded.tool_names[i]) for i in blind_res.ranks.argsort()]print("blinded labels:", blinded.tool_names)print("order, named: ", order_named)print("order, unblinded: ", order_unblinded)print("same order after unblinding:", order_named == order_unblinded)```## Dataset discriminationShen, He and Guan (2026, [10.1371/journal.pcbi.1014008](https://doi.org/10.1371/journal.pcbi.1014008)) score five semi-supervised methods (scANVI, scGEN, STACAS, scDREAMER, ItClust) and five unsupervised ones (Seurat, scVI, Harmony, scanorama, scCRAFT) on six datasets, under annotation scenarios with decreasing label supervision. See the [dataset discrimination explanation](../../docs/explanations/dataset-concordance-and-discrimination.md#dataset-discrimination).[`dataset_discrimination`](../../docs/reference/dataset_discrimination.qmd) takes each (dataset, scenario) pair as a unit and reports the spread of the pooled method scores and Kendall's W over the method-by-metric matrix of the unit.```{python}from beam.datasets import load_openproblems, load_semisupervised_integrationfrom beam.mcda import dataset_discrimination, difficulty_concordanceshen = load_semisupervised_integration()disc = dataset_discrimination(shen.scores, shen.polarity, dataset_ids=shen.unit_names)print(f"{len(shen.unit_names)} units (6 datasets x annotation scenarios), {len(shen.method_names)} methods")print(f"mean discrimination (spread): {disc.mean_spread:.3f}, mean concordance (W): {disc.mean_kendall_w:.2f}")print(f"most discriminating: {disc.most_discriminating}")print(f"least discriminating: {disc.least_discriminating}")display(plot.dataset_discrimination(disc, top=15))```### Difficulty concordance[`difficulty_concordance`](../../docs/reference/difficulty_concordance.qmd) correlates the difficulty profiles of classical and deep-learning methods across the datasets.```{python}dl = {"scVI", "scANVI", "scGEN", "scDREAMER", "scCRAFT", "ItClust"}families = ["DL"if m in dl else"classical"for m in shen.method_names]shen_conc = difficulty_concordance(shen.scores, shen.polarity, families, dataset_ids=shen.unit_names)print(f"Shen (annotation-scenario hardness): concordance {shen_conc.concordance[0, 1]:+.2f}")print(f" families disagree most on: {shen_conc.most_divergent_dataset}")op = load_openproblems("batch_integration")op_metrics = [m for m in op.metric_ids if m !="hvg_overlap"]op_tensor = op.tensor(tuple(op_metrics))op_pol =tuple("higher_is_better"for _ in op_metrics)op_dl = {"scvi", "scanvi", "scalex", "scgpt_finetuned", "scgpt_zeroshot","geneformer", "scprint", "uce", "scimilarity"}op_families = ["DL"if m in op_dl else"classical"for m in op.method_names]op_conc = difficulty_concordance(op_tensor, op_pol, op_families, dataset_ids=op.dataset_names)print(f"OpenProblems (intrinsic-complexity hardness): concordance {op_conc.concordance[0, 1]:+.2f}")display(plot.difficulty_concordance(shen_conc, title="Shen 2026: do DL and classical find the same units hard?"))```### scIB classical and deep-learning methodsscIB scores deep-learning methods (scVI, scANVI, scGen, DESC, SAUCIE, trVAE) on the same five datasets and four metrics as the classical five. `load_scib_integration_families` returns both sets; the deep-learning scores come from the same `scib-reproducibility` table through `reduce_scib_dl.py`.```{python}from beam.datasets import load_scib_integration_familiesfam = load_scib_integration_families()scib_conc = difficulty_concordance(fam.scores, fam.polarity, fam.families, dataset_ids=fam.dataset_names)print(f"scIB (intrinsic complexity, identical data for both families): concordance {scib_conc.concordance[0, 1]:+.2f}")shared =sorted({m.lower() for m in fam.method_names} & {m.lower() for m in shen.method_names})print("methods now shared with Shen:", shared)display(plot.difficulty_concordance(scib_conc, title="scIB: do DL and classical find the same datasets hard?"))```## LimitsThe reported-rank chart and the Spearman comparison use the first three benchmarks, as Tyler publishes no overall ranking. The reported rankings are reconstructed: Tran's final rank directly, scIB's 0.6/0.4 overall and the OpenProblems mean score recomputed. A Spearman over five methods is coarse, and five benchmarks are few for a method-by-benchmark variance estimate. The benchmarks use mostly different datasets; only the human pancreas data overlap, and only in part.