Cross-benchmark meta-analysis

Author

Izaskun Mallona

Published

October 7, 2026

Question

Single-cell data-integration benchmarks rank the same methods differently. The differences can come from the methods, the datasets, or the analysis choices of each benchmark (metrics, scaling, weighting).

Benchmarks

Five benchmarks are pooled. Four publish scores per method on the scIB metrics (ARI, ASW, kBET, LISI). BatchBench has its own batch and cell-type entropy and is used as within-benchmark ranks.

Benchmark Paper Methods covered Metrics covered Scores from
scIB Luecken et al., Nature Methods 2022, 10.1038/s41592-021-01336-8 all five ARI, ASW, kBET, LISI scib-reproducibility repository
OpenProblems batch integration OpenProblems consortium, Nature Biotechnology 2025, 10.1038/s41587-025-02694-w all five ARI, ASW, kBET, LISI bundled CC-BY scores
Tran et al. Tran et al., Genome Biology 2020, 10.1186/s13059-019-1850-9 all five ARI, ASW, kBET, LISI supplementary Tables S4 and S7
Tyler et al. Tyler, Guccione and Schadt, bioRxiv 2023, 10.1101/2021.11.15.468733 harmony, scanorama, liger ARI, ASW, kBET Extended Data Table 2
BatchBench Chazarra-Gil et al., Nucleic Acids Research 2021, 10.1093/nar/gkab004 combat, harmony, fastMNN, scanorama batch entropy, cell-type entropy Supplementary Table 1, provided by the author
Shen 2026 (discrimination axis only) Shen, He and Guan, PLOS Comput Biol 2026, 10.1371/journal.pcbi.1014008 ten semi-supervised and unsupervised DL methods (overlap: harmony, scanorama only) own ARI, NMI, ASW and others authors’ repository, pinned commit

Tyler is a preprint with three of the five methods and three of the four metrics, so it has partial coverage. Its kBET is the raw rejection rate, lower is better, and the loader applies that polarity. BatchBench has four of the five methods (no LIGER) and two metrics: batch entropy, higher is better, and cell-type entropy, lower is better. Ruben Chazarra-Gil provided the scores by personal communication.

Shen 2026 is not pooled: its methods overlap the five only on harmony and scanorama. It is used in the dataset discrimination section.

The five methods shared by the first three benchmarks are combat, harmony, fastMNN, scanorama and LIGER. The excluded benchmarks, provenance, licenses and the dataset crosswalk are in src/beam/data/README.md.

%matplotlib inline
import numpy as np
from collections import defaultdict
from itertools import combinations
from scipy.stats import spearmanr, rankdata
from IPython.display import display

from beam import plot
from beam.datasets import load_integration_benchmarks, load_integration_published_ranks
from beam.reporting import funky_heatmap

CANON = ["combat", "harmony", "fastmnn", "scanorama", "liger"]
BENCH = ["Tran", "scIB", "OpenProblems"]
METRICS = ["ARI", "ASW", "kBET", "LISI"]

ib = load_integration_benchmarks()
published = load_integration_published_ranks()
print("records:", len(ib.rank), "| benchmarks:", sorted(set(ib.benchmark)))
records: 643 | benchmarks: ['BatchBench', 'OpenProblems', 'Tran', 'Tyler', 'scIB']

The reported rankings

Each benchmark reported a ranking of the five methods with its own metrics, scaling and weighting: Tran’s Table S7 final rank, scIB’s 0.6 biological / 0.4 batch overall score, and the OpenProblems mean of scaled scores.

print("reported rank of each method (1 best):")
print(f"  {'method':10s}", "  ".join(f"{b:>12s}" for b in BENCH))
for m in CANON:
    print(f"  {m:10s}", "  ".join(f"{published[b][m]:>12d}" for b in BENCH))

pub_vectors = {b: np.array([published[b][m] for m in CANON]) for b in BENCH}
pub_rhos = [spearmanr(pub_vectors[a], pub_vectors[b]).correlation for a, b in combinations(BENCH, 2)]
print(f"\nmean cross-benchmark Spearman of the reported ranks: {np.mean(pub_rhos):.2f}")
reported rank of each method (1 best):
  method             Tran          scIB  OpenProblems
  combat                5             2             1
  harmony               1             3             2
  fastmnn               4             1             3
  scanorama             3             4             5
  liger                 2             5             4

mean cross-benchmark Spearman of the reported ranks: -0.10

Re-ranking with one rule

The three benchmarks are re-ranked from their raw scores with one rule: four shared metrics, equal weights, ranks within the five methods. The datasets are pooled per benchmark.

cell = defaultdict(list)
for b, d, m, mk, r in zip(ib.benchmark, ib.dataset, ib.method, ib.metric, ib.rank, strict=True):
    cell[(b, m)].append(r)
beam_mean = {(b, m): float(np.mean(cell[(b, m)])) for b in BENCH for m in CANON}
beam_rank = {b: dict(zip(CANON, rankdata([beam_mean[(b, m)] for m in CANON], method="ordinal"), strict=True)) for b in BENCH}

beam_vectors = {b: np.array([beam_mean[(b, m)] for m in CANON]) for b in BENCH}
beam_rhos = [spearmanr(beam_vectors[a], beam_vectors[b]).correlation for a, b in combinations(BENCH, 2)]
print(f"mean cross-benchmark Spearman, beam consistent ranking: {np.mean(beam_rhos):.2f}")
print(f"reported: {np.mean(pub_rhos):+.2f}   beam: {np.mean(beam_rhos):+.2f}   change: {np.mean(beam_rhos) - np.mean(pub_rhos):+.2f}")
mean cross-benchmark Spearman, beam consistent ranking: 0.50
reported: -0.10   beam: +0.50   change: +0.60

Reported ranks and consensus

Reported rank of each method in the three benchmarks (left of the divider) and the consensus of the re-ranked benchmarks (right):

consensus = dict(zip(
    CANON,
    rankdata([np.mean([beam_mean[(b, m)] for b in BENCH]) for m in CANON], method="ordinal"),
    strict=True,
))
columns = [f"{b}\n(reported)" for b in BENCH] + ["beam\nconsensus"]
ranks = np.array([[published[b][m] for b in BENCH] + [int(consensus[m])] for m in CANON])
fig = plot.rank_bump(
    tuple(CANON), tuple(columns), ranks, divider_after=2,
    title="Reported method ranks across three benchmarks, and the beam consensus",
)
display(fig)

Funky heatmaps per benchmark

The funky heatmap of each benchmark. Each circle is the standing of the method on that metric within the benchmark, coloured by the scIB biological and batch groups.

groups = ["bio", "bio", "batch", "batch"]  # ARI, ASW | kBET, LISI
n = len(CANON)
for b in BENCH:
    norm = np.zeros((n, len(METRICS)))
    for j, mk in enumerate(METRICS):
        ranks_bm = [np.mean([r for bb, _d, mm, mkk, r in zip(ib.benchmark, ib.dataset, ib.method, ib.metric, ib.rank, strict=True) if bb == b and mm == m and mkk == mk] or [np.nan]) for m in CANON]
        norm[:, j] = 1.0 - (np.array(ranks_bm) - 1.0) / (n - 1)  # best rank -> 1
    composite = np.nanmean(norm, axis=1)
    order = rankdata(-composite, method="ordinal")
    fig = funky_heatmap(
        norm, tuple(CANON), tuple(METRICS), composite, order,
        metric_groups=tuple(groups), title=f"{b}: standing of the common methods per metric",
    )
    display(fig)

BatchBench

BatchBench has two entropy metrics: batch entropy, higher when the batches are better mixed, and cell-type entropy, higher when the cell types are less separated.

import matplotlib.pyplot as plt

from beam.datasets import load_batchbench

bb = load_batchbench()
bi = bb.metric_ids.index("batch_entropy")
ci = bb.metric_ids.index("cell_type_entropy")
batch_mean = np.nanmean(bb.scores[:, :, bi], axis=1)
cell_mean = np.nanmean(bb.scores[:, :, ci], axis=1)

fig, ax = plt.subplots(figsize=(6.5, 4.5))
ax.scatter(batch_mean, cell_mean, color="tab:blue")
for x, y, name in zip(batch_mean, cell_mean, bb.method_names, strict=True):
    ax.annotate(name, (x, y), xytext=(4, 0), textcoords="offset points", fontsize=8)
ax.set_xlabel("batch entropy (higher = batches better mixed)")
ax.set_ylabel("cell-type entropy (higher = cell types more blurred)")
ax.set_title("BatchBench: batch mixing against biology")
fig.tight_layout()
plt.show()

A method low and to the right mixes batches and keeps the cell types separate.

Variance between benchmarks

The variance decomposition is score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark). The method-by-benchmark component measures how differently the benchmarks rank the methods. The model is fitted on the four scIB-metric sources, then again with BatchBench added. It needs lme4.

from beam.heterogeneity import r_available, source_variance_decomposition

methods, datasets, benchmarks, scores = ib.mean_rank_records()
if r_available():
    def share(keep):
        idx = [i for i in range(len(benchmarks)) if keep(benchmarks[i])]
        rep = source_variance_decomposition(
            [methods[i] for i in idx], [datasets[i] for i in idx],
            [benchmarks[i] for i in idx], [scores[i] for i in idx],
        )
        return rep.method_benchmark_share
    s4 = share(lambda b: b != "BatchBench")
    s5 = share(lambda b: True)
    print(f"method-by-benchmark share: {s4:.2f} without BatchBench, {s5:.2f} with it")
else:
    print("R with lme4 not available; skipping the variance decomposition.")
    print("Provision it with envs/heterogeneity.yml.")
method-by-benchmark share: 0.23 without BatchBench, 0.42 with it

The method-by-benchmark fraction of the variance is 0.23 with four sources and 0.42 with BatchBench. BatchBench ranks four of the five methods, and Tyler three. The fits are singular: with so few benchmarks at least one variance component is estimated at zero.

Network meta-analysis

A network meta-analysis pools the five sources into one ranking. Each method is a treatment and each (benchmark, dataset) pair a study; the effect of a method in a study is its mean rank over the metrics, with the standard deviation across them. netmeta reports a P-score per method (higher is better) and heterogeneity and inconsistency statistics. It needs R’s netmeta.

from beam.heterogeneity import netmeta_available, network_meta_analysis

treatment, study, mean, sd, n = ib.network_arms()
if netmeta_available():
    nma = network_meta_analysis(treatment, study, mean, sd, n)
    print(f"studies: {nma.n_studies}, treatments: {nma.n_treatments}, comparisons: {nma.n_comparisons}")
    print(f"ranking (best first): {', '.join(nma.ranking())}")
    print("P-score per method (higher is better):")
    for m, p in sorted(zip(nma.treatments, nma.pscore), key=lambda x: -x[1]):
        print(f"  {m:10s} {p:.2f}")
    print(f"I-squared: {nma.i2:.2f}")
    print(f"inconsistency Q: {nma.q_inconsistency:.1f} on {nma.df_inconsistency:.0f} df, p = {nma.pval_inconsistency:.1e}")
else:
    print("R with netmeta not available; skipping the network meta-analysis.")
    print("Provision it with envs/heterogeneity.yml.")
studies: 51, treatments: 5, comparisons: 307
ranking (best first): liger, harmony, fastmnn, scanorama, combat
P-score per method (higher is better):
  liger      0.83
  harmony    0.83
  fastmnn    0.58
  scanorama  0.25
  combat     0.00
I-squared: 0.67
inconsistency Q: 103.0 on 19 df, p = 1.5e-13

harmony and liger have the same P-score, 0.83, followed by fastMNN at 0.58, scanorama at 0.25 and combat at 0. I-squared is 0.67 and the inconsistency Q is significant.

Human pancreas in two pipelines

Tran’s Dataset 4 has five pancreas studies (Muraro, Segerstolpe, Baron, Wang and Xin) and the scIB pancreas task has six; four of the studies (Muraro, Segerstolpe, Baron and Xin) are shared. The two pipelines are compared on these overlapping pancreas data.

from beam.datasets import load_pancreas_contrast

pc = load_pancreas_contrast()
tran_top, scib_top = pc.top_method()
print(f"top method on Tran D4:           {tran_top}")
print(f"top method on scIB pancreas:     {scib_top}")
print(f"Spearman of mean ranks (5 methods): {pc.spearman():+.2f}")
print()
print("mean rank per method (1 best, 5 worst):")
for i, m in enumerate(pc.methods):
    print(f"  {m:10s}  Tran D4 {pc.tran_mean_rank[i]:.2f}    scIB pancreas {pc.scib_mean_rank[i]:.2f}")
top method on Tran D4:           harmony
top method on scIB pancreas:     harmony
Spearman of mean ranks (5 methods): +0.46

mean rank per method (1 best, 5 worst):
  combat      Tran D4 4.50    scIB pancreas 4.00
  harmony     Tran D4 1.50    scIB pancreas 1.50
  fastmnn     Tran D4 4.00    scIB pancreas 2.00
  scanorama   Tran D4 3.00    scIB pancreas 3.50
  liger       Tran D4 2.00    scIB pancreas 4.00

The Spearman correlation of the two rankings is 0.46.

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(6, 3.2))
xs = np.arange(len(pc.methods))
ax.bar(xs - 0.18, pc.tran_mean_rank, width=0.34, label="Tran D4")
ax.bar(xs + 0.18, pc.scib_mean_rank, width=0.34, label="scIB pancreas")
ax.set_xticks(xs)
ax.set_xticklabels(pc.methods)
ax.set_ylabel("mean rank across 4 metrics (lower ranks first)")
ax.set_xlabel("method")
ax.set_title("Overlapping pancreas data, two pipelines")
ax.invert_yaxis()
ax.legend(loc="upper right")
fig.tight_layout()
plt.show()

The smallest single-weight change swapping the top method of each benchmark, with the Triantaphyllou-Sanchez closed form on the mean-rank matrix under SAW and equal weights:

from beam.mcda import smallest_weight_perturbation

polarity = ("lower_is_better",) * 4
print("smallest single-weight change that flips the top method:")
print(f"  {'benchmark':12s} {'top':10s} {'flipped to':10s}  {'metric':6s}  abs delta")
for bench in BENCH:
    _, metrics_b, matrix = ib.method_metric_matrix(bench)
    rep = smallest_weight_perturbation(matrix, polarity=polarity, weights="equal", method="saw")
    top_idx = int(rep.base.ranks.argmin())
    p = rep.top_rank_perturbation
    if p is None:
        print(f"  {bench:12s} {CANON[top_idx]:10s} {'-':10s}  {'-':6s}  none found")
    else:
        challenger = CANON[p.lower]
        print(
            f"  {bench:12s} {CANON[top_idx]:10s} {challenger:10s}  "
            f"{metrics_b[p.criterion]:6s}  {p.absolute_delta:.3f}"
        )
smallest single-weight change that flips the top method:
  benchmark    top        flipped to  metric  abs delta
  Tran         harmony    liger       ASW     0.239
  scIB         fastmnn    liger       LISI    0.902
  OpenProblems fastmnn    harmony     ASW     0.114

Specification curve per benchmark

rank_sensitivity ranks under every weighting and aggregation on the mean-rank matrix, and specification_curve reports how often the top method is the same.

from beam.mcda import rank_sensitivity, specification_curve

print("specification curve per benchmark (weighting x aggregation on the mean-rank matrix):")
for bench in BENCH:
    _, _, matrix = ib.method_metric_matrix(bench)
    curve = specification_curve(rank_sensitivity(matrix, polarity, tool_names=CANON))
    dom = curve.tool_names[curve.most_frequent_top_tool]
    print(f"  {bench:12s} {dom:10s} first in {curve.most_frequent_top_fraction * 100:3.0f}% of "
          f"{curve.n_specifications} combinations, {curve.n_distinct_top_tools} method(s) reach the top")
specification curve per benchmark (weighting x aggregation on the mean-rank matrix):
  Tran         harmony    first in 100% of 20 combinations, 1 method(s) reach the top
  scIB         fastmnn    first in 100% of 20 combinations, 1 method(s) reach the top
  OpenProblems harmony    first in  55% of 20 combinations, 3 method(s) reach the top

The rank variance split per method on OpenProblems. Each cell of the matrix is a mean rank over the datasets of the benchmark, so the factors are the weighting and the aggregation only.

op_matrix = ib.method_metric_matrix("OpenProblems")[2]
op_rs = rank_sensitivity(op_matrix, polarity, tool_names=CANON)
plot.rank_sensitivity_by_tool(op_rs, title="OpenProblems: rank variance by factor and method")

Blind analysis

A blind analysis fixes the weighting and the aggregation before the method names are known. These benchmarks are on the mean-rank scale and have no metric cards, so beam.mcda.run is called with explicit polarity. beam.blind relabels and shuffles the rows; the seal maps the labels back.

from beam import blind, Scores
from beam.mcda import run as mcda_run

_, metrics_b, matrix = ib.method_metric_matrix("Tran")
scores = Scores(
    values=matrix, tool_names=tuple(CANON), metric_ids=tuple(metrics_b),
    dataset_names=None, layout="wide",
)
blinded, seal = blind(scores, seed=0)

named = mcda_run(scores.values, polarity, weights="equal", method="saw")
blind_res = mcda_run(blinded.values, polarity, weights="equal", method="saw")
order_named = [scores.tool_names[i] for i in named.ranks.argsort()]
order_unblinded = [seal.true_name(blinded.tool_names[i]) for i in blind_res.ranks.argsort()]
print("blinded labels:", blinded.tool_names)
print("order, named:     ", order_named)
print("order, unblinded: ", order_unblinded)
print("same order after unblinding:", order_named == order_unblinded)
blinded labels: ('method_1', 'method_2', 'method_3', 'method_4', 'method_5')
order, named:      ['harmony', 'liger', 'scanorama', 'fastmnn', 'combat']
order, unblinded:  ['harmony', 'liger', 'scanorama', 'fastmnn', 'combat']
same order after unblinding: True

Dataset discrimination

Shen, He and Guan (2026, 10.1371/journal.pcbi.1014008) score five semi-supervised methods (scANVI, scGEN, STACAS, scDREAMER, ItClust) and five unsupervised ones (Seurat, scVI, Harmony, scanorama, scCRAFT) on six datasets, under annotation scenarios with decreasing label supervision. See the dataset discrimination explanation.

dataset_discrimination takes each (dataset, scenario) pair as a unit and reports the spread of the pooled method scores and Kendall’s W over the method-by-metric matrix of the unit.

from beam.datasets import load_openproblems, load_semisupervised_integration
from beam.mcda import dataset_discrimination, difficulty_concordance

shen = load_semisupervised_integration()
disc = dataset_discrimination(shen.scores, shen.polarity, dataset_ids=shen.unit_names)
print(f"{len(shen.unit_names)} units (6 datasets x annotation scenarios), {len(shen.method_names)} methods")
print(f"mean discrimination (spread): {disc.mean_spread:.3f}, mean concordance (W): {disc.mean_kendall_w:.2f}")
print(f"most discriminating: {disc.most_discriminating}")
print(f"least discriminating: {disc.least_discriminating}")
display(plot.dataset_discrimination(disc, top=15))
104 units (6 datasets x annotation scenarios), 10 methods
mean discrimination (spread): 0.079, mean concordance (W): 0.27
most discriminating: macaque/randomly_wrong_70
least discriminating: human_immune/missing_and_mixing_at_edge_70

Difficulty concordance

difficulty_concordance correlates the difficulty profiles of classical and deep-learning methods across the datasets.

dl = {"scVI", "scANVI", "scGEN", "scDREAMER", "scCRAFT", "ItClust"}
families = ["DL" if m in dl else "classical" for m in shen.method_names]
shen_conc = difficulty_concordance(shen.scores, shen.polarity, families, dataset_ids=shen.unit_names)
print(f"Shen (annotation-scenario hardness): concordance {shen_conc.concordance[0, 1]:+.2f}")
print(f"  families disagree most on: {shen_conc.most_divergent_dataset}")

op = load_openproblems("batch_integration")
op_metrics = [m for m in op.metric_ids if m != "hvg_overlap"]
op_tensor = op.tensor(tuple(op_metrics))
op_pol = tuple("higher_is_better" for _ in op_metrics)
op_dl = {"scvi", "scanvi", "scalex", "scgpt_finetuned", "scgpt_zeroshot",
         "geneformer", "scprint", "uce", "scimilarity"}
op_families = ["DL" if m in op_dl else "classical" for m in op.method_names]
op_conc = difficulty_concordance(op_tensor, op_pol, op_families, dataset_ids=op.dataset_names)
print(f"OpenProblems (intrinsic-complexity hardness): concordance {op_conc.concordance[0, 1]:+.2f}")
display(plot.difficulty_concordance(shen_conc, title="Shen 2026: do DL and classical find the same units hard?"))
Shen (annotation-scenario hardness): concordance +0.32
  families disagree most on: macaque/randomly_wrong_70
OpenProblems (intrinsic-complexity hardness): concordance +0.89

scIB classical and deep-learning methods

scIB scores deep-learning methods (scVI, scANVI, scGen, DESC, SAUCIE, trVAE) on the same five datasets and four metrics as the classical five. load_scib_integration_families returns both sets; the deep-learning scores come from the same scib-reproducibility table through reduce_scib_dl.py.

from beam.datasets import load_scib_integration_families

fam = load_scib_integration_families()
scib_conc = difficulty_concordance(fam.scores, fam.polarity, fam.families, dataset_ids=fam.dataset_names)
print(f"scIB (intrinsic complexity, identical data for both families): concordance {scib_conc.concordance[0, 1]:+.2f}")

shared = sorted({m.lower() for m in fam.method_names} & {m.lower() for m in shen.method_names})
print("methods now shared with Shen:", shared)
display(plot.difficulty_concordance(scib_conc, title="scIB: do DL and classical find the same datasets hard?"))
scIB (intrinsic complexity, identical data for both families): concordance +0.90
methods now shared with Shen: ['harmony', 'scanorama', 'scanvi', 'scgen', 'scvi']

Limits

The reported-rank chart and the Spearman comparison use the first three benchmarks, as Tyler publishes no overall ranking. The reported rankings are reconstructed: Tran’s final rank directly, scIB’s 0.6/0.4 overall and the OpenProblems mean score recomputed. A Spearman over five methods is coarse, and five benchmarks are few for a method-by-benchmark variance estimate. The benchmarks use mostly different datasets; only the human pancreas data overlap, and only in part.