Duo 2018 clustering benchmark

Author

Izaskun Mallona

Published

October 7, 2026

Set-up

The five aggregation methods (SAW, TOPSIS, VIKOR, PROMETHEE II, COMET) come from pymcdm, called on the matrix normalized by beam according to the metric cards. The weighting schemes are beam’s own.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
from IPython.display import display

import beam
from beam.datasets import load_duo2018
from beam.cards import properties_for
from beam.mcda import (
    run_from_registry,
    critical_difference,
    smaa,
    smallest_weight_perturbation,
)

Data

load_duo2018 reads the bundled CSV into a method by dataset by metric tensor, with numpy.nan for the NA cells of the source.

duo = load_duo2018()
print("scores shape:", duo.scores.shape)
print("methods:", ", ".join(duo.method_names))
print("datasets:", ", ".join(duo.dataset_names))
print("metrics:", ", ".join(duo.metric_ids))
scores shape: (14, 12, 4)
methods: CIDR, FlowSOM, monocle, PCAHC, PCAKmeans, pcaReduce, RaceID2, RtsneKmeans, SAFE, SC3, SC3svm, Seurat, TSCAN, ascend
datasets: Koh, KohTCC, Kumar, KumarTCC, SimKumar4easy, SimKumar4hard, SimKumar8hard, Trapnell, TrapnellTCC, Zhengmix4eq, Zhengmix4uneq, Zhengmix8eq
metrics: ari, runtime, shannon_entropy_diff, nclust_deviation

Missing cells

ARI, runtime and Shannon entropy difference each have five missing cells. Cluster-count deviation has 101 of 168 cells missing and is left out.

for metric_id in duo.metric_ids:
    n_missing = int(np.isnan(duo.tensor((metric_id,))[:, :, 0]).sum())
    print(f"{metric_id:22s} missing cells: {n_missing:3d} / {14 * 12}")

analysis_metrics = ["ari", "runtime", "shannon_entropy_diff"]
ari                    missing cells:   5 / 168
runtime                missing cells:   5 / 168
shannon_entropy_diff   missing cells:   5 / 168
nclust_deviation       missing cells: 101 / 168

Metric cards

properties_for reads the polarity, scale type, declared range and cross-dataset aggregation of each metric. ARI is higher is better; runtime and Shannon entropy difference are lower is better. Runtime is a ratio metric over orders of magnitude and takes the geometric mean across datasets (Smith 1988); the other two take the arithmetic mean.

props = properties_for(analysis_metrics)
polarity = [p.polarity for p in props]
for p in props:
    print(
        f"{p.id:22s}  polarity={p.polarity:16s}  scale={p.scale_type:8s}  "
        f"range=({p.range_lower}, {p.range_upper})  "
        f"agg_across_datasets={p.recommended_aggregation_across_datasets}"
    )
ari                     polarity=higher_is_better  scale=interval  range=(-1, 1)  agg_across_datasets=arithmetic_mean
runtime                 polarity=lower_is_better   scale=ratio     range=(0, None)  agg_across_datasets=geometric_mean
shannon_entropy_diff    polarity=lower_is_better   scale=ratio     range=(0, 1)  agg_across_datasets=arithmetic_mean

Pooling across datasets

The standalone functions take one value per method and metric, so the datasets are pooled first with NaN-aware means: the arithmetic mean of ARI and Shannon entropy difference, the geometric mean of runtime.

The five missing cells are on SAFE (two datasets) and ascend (three datasets).

def pool_metric(metric_id, rule):
    matrix = duo.tensor((metric_id,))[:, :, 0]
    if rule == "arithmetic_mean":
        return np.nanmean(matrix, axis=1)
    if rule == "geometric_mean":
        return np.exp(np.nanmean(np.log(matrix), axis=1))
    raise ValueError(f"unsupported pooling rule {rule!r}")

pooled = np.column_stack(
    [pool_metric(p.id, p.recommended_aggregation_across_datasets) for p in props]
)
print("pooled matrix shape:", pooled.shape)
print("pooled matrix all finite:", bool(np.isfinite(pooled).all()))
pooled matrix shape: (14, 3)
pooled matrix all finite: True

card_data_consistency checks the pooled raw scores against the ranges, chance baseline and noise floor on the cards.

from beam.mcda import card_data_consistency, registry_context

ctx = registry_context(analysis_metrics, "saw")
audit = card_data_consistency(
    pooled, ctx.polarity, ctx.bounds,
    baselines=ctx.baselines, targets=ctx.targets, noise_floors=ctx.noise_floors,
    metric_ids=analysis_metrics,
)
print("pooled scores consistent with the cards:", audit.ok)
for finding in audit.findings:
    print(" ", finding.severity, finding.message)
pooled scores consistent with the cards: True

Ranking

run_from_registry takes polarity, bounds and the normalization of each metric from the cards, validates the aggregation against the scale types, and ranks. The default is equal weights with simple additive weighting (SAW).

result = run_from_registry(pooled, analysis_metrics, weights="equal", method="saw")

print(f"weighting={result.weighting}  method={result.method}")
print(f"{'method':12s}  composite  rank")
order = np.argsort(result.ranks)
for i in order:
    print(f"{duo.method_names[i]:12s}  {result.composite[i]:.3f}      {result.ranks[i]}")
weighting=equal  method=saw
method        composite  rank
Seurat        0.947      1
PCAKmeans     0.858      2
PCAHC         0.857      3
CIDR          0.832      4
monocle       0.828      5
RtsneKmeans   0.814      6
TSCAN         0.778      7
ascend        0.753      8
FlowSOM       0.725      9
SC3svm        0.672      10
SC3           0.632      11
pcaReduce     0.599      12
RaceID2       0.581      13
SAFE          0.549      14

Weightings and aggregation methods

Four weightings (equal, entropy, std, critic) by four aggregation methods (saw, topsis, vikor, promethee_ii) give sixteen runs.

weightings = ["equal", "entropy", "std", "critic"]
methods = ["saw", "topsis", "vikor", "promethee_ii"]

combos = [(w, m) for w in weightings for m in methods]
ranks_grid = np.zeros((len(combos), len(duo.method_names)), dtype=int)
row_labels = []
for i, (w, m) in enumerate(combos):
    r = run_from_registry(pooled, analysis_metrics, weights=w, method=m)
    ranks_grid[i] = r.ranks
    row_labels.append(f"{w} / {m}")

Ranks as a heatmap, rank 1 in green and rank 14 in red:

from beam import plot

n_methods = len(duo.method_names)
plot.rank_heatmap(
    ranks_grid,
    row_names=row_labels,
    col_names=duo.method_names,
    row_label="weighting / aggregation method",
    col_label="clustering method",
    title="Rank by configuration",
)

top_per_combo = {duo.method_names[int(np.argmin(row))] for row in ranks_grid}
print("methods that are top-ranked in at least one of the 16 configurations:")
print("  " + ", ".join(sorted(top_per_combo)))
methods that are top-ranked in at least one of the 16 configurations:
  Seurat

Critical difference on ARI

The Demsar (2006) Friedman test ranks the methods on each dataset and tests whether the average ranks differ. Two methods whose average ranks differ by less than the Nemenyi critical difference are not separable.

This runs only on ARI, on the complete observations across the twelve datasets. SAFE and ascend have missing ARI cells, so they are dropped. The rest are complete.

ari = duo.tensor(("ari",))[:, :, 0]
complete = duo.complete_methods(("ari",))
block_idx = [i for i in range(n_methods) if complete[i]]
block = ari[block_idx, :]
block_names = [duo.method_names[i] for i in block_idx]
print(f"ARI block: {block.shape[0]} methods by {block.shape[1]} datasets, all observed: "
      f"{bool(np.isfinite(block).all())}")
print("dropped (incomplete ARI):",
      ", ".join(duo.method_names[i] for i in range(n_methods) if not complete[i]))

cd = critical_difference(block, higher_is_better=True, tool_names=tuple(block_names))
print(f"\nFriedman statistic={cd.friedman_statistic:.2f}  p-value={cd.friedman_pvalue:.2e}")
print(f"Nemenyi critical difference (alpha={cd.alpha}): {cd.critical_difference:.2f}")
print("average ranks (1 = best on ARI):")
for i in cd.order:
    print(f"  {block_names[i]:12s}  {cd.average_ranks[i]:.2f}")
ARI block: 12 methods by 12 datasets, all observed: True
dropped (incomplete ARI): SAFE, ascend

Friedman statistic=47.36  p-value=1.86e-06
Nemenyi critical difference (alpha=0.05): 4.81
average ranks (1 = best on ARI):
  SC3           3.17
  RtsneKmeans   4.79
  Seurat        4.79
  SC3svm        4.96
  pcaReduce     5.46
  PCAHC         5.83
  monocle       6.79
  PCAKmeans     7.21
  TSCAN         7.50
  CIDR          8.67
  FlowSOM       9.21
  RaceID2       9.62

Each method is drawn at its average ARI rank, rank 1 on the left. A bar joins methods within the critical difference.

from beam import plot

plot.critical_difference(cd)

Pairwise superiority on ARI

pairwise_superiority compares the methods two at a time across the datasets, with the ARI noise floor (0.01) as the equivalence band. The probability of superiority is the fraction of datasets with one method outperforming the other.

from beam.mcda import pairwise_superiority

ari_floor = properties_for(["ari"])[0].noise_floor or 0.0
sup = pairwise_superiority(ari, "higher_is_better", rope=ari_floor, method_names=duo.method_names)
top_method = duo.method_names[sup.order[0]]
print(f"highest standing: {top_method} ({sup.standing[sup.order[0]]:.2f})")
print(f"method pairs the sign test cannot separate: {len(sup.equivalent_pairs)} of {len(sup.per_pair)}")
highest standing: SC3 (0.80)
method pairs the sign test cannot separate: 60 of 91

Pairwise transitivity on ARI

pairwise_transitivity tests whether the pairwise majorities agree with one order: the method preferred to every other one, the circular triads, and whether the relation is transitive.

from beam.mcda import pairwise_transitivity

trans = pairwise_transitivity(sup)
choice = duo.method_names[trans.condorcet_choice] if trans.condorcet_choice is not None else "none"
print(f"method preferred to every other one: {choice}")
print(f"transitive: {trans.is_transitive}; circular triads: {trans.n_circular_triads} of {trans.n_triads}")
method preferred to every other one: SC3
transitive: False; circular triads: 1 of 364

The matrix orders the methods by the number of others outperformed. A red cell below the diagonal is a method outperforming one ranked above it.

plot.pairwise_majority(trans)

Bayesian comparison on ARI

bayesian_sign_comparison gives, for each pair, the posterior probabilities that one method is practically better, that the two are equivalent within the floor, and that the other is practically better.

from beam.mcda import bayesian_sign_comparison

bayes = bayesian_sign_comparison(sup)
runner = duo.method_names[bayes.order[1]]
pair = next(p for p in bayes.per_pair if {p.a, p.b} == {bayes.order[0], bayes.order[1]})
p_top = pair.p_a_better if pair.a == bayes.order[0] else pair.p_b_better
print(f"P({top_method} practically better than {runner}): {p_top:.2f}")
print(f"P(the two are practically equivalent): {pair.p_equivalent:.2f}")
P(SC3 practically better than SC3svm): 0.07
P(the two are practically equivalent): 0.93

Posterior probability that the row method scores higher than the column method:

plot.bayesian_comparison(bayes)

SMAA

smaa samples weight vectors from a Dirichlet, ranks once per sample, and reports the fraction of samples with each method ranked first.

smaa_report = smaa(pooled, polarity, n_samples=2000, method="topsis", seed=0)
print("confidence factor (fraction of weight samples ranked first), nonzero only:")
for i in np.argsort(-smaa_report.confidence_factor):
    if smaa_report.confidence_factor[i] > 0:
        print(f"  {duo.method_names[i]:12s}  {smaa_report.confidence_factor[i]:.3f}")
confidence factor (fraction of weight samples ranked first), nonzero only:
  Seurat        0.992
  SC3           0.008
fig, ax = plt.subplots(figsize=(8.0, 3.5))
ax.bar(range(n_methods), smaa_report.confidence_factor, color="tab:green")
ax.set_xticks(range(n_methods))
ax.set_xticklabels(duo.method_names, rotation=90)
ax.set_xlabel("clustering method")
ax.set_ylabel("SMAA confidence factor (share top-ranked)")
ax.set_ylim(0, 1)
ax.set_title("SMAA over 2000 Dirichlet weight draws (method=topsis)")
fig.tight_layout()
plt.show()

Smallest weight perturbation

smallest_weight_perturbation reports, under SAW, the smallest single-weight change swapping a pair, and whether a change below the fragility threshold (default 0.05) displaces the top method.

ts = smallest_weight_perturbation(pooled, polarity, weights="equal", method="saw")
top_method = duo.method_names[int(np.argmin(ts.base.ranks))]
print(f"top-ranked under equal weights / SAW: {top_method}")
print(f"top rank is fragile: {ts.top_rank_is_fragile}")
if ts.most_fragile_pair is not None:
    p = ts.most_fragile_pair
    print(
        f"most fragile ordered pair: {duo.method_names[p.higher]} over "
        f"{duo.method_names[p.lower]}, flips by changing the weight on "
        f"{analysis_metrics[p.criterion]!r} by {p.delta:+.3f}"
    )
top-ranked under equal weights / SAW: Seurat
top rank is fragile: False
most fragile ordered pair: TSCAN over SC3svm, flips by changing the weight on 'runtime' by -0.001

beam.rank

beam.rank takes the tensor, pools it across datasets by the rule on each card, ranks, and runs SMAA, leave-one-metric-out, the smallest-weight perturbation and leave-one-dataset-out.

scores = beam.Scores(
    values=duo.tensor(tuple(analysis_metrics)),
    tool_names=tuple(duo.method_names),
    metric_ids=tuple(analysis_metrics),
    dataset_names=duo.dataset_names,
    layout="long",
)
run = beam.rank(scores, weights="equal", method="saw", seed=0)
print("top-ranked under equal weights / SAW:", run.top_tool)
print("matrix ranked (methods by metrics):", run.matrix.shape)
top-ranked under equal weights / SAW: Seurat
matrix ranked (methods by metrics): (14, 3)

Each dataset is dropped in turn and the other eleven are pooled and re-ranked. The stability of a method is the fraction of those runs with its rank unchanged.

lodo = run.leave_one_dataset_out
print(f"datasets evaluated: {len(lodo.evaluated_datasets)} of {duo.scores.shape[1]}")
base_order = np.argsort(run.result.ranks)
print(f"{'method':12s}  base rank  rank held across leave-one-dataset-out runs")
for i in base_order:
    print(f"{duo.method_names[i]:12s}  {run.result.ranks[i]:>4d}       {lodo.rank_stability[i] * 100:5.0f}%")

influential = lodo.dataset_names[lodo.most_influential_dataset]
print(f"\nmost influential dataset: {influential} (largest rank shift {lodo.max_rank_shift})")
datasets evaluated: 12 of 12
method        base rank  rank held across leave-one-dataset-out runs
Seurat           1         100%
PCAKmeans        2          67%
PCAHC            3          67%
CIDR             4          50%
monocle          5          50%
RtsneKmeans      6         100%
TSCAN            7         100%
ascend           8         100%
FlowSOM          9         100%
SC3svm          10         100%
SC3             11         100%
pcaReduce       12          75%
RaceID2         13          75%
SAFE            14         100%

most influential dataset: KohTCC (largest rank shift 1)
plot.dataset_stability(run)

The funky heatmap of the run, with the rank span across the leave-one-dataset-out runs and across the five aggregations. The brackets group the methods not separated by the Friedman-Nemenyi test on ARI.

from beam.mcda import critical_difference
from beam.reporting import funky_heatmap_from_run

ari_for_cd = duo.tensor(("ari",))[:, :, 0]
complete = ~np.isnan(ari_for_cd).any(axis=0)
cd = critical_difference(ari_for_cd[:, complete], "higher_is_better", tool_names=duo.method_names)
cliques = tuple(tuple(duo.method_names[i] for i in clique) for clique in cd.cliques)

funky_heatmap_from_run(
    run,
    cliques=cliques,
    show_smaa=False,
    show_aggregation_consensus=True,
    title="Duo 2018: scores and rank robustness",
)

Dataset concordance

dataset_concordance ranks the methods within each dataset and compares every pair of per-dataset orderings with Kendall tau-b.

conc = run.dataset_concordance
names = conc.dataset_names
print(f"mean agreement across datasets (Kendall tau-b): {conc.mean_pairwise_tau:.2f}")
print("least typical dataset:", names[conc.most_idiosyncratic_dataset])
print("mutually consistent groups:",
      [tuple(names[d] for d in g) for g in conc.concordant_groups])
print("where methods depart most from their own average rank:")
for cell in conc.notable_cells[:5]:
    side = "lower" if cell.deviation > 0 else "higher"
    print(f"  {conc.tool_names[cell.tool]} on {names[cell.dataset]}: "
          f"rank {cell.rank}, {side} than its mean {cell.mean_rank:.1f}")
mean agreement across datasets (Kendall tau-b): 0.39
least typical dataset: Zhengmix4eq
mutually consistent groups: [('Kumar', 'KumarTCC', 'SimKumar4easy', 'Trapnell', 'TrapnellTCC'), ('Zhengmix4eq', 'Zhengmix4uneq', 'Zhengmix8eq')]
where methods depart most from their own average rank:
  CIDR on Zhengmix4eq: rank 13, lower than its mean 5.5
  FlowSOM on SimKumar4easy: rank 3, higher than its mean 10.5
  CIDR on Zhengmix4uneq: rank 12, lower than its mean 5.5
  CIDR on Zhengmix8eq: rank 12, lower than its mean 5.5
  PCAHC on Zhengmix4eq: rank 10, lower than its mean 4.5
plot.dataset_concordance(run)

Rank of each method per dataset minus its mean rank:

plot.dataset_struggle(run)

Specification curve

rank_sensitivity ranks under every combination of weighting, aggregation and dataset. specification_curve lists the rankings and how often the top method is the same, on the pooled matrix (weighting by aggregation) and on the grid with the datasets.

from beam.mcda import rank_sensitivity, specification_curve

rs = rank_sensitivity(
    scores.values,
    run.context.polarity,
    normalization=list(run.context.normalization),
    bounds=list(run.context.bounds),
    baselines=list(run.context.baselines),
    targets=list(run.context.targets),
    missing="worst",
    tool_names=duo.method_names,
    dataset_names=duo.dataset_names,
)
curve = specification_curve(rs)
dom = curve.tool_names[curve.most_frequent_top_tool]
print(f"specifications (weighting x aggregation x dataset): {curve.n_specifications}")
print(f"{dom} ranks first in {curve.most_frequent_top_fraction * 100:.0f}% of them")
print(f"distinct methods reaching the top: {curve.n_distinct_top_tools}")

pooled = rank_sensitivity(
    run.matrix,
    run.context.polarity,
    normalization=list(run.context.normalization),
    bounds=list(run.context.bounds),
    baselines=list(run.context.baselines),
    targets=list(run.context.targets),
    tool_names=duo.method_names,
)
pooled_curve = specification_curve(pooled)
pdom = pooled_curve.tool_names[pooled_curve.most_frequent_top_tool]
print(
    f"choices only ({pooled_curve.n_specifications} combinations): "
    f"{pdom} first in {pooled_curve.most_frequent_top_fraction * 100:.0f}%"
)
specifications (weighting x aggregation x dataset): 240
Seurat ranks first in 50% of them
distinct methods reaching the top: 6
choices only (20 combinations): Seurat first in 100%

Rank of the top method across the grid with the datasets:

plot.specification_curve(curve)

Blind analysis

beam.blind masks the method names and shuffles the rows under a seed; beam.unblind restores the names. The seal fingerprint is recorded in the manifest.

from beam import blind, unblind

blinded, seal = blind(scores, seed=0)
print("blinded labels:", blinded.tool_names[:3], "...")

blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)
restored = unblind(blind_run, seal)
named_run = beam.rank(scores, weights="equal", method="saw", seed=0, sensitivity=False)

named = dict(zip(named_run.tool_names, named_run.result.ranks))
unblinded = dict(zip(restored.tool_names, restored.result.ranks))
print("ranking identical after unblinding:", named == unblinded)
print("top method after unblinding:", restored.top_tool)
print("blinding fingerprint in manifest:", blind_run.manifest["blinding"]["seal_sha256"][:12])
blinded labels: ('method_01', 'method_02', 'method_03') ...
ranking identical after unblinding: True
top method after unblinding: Seurat
blinding fingerprint in manifest: ab94772c1d60

Mixed-effects on ARI

beam.heterogeneity.mixed_effects fits score ~ method + (1 | dataset) in R’s lme4. See the mixed-effects explanation.

from beam.heterogeneity import mixed_effects_from_matrix, r_available

ari = duo.tensor(("ari",))[:, :, 0]

if r_available():
    me = mixed_effects_from_matrix(ari, duo.method_names, duo.dataset_names)
    print(f"model: {me.formula}")
    print(f"dataset shift (ICC): {me.icc_dataset:.2f} of the ARI variance")
    print(f"residual share:      {me.residual_share:.2f} (interaction confounded with noise at one run per cell)")
    order = np.argsort(-me.method_effects)
    print("\nmethod          ARI marginal mean")
    for i in order[:3]:
        print(f"{me.method_names[i]:14s}  {me.method_effects[i]:.3f} +/- {me.method_effect_se[i]:.3f}")
    print("\nlargest interaction residuals (method, dataset, residual):")
    for m, ds, res in me.top_outliers(3):
        print(f"  {m:10s} {ds:16s} {res:+.2f}")
else:
    me = None
    print("R with lme4 not available; skipping the mixed-effects fit.")
    print("Provision it with envs/heterogeneity.yml (conda or mamba).")
model: score ~ method + (1 | dataset)
dataset shift (ICC): 0.71 of the ARI variance
residual share:      0.29 (interaction confounded with noise at one run per cell)

method          ARI marginal mean
SC3             0.853 +/- 0.083
Seurat          0.847 +/- 0.083
SC3svm          0.823 +/- 0.083

largest interaction residuals (method, dataset, residual):
  RaceID2    SimKumar4hard    -0.50
  RaceID2    SimKumar8hard    -0.43
  RaceID2    Zhengmix4eq      +0.42
if me is not None:
    display(plot.model_effects(me, xlabel="ARI marginal mean over datasets (higher ranks first)"))

Bradley-Terry tree

Each dataset becomes a set of pairwise method comparisons on ARI, and psychotree::bttree splits the datasets by their features (number of cells, number of true clusters, real or simulated, family), with a ranking per leaf. It needs R’s psychotree. See the Bradley-Terry explanation.

from beam.datasets import load_duo2018_features
from beam.heterogeneity import bradley_terry_tree, bttree_available

if bttree_available():
    features = load_duo2018_features()
    numeric, categorical = features.aligned_to(duo.dataset_names)
    bt = bradley_terry_tree(
        ari,
        duo.method_names,
        duo.dataset_names,
        numeric_features=numeric,
        categorical_features=categorical,
        polarity="higher_is_better",
        minsize=4,
    )
    print(f"split found: {bt.did_split}")
    print(f"global ranking (top 3): {', '.join(bt.global_ranking()[:3])}")
    print(bt.summary())
else:
    bt = None
    print("R with psychotree not available; skipping the Bradley-Terry tree.")
    print("Provision it with envs/heterogeneity.yml (conda or mamba).")
split found: False
global ranking (top 3): SC3, RtsneKmeans, Seurat
The Bradley-Terry tree found no dataset feature that splits the method ranking at alpha 0.05 over 12 datasets, so the ranking is reported as one Bradley-Terry model over all of them, led by SC3. With this many datasets the split test has few observations to work with, the same small-sample limit the critical-difference diagram shows; a benchmark with more datasets is where a split can appear.

Rank sensitivity

rank_sensitivity splits the rank variance of each method into the dataset, the weighting, the aggregation and their interactions. Missing cells take the lowest score (missing="worst").

from beam.mcda import rank_sensitivity

rs = rank_sensitivity(
    duo.tensor(tuple(analysis_metrics)),
    ctx.polarity,
    normalization=list(ctx.normalization),
    bounds=list(ctx.bounds),
    baselines=list(ctx.baselines),
    targets=list(ctx.targets),
    missing="worst",
    tool_names=duo.method_names,
    dataset_names=duo.dataset_names,
)
print(f"{rs.n_combinations} combinations")
print(f"  dataset:      {rs.dataset_share:.3f} of the rank variance")
print(f"  weighting:    {rs.weighting_share:.3f}")
print(f"  aggregation:  {rs.aggregation_share:.3f}")
print(f"  interactions: {rs.interaction_share:.3f}")
print(f"  most influential factor: {rs.most_influential_factor}")
240 combinations
  dataset:      0.734 of the rank variance
  weighting:    0.035
  aggregation:  0.033
  interactions: 0.198
  most influential factor: dataset
plot.rank_sensitivity(rs)

The dataset has 0.73 of the rank variance.

The same split per method, with each method’s rank range:

plot.rank_sensitivity_by_tool(rs)

Summary

With equal weights and SAW on ARI, runtime and Shannon entropy difference pooled across the twelve datasets, Seurat ranks first. Across the 240 combinations of weighting, aggregation and dataset it is first in half, and the dataset has 0.73 of the rank variance.