Under TOPSIS aggregation with equal weights over the metrics ari, asw_batch, asw_label, cell_cycle_conservation, clisi, graph_connectivity, ilisi, isolated_label_asw, isolated_label_f1, kbet, nmi and pcr, combat ranks first of 14 tools. Across 1000 weightings drawn at random from the metric simplex, combat ranked first in 46 percent of draws. Its rank held in 11 of 12 leave-one-metric-out runs. Its rank held in 6 of 6 leave-one-dataset-out runs. The top rank is fragile: a weight change of about 0.04 on pcr is enough to overturn it.
| layout | long |
|---|---|
| tools | 14 |
| metrics | ari, asw_batch, asw_label, cell_cycle_conservation, clisi, graph_connectivity, ilisi, isolated_label_asw, isolated_label_f1, kbet, nmi, pcr |
| datasets | 6: cellxgene_census/dkd, cellxgene_census/gtex_v9, cellxgene_census/hypomap, cellxgene_census/immune_cell_atlas, cellxgene_census/mouse_pancreas_atlas, cellxgene_census/tabula_sapiens |
| weighting | equal |
| aggregation | topsis |
| metric | strategy |
|---|---|
| ari | baseline_relative |
| asw_batch | min_max |
| asw_label | min_max |
| cell_cycle_conservation | min_max |
| clisi | min_max |
| graph_connectivity | min_max |
| ilisi | min_max |
| isolated_label_asw | min_max |
| isolated_label_f1 | min_max |
| kbet | min_max |
| nmi | min_max |
| pcr | min_max |
No normalization guard warnings.
| rank | tool | composite |
|---|---|---|
| 1 | combat | 0.8248 |
| 2 | harmonypy | 0.7563 |
| 3 | harmony | 0.7524 |
| 4 | scvi | 0.7397 |
| 5 | scalex | 0.7393 |
| 6 | pyliger | 0.7290 |
| 7 | liger | 0.7273 |
| 8 | scanvi | 0.7231 |
| 9 | batchelor_fastmnn | 0.6759 |
| 10 | scimilarity | 0.6273 |
| 11 | scgpt_zeroshot | 0.6096 |
| 12 | uce | 0.5859 |
| 13 | scanorama | 0.3061 |
| 14 | geneformer | 0.0882 |
Circle size is the normalized score per metric and the bar is the composite. The panels on the right are each tool's rank span across the leave-one-dataset-out runs, its SMAA rank acceptability, and its rank span across the five aggregations. A narrow span means a stable position; a wide one means a position dependent on an analyst choice.
The ranking above uses one weighting (equal) and one aggregation (topsis). The sections below are the same ranking under the alternative choices.
| tool | ranked first (percent of draws) |
|---|---|
| combat | 45.9 |
| harmonypy | 2.8 |
| harmony | 5.7 |
| scvi | 2.2 |
| scalex | 0.9 |
| pyliger | 3.3 |
| liger | 0.8 |
| scanvi | 31.0 |
| batchelor_fastmnn | 6.0 |
| scimilarity | 1.1 |
| scgpt_zeroshot | 0.1 |
| uce | 0.2 |
| scanorama | 0.0 |
| geneformer | 0.0 |
The top-ranked tool kept its rank in 92 percent of the
12 leave-one-metric-out runs. Removing pcr
caused the largest rank change, a shift of 7.
The top-ranked tool kept its rank in 100 percent of the
6 leave-one-dataset-out runs. Omitting dataset
cellxgene_census/gtex_v9 caused the largest rank change, a shift of
6.
Re-ranking under the 4 aggregations gives orderings that agree at a mean Kendall tau-b of 0.55. They do not all rank the same tool first, so the top recommendation depends on which aggregation is used.
Re-ranking under 4 normalizations
(recommended, min_max, rank, zscore) gives orderings that agree at a mean Kendall tau-b of
0.50. The recommended column is the per-metric
default declared on each card (ari: baseline_relative, asw_batch: min_max, asw_label: min_max, cell_cycle_conservation: min_max, clisi: min_max, graph_connectivity: min_max, ilisi: min_max, isolated_label_asw: min_max, isolated_label_f1: min_max, kbet: min_max, nmi: min_max, pcr: min_max); the others apply one
strategy to every metric.
They do not all rank the same tool first, so the top
recommendation depends on which normalization is used.
Across the 96 combinations of weighting, aggregation and dataset, the rank variance splits into the dataset (51 percent), the weighting choice (4 percent), the aggregation choice (16 percent), and their interactions (30 percent). The largest is the dataset.
The top tool overall (combat) ranks first in
36 percent of combinations, with a rank span of
8. Its mean rank per dataset:
| dataset | mean rank of combat |
|---|---|
| cellxgene_census/dkd | 5.8 |
| cellxgene_census/gtex_v9 | 2.0 |
| cellxgene_census/hypomap | 2.9 |
| cellxgene_census/immune_cell_atlas | 5.8 |
| cellxgene_census/mouse_pancreas_atlas | 2.9 |
| cellxgene_census/tabula_sapiens | 3.4 |
Each of the 96 combinations of weighting, aggregation and
dataset is one specification. The most frequent top method
(combat) ranks first in
35 percent of them. The single most common ordering of all tools
holds in 5 percent of specifications, and
9 distinct tool(s) rank first in at least one specification.
The curve below plots that method's rank across the specifications, sorted from its best rank to
its worst; the panel beneath marks the choices of each specification.
The top rank is fragile: a weight change of about 0.037
on pcr overturns it.
Number of tools above a random method on each metric with a declared chance baseline.
| metric | chance score | tools above chance |
|---|---|---|
ari | 0 | 14 of 14 (100 percent) |
Every tool scores above chance on at least one metric.
The top two tools (combat and harmonypy) are separated above the noise floor on at least one metric.
Tool pairs within the noise floor on every metric:
| package | version |
|---|---|
| python | 3.12.14 |
| beam | 0.3.1 |
| numpy | 2.5.3 |
| scipy | 1.18.1 |
| pymcdm | 1.4.0 |
| pyyaml | 6.0.3 |
| jsonschema | 4.26.0 |