Under weighted sum aggregation with equal weights over the metrics ari, runtime and shannon_entropy_diff, Seurat ranks first of 12 tools. Across 1000 weightings drawn at random from the metric simplex, Seurat ranked first in 100 percent of draws. Its rank held in 3 of 3 leave-one-metric-out runs. Its rank held in 12 of 12 leave-one-dataset-out runs. The top rank is stable: the smallest single-metric weight change that would overturn it is about 53.15.
| layout | long |
|---|---|
| tools | 12 |
| metrics | ari, runtime, shannon_entropy_diff |
| datasets | 12: Koh, KohTCC, Kumar, KumarTCC, SimKumar4easy, SimKumar4hard, SimKumar8hard, Trapnell, TrapnellTCC, Zhengmix4eq, Zhengmix4uneq, Zhengmix8eq |
| weighting | equal |
| aggregation | saw |
| metric | strategy |
|---|---|
| ari | baseline_relative |
| runtime | log_min_max |
| shannon_entropy_diff | min_max |
No normalization guard warnings.
| rank | tool | composite |
|---|---|---|
| 1 | Seurat | 0.9470 |
| 2 | PCAKmeans | 0.8562 |
| 3 | PCAHC | 0.8555 |
| 4 | CIDR | 0.8303 |
| 5 | monocle | 0.8232 |
| 6 | RtsneKmeans | 0.8079 |
| 7 | TSCAN | 0.7716 |
| 8 | FlowSOM | 0.7229 |
| 9 | SC3svm | 0.6560 |
| 10 | SC3 | 0.6128 |
| 11 | pcaReduce | 0.5798 |
| 12 | RaceID2 | 0.5671 |
The glyph table shows the per-metric normalized scores as circle size and the overall composite as a bar, alongside panels for how far each tool's rank moves under the robustness checks: the span across leave-one-dataset-out runs, the SMAA rank acceptability, and the span across the five aggregations. A tool with a tight rank span is a stable recommendation; a wide span means the order at that position depends on a choice the analyst made.
The ranking above commits to one weighting (equal) and one aggregation (saw), the way a reader would apply the tool in practice. The sections here re-run it under the alternatives, so you can read how much the recommendation depends on those analyst choices rather than on the data.
| tool | ranked first (percent of draws) |
|---|---|
| Seurat | 99.9 |
| PCAKmeans | 0.0 |
| PCAHC | 0.0 |
| CIDR | 0.0 |
| monocle | 0.0 |
| RtsneKmeans | 0.0 |
| TSCAN | 0.0 |
| FlowSOM | 0.0 |
| SC3svm | 0.0 |
| SC3 | 0.1 |
| pcaReduce | 0.0 |
| RaceID2 | 0.0 |
The top-ranked tool kept its rank in 100 percent of the
3 leave-one-metric-out runs. Removing runtime
caused the largest rank change, a shift of 8.
The top-ranked tool kept its rank in 100 percent of the
12 leave-one-dataset-out runs. Omitting dataset
KohTCC caused the largest rank change, a shift of
1.
Re-ranking under the 5 aggregations gives orderings that
agree at a mean Kendall tau-b of 0.62.
Every aggregation ranks
Seurat first, so the top recommendation does not
depend on the aggregation choice.
Re-ranking under 5 normalizations
(recommended, min_max, log_min_max, rank, zscore) gives orderings that agree at a mean Kendall tau-b of
0.64. The recommended column is the per-metric
default each card declares (ari: baseline_relative, runtime: log_min_max, shannon_entropy_diff: min_max); the others apply one
strategy to every metric.
Every normalization ranks
Seurat first, so the top recommendation does not
depend on the normalization choice.
Across the 240 combinations of weighting, aggregation and dataset, the rank variance splits into the dataset (71 percent), the weighting choice (4 percent), the aggregation choice (4 percent), and their interactions (21 percent). The largest is the dataset.
The top tool overall (Seurat) ranks first in
49 percent of combinations, with a rank span of
10. Its mean rank per dataset:
| dataset | mean rank of Seurat |
|---|---|
| Koh | 1.1 |
| KohTCC | 1.4 |
| Kumar | 4.7 |
| KumarTCC | 4.9 |
| SimKumar4easy | 2.0 |
| SimKumar4hard | 2.0 |
| SimKumar8hard | 1.1 |
| Trapnell | 4.4 |
| TrapnellTCC | 1.8 |
| Zhengmix4eq | 1.0 |
| Zhengmix4uneq | 1.0 |
| Zhengmix8eq | 1.0 |
Each of the 240 combinations of weighting, aggregation and
dataset is one specification. The method that ranks first most often
(Seurat) ranks first in
49 percent of them. The single most common ordering of all tools
holds in 3 percent of specifications, and
5 distinct tool(s) rank first in at least one specification.
The curve below plots that method's rank across the specifications, sorted from its best rank to
its worst; the panel beneath marks which choices each specification used.
The top rank is stable: the smallest single-metric weight change that would overturn it is
about 53.145 on ari.
How many tools score above a random method on each metric that declares a chance baseline.
| metric | chance score | tools above chance |
|---|---|---|
ari | 0 | 12 of 12 (100 percent) |
Every tool scores above chance on at least one metric.
The top two tools (Seurat and PCAKmeans) are separated above the noise floor on at least one metric.
Tool pairs the metric set cannot tell apart above the noise floor:
Friedman test p-value 1.093e-11 over 12 datasets. The Nemenyi critical difference is 4.810 average-rank units; tools closer than this are not separable at the chosen level.
Groups not significantly different:
Effect size: the top-ranked method (Seurat) scores
higher than the next (PCAHC) on
9 of 12 datasets
(probability of superiority 75 percent).
Consistency: the pairwise majorities are transitive, so one order is consistent with them .
Posterior: under the Bayesian sign test (ROPE 0), the probability that
Seurat is practically better than PCAHC is 0.97,
and the probability that the two are practically equivalent is 0.00.
| package | version |
|---|---|
| python | 3.12.13 |
| beam | 0.2.0 |
| numpy | 2.5.1 |
| scipy | 1.18.0 |
| pymcdm | 1.4.0 |
| pyyaml | 6.0.3 |
| jsonschema | 4.26.0 |