Under weighted sum aggregation with equal weights over the metrics cell_type_annotation_agreement and cell_type_annotation_full_match_rate, GPT-4 ranks first of 5 tools. Across 1000 weightings drawn at random from the metric simplex, GPT-4 ranked first in 100 percent of draws. Its rank held in 2 of 2 leave-one-metric-out runs. Its rank held in 36 of 36 leave-one-dataset-out runs. No single-metric weight change within the searched range overturns the top rank.
Consistency: the pairwise majorities are not transitive
(0 circular triads),
so no single order represents them, though
GPT-4 is the method preferred to every other one
individually. See the critical-difference section below.
| layout | long |
|---|---|
| tools | 5 |
| metrics | cell_type_annotation_agreement, cell_type_annotation_full_match_rate |
| datasets | 36: HCL, MCA, HCA_Breast, HCA_Esophagus, HCA_Heart, HCA_Lung, HCA_Prostate, HCA_Skeletal_muscle, HCA_Skin, BCL_B_cell_lymphoma, coloncancer_colon_cancer, lungcancer_lung_cancer_and_brain_metastasis, tabulasapiens_Bladder, tabulasapiens_Blood, tabulasapiens_Bone_Marrow, tabulasapiens_Eye, tabulasapiens_Fat, tabulasapiens_Heart, tabulasapiens_Kidney, tabulasapiens_Large_Intestine, tabulasapiens_Liver, tabulasapiens_Lung, tabulasapiens_Lymph_Node, tabulasapiens_Mammary, tabulasapiens_Muscle, tabulasapiens_Pancreas, tabulasapiens_Prostate, tabulasapiens_Salivary_Gland, tabulasapiens_Skin, tabulasapiens_Small_Intestine, tabulasapiens_Spleen, tabulasapiens_Thymus, tabulasapiens_Tongue, tabulasapiens_Trachea, tabulasapiens_Uterus, tabulasapiens_Vasculature |
| weighting | equal |
| aggregation | saw |
| metric | strategy |
|---|---|
| cell_type_annotation_agreement | min_max |
| cell_type_annotation_full_match_rate | min_max |
No normalization guard warnings.
| rank | tool | composite |
|---|---|---|
| 1 | GPT-4 | 0.5063 |
| 2 | SingleR | 0.3337 |
| 3 | GPT-3.5 | 0.3228 |
| 4 | ScType | 0.3191 |
| 5 | CellMarker2.0 | 0.1829 |
Circle size is the normalized score per metric and the bar is the composite. The panels on the right are each tool's rank span across the leave-one-dataset-out runs, its SMAA rank acceptability, and its rank span across the five aggregations. A narrow span means a stable position; a wide one means a position dependent on an analyst choice.
The ranking above uses one weighting (equal) and one aggregation (saw). The sections below are the same ranking under the alternative choices.
| tool | ranked first (percent of draws) |
|---|---|
| GPT-4 | 100.0 |
| SingleR | 0.0 |
| GPT-3.5 | 0.0 |
| ScType | 0.0 |
| CellMarker2.0 | 0.0 |
The top-ranked tool kept its rank in 100 percent of the
2 leave-one-metric-out runs. Removing cell_type_annotation_full_match_rate
caused the largest rank change, a shift of 1.
The top-ranked tool kept its rank in 100 percent of the
36 leave-one-dataset-out runs. Omitting dataset
HCA_Skeletal_muscle caused the largest rank change, a shift of
1.
Re-ranking under the 5 aggregations gives orderings that
agree at a mean Kendall tau-b of 0.98.
Every aggregation ranks
GPT-4 first, so the top recommendation does not
depend on the aggregation choice.
Re-ranking under 5 normalizations
(recommended, min_max, log_min_max, rank, zscore) gives orderings that agree at a mean Kendall tau-b of
0.98. The recommended column is the per-metric
default declared on each card (cell_type_annotation_agreement: min_max, cell_type_annotation_full_match_rate: min_max); the others apply one
strategy to every metric.
Every normalization ranks
GPT-4 first, so the top recommendation does not
depend on the normalization choice.
Across the 720 combinations of weighting, aggregation and dataset, the rank variance splits into the dataset (96 percent), the weighting choice (0 percent), the aggregation choice (0 percent), and their interactions (4 percent). The largest is the dataset.
The top tool overall (GPT-4) ranks first in
94 percent of combinations, with a rank span of
2. Its mean rank per dataset:
| dataset | mean rank of GPT-4 |
|---|---|
| HCL | 1.0 |
| MCA | 1.0 |
| HCA_Breast | 1.0 |
| HCA_Esophagus | 1.0 |
| HCA_Heart | 1.0 |
| HCA_Lung | 3.0 |
| HCA_Prostate | 2.0 |
| HCA_Skeletal_muscle | 1.0 |
| HCA_Skin | 1.0 |
| BCL_B_cell_lymphoma | 1.0 |
| coloncancer_colon_cancer | 1.0 |
| lungcancer_lung_cancer_and_brain_metastasis | 1.0 |
| tabulasapiens_Bladder | 1.0 |
| tabulasapiens_Blood | 1.0 |
| tabulasapiens_Bone_Marrow | 1.0 |
| tabulasapiens_Eye | 1.0 |
| tabulasapiens_Fat | 1.0 |
| tabulasapiens_Heart | 1.0 |
| tabulasapiens_Kidney | 1.0 |
| tabulasapiens_Large_Intestine | 1.0 |
| tabulasapiens_Liver | 1.0 |
| tabulasapiens_Lung | 1.0 |
| tabulasapiens_Lymph_Node | 1.1 |
| tabulasapiens_Mammary | 1.0 |
| tabulasapiens_Muscle | 1.0 |
| tabulasapiens_Pancreas | 1.0 |
| tabulasapiens_Prostate | 1.0 |
| tabulasapiens_Salivary_Gland | 1.0 |
| tabulasapiens_Skin | 1.0 |
| tabulasapiens_Small_Intestine | 1.0 |
| tabulasapiens_Spleen | 1.0 |
| tabulasapiens_Thymus | 1.0 |
| tabulasapiens_Tongue | 1.0 |
| tabulasapiens_Trachea | 1.0 |
| tabulasapiens_Uterus | 1.0 |
| tabulasapiens_Vasculature | 1.0 |
Each of the 720 combinations of weighting, aggregation and
dataset is one specification. The most frequent top method
(GPT-4) ranks first in
94 percent of them. The single most common ordering of all tools
holds in 17 percent of specifications, and
3 distinct tool(s) rank first in at least one specification.
The curve below plots that method's rank across the specifications, sorted from its best rank to
its worst; the panel beneath marks the choices of each specification.
No single-metric weight change within the searched range overturns the top rank.
Number of tools above a random method on each metric with a declared chance baseline.
| metric | chance score | tools above chance |
|---|---|---|
cell_type_annotation_full_match_rate | 0 | 5 of 5 (100 percent) |
Every tool scores above chance on at least one metric.
Friedman test p-value 1.03e-18 over 36 datasets. The Nemenyi critical difference is 1.017 average-rank units; tools closer than this are not separable at the chosen level.
Groups not significantly different:
Effect size: the top-ranked method (GPT-4) scores
higher than the next (ScType) on
34 of 36 datasets
(probability of superiority 94 percent).
Consistency: the pairwise majorities contain 0 circular triads
out of 10, so no single order agrees with all of them. The reported ranking then depends on the aggregation rule
(though GPT-4 is still preferred to every other method individually).
Posterior: under the Bayesian sign test (ROPE 0), the probability that
GPT-4 is practically better than ScType is 1.00,
and the probability that the two are practically equivalent is 0.00.
| package | version |
|---|---|
| python | 3.12.14 |
| beam | 0.3.1 |
| numpy | 2.5.3 |
| scipy | 1.18.1 |
| pymcdm | 1.4.0 |
| pyyaml | 6.0.3 |
| jsonschema | 4.26.0 |