Hou and Ji 2024 LLM cell type annotation

Generated by beam 0.3.1 at 2026-10-07T14:16:35.496139+00:00.

Under weighted sum aggregation with equal weights over the metrics cell_type_annotation_agreement and cell_type_annotation_full_match_rate, GPT-4 ranks first of 5 tools. Across 1000 weightings drawn at random from the metric simplex, GPT-4 ranked first in 100 percent of draws. Its rank held in 2 of 2 leave-one-metric-out runs. Its rank held in 36 of 36 leave-one-dataset-out runs. No single-metric weight change within the searched range overturns the top rank.

Consistency: the pairwise majorities are not transitive (0 circular triads), so no single order represents them, though GPT-4 is the method preferred to every other one individually. See the critical-difference section below.

Input

layoutlong
tools5
metricscell_type_annotation_agreement, cell_type_annotation_full_match_rate
datasets36: HCL, MCA, HCA_Breast, HCA_Esophagus, HCA_Heart, HCA_Lung, HCA_Prostate, HCA_Skeletal_muscle, HCA_Skin, BCL_B_cell_lymphoma, coloncancer_colon_cancer, lungcancer_lung_cancer_and_brain_metastasis, tabulasapiens_Bladder, tabulasapiens_Blood, tabulasapiens_Bone_Marrow, tabulasapiens_Eye, tabulasapiens_Fat, tabulasapiens_Heart, tabulasapiens_Kidney, tabulasapiens_Large_Intestine, tabulasapiens_Liver, tabulasapiens_Lung, tabulasapiens_Lymph_Node, tabulasapiens_Mammary, tabulasapiens_Muscle, tabulasapiens_Pancreas, tabulasapiens_Prostate, tabulasapiens_Salivary_Gland, tabulasapiens_Skin, tabulasapiens_Small_Intestine, tabulasapiens_Spleen, tabulasapiens_Thymus, tabulasapiens_Tongue, tabulasapiens_Trachea, tabulasapiens_Uterus, tabulasapiens_Vasculature
weightingequal
aggregationsaw

Normalization

metricstrategy
cell_type_annotation_agreementmin_max
cell_type_annotation_full_match_ratemin_max

No normalization guard warnings.

Ranking

composite score per tool
ranktoolcomposite
1GPT-40.5063
2SingleR0.3337
3GPT-3.50.3228
4ScType0.3191
5CellMarker2.00.1829
normalized scores heatmap

Rank robustness

Circle size is the normalized score per metric and the bar is the composite. The panels on the right are each tool's rank span across the leave-one-dataset-out runs, its SMAA rank acceptability, and its rank span across the five aggregations. A narrow span means a stable position; a wide one means a position dependent on an analyst choice.

funky heatmap with rank-robustness panels

Sensitivity

The ranking above uses one weighting (equal) and one aggregation (saw). The sections below are the same ranking under the alternative choices.

SMAA weight sampling

SMAA confidence per tool
toolranked first (percent of draws)
GPT-4100.0
SingleR0.0
GPT-3.50.0
ScType0.0
CellMarker2.00.0

Leave one metric out

The top-ranked tool kept its rank in 100 percent of the 2 leave-one-metric-out runs. Removing cell_type_annotation_full_match_rate caused the largest rank change, a shift of 1.

Leave one dataset out

rank stability per tool across dataset omissions

The top-ranked tool kept its rank in 100 percent of the 36 leave-one-dataset-out runs. Omitting dataset HCA_Skeletal_muscle caused the largest rank change, a shift of 1.

Aggregation agreement

Re-ranking under the 5 aggregations gives orderings that agree at a mean Kendall tau-b of 0.98. Every aggregation ranks GPT-4 first, so the top recommendation does not depend on the aggregation choice.

Normalization agreement

Re-ranking under 5 normalizations (recommended, min_max, log_min_max, rank, zscore) gives orderings that agree at a mean Kendall tau-b of 0.98. The recommended column is the per-metric default declared on each card (cell_type_annotation_agreement: min_max, cell_type_annotation_full_match_rate: min_max); the others apply one strategy to every metric. Every normalization ranks GPT-4 first, so the top recommendation does not depend on the normalization choice.

normalization agreement heatmap

Rank variance by factor

Across the 720 combinations of weighting, aggregation and dataset, the rank variance splits into the dataset (96 percent), the weighting choice (0 percent), the aggregation choice (0 percent), and their interactions (4 percent). The largest is the dataset.

The top tool overall (GPT-4) ranks first in 94 percent of combinations, with a rank span of 2. Its mean rank per dataset:

datasetmean rank of GPT-4
HCL1.0
MCA1.0
HCA_Breast1.0
HCA_Esophagus1.0
HCA_Heart1.0
HCA_Lung3.0
HCA_Prostate2.0
HCA_Skeletal_muscle1.0
HCA_Skin1.0
BCL_B_cell_lymphoma1.0
coloncancer_colon_cancer1.0
lungcancer_lung_cancer_and_brain_metastasis1.0
tabulasapiens_Bladder1.0
tabulasapiens_Blood1.0
tabulasapiens_Bone_Marrow1.0
tabulasapiens_Eye1.0
tabulasapiens_Fat1.0
tabulasapiens_Heart1.0
tabulasapiens_Kidney1.0
tabulasapiens_Large_Intestine1.0
tabulasapiens_Liver1.0
tabulasapiens_Lung1.0
tabulasapiens_Lymph_Node1.1
tabulasapiens_Mammary1.0
tabulasapiens_Muscle1.0
tabulasapiens_Pancreas1.0
tabulasapiens_Prostate1.0
tabulasapiens_Salivary_Gland1.0
tabulasapiens_Skin1.0
tabulasapiens_Small_Intestine1.0
tabulasapiens_Spleen1.0
tabulasapiens_Thymus1.0
tabulasapiens_Tongue1.0
tabulasapiens_Trachea1.0
tabulasapiens_Uterus1.0
tabulasapiens_Vasculature1.0

Specification curve

Each of the 720 combinations of weighting, aggregation and dataset is one specification. The most frequent top method (GPT-4) ranks first in 94 percent of them. The single most common ordering of all tools holds in 17 percent of specifications, and 3 distinct tool(s) rank first in at least one specification. The curve below plots that method's rank across the specifications, sorted from its best rank to its worst; the panel beneath marks the choices of each specification.

specification curve

Smallest weight perturbation

No single-metric weight change within the searched range overturns the top rank.

Reference levels

Chance baseline

Number of tools above a random method on each metric with a declared chance baseline.

metricchance scoretools above chance
cell_type_annotation_full_match_rate0 5 of 5 (100 percent)

Every tool scores above chance on at least one metric.

Critical difference across datasets

Friedman test p-value 1.03e-18 over 36 datasets. The Nemenyi critical difference is 1.017 average-rank units; tools closer than this are not separable at the chosen level.

average ranks with critical difference

Groups not significantly different:

Effect size: the top-ranked method (GPT-4) scores higher than the next (ScType) on 34 of 36 datasets (probability of superiority 94 percent).

Consistency: the pairwise majorities contain 0 circular triads out of 10, so no single order agrees with all of them. The reported ranking then depends on the aggregation rule (though GPT-4 is still preferred to every other method individually).

pairwise dominance matrix ordered by methods outperformed

Posterior: under the Bayesian sign test (ROPE 0), the probability that GPT-4 is practically better than ScType is 1.00, and the probability that the two are practically equivalent is 0.00.

posterior probability that the row method is practically better than the column method

Reproducibility

Input sha256: 442d313924bcc21121eff6e611d1af831d462f14c93ffd87c5755c3f93e8f64d

packageversion
python3.12.14
beam0.3.1
numpy2.5.3
scipy1.18.1
pymcdm1.4.0
pyyaml6.0.3
jsonschema4.26.0

The run manifest written next to this report records the metric card versions and content hashes.