LLM cell type annotation: ranking GPT-4 against classical tools with beam

Author

Izaskun Mallona

Published

October 7, 2026

Data

Hou and Ji (2024) scored GPT-4 and GPT-3.5 against three reference-based annotators (CellMarker2.0, SingleR, ScType). Each method labels the cell types of a dataset, and the score is the agreement with a manual expert annotation.

The single-cell data and the model outputs are not bundled. src/beam/data/gptcelltype2024_agreement.csv and gptcelltype2024_features.csv are reduced from the evaluation table of the repository Winnie09/GPTCelltype_Paper at a pinned commit (CC-BY-4.0, see src/beam/data/README.md). Reference: Hou W, Ji Z. Assessing GPT-4 for cell type annotation in single-cell RNA-seq analysis. Nature Methods 2024, DOI 10.1038/s41592-024-02235-4.

To reproduce, run:

SHA=5944a41511aacd368b45448e256d9625849704df
curl -sf -o all.csv \
  "https://raw.githubusercontent.com/Winnie09/GPTCelltype_Paper/$SHA/anno/compiled/all.csv"

all.csv has one row per annotated cell type and a <method>_agreement column per method (1 full match, 0.5 partial match, 0 mismatch). The reduction script is src/beam/data/reduce_gptcelltype.py:

python src/beam/data/reduce_gptcelltype.py        # fetches the pinned commit
python src/beam/data/reduce_gptcelltype.py all.csv  # or reduces a local copy

Tensor

load_gptcelltype reads the bundled tables. Each (source, tissue) pair is a dataset and each annotated cell type an observation. The two metrics are the mean agreement score and the full match rate, both higher is better.

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

import beam
from beam.datasets import load_gptcelltype
from beam.cards import properties_for

g = load_gptcelltype()
print("scores shape:", g.scores.shape)
print("methods:", ", ".join(g.method_names))
print("datasets:", len(g.dataset_names))
print("metrics:", ", ".join(g.metric_ids))
scores shape: (6, 54, 2)
methods: GPT-4, GPT-4-mar2023, GPT-3.5, CellMarker2.0, SingleR, ScType
datasets: 54
metrics: cell_type_annotation_agreement, cell_type_annotation_full_match_rate

Coverage

SingleR and ScType were not run on the Azimuth and literature datasets, and the March 2023 GPT-4 endpoint was run on a subset of the datasets. Missing cells are numpy.nan and are not imputed.

for i, m in enumerate(g.method_names):
    covered = int(np.sum(~np.isnan(g.scores[i, :, 0])))
    mean_agreement = np.nanmean(g.scores[i, :, 0])
    print(f"{m:14s} covers {covered:2d}/{len(g.dataset_names)} datasets   mean agreement {mean_agreement:.3f}")
GPT-4          covers 54/54 datasets   mean agreement 0.621
GPT-4-mar2023  covers 27/54 datasets   mean agreement 0.625
GPT-3.5        covers 54/54 datasets   mean agreement 0.452
CellMarker2.0  covers 54/54 datasets   mean agreement 0.268
SingleR        covers 36/54 datasets   mean agreement 0.450
ScType         covers 36/54 datasets   mean agreement 0.450
classical = {"CellMarker2.0", "SingleR", "ScType"}
mean_agreement = np.nanmean(g.scores[:, :, 0], axis=1)
o = np.argsort(-mean_agreement)
colors = ["tab:gray" if g.method_names[i] in classical else "tab:blue" for i in o]
fig, ax = plt.subplots(figsize=(7.0, 3.2))
ax.bar([g.method_names[i] for i in o], mean_agreement[o], color=colors)
ax.set_ylabel("mean agreement over datasets")
ax.set_xlabel("method (grey = classical, blue = LLM)")
ax.set_title("Mean annotation agreement per method")
ax.tick_params(axis="x", rotation=45)
fig.tight_layout()
plt.show()

The main comparison uses the five methods of Hou and Ji, without the March 2023 GPT-4 endpoint, on the datasets where all five were run.

head = ["GPT-4", "GPT-3.5", "CellMarker2.0", "SingleR", "ScType"]
idx = [g.method_names.index(m) for m in head]
metrics = list(g.metric_ids)

sub = g.scores[idx]                         # 5 methods x 54 datasets x 2 metrics
complete = ~np.isnan(sub[:, :, 0]).any(axis=0)
block = sub[:, complete, :]                 # 5 x 36 x 2, no missing cells
block_datasets = [d for d, keep in zip(g.dataset_names, complete) if keep]
print(f"complete block: {block.shape[0]} methods, {block.shape[1]} datasets, {block.shape[2]} metrics")
complete block: 5 methods, 36 datasets, 2 metrics

Metric cards

properties_for reads the two annotation cards. Both are higher is better, bounded in [0, 1], with min-max normalization and the arithmetic mean across datasets.

props = properties_for(metrics)
for p in props:
    print(
        f"{p.id:36s} polarity={p.polarity:16s} scale={p.scale_type:8s} "
        f"range=({p.range_lower}, {p.range_upper}) norm={p.recommended_normalization}"
    )
cell_type_annotation_agreement       polarity=higher_is_better scale=interval range=(0, 1) norm=min_max
cell_type_annotation_full_match_rate polarity=higher_is_better scale=ratio    range=(0, 1) norm=min_max

Ranking on the complete cases

The two metrics are pooled across the 36 datasets and the five methods ranked with equal weights and simple additive weighting (SAW).

block_scores = beam.Scores(
    values=block,
    tool_names=tuple(head),
    metric_ids=tuple(metrics),
    dataset_names=tuple(block_datasets),
    layout="long",
)
run = beam.rank(block_scores, weights="equal", method="saw", sensitivity=True)

print(f"weighting={run.result.weighting}  method={run.result.method}")
order = np.argsort(run.result.ranks)
print(f"\n{'method':14s} composite  rank")
for i in order:
    print(f"{head[i]:14s} {run.result.composite[i]:.3f}      {run.result.ranks[i]}")
weighting=equal  method=saw

method         composite  rank
GPT-4          0.506      1
SingleR        0.334      2
GPT-3.5        0.323      3
ScType         0.319      4
CellMarker2.0  0.183      5

GPT-4 ranks first.

Card consistency

card_data_consistency checks the pooled scores against the cards before normalization.

audit = run.card_consistency
print("pooled scores consistent with the cards:", audit.ok)
for finding in audit.findings:
    print(" ", finding.severity, finding.message)
pooled scores consistent with the cards: True

Weightings and aggregation methods

Four weightings (equal, entropy, std, critic) by five aggregation methods give twenty runs.

from beam.mcda import run_from_registry

pooled = np.nanmean(block, axis=1)        # 5 methods x 2 metrics, complete
weightings = ["equal", "entropy", "std", "critic"]
agg_methods = ["saw", "topsis", "vikor", "promethee_ii", "comet"]

combos = [(w, m) for w in weightings for m in agg_methods]
ranks_grid = np.zeros((len(combos), len(head)), dtype=int)
row_labels = []
for i, (w, m) in enumerate(combos):
    r = run_from_registry(pooled, metrics, weights=w, method=m)
    ranks_grid[i] = r.ranks
    row_labels.append(f"{w} / {m}")
from beam import plot

plot.rank_heatmap(
    ranks_grid,
    row_names=row_labels,
    col_names=head,
    row_label="weighting / aggregation method",
    col_label="annotation method",
    title="Rank by configuration (both metrics higher is better)",
)

top_per_combo = {head[int(np.argmin(row))] for row in ranks_grid}
print("methods top-ranked in at least one of the 20 configurations:", ", ".join(sorted(top_per_combo)))
methods top-ranked in at least one of the 20 configurations: GPT-4

aggregation_agreement gives the Kendall tau-b agreement of the rankings of the five aggregations at equal weights.

from beam.mcda import aggregation_agreement

agg = aggregation_agreement(pooled, [p.polarity for p in props], tool_names=head)
print(f"mean pairwise Kendall tau-b across aggregations: {agg.mean_pairwise_tau:.2f}")
print(f"top method unanimous across aggregations: {agg.top_is_unanimous} ({head[agg.top_tool]})")
mean pairwise Kendall tau-b across aggregations: 0.98
top method unanimous across aggregations: True (GPT-4)

Rank stability

SMAA samples weight vectors and records how often each method has each rank. The weight perturbation is the smallest single-weight change displacing the top method.

smaa = run.smaa
top = int(np.argmin(run.result.ranks))
print(f"SMAA: {head[top]} is rank 1 in {smaa.rank_acceptability_index[top, 0] * 100:.0f} percent of weight draws")

wp = run.perturbation
print(f"top rank fragile: {wp.top_rank_is_fragile}")
if wp.top_rank_perturbation is not None:
    print(f"smallest single-weight change at the top pair: {wp.top_rank_perturbation.absolute_delta:.2f}")
else:
    print("no single-weight change within range alters the top rank")
SMAA: GPT-4 is rank 1 in 100 percent of weight draws
top rank fragile: False
no single-weight change within range alters the top rank
plot.smaa(run)

leave_one_metric_out drops each metric in turn.

loo = run.leave_one_out
print(f"top method rank stability under leave-one-metric-out: {loo.rank_stability[top]:.2f}")
print(f"most influential metric: {metrics[loo.most_influential_metric]}")
top method rank stability under leave-one-metric-out: 1.00
most influential metric: cell_type_annotation_full_match_rate

Chance baseline and noise floor

beats_random_baseline uses the chance level on the card, 0 for the full match rate. noise_floor_separation is inactive, as the annotation cards declare no noise floor.

br = run.random_baseline
print("random-baseline check active:", br.active)
if br.active:
    for pm in br.per_metric:
        print(f"  {pm.metric:36s} {pm.n_beating}/{pm.n_observed} methods score above chance ({pm.baseline})")
    print("  methods above chance on no metric:", ", ".join(br.tools_never_beating) or "none")

nf = run.noise_floor
print("noise-floor check active:", nf.active, "(no floor declared on the annotation cards)")
random-baseline check active: True
  cell_type_annotation_full_match_rate 5/5 methods score above chance (0.0)
  methods above chance on no metric: none
noise-floor check active: False (no floor declared on the annotation cards)

Critical difference on agreement

A critical-difference diagram ranks the methods on each of the 36 datasets, with a Friedman test and a Nemenyi post-hoc.

from beam.mcda import critical_difference

cd = critical_difference(block[:, :, 0], "higher_is_better", tool_names=head)
print(f"Friedman p-value: {cd.friedman_pvalue:.2e}")
print(f"critical difference (alpha={cd.alpha}): {cd.critical_difference:.2f}")
print("average rank (lower is better):")
for i in np.argsort(cd.average_ranks):
    print(f"  {head[i]:14s} {cd.average_ranks[i]:.2f}")
print("cliques (methods not separable):")
for clique in cd.cliques:
    print("  " + ", ".join(head[i] for i in clique))
Friedman p-value: 4.21e-20
critical difference (alpha=0.05): 1.02
average rank (lower is better):
  GPT-4          1.17
  ScType         2.94
  SingleR        2.99
  GPT-3.5        3.14
  CellMarker2.0  4.76
cliques (methods not separable):
  ScType, SingleR, GPT-3.5
plot.critical_difference(cd)

pairwise_superiority gives, for each pair, the fraction of the 36 datasets with a higher score for one method.

from beam.mcda import pairwise_superiority

ps = pairwise_superiority(block[:, :, 0], "higher_is_better", method_names=head)
print("standing (Copeland-style, higher means ahead on more pairs):")
for i in np.argsort(-ps.standing):
    print(f"  {head[i]:14s} {ps.standing[i]:.2f}")
standing (Copeland-style, higher means ahead on more pairs):
  GPT-4          0.96
  ScType         0.51
  SingleR        0.50
  GPT-3.5        0.47
  CellMarker2.0  0.06

pairwise_transitivity tests whether one order agrees with the pairwise majorities.

from beam.mcda import pairwise_transitivity

trans = pairwise_transitivity(ps)
print(f"transitive: {trans.is_transitive}; circular triads: {trans.n_circular_triads} of {trans.n_triads}")
transitive: True; circular triads: 0 of 10

The matrix orders the methods by the number of others outperformed. A red cell below the diagonal is a method outperforming one ranked above it.

plot.pairwise_majority(trans)

bayesian_sign_comparison gives, for each pair, the posterior probability that one method is practically better.

from beam.mcda import bayesian_sign_comparison

bayes = bayesian_sign_comparison(ps)
decisive = sum(1 for p in bayes.per_pair if p.decision != "inconclusive")
print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(bayes.per_pair)}")
plot.bayesian_comparison(bayes)
pairs with a decisive posterior at 0.95: 7 of 10

Rank sensitivity

rank_sensitivity ranks the methods on each dataset under every weighting and aggregation and splits the rank variance of each method into the dataset, the weighting, the aggregation and their interactions.

from beam.mcda import rank_sensitivity

ctx = run.context
rs = rank_sensitivity(
    block,
    ctx.polarity,
    normalization=list(ctx.normalization),
    bounds=list(ctx.bounds),
    baselines=list(ctx.baselines),
    targets=list(ctx.targets),
    tool_names=head,
    dataset_names=block_datasets,
)
print(f"{rs.n_combinations} combinations")
print(f"  dataset:      {rs.dataset_share:.3f} of the rank variance")
print(f"  weighting:    {rs.weighting_share:.3f}")
print(f"  aggregation:  {rs.aggregation_share:.3f}")
print(f"  interactions: {rs.interaction_share:.3f}")
print(f"  most influential factor: {rs.most_influential_factor}")
720 combinations
  dataset:      0.962 of the rank variance
  weighting:    0.001
  aggregation:  0.000
  interactions: 0.037
  most influential factor: dataset
plot.rank_sensitivity(rs)

The same split per method, with each method’s rank range:

plot.rank_sensitivity_by_tool(rs)

lodo = run.leave_one_dataset_out
print(f"top method rank stability under leave-one-dataset-out: {lodo.rank_stability[top]:.2f}")
print(f"most influential dataset: {block_datasets[lodo.most_influential_dataset]}")
top method rank stability under leave-one-dataset-out: 1.00
most influential dataset: HCA_Skeletal_muscle

specification_curve lists the rankings of the grid, on the pooled matrix (weighting by aggregation) and with the datasets.

from beam.mcda import specification_curve

curve = specification_curve(rs)
dom = curve.tool_names[curve.most_frequent_top_tool]
print(f"choices plus dataset ({curve.n_specifications} combinations): "
      f"{dom} first in {curve.most_frequent_top_fraction * 100:.0f}%, "
      f"{curve.n_distinct_top_tools} methods reach the top")

pooled = specification_curve(
    rank_sensitivity(
        run.matrix, ctx.polarity,
        normalization=list(ctx.normalization), bounds=list(ctx.bounds),
        baselines=list(ctx.baselines), targets=list(ctx.targets),
        tool_names=head,
    )
)
pdom = pooled.tool_names[pooled.most_frequent_top_tool]
print(f"choices only ({pooled.n_specifications} combinations): "
      f"{pdom} first in {pooled.most_frequent_top_fraction * 100:.0f}%")
choices plus dataset (720 combinations): GPT-4 first in 94%, 3 methods reach the top
choices only (20 combinations): GPT-4 first in 100%

Dataset concordance

dataset_concordance ranks the methods within each dataset and compares every pair of orderings with Kendall tau-b.

conc = run.dataset_concordance
names = conc.dataset_names
print(f"mean agreement across datasets (Kendall tau-b): {conc.mean_pairwise_tau:.2f}")
print("least typical dataset:", names[conc.most_idiosyncratic_dataset])
print("number of mutually consistent dataset groups:", len(conc.concordant_groups))
print("where methods depart most from their own average rank:")
for cell in conc.notable_cells[:5]:
    side = "lower" if cell.deviation > 0 else "higher"
    print(f"  {conc.tool_names[cell.tool]} on {names[cell.dataset]}: "
          f"rank {cell.rank}, {side} than its mean {cell.mean_rank:.1f}")
mean agreement across datasets (Kendall tau-b): 0.53
least typical dataset: HCA_Lung
number of mutually consistent dataset groups: 1
where methods depart most from their own average rank:
  SingleR on tabulasapiens_Heart: rank 5, lower than its mean 3.0
  SingleR on tabulasapiens_Pancreas: rank 5, lower than its mean 3.0
  SingleR on tabulasapiens_Skin: rank 5, lower than its mean 3.0
  ScType on HCA_Heart: rank 5, lower than its mean 3.0
  ScType on HCA_Lung: rank 1, higher than its mean 3.0
plot.dataset_concordance(run)

Rank of each method per dataset minus its mean rank:

plot.dataset_struggle(run)

Blind analysis

A blind analysis fixes the pipeline before the method names are known. beam.blind masks the names and shuffles the rows; beam.unblind restores them. The seal fingerprint is recorded in the manifest.

from beam import blind, unblind

blinded, seal = blind(block_scores, seed=0)
blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)
restored = unblind(blind_run, seal)
named = beam.rank(block_scores, weights="equal", method="saw", seed=0, sensitivity=False)
print("ranking identical after unblinding:",
      dict(zip(named.tool_names, named.result.ranks))
      == dict(zip(restored.tool_names, restored.result.ranks)))
print("top method after unblinding:", restored.top_tool)
print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12])
ranking identical after unblinding: True
top method after unblinding: GPT-4
blinding fingerprint: 50b6fc613b9f

Equal and cell-weighted pooling

By default each dataset has the same weight. The per-cell-type accuracy of Hou and Ji weights a dataset by its number of cell types.

n_cell_types = np.array(g.features.numeric["n_cell_types"])[complete]
agreement = block[:, :, 0]
equal_pool = agreement.mean(axis=1)
weighted_pool = (agreement * n_cell_types).sum(axis=1) / n_cell_types.sum()
print(f"{'method':14s} equal-per-dataset  cell-weighted")
for i in np.argsort(-equal_pool):
    print(f"{head[i]:14s} {equal_pool[i]:.3f}             {weighted_pool[i]:.3f}")
method         equal-per-dataset  cell-weighted
GPT-4          0.603             0.564
SingleR        0.450             0.436
ScType         0.450             0.427
GPT-3.5        0.436             0.399
CellMarker2.0  0.249             0.244

Mixed-effects on agreement

A mixed-effects model fits agreement ~ method + (1 | dataset) in R’s lme4.

from beam.heterogeneity import mixed_effects_from_matrix, r_available

if r_available():
    me = mixed_effects_from_matrix(agreement, head, block_datasets)
    print(f"model: {me.formula}")
    print(f"dataset shift (ICC): {me.icc_dataset:.2f} of the agreement variance")
    order = np.argsort(-me.method_effects)
    print("\nmethod          agreement marginal mean")
    for i in order:
        print(f"{me.method_names[i]:14s}  {me.method_effects[i]:.3f} +/- {me.method_effect_se[i]:.3f}")
else:
    me = None
    print("R with lme4 not available; skipping the mixed-effects fit.")
    print("Provision it with envs/heterogeneity.yml (conda or mamba).")
model: score ~ method + (1 | dataset)
dataset shift (ICC): 0.28 of the agreement variance

method          agreement marginal mean
GPT-4           0.603 +/- 0.021
SingleR         0.450 +/- 0.021
ScType          0.450 +/- 0.021
GPT-3.5         0.436 +/- 0.021
CellMarker2.0   0.249 +/- 0.021

Bradley-Terry tree

A Bradley-Terry tree takes each dataset as a set of pairwise method comparisons and splits the datasets by their features: source, tissue, species, sample type (normal or cancer) and number of cell types. It needs R.

from beam.heterogeneity import bradley_terry_tree, bttree_available

if bttree_available():
    numeric, categorical = g.features.aligned_to(block_datasets)
    bt = bradley_terry_tree(
        agreement,
        head,
        block_datasets,
        numeric_features=numeric,
        categorical_features=categorical,
        polarity="higher_is_better",
        minsize=6,
    )
    print(f"split found: {bt.did_split}")
    print(f"global ranking (top 3): {', '.join(bt.global_ranking()[:3])}")
    print(bt.summary())
else:
    bt = None
    print("R with psychotree not available; skipping the Bradley-Terry tree.")
    print("Provision it with envs/heterogeneity.yml (conda or mamba).")
split found: False
global ranking (top 3): GPT-4, ScType, SingleR
The Bradley-Terry tree found no dataset feature that splits the method ranking at alpha 0.05 over 36 datasets, so the ranking is reported as one Bradley-Terry model over all of them, led by GPT-4. With this many datasets the split test has few observations to work with, the same small-sample limit the critical-difference diagram shows; a benchmark with more datasets is where a split can appear.

plackett_luce fits one full-ranking model over the methods.

from beam.heterogeneity import plackett_luce, plackett_luce_available

if plackett_luce_available():
    pl = plackett_luce(agreement, head, polarity="higher_is_better")
    print("Plackett-Luce ranking (first to last):", ", ".join(pl.ranking()))
    print("top method:", pl.top_tool())
else:
    print("R with PlackettLuce not available; skipping.")
    print("Provision it with envs/heterogeneity.yml (conda or mamba).")
Plackett-Luce ranking (first to last): GPT-4, ScType, SingleR, GPT-3.5, CellMarker2.0
top method: GPT-4

Manual annotation

The agreement metric is defined against the manual annotation. The cell types with a method consensus on a broad lineage different from the manual label:

import csv
import collections
from importlib import resources

text = resources.files("beam.data").joinpath("gptcelltype2024_agreement.csv").read_text("utf-8")
ann_rows = [r for r in csv.DictReader(text.splitlines()) if r["method"] in head]

by_cell = collections.defaultdict(list)
for r in ann_rows:
    if r["broadtype"]:
        by_cell[(r["dataset"], r["cell_type"], r["manual_broadtype"])].append(r["broadtype"])

candidates = []
for (dataset, cell_type, manual), predicted in by_cell.items():
    if len(predicted) < 3:
        continue
    consensus, n = collections.Counter(predicted).most_common(1)[0]
    manual_lineages = {x.strip() for x in manual.split(",")}
    if n / len(predicted) >= 0.8 and consensus not in manual_lineages:
        candidates.append((dataset, cell_type, manual, consensus, f"{n}/{len(predicted)}"))

eligible = sum(1 for v in by_cell.values() if len(v) >= 3)
print(f"cell types where the methods agree on a lineage that differs from the manual one:")
print(f"  {len(candidates)} of {eligible} cell types with at least three running methods")
print("\nexamples (dataset, cell type, manual lineage, method consensus, votes):")
for dataset, cell_type, manual, consensus, votes in candidates[:8]:
    print(f"  {dataset:22s} {cell_type:26s} {manual:18s} -> {consensus} ({votes})")
cell types where the methods agree on a lineage that differs from the manual one:
  40 of 1029 cell types with at least three running methods

examples (dataset, cell type, manual lineage, method consensus, votes):
  Azimuth_Pancreas       immune                     immune cell        -> macrophage/monocyte (3/3)
  Azimuth_Fetal_Development Stromal cells              stromal cell       -> fibroblast (3/3)
  Azimuth_Lung           CD14+ Monocyte             macrophage/monocyte -> granulocyte (3/3)
  Azimuth_Lung           Capillary                  blood vessel cell  -> endothelial cell (3/3)
  Azimuth_Lung           Mucous                     mucosal cell       -> epithelial cell (3/3)
  Azimuth_Lung           Lipofibroblast             fibroblast         -> epithelial cell (3/3)
  Azimuth_Bone_Marrow    Innate Lymphoid Cell       innate lymphoid cell -> t/nk cell (3/3)
  Azimuth_Bone_Marrow    Platelet                   platelet           -> megakaryocyte (3/3)

In most of these the manual label is a fine type (innate lymphoid cell, platelet, capillary) and the methods agree on its parent lineage (t/nk cell, megakaryocyte, endothelial cell).

Partial coverage

The complete cases leave out the 18 datasets where SingleR and ScType were not run. Available-case SAW scores each method on the datasets where it was measured, with the weights renormalized. The Skillings-Mack test is the Friedman-type test for an incomplete matrix.

from beam.mcda import skillings_mack
from beam.reporting import funky_heatmap_from_run

full_scores = beam.Scores(
    values=sub,
    tool_names=tuple(head),
    metric_ids=tuple(metrics),
    dataset_names=tuple(g.dataset_names),
    layout="long",
)
full_run = beam.rank(
    full_scores, weights="equal", method="saw", sensitivity=True, missing="available"
)
print("available-case ranking on all 54 datasets:")
for i in np.argsort(full_run.result.ranks):
    print(f"  {head[i]:14s} rank {full_run.result.ranks[i]}")

sm = skillings_mack(sub[:, :, 0], higher_is_better=True, method_names=head)
print(f"\nSkillings-Mack statistic: {sm.statistic:.1f}  p-value: {sm.p_value:.2e}")
available-case ranking on all 54 datasets:
  GPT-4          rank 1
  GPT-3.5        rank 2
  SingleR        rank 3
  ScType         rank 4
  CellMarker2.0  rank 5

Skillings-Mack statistic: 114.5  p-value: 7.84e-24

The funky heatmap of the available-case run on all 54 datasets:

funky_heatmap_from_run(full_run, title="Cell type annotation: available-case ranking on 54 datasets")
plt.show()
/home/runner/work/beam/beam/src/beam/reporting/figures.py:776: UserWarning: Tight layout not applied. The bottom and top margins cannot be made large enough to accommodate all Axes decorations.
  fig.tight_layout()

Other LLM annotation benchmarks

Three more recent benchmarks score LLMs at cell type annotation: DeepCellSeek (Briefings in Bioinformatics 2025, 10.1093/bib/bbaf677), AnnDictionary (Nature Communications 2025, 10.1038/s41467-025-64511-x) and Single-Cell Omics Arena (SOAR, arXiv:2412.02915). They use different models, datasets and metrics, so they are compared by the within-benchmark rank of the shared methods. The three classical tools (CellMarker2.0, SingleR, ScType) are in GPTCelltype, DeepCellSeek and SOAR; GPT-4o is in DeepCellSeek, AnnDictionary and SOAR. The provenance and license of each bundled table are in src/beam/data/; DeepCellSeek and SOAR scores come from a pinned source, and the SOAR and AnnDictionary folders include a note on their license status.

import csv
from importlib import resources
from beam.datasets import load_deepcellseek

CLASSICAL = {"CellMarker2.0", "SingleR", "ScType", "Cell2Sentence"}

def ranking(scores_by_method):
    order = sorted(scores_by_method, key=lambda m: -scores_by_method[m])
    return {m: i + 1 for i, m in enumerate(order)}

benchmarks = {}
benchmarks["GPTCelltype"] = {m: float(np.nanmean(g.scores[i, :, 0]))
                             for i, m in enumerate(g.method_names)}
dcs = load_deepcellseek()
benchmarks["DeepCellSeek"] = {m: float(np.nanmean(dcs.scores[i, :, 0]))
                              for i, m in enumerate(dcs.method_names)}
ad = {r["model"]: float(r["overall_binary_celltypes_mean"]) for r in csv.DictReader(
    resources.files("beam.data").joinpath(
        "anndictionary/anndictionary_celltype_agreement.csv").read_text().splitlines())}
benchmarks["AnnDictionary"] = ad
soar = {r["model"]: float(r["R-L"]) for r in csv.DictReader(
    resources.files("beam.data").joinpath("soar/soar_readme_scores.csv").read_text().splitlines())
    if r["table_name"] == "SOAR-RNA Zero-shot" and r["R-L"]}
benchmarks["SOAR"] = soar

for name, sc in benchmarks.items():
    rk = ranking(sc)
    n = len(sc)
    classical = [m for m in sc if m in CLASSICAL]
    if classical:
        top = min(rk[m] for m in classical)
        print(f"{name:13s} {n:2d} methods; top-ranked classical ranks {top}/{n}; metric: "
              f"{'ontology agreement' if name in ('GPTCelltype','DeepCellSeek') else 'NLP overlap (ROUGE-L)'}")
    else:
        print(f"{name:13s} {n:2d} methods; LLM-only (no classical comparator)")
GPTCelltype    6 methods; top-ranked classical ranks 4/6; metric: ontology agreement
DeepCellSeek  14 methods; top-ranked classical ranks 12/14; metric: ontology agreement
AnnDictionary 14 methods; LLM-only (no classical comparator)
SOAR          11 methods; top-ranked classical ranks 6/11; metric: NLP overlap (ROUGE-L)
shared_methods = ["GPT-4o", "GPT-4", "CellMarker2.0", "SingleR", "ScType"]
bench_names = list(benchmarks)
grid = np.full((len(shared_methods), len(bench_names)), np.nan)
for j, name in enumerate(bench_names):
    rk = ranking(benchmarks[name])
    n = len(benchmarks[name])
    for i, m in enumerate(shared_methods):
        if m in rk:
            grid[i, j] = rk[m] / n          # normalized rank, 0 first, 1 last

fig, ax = plt.subplots(figsize=(6.0, 3.2))
im = ax.imshow(grid, cmap="RdYlGn_r", aspect="auto", vmin=0, vmax=1)
ax.set_xticks(range(len(bench_names))); ax.set_xticklabels(bench_names, rotation=20, ha="right")
ax.set_yticks(range(len(shared_methods))); ax.set_yticklabels(shared_methods)
ax.set_xlabel("benchmark"); ax.set_ylabel("shared method")
ax.set_title("Normalized rank of the shared methods (0 = first, 1 = last)")
for i in range(len(shared_methods)):
    for j in range(len(bench_names)):
        if not np.isnan(grid[i, j]):
            ax.text(j, i, f"{grid[i, j]:.2f}", ha="center", va="center", fontsize=8)
fig.colorbar(im, ax=ax, fraction=0.05, label="normalized rank")
fig.tight_layout()
plt.show()

A network meta-analysis is not possible: the AnnDictionary and SOAR tables are aggregates, without scores per dataset.

Variance on the shared classical methods

GPTCelltype and DeepCellSeek score the three classical tools per dataset on the same metric. source_variance_decomposition fits score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark) in lme4 over the two benchmarks and the three shared methods. It needs R.

from beam.heterogeneity import source_variance_decomposition, r_available

shared = ["CellMarker2.0", "SingleR", "ScType"]
sv_methods, sv_datasets, sv_benchmarks, sv_scores = [], [], [], []
for bname, b in [("GPTCelltype", g), ("DeepCellSeek", dcs)]:
    for m in shared:
        i = b.method_names.index(m)
        for j, ds in enumerate(b.dataset_names):
            v = b.scores[i, j, 0]
            if not np.isnan(v):
                sv_methods.append(m)
                sv_datasets.append(ds)
                sv_benchmarks.append(bname)
                sv_scores.append(float(v))

if r_available():
    sv = source_variance_decomposition(sv_methods, sv_datasets, sv_benchmarks, sv_scores)
    total = sum(v for v in sv.variance_components.values() if not np.isnan(v))
    print(f"{len(sv_scores)} observations, 2 benchmarks, 3 shared methods")
    for k, v in sv.variance_components.items():
        if not np.isnan(v):
            print(f"  {k:18s} {v / total:.2f} of the score variance")
    print("pooled method effect:", {n: round(e, 3) for n, e in zip(sv.method_names, sv.method_effects)})
else:
    sv = None
    print("R with lme4 not available; skipping the variance decomposition.")
    print("Provision it with envs/heterogeneity.yml (conda or mamba).")
195 observations, 2 benchmarks, 3 shared methods
  benchmark:dataset  0.00 of the score variance
  method:benchmark   0.10 of the score variance
  benchmark          0.24 of the score variance
  Residual           0.66 of the score variance
pooled method effect: {'CellMarker2.0': np.float64(0.389), 'ScType': np.float64(0.483), 'SingleR': np.float64(0.511)}
if sv is not None:
    comps = {k: v for k, v in sv.variance_components.items() if not np.isnan(v)}
    total = sum(comps.values())
    fig, axes = plt.subplots(1, 2, figsize=(9.0, 3.2))
    axes[0].bar(list(comps), [comps[k] / total * 100 for k in comps], color="tab:purple")
    axes[0].set_ylabel("share of score variance (percent)")
    axes[0].set_xlabel("source")
    axes[0].set_title("Variance on the three classical methods")
    axes[0].tick_params(axis="x", rotation=30)
    x = np.arange(len(shared))
    gm = [np.nanmean(g.scores[g.method_names.index(m), :, 0]) for m in shared]
    dm = [np.nanmean(dcs.scores[dcs.method_names.index(m), :, 0]) for m in shared]
    axes[1].bar(x - 0.2, gm, 0.4, label="GPTCelltype")
    axes[1].bar(x + 0.2, dm, 0.4, label="DeepCellSeek")
    axes[1].set_xticks(x)
    axes[1].set_xticklabels(shared, rotation=20, ha="right")
    axes[1].set_ylabel("mean agreement")
    axes[1].set_title("Classical methods per benchmark")
    axes[1].legend()
    fig.tight_layout()
    plt.show()

Summary

On the 36 datasets where all five methods were run, GPT-4 ranks first on the mean agreement and the full match rate with equal weights and SAW. On the critical-difference diagram of the agreement score GPT-4 and CellMarker2.0 are each separate from the other methods, and SingleR, ScType and GPT-3.5 are in one clique.