LLM cell type annotation: ranking GPT-4 against classical tools with beam
Author
Izaskun Mallona
Published
October 7, 2026
Data
Hou and Ji (2024) scored GPT-4 and GPT-3.5 against three reference-based annotators (CellMarker2.0, SingleR, ScType). Each method labels the cell types of a dataset, and the score is the agreement with a manual expert annotation.
The single-cell data and the model outputs are not bundled. src/beam/data/gptcelltype2024_agreement.csv and gptcelltype2024_features.csv are reduced from the evaluation table of the repository Winnie09/GPTCelltype_Paper at a pinned commit (CC-BY-4.0, see src/beam/data/README.md). Reference: Hou W, Ji Z. Assessing GPT-4 for cell type annotation in single-cell RNA-seq analysis. Nature Methods 2024, DOI 10.1038/s41592-024-02235-4.
all.csv has one row per annotated cell type and a <method>_agreement column per method (1 full match, 0.5 partial match, 0 mismatch). The reduction script is src/beam/data/reduce_gptcelltype.py:
python src/beam/data/reduce_gptcelltype.py # fetches the pinned commitpython src/beam/data/reduce_gptcelltype.py all.csv # or reduces a local copy
Tensor
load_gptcelltype reads the bundled tables. Each (source, tissue) pair is a dataset and each annotated cell type an observation. The two metrics are the mean agreement score and the full match rate, both higher is better.
SingleR and ScType were not run on the Azimuth and literature datasets, and the March 2023 GPT-4 endpoint was run on a subset of the datasets. Missing cells are numpy.nan and are not imputed.
for i, m inenumerate(g.method_names): covered =int(np.sum(~np.isnan(g.scores[i, :, 0]))) mean_agreement = np.nanmean(g.scores[i, :, 0])print(f"{m:14s} covers {covered:2d}/{len(g.dataset_names)} datasets mean agreement {mean_agreement:.3f}")
GPT-4 covers 54/54 datasets mean agreement 0.621
GPT-4-mar2023 covers 27/54 datasets mean agreement 0.625
GPT-3.5 covers 54/54 datasets mean agreement 0.452
CellMarker2.0 covers 54/54 datasets mean agreement 0.268
SingleR covers 36/54 datasets mean agreement 0.450
ScType covers 36/54 datasets mean agreement 0.450
classical = {"CellMarker2.0", "SingleR", "ScType"}mean_agreement = np.nanmean(g.scores[:, :, 0], axis=1)o = np.argsort(-mean_agreement)colors = ["tab:gray"if g.method_names[i] in classical else"tab:blue"for i in o]fig, ax = plt.subplots(figsize=(7.0, 3.2))ax.bar([g.method_names[i] for i in o], mean_agreement[o], color=colors)ax.set_ylabel("mean agreement over datasets")ax.set_xlabel("method (grey = classical, blue = LLM)")ax.set_title("Mean annotation agreement per method")ax.tick_params(axis="x", rotation=45)fig.tight_layout()plt.show()
The main comparison uses the five methods of Hou and Ji, without the March 2023 GPT-4 endpoint, on the datasets where all five were run.
head = ["GPT-4", "GPT-3.5", "CellMarker2.0", "SingleR", "ScType"]idx = [g.method_names.index(m) for m in head]metrics =list(g.metric_ids)sub = g.scores[idx] # 5 methods x 54 datasets x 2 metricscomplete =~np.isnan(sub[:, :, 0]).any(axis=0)block = sub[:, complete, :] # 5 x 36 x 2, no missing cellsblock_datasets = [d for d, keep inzip(g.dataset_names, complete) if keep]print(f"complete block: {block.shape[0]} methods, {block.shape[1]} datasets, {block.shape[2]} metrics")
complete block: 5 methods, 36 datasets, 2 metrics
Metric cards
properties_for reads the two annotation cards. Both are higher is better, bounded in [0, 1], with min-max normalization and the arithmetic mean across datasets.
props = properties_for(metrics)for p in props:print(f"{p.id:36s} polarity={p.polarity:16s} scale={p.scale_type:8s} "f"range=({p.range_lower}, {p.range_upper}) norm={p.recommended_normalization}" )
card_data_consistency checks the pooled scores against the cards before normalization.
audit = run.card_consistencyprint("pooled scores consistent with the cards:", audit.ok)for finding in audit.findings:print(" ", finding.severity, finding.message)
from beam.mcda import run_from_registrypooled = np.nanmean(block, axis=1) # 5 methods x 2 metrics, completeweightings = ["equal", "entropy", "std", "critic"]agg_methods = ["saw", "topsis", "vikor", "promethee_ii", "comet"]combos = [(w, m) for w in weightings for m in agg_methods]ranks_grid = np.zeros((len(combos), len(head)), dtype=int)row_labels = []for i, (w, m) inenumerate(combos): r = run_from_registry(pooled, metrics, weights=w, method=m) ranks_grid[i] = r.ranks row_labels.append(f"{w} / {m}")
from beam import plotplot.rank_heatmap( ranks_grid, row_names=row_labels, col_names=head, row_label="weighting / aggregation method", col_label="annotation method", title="Rank by configuration (both metrics higher is better)",)top_per_combo = {head[int(np.argmin(row))] for row in ranks_grid}print("methods top-ranked in at least one of the 20 configurations:", ", ".join(sorted(top_per_combo)))
methods top-ranked in at least one of the 20 configurations: GPT-4
aggregation_agreement gives the Kendall tau-b agreement of the rankings of the five aggregations at equal weights.
from beam.mcda import aggregation_agreementagg = aggregation_agreement(pooled, [p.polarity for p in props], tool_names=head)print(f"mean pairwise Kendall tau-b across aggregations: {agg.mean_pairwise_tau:.2f}")print(f"top method unanimous across aggregations: {agg.top_is_unanimous} ({head[agg.top_tool]})")
mean pairwise Kendall tau-b across aggregations: 0.98
top method unanimous across aggregations: True (GPT-4)
Rank stability
SMAA samples weight vectors and records how often each method has each rank. The weight perturbation is the smallest single-weight change displacing the top method.
smaa = run.smaatop =int(np.argmin(run.result.ranks))print(f"SMAA: {head[top]} is rank 1 in {smaa.rank_acceptability_index[top, 0] *100:.0f} percent of weight draws")wp = run.perturbationprint(f"top rank fragile: {wp.top_rank_is_fragile}")if wp.top_rank_perturbation isnotNone:print(f"smallest single-weight change at the top pair: {wp.top_rank_perturbation.absolute_delta:.2f}")else:print("no single-weight change within range alters the top rank")
SMAA: GPT-4 is rank 1 in 100 percent of weight draws
top rank fragile: False
no single-weight change within range alters the top rank
br = run.random_baselineprint("random-baseline check active:", br.active)if br.active:for pm in br.per_metric:print(f" {pm.metric:36s}{pm.n_beating}/{pm.n_observed} methods score above chance ({pm.baseline})")print(" methods above chance on no metric:", ", ".join(br.tools_never_beating) or"none")nf = run.noise_floorprint("noise-floor check active:", nf.active, "(no floor declared on the annotation cards)")
random-baseline check active: True
cell_type_annotation_full_match_rate 5/5 methods score above chance (0.0)
methods above chance on no metric: none
noise-floor check active: False (no floor declared on the annotation cards)
Critical difference on agreement
A critical-difference diagram ranks the methods on each of the 36 datasets, with a Friedman test and a Nemenyi post-hoc.
from beam.mcda import critical_differencecd = critical_difference(block[:, :, 0], "higher_is_better", tool_names=head)print(f"Friedman p-value: {cd.friedman_pvalue:.2e}")print(f"critical difference (alpha={cd.alpha}): {cd.critical_difference:.2f}")print("average rank (lower is better):")for i in np.argsort(cd.average_ranks):print(f" {head[i]:14s}{cd.average_ranks[i]:.2f}")print("cliques (methods not separable):")for clique in cd.cliques:print(" "+", ".join(head[i] for i in clique))
Friedman p-value: 4.21e-20
critical difference (alpha=0.05): 1.02
average rank (lower is better):
GPT-4 1.17
ScType 2.94
SingleR 2.99
GPT-3.5 3.14
CellMarker2.0 4.76
cliques (methods not separable):
ScType, SingleR, GPT-3.5
plot.critical_difference(cd)
pairwise_superiority gives, for each pair, the fraction of the 36 datasets with a higher score for one method.
from beam.mcda import pairwise_superiorityps = pairwise_superiority(block[:, :, 0], "higher_is_better", method_names=head)print("standing (Copeland-style, higher means ahead on more pairs):")for i in np.argsort(-ps.standing):print(f" {head[i]:14s}{ps.standing[i]:.2f}")
standing (Copeland-style, higher means ahead on more pairs):
GPT-4 0.96
ScType 0.51
SingleR 0.50
GPT-3.5 0.47
CellMarker2.0 0.06
from beam.mcda import pairwise_transitivitytrans = pairwise_transitivity(ps)print(f"transitive: {trans.is_transitive}; circular triads: {trans.n_circular_triads} of {trans.n_triads}")
transitive: True; circular triads: 0 of 10
The matrix orders the methods by the number of others outperformed. A red cell below the diagonal is a method outperforming one ranked above it.
plot.pairwise_majority(trans)
bayesian_sign_comparison gives, for each pair, the posterior probability that one method is practically better.
from beam.mcda import bayesian_sign_comparisonbayes = bayesian_sign_comparison(ps)decisive =sum(1for p in bayes.per_pair if p.decision !="inconclusive")print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(bayes.per_pair)}")plot.bayesian_comparison(bayes)
pairs with a decisive posterior at 0.95: 7 of 10
Rank sensitivity
rank_sensitivity ranks the methods on each dataset under every weighting and aggregation and splits the rank variance of each method into the dataset, the weighting, the aggregation and their interactions.
from beam.mcda import rank_sensitivityctx = run.contextrs = rank_sensitivity( block, ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=head, dataset_names=block_datasets,)print(f"{rs.n_combinations} combinations")print(f" dataset: {rs.dataset_share:.3f} of the rank variance")print(f" weighting: {rs.weighting_share:.3f}")print(f" aggregation: {rs.aggregation_share:.3f}")print(f" interactions: {rs.interaction_share:.3f}")print(f" most influential factor: {rs.most_influential_factor}")
720 combinations
dataset: 0.962 of the rank variance
weighting: 0.001
aggregation: 0.000
interactions: 0.037
most influential factor: dataset
plot.rank_sensitivity(rs)
The same split per method, with each method’s rank range:
top method rank stability under leave-one-dataset-out: 1.00
most influential dataset: HCA_Skeletal_muscle
specification_curve lists the rankings of the grid, on the pooled matrix (weighting by aggregation) and with the datasets.
from beam.mcda import specification_curvecurve = specification_curve(rs)dom = curve.tool_names[curve.most_frequent_top_tool]print(f"choices plus dataset ({curve.n_specifications} combinations): "f"{dom} first in {curve.most_frequent_top_fraction *100:.0f}%, "f"{curve.n_distinct_top_tools} methods reach the top")pooled = specification_curve( rank_sensitivity( run.matrix, ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=head, ))pdom = pooled.tool_names[pooled.most_frequent_top_tool]print(f"choices only ({pooled.n_specifications} combinations): "f"{pdom} first in {pooled.most_frequent_top_fraction *100:.0f}%")
choices plus dataset (720 combinations): GPT-4 first in 94%, 3 methods reach the top
choices only (20 combinations): GPT-4 first in 100%
Dataset concordance
dataset_concordance ranks the methods within each dataset and compares every pair of orderings with Kendall tau-b.
conc = run.dataset_concordancenames = conc.dataset_namesprint(f"mean agreement across datasets (Kendall tau-b): {conc.mean_pairwise_tau:.2f}")print("least typical dataset:", names[conc.most_idiosyncratic_dataset])print("number of mutually consistent dataset groups:", len(conc.concordant_groups))print("where methods depart most from their own average rank:")for cell in conc.notable_cells[:5]: side ="lower"if cell.deviation >0else"higher"print(f" {conc.tool_names[cell.tool]} on {names[cell.dataset]}: "f"rank {cell.rank}, {side} than its mean {cell.mean_rank:.1f}")
mean agreement across datasets (Kendall tau-b): 0.53
least typical dataset: HCA_Lung
number of mutually consistent dataset groups: 1
where methods depart most from their own average rank:
SingleR on tabulasapiens_Heart: rank 5, lower than its mean 3.0
SingleR on tabulasapiens_Pancreas: rank 5, lower than its mean 3.0
SingleR on tabulasapiens_Skin: rank 5, lower than its mean 3.0
ScType on HCA_Heart: rank 5, lower than its mean 3.0
ScType on HCA_Lung: rank 1, higher than its mean 3.0
plot.dataset_concordance(run)
Rank of each method per dataset minus its mean rank:
plot.dataset_struggle(run)
Blind analysis
A blind analysis fixes the pipeline before the method names are known. beam.blind masks the names and shuffles the rows; beam.unblind restores them. The seal fingerprint is recorded in the manifest.
from beam import blind, unblindblinded, seal = blind(block_scores, seed=0)blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)restored = unblind(blind_run, seal)named = beam.rank(block_scores, weights="equal", method="saw", seed=0, sensitivity=False)print("ranking identical after unblinding:",dict(zip(named.tool_names, named.result.ranks))==dict(zip(restored.tool_names, restored.result.ranks)))print("top method after unblinding:", restored.top_tool)print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12])
ranking identical after unblinding: True
top method after unblinding: GPT-4
blinding fingerprint: 50b6fc613b9f
Equal and cell-weighted pooling
By default each dataset has the same weight. The per-cell-type accuracy of Hou and Ji weights a dataset by its number of cell types.
n_cell_types = np.array(g.features.numeric["n_cell_types"])[complete]agreement = block[:, :, 0]equal_pool = agreement.mean(axis=1)weighted_pool = (agreement * n_cell_types).sum(axis=1) / n_cell_types.sum()print(f"{'method':14s} equal-per-dataset cell-weighted")for i in np.argsort(-equal_pool):print(f"{head[i]:14s}{equal_pool[i]:.3f}{weighted_pool[i]:.3f}")
from beam.heterogeneity import mixed_effects_from_matrix, r_availableif r_available(): me = mixed_effects_from_matrix(agreement, head, block_datasets)print(f"model: {me.formula}")print(f"dataset shift (ICC): {me.icc_dataset:.2f} of the agreement variance") order = np.argsort(-me.method_effects)print("\nmethod agreement marginal mean")for i in order:print(f"{me.method_names[i]:14s}{me.method_effects[i]:.3f} +/- {me.method_effect_se[i]:.3f}")else: me =Noneprint("R with lme4 not available; skipping the mixed-effects fit.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")
A Bradley-Terry tree takes each dataset as a set of pairwise method comparisons and splits the datasets by their features: source, tissue, species, sample type (normal or cancer) and number of cell types. It needs R.
from beam.heterogeneity import bradley_terry_tree, bttree_availableif bttree_available(): numeric, categorical = g.features.aligned_to(block_datasets) bt = bradley_terry_tree( agreement, head, block_datasets, numeric_features=numeric, categorical_features=categorical, polarity="higher_is_better", minsize=6, )print(f"split found: {bt.did_split}")print(f"global ranking (top 3): {', '.join(bt.global_ranking()[:3])}")print(bt.summary())else: bt =Noneprint("R with psychotree not available; skipping the Bradley-Terry tree.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")
split found: False
global ranking (top 3): GPT-4, ScType, SingleR
The Bradley-Terry tree found no dataset feature that splits the method ranking at alpha 0.05 over 36 datasets, so the ranking is reported as one Bradley-Terry model over all of them, led by GPT-4. With this many datasets the split test has few observations to work with, the same small-sample limit the critical-difference diagram shows; a benchmark with more datasets is where a split can appear.
plackett_luce fits one full-ranking model over the methods.
from beam.heterogeneity import plackett_luce, plackett_luce_availableif plackett_luce_available(): pl = plackett_luce(agreement, head, polarity="higher_is_better")print("Plackett-Luce ranking (first to last):", ", ".join(pl.ranking()))print("top method:", pl.top_tool())else:print("R with PlackettLuce not available; skipping.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")
Plackett-Luce ranking (first to last): GPT-4, ScType, SingleR, GPT-3.5, CellMarker2.0
top method: GPT-4
Manual annotation
The agreement metric is defined against the manual annotation. The cell types with a method consensus on a broad lineage different from the manual label:
import csvimport collectionsfrom importlib import resourcestext = resources.files("beam.data").joinpath("gptcelltype2024_agreement.csv").read_text("utf-8")ann_rows = [r for r in csv.DictReader(text.splitlines()) if r["method"] in head]by_cell = collections.defaultdict(list)for r in ann_rows:if r["broadtype"]: by_cell[(r["dataset"], r["cell_type"], r["manual_broadtype"])].append(r["broadtype"])candidates = []for (dataset, cell_type, manual), predicted in by_cell.items():iflen(predicted) <3:continue consensus, n = collections.Counter(predicted).most_common(1)[0] manual_lineages = {x.strip() for x in manual.split(",")}if n /len(predicted) >=0.8and consensus notin manual_lineages: candidates.append((dataset, cell_type, manual, consensus, f"{n}/{len(predicted)}"))eligible =sum(1for v in by_cell.values() iflen(v) >=3)print(f"cell types where the methods agree on a lineage that differs from the manual one:")print(f" {len(candidates)} of {eligible} cell types with at least three running methods")print("\nexamples (dataset, cell type, manual lineage, method consensus, votes):")for dataset, cell_type, manual, consensus, votes in candidates[:8]:print(f" {dataset:22s}{cell_type:26s}{manual:18s} -> {consensus} ({votes})")
cell types where the methods agree on a lineage that differs from the manual one:
40 of 1029 cell types with at least three running methods
examples (dataset, cell type, manual lineage, method consensus, votes):
Azimuth_Pancreas immune immune cell -> macrophage/monocyte (3/3)
Azimuth_Fetal_Development Stromal cells stromal cell -> fibroblast (3/3)
Azimuth_Lung CD14+ Monocyte macrophage/monocyte -> granulocyte (3/3)
Azimuth_Lung Capillary blood vessel cell -> endothelial cell (3/3)
Azimuth_Lung Mucous mucosal cell -> epithelial cell (3/3)
Azimuth_Lung Lipofibroblast fibroblast -> epithelial cell (3/3)
Azimuth_Bone_Marrow Innate Lymphoid Cell innate lymphoid cell -> t/nk cell (3/3)
Azimuth_Bone_Marrow Platelet platelet -> megakaryocyte (3/3)
In most of these the manual label is a fine type (innate lymphoid cell, platelet, capillary) and the methods agree on its parent lineage (t/nk cell, megakaryocyte, endothelial cell).
Partial coverage
The complete cases leave out the 18 datasets where SingleR and ScType were not run. Available-case SAW scores each method on the datasets where it was measured, with the weights renormalized. The Skillings-Mack test is the Friedman-type test for an incomplete matrix.
from beam.mcda import skillings_mackfrom beam.reporting import funky_heatmap_from_runfull_scores = beam.Scores( values=sub, tool_names=tuple(head), metric_ids=tuple(metrics), dataset_names=tuple(g.dataset_names), layout="long",)full_run = beam.rank( full_scores, weights="equal", method="saw", sensitivity=True, missing="available")print("available-case ranking on all 54 datasets:")for i in np.argsort(full_run.result.ranks):print(f" {head[i]:14s} rank {full_run.result.ranks[i]}")sm = skillings_mack(sub[:, :, 0], higher_is_better=True, method_names=head)print(f"\nSkillings-Mack statistic: {sm.statistic:.1f} p-value: {sm.p_value:.2e}")
The funky heatmap of the available-case run on all 54 datasets:
funky_heatmap_from_run(full_run, title="Cell type annotation: available-case ranking on 54 datasets")plt.show()
/home/runner/work/beam/beam/src/beam/reporting/figures.py:776: UserWarning: Tight layout not applied. The bottom and top margins cannot be made large enough to accommodate all Axes decorations.
fig.tight_layout()
Other LLM annotation benchmarks
Three more recent benchmarks score LLMs at cell type annotation: DeepCellSeek (Briefings in Bioinformatics 2025, 10.1093/bib/bbaf677), AnnDictionary (Nature Communications 2025, 10.1038/s41467-025-64511-x) and Single-Cell Omics Arena (SOAR, arXiv:2412.02915). They use different models, datasets and metrics, so they are compared by the within-benchmark rank of the shared methods. The three classical tools (CellMarker2.0, SingleR, ScType) are in GPTCelltype, DeepCellSeek and SOAR; GPT-4o is in DeepCellSeek, AnnDictionary and SOAR. The provenance and license of each bundled table are in src/beam/data/; DeepCellSeek and SOAR scores come from a pinned source, and the SOAR and AnnDictionary folders include a note on their license status.
import csvfrom importlib import resourcesfrom beam.datasets import load_deepcellseekCLASSICAL = {"CellMarker2.0", "SingleR", "ScType", "Cell2Sentence"}def ranking(scores_by_method): order =sorted(scores_by_method, key=lambda m: -scores_by_method[m])return {m: i +1for i, m inenumerate(order)}benchmarks = {}benchmarks["GPTCelltype"] = {m: float(np.nanmean(g.scores[i, :, 0]))for i, m inenumerate(g.method_names)}dcs = load_deepcellseek()benchmarks["DeepCellSeek"] = {m: float(np.nanmean(dcs.scores[i, :, 0]))for i, m inenumerate(dcs.method_names)}ad = {r["model"]: float(r["overall_binary_celltypes_mean"]) for r in csv.DictReader( resources.files("beam.data").joinpath("anndictionary/anndictionary_celltype_agreement.csv").read_text().splitlines())}benchmarks["AnnDictionary"] = adsoar = {r["model"]: float(r["R-L"]) for r in csv.DictReader( resources.files("beam.data").joinpath("soar/soar_readme_scores.csv").read_text().splitlines())if r["table_name"] =="SOAR-RNA Zero-shot"and r["R-L"]}benchmarks["SOAR"] = soarfor name, sc in benchmarks.items(): rk = ranking(sc) n =len(sc) classical = [m for m in sc if m in CLASSICAL]if classical: top =min(rk[m] for m in classical)print(f"{name:13s}{n:2d} methods; top-ranked classical ranks {top}/{n}; metric: "f"{'ontology agreement'if name in ('GPTCelltype','DeepCellSeek') else'NLP overlap (ROUGE-L)'}")else:print(f"{name:13s}{n:2d} methods; LLM-only (no classical comparator)")
shared_methods = ["GPT-4o", "GPT-4", "CellMarker2.0", "SingleR", "ScType"]bench_names =list(benchmarks)grid = np.full((len(shared_methods), len(bench_names)), np.nan)for j, name inenumerate(bench_names): rk = ranking(benchmarks[name]) n =len(benchmarks[name])for i, m inenumerate(shared_methods):if m in rk: grid[i, j] = rk[m] / n # normalized rank, 0 first, 1 lastfig, ax = plt.subplots(figsize=(6.0, 3.2))im = ax.imshow(grid, cmap="RdYlGn_r", aspect="auto", vmin=0, vmax=1)ax.set_xticks(range(len(bench_names))); ax.set_xticklabels(bench_names, rotation=20, ha="right")ax.set_yticks(range(len(shared_methods))); ax.set_yticklabels(shared_methods)ax.set_xlabel("benchmark"); ax.set_ylabel("shared method")ax.set_title("Normalized rank of the shared methods (0 = first, 1 = last)")for i inrange(len(shared_methods)):for j inrange(len(bench_names)):ifnot np.isnan(grid[i, j]): ax.text(j, i, f"{grid[i, j]:.2f}", ha="center", va="center", fontsize=8)fig.colorbar(im, ax=ax, fraction=0.05, label="normalized rank")fig.tight_layout()plt.show()
A network meta-analysis is not possible: the AnnDictionary and SOAR tables are aggregates, without scores per dataset.
Variance on the shared classical methods
GPTCelltype and DeepCellSeek score the three classical tools per dataset on the same metric. source_variance_decomposition fits score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark) in lme4 over the two benchmarks and the three shared methods. It needs R.
from beam.heterogeneity import source_variance_decomposition, r_availableshared = ["CellMarker2.0", "SingleR", "ScType"]sv_methods, sv_datasets, sv_benchmarks, sv_scores = [], [], [], []for bname, b in [("GPTCelltype", g), ("DeepCellSeek", dcs)]:for m in shared: i = b.method_names.index(m)for j, ds inenumerate(b.dataset_names): v = b.scores[i, j, 0]ifnot np.isnan(v): sv_methods.append(m) sv_datasets.append(ds) sv_benchmarks.append(bname) sv_scores.append(float(v))if r_available(): sv = source_variance_decomposition(sv_methods, sv_datasets, sv_benchmarks, sv_scores) total =sum(v for v in sv.variance_components.values() ifnot np.isnan(v))print(f"{len(sv_scores)} observations, 2 benchmarks, 3 shared methods")for k, v in sv.variance_components.items():ifnot np.isnan(v):print(f" {k:18s}{v / total:.2f} of the score variance")print("pooled method effect:", {n: round(e, 3) for n, e inzip(sv.method_names, sv.method_effects)})else: sv =Noneprint("R with lme4 not available; skipping the variance decomposition.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")
195 observations, 2 benchmarks, 3 shared methods
benchmark:dataset 0.00 of the score variance
method:benchmark 0.10 of the score variance
benchmark 0.24 of the score variance
Residual 0.66 of the score variance
pooled method effect: {'CellMarker2.0': np.float64(0.389), 'ScType': np.float64(0.483), 'SingleR': np.float64(0.511)}
if sv isnotNone: comps = {k: v for k, v in sv.variance_components.items() ifnot np.isnan(v)} total =sum(comps.values()) fig, axes = plt.subplots(1, 2, figsize=(9.0, 3.2)) axes[0].bar(list(comps), [comps[k] / total *100for k in comps], color="tab:purple") axes[0].set_ylabel("share of score variance (percent)") axes[0].set_xlabel("source") axes[0].set_title("Variance on the three classical methods") axes[0].tick_params(axis="x", rotation=30) x = np.arange(len(shared)) gm = [np.nanmean(g.scores[g.method_names.index(m), :, 0]) for m in shared] dm = [np.nanmean(dcs.scores[dcs.method_names.index(m), :, 0]) for m in shared] axes[1].bar(x -0.2, gm, 0.4, label="GPTCelltype") axes[1].bar(x +0.2, dm, 0.4, label="DeepCellSeek") axes[1].set_xticks(x) axes[1].set_xticklabels(shared, rotation=20, ha="right") axes[1].set_ylabel("mean agreement") axes[1].set_title("Classical methods per benchmark") axes[1].legend() fig.tight_layout() plt.show()
Summary
On the 36 datasets where all five methods were run, GPT-4 ranks first on the mean agreement and the full match rate with equal weights and SAW. On the critical-difference diagram of the agreement score GPT-4 and CellMarker2.0 are each separate from the other methods, and SingleR, ScType and GPT-3.5 are in one clique.
Source Code
---title: "LLM cell type annotation: ranking GPT-4 against classical tools with beam"author: "Izaskun Mallona"date: todayformat: html: theme: cosmo toc: true toc-location: left embed-resources: true code-tools: true fig-width: 6 fig-height: 3.5---## DataHou and Ji (2024) scored GPT-4 and GPT-3.5 against three reference-based annotators (CellMarker2.0, SingleR, ScType). Each method labels the cell types of a dataset, and the score is the agreement with a manual expert annotation.The single-cell data and the model outputs are not bundled. `src/beam/data/gptcelltype2024_agreement.csv` and `gptcelltype2024_features.csv` are reduced from the evaluation table of the repository `Winnie09/GPTCelltype_Paper` at a pinned commit (CC-BY-4.0, see `src/beam/data/README.md`). Reference: Hou W, Ji Z. Assessing GPT-4 for cell type annotation in single-cell RNA-seq analysis. Nature Methods 2024, DOI [10.1038/s41592-024-02235-4](https://doi.org/10.1038/s41592-024-02235-4).To reproduce, run:```bashSHA=5944a41511aacd368b45448e256d9625849704dfcurl-sf-o all.csv \"https://raw.githubusercontent.com/Winnie09/GPTCelltype_Paper/$SHA/anno/compiled/all.csv"````all.csv` has one row per annotated cell type and a `<method>_agreement` column per method (1 full match, 0.5 partial match, 0 mismatch). The reduction script is `src/beam/data/reduce_gptcelltype.py`:```bashpython src/beam/data/reduce_gptcelltype.py # fetches the pinned commitpython src/beam/data/reduce_gptcelltype.py all.csv # or reduces a local copy```## Tensor`load_gptcelltype` reads the bundled tables. Each (source, tissue) pair is a dataset and each annotated cell type an observation. The two metrics are the mean agreement score and the full match rate, both higher is better.```{python}%matplotlib inlineimport numpy as npimport matplotlib.pyplot as pltimport beamfrom beam.datasets import load_gptcelltypefrom beam.cards import properties_forg = load_gptcelltype()print("scores shape:", g.scores.shape)print("methods:", ", ".join(g.method_names))print("datasets:", len(g.dataset_names))print("metrics:", ", ".join(g.metric_ids))```## CoverageSingleR and ScType were not run on the Azimuth and literature datasets, and the March 2023 GPT-4 endpoint was run on a subset of the datasets. Missing cells are `numpy.nan` and [are not imputed](../../docs/explanations/missing-data.md).```{python}for i, m inenumerate(g.method_names): covered =int(np.sum(~np.isnan(g.scores[i, :, 0]))) mean_agreement = np.nanmean(g.scores[i, :, 0])print(f"{m:14s} covers {covered:2d}/{len(g.dataset_names)} datasets mean agreement {mean_agreement:.3f}")``````{python}classical = {"CellMarker2.0", "SingleR", "ScType"}mean_agreement = np.nanmean(g.scores[:, :, 0], axis=1)o = np.argsort(-mean_agreement)colors = ["tab:gray"if g.method_names[i] in classical else"tab:blue"for i in o]fig, ax = plt.subplots(figsize=(7.0, 3.2))ax.bar([g.method_names[i] for i in o], mean_agreement[o], color=colors)ax.set_ylabel("mean agreement over datasets")ax.set_xlabel("method (grey = classical, blue = LLM)")ax.set_title("Mean annotation agreement per method")ax.tick_params(axis="x", rotation=45)fig.tight_layout()plt.show()```The main comparison uses the five methods of Hou and Ji, without the March 2023 GPT-4 endpoint, on the datasets where all five were run.```{python}head = ["GPT-4", "GPT-3.5", "CellMarker2.0", "SingleR", "ScType"]idx = [g.method_names.index(m) for m in head]metrics =list(g.metric_ids)sub = g.scores[idx] # 5 methods x 54 datasets x 2 metricscomplete =~np.isnan(sub[:, :, 0]).any(axis=0)block = sub[:, complete, :] # 5 x 36 x 2, no missing cellsblock_datasets = [d for d, keep inzip(g.dataset_names, complete) if keep]print(f"complete block: {block.shape[0]} methods, {block.shape[1]} datasets, {block.shape[2]} metrics")```## Metric cards[`properties_for`](../../docs/reference/properties_for.qmd) reads the two annotation cards. Both are higher is better, bounded in [0, 1], with [min-max normalization](../../docs/explanations/normalization-and-scales.md) and the arithmetic mean across datasets.```{python}props = properties_for(metrics)for p in props:print(f"{p.id:36s} polarity={p.polarity:16s} scale={p.scale_type:8s} "f"range=({p.range_lower}, {p.range_upper}) norm={p.recommended_normalization}" )```## Ranking on the complete casesThe two metrics are pooled across the 36 datasets and the five methods ranked with equal weights and simple additive weighting (SAW).```{python}block_scores = beam.Scores( values=block, tool_names=tuple(head), metric_ids=tuple(metrics), dataset_names=tuple(block_datasets), layout="long",)run = beam.rank(block_scores, weights="equal", method="saw", sensitivity=True)print(f"weighting={run.result.weighting} method={run.result.method}")order = np.argsort(run.result.ranks)print(f"\n{'method':14s} composite rank")for i in order:print(f"{head[i]:14s}{run.result.composite[i]:.3f}{run.result.ranks[i]}")```GPT-4 ranks first.## Card consistency[`card_data_consistency`](../../docs/explanations/card-data-consistency.md) checks the pooled scores against the cards before normalization.```{python}audit = run.card_consistencyprint("pooled scores consistent with the cards:", audit.ok)for finding in audit.findings:print(" ", finding.severity, finding.message)```## Weightings and aggregation methodsFour [weightings](../../docs/explanations/weighting-schemes.md) (equal, entropy, std, critic) by five [aggregation methods](../../docs/explanations/aggregation-methods.md) give twenty runs.```{python}from beam.mcda import run_from_registrypooled = np.nanmean(block, axis=1) # 5 methods x 2 metrics, completeweightings = ["equal", "entropy", "std", "critic"]agg_methods = ["saw", "topsis", "vikor", "promethee_ii", "comet"]combos = [(w, m) for w in weightings for m in agg_methods]ranks_grid = np.zeros((len(combos), len(head)), dtype=int)row_labels = []for i, (w, m) inenumerate(combos): r = run_from_registry(pooled, metrics, weights=w, method=m) ranks_grid[i] = r.ranks row_labels.append(f"{w} / {m}")``````{python}from beam import plotplot.rank_heatmap( ranks_grid, row_names=row_labels, col_names=head, row_label="weighting / aggregation method", col_label="annotation method", title="Rank by configuration (both metrics higher is better)",)top_per_combo = {head[int(np.argmin(row))] for row in ranks_grid}print("methods top-ranked in at least one of the 20 configurations:", ", ".join(sorted(top_per_combo)))```[`aggregation_agreement`](../../docs/explanations/choice-agreement.md) gives the Kendall tau-b agreement of the rankings of the five aggregations at equal weights.```{python}from beam.mcda import aggregation_agreementagg = aggregation_agreement(pooled, [p.polarity for p in props], tool_names=head)print(f"mean pairwise Kendall tau-b across aggregations: {agg.mean_pairwise_tau:.2f}")print(f"top method unanimous across aggregations: {agg.top_is_unanimous} ({head[agg.top_tool]})")```## Rank stabilitySMAA samples weight vectors and records how often each method has each rank. The weight perturbation is the smallest single-weight change displacing the top method.```{python}smaa = run.smaatop =int(np.argmin(run.result.ranks))print(f"SMAA: {head[top]} is rank 1 in {smaa.rank_acceptability_index[top, 0] *100:.0f} percent of weight draws")wp = run.perturbationprint(f"top rank fragile: {wp.top_rank_is_fragile}")if wp.top_rank_perturbation isnotNone:print(f"smallest single-weight change at the top pair: {wp.top_rank_perturbation.absolute_delta:.2f}")else:print("no single-weight change within range alters the top rank")``````{python}plot.smaa(run)```[`leave_one_metric_out`](../../docs/reference/leave_one_metric_out.qmd) drops each metric in turn.```{python}loo = run.leave_one_outprint(f"top method rank stability under leave-one-metric-out: {loo.rank_stability[top]:.2f}")print(f"most influential metric: {metrics[loo.most_influential_metric]}")```## Chance baseline and noise floor[`beats_random_baseline`](../../docs/reference/beats_random_baseline.qmd) uses the [chance level](../../docs/explanations/reference-levels.md) on the card, 0 for the full match rate. [`noise_floor_separation`](../../docs/reference/noise_floor_separation.qmd) is inactive, as the annotation cards declare no noise floor.```{python}br = run.random_baselineprint("random-baseline check active:", br.active)if br.active:for pm in br.per_metric:print(f" {pm.metric:36s}{pm.n_beating}/{pm.n_observed} methods score above chance ({pm.baseline})")print(" methods above chance on no metric:", ", ".join(br.tools_never_beating) or"none")nf = run.noise_floorprint("noise-floor check active:", nf.active, "(no floor declared on the annotation cards)")```## Critical difference on agreementA [critical-difference diagram](../../docs/explanations/critical-difference.md) ranks the methods on each of the 36 datasets, with a Friedman test and a Nemenyi post-hoc.```{python}from beam.mcda import critical_differencecd = critical_difference(block[:, :, 0], "higher_is_better", tool_names=head)print(f"Friedman p-value: {cd.friedman_pvalue:.2e}")print(f"critical difference (alpha={cd.alpha}): {cd.critical_difference:.2f}")print("average rank (lower is better):")for i in np.argsort(cd.average_ranks):print(f" {head[i]:14s}{cd.average_ranks[i]:.2f}")print("cliques (methods not separable):")for clique in cd.cliques:print(" "+", ".join(head[i] for i in clique))``````{python}plot.critical_difference(cd)```[`pairwise_superiority`](../../docs/explanations/pairwise-method-comparison.md) gives, for each pair, the fraction of the 36 datasets with a higher score for one method.```{python}from beam.mcda import pairwise_superiorityps = pairwise_superiority(block[:, :, 0], "higher_is_better", method_names=head)print("standing (Copeland-style, higher means ahead on more pairs):")for i in np.argsort(-ps.standing):print(f" {head[i]:14s}{ps.standing[i]:.2f}")```[`pairwise_transitivity`](../../docs/reference/pairwise_transitivity.qmd) tests whether one order [agrees with the pairwise majorities](../../docs/explanations/pairwise-method-comparison.md#transitivity).```{python}from beam.mcda import pairwise_transitivitytrans = pairwise_transitivity(ps)print(f"transitive: {trans.is_transitive}; circular triads: {trans.n_circular_triads} of {trans.n_triads}")```The matrix orders the methods by the number of others outperformed. A red cell below the diagonal is a method outperforming one ranked above it.```{python}plot.pairwise_majority(trans)```[`bayesian_sign_comparison`](../../docs/explanations/pairwise-method-comparison.md#bayesian-sign-comparison) gives, for each pair, the posterior probability that one method is practically better.```{python}from beam.mcda import bayesian_sign_comparisonbayes = bayesian_sign_comparison(ps)decisive =sum(1for p in bayes.per_pair if p.decision !="inconclusive")print(f"pairs with a decisive posterior at 0.95: {decisive} of {len(bayes.per_pair)}")plot.bayesian_comparison(bayes)```## Rank sensitivity[`rank_sensitivity`](../../docs/explanations/rank-sensitivity.md) ranks the methods on each dataset under every weighting and aggregation and splits the rank variance of each method into the dataset, the weighting, the aggregation and their interactions.```{python}from beam.mcda import rank_sensitivityctx = run.contextrs = rank_sensitivity( block, ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=head, dataset_names=block_datasets,)print(f"{rs.n_combinations} combinations")print(f" dataset: {rs.dataset_share:.3f} of the rank variance")print(f" weighting: {rs.weighting_share:.3f}")print(f" aggregation: {rs.aggregation_share:.3f}")print(f" interactions: {rs.interaction_share:.3f}")print(f" most influential factor: {rs.most_influential_factor}")``````{python}plot.rank_sensitivity(rs)```The same split per method, with each method's rank range:```{python}plot.rank_sensitivity_by_tool(rs)``````{python}lodo = run.leave_one_dataset_outprint(f"top method rank stability under leave-one-dataset-out: {lodo.rank_stability[top]:.2f}")print(f"most influential dataset: {block_datasets[lodo.most_influential_dataset]}")```[`specification_curve`](../../docs/explanations/rank-sensitivity.md#the-specification-curve) lists the rankings of the grid, on the pooled matrix (weighting by aggregation) and with the datasets.```{python}from beam.mcda import specification_curvecurve = specification_curve(rs)dom = curve.tool_names[curve.most_frequent_top_tool]print(f"choices plus dataset ({curve.n_specifications} combinations): "f"{dom} first in {curve.most_frequent_top_fraction *100:.0f}%, "f"{curve.n_distinct_top_tools} methods reach the top")pooled = specification_curve( rank_sensitivity( run.matrix, ctx.polarity, normalization=list(ctx.normalization), bounds=list(ctx.bounds), baselines=list(ctx.baselines), targets=list(ctx.targets), tool_names=head, ))pdom = pooled.tool_names[pooled.most_frequent_top_tool]print(f"choices only ({pooled.n_specifications} combinations): "f"{pdom} first in {pooled.most_frequent_top_fraction *100:.0f}%")```## Dataset concordance[`dataset_concordance`](../../docs/explanations/dataset-concordance-and-discrimination.md) ranks the methods within each dataset and compares every pair of orderings with Kendall tau-b.```{python}conc = run.dataset_concordancenames = conc.dataset_namesprint(f"mean agreement across datasets (Kendall tau-b): {conc.mean_pairwise_tau:.2f}")print("least typical dataset:", names[conc.most_idiosyncratic_dataset])print("number of mutually consistent dataset groups:", len(conc.concordant_groups))print("where methods depart most from their own average rank:")for cell in conc.notable_cells[:5]: side ="lower"if cell.deviation >0else"higher"print(f" {conc.tool_names[cell.tool]} on {names[cell.dataset]}: "f"rank {cell.rank}, {side} than its mean {cell.mean_rank:.1f}")``````{python}plot.dataset_concordance(run)```Rank of each method per dataset minus its mean rank:```{python}plot.dataset_struggle(run)```## Blind analysisA [blind analysis](../../docs/explanations/analysis-blinding.md) fixes the pipeline before the method names are known. `beam.blind` masks the names and shuffles the rows; `beam.unblind` restores them. The seal fingerprint is recorded in the manifest.```{python}from beam import blind, unblindblinded, seal = blind(block_scores, seed=0)blind_run = beam.rank(blinded, weights="equal", method="saw", seed=0, sensitivity=False)restored = unblind(blind_run, seal)named = beam.rank(block_scores, weights="equal", method="saw", seed=0, sensitivity=False)print("ranking identical after unblinding:",dict(zip(named.tool_names, named.result.ranks))==dict(zip(restored.tool_names, restored.result.ranks)))print("top method after unblinding:", restored.top_tool)print("blinding fingerprint:", blind_run.manifest["blinding"]["seal_sha256"][:12])```## Equal and cell-weighted poolingBy default each dataset has the same weight. The per-cell-type accuracy of Hou and Ji weights a dataset by its number of cell types.```{python}n_cell_types = np.array(g.features.numeric["n_cell_types"])[complete]agreement = block[:, :, 0]equal_pool = agreement.mean(axis=1)weighted_pool = (agreement * n_cell_types).sum(axis=1) / n_cell_types.sum()print(f"{'method':14s} equal-per-dataset cell-weighted")for i in np.argsort(-equal_pool):print(f"{head[i]:14s}{equal_pool[i]:.3f}{weighted_pool[i]:.3f}")```## Mixed-effects on agreementA [mixed-effects model](../../docs/explanations/method-by-dataset-heterogeneity.md) fits `agreement ~ method + (1 | dataset)` in R's lme4.```{python}from beam.heterogeneity import mixed_effects_from_matrix, r_availableif r_available(): me = mixed_effects_from_matrix(agreement, head, block_datasets)print(f"model: {me.formula}")print(f"dataset shift (ICC): {me.icc_dataset:.2f} of the agreement variance") order = np.argsort(-me.method_effects)print("\nmethod agreement marginal mean")for i in order:print(f"{me.method_names[i]:14s}{me.method_effects[i]:.3f} +/- {me.method_effect_se[i]:.3f}")else: me =Noneprint("R with lme4 not available; skipping the mixed-effects fit.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")```## Bradley-Terry treeA [Bradley-Terry tree](../../docs/explanations/method-by-dataset-heterogeneity.md#bradley-terry-trees) takes each dataset as a set of pairwise method comparisons and splits the datasets by their features: source, tissue, species, sample type (normal or cancer) and number of cell types. It needs R.```{python}from beam.heterogeneity import bradley_terry_tree, bttree_availableif bttree_available(): numeric, categorical = g.features.aligned_to(block_datasets) bt = bradley_terry_tree( agreement, head, block_datasets, numeric_features=numeric, categorical_features=categorical, polarity="higher_is_better", minsize=6, )print(f"split found: {bt.did_split}")print(f"global ranking (top 3): {', '.join(bt.global_ranking()[:3])}")print(bt.summary())else: bt =Noneprint("R with psychotree not available; skipping the Bradley-Terry tree.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")```[`plackett_luce`](../../docs/explanations/method-by-dataset-heterogeneity.md#plackett-luce-full-rankings) fits one full-ranking model over the methods.```{python}from beam.heterogeneity import plackett_luce, plackett_luce_availableif plackett_luce_available(): pl = plackett_luce(agreement, head, polarity="higher_is_better")print("Plackett-Luce ranking (first to last):", ", ".join(pl.ranking()))print("top method:", pl.top_tool())else:print("R with PlackettLuce not available; skipping.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")```## Manual annotationThe agreement metric is defined against the manual annotation. The cell types with a method consensus on a broad lineage different from the manual label:```{python}import csvimport collectionsfrom importlib import resourcestext = resources.files("beam.data").joinpath("gptcelltype2024_agreement.csv").read_text("utf-8")ann_rows = [r for r in csv.DictReader(text.splitlines()) if r["method"] in head]by_cell = collections.defaultdict(list)for r in ann_rows:if r["broadtype"]: by_cell[(r["dataset"], r["cell_type"], r["manual_broadtype"])].append(r["broadtype"])candidates = []for (dataset, cell_type, manual), predicted in by_cell.items():iflen(predicted) <3:continue consensus, n = collections.Counter(predicted).most_common(1)[0] manual_lineages = {x.strip() for x in manual.split(",")}if n /len(predicted) >=0.8and consensus notin manual_lineages: candidates.append((dataset, cell_type, manual, consensus, f"{n}/{len(predicted)}"))eligible =sum(1for v in by_cell.values() iflen(v) >=3)print(f"cell types where the methods agree on a lineage that differs from the manual one:")print(f" {len(candidates)} of {eligible} cell types with at least three running methods")print("\nexamples (dataset, cell type, manual lineage, method consensus, votes):")for dataset, cell_type, manual, consensus, votes in candidates[:8]:print(f" {dataset:22s}{cell_type:26s}{manual:18s} -> {consensus} ({votes})")```In most of these the manual label is a fine type (innate lymphoid cell, platelet, capillary) and the methods agree on its parent lineage (t/nk cell, megakaryocyte, endothelial cell).## Partial coverageThe complete cases leave out the 18 datasets where SingleR and ScType were not run. [Available-case SAW](../../docs/explanations/missing-data.md) scores each method on the datasets where it was measured, with the weights renormalized. The [Skillings-Mack test](../../docs/explanations/critical-difference.md#skillings-mack-for-incomplete-blocks) is the Friedman-type test for an incomplete matrix.```{python}from beam.mcda import skillings_mackfrom beam.reporting import funky_heatmap_from_runfull_scores = beam.Scores( values=sub, tool_names=tuple(head), metric_ids=tuple(metrics), dataset_names=tuple(g.dataset_names), layout="long",)full_run = beam.rank( full_scores, weights="equal", method="saw", sensitivity=True, missing="available")print("available-case ranking on all 54 datasets:")for i in np.argsort(full_run.result.ranks):print(f" {head[i]:14s} rank {full_run.result.ranks[i]}")sm = skillings_mack(sub[:, :, 0], higher_is_better=True, method_names=head)print(f"\nSkillings-Mack statistic: {sm.statistic:.1f} p-value: {sm.p_value:.2e}")```The [funky heatmap](../../docs/explanations/funky-heatmaps-and-robustness.md) of the available-case run on all 54 datasets:```{python}funky_heatmap_from_run(full_run, title="Cell type annotation: available-case ranking on 54 datasets")plt.show()```## Other LLM annotation benchmarksThree more recent benchmarks score LLMs at cell type annotation: DeepCellSeek (Briefings in Bioinformatics 2025, [10.1093/bib/bbaf677](https://doi.org/10.1093/bib/bbaf677)), AnnDictionary (Nature Communications 2025, [10.1038/s41467-025-64511-x](https://doi.org/10.1038/s41467-025-64511-x)) and Single-Cell Omics Arena (SOAR, [arXiv:2412.02915](https://arxiv.org/abs/2412.02915)). They use different models, datasets and metrics, so they are compared by the within-benchmark rank of the shared methods. The three classical tools (CellMarker2.0, SingleR, ScType) are in GPTCelltype, DeepCellSeek and SOAR; GPT-4o is in DeepCellSeek, AnnDictionary and SOAR. The provenance and license of each bundled table are in `src/beam/data/`; DeepCellSeek and SOAR scores come from a pinned source, and the SOAR and AnnDictionary folders include a note on their license status.```{python}import csvfrom importlib import resourcesfrom beam.datasets import load_deepcellseekCLASSICAL = {"CellMarker2.0", "SingleR", "ScType", "Cell2Sentence"}def ranking(scores_by_method): order =sorted(scores_by_method, key=lambda m: -scores_by_method[m])return {m: i +1for i, m inenumerate(order)}benchmarks = {}benchmarks["GPTCelltype"] = {m: float(np.nanmean(g.scores[i, :, 0]))for i, m inenumerate(g.method_names)}dcs = load_deepcellseek()benchmarks["DeepCellSeek"] = {m: float(np.nanmean(dcs.scores[i, :, 0]))for i, m inenumerate(dcs.method_names)}ad = {r["model"]: float(r["overall_binary_celltypes_mean"]) for r in csv.DictReader( resources.files("beam.data").joinpath("anndictionary/anndictionary_celltype_agreement.csv").read_text().splitlines())}benchmarks["AnnDictionary"] = adsoar = {r["model"]: float(r["R-L"]) for r in csv.DictReader( resources.files("beam.data").joinpath("soar/soar_readme_scores.csv").read_text().splitlines())if r["table_name"] =="SOAR-RNA Zero-shot"and r["R-L"]}benchmarks["SOAR"] = soarfor name, sc in benchmarks.items(): rk = ranking(sc) n =len(sc) classical = [m for m in sc if m in CLASSICAL]if classical: top =min(rk[m] for m in classical)print(f"{name:13s}{n:2d} methods; top-ranked classical ranks {top}/{n}; metric: "f"{'ontology agreement'if name in ('GPTCelltype','DeepCellSeek') else'NLP overlap (ROUGE-L)'}")else:print(f"{name:13s}{n:2d} methods; LLM-only (no classical comparator)")``````{python}shared_methods = ["GPT-4o", "GPT-4", "CellMarker2.0", "SingleR", "ScType"]bench_names =list(benchmarks)grid = np.full((len(shared_methods), len(bench_names)), np.nan)for j, name inenumerate(bench_names): rk = ranking(benchmarks[name]) n =len(benchmarks[name])for i, m inenumerate(shared_methods):if m in rk: grid[i, j] = rk[m] / n # normalized rank, 0 first, 1 lastfig, ax = plt.subplots(figsize=(6.0, 3.2))im = ax.imshow(grid, cmap="RdYlGn_r", aspect="auto", vmin=0, vmax=1)ax.set_xticks(range(len(bench_names))); ax.set_xticklabels(bench_names, rotation=20, ha="right")ax.set_yticks(range(len(shared_methods))); ax.set_yticklabels(shared_methods)ax.set_xlabel("benchmark"); ax.set_ylabel("shared method")ax.set_title("Normalized rank of the shared methods (0 = first, 1 = last)")for i inrange(len(shared_methods)):for j inrange(len(bench_names)):ifnot np.isnan(grid[i, j]): ax.text(j, i, f"{grid[i, j]:.2f}", ha="center", va="center", fontsize=8)fig.colorbar(im, ax=ax, fraction=0.05, label="normalized rank")fig.tight_layout()plt.show()```A [network meta-analysis](../../docs/explanations/network-meta-analysis.md) is not possible: the AnnDictionary and SOAR tables are aggregates, without scores per dataset.### Variance on the shared classical methodsGPTCelltype and DeepCellSeek score the three classical tools per dataset on the same metric. [`source_variance_decomposition`](../../docs/reference/source_variance_decomposition.qmd) fits `score ~ method + (1 | benchmark) + (1 | benchmark:dataset) + (1 | method:benchmark)` in lme4 over the two benchmarks and the three shared methods. It needs R.```{python}from beam.heterogeneity import source_variance_decomposition, r_availableshared = ["CellMarker2.0", "SingleR", "ScType"]sv_methods, sv_datasets, sv_benchmarks, sv_scores = [], [], [], []for bname, b in [("GPTCelltype", g), ("DeepCellSeek", dcs)]:for m in shared: i = b.method_names.index(m)for j, ds inenumerate(b.dataset_names): v = b.scores[i, j, 0]ifnot np.isnan(v): sv_methods.append(m) sv_datasets.append(ds) sv_benchmarks.append(bname) sv_scores.append(float(v))if r_available(): sv = source_variance_decomposition(sv_methods, sv_datasets, sv_benchmarks, sv_scores) total =sum(v for v in sv.variance_components.values() ifnot np.isnan(v))print(f"{len(sv_scores)} observations, 2 benchmarks, 3 shared methods")for k, v in sv.variance_components.items():ifnot np.isnan(v):print(f" {k:18s}{v / total:.2f} of the score variance")print("pooled method effect:", {n: round(e, 3) for n, e inzip(sv.method_names, sv.method_effects)})else: sv =Noneprint("R with lme4 not available; skipping the variance decomposition.")print("Provision it with envs/heterogeneity.yml (conda or mamba).")``````{python}if sv isnotNone: comps = {k: v for k, v in sv.variance_components.items() ifnot np.isnan(v)} total =sum(comps.values()) fig, axes = plt.subplots(1, 2, figsize=(9.0, 3.2)) axes[0].bar(list(comps), [comps[k] / total *100for k in comps], color="tab:purple") axes[0].set_ylabel("share of score variance (percent)") axes[0].set_xlabel("source") axes[0].set_title("Variance on the three classical methods") axes[0].tick_params(axis="x", rotation=30) x = np.arange(len(shared)) gm = [np.nanmean(g.scores[g.method_names.index(m), :, 0]) for m in shared] dm = [np.nanmean(dcs.scores[dcs.method_names.index(m), :, 0]) for m in shared] axes[1].bar(x -0.2, gm, 0.4, label="GPTCelltype") axes[1].bar(x +0.2, dm, 0.4, label="DeepCellSeek") axes[1].set_xticks(x) axes[1].set_xticklabels(shared, rotation=20, ha="right") axes[1].set_ylabel("mean agreement") axes[1].set_title("Classical methods per benchmark") axes[1].legend() fig.tight_layout() plt.show()```## SummaryOn the 36 datasets where all five methods were run, GPT-4 ranks first on the mean agreement and the full match rate with equal weights and SAW. On the critical-difference diagram of the agreement score GPT-4 and CellMarker2.0 are each separate from the other methods, and SingleR, ScType and GPT-3.5 are in one clique.