beam
Benchmark Evaluation and Metrics
beam ranks methods from the score tables of a benchmark. Each performance metric has a metric card with its scale type, direction, range and allowed transformations, following measurement theory. Methods are ranked with multi-criteria aggregations, and the ranking comes with diagnostics: how much it depends on the weighting, the aggregation and the datasets, whether the methods are separable, and whether one ranking fits all datasets (method by dataset heterogeneity).
The metric cards link to STATO, UO, OBI and HuggingFace evaluate terms where one exists. The OWL file, docs/beam.owl.ttl, is regenerated from the cards on each release.
Where to start
- Quick start. From a CSV to a ranking and an HTML report.
- How to run from beam.yaml. The declarative beam specification.
- Install and run in Python, R, or the CLI.
- Cards and pipeline. What a metric card encodes and how it is processed.
- Critical difference. Whether the methods are separable across datasets (Friedman, Nemenyi, Skillings-Mack).
Worked vignettes
- Duo 2018 clustering: scRNA-seq clustering with 14 methods on 12 datasets.
- Simulated scenarios: checks against a known answer.
- Transportation modes: an example outside bioinformatics, with missing scores.
- M4 forecasting: 25 methods on six frequency bands.
- OpenProblems batch integration: multi-criteria rankings next to the platform’s mean of scores, plus a Bradley-Terry tree on 50 spatial datasets.
- Cross-benchmark meta-analysis: five published scRNA-seq integration benchmarks of the same methods.
- LLM cell type annotation: GPT-4, GPT-3.5 and other language models next to marker and reference based annotation tools.
HTML reports for four of these comparisons:
Source
github.com/imallona/beam. GPL-3 code, CC-BY-4.0 metric cards.