beam

Benchmark Evaluation and Metrics

beam ranks methods from the score tables of a benchmark. Each performance metric has a metric card with its scale type, direction, range and allowed transformations, following measurement theory. Methods are ranked with multi-criteria aggregations, and the ranking comes with diagnostics: how much it depends on the weighting, the aggregation and the datasets, whether the methods are separable, and whether one ranking fits all datasets (method by dataset heterogeneity).

The metric cards link to STATO, UO, OBI and HuggingFace evaluate terms where one exists. The OWL file, docs/beam.owl.ttl, is regenerated from the cards on each release.

Where to start

Worked vignettes

HTML reports for four of these comparisons:

Source

github.com/imallona/beam. GPL-3 code, CC-BY-4.0 metric cards.