Ontology mappings on metric cards
Each metric card has an optional mappings: block that links the metric to external ontologies and registries. It is filled where an upstream term exists.
Ontologies
- STATO, the Statistics Ontology (https://stato-ontology.org): estimators, test statistics, distributions, study designs.
mappings.stato: http://purl.obolibrary.org/obo/STATO_NNNNNNN. - UO, the Units of Measurement Ontology (http://purl.obolibrary.org/obo/uo.owl): runtime in seconds, peak memory in bytes, speed in kilometer per hour, co2 in grams. UO has no monetary units, so
costhas no mapping. - OBI, the Ontology for Biomedical Investigations (https://obi-ontology.org): the assay of the scIB metrics, single-cell RNA sequencing (
OBI_0002631), and principal component regression for pcr (OBI_0200104). - HuggingFace evaluate, a metric card registry:
mappings.huggingface_evaluatehas the URL of the card directory for accuracy, F1, SMAPE, MASE and Spearman correlation. beam does not depend on the library.
Mapping a new card
- Search the EBI Ontology Lookup Service (OLS) in STATO, then UO, then OBI:
https://www.ebi.ac.uk/ols4/api/search?q=<query>&ontology=<slug>.scripts/ols_query.pydoes this for the registry. - Fetch each candidate,
https://www.ebi.ac.uk/ols4/api/ontologies/<slug>/terms/<double-url-encoded-iri>, and check the label and thatis_obsoleteis false.scripts/ols_verify.pyruns this check. - Write the full IRI under
mappings:. The schema types each value as a URI. - Without a term, leave the key absent and add a one-line YAML comment below the block. beam-private IRIs are not minted. STATO takes proposals at https://github.com/ISA-tools/stato/issues.
- Run
python -m pytest tests/test_schema.py -q. - Regenerate the OWL:
python -m beam.owl.generate.
Coverage
| metric_id | stato | uo | obi | huggingface_evaluate |
|---|---|---|---|---|
| accuracy | STATO_0000415 | not in uo | not in obi | metrics/accuracy |
| ari | STATO_0000593 | not in uo | not in obi | not in hf |
| asw_batch | not in stato | not in uo | OBI_0002631 | not in hf |
| asw_label | not in stato | not in uo | OBI_0002631 | not in hf |
| calibration_slope | STATO_0000687 | not in uo | not in obi | not in hf |
| cell_cycle_conservation | not in stato | not in uo | OBI_0002631 | not in hf |
| cell_type_annotation_agreement | not in stato | not in uo | not in obi | not in hf |
| cell_type_annotation_full_match_rate | not in stato | not in uo | not in obi | not in hf |
| clisi | not in stato | not in uo | OBI_0002631 | not in hf |
| co2 | not in stato | UO_0000021 | not in obi | not in hf |
| correlation | STATO_0000201 | not in uo | not in obi | metrics/spearmanr |
| cost | not in stato | not in uo | not in obi | not in hf |
| f1_score | STATO_0000628 | not in uo | not in obi | metrics/f1 |
| graph_connectivity | not in stato | not in uo | OBI_0002631 | not in hf |
| hvg_overlap | not in stato | not in uo | OBI_0002631 | not in hf |
| ilisi | not in stato | not in uo | OBI_0002631 | not in hf |
| isolated_label_asw | not in stato | not in uo | OBI_0002631 | not in hf |
| isolated_label_f1 | STATO_0000628 | not in uo | OBI_0002631 | not in hf |
| kbet | not in stato | not in uo | OBI_0002631 | not in hf |
| mase | not in stato | not in uo | not in obi | metrics/mase |
| nclust_deviation | not in stato | not in uo | not in obi | not in hf |
| nmi | not in stato | not in uo | not in obi | not in hf |
| pcr | not in stato | not in uo | OBI_0200104 | not in hf |
| peak_memory | not in stato | UO_0000233 | not in obi | not in hf |
| runtime | not in stato | UO_0000010 | not in obi | not in hf |
| shannon_entropy_diff | not in stato | not in uo | not in obi | not in hf |
| silhouette | not in stato | not in uo | not in obi | not in hf |
| smape | not in stato | not in uo | not in obi | metrics/smape |
| speed | not in stato | UO_0010008 | not in obi | not in hf |
Summary: STATO covers 6 of 29 cards (ari, accuracy, f1_score, isolated_label_f1, calibration_slope, correlation). UO covers 4 of 29 (runtime, peak_memory, speed, co2). OBI covers 11 of 29 (the scIB family with OBI_0002631 plus pcr with OBI_0200104). HuggingFace evaluate covers 5 of 29 (accuracy, f1_score, smape, mase, correlation).
OWL
src/beam/owl/generate.py reads the cards, builds an rdflib graph, and writes docs/beam.owl.ttl. Each card is a beam:Metric instance. A card with mappings.stato is also an instance of the STATO class (rdf:type and owl:sameAs). UO, OBI, QUDT, OM2 and HuggingFace mappings are rdfs:seeAlso.
SKOS vocabulary
Four card fields are closed enumerations: polarity, scale_type, allowed_transformations and comparability.recommended_normalization. src/beam/owl/skos.py writes them as SKOS concept schemes to docs/beam.skos.ttl, one skos:Concept per value with a prefLabel and a definition. skos:broader links the Stevens scales from nominal to ratio, and the normalizations that are special cases of an affine transform. The values are read from the JSON Schema. Regenerate with python -m beam.owl.skos.
Draft STATO proposals
Metrics without a STATO term, with the parent class and reference for an issue at the STATO tracker. Runtime, peak_memory and the transportation metrics (cost, speed, co2) are out of STATO scope.
- Normalized mutual information (nmi). Mutual information between two partitions divided by a function of their entropies, in the unit interval. Parent: measure of clustering agreement, sibling of the adjusted Rand index (STATO_0000593). Strehl and Ghosh 2002, JMLR 3:583-617.
- Silhouette coefficient (silhouette, asw_batch, asw_label, isolated_label_asw). Mean over points of the difference between the mean nearest-other-cluster distance and the mean within-cluster distance, divided by the larger. Parent: cluster validity index. Rousseeuw 1987, DOI 10.1016/0377-0427(87)90125-7.
- k-nearest-neighbour batch-effect test statistic (kbet). Chi-squared comparison of the batch composition of local k-neighbourhoods with the global composition; the rejection rate is reported. Parent: test statistic. Buttner et al. 2019, DOI 10.1038/s41592-018-0254-1.
- Local inverse Simpson index (clisi, ilisi). Inverse Simpson index of the label or batch composition in a perplexity-weighted neighbourhood. Parent: diversity index. Korsunsky et al. 2019, DOI 10.1038/s41592-019-0619-0.
- Shannon entropy of a partition (shannon_entropy_diff). Parent: information-theoretic measure. Shannon 1948, DOI 10.1002/j.1538-7305.1948.tb01338.x.
- Symmetric mean absolute percentage error (smape). Mean over horizons of the absolute forecast error divided by the average of the absolute actual and forecast values, in percent. Parent: measure of forecast error. Makridakis, Spiliotis and Assimakopoulos 2020, DOI 10.1016/j.ijforecast.2019.04.014.
- Mean absolute scaled error (mase). Mean absolute forecast error divided by the in-sample mean absolute error of a naive one-step forecast. Parent: measure of forecast error. Hyndman and Koehler 2006, DOI 10.1016/j.ijforecast.2006.03.001.