Add a new metric card

Each metric card is one YAML file under src/beam/metrics/<id>/v1.yaml. The card has the metadata the pipeline uses to normalize, weight and aggregate the metric: polarity (higher or lower better), scale type, range, allowed transformations, a recommended normalization, and the ontology mappings (STATO, UO, OBI, HuggingFace evaluate) where an external term exists.

1. Pick the id and the version

The id is a short lowercase string that becomes the column name in score CSVs. The version is a simple v1, v2 and so on:

mkdir -p src/beam/metrics/recall_at_k

2. Write v1.yaml

A card for a retrieval metric. Every required field is set. citations and recommended_normalization are optional, as are score_of_random_baseline and noise_floor; accuracy/v1.yaml shows other optional fields:

id: recall_at_k
version: v1
name: Recall at k
description: |
  Fraction of the relevant items that appear in the top k of a ranked
  retrieval list, averaged over queries.
citations:
  - text: Manning, Raghavan and Schuetze. Introduction to Information Retrieval. Cambridge University Press 2008.
metric_kind: derived
measurand: fraction of relevant items retrieved in the top k
task:
  - information_retrieval
requires_ground_truth: true
ground_truth:
  description: relevant items per query
  shape: set
  dtype: item_id
inputs:
  - role: ranked_items
    shape: vector
    dtype: item_id
  - role: relevant_items
    shape: set
    dtype: item_id
output:
  shape: scalar
  dtype: float
semantics:
  scale_type: ratio
  range:
    lower: 0
    upper: 1
    lower_inclusive: true
    upper_inclusive: true
  polarity: higher_is_better
  monotonic: true
  meaningful_zero: true
  allowed_transformations:
    - affine
    - rank
comparability:
  comparable_within:
    - same_dataset
  recommended_aggregation_across_datasets: arithmetic_mean
  recommended_normalization: min_max
implementations:
  - name: torchmetrics
    language: python
    package: torchmetrics
    function: torchmetrics.retrieval.RetrievalRecall
    license: Apache-2.0
examples:
  - description: one of two relevant items in the top 2
    inputs:
      ranked_items: [a, b, c, d]
      relevant_items: [a, d]
      k: 2
    expected_output: 0.5
    tolerance: 1.0e-12
provenance:
  author: Your Name
  contact: you@example.org
  created: "2026-05-28"
  license: CC-BY-4.0

3. Validate

.venv/bin/python -m pytest tests/test_schema.py -q

The schema check runs against every card in the registry. A missing required field or a polarity outside the polarity enum raises an error. An inverted range (lower > upper) passes the schema check. beam.rank then stops with an error, because every score is outside such a range and the normalization step raises on the first one. To see the cause, call the card-data consistency audit directly on the scores; it reports the inverted range as a malformed_range violation.

4. Optionally add a STATO or UO mapping

A metric with a term in the Statistics Ontology, the Units of Measurement Ontology, the Ontology for Biomedical Investigations, or the HuggingFace evaluate catalogue takes the full IRI under mappings. scripts/ols_query.py searches OLS for candidate IRIs, and scripts/ols_verify.py confirms a candidate is the right term and not obsolete. Only existing IRIs are used.

The cards-and-pipeline page lists which fields the pipeline reads and which are reserved for documentation. See ../explanations/cards-and-pipeline.qmd.

5. Regenerate the OWL release artefact

.venv/bin/python -m beam.owl.generate

This rewrites docs/beam.owl.ttl from the cards plus the schema. The new card appears as an instance of the beam metric class, and also as an instance of its STATO class when the card has a STATO mapping.

6. Add a unit test for the metric (optional)

For a card with implementations, add a small test under tests/ that checks the implementation returns the expected output on a documented input.