Metric cards and the MCDA pipeline

A metric card declares fields for identity, kind, inputs, output, semantics, comparability, implementations, examples, and provenance. See measurement theory and normalization and scales for the scale types and the normalization strategies.

Fields read

  • polarity: normalize inverts lower_is_better columns and rescales each column to [0, 1], higher is better.
  • range_lower, range_upper: when both are declared, run_from_registry passes them to normalize, and min-max and baseline-relative use them instead of the observed extrema. A value outside the declared range raises an error.
  • comparability.recommended_normalization: the strategy for the column, by default min_max. The options are min_max, log_min_max, rank, zscore, baseline_relative and target_relative.
  • semantics.score_of_random_baseline: the chance-level value, used by baseline_relative.
  • semantics.target: the ideal value, used by target_relative.
  • scale_type: validate_for_aggregation refuses nominal and ordinal columns for every aggregation.
  • allowed_transformations: validate_for_aggregation checks that the card allows the transform of the strategy. Min-max, baseline-relative and target-relative need affine or min_max; log_min_max needs log; rank needs rank; zscore needs z_score or affine.
  • comparability.recommended_aggregation_across_datasets: aggregate_across_datasets uses it to reduce a tool by dataset matrix to one value per tool. Runtime and peak memory declare geometric_mean (Smith 1988, 10.1145/63039.63043); bounded metrics declare arithmetic_mean.

For a min-max column run_from_registry warns when a declared bound is missing or the column is heavy-tailed. The warnings are in the Result and the run continues.

Fields declared but not enforced

  • meaningful_zero and monotonic: no reader.
  • uncertainty_model: uncertainty is not propagated through aggregation.
  • comparability.comparable_within and aggregation_rules: notes for readers.