Measurement theory in beam
Every metric card declares a scale_type and a polarity.
Stevens scales
Stevens (1946) orders four measurement scales by the operations they allow:
- Nominal: labels without order, such as a cell type. Only equality is meaningful.
- Ordinal: ordered labels, such as ranks. Comparison is meaningful; differences and ratios are not.
- Interval: numeric, with a unit but no true zero, such as temperature in Celsius. Differences are meaningful; ratios are not.
- Ratio: numeric, with a unit and a true zero, such as runtime in seconds. Ratios are meaningful.
The Adjusted Rand Index (ARI) is interval: its zero is chance-corrected agreement, but its unit depends on the partition pair. Runtime is ratio.
Relevance in benchmarking
Multi-criteria decision analysis combines the metrics into one ranking, and not every operation is allowed on every scale:
- Arithmetic mean: interval and ratio scales.
- Geometric mean: ratio scales with positive values.
- Rank aggregation (Borda, Copeland): any ordered scale.
- Min-max normalization: at least interval.
Velleman and Wilkinson (1993) argue against a rigid use of the Stevens scales. Each card has a free-text scale_rationale for such cases. The polarity (higher_is_better, lower_is_better, target_value) orients normalization and ranking.
References
- Stevens, S. S. (1946). On the theory of scales of measurement. Science, 103(2684), 677-680. DOI 10.1126/science.103.2684.677.
- Velleman, P. F., and Wilkinson, L. (1993). Nominal, ordinal, interval, and ratio typologies are misleading. The American Statistician, 47(1), 65-72. DOI 10.1080/00031305.1993.10475938.