mixed_effects
heterogeneity.mixed_effects
Mixed-effects variance decomposition on benchmark scores.
Following Eugster, Hothorn and Leisch (2008), benchmark results for one metric are modelled as a mixed-effects model with the method as a fixed effect and the dataset as a random effect:
score ~ method + (1 | dataset)The method fixed effects give a global ranking that adjusts for the fact that some datasets are harder than others (a shift that lifts or lowers every method). The variance components split the score variation into a between-dataset part (the random dataset intercept) and a residual part. With one observation per (method, dataset) cell the residual is the method-by-dataset interaction confounded with measurement noise: the two cannot be separated without replicates. When the input does carry replicates (a multi-run benchmark, several runs of the same method on the same dataset), the richer model
score ~ method + (1 | dataset) + (1 | dataset:method)fits the interaction as its own variance component, and its share of the total is the formal answer to “how much of the apparent ranking is dataset-dependent”. This is the diagnostic counterpart to beam.mcda.leave_one_dataset_out: the leave-one-out check asks whether the pooled ranking hangs on any single dataset; this asks how much of the score variance lives in the interaction at all.
The model is fit by R’s lme4 in a one-shot subprocess. The score values are taken as supplied for a single metric. Polarity does not enter a variance decomposition, but scale does, so do not mix metrics: pass one metric’s scores per call. lme4 uses a Gaussian likelihood; for a bounded metric this is an approximation, and glmmTMB with a beta family is the documented future extension.
Classes
| Name | Description |
|---|---|
| MixedEffectsReport | Outcome of a mixed-effects variance decomposition on one metric. |
MixedEffectsReport
heterogeneity.mixed_effects.MixedEffectsReport(self, method_names, method_effects, method_effect_se, variance_components, residuals, residual_methods, residual_datasets, formula, formula_kind, has_replicates, singular, n_obs, n_methods, n_datasets, loglik, aic, warnings, engine='lmer', scale='response')
Outcome of a mixed-effects variance decomposition on one metric.
Attributes
| Name | Type | Description |
|---|---|---|
| method_names | tuple[str, …] | Method labels in the order the effect estimates are reported (the sorted factor levels lme4 used, not the input order). |
| method_effects | np.ndarray | Estimated marginal mean per method over datasets, aligned with method_names. Higher means a higher score on the metric as supplied, regardless of the metric’s polarity. |
| method_effect_se | np.ndarray | Standard error of each marginal mean, aligned with method_names. |
| variance_components | dict[str, float] | Map from grouping factor to variance: "dataset", optionally "dataset:method" (only in the interaction model), and "Residual". |
| residuals | np.ndarray | Per-observation residual, aligned with residual_methods and residual_datasets (the rows fed to the model, with NaN scores dropped). |
| residual_methods, residual_datasets | Method and dataset label per residual row. | |
| formula | str | The R model formula that was fit. |
| formula_kind | str | "main" or "interaction", after resolving "auto". |
| has_replicates | bool | True when at least one (dataset, method) cell held more than one observation, so the interaction model is identifiable. |
| singular | bool | lme4’s singular-fit flag. A singular fit usually means a variance component collapsed to its boundary (often zero) and the estimate should be read with caution. |
| n_obs, n_methods, n_datasets | Observation and factor-level counts after dropping NaN scores. | |
| loglik, aic | Model fit statistics. | |
| warnings | tuple[str, …] | Convergence or other warnings raised by lme4 during the fit. |
Methods
| Name | Description |
|---|---|
| top_outliers | Return the k (method, dataset, residual) cells with the largest absolute residual. |
top_outliers
heterogeneity.mixed_effects.MixedEffectsReport.top_outliers(k=10)
Return the k (method, dataset, residual) cells with the largest absolute residual.
These are the cells where a method departs most from what its global effect predicts on that dataset, the strongest single signals of method-by-dataset interaction.
Functions
| Name | Description |
|---|---|
| mixed_effects | Fit a mixed-effects model on one metric’s scores and decompose its variance. |
| mixed_effects_from_matrix | Fit the mixed-effects model from a method by dataset score matrix for one metric. |
| r_available | Return True when Rscript and the lme4 and jsonlite packages are present. |
mixed_effects
heterogeneity.mixed_effects.mixed_effects(methods, datasets, scores, formula_kind='auto', engine='lmer', family=None)
Fit a mixed-effects model on one metric’s scores and decompose its variance.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
methods |
Sequence[str] | Three parallel sequences, one entry per observation: the method label, the dataset label, and the metric score. Rows with a NaN score are dropped before fitting. Several rows sharing a (method, dataset) pair are replicates and enable the interaction model. | required |
datasets |
Sequence[str] | Three parallel sequences, one entry per observation: the method label, the dataset label, and the metric score. Rows with a NaN score are dropped before fitting. Several rows sharing a (method, dataset) pair are replicates and enable the interaction model. | required |
scores |
Sequence[str] | Three parallel sequences, one entry per observation: the method label, the dataset label, and the metric score. Rows with a NaN score are dropped before fitting. Several rows sharing a (method, dataset) pair are replicates and enable the interaction model. | required |
formula_kind |
str | "auto" (default) fits the interaction model when the input has replicates and the main-effects model otherwise. "main" forces score ~ method + (1 \| dataset). "interaction" forces score ~ method + (1 \| dataset) + (1 \| dataset:method), which is only identifiable with replicates and otherwise yields a singular fit. |
'auto' |
engine |
str | "lmer" (default) fits a Gaussian linear mixed model in lme4. "glmmtmb" fits the same structure in glmmTMB, which allows a non-Gaussian family for a bounded metric. The variance components and marginal means it reports are then on the model’s link scale, not the response scale, so they are not directly comparable to the lmer numbers; the method ordering is comparable. |
'lmer' |
family |
str | None | Only used when engine="glmmtmb". None (default) resolves to "beta" when every score lies strictly in (0, 1) and "gaussian" otherwise. Pass "beta" or "gaussian" to force one. The beta family models a metric bounded in (0, 1) such as ARI; scores exactly at 0 or 1 are squeezed inside the open interval before the fit. |
None |
Returns
| Type | Description |
|---|---|
| MixedEffectsReport | With engine set to the engine used and scale set to "response" for the Gaussian fits or "link" for a beta fit. |
Raises
| Type | Description |
|---|---|
| ValueError | If the three sequences differ in length, fewer than two methods or two datasets remain after dropping NaN scores, or engine or family is not recognised. |
| RNotAvailableError | If the R toolchain for the chosen engine is not available. |
| RExecutionError | If the R subprocess fails. |
mixed_effects_from_matrix
heterogeneity.mixed_effects.mixed_effects_from_matrix(matrix, method_names, dataset_names, formula_kind='auto', engine='lmer', family=None)
Fit the mixed-effects model from a method by dataset score matrix for one metric.
Convenience wrapper that flattens a 2D matrix into the long-format sequences mixed_effects expects. NaN cells are carried through and dropped by mixed_effects.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
matrix |
Array-like of shape (n_methods, n_datasets) holding one metric’s scores. |
required | |
method_names |
Sequence[str] | Length n_methods row labels. |
required |
dataset_names |
Sequence[str] | Length n_datasets column labels. |
required |
formula_kind |
str | Forwarded to mixed_effects. A single matrix has one observation per cell, so only "auto" (resolving to main) or "main" are meaningful here. |
'auto' |
Returns
| Type | Description |
|---|---|
| MixedEffectsReport |
r_available
heterogeneity.mixed_effects.r_available()
Return True when Rscript and the lme4 and jsonlite packages are present.
Tests and vignettes use this to skip the analysis cleanly on a machine without the R toolchain.