skillings_mack
mcda.skillings_mack
Skillings-Mack (1981) test: coverage-aware Friedman for incomplete blocks.
The Friedman test in beam.mcda.cd refuses any tool by dataset matrix with missing cells, because the per-dataset ranks 1..k only make sense over a complete column. Skillings and Mack (1981) generalize the Friedman statistic to unbalanced block designs by ranking within each block over only the methods that are present, standardizing each within-block rank deviation by the block size, and assembling a chi-squared form from the resulting per-method sums. The test gives a global “are the methods separable” answer on a partial matrix without imputing the missing scores.
Reference: Skillings JH, Mack GA. On the use of a Friedman-type statistic in balanced and unbalanced block designs. Technometrics 1981, 23(2):171-177. DOI 10.1080/00401706.1981.10486261.
Classes
| Name | Description |
|---|---|
| SkillingsMackReport | Outcome of a Skillings-Mack test on a tool by dataset matrix with NaN. |
SkillingsMackReport
mcda.skillings_mack.SkillingsMackReport(self, statistic, df, p_value, adjusted_rank_sums, coverage, n_methods, n_blocks, method_names=None, nemenyi_cliques=None, note='Skillings-Mack is a global test on incomplete blocks. The Nemenyi pairwise post-hoc needs a complete matrix; restrict the comparison to the block of methods and datasets where all of them ran.')
Outcome of a Skillings-Mack test on a tool by dataset matrix with NaN.
Fields: statistic: chi-squared test statistic. df: degrees of freedom, n_methods - 1. p_value: tail probability under the null that the methods rank equally. adjusted_rank_sums: length n_methods array of the centered standardized per-method statistics A_i. They sum to zero by construction; a large absolute value flags a method whose within-block ranks systematically deviate from the block centre. coverage: length n_methods integer array recording the number of blocks (columns with at least two methods present) each method appears in. n_methods, n_blocks: matrix shape. method_names: optional labels carried for reporting. nemenyi_cliques: always None. The Nemenyi post-hoc needs a complete matrix and is not generalized here; the field exists to mirror :class:beam.mcda.cd.CriticalDifferenceReport for callers that branch on the test’s output. note: a short message explaining the lack of pairwise post-hoc and pointing to the complete-case block.
Functions
| Name | Description |
|---|---|
| coverage_aware_critical_difference | Convenience wrapper around :func:skillings_mack for the CD use case. |
| skillings_mack | Run the Skillings-Mack test on a tool by dataset matrix that may have NaN. |
coverage_aware_critical_difference
mcda.skillings_mack.coverage_aware_critical_difference(scores, higher_is_better=True, method_names=None)
Convenience wrapper around :func:skillings_mack for the CD use case.
A drop-in for :func:beam.mcda.cd.critical_difference when the matrix has missing cells. Returns a :class:SkillingsMackReport with the global chi-squared test only; nemenyi_cliques is None and the note field explains that the pairwise post-hoc needs a complete matrix. On a matrix with no NaN this gives the same chi-squared statistic as :func:critical_difference, so the global “are the methods separable” answer is unchanged; only the cliques are missing.
skillings_mack
mcda.skillings_mack.skillings_mack(scores, higher_is_better=True, method_names=None)
Run the Skillings-Mack test on a tool by dataset matrix that may have NaN.
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
scores |
np.ndarray | 2D array of shape (n_methods, n_blocks). NaN marks a method that did not run on a block. Blocks with fewer than two observed methods contribute nothing. |
required |
higher_is_better |
bool | Whether a larger score is better. When False the ranking is reversed inside each block so the better method earns the higher within-block rank. The chi-squared statistic itself does not depend on the direction. |
True |
method_names |
Sequence[str] | None | Optional length n_methods labels, carried in the report. |
None |
Returns
| Type | Description |
|---|---|
| SkillingsMackReport |
Notes
Within each block j with k_j >= 2 observed methods, the present scores are ranked from 1 (lowest) to k_j (highest) with ties averaged. For each method i, the per-block contribution is
``(R_ij - (k_j + 1) / 2) * sqrt(12 / (k_j + 1))``and A_i is the sum of these contributions over the blocks where method i is present. The covariance matrix of A is
``Sigma_ii = sum over blocks j containing i of (k_j - 1)``
``Sigma_ij = -(count of blocks containing both i and j)``The full Sigma is rank-deficient (every row sums to zero), so one row and column are dropped before inverting the reduced submatrix. The test statistic T = A_red.T @ Sigma_red^-1 @ A_red is chi-squared with n_methods - 1 degrees of freedom under the null that the methods rank identically. On a complete matrix the result equals the Friedman chi-squared statistic from :func:beam.mcda.cd.critical_difference.