TuiML Documentation

Parametric significance tests for comparing learning algorithms.

A parametric test assumes an explicit distributional form for the data. The tests here assume the quantity being tested (the per-fold or per-dataset score difference, or the within-group residual) is drawn from a normal distribution. When that assumption is reasonable these tests are the most powerful option available; when it fails -- heavy tails, a handful of datasets, accuracies saturating near 1.0, or an outlier dataset -- they become anti-conservative and report "significant" far too often. In that regime use the rank-based tests in nonparametric instead.

Contents:

paired_t_test            Student's paired t-test on matched scores
corrected_paired_t_test  Paired t-test with the Nadeau & Bengio variance
                         correction for resampled / cross-validated scores
one_way_anova            Omnibus F-test across k independent groups
PairedStats              Result container returned by the paired tests
SignificanceLevel        WIN / LOSS / TIE verdict enum

Notes

Choosing a test. Two algorithms scored on the same folds or the same datasets are paired -- use paired_t_test. If those scores come from resampling that reuses training data (k-fold CV, repeated random splits), the folds are not independent, the ordinary t-test's variance estimate is too small and its Type I error rate is badly inflated; use corrected_paired_t_test. For more than two algorithms an omnibus test comes first: one_way_anova for independent groups, or -- much more commonly in machine learning, where the same datasets are reused across algorithms -- friedman_test.

Multiplicity. Running every pairwise t-test over k algorithms performs $k(k-1)/2$ tests, so the probability of at least one false positive grows quickly. Always feed the resulting p-values through one of the procedures in corrections.

References

Student1908

Student (W. S. Gosset) (1908). "The Probable Error of a Mean". Biometrika, 6(1), 1-25.

Dietterich1998

Dietterich, T. G. (1998). "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms". Neural Computation, 10(7), 1895-1923.

Demsar2006

Demsar, J. (2006). "Statistical Comparisons of Classifiers over Multiple Data Sets". Journal of Machine Learning Research, 7, 1-30.

Classes

SignificanceLevel

class evaluation.statistics.parametric.SignificanceLevel(Enum)

Verdict of a two-sided paired comparison between two algorithms.

A test does not simply return "different"/"not different": once the null hypothesis of equal performance is rejected, the sign of the mean difference says which algorithm won. This enum encodes both facts in one value, always from the point of view of the first argument (x) of the test.

Attributes

TIE

int

Value 0. The null hypothesis was not rejected (:math:`p \geq \alpha`). This is not evidence that the two algorithms are equal, only that the data are too few or too noisy to distinguish them.

WIN

int

Value 1. The null hypothesis was rejected and x is the better algorithm. "Better" is decided by the higher_better flag of the test: with higher_better=True (accuracy, F1) this means mean(x) > mean(y); with higher_better=False (error rate, RMSE) it means mean(x) < mean(y).

LOSS

int

Value -1. The null hypothesis was rejected and y is the better algorithm.

Notes

The comparison performed is always two-sided, so WIN and LOSS partition the same rejection region; the enum records which tail the observed difference fell into. Do not read a WIN at

\alpha = 0.05

as a one-sided claim at

\alpha = 0.025

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for the SignificanceLevel enum.

Returns

dict

JSON Schema describing the enum values.

PairedStats

class evaluation.statistics.parametric.PairedStats

Complete result of a paired comparison between two algorithms.

Returned by every paired test in this package -- paired_t_test, corrected_paired_t_test and wilcoxon_signed_rank_test -- so that the three are drop-in interchangeable. It bundles the descriptive statistics of both samples, the test statistic, the two-sided p-value and the WIN/LOSS/TIE verdict.

Attributes

x_mean

float

Sample mean of the first algorithm's scores.

y_mean

float

Sample mean of the second algorithm's scores.

x_std

float

Sample standard deviation of the first algorithm's scores, computed with ddof=1 (unbiased, divides by :math:`n - 1`).

y_std

float

Sample standard deviation of the second algorithm's scores (ddof=1).

diff_mean

float

Mean of the paired differences :math:`d_i = x_i - y_i`. This is the effect size in the original units of the score; its sign decides WIN vs LOSS. Note it is x_mean - y_mean, not the other way round.

diff_std

float

Spread of the paired differences. For paired_t_test and wilcoxon_signed_rank_test this is the ordinary ddof=1 standard deviation of :math:`d_i`. For corrected_paired_t_test it is instead the corrected standard error :math:`\sqrt{(1/n + n_{test}/n_{train})\, s_d^2}`, which is why the two are not comparable across tests.

t_statistic

float

The test statistic. For the two t-tests this is Student's :math:`t`, distributed with :math:`n - 1` degrees of freedom under the null. For the Wilcoxon test the field instead carries the normal approximation z-score of the signed-rank statistic (the name is kept only so the container stays uniform).

p_value

float

Two-sided p-value: the probability, assuming the null hypothesis of no difference is true, of observing a statistic at least as extreme as this one in either direction. It is not the probability that the null is true, and 1 - p_value is not the probability that the difference is real. Lies in :math:`[0, 1]`.

correlation

float

Pearson correlation between the two score vectors, in :math:`[-1, 1]`. High positive correlation (the usual case when both algorithms are scored on the same folds) is precisely what makes the paired design more powerful than an unpaired one, because the dataset-to-dataset variance cancels in :math:`d_i`. Reported as 0.0 when either sample is constant.

significance

SignificanceLevel

Verdict at the significance_level supplied to the test: WIN (x better), LOSS (y better) or TIE (null not rejected). See SignificanceLevel.

n

int

Number of paired observations actually used, after dropping pairs containing NaN. For the Wilcoxon test, pairs with a zero difference are dropped as well, so n can be smaller than the input length.

Notes

The verdict is computed with a strict comparison, p_value < significance_level; a p-value exactly equal to $\alpha$ is reported as TIE.

Because the whole object is derived from one significance_level, re-reading significance under a different $\alpha$ is invalid -- re-run the test, or threshold p_value yourself.

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for PairedStats dataclass fields.

Returns

dict

JSON Schema describing all dataclass fields.

is_significant (self) -> bool

Report whether the null hypothesis of no difference was rejected.

Returns

significant

bool

True if p_value fell below the significance level supplied to the test (equivalently, significance is not TIE). False means the evidence was insufficient -- not that the two algorithms perform equally.

x_better (self) -> bool

Report whether the first algorithm won significantly.

Returns

better

bool

True only if the difference was significant and pointed in x's favour, as judged by the higher_better flag of the test. False covers both "y won" and "tie".

y_better (self) -> bool

Report whether the second algorithm won significantly.

Returns

better

bool

True only if the difference was significant and pointed in y's favour. False covers both "x won" and "tie".

Functions

Func

paired_t_test

Line 465

paired_t_test(x: np.ndarray, y: np.ndarray, significance_level: float=0.05, higher_better: bool=True) -> PairedStats

Student's paired t-test comparing two algorithms on matched scores.

Tests whether the mean of the per-pair differences

d_i = x_i - y_i

differs from zero. Pairing means observation

i

of x and observation

i

of y must refer to the same fold, split or dataset; the shared difficulty of that fold then cancels out of

d_i

, which is what makes the paired design far more sensitive than comparing two independent samples.

Hypotheses

The test is two-sided:

$H_0: \mu_d = 0$ -- the two algorithms have the same expected

score, any observed gap is sampling noise.

$H_1: \mu_d \neq 0$ -- their expected scores differ, in either

direction.

Theory

With $\bar{d}$ the mean and $s_d$ the ddof=1 standard deviation of the $n$ differences, the statistic is

t = \frac{\bar{d}}{s_d / \sqrt{n}}

which follows a Student t-distribution with $\nu = n - 1$ degrees of freedom under $H_0$ . The two-sided p-value is $p = 2\,[1 - F_{\nu}(|t|)]$ .

Parameters

x

ndarray of shape (n,)

Scores of the first algorithm, one entry per fold or dataset (e.g. accuracies of model A).

y

ndarray of shape (n,)

Scores of the second algorithm, aligned element-wise with x. Must have the same length as x.

significance_level

float = 0.05

The :math:`\alpha` against which the p-value is thresholded to produce the WIN/LOSS/TIE verdict. It does not change the p-value itself.

higher_better

bool = True

Orientation of the score. True for accuracy, F1, AUC; False for error rate, RMSE, log-loss. Only affects which of WIN/LOSS is reported, never the statistic or the p-value.

Returns

stats

PairedStats

Full result: t_statistic, the two-sided p_value, the means and standard deviations of both samples, diff_mean, correlation, the significance verdict and the effective sample size n. Read stats.p_value < alpha or stats.is_significant() for the decision and stats.x_better() for its direction.

Raises

ValueError

If x and y have different lengths, if fewer than 2 observations are supplied, or if fewer than 2 complete pairs survive NaN removal.

Notes

Assumptions.

Paired data -- x[i] and y[i] measured on the same fold or

dataset. Comparing unrelated runs violates this and inflates significance.

The differences $d_i$ are approximately normally distributed.

The test is fairly robust for

n \gtrsim 30

, but with the 5-10 datasets typical of a machine-learning study a single outlier dominates both

\bar{d}

and

s_d

The differences are independent across $i$ . This is the

assumption that k-fold cross-validation breaks, because the training sets overlap; see corrected_paired_t_test.

Scores are on an interval scale and commensurable across datasets.

Averaging accuracy differences over heterogeneous datasets is exactly the practice [Demsar2006]_ argues against.

Complexity. $O(n)$ time and memory.

Handling of degenerate input. Pairs containing NaN in either vector are dropped. If every difference is identical (zero standard error), the test returns t = 0 and p = 1 rather than dividing by zero -- note this makes a perfect, perfectly consistent win look like a tie.

When to prefer something else. Use wilcoxon_signed_rank_test when normality is doubtful or $n$ is small -- [Demsar2006]_ recommends it as the default for comparing two classifiers over multiple datasets. Use corrected_paired_t_test for cross-validated or repeated-resampling scores. For three or more algorithms run an omnibus test first (friedman_test) rather than all pairwise t-tests.

References

Student1908

Student (W. S. Gosset) (1908). "The Probable Error of a Mean". Biometrika, 6(1), 1-25.

Dietterich1998

Dietterich, T. G. (1998). "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms". Neural Computation, 10(7), 1895-1923.

Demsar2006

Demsar, J. (2006). "Statistical Comparisons of Classifiers over Multiple Data Sets". Journal of Machine Learning Research, 7, 1-30.

corrected_paired_t_test

Line 682

corrected_paired_t_test(x: np.ndarray, y: np.ndarray, n_train: int, n_test: int, significance_level: float=0.05, higher_better: bool=True) -> PairedStats

Corrected resampled paired t-test for cross-validated scores.

The ordinary paired t-test assumes the

n

score differences are independent. Scores produced by k-fold cross-validation or repeated random subsampling are not: the training sets overlap, so the differences are positively correlated. The usual variance estimator is therefore too small,

|t|

is too large, and the test declares differences significant far more often than

\alpha

allows -- [Dietterich1998]_ measured Type I error rates several times the nominal level. This function applies the [NadeauBengio2003]_ correction, which inflates the variance estimate to account for that overlap.

Hypotheses

Identical to paired_t_test and still two-sided:

$H_0: \mu_d = 0$ -- both algorithms have the same expected

generalization performance.

$H_1: \mu_d \neq 0$ .

Only the standard error in the denominator changes.

Theory

With $s_d^2$ the ordinary ddof=1 variance of the differences, the corrected statistic replaces $s_d^2/n$ by

\widehat{\sigma}^2 = \left(\frac{1}{n} + \frac{n_{test}}{n_{train}}\right) s_d^2

giving

t = \frac{\bar{d}}{\sqrt{\widehat{\sigma}^2}}

compared against a t-distribution with $n - 1$ degrees of freedom. The extra $n_{test}/n_{train}$ term is the price of reusing training data; it never vanishes with more resampling rounds, which is why simply repeating cross-validation cannot buy unlimited significance.

Parameters

x

ndarray of shape (n,)

Per-fold scores of the first algorithm.

y

ndarray of shape (n,)

Per-fold scores of the second algorithm, aligned element-wise with x (same folds, same order).

n_train

int

Number of training examples in one resampling round. For 10-fold CV on 1000 examples this is 900.

n_test

int

Number of test examples in one resampling round. For 10-fold CV on 1000 examples this is 100. The ratio :math:`n_{test}/n_{train}` -- not the absolute sizes -- drives the correction; for k-fold CV it equals :math:`1/(k-1)`.

significance_level

float = 0.05

:math:`\alpha` used to turn the p-value into the WIN/LOSS/TIE verdict.

higher_better

bool = True

True when larger scores are better (accuracy), False for error metrics. Affects only the direction of the verdict.

Returns

stats

PairedStats

Same container as paired_t_test, with two differences worth noting: t_statistic and p_value are the corrected ones, and diff_std holds the corrected standard error :math:`\sqrt{\widehat{\sigma}^2}` rather than a standard deviation, so it is not comparable to the diff_std of the uncorrected test.

Raises

ValueError

If x and y have different lengths, or if fewer than 2 complete pairs survive NaN removal.

Notes

Assumptions.

Paired, element-wise aligned scores from a resampling scheme with a

constant train/test split ratio.

Approximately normal differences, as for the uncorrected test.
n_train and n_test describe a single round, not the totals

accumulated over all rounds. Passing the totals silently shrinks the correction toward nothing.

Complexity. $O(n)$ time and memory.

When to prefer it. Use this instead of paired_t_test whenever the scores come from k-fold CV, repeated k-fold CV, or 5x2 CV -- i.e. almost every model comparison run on a single dataset. If the $n$ scores are one-per-dataset over $n$ genuinely distinct datasets there is no training-set overlap and the plain paired t-test is appropriate. [BouckaertFrank2004]_ found 10x10-fold CV with this correction to give the best replicability among the schemes they compared.

Degenerate input. If the corrected variance is zero the function returns t = 0 and p = 1.

References

NadeauBengio2003

Nadeau, C., & Bengio, Y. (2003). "Inference for the Generalization Error". Machine Learning, 52(3), 239-281.

Dietterich1998

Dietterich, T. G. (1998). "Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms". Neural Computation, 10(7), 1895-1923.

BouckaertFrank2004

Bouckaert, R. R., & Frank, E. (2004). "Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms". Advances in Knowledge Discovery and Data Mining (PAKDD), LNCS 3056, 3-12.

one_way_anova

Line 904

one_way_anova(*groups, significance_level: float=0.05) -> Tuple[float, float, bool]

One-way ANOVA: omnibus F-test across :math:`k` independent groups.

Answers a single question -- "is any of these groups different from the others?" -- without saying which. It is the multi-group generalisation of the unpaired t-test, and the parametric analogue of friedman_test.

Hypotheses

$H_0: \mu_1 = \mu_2 = \dots = \mu_k$ -- every group has the

same population mean.

$H_1$ : at least one $\mu_i$ differs. Rejecting says nothing

about which group, or about how many; that needs a post-hoc test.

Theory

The total variability is split into a between-group and a within-group part. With $\bar{X}$ the grand mean, $n_i$ and $\bar{X}_i$ the size and mean of group $i$ , and $N = \sum_i n_i$ :

SS_B = \sum_{i=1}^{k} n_i (\bar{X}_i - \bar{X})^2, \quad SS_W = \sum_{i=1}^{k} \sum_{j=1}^{n_i} (X_{ij} - \bar{X}_i)^2

F = \frac{SS_B / (k - 1)}{SS_W / (N - k)}

Under $H_0$ , $F$ follows an F-distribution with $(k-1, N-k)$ degrees of freedom. Large $F$ means the group means are spread out relative to the noise within groups. The p-value is the upper tail $P(F_{k-1,\,N-k} > F)$ -- one-sided by construction, because only large $F$ contradicts $H_0$ , even though the underlying alternative is two-sided in each mean.

Parameters

*groups

array-like

Two or more 1-D arrays of scores, passed as separate positional arguments (one_way_anova(a, b, c)). Groups may have different lengths and are treated as mutually independent samples -- entries are not paired across groups.

significance_level

float = 0.05

:math:`\alpha` used only to compute the returned boolean; it does not affect f_statistic or p_value. Keyword-only.

Returns

f_statistic

float

The F-ratio :math:`MS_B / MS_W`. Always non-negative; 0.0 when the within-group mean square is zero. Values near 1 are what :math:`H_0` predicts.

p_value

float

Upper-tail probability :math:`P(F_{k-1,\,N-k} > F)` under :math:`H_0`, in :math:`[0, 1]`.

significant

bool

p_value < significance_level. True means "the groups are not all equal"; it does not identify the winner.

Raises

ValueError

If fewer than two groups are supplied.

Notes

Assumptions.

Independence between and within groups. This is the assumption that

rules ANOVA out for the usual machine-learning layout, where the same datasets or folds are reused by every algorithm -- those samples are matched, not independent. Use friedman_test (or a repeated measures design) there.

Normality of the residuals within each group.
Homoscedasticity -- equal variance across groups. ANOVA tolerates

moderate violations when group sizes are balanced and degrades quickly when they are not.

Complexity. $O(N)$ in the total number of observations.

After a rejection. A significant omnibus result licenses post-hoc pairwise comparisons, but each of those is a fresh test, so their p-values must be adjusted -- see holm_correction.

References

Fisher1925

Fisher, R. A. (1925). "Statistical Methods for Research Workers". Oliver and Boyd, Edinburgh.

Demsar2006

Demsar, J. (2006). "Statistical Comparisons of Classifiers over Multiple Data Sets". Journal of Machine Learning Research, 7, 1-30.

parametric.py

Notes

References

See Also

Classes

SignificanceLevel

Attributes

Notes

See Also

Methods

Returns

PairedStats

Attributes

Notes

See Also

Methods

Returns

Returns

Returns

Returns

Functions

paired_t_test

Hypotheses

Theory

Parameters

Returns

Raises

Notes

References

See Also

corrected_paired_t_test

Hypotheses

Theory

Parameters

Returns

Raises

Notes

References

See Also

one_way_anova

Hypotheses

Theory

Parameters

Returns

Raises

Notes

References

See Also