Test Statistics Command - Significance tests over experiment results.
Functions
tuiml test-statistics [OPTIONS]
Run statistical significance tests over experiment results.
Compares algorithms using their cross-validation scores and reports the test statistic, the p-value and whether the difference is significant, so a benchmark result can be defended rather than eyeballed. The Friedman test, Friedman aligned ranks, the Quade test, one-way ANOVA, the Nemenyi post-hoc test, the Wilcoxon signed-rank test and the paired t-test are all available. Scores are given as a JSON object mapping each algorithm name to its list of fold scores, and --significance-level moves alpha away from its 0.05 default.
Compare three or more algorithms:
bash
$ tuiml test-statistics --test friedman --results "$SCORES"Find which pairs actually differ, once Friedman is significant:
bash
$ tuiml test-statistics --test nemenyi --results "$SCORES"Compare exactly two algorithms without assuming normality:
bash
$ tuiml test-statistics --test wilcoxon --results "$SCORES"Options
--test
str required
Test to run. Across 3+ algorithms: 'friedman', 'friedman_aligned' (more powerful variant), 'quade' (accounts for dataset difficulty), 'anova' (parametric). Post-hoc after Friedman: 'nemenyi'. Between 2 algorithms: 'wilcoxon', 'paired_t' (parametric)
--results
str required
Cross-validation scores as a JSON object mapping each algorithm name to its list of fold scores
--significance-level
float
Significance level alpha below which a difference counts as real (default: 0.05)
--higher-better/--no-higher-better
= True
Whether a higher score means a better model, which sets the ranking direction (default: higher is better)
--json-output
flag
Output raw JSON