Statistical significance tests.
Functions
execute_statistical_test(**kwargs) -> Dict[str, Any]
Run statistical significance tests on experiment results.
Backs the
tuiml_test_statistics tool. Supported tests: friedman, nemenyi, wilcoxon, paired_t, anova, friedman_aligned, quade. The pairwise tests (wilcoxon, paired_t) compare the first two algorithms in results.Parameters
test
str
Name of the test to run (arrives via
**kwargs, like all parameters below).
results
dict
{algorithm_name: [scores...]} mapping of per-fold scores.
significance_level
float
= 0.05
Alpha level for significance.
higher_better
bool
= True
Whether higher scores are better (pairwise tests only).
Returns
result
dict
On success:
status ('success'), test and significant; most tests add statistic and p_value; pairwise tests add algorithms and a details dict of means/stds; nemenyi returns per-pair significance in details. On failure: status ('error'), error and optionally suggestion / error_type.