Non-parametric statistical tests.
Tests that don't assume normal distribution.
Functions
wilcoxon_signed_rank_test(x: np.ndarray, y: np.ndarray, significance_level: float=0.05, higher_better: bool=True) -> PairedStats
Wilcoxon signed-rank test (non-parametric alternative to paired t-test).
Parameters
x, y
ndarray
Results from two models.
significance_level
float
Significance level.
higher_better
bool
If True, higher is better.
Returns
stats
PairedStats
friedman_test(results: Dict[str, np.ndarray], significance_level: float=0.05) -> Tuple[float, float, bool]
Friedman test for comparing multiple models.
Non-parametric test for comparing more than two related samples.
Parameters
results
dict
Dictionary of {model_name: scores_array}. All arrays must have the same length.
significance_level
float
Significance level.
Returns
chi2
float
Chi-squared statistic.
p_value
float
P-value.
significant
bool
Whether the difference is significant.
python
>>> import numpy as np
>>> from tuiml.evaluation.statistics import friedman_test
>>> results = {
... 'ModelA': np.array([0.85, 0.87, 0.83]),
... 'ModelB': np.array([0.82, 0.84, 0.81]),
... 'ModelC': np.array([0.80, 0.82, 0.79])
... }
>>> chi2, p_value, sig = friedman_test(results)nemenyi_post_hoc(results: Dict[str, np.ndarray], significance_level: float=0.05) -> Dict[Tuple[str, str], bool]
Nemenyi post-hoc test after Friedman test.
Parameters
results
dict
Dictionary of {model_name: scores_array}.
significance_level
float
Significance level.
Returns
pairwise
dict
Dictionary of {(model1, model2): is_significant}.
friedman_aligned_ranks_test(results: Dict[str, np.ndarray], significance_level: float=0.05) -> Tuple[float, float, bool]
Friedman Aligned Ranks test.
More powerful alternative to standard Friedman test.
Parameters
results
dict
Dictionary of {model_name: scores_array}.
significance_level
float
Significance level.
Returns
statistic
float
Test statistic.
p_value
float
P-value.
significant
bool
Whether the difference is significant.
quade_test(results: Dict[str, np.ndarray], significance_level: float=0.05) -> Tuple[float, float, bool]
Quade test for comparing multiple algorithms.
Similar to Friedman but accounts for dataset difficulty.
Parameters
results
dict
Dictionary of {model_name: scores_array}.
significance_level
float
Significance level.
Returns
f_statistic
float
F statistic.
p_value
float
P-value.
significant
bool
Whether the difference is significant.