evaluation/

Model evaluation: metrics, resampling, significance tests, and visualization.

Everything needed to measure how well a model performs and to decide whether a difference between two models is real. The subpackages are:

metrics: classification, regression, clustering, and information-theoretic

scores.

splitting: resampling strategies (KFold,

train_test_split, and friends).

statistics: significance tests for comparing models across datasets, plus

multiple-comparison corrections.

tuning: hyperparameter search (grid, random, and Bayesian).
visualization: plots and critical-difference diagrams.

The most common metrics and splitters are re-exported here, so from tuiml.evaluation import accuracy_score works without naming the subpackage.

Score predictions:

python

>>> import numpy as np
>>> from tuiml.evaluation import accuracy_score
>>> y_true = np.array([0, 1, 1, 0])
>>> y_pred = np.array([0, 1, 0, 0])
>>> accuracy_score(y_true, y_pred)
0.75

Hold out a test set:

python

>>> from tuiml.datasets import load_iris
>>> from tuiml.evaluation import train_test_split
>>> X, y = load_iris()
>>> X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
>>> len(X_train), len(X_test)
(120, 30)

Cross-validate:

python

>>> from tuiml.evaluation import KFold
>>> cv = KFold(n_splits=5, shuffle=True, random_state=0)
>>> sum(1 for _ in cv.split(X))
5

Search hyperparameters:

python

>>> from tuiml.evaluation import GridSearchCV
>>> from tuiml.algorithms.bayesian import NaiveBayesClassifier
>>> search = GridSearchCV(NaiveBayesClassifier(),
...                       param_grid={'use_kernel_estimator': [True, False]})
>>> search = search.fit(X, y)                              # doctest: +SKIP
>>> search.best_params_                                    # doctest: +SKIP
{'use_kernel_estimator': False}

Packages