API Reference / cli /

benchmark.py

Benchmark Command - Compare models across datasets via CLI.

Functions

Command

tuiml benchmark

Line 24
tuiml benchmark [OPTIONS]

Benchmark models across datasets with cross-validation.

Runs every model on every dataset, reports mean and spread per metric, marks winners, and optionally tests significance against a baseline.

Benchmark three models with 10-fold cross-validation:

bash 
$ tuiml benchmark -a RandomForestClassifier -a SVC -a NaiveBayesClassifier -d iris.csv -t class --cv 10

Compare across multiple datasets against a baseline:

bash 
$ tuiml benchmark -a SVC -a RandomForestClassifier -d iris -d glass --baseline SVC

Use a repeated holdout split instead of cross-validation:

bash 
$ tuiml benchmark -a SVC -a RandomForestClassifier -d iris --test-size 0.3 --repeats 5

Export a LaTeX table and show the critical difference diagram:

bash 
$ tuiml benchmark -a SVC -a RandomForestClassifier -d iris -o results.tex -f latex --plot

Options

--models, -a
repeatable required
Model to include, by exact class name, e.g. RandomForestClassifier. Repeatable.
--data, -d
repeatable required
Dataset file path or built-in dataset name. Repeatable; every model runs on every dataset.
--target, -t
Target column name. Applies to dataset files only; built-in datasets define their own.
--cv
int = 10
Number of cross-validation folds (default: 10). Ignored when --test-size is given.
--test-size
float
Use a holdout split of this fraction instead of cross-validation.
--repeats
int = 1
Repeat the whole split scheme this many times and pool the results (default: 1).
--metrics, -m
repeatable
Metric to report, by function name. Repeatable; defaults to metrics chosen for the task.
--random-seed
int
Random seed, so the splits and therefore the comparison are reproducible.
--output, -o
Write the results table to this file, in the format given by --format.
--format, -f
choice: markdown | latex | csv | html = 'markdown'
Format of the file written by --output (default: markdown).
--baseline
Model to use as the reference for pairwise significance tests.
--plot
flag
Display a critical difference diagram of the model rankings.
--verbose, -v
flag
Report progress while running and re-raise full tracebacks on error.