API Reference /

benchmarking.py

Benchmark models across datasets.

Benchmark takes a set of model specs and a set of dataset specs; run scores every model on every dataset with cross-validation or repeated holdout, optionally tuning each model inside the training folds. Results live on the instance: tidy per-fold scores, ranked tables, significance tests, paper-ready exports, and comparison plots.

Everything is declarative, in the same {"name": ..., "params": {...}} convention as train, so a whole benchmark can be stored as JSON and replayed.

Classes

Benchmark

class benchmarking.Benchmark

A declarative model comparison: configure it, then run it.

The constructor validates the whole configuration up front, so a typo fails immediately, not minutes into the run. run executes the comparison and stores the results on the instance: scores_, best_params_, and every table, test, export, and plot method.
Constructor 
__init__(
    self,
    models: List[Dict[str, Any]],
    datasets: List[Dict[str, Any]],
    *,
    pipeline: Optional[Union[str, List[Dict]]] = None,
    evaluation: Optional[Dict[str, Any]] = None,
    random_seed: Optional[int] = None,
    task: Optional[str] = None,
)

Parameters

models
list of dict

Model specs, each {"name": ..., "params": {...}} plus optional keys:

  • •"tune": ``{"method": "grid" | "random", "space": {param:

[values]}, "iterations": 25}``. Tuning runs on the training side of every fold with an inner 3-fold split, so reported scores stay honest.

  • •"pipeline": per-model step list overriding the shared

pipeline (for example scaling for an SVM but not for trees).

  • •"label": display name; defaults to the model name.
datasets
list of dict

Dataset specs, always in spec form:

  • •{"source": "iris"}: a builtin dataset (builtins define their

own target).

  • •``{"source": "sales.csv", "target": "label", "features": [...],

"name": "sales"}: a data file; "features" restricts the columns, "name"`` sets the display name.

  • •{"name": "custom", "X": X_array, "y": y_array}: in-memory
arrays.
pipeline
str or list of dict
Steps applied before every model, as component specs, or a preset name from PRESETS.
evaluation
dict
{"cv": 10} for k-fold (the default), or {"test_size": 0.2} for holdout. "repeats" repeats either scheme with fresh seeds, which also gives holdout a spread for statistics. "stratify" (default True) keeps class balance, and "metrics" lists metric function names from tuiml.evaluation.metrics. Forecasters (models tagged timeseries) never shuffle: "cv" becomes an expanding-window TimeSeriesSplit, "test_size" a single tail holdout, and "repeats", "stratify" and the shared pipeline do not apply. The series is the dataset's target column, or its first feature column when there is no target.
random_seed
int
Seeds the splits and every component that accepts a seed, making the run reproducible. Falls back to the global seed, then 42.
task
{"classification", "regression", "clustering", "forecasting"}
Overrides the task inferred from the models.

Attributes

scores_
pd.DataFrame
Tidy per-fold scores with columns dataset, model, metric, fold, value. Set by run.
best_params_
pd.DataFrame
Tuning choices per dataset, model, and fold. Set by run.
task
str
The resolved task.
metrics
list of str
The metric names to compute; the first is the primary one.
random_seed
int
The resolved seed.

Step 1, import and configure:

python 
>>> from tuiml import Benchmark
>>> bench = Benchmark(
...     models=[
...         {"name": "NaiveBayesClassifier"},
...         {"name": "RandomForestClassifier",
...          "tune": {"method": "grid",
...                   "space": {"n_estimators": [50, 100]}}},
...     ],
...     datasets=[
...         {"source": "iris"},
...         {"source": "sales.csv", "target": "label"},
...     ],
...     evaluation={"cv": 10, "metrics": ["accuracy_score"]},
...     random_seed=42,
... )

Step 2, execute:

python 
>>> bench.run()                                        # doctest: +SKIP

Step 3, read the results:

python 
>>> print(bench.summary())                             # doctest: +SKIP
>>> bench.table()                                      # doctest: +SKIP
>>> bench.compare(baseline="NaiveBayesClassifier")     # doctest: +SKIP
>>> bench.to_latex()                                   # doctest: +SKIP

Methods

run (self, verbose: int=0, progress_callback: Optional[Callable]=None) -> 'Benchmark'

Execute the benchmark and store the results on this instance.

Parameters
verbose
int = 0
Print progress while running.
progress_callback
callable
Called after every fold with a progress dict.
Returns
self
Benchmark
With scores_ and best_params_ populated.
table (self, metric: Optional[str]=None, formatted: bool=True) -> pd.DataFrame

Datasets by models table of scores for one metric.

Parameters
metric
str
Which metric to tabulate; defaults to the primary one.
formatted
bool = True
True gives "mean ± std" strings; False gives numeric means.
Returns
pd.DataFrame
One row per dataset, one column per model.
best (self, metric: Optional[str]=None) -> pd.DataFrame

The winning model per dataset, plus the overall mean rank.

Parameters
metric
str
Which metric decides; defaults to the primary one.
Returns
pd.DataFrame
Columns best_model and best_score indexed by dataset, with an overall row naming the best mean rank.
compare (self, baseline: str, metric: Optional[str]=None, significance_level: float=0.05) -> pd.DataFrame

Paired significance tests of every model against a baseline.

Parameters
baseline
str
The model label to compare against.
metric
str
Which metric to test; defaults to the primary one.
significance_level
float = 0.05
Alpha for the paired t-test.
Returns
pd.DataFrame
One row per (dataset, model): means, p-value, significance, and the winner (or "tie").
summary (self) -> str

A readable text report of the primary metric per dataset.

Returns
str
The report.
to_markdown (self, metric: Optional[str]=None) -> str

The scores table as GitHub-flavored markdown, winners in bold.

Parameters
metric
str
Which metric to export; defaults to the primary one.
Returns
str
A markdown table.
to_latex (self, metric: Optional[str]=None) -> str

The scores table as a LaTeX table, winners in bold.

Parameters
metric
str
Which metric to export; defaults to the primary one.
Returns
str
A LaTeX table environment.
to_csv (self, path: Optional[str]=None) -> Optional[str]

Write (or return) the tidy per-fold scores as CSV.

Parameters
path
str
File to write. When omitted, the CSV text is returned.
Returns
str or None
The CSV text when no path was given.
to_html (self, metric: Optional[str]=None) -> str

The scores table as an HTML table.

Parameters
metric
str
Which metric to export; defaults to the primary one.
Returns
str
An HTML table.
plot_critical_difference (self, metric: Optional[str]=None, **kwargs)

Critical difference diagram over datasets (Demsar style).

Parameters
metric
str
Which metric to rank by; defaults to the primary one.
**kwargs
Forwarded to plot_critical_difference.
plot_boxplot (self, metric: Optional[str]=None, **kwargs)

Per-model box plots of every fold score.

Parameters
metric
str
Which metric to plot; defaults to the primary one.
**kwargs
Forwarded to plot_boxplot_comparison.
plot_ranking (self, metric: Optional[str]=None, **kwargs)

Mean-rank table plot across datasets.

Parameters
metric
str
Which metric to rank by; defaults to the primary one.
**kwargs
Forwarded to plot_ranking_table.
__repr__ (self) -> str