API Reference /

benchmarking.py

Benchmark models across datasets.

Benchmark takes a set of model specs and a set of dataset specs; run scores every model on every dataset with cross-validation or repeated holdout, optionally tuning each model inside the training folds. Results live on the instance: tidy per-fold scores, ranked tables, significance tests, paper-ready exports, and comparison plots.

Everything is declarative, in the same {"name": ..., "params": {...}} convention as train, so a whole benchmark can be stored as JSON and replayed.

Classes

Benchmark

class benchmarking.Benchmark

A declarative model comparison: configure it, then run it.

The constructor validates the whole configuration up front, so a typo fails immediately, not minutes into the run. run executes the comparison and stores the results on the instance: scores_, best_params_, and every table, test, export, and plot method.
Constructor 
__init__(
    self,
    models: List[Dict[str, Any]],
    datasets: List[Dict[str, Any]],
    *,
    pipeline: Optional[Union[str, List[Dict]]] = None,
    evaluation: Optional[Dict[str, Any]] = None,
    random_seed: Optional[int] = None,
    task: Optional[str] = None,
)

Parameters

models
list of dict

Model specs, each {"name": ..., "params": {...}} plus optional keys:

  • •"tune": ``{"method": "grid" | "random", "space": {param:

[values]}, "iterations": 25}``. Tuning runs on the training side of every fold with an inner 3-fold split, so reported scores stay honest.

  • •"pipeline": per-model step list overriding the shared

pipeline (for example scaling for an SVM but not for trees).

  • •"label": display name; defaults to the model name.
datasets
list of dict

Dataset specs, always in spec form:

  • •{"source": "iris"}: a builtin dataset (builtins define their

own target).

  • •``{"source": "sales.csv", "target": "label", "features": [...],

"name": "sales"}: a data file; "features" restricts the columns, "name"`` sets the display name.

  • •{"name": "custom", "X": X_array, "y": y_array}: in-memory
arrays.
pipeline
str or list of dict
Steps applied before every model, as component specs, or a preset name from PRESETS.
evaluation
dict
{"cv": 10} for k-fold (the default), or {"test_size": 0.2} for holdout. "repeats" repeats either scheme with fresh seeds, which also gives holdout a spread for statistics. "stratify" (default True) keeps class balance, and "metrics" lists metric function names from tuiml.evaluation.metrics.
random_seed
int
Seeds the splits and every component that accepts a seed, making the run reproducible. Falls back to the global seed, then 42.
task
{"classification", "regression", "clustering"}
Overrides the task inferred from the models.

Attributes

scores_
pd.DataFrame
Tidy per-fold scores with columns dataset, model, metric, fold, value. Set by run.
best_params_
pd.DataFrame
Tuning choices per dataset, model, and fold. Set by run.
task
str
The resolved task.
metrics
list of str
The metric names to compute; the first is the primary one.
random_seed
int
The resolved seed.

Step 1, import and configure:

python 
>>> from tuiml import Benchmark
>>> bench = Benchmark(
...     models=[
...         {"name": "NaiveBayesClassifier"},
...         {"name": "RandomForestClassifier",
...          "tune": {"method": "grid",
...                   "space": {"n_estimators": [50, 100]}}},
...     ],
...     datasets=[
...         {"source": "iris"},
...         {"source": "sales.csv", "target": "label"},
...     ],
...     evaluation={"cv": 10, "metrics": ["accuracy_score"]},
...     random_seed=42,
... )

Step 2, execute:

python 
>>> bench.run()                                        # doctest: +SKIP

Step 3, read the results:

python 
>>> print(bench.summary())                             # doctest: +SKIP
>>> bench.table()                                      # doctest: +SKIP
>>> bench.compare(baseline="NaiveBayesClassifier")     # doctest: +SKIP
>>> bench.to_latex()                                   # doctest: +SKIP

Methods

run (self, verbose: int=0, progress_callback: Optional[Callable]=None) -> 'Benchmark'

Execute the benchmark and store the results on this instance.

Parameters
verbose
int = 0
Print progress while running.
progress_callback
callable
Called after every fold with a progress dict.
Returns
self
Benchmark
With scores_ and best_params_ populated.
table (self, metric: Optional[str]=None, formatted: bool=True) -> pd.DataFrame

Datasets by models table of scores for one metric.

Parameters
metric
str
Which metric to tabulate; defaults to the primary one.
formatted
bool = True
True gives "mean ± std" strings; False gives numeric means.
Returns
pd.DataFrame
One row per dataset, one column per model.
best (self, metric: Optional[str]=None) -> pd.DataFrame

The winning model per dataset, plus the overall mean rank.

Parameters
metric
str
Which metric decides; defaults to the primary one.
Returns
pd.DataFrame
Columns best_model and best_score indexed by dataset, with an overall row naming the best mean rank.
compare (self, baseline: str, metric: Optional[str]=None, significance_level: float=0.05) -> pd.DataFrame

Paired significance tests of every model against a baseline.

Parameters
baseline
str
The model label to compare against.
metric
str
Which metric to test; defaults to the primary one.
significance_level
float = 0.05
Alpha for the paired t-test.
Returns
pd.DataFrame
One row per (dataset, model): means, p-value, significance, and the winner (or "tie").
summary (self) -> str

A readable text report of the primary metric per dataset.

Returns
str
The report.
to_markdown (self, metric: Optional[str]=None) -> str

The scores table as GitHub-flavored markdown, winners in bold.

Parameters
metric
str
Which metric to export; defaults to the primary one.
Returns
str
A markdown table.
to_latex (self, metric: Optional[str]=None) -> str

The scores table as a LaTeX table, winners in bold.

Parameters
metric
str
Which metric to export; defaults to the primary one.
Returns
str
A LaTeX table environment.
to_csv (self, path: Optional[str]=None) -> Optional[str]

Write (or return) the tidy per-fold scores as CSV.

Parameters
path
str
File to write. When omitted, the CSV text is returned.
Returns
str or None
The CSV text when no path was given.
to_html (self, metric: Optional[str]=None) -> str

The scores table as an HTML table.

Parameters
metric
str
Which metric to export; defaults to the primary one.
Returns
str
An HTML table.
plot_critical_difference (self, metric: Optional[str]=None, **kwargs)

Critical difference diagram over datasets (Demsar style).

Parameters
metric
str
Which metric to rank by; defaults to the primary one.
**kwargs
Forwarded to plot_critical_difference.
plot_boxplot (self, metric: Optional[str]=None, **kwargs)

Per-model box plots of every fold score.

Parameters
metric
str
Which metric to plot; defaults to the primary one.
**kwargs
Forwarded to plot_boxplot_comparison.
plot_ranking (self, metric: Optional[str]=None, **kwargs)

Mean-rank table plot across datasets.

Parameters
metric
str
Which metric to rank by; defaults to the primary one.
**kwargs
Forwarded to plot_ranking_table.
__repr__ (self) -> str