API Reference / evaluation / splitting /

holdout.py

Single train/test holdout splitting.

Holdout evaluation carves the data into one train part and one test part instead of rotating over folds. It costs a single fit, so it is the right default for a quick sanity check, for a large dataset where one split is already a reliable estimate, or for carving off a final untouched test set before any cross-validation happens.

This module offers two shapes of the same idea:

train_test_split -- an eager helper that slices the arrays you hand it and returns the pieces directly. HoldoutSplit and StratifiedHoldoutSplit -- lazy splitters implementing the BaseSplitter protocol, so they can be dropped into cross_val_score or any other code that expects split()/get_n_splits(). Both yield exactly one pair.

Use the stratified variants when y is a class label; the unstratified ones can leave a rare class entirely out of the training half. When one split is too noisy to trust, move to KFold or ShuffleSplit.

Classes

HoldoutSplit

class evaluation.splitting.holdout.HoldoutSplit(BaseSplitter)

Single, unstratified train/test holdout exposed as a splitter.

split() yields exactly one (train_index, test_index) pair and get_n_splits() always returns 1, so this class plugs a plain holdout into any code written against the cross-validator protocol.

The first int(n_samples * test_size) positions of the (optionally shuffled) index vector become the test set and the rest become the training set. Labels are ignored, so on class-sorted data an unshuffled holdout can put an entire class on one side; use StratifiedHoldoutSplit for classification.

Constructor 
__init__(
    self,
    test_size: float = 0.3,
    shuffle: bool = True,
    random_state: Optional[int] = None,
)

Parameters

test_size
float = 0.3
Proportion of rows held out for testing. Must be strictly between 0 and 1; unlike train_test_split, an absolute count is not accepted here.
shuffle
bool = True
Whether to permute the sample order before cutting. With shuffle=False the test set is the leading block of rows.
random_state
int
Random seed for reproducibility. Only used when shuffle=True.

Notes

Layout for test_size=0.3 over 10 samples, one column per position of the (possibly shuffled) index vector: 

position:  0 1 2 3 4 5 6 7 8 9
split 0:   T T T . . . . . . .

T = test, . = train

See Also

StratifiedHoldoutSplit

Same, with class proportions preserved.
train_test_split

Eager function returning the sliced arrays.
ShuffleSplit

Repeat this random holdout n_splits times.
python 
>>> import numpy as np
>>> from tuiml.evaluation.splitting import HoldoutSplit
>>> X = np.arange(20).reshape(10, 2)
>>> cv = HoldoutSplit(test_size=0.3, random_state=0)
>>> print(cv.get_n_splits())
1
>>> for train_idx, test_idx in cv.split(X):
...     print(len(train_idx), len(test_idx), sorted(test_idx.tolist()))
7 3 [2, 4, 8]

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for constructor parameters.

split (self, X: np.ndarray, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> Iterator[Tuple[np.ndarray, np.ndarray]]

Yield the single train/test index pair of an unstratified holdout.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Data to split. Only its length is used.
y
np.ndarray of shape (n_samples,)
Ignored; accepted for a uniform splitter API.
groups
np.ndarray of shape (n_samples,)
Ignored; accepted for a uniform splitter API.
Yields
train_index
np.ndarray of shape (n_train,)
Positional indices of the training rows.
test_index
np.ndarray of shape (n_test,)
Positional indices of the held-out rows, where n_test == int(n_samples * test_size). Exactly one pair is produced.
get_n_splits (self, X: Optional[np.ndarray]=None, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> int

Return 1: a holdout produces a single train/test pair.

Parameters
X
np.ndarray
Ignored.
y
np.ndarray
Ignored.
groups
np.ndarray
Ignored.
Returns
n_splits
int
Always 1.
__repr__ (self) -> str

Return a short string showing the test proportion and shuffle flag.

StratifiedHoldoutSplit

class evaluation.splitting.holdout.StratifiedHoldoutSplit(BaseSplitter)

Single, stratified train/test holdout exposed as a splitter.

Like HoldoutSplit, split() yields exactly one (train_index, test_index) pair and get_n_splits() returns 1. The difference is that y is required and the holdout is taken within each class: from every class, max(1, int(n_class * test_size)) samples go to the test part. Both parts therefore keep roughly the class distribution of y, and every class is guaranteed at least one test sample even when it is tiny.
Constructor 
__init__(
    self,
    test_size: float = 0.3,
    shuffle: bool = True,
    random_state: Optional[int] = None,
)

Parameters

test_size
float = 0.3
Proportion of each class held out for testing. Must be strictly between 0 and 1.
shuffle
bool = True
Whether to permute each class's indices before slicing, and to shuffle the two resulting index vectors. With shuffle=False the test part of each class is its leading block.
random_state
int
Random seed for reproducibility. Only used when shuffle=True.

Notes

10 samples, classes 0 0 0 0 0 1 1 1 1 1, test_size=0.4. Two samples are drawn from each class rather than four from one end: 

class:    0 0 0 0 0 1 1 1 1 1
split 0:  T T . . . T T . . .

T = test, . = train

Because the per-class count is rounded up with max(1, ...), the total test size can slightly exceed test_size * n_samples when there are many small classes.

See Also

HoldoutSplit

Same, ignoring class labels.
train_test_split

Eager equivalent via stratify=y.
StratifiedKFold

Rotate over stratified folds instead.
python 
>>> import numpy as np
>>> from tuiml.evaluation.splitting import StratifiedHoldoutSplit
>>> X = np.arange(20).reshape(10, 2)
>>> y = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
>>> cv = StratifiedHoldoutSplit(test_size=0.4, random_state=0)
>>> print(cv.get_n_splits())
1
>>> for train_idx, test_idx in cv.split(X, y):
...     print(len(train_idx), len(test_idx), np.bincount(y[test_idx]).tolist())
6 4 [2, 2]

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for constructor parameters.

split (self, X: np.ndarray, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> Iterator[Tuple[np.ndarray, np.ndarray]]

Yield the single train/test index pair of a stratified holdout.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Data to split. Only its length is used.
y
np.ndarray of shape (n_samples,)
Class labels driving the stratification. Required.
groups
np.ndarray of shape (n_samples,)
Ignored; accepted for a uniform splitter API.
Yields
train_index
np.ndarray of shape (n_train,)
Positional indices of the training rows.
test_index
np.ndarray of shape (n_test,)
Positional indices of the held-out rows, holding approximately the class proportions of y and at least one sample per class. Exactly one pair is produced.
Raises
ValueError
If y is None.
get_n_splits (self, X: Optional[np.ndarray]=None, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> int

Return 1: a holdout produces a single train/test pair.

Parameters
X
np.ndarray
Ignored.
y
np.ndarray
Ignored.
groups
np.ndarray
Ignored.
Returns
n_splits
int
Always 1.
__repr__ (self) -> str

Return a short string showing the test proportion.

Functions

Func

train_test_split

Line 29
train_test_split(*arrays, test_size: Optional[Union[float, int]]=None, train_size: Optional[Union[float, int]]=None, shuffle: bool=True, stratify: Optional[np.ndarray]=None, random_state: Optional[int]=None)

Split one or more equal-length arrays into a single train and test part.

All input arrays are indexed with the same pair of index vectors, so rows stay aligned across X, y and any extra arrays (sample weights, ids). The output is flattened: for inputs (X, y) the return order is X_train, X_test, y_train, y_test.

Parameters

*arrays
sequence of array-like
One or more arrays to split. All must have the same first dimension.
test_size
float or int
If float in (0, 1), the proportion of rows held out for test. If int, the absolute number of test rows. Defaults to 0.25 when neither test_size nor train_size is given. The resulting count is clipped to [1, n_samples - 1] so neither part is ever empty.
train_size
float or int
Complementary specification of the training part, used only when test_size is None.
shuffle
bool = True
Whether to permute the rows before splitting. With shuffle=False and no stratify, the test part is the leading block of rows, which is only meaningful if the row order already carries meaning.
stratify
array-like of shape (n_samples,)
Labels to stratify on (normally y). When given, the split is done per class so both parts keep roughly the class proportions of stratify, and at least one row of each class lands in the test part.
random_state
int
Random seed for reproducible shuffling.

Returns

splits
list of np.ndarray
2 * len(arrays) arrays, alternating train part then test part for each input array in the order given.

See Also

HoldoutSplit

Same split exposed as a lazy splitter object.
KFold

Rotate over folds instead of one split.
python 
>>> import numpy as np
>>> from tuiml.evaluation.splitting import train_test_split
>>> X = np.arange(20).reshape(10, 2)
>>> y = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
>>> X_train, X_test, y_train, y_test = train_test_split(
...     X, y, test_size=0.3, random_state=0
... )
>>> print(X_train.shape, X_test.shape)
(7, 2) (3, 2)
>>> print(sorted(y_test.tolist()))
[0, 0, 1]

Passing stratify=y forces both parts to mirror the class balance:

python 
>>> X_train, X_test, y_train, y_test = train_test_split(
...     X, y, test_size=0.4, stratify=y, random_state=0
... )
>>> print(np.bincount(y_test).tolist(), np.bincount(y_train).tolist())
[2, 2] [3, 3]