Random-permutation cross-validators (Monte-Carlo cross-validation).
A shuffle splitter draws a fresh random train/test partition on every iteration instead of rotating over a fixed set of folds. That is the key difference from KFold: the number of iterations and the test proportion are decoupled, so you can ask for 50 splits with 10% test each, and the test sets are independent draws that may overlap rather than a disjoint cover of the data. Some samples may be tested many times, others never.
Use these when you want more repetitions than 1 / test_size folds would allow, or a test fraction that K-fold cannot express. Prefer KFold when you need each sample tested exactly once.
ShuffleSplit -- ignores y. StratifiedShuffleSplit -- requires y and samples within each class so every draw keeps the class proportions.
Both also allow train_size to be smaller than the complement of test_size, which yields a sub-sampled training set -- handy for learning curves.
Classes
Unstratified random-permutation splitter: n_splits independent draws.
Each iteration reshuffles all sample indices and cuts the permutation into a test block of n_test rows followed by a train block of n_train rows. Unlike KFold, the test sets of different iterations are not disjoint and need not cover the data: a sample can be tested repeatedly or never. Test size and iteration count are independent knobs, so n_splits may be far larger than 1 / test_size.
Labels are ignored -- a draw can be badly class-skewed on imbalanced data. Use StratifiedShuffleSplit for classification.
__init__( self, n_splits: int = 10, test_size: Optional[Union[float, int]] = 0.1, train_size: Optional[Union[float, int]] = None, random_state: Optional[int] = None, )
Parameters
n_splits
test_size
[1, n_samples - 1].
train_size
random_state
n_splits permutations, so a fixed seed reproduces the whole sequence.
Notes
Three draws with test_size=0.3 over 10 samples. Test sets overlap across iterations, which never happens with K-fold:
sample: 0 1 2 3 4 5 6 7 8 9 split 0: . . T . T . . . T . split 1: . T . T . T . . . . split 2: . . T T . . . . T . T = test, . = train
See Also
>>> import numpy as np
>>> from tuiml.evaluation.splitting import ShuffleSplit
>>> X = np.arange(20).reshape(10, 2)
>>> cv = ShuffleSplit(n_splits=3, test_size=0.3, random_state=0)
>>> print(cv.get_n_splits())
3
>>> for train_idx, test_idx in cv.split(X):
... print(len(train_idx), len(test_idx), sorted(test_idx.tolist()))
7 3 [2, 4, 8]
7 3 [1, 3, 5]
7 3 [2, 3, 8]Note that samples 2, 3 and 8 are tested more than once while others are
never tested -- the defining behaviour of a shuffle split.
Methods
Stratified random permutation cross-validation.
__init__( self, n_splits: int = 10, test_size: Optional[Union[float, int]] = 0.1, train_size: Optional[Union[float, int]] = None, random_state: Optional[int] = None, )
Parameters
n_splits
test_size
train_size
random_state
>>> from tuiml.evaluation.splitting import StratifiedShuffleSplit
>>> import numpy as np
>>> X = np.arange(10).reshape(-1, 1)
>>> y = np.array([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])
>>> sss = StratifiedShuffleSplit(n_splits=5, test_size=0.3)
>>> for train_idx, test_idx in sss.split(X, y):
... print(f"Test class distribution: {np.bincount(y[test_idx])}")
Test class distribution: [1 1]
Test class distribution: [1 1]
Test class distribution: [1 1]
Test class distribution: [1 1]
Test class distribution: [1 1]