API Reference / evaluation / splitting /

group.py

Group-aware cross-validation: never split a group across train and test.

Reach for these splitters whenever rows are not independent because several of them come from the same source -- multiple readings per patient, several sessions per user, many sentences from one document, repeated measurements of one subject. With an ordinary KFold, sibling rows land on both sides of the split and the model can score well simply by recognising the source, which silently inflates the estimate. Grouped splitters keep every group whole inside a single fold, so the test fold always measures generalisation to unseen groups.

Two variants are provided:

GroupKFold -- assigns groups to folds round-robin; ignores y. StratifiedGroupKFold -- also tries to balance the class distribution across folds; requires y.

Both require the groups argument to split() and need at least n_splits distinct groups.

Classes

GroupKFold

class evaluation.splitting.group.GroupKFold(BaseSplitter)

K-fold over whole groups: a group never spans train and test.

Folding happens at the level of the distinct values in groups, not at the level of rows. Each unique group is assigned to exactly one fold, so when that fold is the test set every row of the group is held out together, and when it is not, every row is in training. As a result no group ever appears on both sides of a split -- the property that makes this splitter the right choice for clustered data.

Groups are assigned round-robin in sorted order (i % n_splits over np.unique(groups)), so folds hold a similar number of groups but not necessarily a similar number of rows: unequal group sizes give unequal fold sizes. Labels are ignored; use StratifiedGroupKFold if class balance matters too. groups is required and there must be at least n_splits distinct groups.

Constructor 
__init__(
    self,
    n_splits: int = 5,
)

Parameters

n_splits
int = 5
Number of folds. Must be at least 2 and no larger than the number of distinct groups.

Notes

6 samples in 3 groups, n_splits=3. Rows of group 1 always move together: 

sample:  0 1 2 3 4 5
group:   1 1 2 2 3 3
fold 0:  T T . . . .      test group {1}
fold 1:  . . T T . .      test group {2}
fold 2:  . . . . T T      test group {3}

T = test, . = train

See Also

StratifiedGroupKFold

Grouped folds that also balance classes.
KFold

Plain folds, no group awareness.
python 
>>> import numpy as np
>>> from tuiml.evaluation.splitting import GroupKFold
>>> X = np.arange(12).reshape(6, 2)
>>> y = np.array([0, 0, 0, 1, 1, 1])
>>> groups = np.array([1, 1, 2, 2, 3, 3])
>>> cv = GroupKFold(n_splits=3)
>>> print(cv.get_n_splits())
3
>>> for train_idx, test_idx in cv.split(X, y, groups):
...     print(test_idx.tolist(),
...           sorted(set(groups[test_idx].tolist())),
...           sorted(set(groups[train_idx].tolist())))
[0, 1] [1] [2, 3]
[2, 3] [2] [1, 3]
[4, 5] [3] [1, 2]

The test and train group sets are always disjoint.

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for constructor parameters.

split (self, X: np.ndarray, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> Iterator[Tuple[np.ndarray, np.ndarray]]

Yield the n_splits train/test index pairs of a grouped K-fold.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data. Only its length is used.
y
np.ndarray of shape (n_samples,)
Ignored; accepted for a uniform splitter API.
groups
np.ndarray of shape (n_samples,)
Group label of each sample. Required. Samples sharing a label always end up in the same fold.
Yields
train_index
np.ndarray of shape (n_train,)
Positional indices of the training rows for this fold.
test_index
np.ndarray of shape (n_test,)
Positional indices of the held-out rows: every row of the groups assigned to this fold, and no row of any other group.
Raises
ValueError
If groups is None, has a different length than X, or contains fewer distinct values than n_splits.
get_n_splits (self, X: Optional[np.ndarray]=None, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> int

Return the number of folds, i.e. how many pairs split will yield.

Parameters
X
np.ndarray
Ignored; the count is fixed at construction time.
y
np.ndarray
Ignored.
groups
np.ndarray
Ignored.
Returns
n_splits
int
The configured n_splits.
__repr__ (self) -> str

Return a short string showing the fold count.

StratifiedGroupKFold

class evaluation.splitting.group.StratifiedGroupKFold(BaseSplitter)

Grouped K-fold that also tries to balance classes across folds.

Combines the two guarantees of GroupKFold and StratifiedKFold, but only the first is exact. A group is still never split across train and test -- that is a hard constraint. Class balance is best effort: groups are sorted by their majority class and then assigned greedily, each group going to whichever fold currently holds the fewest samples of that group's majority class.

Because whole groups are indivisible, exact stratification is generally impossible, and the greedy pass optimises class counts rather than fold sizes. Expect folds that differ noticeably in the number of groups and rows when group sizes or class mixes vary. Both y and groups are required, and there must be at least n_splits distinct groups.

Constructor 
__init__(
    self,
    n_splits: int = 5,
    shuffle: bool = False,
    random_state: Optional[int] = None,
)

Parameters

n_splits
int = 5
Number of folds. Must be at least 2 and no larger than the number of distinct groups.
shuffle
bool = False
Whether to shuffle the group order before the greedy assignment, which varies the resulting partition between runs.
random_state
int
Random seed for reproducibility. Only used when shuffle=True.

Notes

Fold assignment is driven by each group's majority class, so a group whose labels are mixed contributes its minority rows wherever its majority class sends it. Verify the realised balance rather than assuming it: 

for train_idx, test_idx in cv.split(X, y, groups):
    print(np.bincount(y[test_idx]))

See Also

GroupKFold

Grouped folds without stratification.
StratifiedKFold

Exact stratification, no group awareness.
python 
>>> import numpy as np
>>> from tuiml.evaluation.splitting import StratifiedGroupKFold
>>> X = np.arange(24).reshape(12, 2)
>>> y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1])
>>> groups = np.array([1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6])
>>> cv = StratifiedGroupKFold(n_splits=3)
>>> print(cv.get_n_splits())
3
>>> for train_idx, test_idx in cv.split(X, y, groups):
...     print(sorted(set(groups[test_idx].tolist())),
...           np.bincount(y[test_idx], minlength=2).tolist())
[1, 3, 6] [2, 4]
[2, 4] [2, 2]
[5] [0, 2]

Groups stay intact, but the folds are visibly uneven -- the best-effort

nature of the stratification in action.

Methods

get_parameter_schema (cls) -> dict

Return JSON Schema for constructor parameters.

split (self, X: np.ndarray, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> Iterator[Tuple[np.ndarray, np.ndarray]]

Yield the n_splits grouped, class-balanced train/test index pairs.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data. Only its length is used.
y
np.ndarray of shape (n_samples,)
Class labels used for the best-effort balancing. Required.
groups
np.ndarray of shape (n_samples,)
Group label of each sample. Required. Samples sharing a label always end up in the same fold.
Yields
train_index
np.ndarray of shape (n_train,)
Positional indices of the training rows for this fold.
test_index
np.ndarray of shape (n_test,)
Positional indices of the held-out rows: all rows of the groups assigned to this fold. Group-disjoint from the training rows; class proportions are approximate, not guaranteed.
Raises
ValueError
If y or groups is None, if groups has a different length than X, or if there are fewer distinct groups than n_splits.
get_n_splits (self, X: Optional[np.ndarray]=None, y: Optional[np.ndarray]=None, groups: Optional[np.ndarray]=None) -> int

Return the number of folds, i.e. how many pairs split will yield.

Parameters
X
np.ndarray
Ignored; the count is fixed at construction time.
y
np.ndarray
Ignored.
groups
np.ndarray
Ignored.
Returns
n_splits
int
The configured n_splits.
__repr__ (self) -> str

Return a short string showing the fold count and shuffle flag.