API Reference / preprocessing / encoding /

rare_category.py

RareCategoryEncoder filter.

Merge infrequent (rare) nominal values into a single category.

Classes

RareCategoryEncoder

class preprocessing.encoding.rare_category.RareCategoryEncoder(Transformer)

Merge infrequent nominal values into a single category.

Values that appear below a frequency threshold are merged into a single "other" category.
Constructor 
__init__(
    self,
    min_frequency: float = 5,
    columns: Optional[List[int]] = None,
    merged_value: int = ...,
)

Parameters

min_frequency
int or float = 5

Minimum frequency for a value to be kept.

  • •int: Absolute count threshold
  • •float (0-1): Proportion threshold
columns
list of int
Indices of columns to process. If None, processes all.
merged_value
str or int = -1

Value to use for merged categories.

  • •For numeric encoding: typically -1 or max+1
  • •For string: "other", "rare", etc.

Attributes

value_maps_
dict
Mapping of original values to merged values for each column.
python 
>>> import numpy as np
>>> from tuiml.preprocessing.encoding import RareCategoryEncoder
python 
>>> # Values: 0 appears 5 times, 1 appears 3 times, 2 appears 1 time
>>> X = np.array([[0], [0], [0], [0], [0], [1], [1], [1], [2]])
python 
>>> # Merge values appearing less than 3 times
>>> merger = RareCategoryEncoder(min_frequency=3)
>>> X_merged = merger.fit_transform(X)
>>> # Value 2 is merged into -1

Methods

get_parameter_schema (cls)
fit (self, X: np.ndarray, y: Optional[np.ndarray]=None, feature_names: Optional[List[str]]=None) -> 'RareCategoryEncoder'

Learn which category values are too rare to keep.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input data holding categorical values.
y
np.ndarray
Ignored, present for API consistency.
feature_names
list of str
Names of the input columns.
Returns
self
object
The fitted encoder, holding the retained values per column.
transform (self, X: np.ndarray) -> np.ndarray

Replace infrequent category values with the merged placeholder.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input data with the same number of columns seen during fit.
Returns
X_out
np.ndarray of shape (n_samples, n_features)
Data where every value below the frequency threshold has been replaced by merged_value.
Raises
ValueError
If X has a different number of columns than seen during fit.
value_maps_ (self) -> Dict[int, Dict]

Mapping of original values to merged values for each column.

__repr__ (self) -> str