Subset-based feature selection methods.
This module provides feature selectors that evaluate subsets of features rather than individual features.
- CFSSelector: correlation-based subset evaluation (Hall, 1999).
- WrapperSelector: subset scored by cross-validating the target estimator.
Classes
Correlation-based Feature Selection (CFS).
CFS evaluates subsets of features by considering the individual predictive ability of each feature along with the degree of redundancy between them.
Constructor
__init__( self, n_bins: int = 10, search_method: str = 'best_first', search_termination: int = 5, locally_predictive: bool = True, )
Overview
The merit of a feature subset S is computed as:
\text{merit}(S) = \frac{k \bar{r}_{cf}}{\sqrt{k + k(k-1)\bar{r}_{ff}}}
where:
- k is the number of features in subset S.
- \bar{r}_{cf} is the average feature-class correlation.
- \bar{r}_{ff} is the average feature-feature inter-correlation.
Parameters
n_bins
int
= 10
Number of bins for discretizing continuous features before computing symmetrical uncertainty.
search_method
{"best_first", "greedy_forward"}
= "best_first"
Search method for finding the subset with maximum merit:
- •
"best_first": Best-first search with backtracking. - •
"greedy_forward": Simple forward greedy search.
search_termination
int
= 5
For
"best_first": number of non-improving nodes before terminating.
locally_predictive
bool
= True
If True, include locally predictive attributes (features that correlate more with the class than with any already selected feature).
Attributes
selected_features_
np.ndarray
Indices of the selected features.
merit_
float
The CFS merit score of the final selected subset.
Notes
When to use:
- When you want a fast, filter-based subset selection that handles redundancy.
- When you have many features and want to reduce them without training a model.
- Only handles linear/monotone relationships via symmetrical uncertainty.
- May struggle with complex non-linear feature interactions that a wrapper
References
Hall1998
Hall, M. A. (1998). Correlation-based Feature Subset Selection
for Machine Learning. PhD Thesis, University of Waikato.
Basic usage with search method configuration:
python
>>> from tuiml.features.selection import CFSSelector
>>> import numpy as np
>>> X, y = np.random.randn(50, 10), np.random.randint(0, 2, 50)
>>> selector = CFSSelector(search_method="best_first", search_termination=3)
>>> X_new = selector.fit_transform(X, y)
>>> print(f"Selected: {selector.selected_features_}")
>>> print(f"Merit: {selector.merit_:.4f}")Methods
Wrapper-based Feature Selection using cross-validation.
Evaluates feature subsets by training a learning algorithm and using its performance (e.g., accuracy) as the merit of the subset.
Constructor
__init__( self, estimator: Any, cv: int = 5, scoring: str = 'accuracy', search_method: str = 'greedy_forward', search_termination: int = 5, random_state: Optional[int] = None, )
Overview
Wrapper selection is a comprehensive approach that considers feature interactions by treating the learning algorithm as a "black box" to score subsets. It uses cross-validation to provide a robust performance estimate.
Search Strategies
-
"greedy_forward": Start with no features and add one at a time. -
"greedy_backward": Start with all features and remove one at a time. -
"best_first": Search through subsets using priority queue with backtracking.
Parameters
estimator
object
A classifier or regressor with
fit and predict methods. Must follow the scikit-learn estimator interface.
cv
int
= 5
Number of cross-validation folds for evaluating each subset.
scoring
{"accuracy", "f1", "precision", "recall"}
= "accuracy"
The performance metric used to rank subsets.
search_method
{"greedy_forward", "greedy_backward", "best_first"}
= "greedy_forward"
The algorithm used to explore the subset space.
search_termination
int
= 5
For
"best_first": how many non-improving expansions to allow.
random_state
int
Random seed for cross-validation shuffling.
Attributes
selected_features_
np.ndarray
Indices of the features in the best subset found.
cv_score_
float
The cross-validation score achieved by the final selected subset.
Notes
When to use:
- When you want to find the absolute best subset for a specific model.
- When feature interactions are crucial (e.g., XOR-like problems).
- Computationally very expensive, especially with many features or slow models.
- High risk of overfitting to the validation set if the dataset is small.
References
Kohavi1997
Kohavi, R. and John, G. (1997). Wrappers for feature subset
selection. Artificial Intelligence, 97(1-2), 273-324.
Using a Decision Tree with forward selection:
python
>>> from tuiml.features.selection import WrapperSelector
>>> from tuiml.algorithms.trees import DecisionTreeClassifier
>>> import numpy as np
>>> X, y = np.random.randn(20, 6), np.random.randint(0, 2, 20)
>>> selector = WrapperSelector(
... estimator=DecisionTreeClassifier(),
... cv=3,
... search_method="greedy_forward"
... )
>>> X_new = selector.fit_transform(X, y)
>>> print(f"Selected: {selector.selected_features_}")
>>> print(f"Best CV Score: {selector.cv_score_:.4f}")