API Reference / features / selection /

sequential.py

Sequential feature selection methods.

This module provides sequential/greedy feature selection methods that iteratively add or remove features based on model performance.

  • SequentialFeatureSelector: greedy forward/backward stepwise search.
  • BestFirstSelector: best-first search with backtracking.

Classes

SequentialFeatureSelector

class features.selection.sequential.SequentialFeatureSelector(FeatureSelector, SelectorMixin)

Sequential feature selector (forward or backward selection).

Performs a greedy search through the space of feature subsets by iteratively adding (forward) or removing (backward) features based on a model's performance.
Constructor 
__init__(
    self,
    estimator: Any = None,
    n_features_to_select: Union[int, float, str] = 'auto',
    direction: Literal['forward', 'backward'] = 'forward',
    scoring: Optional[Any] = None,
    cv: int = 5,
    tol: float = 0.0,
    random_state: Optional[int] = None,
)

Overview

Sequential feature selection is a wrapper method that evaluates feature subsets using a learning algorithm. It starts with an empty set of features (forward) or the full set (backward) and greedily modifies the subset to maximize a performance metric.

Process

  1. Forward Selection: Start with zero features. In each step, evaluate all
features not yet in the set. Add the one that improves the cross-validated score the most.
  1. Backward Selection: Start with all features. In each step, try removing
each feature. Remove the one whose absence results in the highest score.

The process continues until the desired number of features is reached or no improvement above tol is found.

Parameters

estimator
object
A supervised learning estimator with fit and predict methods.
n_features_to_select
int, float, or "auto" = "auto"

Number of features to select:

  • •int: Select exactly this many features.
  • •float: Select this fraction of total features (0 < x < 1).
  • •"auto": Stop when the score doesn't improve by at least tol.
direction
{"forward", "backward"} = "forward"
Search direction. Forward adds features, backward removes them.
scoring
callable
Scoring function. If None, uses accuracy for classification.
cv
int = 5
Number of cross-validation folds.
tol
float = 0.0
Tolerance for improvement. Selection stops if the score doesn't improve by at least tol.
random_state
int
Random seed for cross-validation splits.

Attributes

n_features_to_select_
int
Actual number of features selected.
support_
np.ndarray of shape (n_features,)
Boolean mask of selected features.

Notes

Complexity:
  • Roughly O(n_{features} \cdot n_{select} \cdot CV) model fits.
  • More expensive than filter methods but captures feature interactions.
When to use:
  • For small to medium datasets where feature interactions are important.
  • When you want to find a sparse set of highly predictive features.
Limitations:
  • Computationally expensive for many features.
  • Greedy search may get stuck in local optima.

References

Ferri1994
Ferri, F. J., et al. (1994). Comparative study of techniques for large-scale feature selection. Pattern Recognition in Practice IV, pp. 403-413.

See Also

BestFirstSelector

Search with backtracking.
WrapperSelector

Comprehensive wrapper selection.

Select 2 features forward using a simple estimator:

python 
>>> from tuiml.features.selection import SequentialFeatureSelector
>>> from tuiml.algorithms.linear import LogisticRegression
>>> import numpy as np
>>> X, y = np.random.randn(20, 5), np.random.randint(0, 2, 20)
>>> selector = SequentialFeatureSelector(
...     estimator=LogisticRegression(),
...     n_features_to_select=2,
...     direction='forward'
... )
>>> X_new = selector.fit_transform(X, y)
>>> print(selector.n_features_to_select_)
2

Methods

fit (self, X: np.ndarray, y: Optional[np.ndarray]=None) -> 'SequentialFeatureSelector'

Fit the feature selector.

Parameters
X
ndarray of shape (n_samples, n_features)
Training data.
y
ndarray of shape (n_samples,)
Target values.
Returns
self
SequentialFeatureSelector
The fitted selector.
transform (self, X: np.ndarray) -> np.ndarray

Transform X by selecting the sequentially chosen features.

Parameters
X
ndarray of shape (n_samples, n_features)
Input data.
Returns
X_new
ndarray of shape (n_samples, n_selected_features)
Data with only the selected features.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.

BestFirstSelector

class features.selection.sequential.BestFirstSelector(FeatureSelector, SelectorMixin)

Best-first feature selector with backtracking.

Performs a search through the space of feature subsets using a best-first heuristic, which allows backtracking when a selected path leads to no improvement.
Constructor 
__init__(
    self,
    estimator: Any = None,
    direction: Literal['forward', 'backward', 'bidirectional'] = 'forward',
    search_termination: int = 5,
    cv: int = 5,
    random_state: Optional[int] = None,
)

Overview

Best-first search uses a priority queue to explore the most promising feature subsets first. Unlike greedy sequential selection, it can jump back to a previously explored node if it looks more promising than the current expansion.

Search Process

  1. Maintain a list of "open" nodes (feature subsets) ranked by their CV score.
  2. Expand the best node by adding/removing one feature.
  3. If the best score hasn't improved for search_termination expansions,
stop and return the overall best subset.

This strategy balances between greedy search and exhaustive exploration.

Parameters

estimator
object
A supervised learning estimator with fit and predict methods.
direction
{"forward", "backward", "bidirectional"} = "forward"

Search direction:

  • •"forward": Start with no features.
  • •"backward": Start with all features.
  • •"bidirectional": Can add or remove features at any step.
search_termination
int = 5
Maximum number of consecutive non-improving expansions allowed before the search terminates.
cv
int = 5
Number of cross-validation folds for subset evaluation.
random_state
int
Random seed for cross-validation splits.

Attributes

n_features_selected_
int
Number of features in the best subset found.

Notes

Complexity:
  • Can be highly variable depending on search_termination.
  • Generally more expensive than SequentialFeatureSelector but potentially
more effective at finding global optima. When to use:
  • When greedy selection fails to find a good subset.
  • When you have moderate number of features and computational budget.

See Also

SequentialFeatureSelector

Pure greedy selection.
CFSSelector

Correlation-based filter.

Perform best-first search for 10 nodes:

python 
>>> from tuiml.features.selection import BestFirstSelector
>>> from tuiml.algorithms.trees import DecisionTreeClassifier
>>> import numpy as np
>>> X, y = np.random.randn(20, 8), np.random.randint(0, 2, 20)
>>> selector = BestFirstSelector(
...     estimator=DecisionTreeClassifier(),
...     direction='forward',
...     search_termination=3
... )
>>> X_new = selector.fit_transform(X, y)
>>> print(f"Selected {selector.n_features_selected_} features")

Methods

fit (self, X: np.ndarray, y: Optional[np.ndarray]=None) -> 'BestFirstSelector'

Fit the feature selector using best-first search.

Parameters
X
ndarray of shape (n_samples, n_features)
Training data.
y
ndarray of shape (n_samples,)
Target values.
Returns
self
BestFirstSelector
The fitted selector.
transform (self, X: np.ndarray) -> np.ndarray

Transform X by selecting the best-first-chosen features.

Parameters
X
ndarray of shape (n_samples, n_features)
Input data.
Returns
X_new
ndarray of shape (n_samples, n_selected_features)
Data with only the selected features.
get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for constructor parameters.