API Reference / algorithms / ensemble /

stacking.py

Stacking (Stacked Generalization) ensemble implementations for classification and regression.

Classes

StackingClassifier

class algorithms.ensemble.stacking.StackingClassifier(Classifier)

StackingClassifier for stacked generalization meta-learning.

StackingClassifier trains a meta-classifier on the outputs of diverse base classifiers. Level-one meta-features are generated using cross-validation to avoid overfitting in the stacking process.
Constructor 
__init__(
    self,
    classifiers: List[Any] = None,
    meta_classifier: Any = 'LogisticRegression',
    num_folds: int = 10,
    use_probabilities: bool = True,
    random_state: Optional[int] = None,
)

Overview

The algorithm proceeds in two levels:

  1. Level-0 (Base classifiers):
a. Use k-fold cross-validation on the training data b. For each fold, train all base classifiers on the training portion c. Generate meta-features (predictions or probabilities) on the validation portion
  1. Level-1 (Meta-classifier):
a. Train all base classifiers on the full training set (for future prediction) b. Train the meta-classifier on the cross-validated meta-features
  1. To predict new data, generate meta-features from level-0 classifiers
and pass them through the level-1 meta-classifier

Theory

Given L base classifiers h_1, \ldots, h_L, the meta-features for sample x_i are constructed as:

z_i = \bigl(h_1(x_i), h_2(x_i), \ldots, h_L(x_i)\bigr)

When use_probabilities=True, each h_l(x_i) is a K-dimensional probability vector, yielding L \times K meta-features.

The meta-classifier g learns the combination function:

H(x) = g(z) = g\bigl(h_1(x), h_2(x), \ldots, h_L(x)\bigr)

Cross-validation prevents overfitting by ensuring that meta-features for training sample x_i are generated by classifiers that did not see x_i during training.

Parameters

classifiers
list, 'DecisionTreeClassifier'] = ['
The base classifiers to use. Can be classifier names (strings), classes, or instances.
meta_classifier
str or object = 'LogisticRegression'
The meta-classifier used to combine base classifier predictions.
num_folds
int = 10
The number of folds for cross-validation to generate meta-features. Must be at least 2.
use_probabilities
bool = True
Whether to use class probabilities (True) or one-hot encoded hard labels (False) as meta-features.
random_state
int or None = None
Random seed for reproducibility of the cross-validation splits.

Attributes

base_estimators_
list
The collection of fitted base classifiers (trained on full data).
meta_estimator_
Classifier
The fitted meta-classifier.
classes_
np.ndarray
The unique class labels discovered during fit().

Notes

Complexity:

  • Training: O(k \cdot \sum_{l=1}^{L} C_l + C_{\text{meta}}) where k = num_folds,
C_l = complexity of base classifier l, C_{\text{meta}} = meta-classifier complexity
  • Prediction: O(\sum_{l=1}^{L} C_l^{\text{pred}} + C_{\text{meta}}^{\text{pred}}) per sample
When to use StackingClassifier:
  • When you have diverse base classifiers with complementary strengths
  • When simple voting does not capture complex relationships between classifiers
  • When you have enough data to support cross-validation without underfitting
  • When you want the meta-learner to learn optimal combination weights automatically

References

Wolpert1992
Wolpert, D.H. (1992). Stacked Generalization. Neural Networks, 5(2), 241-259. DOI: 10.1016/S0893-6080(05)80023-1
Breiman1996c
Breiman, L. (1996). Stacked Regressions. Machine Learning, 24(1), 49-64. DOI: 10.1007/BF00117832
Ting1999
Ting, K.M. and Witten, I.H. (1999). Issues in Stacked Generalization. Journal of Artificial Intelligence Research, 10, 271-289.

See Also

VotingClassifier

Simpler combination via voting rules.
AdaBoostClassifier

Sequential boosting ensemble.
BaggingClassifier

Parallel bootstrap aggregating.

Basic usage for stacked generalization:

python 
>>> from tuiml.algorithms.ensemble import StackingClassifier
>>> import numpy as np
>>>
>>> # Create sample training data
>>> X_train = np.array([[1, 2], [2, 3], [3, 1], [4, 3], [5, 2]])
>>> y_train = np.array([0, 0, 1, 1, 1])
>>>
>>> # Fit the Stacking classifier
>>> clf = StackingClassifier(
...     classifiers=['NaiveBayesClassifier', 'DecisionTreeClassifier'],
...     meta_classifier='LogisticRegression'
... )
>>> clf.fit(X_train, y_train)
StackingClassifier(...)
>>> predictions = clf.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]
get_capabilities (cls) -> List[str]
get_complexity (cls) -> str
get_references (cls) -> List[str]
fit (self, X: np.ndarray, y: np.ndarray) -> 'StackingClassifier'

Fit the StackingClassifier classifier.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data.
y
np.ndarray of shape (n_samples,)
Target labels.
Returns
self
StackingClassifier
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test data.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test data.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
The class probabilities of the input samples.
__repr__ (self) -> str

StackingRegressor

class algorithms.ensemble.stacking.StackingRegressor(Regressor)

StackingRegressor for stacked generalization meta-learning regression.

StackingRegressor trains a meta-regressor on the outputs of diverse base regressors. Level-one meta-features are generated using cross-validation to avoid overfitting in the stacking process.
Constructor 
__init__(
    self,
    regressors: List[Any] = None,
    meta_regressor: Any = 'GradientBoostingRegressor',
    num_folds: int = 10,
    random_state: Optional[int] = None,
)

Overview

The algorithm proceeds in two levels:

  1. Level-0 (Base regressors):
a. Use k-fold cross-validation on the training data b. For each fold, train all base regressors on the training portion c. Generate meta-features (predictions) on the validation portion
  1. Level-1 (Meta-regressor):
a. Train all base regressors on the full training set (for future prediction) b. Train the meta-regressor on the cross-validated meta-features
  1. To predict new data, generate meta-features from level-0 regressors
and pass them through the level-1 meta-regressor

Theory

Given L base regressors h_1, \ldots, h_L, the meta-features for sample x_i are constructed as:

z_i = \bigl(h_1(x_i), h_2(x_i), \ldots, h_L(x_i)\bigr)

The meta-regressor g learns the combination function:

H(x) = g(z) = g\bigl(h_1(x), h_2(x), \ldots, h_L(x)\bigr)

Cross-validation prevents overfitting by ensuring that meta-features for training sample x_i are generated by regressors that did not see x_i during training.

Parameters

regressors
list = ['
The base regressors to use. Can be regressor names (strings), classes, or instances.
meta_regressor
str or object = 'GradientBoostingRegressor'
The meta-regressor used to combine base regressor predictions.
num_folds
int = 10
The number of folds for cross-validation to generate meta-features. Must be at least 2.
random_state
int or None = None
Random seed for reproducibility of the cross-validation splits.

Attributes

base_estimators_
list
The collection of fitted base regressors (trained on full data).
meta_estimator_
Regressor
The fitted meta-regressor.

Notes

Complexity:

  • Training: O(k \cdot \sum_{l=1}^{L} C_l + C_{\text{meta}}) where k = num_folds,
C_l = complexity of base regressor l, C_{\text{meta}} = meta-regressor complexity
  • Prediction: O(\sum_{l=1}^{L} C_l^{\text{pred}} + C_{\text{meta}}^{\text{pred}}) per sample
When to use StackingRegressor:
  • When you have diverse base regressors with complementary strengths
  • When simple averaging does not capture complex relationships between regressors
  • When you have enough data to support cross-validation without underfitting
  • When you want the meta-learner to learn optimal combination weights automatically

References

Wolpert1992
Wolpert, D.H. (1992). Stacked Generalization. Neural Networks, 5(2), 241-259. DOI: 10.1016/S0893-6080(05)80023-1
Breiman1996c
Breiman, L. (1996). Stacked Regressions. Machine Learning, 24(1), 49-64. DOI: 10.1007/BF00117832

See Also

StackingClassifier

Stacked generalization for classification.
VotingRegressor

Simpler combination via averaging.
BaggingRegressor

Parallel bootstrap aggregating for regression.

Basic usage for stacked generalization regression:

python 
>>> from tuiml.algorithms.ensemble import StackingRegressor
>>> import numpy as np
>>>
>>> # Create sample training data
>>> X_train = np.array([[1, 2], [2, 3], [3, 1], [4, 3], [5, 2]])
>>> y_train = np.array([1.5, 2.3, 3.1, 4.2, 5.0])
>>>
>>> # Fit the Stacking regressor
>>> reg = StackingRegressor(
...     regressors=['GradientBoostingRegressor'],
...     meta_regressor='GradientBoostingRegressor'
... )
>>> reg.fit(X_train, y_train)
StackingRegressor(...)
>>> predictions = reg.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return JSON Schema for constructor parameters.

get_capabilities (cls) -> List[str]

Return algorithm capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'StackingRegressor'

Fit the StackingRegressor.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
StackingRegressor
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test data.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted target values.
score (self, X: np.ndarray, y: np.ndarray) -> float

Compute R-squared score.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
y
np.ndarray of shape (n_samples,)
True target values.
Returns
score
float
R-squared score.
__repr__ (self) -> str

Return string representation.