Bagging (Bootstrap Aggregating) ensemble implementations for classification and regression.
Classes
BaggingClassifier for bootstrap aggregating ensemble classification.
BaggingClassifier improves stability and accuracy by training multiple base classifiers on different bootstrap samples of the training data and combining their predictions through majority voting.
Constructor
__init__( self, base_classifier: Any = 'DecisionTreeClassifier', n_estimators: int = 10, bag_size_percent: int = 100, random_state: Optional[int] = None, n_jobs: int = 1, )
Overview
The algorithm proceeds as follows:
- For each of T ensemble members:
- To predict, aggregate predictions from all T classifiers via majority vote
Theory
Each bootstrap sample S_t is drawn with replacement from the original dataset D of size n:
S_t = \{(x_{i_1}, y_{i_1}), \ldots, (x_{i_{n'}}, y_{i_{n'}})\}, \quad i_j \sim \text{Uniform}(1, n)
where n' = n \cdot \text{bag\_size\_percent} / 100.
The final ensemble prediction uses majority voting:
H(x) = \arg\max_{k} \sum_{t=1}^{T} \mathbb{1}[h_t(x) = k]
The variance reduction from bagging is:
\text{Var}(H) = \rho \cdot \sigma^2 + \frac{1 - \rho}{T} \cdot \sigma^2
where \rho is the pairwise correlation between base learners and \sigma^2 is the variance of a single base learner.
Parameters
base_classifier
str or class
= 'DecisionTreeClassifier'
The base classifier to use. Unstable learners (e.g., decision trees) benefit most from bagging.
n_estimators
int
= 10
The number of base classifiers in the ensemble. More estimators generally improve accuracy at the cost of computation time.
bag_size_percent
int
= 100
Size of each bootstrap sample as a percentage of the training set. 100 means each bag is the same size as the original dataset.
random_state
int or None
= None
Random seed for reproducibility.
n_jobs
int
= 1
The number of jobs to run in parallel for fitting base classifiers.
-1 means use all available processors.
Attributes
estimators_
list
The collection of fitted base classifiers.
classes_
np.ndarray
The unique class labels discovered during
fit().
Notes
Complexity:
- Training: O(T \cdot n' \cdot C_{\text{base}}) where T = n_estimators,
- Prediction: O(T \cdot C_{\text{predict}}) per sample
- When the base learner is unstable (high variance), such as decision trees
- When you want to reduce overfitting without increasing bias
- When parallel training is desirable (each estimator is independent)
- As a building block for more complex ensemble methods (e.g., Random Forest)
References
Breiman1996
Breiman, L. (1996).
Bagging Predictors.
Machine Learning, 24(2), 123-140.
DOI: 10.1007/BF00058655
Breiman1996b
Breiman, L. (1996).
Heuristics of Instability and Stabilization in Model Selection.
The Annals of Statistics, 24(6), 2350-2383.
Basic usage for classification with bootstrap aggregating:
python
>>> from tuiml.algorithms.ensemble import BaggingClassifier
>>> import numpy as np
>>>
>>> # Create sample training data
>>> X_train = np.array([[1, 2], [2, 3], [3, 1], [4, 3], [5, 2]])
>>> y_train = np.array([0, 0, 1, 1, 1])
>>>
>>> # Fit the Bagging classifier
>>> clf = BaggingClassifier(base_classifier='DecisionTreeClassifier', n_estimators=10)
>>> clf.fit(X_train, y_train)
BaggingClassifier(...)
>>> predictions = clf.predict(X_train)Methods
get_parameter_schema
(cls) -> Dict[str, Dict[str, Any]]
get_capabilities
(cls) -> List[str]
get_complexity
(cls) -> str
get_references
(cls) -> List[str]
__repr__
(self) -> str
BaggingRegressor for bootstrap aggregating ensemble regression.
BaggingRegressor improves stability and accuracy by training multiple base regressors on different bootstrap samples of the training data and combining their predictions through averaging.
Constructor
__init__( self, base_regressor: Any = 'GradientBoostingRegressor', n_estimators: int = 10, bag_size_percent: int = 100, random_state: Optional[int] = None, n_jobs: int = 1, )
Overview
The algorithm proceeds as follows:
- For each of T ensemble members:
- To predict, average predictions from all T regressors
Theory
Each bootstrap sample S_t is drawn with replacement from the original dataset D of size n:
S_t = \{(x_{i_1}, y_{i_1}), \ldots, (x_{i_{n'}}, y_{i_{n'}})\}, \quad i_j \sim \text{Uniform}(1, n)
where n' = n \cdot \text{bag\_size\_percent} / 100.
The final ensemble prediction averages across all base regressors:
H(x) = \frac{1}{T} \sum_{t=1}^{T} h_t(x)
The variance reduction from bagging is:
\text{Var}(H) = \rho \cdot \sigma^2 + \frac{1 - \rho}{T} \cdot \sigma^2
where \rho is the pairwise correlation between base learners and \sigma^2 is the variance of a single base learner.
Parameters
base_regressor
str or class
= 'GradientBoostingRegressor'
The base regressor to use. Unstable learners (e.g., decision trees) benefit most from bagging.
n_estimators
int
= 10
The number of base regressors in the ensemble. More estimators generally improve accuracy at the cost of computation time.
bag_size_percent
int
= 100
Size of each bootstrap sample as a percentage of the training set. 100 means each bag is the same size as the original dataset.
random_state
int or None
= None
Random seed for reproducibility.
n_jobs
int
= 1
The number of jobs to run in parallel for fitting base regressors.
-1 means use all available processors.
Attributes
estimators_
list
The collection of fitted base regressors.
Notes
Complexity:
- Training: O(T \cdot n' \cdot C_{\text{base}}) where T = n_estimators,
- Prediction: O(T \cdot C_{\text{predict}}) per sample
- When the base learner is unstable (high variance), such as decision trees
- When you want to reduce overfitting without increasing bias
- When parallel training is desirable (each estimator is independent)
- When prediction averaging can smooth out individual model noise
References
Breiman1996
Breiman, L. (1996).
Bagging Predictors.
Machine Learning, 24(2), 123-140.
DOI: 10.1007/BF00058655
See Also
Basic usage for regression with bootstrap aggregating:
python
>>> from tuiml.algorithms.ensemble import BaggingRegressor
>>> import numpy as np
>>>
>>> # Create sample training data
>>> X_train = np.array([[1, 2], [2, 3], [3, 1], [4, 3], [5, 2]])
>>> y_train = np.array([1.5, 2.3, 3.1, 4.2, 5.0])
>>>
>>> # Fit the Bagging regressor
>>> reg = BaggingRegressor(n_estimators=10, random_state=42)
>>> reg.fit(X_train, y_train)
BaggingRegressor(...)
>>> predictions = reg.predict(X_train)