API Reference / algorithms / ensemble /

adaboost.py

AdaBoostClassifier classifier implementation.

Classes

AdaBoostClassifier

class algorithms.ensemble.adaboost.AdaBoostClassifier(Classifier)

AdaBoostClassifier for adaptive boosting in multiclass classification.

AdaBoostClassifier (also known as AdaBoost.M1) builds an ensemble by iteratively training weak learners on weighted versions of the data, focusing on previously misclassified instances, and combining their predictions using weighted majority voting.
Constructor 
__init__(
    self,
    base_classifier: Any = 'DecisionStumpClassifier',
    n_estimators: int = 50,
    weight_threshold: float = 100,
    random_state: Optional[int] = None,
)

Overview

The algorithm proceeds as follows:

  1. Initialize uniform sample weights w_i = 1/n for all training instances
  2. For each boosting iteration t = 1, \ldots, T:
a. Train a weak learner h_t on the weighted training data b. Compute the weighted error \epsilon_t c. Calculate the estimator weight \alpha_t d. Update sample weights, increasing weights on misclassified instances
  1. Combine all weak learners via weighted majority vote

Theory

The weighted classification error at iteration t is:

\epsilon_t = \sum_{i: h_t(x_i) \neq y_i} w_i

The estimator weight for the multiclass case (SAMME) is:

\alpha_t = \ln\!\left(\frac{1 - \epsilon_t}{\epsilon_t}\right) + \ln(K - 1)

where K is the number of classes. Sample weights are updated as:

w_i \leftarrow w_i \cdot \exp\!\bigl(\alpha_t \cdot \mathbb{1}[h_t(x_i) \neq y_i]\bigr)

The final prediction is obtained by weighted majority vote:

H(x) = \arg\max_{k} \sum_{t=1}^{T} \alpha_t \cdot \mathbb{1}[h_t(x) = k]

Parameters

base_classifier
str or class = 'DecisionStumpClassifier'
The base classifier to use as weak learner. Typically a simple classifier such as a decision stump.
n_estimators
int = 50
The maximum number of estimators at which boosting is terminated. More estimators can improve accuracy but may lead to overfitting.
weight_threshold
float = 100
Weight threshold for triggering resampling. When the ratio between maximum and minimum sample weights exceeds this value, the data is resampled according to the current weight distribution.
random_state
int or None = None
Random seed for reproducibility. Controls the resampling process.

Attributes

estimators_
list
The collection of fitted weak learners.
estimator_weights_
np.ndarray
Weights :math:`\alpha_t` assigned to each estimator.
classes_
np.ndarray
The unique class labels discovered during fit().

Notes

Complexity:

  • Training: O(T \cdot n \cdot C_{\text{base}}) where T = n_estimators,
n = number of samples, C_{\text{base}} = base classifier complexity
  • Prediction: O(T \cdot C_{\text{predict}}) per sample
When to use AdaBoostClassifier:
  • When a simple base learner (e.g., decision stump) needs to be boosted
  • Binary or multiclass classification tasks with moderate noise
  • When you want an interpretable ensemble (weighted sum of simple rules)
  • When training data is relatively clean (AdaBoost is sensitive to outliers)

References

Freund1996
Freund, Y. and Schapire, R.E. (1996). Experiments with a New Boosting Algorithm. Proceedings of the 13th International Conference on Machine Learning, pp. 148-156.
Freund1997
Freund, Y. and Schapire, R.E. (1997). A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. Journal of Computer and System Sciences, 55(1), 119-139. DOI: 10.1006/jcss.1997.1504
Hastie2009
Hastie, T., Tibshirani, R. and Friedman, J. (2009). The Elements of Statistical Learning. Springer, Chapter 10.

See Also

BaggingClassifier

Bootstrap aggregating ensemble method.
VotingClassifier

Combines classifiers via voting rules.
StackingClassifier

Stacked generalization meta-classifier.

Basic usage for multiclass classification with AdaBoost:

python 
>>> from tuiml.algorithms.ensemble import AdaBoostClassifier
>>> import numpy as np
>>>
>>> # Create sample training data
>>> X_train = np.array([[1, 2], [2, 3], [3, 1], [4, 3], [5, 2]])
>>> y_train = np.array([0, 0, 1, 1, 1])
>>>
>>> # Fit the AdaBoost classifier
>>> clf = AdaBoostClassifier(n_estimators=50, random_state=42)
>>> clf.fit(X_train, y_train)
AdaBoostClassifier(...)
>>> predictions = clf.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]
get_capabilities (cls) -> List[str]
get_complexity (cls) -> str
get_references (cls) -> List[str]
fit (self, X: np.ndarray, y: np.ndarray) -> 'AdaBoostClassifier'

Fit the AdaBoostClassifier.M1 classifier.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data.
y
np.ndarray of shape (n_samples,)
Target labels.
Returns
self
AdaBoostClassifier
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test data.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test data.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
The class probabilities of the input samples.
__repr__ (self) -> str

AdaBoostRegressor

class algorithms.ensemble.adaboost.AdaBoostRegressor(Regressor)

AdaBoost.R2 regressor for adaptive boosting in regression tasks.

AdaBoostRegressor implements the AdaBoost.R2 algorithm, which builds an ensemble by iteratively training base regressors on weighted versions of the data. Instances with larger prediction errors receive higher weights in subsequent iterations, and the final prediction is computed as a weighted median of the individual estimator predictions.
Constructor 
__init__(
    self,
    base_regressor: Any = 'GradientBoostingRegressor',
    n_estimators: int = 10,
    random_state: Optional[int] = None,
)

Overview

The algorithm proceeds as follows:

  1. Initialize uniform sample weights w_i = 1/n
  2. For each boosting iteration t = 1, \ldots, T:
a. Fit a base regressor h_t on the weighted training data b. Compute predictions and the maximum absolute error D c. Compute relative losses L_i = |y_i - h_t(x_i)| / D d. Compute weighted average loss \bar{L} e. If \bar{L} \geq 0.5, stop boosting f. Compute \beta_t = \bar{L} / (1 - \bar{L}) g. Update weights: w_i \leftarrow w_i \cdot \beta_t^{1 - L_i} h. Normalize weights
  1. Combine estimators using the weighted median

Theory

The relative loss for each sample at iteration t is:

L_i = \frac{|y_i - h_t(x_i)|}{D_t}

where D_t = \max_i |y_i - h_t(x_i)| is the maximum absolute error. The weighted average loss is:

\bar{L}_t = \sum_{i=1}^{n} w_i \cdot L_i

The estimator confidence is:

\beta_t = \frac{\bar{L}_t}{1 - \bar{L}_t}

Weights are updated as:

w_i \leftarrow w_i \cdot \beta_t^{1 - L_i}

The final prediction is the weighted median of all estimator predictions, using \log(1/\beta_t) as the estimator weight.

Parameters

base_regressor
str or class = 'GradientBoostingRegressor'
The base regressor to use as weak learner. Can be a string name (resolved via the hub registry) or a class/instance.
n_estimators
int = 10
The maximum number of boosting iterations.
random_state
int or None = None
Random seed for reproducibility.

Attributes

estimators_
list
The collection of fitted base regressors.
estimator_weights_
np.ndarray
Log-confidence weights :math:`\log(1/\beta_t)` for each estimator.

Notes

Complexity:

  • Training: O(T \cdot n \cdot C_{\text{base}}) where T = n_estimators
  • Prediction: O(T \cdot n \cdot \log T) per batch due to weighted median
When to use AdaBoostRegressor:
  • When a simple base regressor needs to be boosted for better accuracy
  • Regression tasks with moderate noise levels
  • When you want an interpretable ensemble of simple models
  • When training data is relatively clean (AdaBoost is sensitive to outliers)

References

Drucker1997
Drucker, H. (1997). Improving Regressors Using Boosting Techniques. Proceedings of the 14th International Conference on Machine Learning, pp. 107-115.
Freund1997
Freund, Y. and Schapire, R.E. (1997). A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. Journal of Computer and System Sciences, 55(1), 119-139. DOI: 10.1006/jcss.1997.1504

See Also

AdaBoostClassifier

AdaBoost for classification tasks.
GradientBoostingRegressor

Gradient boosting for regression.
BaggingRegressor

Bootstrap aggregating for regression.

Basic usage for regression with AdaBoost.R2:

python 
>>> from tuiml.algorithms.ensemble import AdaBoostRegressor
>>> import numpy as np
>>> X_train = np.array([[1], [2], [3], [4], [5]])
>>> y_train = np.array([1.0, 4.0, 9.0, 16.0, 25.0])
>>> reg = AdaBoostRegressor(n_estimators=50, random_state=42)
>>> reg.fit(X_train, y_train)
AdaBoostRegressor(...)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return regressor capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'AdaBoostRegressor'

Fit the AdaBoost.R2 regressor.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training data.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
AdaBoostRegressor
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values using weighted median of estimators.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test data.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted target values.
score (self, X: np.ndarray, y: np.ndarray) -> float

Compute the R-squared (coefficient of determination) score.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
y
np.ndarray of shape (n_samples,)
True target values.
Returns
score
float
R-squared score.
__repr__ (self) -> str

String representation.