XGBoost (eXtreme Gradient Boosting) implementation.
Classes
XGBoost classifier for high-performance gradient boosting.
XGBoost (eXtreme Gradient Boosting) uses a regularized gradient boosting framework to build an ensemble of decision trees with sparsity awareness and cache-aware access patterns to achieve state-of-the-art results.
Constructor
__init__( self, n_estimators: int = 100, max_depth: int = 6, learning_rate: float = 0.3, subsample: float = 1.0, colsample_bytree: float = 1.0, min_child_weight: float = 1.0, gamma: float = 0.0, reg_alpha: float = 0.0, reg_lambda: float = 1.0, objective: str = 'binary:logistic', random_state: Optional[int] = None, )
Overview
The algorithm builds an additive ensemble of decision trees:
- Initialize the model with a constant prediction (e.g., log-odds for classification)
- For each boosting round, compute the negative gradient (pseudo-residuals) and
- Fit a new regression tree to the negative gradient using an
- Prune the tree using the \gamma (minimum loss reduction) threshold
- Add the new tree to the ensemble, scaled by the learning rate \eta
- Repeat until the specified number of boosting rounds is reached
Theory
At boosting round t, XGBoost minimizes the regularized objective:
\mathcal{L}^{(t)} = \sum_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega(f_t)
where the regularization term is:
\Omega(f) = \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T} w_j^2 + \alpha \sum_{j=1}^{T} |w_j|
Using a second-order Taylor expansion, the objective becomes:
\tilde{\mathcal{L}}^{(t)} \approx \sum_{i=1}^{n} \left[ g_i f_t(x_i) + \frac{1}{2} h_i f_t^2(x_i) \right] + \Omega(f_t)
where g_i = \partial_{\hat{y}} l(y_i, \hat{y}_i^{(t-1)}) and h_i = \partial^2_{\hat{y}} l(y_i, \hat{y}_i^{(t-1)}) are the first and second order gradients. The optimal leaf weight for leaf j is:
w_j^* = -\frac{\sum_{i \in I_j} g_i}{\sum_{i \in I_j} h_i + \lambda}
Parameters
n_estimators
int
= 100
Number of boosting rounds (trees to build).
max_depth
int
= 6
Maximum tree depth for base learners.
learning_rate
float
= 0.3
Step size shrinkage used in update to prevent overfitting (eta).
subsample
float
= 1.0
Fraction of training samples to randomly sample for each tree.
colsample_bytree
float
= 1.0
Fraction of columns to randomly sample for each tree.
min_child_weight
float
= 1.0
Minimum sum of instance weight (hessian) needed in a child.
gamma
float
= 0.0
Minimum loss reduction required to make a further partition.
reg_alpha
float
= 0.0
L1 regularization term on weights.
reg_lambda
float
= 1.0
L2 regularization term on weights.
objective
str
= "binary:logistic"
Learning objective. Automatically switches to
"multi:softprob" for multi-class tasks.
random_state
int
Seed for reproducibility.
Attributes
model_
xgb.XGBClassifier
The underlying fitted XGBoost model object.
classes_
ndarray of shape (n_classes,)
Unique class labels discovered during
fit().
n_classes_
int
Number of unique classes discovered during
fit().
Notes
Complexity:
- Training: O(T \cdot n \cdot d \cdot D) where T = n_estimators,
- Prediction: O(T \cdot D) per sample
- Structured / tabular classification tasks with moderate-to-large datasets
- When you need built-in handling of missing values
- Competitions and benchmarks where predictive accuracy is paramount
- When L1 and L2 regularization are needed to control model complexity
- Datasets that benefit from second-order gradient optimization
References
Chen2016
Chen, T. and Guestrin, C. (2016).
XGBoost: A Scalable Tree Boosting System.
Proceedings of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pp. 785-794.
DOI: 10.1145/2939672.2939785
Friedman2001
Friedman, J.H. (2001).
Greedy Function Approximation: A Gradient Boosting Machine.
The Annals of Statistics, 29(5), pp. 1189-1232.
DOI: 10.1214/aos/1013203451
See Also
Train an XGBoost classifier on a binary classification task:
python
>>> from tuiml.algorithms.gradient_boosting import XGBoostClassifier
>>> import numpy as np
>>>
>>> X_train = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
>>> y_train = np.array([0, 0, 1, 1])
>>> clf = XGBoostClassifier(n_estimators=100, learning_rate=0.1)
>>> clf.fit(X_train, y_train)
>>> y_pred = clf.predict(X_train)Methods
__repr__
(self) -> str
XGBoost regressor for high-performance gradient boosting on continuous targets.
Implementation of the XGBoost regression algorithm using regularized gradient boosted decision trees with second-order optimization.
Constructor
__init__( self, n_estimators: int = 100, max_depth: int = 6, learning_rate: float = 0.3, subsample: float = 1.0, colsample_bytree: float = 1.0, min_child_weight: float = 1.0, gamma: float = 0.0, reg_alpha: float = 0.0, reg_lambda: float = 1.0, objective: str = 'reg:squarederror', random_state: Optional[int] = None, )
Overview
The regression variant follows the same additive training procedure:
- Initialize predictions with a constant value (e.g., mean of targets)
- For each boosting round, compute the gradient and Hessian of the
- Fit a regression tree to the negative gradient using approximate
- Prune the tree using the \gamma threshold and regularization
- Update predictions by adding the new tree scaled by learning rate \eta
- Repeat until all boosting rounds are completed
Theory
For the default squared-error objective, the loss for sample i is:
l(y_i, \hat{y}_i) = \frac{1}{2}(y_i - \hat{y}_i)^2
The gradients are g_i = \hat{y}_i - y_i and h_i = 1. The regularized objective at round t is:
\mathcal{L}^{(t)} = \sum_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T} w_j^2 + \alpha \sum_{j=1}^{T} |w_j|
where T is the number of leaves, w_j are leaf weights, \lambda is L2 regularization, and \alpha is L1 regularization.
Parameters
n_estimators
int
= 100
Number of boosting rounds (trees to build).
max_depth
int
= 6
Maximum tree depth for base learners.
learning_rate
float
= 0.3
Step size shrinkage (eta) to prevent overfitting.
subsample
float
= 1.0
Fraction of training samples to randomly sample for each tree.
colsample_bytree
float
= 1.0
Fraction of columns to randomly sample for each tree.
min_child_weight
float
= 1.0
Minimum sum of instance weight needed in a child.
gamma
float
= 0.0
Minimum loss reduction required to make a split.
reg_alpha
float
= 0.0
L1 regularization term on weights.
reg_lambda
float
= 1.0
L2 regularization term on weights.
objective
str
= "reg:squarederror"
Regression learning objective.
random_state
int
Seed for reproducibility.
Attributes
model_
xgb.XGBRegressor
The underlying fitted XGBoost regressor object.
Notes
Complexity:
- Training: O(T \cdot n \cdot d \cdot D) where T = n_estimators,
- Prediction: O(T \cdot D) per sample
- Structured / tabular regression tasks with moderate-to-large datasets
- When the data contains missing values that should be handled natively
- When L1 and L2 regularization are needed for controlling overfitting
- Benchmarks where predictive accuracy on continuous targets is paramount
References
Chen2016
Chen, T. and Guestrin, C. (2016).
XGBoost: A Scalable Tree Boosting System.
Proceedings of the 22nd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining, pp. 785-794.
DOI: 10.1145/2939672.2939785
Friedman2001
Friedman, J.H. (2001).
Greedy Function Approximation: A Gradient Boosting Machine.
The Annals of Statistics, 29(5), pp. 1189-1232.
DOI: 10.1214/aos/1013203451
See Also
Train an XGBoost regressor on a simple regression task:
python
>>> from tuiml.algorithms.gradient_boosting import XGBoostRegressor
>>> import numpy as np
>>>
>>> X_train = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
>>> y_train = np.array([1.5, 3.5, 5.5, 7.5])
>>> reg = XGBoostRegressor(n_estimators=100, max_depth=5)
>>> reg.fit(X_train, y_train)
>>> y_pred = reg.predict(X_train)Methods
__repr__
(self) -> str