API Reference / algorithms / gradient_boosting /

lightgbm.py

LightGBM (Light Gradient Boosting Machine) implementation.

Classes

LightGBMClassifier

class algorithms.gradient_boosting.lightgbm.LightGBMClassifier(Classifier)

LightGBM classifier for distributed, high-performance gradient boosting.

LightGBM is a gradient boosting framework that uses leaf-wise tree growth and histogram-based split finding. It is designed for distributed training with fast speed, low memory usage, and support for billion-level data.
Constructor 
__init__(
    self,
    n_estimators: int = 100,
    max_depth: int = ...,
    learning_rate: float = 0.1,
    num_leaves: int = 31,
    subsample: float = 1.0,
    colsample_bytree: float = 1.0,
    reg_alpha: float = 0.0,
    reg_lambda: float = 0.0,
    min_child_samples: int = 20,
    min_split_gain: float = 0.0,
    verbose: int = ...,
    random_state: Optional[int] = None,
)

Overview

The algorithm builds an ensemble of decision trees using leaf-wise growth:

  1. Initialize the model with a constant prediction (e.g., log-odds for
classification)
  1. For each boosting iteration, compute the negative gradient of the
loss function for every training sample
  1. Build histograms of feature values using gradient-based one-side
sampling (GOSS) and exclusive feature bundling (EFB)
  1. Grow the tree leaf-wise by splitting the leaf with the highest
gain, rather than level-wise, controlled by num_leaves
  1. Add the new tree to the ensemble, scaled by the learning rate
  2. Repeat until the specified number of boosting rounds is reached

Theory

At each boosting round t, LightGBM minimizes:

\mathcal{L}^{(t)} = \sum_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \Omega(f_t)

where the regularization term is:

\Omega(f) = \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T} w_j^2 + \alpha \sum_{j=1}^{T} |w_j|

GOSS keeps all instances with large gradients and randomly samples from instances with small gradients, multiplying them by \frac{1-a}{b} to compensate:

\tilde{V}_j(d) = \frac{1}{n} \left( \sum_{x_i \in A_l} g_i + \frac{1-a}{b} \sum_{x_i \in B_l} g_i \right)^2

where A is the set of top-a instances and B is sampled from the remaining instances with ratio b.

Parameters

n_estimators
int = 100
Number of boosting iterations (trees to build).
max_depth
int = -1
Maximum tree depth. -1 means no limit.
learning_rate
float = 0.1
Step size shrinkage to prevent overfitting.
num_leaves
int = 31
Maximum number of leaves in one tree.
subsample
float = 1.0
Fraction of samples to randomly sample for each tree.
colsample_bytree
float = 1.0
Fraction of features to randomly sample for each tree.
reg_alpha
float = 0.0
L1 regularization term on weights.
reg_lambda
float = 0.0
L2 regularization term on weights.
min_child_samples
int = 20
Minimum number of samples required in a leaf node.
min_split_gain
float = 0.0
Minimum loss reduction required to make a further partition.
verbose
int = -1
Verbosity level. -1: Quiet, 0: Warnings, 1: Info.
random_state
int
Seed for reproducibility.

Attributes

model_
lgb.LGBMClassifier
The underlying fitted LightGBM model object.
classes_
ndarray of shape (n_classes,)
Unique class labels discovered during fit().
n_classes_
int
Number of unique classes discovered during fit().

Notes

Complexity:

  • Training: O(T \cdot n \cdot d \cdot L) where T = n_estimators,
n = n_samples, d = n_features, L = num_leaves. In practice, GOSS and EFB significantly reduce effective n and d.
  • Prediction: O(T \cdot L) per sample
When to use LightGBMClassifier:
  • Large-scale classification tasks where training speed is critical
  • High-dimensional datasets where exclusive feature bundling reduces cost
  • When memory efficiency is important (histogram-based approach)
  • Distributed training scenarios across multiple machines

References

Ke2017
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. and Liu, T.Y. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. Advances in Neural Information Processing Systems (NeurIPS), 30, pp. 3146-3154.
Friedman2001
Friedman, J.H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics, 29(5), pp. 1189-1232. DOI: 10.1214/aos/1013203451

See Also

LightGBMRegressor

LightGBM for regression tasks.
XGBoostClassifier

XGBoost classifier with second-order optimization.
CatBoostClassifier

CatBoost classifier with native categorical support.

Train a LightGBM classifier on a binary classification task:

python 
>>> from tuiml.algorithms.gradient_boosting import LightGBMClassifier
>>> import numpy as np
>>>
>>> X_train = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
>>> y_train = np.array([0, 0, 1, 1])
>>> clf = LightGBMClassifier(n_estimators=100, num_leaves=31)
>>> clf.fit(X_train, y_train)
>>> y_pred = clf.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for algorithm parameters.

get_capabilities (cls) -> List[str]

Return supported capabilities.

get_complexity (cls) -> str

Return complexity analysis.

get_references (cls) -> List[str]

Return academic citations.

fit (self, X: np.ndarray, y: np.ndarray) -> 'LightGBMClassifier'

Fit the LightGBM classifier to training data.

Parameters
X
ndarray of shape (n_samples, n_features)
Training feature matrix.
y
ndarray of shape (n_samples,)
Target class labels.
Returns
self
LightGBMClassifier
Fitted estimator.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
ndarray of shape (n_samples, n_features)
Samples to classify.
Returns
ndarray of shape (n_samples,)
Predicted class labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
ndarray of shape (n_samples, n_features)
Samples to predict probabilities for.
Returns
ndarray of shape (n_samples, n_classes)
Class probability matrix.
__repr__ (self) -> str

LightGBMRegressor

class algorithms.gradient_boosting.lightgbm.LightGBMRegressor(Regressor)

LightGBM regressor for distributed, high-performance gradient boosting on continuous targets.

Implementation of the LightGBM regression algorithm using leaf-wise tree growth and histogram-based split finding for fast, memory-efficient training.
Constructor 
__init__(
    self,
    n_estimators: int = 100,
    max_depth: int = ...,
    learning_rate: float = 0.1,
    num_leaves: int = 31,
    subsample: float = 1.0,
    colsample_bytree: float = 1.0,
    reg_alpha: float = 0.0,
    reg_lambda: float = 0.0,
    min_child_samples: int = 20,
    min_split_gain: float = 0.0,
    verbose: int = ...,
    random_state: Optional[int] = None,
)

Overview

The regression variant follows the same leaf-wise boosting procedure:

  1. Initialize predictions with a constant value (e.g., mean of targets)
  2. For each boosting iteration, compute the negative gradient of the
loss (e.g., squared error) for every training sample
  1. Build feature histograms using GOSS (gradient-based one-side
sampling) and EFB (exclusive feature bundling)
  1. Grow the tree leaf-wise by splitting the leaf with the highest
gain, controlled by num_leaves
  1. Add the new tree to the ensemble, scaled by the learning rate
  2. Repeat until the specified number of boosting rounds is reached

Theory

For the default squared-error objective, the loss for sample i is:

l(y_i, \hat{y}_i) = \frac{1}{2}(y_i - \hat{y}_i)^2

The regularized objective at round t is:

\mathcal{L}^{(t)} = \sum_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \gamma T + \frac{1}{2} \lambda \sum_{j=1}^{T} w_j^2 + \alpha \sum_{j=1}^{T} |w_j|

The optimal leaf weight for leaf j is:

w_j^* = -\frac{\sum_{i \in I_j} g_i}{|I_j| + \lambda}

where g_i is the gradient, \lambda is the L2 regularization term (reg_lambda), and \alpha is the L1 term (reg_alpha).

Parameters

n_estimators
int = 100
Number of boosting iterations (trees to build).
max_depth
int = -1
Maximum tree depth. -1 means no limit.
learning_rate
float = 0.1
Step size shrinkage to prevent overfitting.
num_leaves
int = 31
Maximum number of leaves in one tree.
subsample
float = 1.0
Fraction of samples to randomly sample for each tree.
colsample_bytree
float = 1.0
Fraction of features to randomly sample for each tree.
reg_alpha
float = 0.0
L1 regularization term on weights.
reg_lambda
float = 0.0
L2 regularization term on weights.
min_child_samples
int = 20
Minimum number of samples required in a leaf node.
min_split_gain
float = 0.0
Minimum loss reduction required to make a split.
verbose
int = -1
Verbosity level. -1: Quiet, 0: Warnings, 1: Info.
random_state
int
Seed for reproducibility.

Attributes

model_
lgb.LGBMRegressor
The underlying fitted LightGBM regressor object.

Notes

Complexity:

  • Training: O(T \cdot n \cdot d \cdot L) where T = n_estimators,
n = n_samples, d = n_features, L = num_leaves. GOSS and EFB reduce effective n and d in practice.
  • Prediction: O(T \cdot L) per sample
When to use LightGBMRegressor:
  • Large-scale regression tasks where training speed is critical
  • High-dimensional datasets where exclusive feature bundling reduces cost
  • When memory efficiency is important (histogram-based approach)
  • Distributed training scenarios across multiple machines

References

Ke2017
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q. and Liu, T.Y. (2017). LightGBM: A Highly Efficient Gradient Boosting Decision Tree. Advances in Neural Information Processing Systems (NeurIPS), 30, pp. 3146-3154.
Friedman2001
Friedman, J.H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics, 29(5), pp. 1189-1232. DOI: 10.1214/aos/1013203451

See Also

LightGBMClassifier

LightGBM for classification tasks.
XGBoostRegressor

XGBoost regressor with second-order optimization.
CatBoostRegressor

CatBoost regressor with native categorical support.

Train a LightGBM regressor on a simple regression task:

python 
>>> from tuiml.algorithms.gradient_boosting import LightGBMRegressor
>>> import numpy as np
>>>
>>> X_train = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
>>> y_train = np.array([1.5, 3.5, 5.5, 7.5])
>>> reg = LightGBMRegressor(n_estimators=100, learning_rate=0.05)
>>> reg.fit(X_train, y_train)
>>> y_pred = reg.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for algorithm parameters.

get_capabilities (cls) -> List[str]

Return supported capabilities.

get_complexity (cls) -> str

Return complexity analysis.

get_references (cls) -> List[str]

Return academic citations.

fit (self, X: np.ndarray, y: np.ndarray) -> 'LightGBMRegressor'

Fit the LightGBM regressor to training data.

Parameters
X
ndarray of shape (n_samples, n_features)
Training feature matrix.
y
ndarray of shape (n_samples,)
Target values.
Returns
self
LightGBMRegressor
Fitted estimator.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values for samples.

Parameters
X
ndarray of shape (n_samples, n_features)
Samples to predict.
Returns
ndarray of shape (n_samples,)
Predicted values.
__repr__ (self) -> str