API Reference / algorithms / gradient_boosting /

catboost.py

CatBoost (Categorical Boosting) implementation.

Classes

CatBoostClassifier

class algorithms.gradient_boosting.catboost.CatBoostClassifier(Classifier)

CatBoost classifier with native support for categorical features.

CatBoost is a gradient boosting algorithm that provides advanced handling of categorical features using ordered target statistics, robust regularization, and ordered boosting to reduce prediction shift.
Constructor 
__init__(
    self,
    iterations: int = 100,
    depth: int = 6,
    learning_rate: float = 0.03,
    l2_leaf_reg: float = 3.0,
    border_count: int = 128,
    bagging_temperature: float = 1.0,
    random_strength: float = 1.0,
    cat_features: Optional[List[int]] = None,
    verbose: bool = False,
    random_state: Optional[int] = None,
)

Overview

The algorithm builds an ensemble of symmetric (oblivious) decision trees:

  1. Encode categorical features using ordered target statistics computed
on a random permutation of the training data to avoid target leakage
  1. For each boosting iteration, compute the negative gradient of the
loss function using the ordered boosting scheme
  1. Build a symmetric (oblivious) decision tree where all nodes at the
same depth use the same split condition
  1. Compute optimal leaf values with L2 regularization on the leaf weights
  2. Add the new tree to the ensemble, scaled by the learning rate
  3. Repeat until the specified number of iterations is reached

Theory

CatBoost addresses prediction shift (a form of target leakage in gradient boosting) through ordered boosting. For each sample x_i, the model M_i used to compute gradients is trained only on samples appearing before x_i in a random permutation \sigma.

The ordered target statistic for a categorical feature value c is:

\hat{x}_k^i = \frac{\sum_{j=1}^{p-1} [x_{\sigma_j}^i = x_{\sigma_p}^i] \cdot y_{\sigma_j} + a \cdot P}{\sum_{j=1}^{p-1} [x_{\sigma_j}^i = x_{\sigma_p}^i] + a}

where a is a prior weight and P is the prior value.

The regularized loss at iteration t is:

\mathcal{L}^{(t)} = \sum_{i=1}^{n} l(y_i, \hat{y}_i^{(t-1)} + f_t(x_i)) + \frac{\lambda}{2} \sum_{j=1}^{T} w_j^2

where \lambda is the L2 leaf regularization coefficient (l2_leaf_reg).

Parameters

iterations
int = 100
Number of boosting iterations (trees to build).
depth
int = 6
Depth of the decision trees.
learning_rate
float = 0.03
Step size shrinkage to prevent overfitting.
l2_leaf_reg
float = 3.0
L2 regularization coefficient for the leaves.
border_count
int = 128
Number of discretization splits for numerical features.
bagging_temperature
float = 1.0
Controls Bayesian bagging intensity. Higher values increase randomness.
random_strength
float = 1.0
Randomness used for scoring splits.
cat_features
list of int
Indices of categorical columns in the input data.
verbose
bool = False
Whether to print training progress and metrics.
random_state
int
Seed for reproducibility.

Attributes

model_
cb.CatBoostClassifier
The underlying fitted CatBoost model object.
classes_
ndarray of shape (n_classes,)
Unique class labels discovered during fit().
n_classes_
int
Number of unique classes discovered during fit().

Notes

Complexity:

  • Training: O(T \cdot n \cdot d \cdot D) where T = iterations,
n = n_samples, d = n_features, D = depth
  • Prediction: O(T \cdot D) per sample (very fast due to oblivious trees)
When to use CatBoostClassifier:
  • Datasets with categorical features that should not be one-hot encoded
  • When minimal hyperparameter tuning is desired (strong defaults)
  • When reducing prediction shift (target leakage) is important
  • Production systems where fast inference with oblivious trees is beneficial

References

Prokhorenkova2018
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V. and Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems (NeurIPS), 31.
Dorogush2018
Dorogush, A.V., Ershov, V. and Gulin, A. (2018). CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363.

See Also

CatBoostRegressor

CatBoost for regression tasks.
XGBoostClassifier

XGBoost classifier with second-order optimization.
LightGBMClassifier

LightGBM classifier with leaf-wise growth.

Train a CatBoost classifier with categorical feature support:

python 
>>> from tuiml.algorithms.gradient_boosting import CatBoostClassifier
>>> import numpy as np
>>>
>>> X_train = np.array([[1, 0], [2, 1], [3, 0], [4, 1]])
>>> y_train = np.array([0, 1, 0, 1])
>>> clf = CatBoostClassifier(iterations=500, learning_rate=0.01)
>>> clf.fit(X_train, y_train)
>>> y_pred = clf.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for algorithm parameters.

get_capabilities (cls) -> List[str]

Return supported capabilities.

get_complexity (cls) -> str

Return complexity analysis.

get_references (cls) -> List[str]

Return academic citations.

fit (self, X: np.ndarray, y: np.ndarray) -> 'CatBoostClassifier'

Fit the CatBoost classifier to training data.

Parameters
X
ndarray of shape (n_samples, n_features)
Training feature matrix.
y
ndarray of shape (n_samples,)
Target class labels.
Returns
self
CatBoostClassifier
Fitted estimator.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
ndarray of shape (n_samples, n_features)
Samples to classify.
Returns
ndarray of shape (n_samples,)
Predicted class labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
ndarray of shape (n_samples, n_features)
Samples to predict probabilities for.
Returns
ndarray of shape (n_samples, n_classes)
Class probability matrix.
__repr__ (self) -> str

CatBoostRegressor

class algorithms.gradient_boosting.catboost.CatBoostRegressor(Regressor)

CatBoost regressor with native support for categorical features.

Implementation of the CatBoost algorithm for regression tasks, using ordered target statistics and oblivious decision trees to handle categorical features efficiently without manual encoding.
Constructor 
__init__(
    self,
    iterations: int = 100,
    depth: int = 6,
    learning_rate: float = 0.03,
    l2_leaf_reg: float = 3.0,
    border_count: int = 128,
    bagging_temperature: float = 1.0,
    random_strength: float = 1.0,
    cat_features: Optional[List[int]] = None,
    verbose: bool = False,
    random_state: Optional[int] = None,
)

Overview

The regression variant follows the same ordered boosting procedure:

  1. Encode categorical features using ordered target statistics to
avoid target leakage during gradient computation
  1. For each iteration, compute the negative gradient of the loss
(e.g., squared error) using ordered boosting
  1. Build a symmetric (oblivious) decision tree where all nodes at
the same depth share the same split condition
  1. Compute optimal leaf values with L2 regularization
  2. Add the tree to the ensemble, scaled by the learning rate
  3. Repeat for the specified number of iterations

Theory

For the default RMSE objective, the loss for sample i is:

l(y_i, \hat{y}_i) = \frac{1}{2}(y_i - \hat{y}_i)^2

The ordered boosting scheme trains model M_i on a prefix of a random permutation \sigma to compute the gradient for sample x_{\sigma_i}:

g_i = \frac{\partial l(y_i, s)}{\partial s}\bigg|_{s=M_{\sigma_i}(x_i)}

The regularized leaf weight for leaf j is:

w_j^* = -\frac{\sum_{i \in I_j} g_i}{|I_j| + \lambda}

where \lambda is the l2_leaf_reg parameter.

Parameters

iterations
int = 100
Number of boosting iterations (trees to build).
depth
int = 6
Depth of the decision trees.
learning_rate
float = 0.03
Step size shrinkage to prevent overfitting.
l2_leaf_reg
float = 3.0
L2 regularization coefficient for the leaves.
border_count
int = 128
Number of discretization splits for numerical features.
bagging_temperature
float = 1.0
Controls Bayesian bagging intensity.
random_strength
float = 1.0
Randomness used for scoring splits.
cat_features
list of int
Indices of categorical columns in the input data.
verbose
bool = False
Whether to print training progress and metrics.
random_state
int
Seed for reproducibility.

Attributes

model_
cb.CatBoostRegressor
The underlying fitted CatBoost regressor object.

Notes

Complexity:

  • Training: O(T \cdot n \cdot d \cdot D) where T = iterations,
n = n_samples, d = n_features, D = depth
  • Prediction: O(T \cdot D) per sample (very fast due to oblivious trees)
When to use CatBoostRegressor:
  • Regression tasks with categorical features that should not be one-hot encoded
  • When minimal hyperparameter tuning is desired (strong defaults)
  • Datasets with mixed numerical and categorical features
  • When fast inference with oblivious trees is needed in production

References

Prokhorenkova2018
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V. and Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems (NeurIPS), 31.
Dorogush2018
Dorogush, A.V., Ershov, V. and Gulin, A. (2018). CatBoost: gradient boosting with categorical features support. arXiv preprint arXiv:1810.11363.

See Also

CatBoostClassifier

CatBoost for classification tasks.
XGBoostRegressor

XGBoost regressor with second-order optimization.
LightGBMRegressor

LightGBM regressor with leaf-wise growth.

Train a CatBoost regressor with categorical feature support:

python 
>>> from tuiml.algorithms.gradient_boosting import CatBoostRegressor
>>> import numpy as np
>>>
>>> X_train = np.array([[1, 0], [2, 1], [3, 0], [4, 1]])
>>> y_train = np.array([1.5, 3.5, 2.5, 4.5])
>>> reg = CatBoostRegressor(iterations=1000, depth=8)
>>> reg.fit(X_train, y_train)
>>> y_pred = reg.predict(X_train)

Methods

get_parameter_schema (cls) -> Dict[str, Any]

Return JSON Schema for algorithm parameters.

get_capabilities (cls) -> List[str]

Return supported capabilities.

get_complexity (cls) -> str

Return complexity analysis.

get_references (cls) -> List[str]

Return academic citations.

fit (self, X: np.ndarray, y: np.ndarray) -> 'CatBoostRegressor'

Fit the CatBoost regressor to training data.

Parameters
X
ndarray of shape (n_samples, n_features)
Training feature matrix.
y
ndarray of shape (n_samples,)
Target values.
Returns
self
CatBoostRegressor
Fitted estimator.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values for samples.

Parameters
X
ndarray of shape (n_samples, n_features)
Samples to predict.
Returns
ndarray of shape (n_samples,)
Predicted values.
__repr__ (self) -> str