API Reference / algorithms / trees /

random_forest.py

RandomForestClassifier and RandomForestRegressor using C++ tree builders.

Classes

RandomForestClassifier

class algorithms.trees.random_forest.RandomForestClassifier(Classifier)

Random Forest classifier - ensemble of random trees.

Random Forest is an ensemble method that fits multiple randomized decision trees on bootstrapped subsamples of the dataset and uses majority voting (or probability averaging) to improve predictive accuracy and control overfitting.
Constructor 
__init__(
    self,
    n_estimators: int = 100,
    max_features: Any = 'sqrt',
    max_depth: Optional[int] = None,
    min_samples_split: int = 2,
    min_samples_leaf: int = 1,
    bootstrap: bool = True,
    oob_score: bool = False,
    random_state: Optional[int] = None,
    n_jobs: int = ...,
    criterion: str = 'gini',
)

Overview

The Random Forest algorithm works as follows:

  1. For each of the T trees, draw a bootstrap sample
(sampling with replacement) of size n from the training data
  1. Build a fully-grown randomized tree on each bootstrap sample,
selecting k random features at each split
  1. For prediction, aggregate results via majority voting (classification)
or averaging (probability estimation)
  1. Optionally compute the out-of-bag (OOB) score using samples not
included in each tree's bootstrap

Theory

Each tree h_t is trained on a bootstrap sample S_t. The ensemble prediction is determined by majority vote:

\hat{y}(x) = \arg\max_c \sum_{t=1}^{T} \mathbb{1}[h_t(x) = c]

The generalization error of a Random Forest is bounded by:

PE^* \leq \bar{\rho} \cdot \frac{(1 - s^2)}{s^2}

where \bar{\rho} is the mean correlation between trees and s is the strength (margin) of individual trees.

The out-of-bag error is computed using each sample's predictions only from trees that did not include it in their bootstrap:

OOB_{error} = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1}[\hat{y}_{OOB}(x_i) \neq y_i]

Parameters

n_estimators
int = 100
Number of trees in the forest.
max_features
{'sqrt', 'log2'}, int or float = 'sqrt'
Number of features to consider at each split.
max_depth
int
Maximum depth of the trees. None means unlimited.
min_samples_split
int = 2
Minimum samples required to split a node.
min_samples_leaf
int = 1
Minimum samples required at a leaf node.
bootstrap
bool = True
Whether to use bootstrap samples when building trees.
oob_score
bool = False
Whether to calculate out-of-bag score.
random_state
int
Random seed for reproducibility.
n_jobs
int = 1
Number of parallel jobs (-1 for all CPUs).
criterion
str = 'gini'
Splitting criterion ('gini' or 'entropy').

Attributes

estimators_
list of TreeNode
Fitted tree root nodes.
classes_
np.ndarray
Unique class labels.
n_features_
int
Number of features seen during fit.
oob_score_
float
Out-of-bag score (if oob_score=True).
feature_importances_
np.ndarray or None
Feature importance scores (impurity-based).

Notes

Complexity:

  • Training: O(T \cdot n \cdot k \cdot \log(n)) where T =
n_estimators, n = samples, k = max_features
  • Prediction: O(T \cdot \log(n)) per sample
When to use RandomForestClassifier:
  • When you need a robust, general-purpose classifier
  • High-dimensional datasets where feature selection is implicit
  • When out-of-bag error estimation is desired (no separate validation set)
  • When training can be parallelized across multiple cores
  • When individual tree interpretability is less important than accuracy

References

Breiman2001
Breiman, L. (2001). Random Forests. Machine Learning, 45(1), pp. 5-32. DOI: 10.1023/A:1010933404324
Breiman1996
Breiman, L. (1996). Bagging Predictors. Machine Learning, 24(2), pp. 123-140. DOI: 10.1007/BF00058655

See Also

Basic usage for classification with OOB score:

python 
>>> from tuiml.algorithms.trees import RandomForestClassifier
>>> import numpy as np
>>>
>>> # Create sample data
>>> X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [2, 3], [4, 5]])
>>> y = np.array([0, 0, 1, 1, 0, 1])
>>>
>>> # Fit a random forest
>>> clf = RandomForestClassifier(n_estimators=100, oob_score=True, random_state=42)
>>> clf.fit(X, y)
RandomForestClassifier(...)
>>> predictions = clf.predict(X)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return classifier capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'RandomForestClassifier'

Fit the Random Forest classifier.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target labels.
Returns
self
RandomForestClassifier
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
proba
np.ndarray of shape (n_samples, n_classes)
Class probabilities.
__repr__ (self) -> str

String representation.

RandomForestRegressor

class algorithms.trees.random_forest.RandomForestRegressor(Regressor)

Random Forest regressor - ensemble of random regression trees.

Random Forest regressor is an ensemble method that fits multiple randomized regression trees on bootstrapped subsamples and uses averaging of predictions to improve accuracy and control overfitting. Each base tree uses variance reduction (MSE) as its splitting criterion.
Constructor 
__init__(
    self,
    n_estimators: int = 100,
    max_features: Any = 'sqrt',
    max_depth: Optional[int] = None,
    min_samples_split: int = 2,
    min_samples_leaf: int = 1,
    bootstrap: bool = True,
    oob_score: bool = False,
    random_state: Optional[int] = None,
    n_jobs: int = ...,
    criterion: str = 'squared_error',
)

Overview

The Random Forest regression algorithm works as follows:

  1. For each of the T trees, draw a bootstrap sample
(sampling with replacement) of size n from the training data
  1. Build a fully-grown randomized tree on each bootstrap sample,
selecting k random features at each split
  1. For prediction, average the outputs of all trees
  2. Optionally compute the out-of-bag (OOB) R-squared using samples not
included in each tree's bootstrap

Theory

Each tree h_t is trained on a bootstrap sample S_t. The ensemble prediction is the mean of individual tree predictions:

\hat{y}(x) = \frac{1}{T} \sum_{t=1}^{T} h_t(x)

The out-of-bag R-squared is computed using each sample's predictions only from trees that did not include it in their bootstrap:

R^2_{OOB} = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y}_{OOB}(x_i))^2} {\sum_{i=1}^{n} (y_i - \bar{y})^2}

Parameters

n_estimators
int = 100
Number of trees in the forest.
max_features
{'sqrt', 'log2'}, int or float = 'sqrt'
Number of features to consider at each split.
max_depth
int
Maximum depth of the trees. None means unlimited.
min_samples_split
int = 2
Minimum samples required to split a node.
min_samples_leaf
int = 1
Minimum samples required at a leaf node.
bootstrap
bool = True
Whether to use bootstrap samples when building trees.
oob_score
bool = False
Whether to calculate out-of-bag R-squared score.
random_state
int
Random seed for reproducibility.
n_jobs
int = 1
Number of parallel jobs (-1 for all CPUs).
criterion
str = 'squared_error'
Splitting criterion ('squared_error' or 'friedman_mse').

Attributes

estimators_
list of TreeNode
Fitted tree root nodes.
n_features_
int
Number of features seen during fit.
oob_score_
float
Out-of-bag R-squared score (if oob_score=True).
feature_importances_
np.ndarray or None
Feature importance scores (impurity-based).

Notes

Complexity:

  • Training: O(T \cdot n \cdot k \cdot \log(n)) where T =
n_estimators, n = samples, k = max_features
  • Prediction: O(T \cdot \log(n)) per sample
When to use RandomForestRegressor:
  • When you need a robust, general-purpose regressor
  • High-dimensional datasets where feature selection is implicit
  • When out-of-bag R-squared estimation is desired
  • When training can be parallelized across multiple cores

References

Breiman2001
Breiman, L. (2001). Random Forests. Machine Learning, 45(1), pp. 5-32. DOI: 10.1023/A:1010933404324

See Also

RandomForestClassifier

Classifier counterpart.
python 
>>> from tuiml.algorithms.trees import RandomForestRegressor
>>> import numpy as np
>>> X = np.array([[1, 2], [3, 4], [5, 6], [7, 8], [2, 3], [4, 5]])
>>> y = np.array([1.0, 2.0, 3.0, 4.0, 1.5, 2.5])
>>> reg = RandomForestRegressor(n_estimators=100, oob_score=True, random_state=42)
>>> reg.fit(X, y)
RandomForestRegressor(...)
>>> predictions = reg.predict(X)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return JSON Schema for constructor parameters.

get_capabilities (cls) -> List[str]

Return regressor capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'RandomForestRegressor'

Fit the Random Forest regressor.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
RandomForestRegressor
Returns the fitted instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict target values for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted target values.
score (self, X: np.ndarray, y: np.ndarray) -> float

Return R-squared score.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
y
np.ndarray of shape (n_samples,)
True target values.
Returns
r2
float
R-squared score.
__repr__ (self) -> str

String representation.