Voting ensemble implementations for classification and regression.
Classes
VotingClassifier for combining heterogeneous classifiers via voting rules.
VotingClassifier (also known as Vote) combines predictions from multiple diverse classifiers using various combination rules such as average probabilities, product rule, or majority voting.
Constructor
__init__( self, classifiers: List[Any] = None, combination_rule: str = 'average', )
Overview
The algorithm proceeds as follows:
- Train each of the L base classifiers independently on the full training set
- To predict, collect outputs (predictions or probabilities) from all classifiers
- Apply the selected combination rule to aggregate the outputs
- Return the class with the highest aggregated score
Theory
Let p_l(k|x) denote the posterior probability estimate from classifier h_l for class k given input x. The combination rules are:
Average rule:
P(k|x) = \frac{1}{L} \sum_{l=1}^{L} p_l(k|x)
Product rule:
P(k|x) = \frac{\prod_{l=1}^{L} p_l(k|x)}{\sum_{k'} \prod_{l=1}^{L} p_l(k'|x)}
Majority voting rule:
H(x) = \arg\max_{k} \sum_{l=1}^{L} \mathbb{1}[h_l(x) = k]
Median rule:
P(k|x) = \text{median}_{l=1}^{L}\, p_l(k|x)
Parameters
classifiers
list, 'DecisionTreeClassifier']
= ['
The collection of classifiers to be combined. Can be classifier names (strings), classes, or instances.
combination_rule
{'average', 'product', 'majority', 'median', 'max', 'min'}
= 'average'
The rule used to combine the predictions of the base classifiers.
Attributes
estimators_
list
The collection of fitted base classifiers.
classes_
np.ndarray
The unique class labels discovered during
fit().
Notes
Complexity:
- Training: O(\sum_{l=1}^{L} C_l) where C_l is the training
- Prediction: O(\sum_{l=1}^{L} C_l^{\text{pred}}) per sample
- When you have multiple diverse classifiers with comparable performance
- When you want a simple combination without learning combination weights
- When base classifiers make independent errors (low correlation)
- As a baseline before trying more complex methods like stacking
References
Kittler1998
Kittler, J., Hatef, M., Duin, R.P.W. and Matas, J. (1998).
On Combining Classifiers.
IEEE Transactions on Pattern Analysis and Machine Intelligence,
20(3), 226-239.
DOI: 10.1109/34.667881
Polikar2006
Polikar, R. (2006).
Ensemble Based Systems in Decision Making.
IEEE Circuits and Systems Magazine, 6(3), 21-45.
DOI: 10.1109/MCAS.2006.1688199
See Also
Basic usage for combining classifiers with voting:
python
>>> from tuiml.algorithms.ensemble import VotingClassifier
>>> import numpy as np
>>>
>>> # Create sample training data
>>> X_train = np.array([[1, 2], [2, 3], [3, 1], [4, 3], [5, 2]])
>>> y_train = np.array([0, 0, 1, 1, 1])
>>>
>>> # Fit the Voting classifier with average rule
>>> clf = VotingClassifier(
... classifiers=['NaiveBayesClassifier', 'DecisionTreeClassifier'],
... combination_rule='average'
... )
>>> clf.fit(X_train, y_train)
VotingClassifier(...)
>>> predictions = clf.predict(X_train)Methods
get_parameter_schema
(cls) -> Dict[str, Dict[str, Any]]
get_capabilities
(cls) -> List[str]
get_complexity
(cls) -> str
get_references
(cls) -> List[str]
__repr__
(self) -> str
VotingRegressor for combining heterogeneous regressors via aggregation rules.
VotingRegressor combines predictions from multiple diverse regressors using aggregation rules such as averaging, median, or weighted average.
Constructor
__init__( self, regressors: List[Any] = None, combination_rule: str = 'average', )
Overview
The algorithm proceeds as follows:
- Train each of the L base regressors independently on the full training set
- To predict, collect outputs (predictions) from all regressors
- Apply the selected combination rule to aggregate the outputs
Theory
Let h_l(x) denote the prediction from regressor h_l for input x. The combination rules are:
Average rule:
H(x) = \frac{1}{L} \sum_{l=1}^{L} h_l(x)
Median rule:
H(x) = \text{median}_{l=1}^{L}\, h_l(x)
Max / Min rules:
H(x) = \max_{l} h_l(x) \quad \text{or} \quad H(x) = \min_{l} h_l(x)
Parameters
regressors
list
= ['
The collection of regressors to be combined. Can be regressor names (strings), classes, or instances.
combination_rule
{'average', 'median', 'max', 'min'}
= 'average'
The rule used to combine the predictions of the base regressors.
Attributes
estimators_
list
The collection of fitted base regressors.
Notes
Complexity:
- Training: O(\sum_{l=1}^{L} C_l) where C_l is the training
- Prediction: O(\sum_{l=1}^{L} C_l^{\text{pred}}) per sample
- When you have multiple diverse regressors with comparable performance
- When you want a simple combination without learning combination weights
- When base regressors make independent errors (low correlation)
- As a baseline before trying more complex methods like stacking
References
Perrone1993
Perrone, M.P. and Cooper, L.N. (1993).
When Networks Disagree: Ensemble Methods for Hybrid Neural Networks.
Neural Networks for Speech and Image Processing, Chapman & Hall.
See Also
Basic usage for combining regressors with voting:
python
>>> from tuiml.algorithms.ensemble import VotingRegressor
>>> import numpy as np
>>>
>>> # Create sample training data
>>> X_train = np.array([[1, 2], [2, 3], [3, 1], [4, 3], [5, 2]])
>>> y_train = np.array([1.5, 2.3, 3.1, 4.2, 5.0])
>>>
>>> # Fit the Voting regressor with average rule
>>> reg = VotingRegressor(
... regressors=['GradientBoostingRegressor'],
... combination_rule='average'
... )
>>> reg.fit(X_train, y_train)
VotingRegressor(...)
>>> predictions = reg.predict(X_train)