API Reference / algorithms / neural /

multilayer_perceptron.py

MultilayerPerceptronClassifier (Neural Network) classifier implementation.

Classes

MultilayerPerceptronClassifier

class algorithms.neural.multilayer_perceptron.MultilayerPerceptronClassifier(Classifier)

Multilayer Perceptron (Neural Network) classifier.

A feedforward artificial neural network trained using backpropagation. It supports multiple hidden layers, various activation functions, momentum-based updates, and early stopping.
Constructor 
__init__(
    self,
    hidden_layers: List[int] = None,
    learning_rate: float = 0.1,
    momentum: float = 0.9,
    max_epochs: int = 500,
    validation_threshold: int = 20,
    decay: bool = True,
    activation: str = 'relu',
    random_state: Optional[int] = None,
)

Overview

The Multilayer Perceptron trains by iteratively adjusting weights through forward and backward passes:

  1. Initialize weights using Xavier initialization for each layer
  2. Standardize input features to zero mean and unit variance
  3. Perform a forward pass: propagate inputs through hidden layers
applying activation functions, then softmax at the output layer
  1. Compute the cross-entropy loss between predictions and targets
  2. Perform a backward pass (backpropagation): compute gradients of the
loss with respect to each weight and bias
  1. Update weights using gradient descent with momentum
  2. Optionally decay the learning rate over epochs
  3. Stop early if the loss does not improve for validation_threshold
consecutive epochs

Theory

For a network with L layers, the forward pass computes activations layer by layer. For hidden layer l:

\mathbf{z}^{(l)} = \mathbf{a}^{(l-1)} \mathbf{W}^{(l)} + \mathbf{b}^{(l)}
\mathbf{a}^{(l)} = g(\mathbf{z}^{(l)})

where g is the activation function (sigmoid or ReLU). The output layer uses softmax:

\hat{y}_k = \frac{e^{z_k}}{\sum_j e^{z_j}}

The network is trained by minimizing the cross-entropy loss:

\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} y_{ik} \log(\hat{y}_{ik})

Weights are updated using momentum-based gradient descent:

\mathbf{v}_t = \mu \, \mathbf{v}_{t-1} - \eta \, \nabla \mathcal{L}
\mathbf{W}_t = \mathbf{W}_{t-1} + \mathbf{v}_t

where \mu is the momentum and \eta is the learning rate.

Parameters

hidden_layers
list of int = [100]
Sizes of the hidden layers. Each entry in the list represents the number of neurons in that hidden layer.
learning_rate
float = 0.1
Learning rate for weight updates.
momentum
float = 0.9
Momentum for gradient descent to accelerate convergence and avoid local minima.
max_epochs
int = 500
Maximum number of training epochs.
validation_threshold
int = 20
Number of epochs to wait for improvement in loss before stopping (patience).
decay
bool = True
Whether to decay the learning rate over time.
activation
{'sigmoid', 'relu'} = 'relu'
Activation function for hidden layers.
random_state
int or None = None
Seed for the random number generator.

Attributes

weights_
list of np.ndarray
Weight matrices connecting each layer.
biases_
list of np.ndarray
Bias vectors for each layer.
classes_
np.ndarray of shape (n_classes,)
Unique class labels discovered during fit.

Notes

Complexity:

  • Training: O(n \cdot E \cdot \sum_{l=1}^{L} d_l \cdot d_{l+1})
where n = number of samples, E = number of epochs, and d_l = size of layer l
  • Prediction: O(n \cdot \sum_{l=1}^{L} d_l \cdot d_{l+1}) per
batch

When to use MultilayerPerceptronClassifier:

  • When the decision boundary is non-linear
  • When you need a flexible model that can approximate complex functions
  • For problems like XOR that are not linearly separable
  • When sufficient training data is available to fit the network parameters
  • As a lightweight alternative to deep learning frameworks

References

Rumelhart1986
Rumelhart, D.E., Hinton, G.E. and Williams, R.J. (1986). Learning Representations by Back-Propagating Errors. Nature, 323, 533-536. DOI: 10.1038/323533a0
Glorot2010
Glorot, X. and Bengio, Y. (2010). Understanding the Difficulty of Training Deep Feedforward Neural Networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 249-256.

See Also

PerceptronClassifier

Single-layer perceptron for linearly separable problems.

Train a Multilayer Perceptron on the XOR problem:

python 
>>> from tuiml.algorithms.neural import MultilayerPerceptronClassifier
>>> import numpy as np
>>> X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
>>> y = np.array([0, 1, 1, 0])  # XOR problem
>>> clf = MultilayerPerceptronClassifier(hidden_layers=[5], activation='relu')
>>> clf.fit(X, y)
MultilayerPerceptronClassifier(layers=[5])

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]
get_capabilities (cls) -> List[str]
get_complexity (cls) -> str
get_references (cls) -> List[str]
fit (self, X: np.ndarray, y: np.ndarray) -> 'MultilayerPerceptronClassifier'

Fit the MultilayerPerceptronClassifier classifier.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target labels.
Returns
self
MultilayerPerceptronClassifier
Returns the fitted instance.
partial_fit (self, X: np.ndarray, y: np.ndarray, classes: Optional[np.ndarray]=None) -> 'MultilayerPerceptronClassifier'

Incrementally fit the MultilayerPerceptronClassifier classifier on a batch of samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Incremental training features.
y
np.ndarray of shape (n_samples,)
Incremental target labels.
classes
np.ndarray of shape (n_classes,) = None
List of all classes expected. Must be provided at the first call, can be omitted afterwards.
Returns
self
MultilayerPerceptronClassifier
Returns the updated instance.
predict_proba (self, X: np.ndarray) -> np.ndarray

Predict class probabilities for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input features.
Returns
probabilities
np.ndarray of shape (n_samples, n_classes)
Predicted class probabilities.
predict (self, X: np.ndarray) -> np.ndarray

Predict class labels for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted class labels.
__repr__ (self) -> str

MultilayerPerceptronRegressor

class algorithms.neural.multilayer_perceptron.MultilayerPerceptronRegressor(Regressor)

Multilayer Perceptron (Neural Network) regressor.

A feedforward artificial neural network trained using backpropagation for regression tasks. It supports multiple hidden layers, various activation functions, momentum-based updates, and early stopping. The output layer uses a single linear neuron to produce continuous predictions, trained with mean squared error loss.
Constructor 
__init__(
    self,
    hidden_layers: List[int] = None,
    learning_rate: float = 0.1,
    momentum: float = 0.9,
    max_epochs: int = 500,
    validation_threshold: int = 20,
    decay: bool = True,
    activation: str = 'relu',
    random_state: Optional[int] = None,
)

Overview

The Multilayer Perceptron regressor trains by iteratively adjusting weights through forward and backward passes:

  1. Initialize weights using Xavier initialization for each layer
  2. Standardize input features and target values to zero mean and unit variance
  3. Perform a forward pass: propagate inputs through hidden layers
applying activation functions, then a linear activation at the output
  1. Compute the MSE loss between predictions and targets
  2. Perform a backward pass (backpropagation): compute gradients of the
loss with respect to each weight and bias
  1. Update weights using gradient descent with momentum
  2. Optionally decay the learning rate over epochs
  3. Stop early if the loss does not improve for validation_threshold
consecutive epochs

Theory

For a network with L layers, the forward pass computes activations layer by layer. For hidden layer l:

\mathbf{z}^{(l)} = \mathbf{a}^{(l-1)} \mathbf{W}^{(l)} + \mathbf{b}^{(l)}
\mathbf{a}^{(l)} = g(\mathbf{z}^{(l)})

where g is the activation function (sigmoid or ReLU). The output layer uses a linear activation (identity function):

\hat{y} = \mathbf{a}^{(L-1)} \mathbf{W}^{(L)} + \mathbf{b}^{(L)}

The network is trained by minimizing the mean squared error loss:

\mathcal{L} = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2

Parameters

hidden_layers
list of int = [100]
Sizes of the hidden layers. Each entry represents the number of neurons in that hidden layer.
learning_rate
float = 0.1
Learning rate for weight updates.
momentum
float = 0.9
Momentum for gradient descent to accelerate convergence.
max_epochs
int = 500
Maximum number of training epochs.
validation_threshold
int = 20
Number of epochs to wait for improvement in loss before stopping (patience).
decay
bool = True
Whether to decay the learning rate over time.
activation
{'sigmoid', 'relu'} = 'relu'
Activation function for hidden layers.
random_state
int or None = None
Seed for the random number generator.

Attributes

weights_
list of np.ndarray
Weight matrices connecting each layer.
biases_
list of np.ndarray
Bias vectors for each layer.

Notes

Complexity:

  • Training: O(n \cdot E \cdot \sum_{l=1}^{L} d_l \cdot d_{l+1})
where n = samples, E = epochs, d_l = layer size
  • Prediction: O(n \cdot \sum_{l=1}^{L} d_l \cdot d_{l+1})
When to use MultilayerPerceptronRegressor:
  • When the relationship between features and target is non-linear
  • When you need a flexible model for continuous value prediction
  • When sufficient training data is available to fit the network
  • As a lightweight alternative to deep learning frameworks

References

Rumelhart1986
Rumelhart, D.E., Hinton, G.E. and Williams, R.J. (1986). Learning Representations by Back-Propagating Errors. Nature, 323, 533-536. DOI: 10.1038/323533a0

See Also

MultilayerPerceptronClassifier

MLP for classification tasks.
LinearRegression

Linear regression baseline.

Train a Multilayer Perceptron regressor on a simple problem:

python 
>>> from tuiml.algorithms.neural import MultilayerPerceptronRegressor
>>> import numpy as np
>>> X = np.array([[1], [2], [3], [4], [5]])
>>> y = np.array([1.0, 4.0, 9.0, 16.0, 25.0])
>>> reg = MultilayerPerceptronRegressor(hidden_layers=[10], activation='relu')
>>> reg.fit(X, y)
MultilayerPerceptronRegressor(layers=[10])

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return parameter schema.

get_capabilities (cls) -> List[str]

Return regressor capabilities.

get_complexity (cls) -> str

Return time/space complexity.

get_references (cls) -> List[str]

Return academic references.

fit (self, X: np.ndarray, y: np.ndarray) -> 'MultilayerPerceptronRegressor'

Fit the MultilayerPerceptronRegressor.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Training features.
y
np.ndarray of shape (n_samples,)
Target values.
Returns
self
MultilayerPerceptronRegressor
Returns the fitted instance.
partial_fit (self, X: np.ndarray, y: np.ndarray) -> 'MultilayerPerceptronRegressor'

Incrementally fit the MultilayerPerceptronRegressor regressor on a batch of samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Incremental training features.
y
np.ndarray of shape (n_samples,)
Incremental target values.
Returns
self
MultilayerPerceptronRegressor
Returns the updated instance.
predict (self, X: np.ndarray) -> np.ndarray

Predict continuous target values for samples.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Input features.
Returns
y_pred
np.ndarray of shape (n_samples,)
Predicted continuous values.
score (self, X: np.ndarray, y: np.ndarray) -> float

Compute the R-squared (coefficient of determination) score.

Parameters
X
np.ndarray of shape (n_samples, n_features)
Test features.
y
np.ndarray of shape (n_samples,)
True target values.
Returns
score
float
R-squared score.
__repr__ (self) -> str

String representation.