API Reference / algorithms / timeseries / deep /

patchtst.py

PatchTST: a patch-based Transformer forecaster with channel independence.

Classes

PatchTSTForecaster

class algorithms.timeseries.deep.patchtst.PatchTSTForecaster(DeepForecaster)

PatchTST: a Transformer that attends over patches, not time steps.

Point-wise Transformers on time series attend over individual time steps, which are semantically thin — a single reading carries almost no meaning on its own — and give attention a sequence as long as the lookback. PatchTST instead cuts the lookback into patches, short subseries of patch_len points, and treats each patch as one token. A patch carries local shape, so attention finally has something meaningful to relate, and the sequence length drops by roughly stride, making attention's quadratic cost about \text{stride}^2 cheaper.

The second idea is channel independence. A multivariate series is not mixed inside the model; every channel is pushed through the same shared backbone on its own. This helps for two reasons. It cuts parameters sharply — one backbone rather than one per channel pair, and a head sized by patches rather than by channels — and it removes the chance to overfit spurious cross-channel correlations, which are abundant and unstable in real series. Each channel also contributes its own training examples to the same weights, so the effective dataset grows with the channel count instead of the parameter count. The public API here is univariate, but the backbone implements this folding, so a channel dimension is handled by construction.

Instance normalisation (RevIN) wraps the whole thing: each window is standardised on its own mean and standard deviation and the forecast is mapped back with the same statistics, which is what lets one set of weights serve windows at wildly different levels.

Constructor 
__init__(
    self,
    lookback: int = 24,
    horizon: int = 8,
    patch_len: int = 8,
    stride: int = 4,
    d_model: int = 32,
    n_heads: int = 4,
    n_layers: int = 1,
    dim_feedforward: int = 64,
    dropout: float = 0.0,
    revin: bool = True,
    n_epochs: int = 100,
    batch_size: int = 32,
    learning_rate: float = 0.001,
    patience: int = 15,
    device: str = 'cpu',
    random_state: Optional[int] = None,
)

Overview

  1. Slide a (lookback, horizon) window over the series.
  2. Normalise each window by its own statistics (RevIN).
  3. Cut the window into n_patches patches, padding the end by repeating
the last value when the arithmetic does not divide.
  1. Embed each patch to d_model and add a learned positional encoding.
  2. Run a Transformer encoder over the patch tokens.
  3. Flatten the encoded tokens and project linearly to the horizon.
  4. Denormalise with the window's own statistics.

Theory

A window x \in \mathbb{R}^{L} is normalised,

\tilde{x} = \frac{x - \mu}{\sigma}, \qquad \mu = \frac{1}{L}\sum_t x_t, \quad \sigma = \sqrt{\frac{1}{L}\sum_t (x_t - \mu)^2},

and cut into N = \lceil (L - P)/S \rceil + 1 patches of length P at stride S. Each patch is embedded and encoded,

z_i = W_e p_i + e_i, \qquad Z = \text{TransformerEncoder}(z_1, \dots, z_N),

where attention costs O(N^2 d) rather than the O(L^2 d) of a point-wise model. The head flattens and projects,

\hat{y} = W_h\, \text{vec}(Z) \in \mathbb{R}^{H},

and the forecast is returned to the original units by \sigma \hat{y} + \mu, which is an exact inverse of the normalisation.

Parameters

lookback
int = 24
Length of the input window. Automatically shrunk when the series is too short to yield training windows.
horizon
int = 8
Number of steps the network is trained to emit at once. Longer forecasts are produced by autoregressive rollout.
patch_len
int = 8
Points per patch. Clipped to the resolved lookback.
stride
int = 4
Step between patch starts; equal to patch_len gives disjoint patches, smaller gives overlap.
d_model
int = 32
Token embedding width. Must be divisible by n_heads.
n_heads
int = 4
Attention heads per encoder layer.
n_layers
int = 1
Transformer encoder layers. The paper uses 3; the default here keeps the generic algorithm contract, which fits every model on sixty points, fast. Real use wants 2-4.
dim_feedforward
int = 64
Width of the position-wise feed-forward network.
dropout
float = 0.0
Dropout inside the encoder and before the head. The default is 0 so a fit is exactly reproducible; 0.1-0.3 is usual for real training runs.
revin
bool = True
Apply reversible instance normalisation to each window. Turning this off is almost always a mistake on a trending series.
n_epochs
int = 100
Maximum passes over the window dataset. Real use wants several hundred.
batch_size
int = 32
Windows per gradient step.
learning_rate
float = 0.001
Adam step size.
patience
int = 15
Epochs without an improvement in training loss before stopping early.
device
{"cpu", "auto", "cuda", "mps"} = "cpu"
Where to run. The default is "cpu" because it is the only setting that gives bit-identical forecasts across machines; use "auto" when speed matters more than exact reproducibility.
random_state
int = None
Seed for weight initialisation and batch shuffling.

Attributes

module_
torch.nn.Module
The trained network.
lookback_
int
Lookback actually used, after any shrinking.
horizon_
int
Horizon actually used, after any shrinking.
n_windows_
int
Number of training windows built from the series.
offset_
float
Mean of the training series, removed before fitting.
scale_
float
Standard deviation of the training series, divided out before fitting.
series_
np.ndarray of shape (n_samples,)
The globally scaled training series, kept for the rollout.
loss_curve_
np.ndarray of shape (n_epochs_run_,)
Mean training loss per epoch.
n_epochs_run_
int
Epochs actually run before early stopping.
device_
str
Resolved device string.

Notes

Requires PyTorch: pip install 'tuiml[torch]'. The class imports, constructs, registers and reports its schema without torch; only fit needs it.

Complexity:

  • Training: O(E \cdot W \cdot (N^2 d + N d^2)) for E
epochs, W windows, N patches and width d. The patching is what turns L^2 into N^2.
  • Prediction: O(\lceil s / H \rceil \cdot (N^2 d + N d^2)).
When to use PatchTSTForecaster:
  • Long lookbacks, where a point-wise Transformer's attention is both
expensive and unfocused.
  • Series with repeating local shapes that patches can capture as units.
  • Multivariate problems where channel mixing has been found to overfit.
  • Not for very short series: with a few dozen points there are too few
patches for attention to say anything, and ExponentialSmoothing will win.

References

Nie2023
Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2023). A time series is worth 64 words: Long-term forecasting with Transformers. International Conference on Learning Representations (ICLR). https://doi.org/10.48550/arXiv.2211.14730
Kim2022
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., & Choo, J. (2022). Reversible instance normalization for accurate time-series forecasting against distribution shift. ICLR.

See Also

NBEATSForecaster

Fully connected doubly-residual forecaster.
NHITSForecaster

Multi-rate variant of N-BEATS.
ARIMA

Classical alternative for short series.
python 
>>> import numpy as np
>>> from tuiml.algorithms.timeseries.deep import PatchTSTForecaster
>>> from tuiml.utils.torch_backend import has_torch
>>> model = PatchTSTForecaster(lookback=24, horizon=6, random_state=0)
>>> from tuiml.algorithms.timeseries.deep.patchtst import num_patches
>>> num_patches(24, model.patch_len, model.stride)
5
>>> if has_torch():
...     y = np.sin(np.arange(200) / 5.0)
...     _ = model.fit(y)
...     print(model.predict(steps=6).shape)
... else:
...     print("(6,)")
(6,)

Methods

get_parameter_schema (cls) -> Dict[str, Dict[str, Any]]

Return JSON Schema for algorithm parameters.

get_complexity (cls) -> str

Return complexity analysis.

get_references (cls) -> List[str]

Return academic citations.

Functions

Func

num_patches

Line 23
num_patches(length: int, patch_len: int, stride: int) -> int

Return how many patches a window of length is split into.

The end of the window is padded by repeating its last value so the final patch lands exactly on the padded end. Padding the end rather than the start matters: the most recent points are the ones a forecast leans on, and truncating them to make the arithmetic divide would throw away the most informative part of the window.

Parameters

length
int
Window length.
patch_len
int
Points per patch. Clipped to length when the window is shorter.
stride
int
Step between consecutive patch starts. Equal to patch_len gives disjoint patches; smaller gives overlap.

Returns

n_patches
int
Number of patches, at least 1.
python 
>>> from tuiml.algorithms.timeseries.deep.patchtst import num_patches
>>> num_patches(24, patch_len=8, stride=4)   # divides exactly
5
>>> num_patches(25, patch_len=8, stride=4)   # padded to 29
6
>>> num_patches(5, patch_len=8, stride=4)    # window shorter than a patch
1
Func

padded_length

Line 65
padded_length(length: int, patch_len: int, stride: int) -> int

Return the window length after end padding.

Parameters

length
int
Window length.
patch_len
int
Points per patch.
stride
int
Step between patch starts.

Returns

padded
int
patch_len + (n_patches - 1) * stride, never less than length.
python 
>>> from tuiml.algorithms.timeseries.deep.patchtst import padded_length
>>> padded_length(25, patch_len=8, stride=4)
28
Func

patchify

Line 94
patchify(torch: Any, x: Any, patch_len: int, stride: int) -> Any

Split each window into overlapping or disjoint patches.

Parameters

torch
module
The imported torch module, passed in so this function never imports it.
x
torch.Tensor of shape (batch, length)
Input windows.
patch_len
int
Points per patch.
stride
int
Step between patch starts.

Returns

patches
torch.Tensor of shape (batch, n_patches, patch_len)
The patch tokens, oldest first.