preprocessing/

Data Preprocessing and Transformation.

The tuiml.preprocessing module provides a wide array of tools for preparing dataset for machine learning. It follows a consistent API where most components are Transformers that can be used in pipelines.

Overview

This module is organized into several functional categories:

Scaling: Standardize or normalize numerical features (e.g., Z-score, Min-Max).
Encoding: Convert categorical/nominal data into numerical formats (e.g., One-Hot, Ordinal).
Imputation: Handle missing data using statistical or distance-based strategies.
Discretization: Partition continuous features into discrete bins (e.g., Equal-Width, MDL).
Outliers: Detect and manage extreme values (e.g., IQR clipping).
Sampling: Balance classes (SMOTE, ADASYN) or reduce dataset size.
Text: Convert raw text into numerical vectors (TF-IDF, Hashing).
Time Series: Extract temporal features through lags and differences.

Basic Usage

Impute missing values and scale features in sequence:

>>> from tuiml.preprocessing import SimpleImputer, StandardScaler >>> import numpy as np >>> X = np.array([[1, 2], [np.nan, 3], [7, 6]]) >>> imputer = SimpleImputer(strategy="mean") >>> scaler = StandardScaler() >>> X_clean = scaler.fit_transform(imputer.fit_transform(X))

Imbalanced Learning

Use SMOTE to generate synthetic samples for a minority class:

>>> from tuiml.preprocessing.sampling import SMOTESampler >>> X = np.random.rand(10, 2) >>> y = np.array([0, 0, 0, 0, 0, 0, 1, 1, 1, 1]) >>> sampler = SMOTESampler(k_neighbors=2) >>> X_res, y_res = sampler.fit_resample(X, y)

Packages