3. Preprocessing and the Leakage Trap

Chapter 1 found 652 missing values hiding as zeros. This notebook fixes them, and then asks the question tutorials usually skip: did that actually help? The answer here is "barely", and the reason why is more useful than a win would have been.

Then we cover the mistake that makes preprocessing dangerous rather than merely tedious.

You will learn:

• how to turn disguised missing values into real ones, and impute them
• how to measure whether a preprocessing step earned its place
• why scaling transforms one algorithm's score by four points and another's by exactly zero
• what data leakage is, how to avoid it, and how big the leak actually is here
• outliers, encoding and discretisation, briefly

Prerequisites: chapters 1 and 2.

import numpy as np
import matplotlib.pyplot as plt

from tuiml.datasets import load_diabetes
from tuiml.evaluation import StratifiedKFold, accuracy_score, roc_auc_score
from tuiml.algorithms.trees import RandomForestClassifier
from tuiml.algorithms.neighbors import KNearestNeighborsClassifier

data = load_diabetes()
X, y = data.X, data.y
print(data.shape)

(768, 8)

3.1 Making missing values missing

The five columns where a zero is physiologically impossible were plas, pres, skin, insu and mass. Turn those zeros into np.nan, which is what "missing" is supposed to look like.

Note that preg is deliberately excluded: its zeros are real.

IMPOSSIBLE_ZERO = ["plas", "pres", "skin", "insu", "mass"]
columns = [data.feature_names.index(c) for c in IMPOSSIBLE_ZERO]

X_nan = X.copy()
for col in columns:
    X_nan[X_nan[:, col] == 0, col] = np.nan

print(f"missing values now visible: {int(np.isnan(X_nan).sum())}")
print()
for name, col in zip(IMPOSSIBLE_ZERO, columns):
    n = int(np.isnan(X_nan[:, col]).sum())
    print(f"  {name:6s} {n:4d}  ({100 * n / len(X):.1f}%)")

missing values now visible: 652

  plas      5  (0.7%)
  pres     35  (4.6%)
  skin    227  (29.6%)
  insu    374  (48.7%)
  mass     11  (1.4%)

3.2 Imputation

You now have holes, and most algorithms cannot consume a NaN. The options are to drop the rows, drop the columns, or fill the gaps. Dropping rows here would cost you half the dataset, because insu alone is missing for 374 of 768 patients. So: fill.

SimpleImputer replaces each gap with a summary of the column.

from tuiml.preprocessing import SimpleImputer

imputer = SimpleImputer(strategy="median")
X_imputed = imputer.fit_transform(X_nan)

insu = data.feature_names.index("insu")
print("insu before:", X_nan[:6, insu])
print("insu after :", X_imputed[:6, insu])
print()
print("remaining NaNs:", int(np.isnan(X_imputed).sum()))

insu before: [ nan  nan  nan  94. 168.  nan]
insu after : [125. 125. 125.  94. 168. 125.]

remaining NaNs: 0

strategy may be "mean", "median", "most_frequent" or "constant" (with fill_value). Median is the safer default for skewed numeric data — insulin has a long right tail, and the mean of a skewed column sits somewhere no patient actually is.

KNNImputer is the smarter option: it fills each gap using the most similar complete rows, so a missing insulin value is estimated from patients with comparable glucose and BMI rather than from the population as a whole.

from tuiml.preprocessing import KNNImputer

knn_imputed = KNNImputer(n_neighbors=5).fit_transform(X_nan)

print(f"median fill for insu : {X_imputed[2, insu]:.1f}")
print(f"KNN fill for insu    : {knn_imputed[2, insu]:.1f}")
print()
print("The median gives every patient the same value. KNN does not.")
print("distinct imputed insu values — median:",
      len(np.unique(X_imputed[np.isnan(X_nan[:, insu]), insu])),
      "| KNN:", len(np.unique(knn_imputed[np.isnan(X_nan[:, insu]), insu])))

median fill for insu : 125.0
KNN fill for insu    : 164.6

The median gives every patient the same value. KNN does not.
distinct imputed insu values — median: 1 | KNN: 301

3.3 Did it help?

Here is the step almost every tutorial omits. We have done some careful, well-motivated cleaning. Measure it.

def cross_validate(make_model, X_data, steps=(), n_splits=10, seed=42):
    """Cross-validate, fitting every preprocessing step inside the fold.

    steps is a sequence of zero-argument factories, so each fold gets a
    fresh, unfitted transformer — which is the entire point of section 3.7.
    """
    splitter = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=seed)
    accuracies, aucs = [], []

    for train_idx, test_idx in splitter.split(X_data, y):
        X_tr, X_te = X_data[train_idx].copy(), X_data[test_idx].copy()

        for make_step in steps:
            step = make_step()
            X_tr = step.fit_transform(X_tr)     # fit on train
            X_te = step.transform(X_te)         # apply to test

        model = make_model().fit(X_tr, y[train_idx])
        accuracies.append(accuracy_score(y[test_idx], model.predict(X_te)))
        aucs.append(roc_auc_score(y[test_idx], model.predict_proba(X_te)[:, 1]))

    return np.mean(accuracies), np.mean(aucs)


forest = lambda: RandomForestClassifier(n_estimators=200, random_state=42)
median = lambda: SimpleImputer(strategy="median")

raw_acc, raw_auc = cross_validate(forest, X)
clean_acc, clean_auc = cross_validate(forest, X_nan, steps=[median])

print(f"{'':28s} {'accuracy':>9s} {'AUC':>8s}")
print(f"{'raw zeros, untouched':28s} {raw_acc:9.4f} {raw_auc:8.4f}")
print(f"{'zeros -> NaN -> median':28s} {clean_acc:9.4f} {clean_auc:8.4f}")
print(f"{'difference':28s} {clean_acc - raw_acc:+9.4f} {clean_auc - raw_auc:+8.4f}")

                              accuracy      AUC
raw zeros, untouched            0.7668   0.8226
zeros -> NaN -> median          0.7695   0.8263
difference                     +0.0027  +0.0037

Essentially nothing. A few thousandths, well inside the fold-to-fold noise we measured in chapter 2.

This is worth sitting with rather than explaining away, because the honest conclusion is not "cleaning is pointless". It is this: a random forest did not need the help. A tree splits on thresholds, and insu <= 0.5 is a perfectly good split — so the forest had already learned to treat the zero group separately. We told it something it had worked out on its own.

The lesson generalises, and it is the most useful thing in this chapter:

Remark — preprocessing requirements are a property of the algorithm, not of the data. There is no such thing as a correctly preprocessed dataset in the abstract. There is only a dataset prepared for a particular model. The next section makes that concrete.

3.4 Scaling, and why it depends entirely on the model

Features on wildly different numeric ranges break any algorithm that measures distance. In this dataset insu runs to 846 while pedi stays under 2.5, so a Euclidean distance is almost entirely a statement about insulin.

To make the effect unmissable, imagine someone recorded age in a different unit — a mistake that happens constantly in real data pipelines:

X_bad_units = X.copy()
X_bad_units[:, data.feature_names.index("age")] *= 1000

knn = lambda: KNearestNeighborsClassifier(k=5)

print(f"{'':34s} {'KNN':>8s} {'forest':>8s}")
for label, X_variant in [("original units", X), ("age recorded x1000", X_bad_units)]:
    k_acc, _ = cross_validate(knn, X_variant)
    f_acc, _ = cross_validate(forest, X_variant)
    print(f"{label:34s} {k_acc:8.4f} {f_acc:8.4f}")

                                        KNN   forest

original units                       0.7226   0.7668

age recorded x1000                   0.6875   0.7668

The unit error costs the nearest-neighbour classifier several points — it drops most of the way to the 0.65 do-nothing baseline, because age now dominates every distance computation and the other seven features may as well not exist.

The random forest scores exactly the same number in both rows. Not approximately; identically. Trees split on one feature at a time and only care about the order of values, which multiplying by 1000 does not change.

Now add a scaler:

from tuiml.preprocessing import StandardScaler

scaled_acc, _ = cross_validate(knn, X_bad_units, steps=[StandardScaler])
print(f"KNN, age x1000, standardised: {scaled_acc:.4f}   (repaired)")

KNN, age x1000, standardised: 0.7265   (repaired)

StandardScaler subtracts the mean and divides by the standard deviation, so every column arrives centred at 0 with unit variance and the damage is undone. MinMaxScaler squeezes each column into a fixed range instead, which is what you want when an algorithm expects bounded input or when you need to preserve exact zeros.

Scale for: k-nearest neighbours, SVMs, neural networks, PCA, k-means, and any regularised linear model — anything that computes a distance or penalises coefficient size.

Do not bother for: decision trees, random forests, gradient boosting. It cannot help them, and it costs you the interpretable units on your feature values.

3.5 The leakage trap

Look again at cross_validate in section 3.3. Inside the fold loop it calls fit_transform on the training rows and transform on the test rows. Never fit_transform on both.

That ordering is the whole ballgame. Here is the wrong version — impute once, up front, on the entire dataset, and then cross-validate:

# WRONG: the imputer sees every row, including rows that will later be test data.
X_leaked = SimpleImputer(strategy="median").fit_transform(X_nan.copy())
leak_acc, leak_auc = cross_validate(forest, X_leaked)

print(f"{'':34s} {'accuracy':>9s} {'AUC':>8s}")
print(f"{'imputed inside the fold (right)':34s} {clean_acc:9.4f} {clean_auc:8.4f}")
print(f"{'imputed before splitting (wrong)':34s} {leak_acc:9.4f} {leak_auc:8.4f}")

                                    accuracy      AUC
imputed inside the fold (right)       0.7695   0.8263
imputed before splitting (wrong)      0.7669   0.8266

The difference is a rounding error, and I am not going to pretend otherwise.

That is because of what leaked. The median of a column is a very stable statistic: computed on 90% of the rows or 100% of them, it is nearly the same number, so the test set learned almost nothing about itself. The leak is real but tiny.

It is not always tiny. The size of a leak scales with how much the transform depends on individual rows:

Transform	What leaks	Size of leak
Median / mean imputation	one number per column	negligible, as above
Standardisation	two numbers per column	small
Feature selection	which columns survive, chosen using test labels	large — chapter 5
Resampling (SMOTE)	synthetic rows built from test rows	large — chapter 6
Target encoding	the target itself, directly	catastrophic

Chapter 5 builds a dataset where doing selection outside the fold produces a model that scores far above chance on pure noise. That is the demonstration this section cannot honestly give you with a median.

Remark — get the discipline right anyway. The correct habit costs nothing and the failure is silent. Do not learn to eyeball which transforms are "safe enough" to fit on everything; fit every one of them inside the fold, and you never have to make that judgement. Chapter 4 introduces Workflow, which makes the correct version the easy version — you stop writing the fold loop at all.

3.6 Outliers, encoding, discretisation

The rest of the preprocessing catalog, briefly. Each of these is a Transformer with the same fit / transform / fit_transform interface, so everything above applies unchanged.

Outliers. IQROutlierDetector finds values beyond factor interquartile ranges of the quartiles and either clips or removes them. ValueClipper caps at bounds or percentiles you choose.

Watch what happens when we apply it to the column we just imputed:

from tuiml.preprocessing import IQROutlierDetector, ValueClipper

clipped = IQROutlierDetector(factor=1.5, action="clip").fit_transform(X_imputed.copy())

print(f"insu before clipping: min {X_imputed[:, insu]. min():6.1f}  "
      f"max {X_imputed[:, insu].max():6.1f}")
print(f"insu after clipping : min {clipped[:, insu].min():6.1f}  "
      f"max {clipped[:, insu].max():6.1f}")
print()
print("surviving distinct values:", len(np.unique(clipped[:, insu])))

insu before clipping: min   14.0  max  846.0
insu after clipping : min  112.9  max  135.9

surviving distinct values: 16

The column has been destroyed — a range of 14 to 846 squeezed into a band about 23 units wide.

This is not a bug in the detector, it is the order of our steps. We filled 374 of 768 insulin values with exactly 125.0. That single value is now the median, the 25th percentile and the 75th percentile all at once, so the interquartile range collapsed to nearly zero and "1.5 IQRs from the quartiles" became a very narrow window. Every real measurement got clipped as an outlier.

Do it the other way round and the column survives:

detect_first = IQROutlierDetector(factor=1.5, action="clip").fit_transform(X_nan.copy())
then_impute = SimpleImputer(strategy="median").fit_transform(detect_first)

print(f"clip first, then impute: min {then_impute[:, insu].min():6.1f}  "
      f"max {then_impute[:, insu].max():6.1f}")
print("surviving distinct values:", len(np.unique(then_impute[:, insu])))

clip first, then impute: min   14.0  max  360.6
surviving distinct values: 164

Remark — step order is part of the pipeline, not a detail of it. Steps interact. Imputing before outlier detection changes what counts as an outlier; scaling before imputation changes what the fill value means. This is a large part of why chapter 4 makes the pipeline an explicit object you can print and inspect, rather than a sequence of statements you have to hold in your head.

And be careful with outlier removal generally. An outlier is not automatically an error — in a medical dataset the extreme values are frequently the patients you most need to get right. Clip because you have a reason to believe the value is wrong, not because it is far from the mean.

Encoding. Models consume numbers, so categorical columns must be converted. OneHotEncoder gives each category its own binary column and implies no ordering. OrdinalEncoder maps categories to integers, which is right for genuinely ordered categories (small < medium < large) and wrong for unordered ones — encoding {red: 0, green: 1, blue: 2} tells the model that green sits between red and blue.

from tuiml.preprocessing import OneHotEncoder

colours = np.array([["red"], ["green"], ["blue"], ["red"]], dtype=object)
encoded = OneHotEncoder().fit_transform(colours)

print(encoded)

[[0. 0. 1.]
 [0. 1. 0.]
 [1. 0. 0.]
 [0. 0. 1.]]

Discretisation. EqualWidthDiscretizer, QuantileDiscretizer and MDLDiscretizer turn a continuous column into bins. Useful for rule-based models that want categorical input, and for making a model's output explainable in human terms ("patients over 50"). It always destroys information, so it needs a reason.

from tuiml.preprocessing import EqualWidthDiscretizer

age_col = data.feature_names.index("age")
binned = EqualWidthDiscretizer(n_bins=4).fit_transform(X_imputed.copy())

print("age raw   :", X_imputed[:8, age_col])
print("age binned:", binned[:8, age_col])
print()

bins, counts = np.unique(binned[:, age_col], return_counts=True)
for b, n in zip(bins, counts):
    print(f"  bin {int(b)}: {n:3d} patients")

age raw   : [50. 31. 32. 21. 33. 30. 26. 29.]
age binned: [1. 0. 0. 0. 0. 0. 0. 0.]

  bin 0: 498 patients
  bin 1: 189 patients
  bin 2:  68 patients
  bin 3:  13 patients

Note how lopsided those bins are. Equal width means each bin spans the same number of years, not that each holds the same number of patients — and since most of this cohort is young, the first bin swallows the majority. QuantileDiscretizer bins by equal frequency instead, which is usually what people actually want.

Recap

• Turn disguised missing values into np.nan, then impute. SimpleImputer(strategy="median") for a robust default, KNNImputer when you want each gap filled from similar rows.
• Measure whether a preprocessing step helped. Here, imputation moved a random forest by less than the fold-to-fold noise, because trees can already split the missing group off by themselves.
• Preprocessing requirements belong to the algorithm, not the dataset. A unit error costs KNN several points and costs a random forest exactly zero.
• Scale for distance-based and regularised models; skip it for trees.
• Fit every transform on the training fold only, then transform the test fold. The leak from a median is negligible; the leak from feature selection or resampling is not.
• Outlier clipping, encoding and discretisation all share the same interface — and all belong inside the fold.

Next: chapter 4 replaces the hand-written fold loop in this notebook with Workflow, which does the fit-on-train-only bookkeeping for you.