13. Serving and Deploying

A model that lives in a notebook has not done anything yet. This chapter puts one behind an HTTP endpoint, calls it, and then covers the parts of production that the endpoint does not solve for you.

You will learn:

• how to serve a fitted pipeline in one call
• the REST endpoints you get, and their exact request shapes
• why serving the pipeline rather than the model is the thing that prevents a whole class of production bug
• how to apply chapter 2's threshold decision at serving time
• workers, Docker, and what serving deliberately does not give you

Prerequisites: chapters 4 and 6.

import json
import time
import urllib.request

import numpy as np
import tuiml

13.1 Train something worth serving

The model from chapter 7's bake-off — logistic regression, with imputation and scaling in the pipeline.

model = tuiml.train({
    "model": {"name": "LogisticRegression"},
    "data": "diabetes",
    "pipeline": [
        {"name": "SimpleImputer", "params": {"strategy": "median"}},
        {"name": "StandardScaler"},
    ],
    "evaluation": {"cv": 10},
    "random_seed": 42,
})

print(f"accuracy: {model.metrics_['cv_accuracy_score_mean']:.4f} "
      f"± {model.metrics_['cv_accuracy_score_std']:.4f}")

accuracy: 0.7708 ± 0.0610

13.2 Serving it

info = tuiml.serve(model, port=8850, model_id="diabetes")

time.sleep(2)   # give the server a moment to bind

print(json.dumps(info, indent=2))

{
  "server_id": "127.0.0.1:8850",
  "host": "127.0.0.1",
  "port": 8850,
  "model_id": "diabetes",
  "url": "http://127.0.0.1:8850",
  "endpoints": {
    "predict": "http://127.0.0.1:8850/models/diabetes/predict",
    "health": "http://127.0.0.1:8850/health",
    "docs": "http://127.0.0.1:8850/docs"
  }
}

The server runs in a background thread and serve returns immediately with its address. background=False blocks instead, which is what you want when the server is your process rather than something you started from a notebook.

What is now behind that port is the entire pipeline — imputer, scaler and model. That is the detail worth pausing on.

A raw patient record arrives with a 0 where insulin was not measured, on the original numeric scales. The endpoint imputes it and standardises it using the statistics learned at training time, and only then predicts. Had you serialised just the LogisticRegression, you would now be reimplementing median imputation and standardisation in your serving code, from memory, and every future retrain would silently invalidate the constants you hard-coded there.

That mismatch — training preprocessing and serving preprocessing drifting apart — is one of the most common ways a model that validated well behaves badly in production. Chapter 4's save/serve carrying the whole pipeline is the structural fix.

13.3 The endpoints

base = info["url"]


def get(path):
    """GET a JSON endpoint on the running server."""
    with urllib.request.urlopen(base + path, timeout=10) as response:
        return json.loads(response.read())


def post(path, payload):
    """POST JSON to the running server and return the decoded reply."""
    request = urllib.request.Request(
        base + path,
        data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json"},
    )
    with urllib.request.urlopen(request, timeout=10) as response:
        return json.loads(response.read())


print("health:", get("/health"))
print("models:", get("/models"))

health: {'status': 'healthy', 'version': '0.2.0', 'models_loaded': 1}
models: {'models': ['diabetes'], 'count': 1}

Method	Path	Purpose
GET	/health	Liveness — status, version, models loaded
GET	/models	Which models this server holds
GET	/models/{id}	Metadata for one model
POST	/models/{id}/predict	Predictions
POST	/models/{id}/predict_proba	Class probabilities
GET	/stats	Request counts
GET	/docs	Interactive OpenAPI documentation

/docs is worth opening in a browser while you develop — it is a live, executable description of the API.

13.4 Predicting

The request body is {"features": [[...], [...]]} — always a 2-D array, even for one row.

patients = [
    [6, 148, 72, 35, 0, 33.6, 0.627, 50],    # high glucose, older
    [1, 85, 66, 29, 0, 26.6, 0.351, 31],     # unremarkable
    [10, 125, 70, 26, 115, 31.1, 0.205, 41],  # borderline — actually diabetic
]

print(post("/models/diabetes/predict", {"features": patients}))

{'predictions': [1, 0, 0], 'model_id': 'diabetes', 'model_class': 'Workflow'}

Note that those rows contain a raw 0 for insulin — untouched, exactly as it would arrive from a clinical system. The server's imputer handled it.

The same thing with curl:

curl -X POST http://127.0.0.1:8850/models/diabetes/predict \
  -H 'Content-Type: application/json' \
  -d '{"features": [[6, 148, 72, 35, 0, 33.6, 0.627, 50]]}'

13.5 Probabilities, and the threshold

Chapters 2 and 6 established that the 0.5 cut-off is a policy decision, not a property of the model — and that on a medical screening problem it is the wrong one.

/predict applies 0.5. To make your own decision, ask for probabilities:

proba = post("/models/diabetes/predict_proba", {"features": patients})
print(json.dumps(proba, indent=2))

{
  "probabilities": [
    [
      0.28211443260188673,
      0.7178855673981133
    ],
    [
      0.9499578256495229,
      0.05004217435047713
    ],
    [
      0.5589866618394641,
      0.4410133381605359
    ]
  ],
  "classes": null,
  "model_id": "diabetes"
}

THRESHOLD = 0.35        # chosen in chapter 6, where missing a case costs more

positive = [p[1] for p in proba["probabilities"]]

print(f"{'patient':>8s} {'P(positive)':>12s} {'at 0.5':>8s} {'at 0.35':>9s}")
for i, p in enumerate(positive):
    print(f"{i:>8d} {p:12.3f} {int(p >= 0.5):>8d} {int(p >= THRESHOLD):>9d}")

 patient  P(positive)   at 0.5   at 0.35
       0        0.718        1         1
       1        0.050        0         0
       2        0.441        0         1

Patient 2 is the one to look at. The model puts them below 0.5, so /predict reports "negative" — and this row is taken from the dataset, where the recorded outcome is positive. At a threshold of 0.35 they are flagged.

One row is an anecdote, not evidence; chapter 6 has the aggregate version, where dropping the threshold moved recall from 0.53 to 0.83. But it is a concrete picture of what that aggregate means: a real person whom the default cut-off sends home.

Remark — put the threshold where you can change it. It belongs in your application's configuration, not baked into the model artefact, because it is the number most likely to need adjusting after the model meets real traffic. Serving probabilities and thresholding downstream means changing the policy does not require a redeploy of the model.

Remark. predict_proba returns "classes": null here rather than the class labels, so the caller has to know that index 1 is the positive class. Worth pinning down in your client code with a named constant rather than a bare [1].

13.6 Shutting down

print("running:", tuiml.server_status())
tuiml.stop_server()
time.sleep(1)
print("after stop:", tuiml.server_status())

running: [{'server_id': '127.0.0.1:8850', 'host': '127.0.0.1', 'port': 8850, 'model_id': 'diabetes', 'url': 'http://127.0.0.1:8850'}]

after stop: []

13.7 Serving without a notebook

A saved pipeline serves directly from disk — this is the normal deployment path, where training and serving are different processes on different machines.

model.save("diabetes.joblib")

tuiml serve diabetes.joblib --port 8000
tuiml serve diabetes.joblib --host 0.0.0.0 --port 8000 --workers 4
tuiml status          # what is running
tuiml stop-server     # stop it

Or from Python, in a process whose whole job is to serve:

import tuiml

tuiml.serve("diabetes.joblib", host="0.0.0.0", port=8000, background=False)

background=False blocks, which is what a container's entrypoint should do.

13.8 Production notes

Workers. One worker is one Python process and therefore one CPU core under load. --workers 4 forks four, each with its own copy of the model — so memory scales with worker count, which matters for a large ensemble.

Bind address. 127.0.0.1 accepts only local connections. 0.0.0.0 accepts anything that can reach the port, which inside a container is what you want and on a public host is not.

There is no authentication. The endpoint is open to whoever can reach it. Put it behind a reverse proxy, an API gateway or a service mesh — whatever your infrastructure already uses for this — and do not expose it directly.

Docker. A minimal image:

FROM python:3.12-slim

# TuiML compiles C++ extensions at install time.
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential cmake && rm -rf /var/lib/apt/lists/*

RUN pip install --no-cache-dir tuiml

COPY diabetes.joblib /app/model.joblib
WORKDIR /app

EXPOSE 8000
CMD ["tuiml", "serve", "model.joblib", "--host", "0.0.0.0", "--port", "8000"]

docker build -t diabetes-model .
docker run -p 8000:8000 diabetes-model

Reverse proxy. Behind nginx:

location /model/ {
    proxy_pass http://127.0.0.1:8000/;
    proxy_set_header Host $host;
}

13.9 What serving does not give you

Being explicit, because an endpoint that returns 200 can feel like the finish line.

Monitoring. /stats counts requests. It does not tell you whether the predictions are any good — for that you need outcomes, which arrive later and have to be joined back to the predictions that anticipated them. Log every request with its model version and a correlation id, or you will not be able to reconstruct this later.

Drift detection. The world changes; the training data does not. Watch the distribution of incoming features against the training distribution, and watch your metrics against the numbers from chapter 8. A model whose input distribution has moved is a model whose validation is out of date.

Versioning. model_id distinguishes models on one server, not versions of one model over time. Deploy diabetes-v3 alongside diabetes-v2 and shift traffic deliberately; never quietly replace a file.

Retraining. No schedule, no trigger, no pipeline. That is your infrastructure's job — and chapter 9's specs are what make it tractable, since the retraining job is "run this JSON again on newer data".

Remark — the whole book applies at 3am. A model in production still has the class imbalance from chapter 6, the threshold decision from chapter 2, and the honest error bars from chapter 8. Deployment does not resolve any of them; it just means someone is now depending on the answers.

Recap

• tuiml.serve(model, port=...) starts a REST server in a background thread; background=False blocks.
• The whole pipeline is served, so raw records get exactly the preprocessing they got in training. This prevents training/serving skew.
• POST /models/{id}/predict with {"features": [[...]]} — always 2-D.
• /predict applies a 0.5 threshold. Use /predict_proba and threshold in your application, where you can change it without redeploying.
• tuiml serve model.joblib --workers 4 for the deployment path; background=False for a container entrypoint.
• No authentication. Put it behind a proxy.
• Serving gives you an endpoint. Monitoring, drift detection, versioning and retraining are still yours.

Next: chapter 14 runs the entire book end to end on one problem, from the raw file to the served endpoint.