11. Working with an Agent

The agent is connected. It can profile your data, train models, tune them, benchmark a field of candidates and serve the winner — and it will do all of that faster than you can type.

This chapter is about the two skills that decide whether that is useful or dangerous: asking well, and checking afterwards.

The checking half is the important one. Everything chapters 2 through 8 said about not fooling yourself applies unchanged when the person running the experiment is a language model, except that the model is more fluent than you are and will describe a flawed result in confident, well-organised prose.

You will learn:

• prompt patterns that produce defensible experiments
• the specific ways an agent's ML goes wrong
• how to read what actually ran, rather than what you were told ran
• a verification checklist you can apply in a minute

Prerequisites: chapters 8 and 10.

import json
import numpy as np
from tuiml.agent import execute_tool

11.1 A session that goes well

Every tool call below is real; the assistant's turns are what a competent model does with them.

You: I have the built-in diabetes dataset. Profile it and tell me what you'd worry about before modelling.

profile = execute_tool("tuiml_profile_data", data="diabetes")

print("declared missing values:", profile["missing_values"] or "none")
for column in ["plas", "pres", "skin", "insu", "mass"]:
    print(f"  {column:6s} min={profile['numeric_stats'][column]['min']:6.1f}")
print("class distribution:", profile["class_distribution"])

declared missing values: none
  plas   min=   0.0
  pres   min=   0.0
  skin   min=   0.0
  insu   min=   0.0
  mass   min=   0.0
class distribution: {'0': 500, '1': 268}

Assistant: 768 rows, 8 numeric features. The dataset declares no missing values, but plas, pres, skin, insu and mass all have a minimum of 0, which is physiologically impossible — those are missing values encoded as zeros, and insu in particular will be affected heavily. preg also has zeros but those are real. The target is 65/35, so accuracy will be misleading; I'll report F1 and recall too. I suggest imputing the disguised zeros inside the CV fold and comparing a few algorithms.

That is the right answer, and it came from noticing a contradiction between two fields of the profile. Now the modelling:

You: Do that. Include a baseline, use 10-fold CV, and fix the seed.

baseline = execute_tool(
    "tuiml_train", algorithm="ZeroRuleClassifier", data="diabetes",
    cv=10, random_seed=42,
)
candidate = execute_tool(
    "tuiml_train", algorithm="LogisticRegression", data="diabetes",
    preprocessing=["SimpleImputer", "StandardScaler"], cv=10, random_seed=42,
)

print(f"baseline (ZeroRule) : {baseline['metrics']['cv_accuracy_score_mean']:.4f}")
print(f"logistic regression : {candidate['metrics']['cv_accuracy_score_mean']:.4f} "
      f"± {candidate['metrics']['cv_accuracy_score_std']:.4f}")

baseline (ZeroRule) : 0.6510
logistic regression : 0.7708 ± 0.0610

11.2 Prompt patterns

The difference between a good session and a bad one is mostly in what you asked for. Five patterns carry most of the weight.

Ask for the profile first, and read it. An assistant that starts by training has skipped the step where the problems live. "Profile it and tell me what you'd worry about" is a better opener than "train a model".

Specify the evaluation, or you will get the default. Section 11.3 shows exactly what the default is. Say "10-fold cross-validation, seed 42" and you have removed the largest source of unreliable numbers.

Demand a baseline. "Include ZeroRuleClassifier" costs nothing and calibrates everything else. An assistant will rarely add one unprompted.

Name the metric, and say why. "This is medical screening, so optimise recall and report precision alongside it." Otherwise you get accuracy, which chapter 2 established is the wrong metric here.

Ask it to compare, not to choose. "Benchmark these four with a significance test" produces evidence. "What's the best model?" produces an opinion, delivered with the same confidence either way.

A prompt combining all five:

Profile the data first and tell me what's wrong with it. Then benchmark logistic regression, a random forest, naive Bayes and ZeroRuleClassifier, with imputation inside the pipeline, 10-fold CV, seed 42. Report accuracy, F1 and recall with standard deviations, and run a significance test against the best model. Tell me which differences are real.

11.3 How agent-run ML goes wrong

Four failure modes, in rough order of how often they bite.

Underspecified calls silently take the defaults

Ask for "train a random forest on the diabetes data" with nothing else, and this is what runs:

lazy = execute_tool("tuiml_train", algorithm="RandomForestClassifier", data="diabetes")

print("evaluation method:", lazy["metadata"]["evaluation_method"])
print("seed used        :", lazy["random_seed"])
print("accuracy         :", round(lazy["metrics"]["accuracy_score"], 4))

evaluation method: holdout
seed used        : 478163327
accuracy         : 0.7763

A single holdout split, with a randomly chosen seed. Chapter 2 measured an 11-point spread across holdout splits on this dataset. So this number is one draw from a wide distribution, and because the seed was random, neither you nor the assistant can reproduce it.

Watch:

draws = [
    execute_tool("tuiml_train", algorithm="RandomForestClassifier",
                 data="diabetes")["metrics"]["accuracy_score"]
    for _ in range(5)
]

print("five identical requests:", [round(d, 4) for d in draws])
print(f"spread: {max(draws) - min(draws):.4f}")

five identical requests: [0.7829, 0.7829, 0.75, 0.7829, 0.7566]
spread: 0.0329

Five identical requests, five different answers. An assistant that ran this once and reported "77.6% accuracy" was not lying — it was reporting the number it got. It simply had no way to know the number was that unstable, because it only ran it once.

Remark — this is not a model failing to reason. It is the tool's defaults being permissive: recall from chapter 10 that tuiml_train has no required parameters. Underspecified calls succeed. The fix is on your side of the conversation — say cv=10, seed 42 — or in an instruction you give the assistant once, at the start.

Optimising against the number it reports

Chapter 7's lesson survives the transition to agents and gets worse. An assistant asked to "improve accuracy" will try configurations in a loop and report the best one it found — which is best_score from chapter 7.6, selected by maximisation, optimistically biased, and now produced by something that can try thirty variants while you make coffee.

Ask for the honest number explicitly: "report nested CV, or hold out a test set at the start and only touch it once at the end."

Preprocessing outside the pipeline

Chapters 3 to 6 established that transforms belong inside the fold. An assistant can call tuiml_preprocess on a whole dataset and then tuiml_train on the result — which is precisely the chapter 5 mistake, and it produced 82% accuracy on pure noise.

The preprocessing argument to tuiml_train does it correctly, inside the fold. A session that used a separate preprocessing call is the thing to look for.

Fluent narration of a weak result

The most insidious one. A model will write "the random forest achieved strong performance at 77.6% accuracy, substantially outperforming the baseline" about a number that is one unstable draw and eleven points above a do-nothing classifier. Nothing in that sentence is false. All of it is unearned.

11.4 Verifying what actually ran

Every tool result carries metadata about what was really done. This is the ground truth, and it does not depend on the summary you were given.

careful = execute_tool(
    "tuiml_train", algorithm="LogisticRegression", data="diabetes",
    preprocessing=["SimpleImputer", "StandardScaler"], cv=10, random_seed=42,
)

print(json.dumps(careful["metadata"], indent=2))
print()
print("seed:", careful["random_seed"])

{
  "algorithm": "LogisticRegression",
  "steps": [
    "simpleimputer",
    "standardscaler",
    "logisticregression"
  ],
  "evaluation_method": "cross_validate",
  "n_samples": 768
}

seed: 42

Three fields answer most questions.

evaluation_method — cross_validate or holdout. If it says holdout, you have one draw, and chapter 2 says one draw is not a result.

steps — the pipeline that actually ran, in order. If the assistant said it imputed and this list contains only the model, it did not.

random_seed — if this is a large arbitrary number rather than the one you asked for, the run is not reproducible.

And the per-fold scores are returned in full, so you never have to accept a mean on its own:

folds = careful["cv_results"]["scores"]["accuracy_score"]

print("per-fold accuracy:", [round(f, 3) for f in folds])
print(f"mean {np.mean(folds):.4f}  std {np.std(folds):.4f}  "
      f"range {min(folds):.3f} to {max(folds):.3f}")

per-fold accuracy: [0.701, 0.805, 0.727, 0.844, 0.831, 0.675, 0.857, 0.779, 0.711, 0.776]
mean 0.7708  std 0.0610  range 0.675 to 0.857

That spread is the context every reported mean needs. Two models whose means differ by less than this are not distinguishable, and chapter 8's tuiml_test_statistics will say so formally.

11.5 Reproducing the claim

The strongest check is to re-run the assistant's own configuration and confirm you get its number.

import tuiml

check = tuiml.train({
    "model": {"name": "LogisticRegression"},
    "data": "diabetes",
    "pipeline": [{"name": "SimpleImputer"}, {"name": "StandardScaler"}],
    "evaluation": {"cv": 10},
    "random_seed": 42,
})

agent_score = careful["metrics"]["cv_accuracy_score_mean"]
my_score = check.metrics_["cv_accuracy_score_mean"]

print(f"agent reported : {agent_score:.4f}")
print(f"reproduced     : {my_score:.4f}")
print(f"match          : {np.isclose(agent_score, my_score)}")

agent reported : 0.7708
reproduced     : 0.7708
match          : True

This works because of chapter 9: the configuration is data, so you can take what the assistant ran and run it yourself. A disagreement here means the assistant did something other than what it described — which is exactly the thing that is hard to detect from prose alone.

11.6 The checklist

Before acting on a result an assistant produced, in about a minute:

1. Was there a baseline? If ZeroRuleClassifier was not run, the headline number has no scale.
2. evaluation_method — CV or holdout? Holdout means one draw.
3. Is the seed the one you asked for? If not, the result is not reproducible.
4. Do steps match what you were told? The pipeline that ran, not the pipeline described.
5. Are the per-fold scores reported? A mean without a spread is not a result.
6. Is the metric the right one? Accuracy on imbalanced data is the default and usually wrong.
7. Was the winning configuration selected by search? Then the reported score is optimistic — ask for nested CV.
8. Can you reproduce it? Re-run the spec.

The pattern behind all eight: the assistant's summary is a claim; the tool metadata is evidence. Read the evidence.

Remark — this is not a reason to distrust agents. It is the same checklist you should apply to a colleague's result, or to your own from three weeks ago. What changes with an agent is throughput: it can generate more results per hour than you can carefully check, so the checking has to become a habit rather than an occasional impulse.

Recap

• Ask for the profile first, and read it — the problems live there.
• Specify evaluation, seed, baseline and metric. Unspecified means default, and the defaults are permissive.
• tuiml_train with no evaluation argument runs a single holdout with a random seed. Five identical requests gave five different answers.
• Ask an agent to compare with a significance test, not to choose.
• Failure modes: underspecified calls, optimising against the reported score, preprocessing outside the fold, and fluent narration of weak results.
• metadata.evaluation_method, metadata.steps and random_seed tell you what really ran.
• Per-fold scores come back in full. Never accept a mean alone.
• Re-run the spec to reproduce the claim. Chapter 9's format is what makes that a one-liner.

Next: chapter 12 moves the agent out of the chat window and into your own code — LangChain, Pydantic-AI, and a raw tool loop.