Week 15 — The Pickle: A Training Script Is Not Production¶
You have a model that beats the dummy and a ranked list Priya likes. Ana asks the question that ends the notebook phase: "what happens when your laptop is off and it's Tuesday night?" Nothing, currently. That's this week's problem.
Course details
Course: Applied ML Foundations for SaaS Analytics Who this is for: Engineers about to pickle a model and get paged for it.
This is still a training script. We will be explicit about what it is not: no feature store, no canary, no GDPR review, no CI. The GPU coding-specialist capstone is a different page (docs/ml/capstone.md).
🎯 What you will be able to do¶
- Wire
build_features→ time-based split → GBT → a versioned artifact - Call
pipelines.contract.validate/predict— extra keys and NaN fail loud - Pick a threshold from CS capacity, not from 0.5
- Attach reason codes to every listed customer — and read them before you ship
- Draw a crude drift picture (
tenure_so_far, this month vs train) - Time 80
predict()calls so you have a latency number, not a vibe - List what you are not deploying
Think of it like… a build artifact + an API contract.
The model file is a binary. The scaler is part of that binary. The request body is FEATURE_COLS. If any of those three drift independently, production is a silent wrong-number generator.
If you already write software¶
A training script is a scratchpad. Production is a contract.
This week is the one that separates “I trained a thing” from “we can call it on Tuesday.”
Explore (weeks 6–14)
│ time-based split, pick a model, write predict()
▼
Ship
predict(payload) -> {churn_score, flag_for_cs, model_version}
the same features as training (tenure_so_far, not lifetime tenure_days)
the scaler travels with the joblib
a latency budget (we time 80 calls)
a drift picture (did live traffic stop looking like train?)
Time-based split is not optional¶
train_test_split(..., shuffle=True) is fine for a homework iris set. It is a lie for SaaS. Customers in the “test” set would include people from the same week as train — and tomorrow’s traffic is next week. Split on time — an earlier snapshot for training, a later one for testing (not a signup_date cut; see the foot-gun below). Train on the past. Test on the future. Same rule as backtesting a trading strategy, or as not using tomorrow’s logs to tune today’s alert.
Picture the contract¶
from pipelines.contract import predict
# payload keys == FEATURE_COLS
# extra key → ValueError, missing field → ValueError, NaN → ValueError
# returns {"churn_score": float, "flag_for_cs": bool, "model_version": str}
If you cannot call that function without reaching back into a throwaway script, you do not have a model. You have a souvenir.
Laptop budget
No GPU. Aimed at ~8 GB RAM. Training uses a few thousand sampled customers (or short sequences) so this week should finish in a few minutes on CPU. The ideas are the same if you later set n=None and train on all ~49k rows.
import time
from datetime import datetime
from pathlib import Path
import joblib
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import (
average_precision_score,
precision_recall_curve,
roc_auc_score,
)
from sklearn.pipeline import Pipeline
from pipelines.contract import predict, validate
from pipelines.features import (
AS_OF_DEFAULT,
FEATURE_COLS,
build_features,
make_preprocessor,
)
from pipelines.split import snapshot_split
Architecture (the only diagram that matters)¶
warehouse CSVs
│ nightly job
▼
build_features(as_of) ──► backtest ──► train pipeline ──► artifact (joblib + metrics)
│ │
└── holdout report ▼
predict(payload) {score, flag, version}
│
▼
monitor: score volume,
feature histograms, weekly PR-AUC
Engineer mental model
Train on the past, test on the more recent past. Random shuffle is a unit test. A time wall is an integration test against reality.
as_of = AS_OF_DEFAULT
HORIZON = 30
# Stand 30 days in the past, learn from what happened by today, then score
# today's at-risk customers and check the next 30 days. Write the horizon on
# the artifact — it is part of what the score means.
train_df, y_train, test_df, y_test = snapshot_split(as_of, horizon_days=HORIZON)
train_as_of = as_of - pd.Timedelta(days=HORIZON)
print(f"train: at-risk on {train_as_of.date()}, labelled by {as_of.date()} n={len(train_df):,}")
print(f"test: at-risk on {as_of.date()}, labelled over the next {HORIZON} days n={len(test_df):,}")
print("Train rate", float(y_train.mean()), "Test rate", float(y_test.mean()))
pipe = Pipeline(
[
("prep", make_preprocessor()),
("model", GradientBoostingClassifier(
n_estimators=40, learning_rate=0.1, max_depth=2, random_state=42
)),
]
)
pipe.fit(train_df[FEATURE_COLS], y_train)
proba = pipe.predict_proba(test_df[FEATURE_COLS])[:, 1]
print(f"Holdout AUC: {roc_auc_score(y_test, proba):.3f}")
print(f"Holdout PR-AUC: {average_precision_score(y_test, proba):.3f}")
print(f"dummy PR-AUC: {float(y_test.mean()):.3f}")
Foot-gun: a signup-date cut is a tenure cut¶
The tempting time wall is one snapshot, sorted by signup_date, oldest 80% in train. It looks like "train on the past." It is not. At a fixed as_of, tenure_so_far = as_of − signup_date, so cutting on signup date is cutting on tenure.
one snapshot at as_of, cut on signup_date
tenure_so_far: 0 ──── 109 days │ 110 ─────────────── 882 days
test (new) │ train (old)
3.8% churn (209) │ 1.4% churn (323)
the model never sees a short tenure in training, then scores only short tenures
old_cut = build_features(as_of=as_of, n=None)["signup_date"].quantile(0.80)
snap = build_features(as_of=as_of, n=None)
early, late = snap[snap["signup_date"] <= old_cut], snap[snap["signup_date"] > old_cut]
print(f"signup cut → train tenure {early['tenure_so_far'].min()}–{early['tenure_so_far'].max()} days, "
f"test tenure {late['tenure_so_far'].min()}–{late['tenure_so_far'].max()} days")
print(f"backtest → train tenure {train_df['tenure_so_far'].min()}–{train_df['tenure_so_far'].max()} days, "
f"test tenure {test_df['tenure_so_far'].min()}–{test_df['tenure_so_far'].max()} days")
The first line prints two ranges that do not touch. On this file the signup-cut model scores AUC ≈ 0.70 against ≈ 0.75 for the backtest. That is not a disaster on the dashboard, and that is exactly the danger.
Everything the model knows about tenure it learned on customers older than 110 days. New customers churn at 2.6× the rate of old ones, and the model has never seen one. The number looks like a small regression; the real problem is a model scoring a population it was never trained on. It will look fine offline and fail the first month new signups dominate the list (Week 17 is what that month feels like on-call).
A backtest keeps both sides on the same tenure range and asks the question production will: standing on a date, who leaves next?
Ship / don't ship
Ship a model evaluated on a later snapshot than it trained on, with the horizon written in metrics.json. Don't ship a split on any column that is a function of as_of — signup_date, tenure_so_far, "days since last login" — without checking the two sides overlap.
Threshold from a staffing number¶
CS can call 80 accounts from this test window. We take the 80 highest scores and measure precision. That is the meeting.
BUDGET = 80
order = np.argsort(-proba)
top = order[:BUDGET]
picked = y_test.to_numpy()[top]
hits = picked.sum()
print(f"Calling {BUDGET} highest-risk test users catches {int(hits)} actuals "
f"({hits/BUDGET:.0%} precision at this budget).")
print(f"There were {int(y_test.sum())} events in the window; "
f"recall={hits / max(float(y_test.sum()), 1):.0%}.")
threshold = float(np.partition(proba, -BUDGET)[-BUDGET]) if len(proba) >= BUDGET else 1.0
# One backtest is one draw. Put the interval on the report (Week 11's bootstrap).
rng = np.random.default_rng(0)
y_np = y_test.to_numpy()
boot = []
for _ in range(500):
i = rng.integers(0, len(y_np), len(y_np))
boot.append(y_np[i][np.argsort(-proba[i])[:BUDGET]].mean())
print(f"precision@{BUDGET} 95% CI: {np.percentile(boot, 2.5):.0%}–{np.percentile(boot, 97.5):.0%}")
prec, rec, _thr = precision_recall_curve(y_test, proba)
fig, ax = plt.subplots(figsize=(6.5, 3.8))
ax.plot(rec, prec, color="#1d4ed8")
ax.set_xlabel("recall (catch rate)")
ax.set_ylabel("precision (when we call, we were right)")
ax.set_title("Precision–recall — pick a point your team can staff")
ax.scatter([hits / max(float(y_test.sum()), 1)], [hits / BUDGET], color="#b91c1c", zorder=5)
ax.annotate("80-call budget", xy=(hits / max(float(y_test.sum()), 1), hits / BUDGET),
xytext=(0.35, 0.55), textcoords="axes fraction",
arrowprops={"arrowstyle": "->", "color": "#b91c1c"})
plt.tight_layout()
plt.show()
Why is this customer on the list?¶
Priya's first question about any name is why. "The model said 0.99" is not an answer she can open a call with. Give every listed customer reason codes: the two or three columns that push their score up the most.
The simplest honest version needs no new library. For each column, reset it to a typical customer's value and rescore. The bigger the drop, the more that column is carrying this customer's score. It is a what-if test, one row at a time.
from pipelines.features import NUMERIC
typical = train_df[NUMERIC].median()
def reasons(model, row, k=2):
"""Columns whose typical value would lower this customer's score the most."""
base = model.predict_proba(row.to_frame().T[FEATURE_COLS])[:, 1][0]
drops = {}
for col in NUMERIC:
what_if = row.copy()
what_if[col] = typical[col]
drops[col] = base - model.predict_proba(what_if.to_frame().T[FEATURE_COLS])[:, 1][0]
top_cols = sorted(drops, key=drops.get, reverse=True)[:k]
return base, [f"{c}={row[c]:.4g} (+{drops[c]:.2f})" for c in top_cols]
for i in np.argsort(-proba)[:5]:
row = test_df.iloc[i]
score, why = reasons(pipe, row)
print(f"{row['user_id']} {row['plan_type']:<10} score={score:.3f} because {', '.join(why)}")
Read the output before you ship. On this file the top of the list is a row of pro accounts with scores near 1.0, and the reason is the same every time: mrr of about $101.1–$101.3. That is not a churn pattern. It is three training churners who happened to pay $101.14, $101.15 and $101.32, and a boosted tree that carved a leaf just for them. In the test month, none of the ~100 customers in that price band churn.
No aggregate metric showed this. AUC was fine. The reason codes showed it in five lines, because they force you to read the list the way Priya will. The fix is a guard rail on the tree, not a patch on the list: forbid leaves smaller than, say, 50 customers.
guarded = Pipeline([
("prep", make_preprocessor()),
("model", GradientBoostingClassifier(
n_estimators=40, learning_rate=0.1, max_depth=2, min_samples_leaf=50, random_state=42
)),
]).fit(train_df[FEATURE_COLS], y_train)
guarded_proba = guarded.predict_proba(test_df[FEATURE_COLS])[:, 1]
top_guarded = y_test.to_numpy()[np.argsort(-guarded_proba)[:BUDGET]].sum()
print(f"top score {proba.max():.3f} → {guarded_proba.max():.3f}; "
f"hits in the top {BUDGET}: {int(hits)} → {int(top_guarded)}")
for i in np.argsort(-guarded_proba)[:3]:
row = test_df.iloc[i]
score, why = reasons(guarded, row)
print(f"{row['user_id']} {row['plan_type']:<10} score={score:.3f} because {', '.join(why)}")
The top score falls from “certain” to a modest number that matches reality (the base rate is 2%), and the reasons now read like churn: brand-new free accounts that logged one event and never came back. The hit count barely moves — the guarded top 80 sits inside a plateau of ~2,000 customers with identical scores, so which 80 you call there is decided by sort order (the capstone makes that tie-break explicit). What changed is that the five names at the top are no longer noise. pipelines/train.py ships with this guard.
Math, translated
The what-if test is a crude, one-column-at-a-time cousin of SHAP values, which split a score into per-column contributions that add up exactly to the prediction, accounting for columns that interact. Reach for the shap library when you need contributions that sum correctly or have many interacting columns. Either way the contract is the same: every name on the list comes with a reason a human can check.
Watch out — a reason code is a description of the model, not of the customer
"Low usage" as a reason means the model leans on low usage for this person. It does not mean usage caused the risk, or that raising usage will save them (Week 11). Write reasons as “flagged because…”, never “at risk because…”.
The contract: validate + predict¶
Watch out
If the API re-implements feature math differently from build_features, you have two products. One function. The handler imports pipelines.contract. Extra keys (churn_date, email) and NaN fail loud — they do not become a silent 0.
version = as_of.strftime("%Y%m%d")
artifact = {
"pipeline": pipe,
"metrics": {"threshold": round(threshold, 4), "model_version": version},
}
demo = {k: train_df.iloc[0][k] for k in FEATURE_COLS}
demo["plan_type"] = str(demo["plan_type"])
print("demo payload →", predict(demo, artifact))
try:
validate({"mrr": 10})
except Exception as exc:
print("missing keys →", type(exc).__name__, exc)
try:
validate({**demo, "churn_date": "2024-07-01"})
except Exception as exc:
print("unknown key →", type(exc).__name__, exc)
t0 = time.perf_counter()
for _ in range(80):
predict(demo, artifact)
print("80 predict() calls", round(time.perf_counter() - t0, 3), "s")
predict already returns churn_score, flag_for_cs, and model_version. Do not invent a CustomerFeatures type unless you define it.
Drift — did this month stop looking like train?¶
We will not implement a full PSI monitor. We will overlay histograms. If the orange fill walks away from the blue, someone should get a Slack.
Math, translated (optional)
Population Stability Index is a fancy “how different are two histograms.” If you want a number, compare bin shares: sum( (p − q) * log(p / q) ). If you want a decision, look at the picture first.
fig, axes = plt.subplots(1, 3, figsize=(12, 3.3))
for ax, col in zip(axes, ["mrr", "log_usage", "tenure_so_far"]):
ax.hist(train_df[col], bins=30, density=True, alpha=0.55, label="train", color="#3b82f6")
ax.hist(test_df[col], bins=30, density=True, alpha=0.55, label="today's snapshot", color="#f59e0b")
ax.set_title(col)
ax.legend(fontsize=8)
plt.suptitle("If orange leaves blue, the world moved — re-check PR-AUC before celebrating")
plt.tight_layout()
plt.show()
dest = Path.cwd() / "artifacts" / version
dest.mkdir(parents=True, exist_ok=True)
joblib.dump({"pipeline": pipe, "features": FEATURE_COLS}, dest / "model.joblib")
print("Wrote", dest / "model.joblib")
print("Ship the joblib AND contract.py AND this week's commit hash. Same layout as pipelines.train.")
What this lesson is not¶
| Claimed in many “production” tutorials | Reality here |
|---|---|
| Feature versioning | FEATURE_COLS in a joblib |
| Error handling | validate() on types, extras, NaN |
| Drift detection | Three histograms of tenure_so_far etc. |
| Retraining triggers | “Look at the histograms + weekly PR-AUC” |
| Deployment | A directory on disk (artifacts/<date>/) |
Ship / don’t ship
You can ship a batch score from this pipeline: score tonight’s accounts, hand CS a CSV of 80 names. Do not ship a public HTTP API until the contract lives in a tested module, the artifact is in a registry, and someone owns the weekly PR-AUC dashboard.
Course recap (the actual skills, weeks 0–15)
| Week | How to think about it |
|---|---|
| 0 Python | Glue: dataclasses, a fit/predict class |
| 1 NumPy | SIMD / SQL on a typed column |
| 2 Pandas | Joins you already know; never explode the grain |
| 3 SQL / as_of | The warehouse is source of truth |
| 4 Charts | Pick the shape that matches the question |
| 5 Stats | "How often would luck look like this?" |
| 6 Features | API contract + a wall against the future |
| 7 Classifiers | Score, then a staffed threshold |
| 8 Labels | Horizon, imbalance, calibration |
| 9 Regression | MAE in real units |
| 10 Clusters | Personas, not APIs |
| 11 Ranking | Precision@k; beat a SQL sort |
| 12 PCA | JPEG / rotate the cloud |
| 13 Ensembles | Reviewers voting vs hunting leftovers |
| 14 Nets | Mixers + the four-line training step |
| 15 Pickle | Artifact + contract + capacity |
✍️ Write-up¶
In one page: (1) the time wall you used, (2) holdout AUC vs a dummy, (3) the 80-call precision with its bootstrap interval, (4) one drift risk, (5) what you refused to over-claim.
✍️ Exercise¶
When you can explain the week out loud, do the exercises. Starter: python exercises/ml/week-15/starter.py from the repo root.
Before you leave¶
Try one self-check (Predict / Diagnose / Choose / Defend). Write the answer before you open the block.
🔗 Next: the job, then (optionally) deep learning¶
You have a joblib and a contract. Ana's next sentence: "this doesn't ship until it can't lose to the dummy, automatically, every night." Week 16 is that gate. After that:
- 17 on-call + the score as a bot tool
- 18–20 optional pictures (CNN / RNN / attention) — not how CloudWave ships churn
- Capstone (optional, GPU): a coding-tool-use specialist, not this pickle