Week 7 — Classification: A Score, Then a Threshold¶

Course: Applied ML Foundations for SaaS Analytics
Who this is for: Engineers who have written a spam filter, a linter, or a “risk score.” Same shape.

🎯 What you will be able to do¶

Explain a model as f(features) → score in [0, 1], then a threshold
Always beat a baseline (predict “nobody churns”) before celebrating AUC
Read a confusion matrix in customers, not jargon
See why precision vs recall is a staffing problem
Glance at a decision tree — the only model you can literally read
Recognize underfit (high bias) vs overfit (high variance) on a picture

Think of it like… a code-review bot.

The model does not “know” who will churn. It outputs a risk score, like a linter warning level. You choose the cutoff: flag everything above 0.3 (noisy, catch more) or only above 0.7 (quiet, miss more). The algorithm did not make that product decision. You did.

If you already write software¶

A classifier is not a fortune teller. It is a function that returns a score, and then you pick a threshold — exactly like a linter warning level or a WAF rule.

features  →  model  →  score in [0, 1]  →  if score >= t: flag
                                              ↑
                                    this is a product decision
                                    not an algorithm decision

Low threshold (0.3): noisy Slack channel, catch more real churn
High threshold (0.7): quiet channel, miss more real churn

Precision vs recall is a staffing problem. High recall means the CS team gets more names. Can they call them? If not, you did not “improve the model.” You created a junk queue.

Always beat a dummy¶

The dummy baseline here is “predict nobody churns” (or “predict the majority class”). If your fancy model cannot beat that, you built a weather app that says “today’s weather will be like yesterday” and lost to it.

Picture underfit vs overfit¶

Underfit (high bias)     a one-line linter that only flags `== null`
                         misses almost everything, stable and useless

Overfit (high variance)  a linter that memorized last Tuesday’s PR
                         perfect on the training set, random on the next one

The picture you want: a model that is slightly wrong on train and similarly wrong on a held-out week. That is generalization. Memorizing the training customers is not intelligence.

Laptop budget

No GPU. Aimed at ~8 GB RAM. Training uses a few thousand sampled customers (or short sequences) so this week should finish in a few minutes on CPU. The ideas are the same if you later set n=None and train on all ~49k rows.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from pipelines.features import build_features

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (roc_auc_score, roc_curve, confusion_matrix,
                             precision_score, recall_score, ConfusionMatrixDisplay)

What “logistic regression” actually is¶

Ignore the word regression. This is a classifier.

score = sigmoid( w1*mrr + w2*usage + w3*tenure + ... + b )
         └── squash any number into (0, 1)  — a score, not a calibrated chance

If the weighted sum is large and positive → score near 1 (likely churn).
If it is large and negative → score near 0.

A number in [0, 1] is not a calibrated probability

A classifier can output a number in [0, 1] without that number being empirically calibrated. For a calibrated model: predictions near 0.8 → about 80% actually positive. Until you check (Week 8, scikit-learn Probability Calibration), treat the score as a rank, not “80% chance they churn.”

Math, translated

The sigmoid is just a soft on/off switch. In symbols: z = w·x + b is the weighted sum (same z as any linear model), and p = σ(z) = 1 / (1 + e^-z) squashes it into (0, 1). Training picks w and b to minimize log loss — -(y·log(p) + (1-y)·log(1-p)) — which is just "how surprised was the model, on average, given what actually happened." A confident wrong answer (p=0.99 on a customer who stayed) costs a lot; a wobbly p=0.5 costs a little either way.

Before reaching for sklearn, here is the whole thing in NumPy — one score, one loss, one gradient step:

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# 3 toy customers, 2 features each (scaled mrr, scaled usage), plus a bias column of 1s
X_toy = np.array([[1.0, 0.2, 1.0], [0.1, 0.9, 1.0], [0.8, 0.7, 1.0]])
y_toy = np.array([0, 1, 1])
w = np.zeros(3)  # start with "no opinion" — every score is 0.5

p = sigmoid(X_toy @ w)
loss = -np.mean(y_toy * np.log(p) + (1 - y_toy) * np.log(1 - p))
grad = X_toy.T @ (p - y_toy) / len(y_toy)  # (prediction - truth), averaged, fed back through X
w = w - 0.5 * grad  # one gradient-descent step: nudge w opposite the gradient

print("start loss:", round(loss, 3))
print("weights after one step:", w.round(3))

That p - y_toy term is the whole engine: score too high on a churner → push weights down; score too low → push up. LogisticRegression.fit() below just repeats this thousands of times on real features until the gradient is ~0.

loss
 ^
 |   ● start (random w)         prediction error
 |    \                               ↓
 |     ●                             loss
 |      \                             ↓
 |       ●                        gradient  (which way is downhill?)
 |        \                           ↓
 |         ● minimum (fit w)    parameter update  (w = w - lr * grad)
 +----------------------→ w            ↓
                                  new prediction (repeat)

Same loop every epoch: score the batch, turn errors into a loss number, use the gradient to find downhill, nudge every weight a little, re-score. max_iter=1000 below is just "how many trips around that loop before we give up."

AUC (area under the ROC curve) is ranking quality: if you sort users by score, do the actual churners tend to sit at the top? Formally, it is the probability that a randomly chosen churner scores higher than a randomly chosen non-churner (ties split their credit 50/50). 0.5 is a coin flip. 1.0 is a perfect ranking. It is not accuracy, and it is not “percent chance they churn.”

df = build_features(as_of="2024-06-01")  # ~8k laptop sample; features stop at as_of
numeric = ["mrr", "tenure_so_far", "log_usage", "features_adopted", "total_events", "n_support"]
categorical = ["plan_type"]
X = df[numeric + categorical]
y = df["is_churned"].astype(int)  # lifetime flag — Week 8 replaces this label

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

prep = ColumnTransformer([
    ("num", StandardScaler(), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])

def pipe(model):
    return Pipeline([("prep", prep), ("model", model)])

# Baseline first. Always.
dummy = pipe(DummyClassifier(strategy="most_frequent"))
dummy.fit(X_train, y_train)
print(f"Majority-class accuracy: {dummy.score(X_test, y_test):.3f}")
print(f"Majority-class AUC:      {roc_auc_score(y_test, dummy.predict_proba(X_test)[:,1]):.3f}  (0.5 = coin flip on ranking)")
print("If your fancy model cannot beat this, it is not fancy.")

Train three models, read one of them¶

A shallow decision tree is a flowchart. Random forest is a committee of those flowcharts. Logistic regression is the weighted sum.

logreg = pipe(LogisticRegression(max_iter=1000))
tree = pipe(DecisionTreeClassifier(max_depth=3, min_samples_leaf=200, random_state=42))
forest = pipe(RandomForestClassifier(n_estimators=40, max_depth=6, random_state=42, n_jobs=2))

rows = []
for name, model in [("logreg", logreg), ("tree", tree), ("forest", forest)]:
    model.fit(X_train, y_train)
    proba = model.predict_proba(X_test)[:, 1]
    pred = (proba >= 0.5).astype(int)
    rows.append({
        "model": name,
        "AUC": roc_auc_score(y_test, proba),
        "precision": precision_score(y_test, pred, zero_division=0),
        "recall": recall_score(y_test, pred, zero_division=0),
    })
print(pd.DataFrame(rows).round(3).to_string(index=False))

fig, ax = plt.subplots(figsize=(10, 5))
# plot the raw tree (need the trained DecisionTree inside the pipeline)
ohe_names = list(tree.named_steps["prep"].named_transformers_["cat"].get_feature_names_out(categorical))
plot_tree(tree.named_steps["model"], feature_names=numeric + ohe_names,
          class_names=["stay", "churn"], filled=True, max_depth=3, fontsize=7, ax=ax)
ax.set_title("A 3-level tree — read it like a product flowchart")
plt.tight_layout()
plt.show()

Confusion matrix + threshold slider¶

At threshold 0.5 the library yells “positive.” Your CS team can call 80 accounts a week. That is the real threshold.

                    predicted stay     predicted churn
actually stay       true negative      false alarm      ← wasted CS time
actually churned    miss               catch            ← saved revenue

Engineer mental model

Precision = “when we page CS, how often were we right?” Recall = “of everyone who churned, how many did we catch?” You cannot max both at a fixed staffing level. Pick the one that matches the cost of a miss vs a wasted call.

Before you run this¶

Predict:

If you raise the threshold from 0.2 to 0.7, which metric will improve (precision or recall)?
Which will worsen?
Why?

Run it¶

Compare the three confusion matrices with your prediction.

Explain the difference¶

If your prediction was wrong, what assumption was wrong?

proba = forest.predict_proba(X_test)[:, 1]

fig, axes = plt.subplots(1, 3, figsize=(12, 3.6))
for ax, thr in zip(axes, [0.2, 0.5, 0.7]):
    pred = (proba >= thr).astype(int)
    ConfusionMatrixDisplay(confusion_matrix(y_test, pred)).plot(ax=ax, colorbar=False)
    prec = precision_score(y_test, pred, zero_division=0)
    rec = recall_score(y_test, pred, zero_division=0)
    ax.set_title(f"thr={thr}  P={prec:.2f} R={rec:.2f}\nflagged={pred.sum()}")
plt.tight_layout()
plt.show()

fpr, tpr, _ = roc_curve(y_test, proba)
fig, ax = plt.subplots(figsize=(5, 4))
ax.plot(fpr, tpr, color="#1d4ed8", label=f"forest AUC={roc_auc_score(y_test, proba):.3f}")
ax.plot([0, 1], [0, 1], ls="--", color="#94a3b8", label="coin flip AUC=0.50")
ax.set_xlabel("false alarm rate")
ax.set_ylabel("catch rate (recall)")
ax.set_title("ROC: ranking quality, independent of one threshold")
ax.legend()
plt.tight_layout()
plt.show()

Watch out

A random split is convenient and slightly dishonest for time-stamped customers. We fix that in Week 15.
Never rank “top 20% risk” on the training rows and call it a holdout result.
0.5 is not a sacred threshold. It is sklearn’s default because someone had to pick a number.

Ship / don’t ship

Ship a classifier when it beats the dummy on AUC and you have picked a threshold from a capacity number (“CS can call 50/week”). AUC alone does not page anyone.

CloudWave’s lifetime churn is ~6.4%. Accuracy is a trap and a 0.7 score is not “70% chance.” Week 8 is labels, PR-AUC, and calibration. Week 11 is the list CS actually uses.

Overfitting, bias, and variance — the three words on every ML interview¶

A model can fail in two opposite ways:

UNDERFIT (high bias)              OVERFIT (high variance)
a line through a curve            a scribble through every point
too simple — misses the shape     too clingy — memorizes noise
train error HIGH                  train error TINY
test error HIGH                   test error HIGH  ← the tell

Think of it like… studying for an exam.

Bias is showing up with only one idea (“everyone churns if they are free”). You are systematically wrong, even on the homework.

Variance is memorizing last year’s answer key, typos included. Homework is perfect. The real exam (new customers) is a mess.

Overfitting is the name for that second failure. Underfitting is the first.

# Toy picture: a smooth truth, noisy homework, three students
rng = np.random.default_rng(0)
x = np.linspace(0, 1, 40)
truth = np.sin(2 * np.pi * x)
y = truth + rng.normal(0, 0.18, size=len(x))

fig, axes = plt.subplots(1, 3, figsize=(12, 3.4), sharey=True)
for ax, deg, title in [
    (axes[0], 1, "Underfit — high bias"),
    (axes[1], 3, "About right"),
    (axes[2], 14, "Overfit — high variance"),
]:
    coef = np.polyfit(x, y, deg)
    xx = np.linspace(0, 1, 200)
    ax.scatter(x, y, s=12, color="#64748b", label="train points")
    ax.plot(xx, np.sin(2 * np.pi * xx), color="#0f766e", lw=2, label="truth")
    ax.plot(xx, np.polyval(coef, xx), color="#dc2626", lw=2, label=f"poly deg {deg}")
    ax.set_title(title)
    ax.set_ylim(-1.8, 1.8)
axes[0].legend(fontsize=8, loc="lower left")
plt.tight_layout()
plt.show()
print("Same data. Only the model's freedom changed. That freedom is 'capacity.'")

Math, translated

Bias ≈ how far the model’s average answer sits from the truth (systematic miss).

Variance ≈ how much the answer would jump if you retrained on a different sample of customers.

You cannot drive both to zero. A deeper tree / bigger net lowers bias and raises variance. Regularization, more data, and ensembles are how you buy the pair you can live with.

Engineer mental model

Watch two curves: train vs holdout. If both are bad → underfit (add features, more capacity). If train is great and holdout is not → overfit (simpler model, more data, regularization). Never tune on the number you will report.

✍️ Exercise¶

When you can explain the week out loud, do the exercises. Starter: python exercises/ml/week-07/starter.py from the repo root.

🤔 Reflection¶

Why can accuracy be ~94% while the model is useless? (Hint: ~6.4% of users churn.)
A PM wants “both high precision and high recall.” What resource do they need to give you?
Would you rather explain a depth-3 tree or a 150-tree forest to legal?

🔗 Next week¶

Week 8 — labels. Lifetime is_churned plus lifetime tenure_days is a tenure detector. We will cut the answer key at as_of the same way Week 6 cut the features. Regression is Week 9.