Skip to content

Week 14 — Neural Nets, Without the Mystique

Marcus asks the obvious follow-up to last week's bake-off: could a neural net beat the tree? You already know how to build one. This week you find out, on the same seven-column table, whether it is worth the extra machinery — and you are learning nets now so Weeks 18–20 make sense later.

Course details

Course: Applied ML Foundations for SaaS Analytics Who this is for: Engineers who keep hearing “just use a network.” This week is permission to not, plus a picture of what a net actually is.

We will not pretend we taught calculus-level backpropagation.

🎯 What you will be able to do

  • Draw a net as an assembly line of mixers + on/off switches
  • See why stacked linear layers collapse to one linear layer (so we need activations)
  • Regularize with dropout + early stopping, and read a train/val loss curve
  • Write the four-line PyTorch training step: zero_grad → forward → backward → step
  • Decide GBDT vs net on a tabular SaaS problem honestly

Think of it like… an assembly line of mixers.

Each hidden layer takes the previous numbers, mixes them (weighted sum), then puts each mix through a cheap non-linear switch (ReLU: “if negative, make it 0”). The last mixer outputs a churn score. Training is credit assignment: nudge every weight a tiny bit so tomorrow’s score is less wrong. The library does the calculus (backprop). You do the architecture and the data.

If you already write software

A neural net is not magic and not a brain. It is mixers + switches, stacked.

input numbers
    │
    ▼
linear mix     (weighted sum — the same idea as a spreadsheet SUMPRODUCT)
    │
    ▼
switch         (ReLU: if negative, emit 0; else pass through)
    │
    ▼
linear mix
    │
    ▼
switch
    │
    ▼
one number     (a churn score)

Without the switch, stacked mixers collapse into one mixer — you paid for depth and got a linear model. ReLU is the cheap non-linearity that makes depth mean something.

Why a GBT still wins on this table

CloudWave’s scoring row is 7 columns (FEATURE_COLS). A gradient-boosted tree will usually beat a small net here, train faster, and be easier to ship. You are learning nets this week so weeks 18–20 (usage-over-time, sequences, text) make sense — not because a net is the right churn model.

Picture the training loop

for each batch of customers:
    optimizer.zero_grad()                   # do not accumulate last batch's blame
    score = model(features)                 # forward
    loss  = how wrong is the score          # the complaint
    loss.backward()                         # fill in .grad on every weight
    optimizer.step()                        # nudge weights to be less wrong

That is the whole mystery. The library does the calculus (backprop). You pick the architecture, the data, the loss, and when to stop.

Laptop budget

No GPU. Aimed at ~8 GB RAM. Training uses a few thousand sampled customers (or short sequences) so this week should finish in a few minutes on CPU. The ideas are the same if you later set n=None and train on all ~49k rows.

import copy

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

from pipelines.features import AS_OF_DEFAULT, FEATURE_COLS, build_features, make_preprocessor
from pipelines.split import snapshot_split

Why activations exist — a 30-second proof

If every layer is just x → Wx + b, then two layers are W2(W1x + b1) + b2, which is some other W'x + b'. You built a logistic regression with extra typing.

ReLU (and friends) break that collapse. That is the whole reason they exist.

features ─► mix ─► ReLU ─► mix ─► ReLU ─► mix ─► sigmoid ─► score
              ▲                  ▲
              └── without these switches, the whole tower is one line

Math, translated — backprop in one sentence

After a batch of examples, we know how wrong the score was. Backprop walks backward through the assembly line and estimates “if I wiggle this weight, does the error go up or down?” Then we wiggle it the helpful way (gradient descent). You will not derive it this week. You will treat it like the compiler: necessary, already written.

# Collapse demo: two linear maps == one linear map
rng = np.random.default_rng(0)
X = rng.normal(size=(5, 3))
W1, b1 = rng.normal(size=(3, 4)), rng.normal(size=4)
W2, b2 = rng.normal(size=(4, 2)), rng.normal(size=2)
two_layers = (X @ W1 + b1) @ W2 + b2
W_eq, b_eq = W1 @ W2, b1 @ W2 + b2
one_layer = X @ W_eq + b_eq
print("Max difference between 2 linear layers and 1 equivalent layer:",
      np.max(np.abs(two_layers - one_layer)))
print("That number should be ~0. Activations are what make depth real.")

CloudWave bake-off — sklearn MLP vs logistic vs GBT

MLPClassifier is the sklearn stand-in. Real dropout lives in PyTorch (nn.Dropout) below. sklearn’s alpha is L2, not dropout.

train, y_train, test, y_test = snapshot_split(AS_OF_DEFAULT, horizon_days=30)  # the backtest from Week 11
prep = make_preprocessor()
X_train_t = prep.fit_transform(train[FEATURE_COLS])
X_test_t = prep.transform(test[FEATURE_COLS])

for name, model in [
    ("logreg", LogisticRegression(max_iter=1000)),
    ("gbt", GradientBoostingClassifier(n_estimators=40, max_depth=2, random_state=42)),
    ("mlp", MLPClassifier(hidden_layer_sizes=(16, 8), activation="relu",
                          max_iter=20, random_state=42)),
    ("mlp+L2", MLPClassifier(hidden_layer_sizes=(16, 8), activation="relu",
                             alpha=0.01, max_iter=20, random_state=42)),
]:
    model.fit(X_train_t, y_train)
    auc = roc_auc_score(y_test, model.predict_proba(X_test_t)[:, 1])
    print(f"{name:<18} AUC={auc:.3f}")

The one plot a net owes you: train vs validation loss

If train loss keeps falling and val loss turns up, you are memorizing. Early stopping = take the checkpoint when val was best. sklearn’s MLP gives you loss_curve_ (log-loss). Do not plot it against 1 − accuracy — different units, different story. Dropout is “randomly break mixers”; it is the nn.Dropout layer in the PyTorch net below, not alpha.

for early in [True, False]:
    mlp = MLPClassifier(hidden_layer_sizes=(24, 12), activation="relu",
                        max_iter=25, random_state=42, early_stopping=early,
                        validation_fraction=0.2, n_iter_no_change=5)
    mlp.fit(X_train_t, y_train)
    auc = roc_auc_score(y_test, mlp.predict_proba(X_test_t)[:, 1])
    print(f"early_stopping={early!s:<5}  epochs run={mlp.n_iter_:>2}  test AUC={auc:.3f}")

fig, ax = plt.subplots(figsize=(8, 3.4))
ax.plot(mlp.loss_curve_, label="train log-loss", color="#1d4ed8")
ax.set_xlabel("epoch (one pass over the data)")
ax.set_title("Learning curve — the thing early stopping is supposed to watch")
ax.legend()
plt.tight_layout()
plt.show()

The early_stopping=True run quits after a handful of epochs and scores below a coin flip. It is not a bad net — it is a net that never trained. sklearn's early stopping watches validation accuracy, and on a label that is 98% zeros, accuracy is flat from the first epoch (“predict nobody churns” is already 98% right). Flat looks like “stopped improving,” so it stops. It is the Week 8 accuracy trap, hiding inside a convenience flag.

Watch out — early stopping is only as good as the number it watches

Stop on a metric that moves when the model gets better at your job: validation log-loss or PR-AUC, not accuracy. If the library only offers accuracy, write the loop yourself — the PyTorch loop below is twelve lines.

PyTorch — NumPy with a tape recorder

Week 0 promised this. A torch.tensor is a NumPy array that remembers the recipe. loss.backward() walks the recipe and fills .grad on every weight. optimizer.step() nudges the weights the helpful way.

batch of rows
    │  zero_grad → forward
    ▼
  logits → loss
    │  backward  (the library’s calculus)
    ▼
  .grad on every weight
    │  step
    ▼
  slightly less-wrong weights

Install once: pip install torch (CPU is enough for this course).

try:
    import torch
    import torch.nn as nn
except ImportError:
    raise SystemExit("PyTorch is missing. Install with:  pip install torch") from None

torch.manual_seed(0)
x = torch.tensor([2.0, 3.0], requires_grad=True)
y = (x ** 2).sum()          # 4 + 9 = 13
y.backward()
print("x      ", x.tolist())
print("y      ", float(y))
print("x.grad ", x.grad.tolist(), "  ← d(x1²+x2²)/dx = 2x")

# Same CloudWave table, now as tensors. Carve a validation slice out of the
# TRAIN snapshot: we pick the best epoch on it, and open the test set once (Week 13).
X_fit, X_val, y_fit, y_val = train_test_split(
    np.asarray(X_train_t, dtype=np.float32), y_train.to_numpy(),
    test_size=0.2, stratify=y_train, random_state=0,
)
Xt = torch.tensor(X_fit)
yt = torch.tensor(y_fit, dtype=torch.float32).unsqueeze(1)
Xv = torch.tensor(X_val)

net = nn.Sequential(
    nn.Linear(Xt.shape[1], 16),
    nn.ReLU(),
    nn.Dropout(0.2),           # the actual dropout; off later with net.eval()
    nn.Linear(16, 1),          # one logit; sigmoid lives in the loss
)
opt = torch.optim.Adam(net.parameters(), lr=1e-2)
loss_fn = nn.BCEWithLogitsLoss()

train_losses, val_prs = [], []
best_pr, best_state = -1.0, None
for epoch in range(300):
    net.train()
    opt.zero_grad()
    loss = loss_fn(net(Xt), yt)
    loss.backward()
    opt.step()
    train_losses.append(float(loss))
    net.eval()
    with torch.no_grad():
        pr = average_precision_score(y_val, net(Xv).numpy().ravel())
    val_prs.append(pr)
    if pr > best_pr:                                   # early stopping, by hand:
        best_pr, best_state = pr, copy.deepcopy(net.state_dict())  # keep the best checkpoint

fig, axes = plt.subplots(1, 2, figsize=(10, 3.4))
axes[0].plot(train_losses, color="#1d4ed8")
axes[0].set_title("PyTorch train loss")
axes[0].set_xlabel("epoch")
axes[1].plot(val_prs, color="#0f766e")
axes[1].set_title("Validation PR-AUC — pick the checkpoint here")
axes[1].set_xlabel("epoch")
plt.tight_layout()
plt.show()

net.load_state_dict(best_state)
net.eval()
with torch.no_grad():
    test_scores = net(torch.tensor(np.asarray(X_test_t, dtype=np.float32))).numpy().ravel()
print(f"best epoch {int(np.argmax(val_prs))}   test AUC {roc_auc_score(y_test, test_scores):.3f}   "
      f"test PR-AUC {average_precision_score(y_test, test_scores):.3f}")
print("Remember the four calls: zero_grad → forward → backward → step.")

Watch out

Forget optimizer.zero_grad() and gradients pile up — the model walks off a cliff. Forget model.eval() later and dropout will stay on at serve time. A 32-16-8 net on 5 columns is still theatre: depth does not invent information that is not in the features.

Ship / don’t ship

Tabular SaaS, < ~100k rows, mixed columns → gradient-boosted trees.

Images, text, audio, long sequences → deep learning.

A net is not “more serious.” It is a different tool. Pick the one you can monitor at 3 a.m.

✍️ Exercise

When you can explain the week out loud, do the exercises. Starter: python exercises/ml/week-14/starter.py from the repo root.

🤔 Reflection

  1. What problem is the activation function solving, in your own words?
  2. Who owns backprop on your team — you, or the library? What do you still own?
  3. Name one product surface at CloudWave where a net would be the right call (e.g. search ranking on ticket text).
  4. In one sentence: what does loss.backward() put on each weight?

🔗 Next week

GBT is still what ships. Week 15 turns the winning model into something more durable than a notebook variable — a time split, predict(), and a versioned artifact. After that: the job pipeline, then (optionally) CNNs, RNNs, and Transformers.