Module 10 — Security operations center¶

02:07. One alert:

DET-003:alice  critical  "SSRF to instance metadata"  status=new

Two stories fit it. Alice’s account is being used by someone else to steal cloud credentials. Or a developer is testing the fetch feature on a workstation and forgot which mode the lab is in. Both explanations produce the same line. You have minutes, a playbook, and whatever the logs can actually show you.

The metric your manager watches is “time to acknowledge.” It will look great if you close the alert in thirty seconds, and it will look exactly as great if you close it wrongly. This module is about what happens between the alert and the decision.

Why it matters to a software engineer¶

When your service pages at 2 a.m., the people on the other side of the ticket speak “alerts, cases, severity, containment.” If you cannot walk an event from log line to case to fix, you will either ignore the SOC or drown it. This module is the operating model.

Visual overview¶

flowchart TB
  T[Telemetry] --> D[Detection]
  D --> A[Alert]
  A --> TRI[Triage]
  TRI -->|benign / false positive| TUNE[Tune with expiry and evidence]
  TRI --> INV[Investigation]
  INV --> INC[Incident]
  INC --> CON[Contain]
  CON --> ERA[Eradicate cause]
  ERA --> REC[Recover]
  REC --> LEARN[Lessons + control change]

Intuition

The TUNE branch is easy to skim past but it's where most SOCs quietly fail: every alert triaged as a false positive is a fork in the road. Take the lazy fork (silently dismiss) enough times and you get alert fatigue; take the disciplined fork (tune the rule, with an expiry so the exception doesn't outlive its reason) and the signal-to-noise ratio actually improves over time instead of decaying.

SIEM	EDR	NDR	XDR	SOAR
Correlates stored events	Endpoint behavior and response	Network behavior/metadata	Correlates endpoint+network+cloud+identity, analytics built in	Orchestrates defined workflows
Broad context, data-cost risk	Host depth, agent dependency	Useful where host visibility is weak	Correlation only as good as what's ingested; vendor-native model adds lock-in	Speeds repetition, amplifies bad logic

Measure MTTD/MTTA/MTTR with explicit start/end definitions, plus fidelity, investigation quality, and control effectiveness. Ticket closure alone rewards the wrong behavior and contributes to fatigue.

Hint

Before quoting an MTTR number, ask "detected-to-contained, or reported-to-closed?" Teams that optimize the metric instead of the outcome tend to gravitate toward whichever start/end pair makes the number look best, which is exactly the "ticket closure rewards the wrong behavior" trap the last sentence is warning about.

Learning objectives¶

Explain why a SOC exists and how work flows through tiers.
Use SIEM/EDR/NDR/XDR/SOAR as categories, not as shopping lists.
Name metrics and how they are abused.
Run a simple SOC workflow on soc-lite.

Key concepts¶

Full narrative and comparison tables: COURSE.md section 5.

Why a SOC. Prevention is incomplete; someone correlates identity + app + cloud; someone is accountable for detection SLAs; someone coordinates IR.

Tiers (typical, not mandatory).

Tier	Job	Timebox
L1	Triage: duplicate? obvious FP? enough to page?	minutes
L2	Investigate, enrich, contain with playbook	tens of minutes to hours
L3 / IR	Novel, severe, or failed playbooks	hours to days
Detection engineering	Rules as code, tuning, purple tests	sprint cadence
You (service owner)	Fix the system, confirm recovery	parallel

Flow. Alert → triage → enrich → investigate → escalate → contain → eradicate → recover → post-incident review.

Playbook (idea 12). A playbook is the numbered card for one alert type. It says when the card applies, what to do in order, when to stop and hand the case upward, and which evidence to keep. It is a procedure for a person. It is not a compliance document, and it is not a product that closes the alert by itself.

flowchart TB
  alert["alert DET-002"] --> when["applies when actor read someone else's note"]
  when --> steps["1. open the event  2. name owner and actor  3. preserve the log"]
  steps --> stop["stop and escalate if more than one note, or the actor is admin"]
  stop --> keep["keep the raw event; do not delete the container"]

The lab card for this alert is labs/soc-lite/playbooks/broken-access-control.md. The rule that points at it:

id: DET-002
event: cross_user_note_access
playbook: broken-access-control.md

SIEM, EDR, NDR, XDR, SOAR, TI, cases, VM, detection engineering — see COURSE tables. soc-lite is a toy SIEM + case system.

Alert fatigue. Too many low-fidelity alerts. Humans learn to click “close.” Fix: fewer, better detections; enrichment; suppression with expiry; staffing honesty.

False positives. A cost paid in analyst hours and missed true positives. Track them as defects in detection-as-code.

Burnout. Shift work + hostility + un-actionable queues. Metrics that only count closed tickets make this worse.

Metrics. MTTD, MTTA, MTTR, dwell time, fidelity, investigation quality, control effectiveness. Define MTTR as respond or recover explicitly.

Worked scene — triaging DET-003:alice¶

Asset and identity. notes-api; alice. The playbook is ssrf-metadata.md.
Fidelity. GET /alerts/DET-003:alice shows the fetch URL and ssrf_metadata_access evidence. That is the event itself, not a heuristic. It’s a true positive on a simulated attack.
Context. GET /events?q=alice shows a login_success shortly before. The session is valid. That doesn’t tell you who was using it.
Decision. Open a case, severity critical. Credentials of the workload role are treated as burned, even though they’re dummies.
Action. snapshot_logs with approval: APPROVE writes an audit row. The same call with "nope" gets 403.

What that implies. The fast part was deciding it was real. The slow, honest part is writing down what you don’t know: whether Alice was at her keyboard. The case timeline should say so.

Architecture connection¶

services --> logs --> detections --> alert queue --> case --> owners
                                      |               |
                                      +--> playbooks  +--> change ticket

The SOC does not own your service. You own the fix. The SOC owns the process to notice and coordinate.

Hands-on lab — mini SOC workflow¶

AUTHORIZED LAB USE ONLY.

Prerequisites¶

Dirty lab with alerts (run simulate + ingest if empty).

Before you run this¶

Write down three answers before you run anything:

Which alert will you pick, and what are its asset, identity, and severity?
What status will the alert have before and after you open a case?
What will /actions/simulate return with "approval":"nope", and with the field left out entirely?

Then run the steps. If a result surprises you, which assumption was wrong?

Steps¶

curl -s -X POST http://127.0.0.1:8090/ingest
List alerts. Pick the critical SSRF one if present, else IDOR.
Triage notes (write them down):
What asset? notes-api
What identity? alice
True positive vs lab-generated? both: it is a real TP on a simulated attack
Severity vs business: dummy payroll note → treat as high for practice
Enrich: GET /alerts/{id}, GET /events?q=alice, GET /playbooks/ssrf-metadata.md
Open a case linking alert ids.
Add a timeline note via POST /cases/{id}/update with a hypothesis.
Simulated containment (still no production change):

curl -s -X POST http://127.0.0.1:8090/actions/simulate -H 'Content-Type: application/json' \
  -d '{"action":"snapshot_logs","target":"notes-api","approval":"APPROVE","actor":"l2-analyst"}'

snapshot_logs writes a SOC audit row. It does not copy files — use preserve-logs.sh for a real snapshot (labs/evidence/).

Try "approval":"nope"; expect 403. Omitting the approval field entirely is 422 (validation), not 403. Unknown action on this endpoint is 400. Alert ids in path/body look like DET-003:alice (colon is fine). A case with the wrong alert_ids still opens; those alerts stay new. 8. Record MTTA-like time: wall clock from first alert created_at to case open. This is a toy measurement.

Expected observations¶

Alerts have statuses new then cased. Audit rows exist for actions. Approval gate refuses missing APPROVE.

Security lessons¶

Triage is a decision under uncertainty. Playbooks beat heroics. Approval on response actions is a SOC control, not bureaucracy.

Common mistakes¶

Closing as FP because “it’s the lab.”
Paging on every DET-005 regex hit without impact.
Measuring only ticket volume.

Keep for the capstone¶

Copy the incident-report template to docs/capstone/work/incident-report.md. Note the case ID, the alert IDs you cased, and the MTTA figure. Module 11 adds the timeline and RCA, and the finished report is the capstone’s Artifact E item.

Cleanup¶

Leave cases for module 11 or reset.

Knowledge check¶

SIEM vs EDR in one sentence each.
Why can MTTD go down while risk goes up?
What is L1 not supposed to do?
Name one SOAR failure mode.
Who fixes the IDOR, SOC or engineering?

Answers: (1) SIEM correlates logs; EDR watches endpoints/processes. (2) You detect only noisy easy alerts and miss slow data theft. (3) Novel containment without a playbook / destroying evidence. (4) Auto-close or auto-block on a bad IOC list. (5) Engineering owns the code; SOC coordinates the incident.

Engineering assignment¶

Draft an L1 playbook card (half page) for DET-002: when to escalate, what to never do, who owns the service.

Self-check¶

Answer before expanding. These are the assessment moves on this module's Acme Notes lab, not trivia.

Explain: Who owns fixing the IDOR — SOC or engineering?

Engineering owns the code. The SOC owns noticing, coordinating, and the case. Playbooks beat heroics; L1 is not supposed to invent containment that destroys evidence.

Predict: After you open a case from a new alert, what statuses should you see?

Alert moves from new to cased. Audit rows exist for the actions. Closing as FP "because it's the lab" skips the workflow you came to practice.

Diagnose: MTTD went down but risk went up. What happened?

You detect only noisy easy alerts and miss slow data theft — or you page on every DET-005 regex hit until humans click close. Metrics without fidelity are a failure mode.

Design: On an L1 card for DET-002, what must L1 never do?

Do not dump note bodies into the ticket, do not down -v before preserve, do not declare Alice guilty from one alert. Escalate when actor ≠ owner is confirmed and impact includes dummy payroll-class notes.

Defend: Why is approval=APPROVE a SOC control, not bureaucracy?

Response actions change the system. A SOAR that auto-blocks on a bad IOC list is a documented failure mode. Residual: a human can still approve a bad action; the gate is necessary, not sufficient.

Before you leave¶

Diagnose — triage from impact and fidelity, not from alert volume.
Build — ingest, open a case, record a toy MTTA, leave or reset per the lab.

How these are graded: assessment.