Module 05 — Context Engineering & Memory

Time: 5–7 days · Depends on: 01–04 · Next: Fine-tuning

Learning objectives

• Treat the context window as a scarce, ordered resource with hard token budgets
• Design a packing hierarchy (system → task → tools/RAG → memory → dumps)
• Implement memory tiers (working, session, user, world/RAG) instead of “stuff the transcript”
• Use SessionMemory / src.context_memory to budget and assemble messages in code
• Distinguish context engineering from prompt engineering and know when each fails

Why this matters (CS engineer view)

Day 19 of a “simple” support chat: the system policy still says never invent account IDs. The window is 90% old tool JSON and small talk. The model invents an account ID anyway—not because the policy vanished from the product, but because it drowned under clutter on the desk. Cost per turn climbed; nobody owned the packer.

The running app enters Gate 3 here: it's tested and evaluated, but still ignorant of anything outside training data — this module and 07/09 are what ground it.

You already budget CPU, memory, and bandwidth. The LLM context window is the same class of resource: finite, ordered, and expensive.

In production, models rarely fail only because the instruction was poorly worded. They fail because safety policy is buried under a 40-turn log, tool dumps crowd out the user question, stale RAG chunks linger, or “memory” is an unbounded array with no summary path.

Prompt engineering shapes how the model is instructed.
Context engineering decides what enters the window, in what order, at what fidelity, under what budget.

Ship multi-turn chat or agents without a packing policy and you get ignored instructions, rising cost per turn, and silent constraint loss.

Mental model

Think of each model call as filling a fixed-size buffer. Priority order is not chronological order — it is product order.

flowchart TB
  subgraph budget["Context window (token budget)"]
    direction TB
    S["1. System policy & non-negotiables"]
    T["2. Task instructions for this turn"]
    R["3. High-signal tools / RAG facts"]
    M["4. Compact session memory"]
    H["5. Low-signal history / raw dumps"]
  end
  Headroom["Reserved completion headroom\n(10–20% of window)"]
  budget --> Headroom
  Drop["Over budget? Drop from bottom first.\nNever drop safety policy."]
  H -.-> Drop

Layer	Analogy	Drop priority
System policy	Kernel / capabilities	Last (never)
Task instructions	Current syscall args	Keep for this turn
Tools / RAG	Hot cache of facts	Cap size; refresh
Session memory	Working set summary	Compress
Raw history / dumps	Cold storage spill	First to drop

Intuition lock

Sticky picture: The context window is a desk surface—working memory with hard edges, not an infinite backpack. You cannot leave every sticky note, PDF, and tool printout on it and still find the safety card. Packing is a priority queue under a token budget: pin policy and this turn’s task first, cap RAG/tools, compress session memory, spill raw dumps first. Headroom is empty space reserved for the answer so the model isn’t forced to whisper a truncated reply.

Kill this idea: “Bigger window means I can paste everything and the model will figure it out.” → Replace with: Order and fidelity under budget are product policy; more room without a packer just delays a more expensive mess.

Core tutorial

1. Prompt engineering vs context engineering

Dimension	Prompt engineering	Context engineering
Question	How do we instruct?	What is in the window?
Artifacts	System text, few-shots, formats	Packer, memory store, retriever, budgeter
Failure mode	Vague task, bad format	Wrong/stale/noisy facts, drowned policy
Iteration speed	Edit strings	Change ranking, caps, summary policy
Measurable	Style, schema pass rate	Tokens/turn, recall of constraints, cost

You need both. A perfect prompt with a garbage window still fails.

Explainer

Ordered resource, not a bag. Most chat APIs send a list of messages. Models attend over the whole sequence, but position and volume still matter: long tool results can dominate, and late contradictory instructions confuse both the model and your evaluators. Treat order as an API contract your packer owns — not as “whatever messages.append produced.”

2. Information hierarchy (recommended packing order)

When assembling a request, prefer this order (high → low priority):

1. System policy & non-negotiables
2. Task instructions for this turn
3. High-signal retrieved facts / tool results
4. Compact conversation memory (summary + last k)
5. Low-signal history / raw dumps (first to drop)

Over budget rule: drop from the bottom. Never drop safety policy. Truncate or summarize dumps; do not silently omit the user question.

3. Token budgeting in code

This course ships a dependency-free estimator and priority packer in src.context_memory. Production systems often swap in tiktoken or provider-native counters — the policy stays the same.

from src.context_memory import SessionMemory, estimate_tokens, fit_budget

parts = [
    ("system", "You are a careful assistant. Never invent account IDs."),
    ("summary", "User prefers concise answers. Dark mode preference."),
    ("history", "..." * 200),  # low priority: listed last → dropped first if over budget
]
kept = fit_budget(parts, budget=50)
assert kept[0][0] == "system"
print(estimate_tokens("hello world"))

How fit_budget works: walk the list in order, keep each part while cumulative estimate_tokens ≤ budget, then stop. So list order is priority.

That is stricter than the diagram’s “drop from the bottom.” The diagram is the product rule (dumps go first). The teaching helper implements a simple greedy keep-from-the-front: if a middle part is huge, everything after it is dropped, even if a truncated version of that middle part plus history would have fit. Production packers hard-cap oversized parts (truncate tool JSON to 2k tokens) and then continue. The stretch lab asks you to do that.

# Sketch of the shipped logic (see src/context_memory.py)
def estimate_tokens(text: str) -> int:
    """Rough ~4 chars/token without external deps.

    English prose is often ~4 chars/token. Code, URLs, and CJK are denser
    (more tokens per character). Use tiktoken or the provider usage field
    in production — this helper is for packing *policy*, not invoices.
    """
    if not text:
        return 0
    return max(1, (len(text) + 3) // 4)

def fit_budget(parts: list[tuple[str, str]], budget: int) -> list[tuple[str, str]]:
    kept, used = [], 0
    for label, text in parts:
        n = estimate_tokens(text)
        if used + n <= budget:
            kept.append((label, text))
            used += n
        else:
            break
    return kept

Headroom: leave 10–20% of the total window for the completion. If you fill the window with input, you force short or truncated answers.

Think about it

Question: Your window is 128k. A support agent ships with 2k system policy, 500 tokens of task, 80k of past tickets “just in case,” and a 200-token user question. Latency feels like molasses; the bot still invents a policy clause. What fails first—accuracy, cost, or both—and what is the minimal packing fix before finance and trust both melt?

Reveal a strong answer

Both fail. Cost scales with input tokens every turn; accuracy fails as attention and instruction priority drown under low-signal dumps—the desk is full of junk mail. Minimal fix: cap tickets (top-k by embedding + recency), rolling session summary, pin system policy first, reserve completion headroom. “Paste the CRM export” is not a product strategy.

4. Memory tiers

Tier	Contents	Lifetime	Storage
Working	Current turn + live tool results	Turn	In request only
Session	Rolling summary + last k turns	Session	Server/session store
User profile	Preferences, stable facts	Long-lived	Explicit write (DB)
World / RAG	Docs, tickets, code, policies	External	Vector/DB/search

Rules of thumb:

• Working is rebuilt every call; do not persist raw tool dumps forever.
• Session must compress — unbounded chat history is a cost and quality bug.
• User profile is written only when the product intends it (settings, confirmed facts) — not every model guess.
• World / RAG is the database of truth for private knowledge; the window only holds retrieved slices.

Explainer

Librarian, not a hoarder. Session summary is the index card: goals, decisions, constraints. Profile is the card catalog entry you deliberately file. RAG is the closed stacks—you fetch a few volumes per turn, you do not wheel the whole library onto the desk. If every guess the model makes becomes “memory,” you will re-feed hallucinations as if they were user preferences.

5. SessionMemory (course package)

from src.context_memory import SessionMemory

mem = SessionMemory(summary="User prefers dark mode; account # never invent.", max_recent=10)
mem.add("user", "Can you summarize our last decision?")
mem.add("assistant", "We agreed to ship dark mode first.")

messages = mem.build_messages(
    system="You are a product assistant. Follow policy.",
    user="Remind me what we decided.",
)
# messages: [system, summary-as-system, ...recent, user]

Behavior worth internalizing:

• add validates roles (system | user | assistant | tool) and caps recent to max_recent.
• build_messages always puts system first, then optional summary, then recent turns, then the new user message.
• should_summarize(max_messages=20) is a simple trigger for a summarization job.
• transcript() is for feeding a summarizer, not for stuffing into every call.

6. Rolling summary pattern

When len(recent) grows, compress older turns into summary and keep only the tail.

def should_summarize(history: list[dict], max_messages: int = 20) -> bool:
    return len(history) > max_messages

SUMMARY_PROMPT = """Summarize the conversation for future turns.
Keep: user goals, decisions, constraints, open questions, names/IDs.
Drop: chit-chat, duplicate clarifications.
Max 200 words.

Transcript:
{transcript}
"""

Hard requirement: treat summary as state you own. Write it to your store. Do not rely on the model “remembering” across sessions without an explicit memory write.

After summarization, a regression-style check (Module 04 mindset): a constraint from turn 1 must still appear in summary after turn 15 (you can unit-test the summary string or a structured memory record).

7. Context packing for RAG (preview of 07/09)

Even before full RAG systems:

• Chunk for retrieval (semantic + structure-aware)
• Rerank top-k; dedupe near-identical chunks
• Cap total retrieved tokens so tools + RAG cannot starve the task
• Cite sources with stable IDs the UI can open

Pack order still applies: policy and task before retrieved text. Retrieved text is data, not instructions (Module 02).

8. Putting it together: a packer sketch

from src.context_memory import SessionMemory, estimate_tokens, fit_budget

def pack_turn(
    *,
    system: str,
    task: str,
    tool_or_rag: str,
    mem: SessionMemory,
    user: str,
    input_budget: int = 6000,
) -> list[dict]:
    """Assemble messages under a hard input budget."""
    # Priority list for text budget (order matters)
    parts = [
        ("system", system),
        ("task", task),
        ("tools_rag", tool_or_rag),
        ("summary", mem.summary or ""),
        ("history", mem.transcript()),
    ]
    kept = {label: text for label, text in fit_budget(
        [(l, t) for l, t in parts if t], budget=input_budget
    )}
    # Rebuild message list from what survived
    msgs = [{"role": "system", "content": kept.get("system", system)}]
    if "task" in kept:
        msgs.append({"role": "system", "content": f"Task:\n{kept['task']}"})
    if "tools_rag" in kept:
        msgs.append({"role": "system", "content": f"Context:\n{kept['tools_rag']}"})
    if "summary" in kept and kept["summary"]:
        msgs.append({"role": "system", "content": f"Conversation summary:\n{kept['summary']}"})
    # Prefer structured recent from SessionMemory if budget allows history
    if "history" in kept:
        msgs.extend(mem.recent)
    msgs.append({"role": "user", "content": user})
    # Optional: assert estimate_tokens of full payload + headroom
    return msgs

In production you will measure exact tokens and may truncate within a part (e.g. tool JSON) rather than dropping the whole part. The priority list is still the design center.

Quiz · +25 XP

You are over the token budget. What should you drop first?

System safety policy The current user question Low-signal history / raw dumps Task instructions for this turn

Common failure modes

Symptom	Likely cause	Fix
Ignores instructions mid-chat	History / tools outrank system	Re-pin system; cap history; reorder packer
Contradicts earlier decision	Summary lost a constraint	Structured memory fields; summary QA
High cost, mediocre quality	Dumping full PDFs / CRM every turn	Retrieve top-k; budget retrieved tokens
Hallucinated IDs	No structured memory write	Explicit profile store; “never invent IDs” + tools
Answers cut off	No completion headroom	Reserve 10–20% of window
“Forgot” after refresh	Memory only in client RAM	Persist session + user tiers server-side

Lab

Lab · Session memory under budget

Goal: Prove that constraints survive compression and that packing respects priority.

1. Run the course tests:

   poetry run pytest tests/test_context_memory.py -v
   

2. Extend or script a demo that:
3. Creates SessionMemory(max_recent=6)
4. Adds a constraint in turn 1: e.g. “Never use my real name; call me Rivet.”
5. Adds 12+ filler turns
6. When should_summarize() is true, call a model (or a stub) with SUMMARY_PROMPT and set mem.summary
7. Assert (test or manual):
8. len(mem.recent) <= max_recent
9. Summary still contains the “Rivet” constraint (string contains or structured field)
10. fit_budget with a tiny budget keeps system and drops long history
11. Measure tokens/turn before vs after summarization with estimate_tokens on build_messages(...) payloads.

Stretch: Implement partial truncation of the lowest-priority part instead of dropping it entirely when it barely overflows.

Knowledge check

Quiz · +25 XP

Which statement best separates the two disciplines?

Context engineering only means writing longer system prompts Prompt engineering shapes instructions; context engineering decides what enters the window, in what order, at what fidelity Context engineering replaces the need for evaluation suites Prompt engineering is only for fine-tuned models

Quiz · +25 XP

Where should a stable preference like “always use metric units” live for a multi-session product?

Only in the last 10 chat messages forever User-profile tier with an explicit write path, optionally mirrored into session summary Only in model weights after fine-tuning Inside every RAG chunk of the company wiki

Think about it

Question: fit_budget stops at the first part that does not fit. Why might that be too blunt for tool results, and what policy would you add?

Reveal a strong answer

A single huge tool result can block all lower-priority parts even if you could keep a truncated tool payload + history. Better policy: for designated parts, **hard-cap** (e.g. 2k tokens of tool JSON), then continue packing. Priority remains; fidelity degrades gracefully instead of all-or-nothing drops.

Open source materials

Curated reading (concepts — verify current URLs and versions):

1. LangChain — Context engineering for agents — search “context engineering”; packing, write/select/compress patterns for agent windows
2. HumanLayer — 12-Factor Agents — practical factors for agent context, tool ownership, and not treating the transcript as a database
3. OpenAI / Anthropic / Gemini docs — context windows & token counting — provider-native budgets and headroom (pick your stack)
4. tiktoken — production-grade token counting for OpenAI-compatible models
5. Course code: src/context_memory.py and tests/test_context_memory.py (repo root; not part of the docs site)

Checkpoint

• You can draw your app’s memory tiers (working / session / profile / world)
• You enforce a token budget in code (priority order + headroom)
• History is not your only “database”
• You can explain context engineering vs prompt engineering in one sentence each

When the above checklist is true for your project (or a lab demo), mark this module complete.

Complete module · +120 XP

Exercise

• Catalog: EX-05 — Memory budget
• Prove: Session memory stays bounded; fit_budget drops low-priority history instead of silent truncation.
• Test: pytest tests/test_context_memory.py -v

Next: Module 06 — Fine-tuning · or jump to Tools & basic RAG on a faster path (you will still need context packing).