Day 3 · Lesson 4 — Context Engineering: Sessions & Memory
Turning a noisy transcript into curated knowledge, and knowing when to forget.
A memory manager is more than a passive vector database. It's an active pipeline that decides what to remember, reconciles it with what it already knows, and prunes what's gone stale. That self-curation, rather than similarity search, is what makes memory intelligent.
From Lesson 3: what makes a memory manager more than a vector DB?
Active curation over time. This lesson is exactly that machinery, an LLM-driven ETL, not a dumb store.
Memory generation is Extract–Transform–Load with an LLM doing the thinking:1
INGEST raw source (usually the conversation) arrives
EXTRACT an LLM pulls ONLY info matching predefined TOPIC definitions
— nothing matches the topics? no memory is created
CONSOLIDATE compare new info to existing memories → MERGE / UPDATE /
CREATE / DELETE (the "self-editing" step)
STORE persist to a vector DB or knowledge graphThe gardener analogy: extraction is receiving new seeds; consolidation is pulling weeds (deleting redundant/conflicting data), pruning overgrowth (refining existing memories), and planting the new in the right spot. Without it, memory becomes an overgrown, unusable mess.
Extraction is targeted filtering of signal from noise, not summarisation. And "meaningful" is not universal: a support agent remembers order numbers; a wellness coach remembers goals and moods.1 You define it yourself: a JSON schema, a natural-language topic definition, or few-shot examples of ideal memories. No topic match, no memory. This is the single biggest lever on memory quality.
The hardest, most valuable stage. An LLM sees the new insight and the existing memories and decides what to do, resolving four problems conversations create:1
Forgetting is a feature, not a bug. A memory that only ever grows is a memory that rots.
"Garbage in, confident garbage out." So each memory tracks its provenance (origin and freshness), which weights it during consolidation and inference:1
Bootstrapped (CRM, internal) ── high trust (great for cold-start)
User input, explicit (a form) ── high trust
User input, implicit (inferred from chat) ── lower trust
Tool output ── brittle/stale → prefer short-term cache, not memoryWhen sources conflict, provenance sets the hierarchy: prefer the most trusted, or the most recent, or look for corroboration. And when a user revokes a data source, memories derived from it must go.
When to generate — session-end, turn cadence, real-time, or explicit command. A
cost/fidelity trade: real-time is fresh but expensive; session-end is cheap but coarse. Or
memory-as-a-tool: expose create_memory and let the agent decide.1
Blocking? Almost never: memory generation is expensive, so run it
asynchronously in the background after the reply is sent. (The "upload" step from Lesson 1.)
Tracking a memory's origin, freshness, and confidence is E Evidence applied to knowledge itself. It's how the system (and an auditor) can say why the agent believed something. And because a memory, once stored, is trusted implicitly at inference, the write path is a V Validation boundary: a malicious user can poison long-term memory via prompt injection, so information must be sanitised before it's committed (Lesson 5 goes deep). Revoking source-derived memories on request is C Compliance (data lineage / right-to-delete).
Ladder read: a memory store that records provenance and gates writes is auditable and defensible (L3); one that ingests anything a user says, unfiltered, is a self-corrupting L1 liability.
Invest in your topic definitions + few-shot examples: extraction quality dwarfs retrieval tricks. And treat tool output as cache, not memory: it goes stale fast.
Agree a consolidation + forgetting policy: how conflicts resolve, what TTLs apply. Run generation in the background as a shared service, not inline in each agent.
Require provenance tracking and pre-commit sanitisation on the memory store: a trust hierarchy for conflicts, and a right-to-delete that respects data lineage. Memory is regulated data.
Recall, don't re-read.
The most sophisticated stage of memory generation is —
Consolidation: the LLM reconciles new info with existing memories, resolving duplication, conflict, and evolution, and forgetting the stale. That self-editing is memory's core value.
Which source should generally NOT become long-term memory?
Tool output is brittle and stale, better as a short-term cache. Provenance sets trust: bootstrapped and explicit-user data rank high; inferred and tool-derived rank low.
Memory generation should usually run —
Background, after the reply is sent: it's expensive (LLM calls + DB writes) and would otherwise stall the user. This is Lesson 1's "upload" step.
Look at what one of your agents chooses to remember. Is "meaningful" actually defined (topics, examples), or is it remembering everything (noise) or nothing (amnesia)? Then ask: could a user poison it by just telling it something false? If yes, you need a pre-commit filter.
Context Engineering (Day 3), "Memory Generation: Extraction and Consolidation" pp.41–56; Agent Engine Memory Bank automates this pipeline.
Up next → Day 3 · Lesson 5: Retrieval, Inference & Risk — scoring which memories to surface, where to place them in context, procedural memory, evaluation, and the privacy/security risks (isolation, poisoning, cross-user exfiltration) that close the day.
Related: ← Day 3 L3: Memory vs RAG · Day 2 L5: poisoning & injection · Course home