Day 3 · Lesson 5 — Context Engineering: Sessions & Memory

Retrieval, Inference & Risk

Finding the right memory, placing it well, and not leaking it.

Generating memories is half the job. The other half: surfacing the right ones at the right moment, putting them where they'll help and not mislead, and doing it all without letting one user's memories bleed into another's. This lesson closes Day 3 on its sharpest edge: the risks.

Recall first (spacing)

From Lesson 4: what's the highest-leverage way to improve retrieved memories?

A faster vector database
Better memory generation upstream
A larger context window

Quality in, quality out, a clean corpus makes any retrieval better. Retrieval tricks can't rescue a noisy store. Now, given a good corpus, how do we surface the right piece?

Retrieval: score on three dimensions

Relying on semantic similarity alone is the common trap: it surfaces memories that are related but old or trivial. Blend three scores:¹

RELEVANCE   semantic similarity to the current conversation
RECENCY     how recently the memory was created
IMPORTANCE  how significant it is (often set at generation time)

For accuracy-critical cases, query rewriting and reranking help but add LLM-call latency, usually too slow for real time unless cached. And when to retrieve is its own choice: proactive (load every turn, always available, sometimes wasteful) vs reactive / memory-as-a-tool (the agent decides, efficient but an extra call).

Inference: placement changes behaviour

Where you put a retrieved memory shapes how the model uses it. Two options, each with a failure mode:¹

Placement	Good for	Failure mode
System instructions	Stable, global facts (user profile) — high authority	Over-influence: agent forces every topic back to the memory
Conversation history	Transient, episodic memories, in-the-moment	Dialogue injection: model thinks a memory was actually said

The practical answer is hybrid: profile in the system prompt, episodic memories injected per turn.

Procedural memory (the "knowing how" from Lesson 3) is a different beast: retrieving a playbook, not a fact. It's reasoning augmentation: dynamically injecting a proven strategy is a fast, online alternative to fine-tuning; the agent adapts without touching model weights.¹

Evaluation: is the memory actually helping?

Three layers, mapping straight onto Day 1's tests-vs-evals thinking:¹

Generation quality — precision / recall / F1 vs a golden set ("remembering the right things?").
Retrieval performance — Recall@K and latency ("finding them, fast enough?").
End-to-end task success — an LLM judge on downstream tasks ("does memory make the agent better?").

The governance layer — where Day 3 lands

Governance overlay · Day 3 Lesson 5

Memory is personal data with an attack surface

Three risks, three controls, the archivist's rules:¹

Cross-user access → strict isolation via ACLs; users can opt out and delete. (I Identity, C Compliance)
Memory poisoning — a user injects false "facts" to corrupt persistent memory → validate + sanitise before committing (Model Armor / filters). (V Validation)
Cross-user exfiltration — shared/procedural memory leaks one user's data to another → rigorous anonymisation of app-scoped memory. (C Compliance)

And PII redaction before persistence throughout. Ladder read: isolation + pre-commit validation + anonymised sharing is the L3 Controlled memory system; skip them and personalization becomes a breach vector. This is Day 2's tool-security thinking, now applied to the agent's knowledge.

Through your three lenses

Individual (IC)

Blend relevance + recency + importance; never rank on similarity alone. Choose placement on purpose: profile in the system prompt, episodic in the dialogue. Watch for over-influence.

Team

Evaluate memory like any system: golden set, Recall@K, task-success judge, latency budget on the hot path. Make "does memory help?" a measured question, not a vibe.

Organisation

Treat the three risks (cross-user access, poisoning, exfiltration) as a required controls checklist: ACL isolation, pre-commit sanitisation, anonymised app-scope. Memory is regulated personal data.

Check the reflex

Recall, don't re-read.

Ranking memories on semantic similarity alone tends to surface —

Only the freshest memories
Related but old or trivial ones
Perfectly relevant ones

Similarity ≠ usefulness. Blend relevance with recency and importance, or you'll keep pulling conceptually-close-but-stale memories.

A user injecting false "facts" to corrupt an agent's long-term memory is —

Memory poisoning
Dialogue injection
Over-influence

Memory poisoning, defended by validating/sanitising before committing to memory. (Dialogue injection = the model mistaking a memory for something said; over-influence = a system-prompt memory dominating reasoning.)

Placing a memory in the system prompt risks —

Dialogue injection
Faster forgetting
Over-influence

Over-influence: high-authority placement can make the agent relate every topic back to that memory. Conversation-history placement risks dialogue injection instead.

Carry this into the week

Run the archivist's three checks on one memory-enabled agent: can user A reach user B's memories? Can a user poison its memory by lying to it? Does any shared memory carry someone's personal data? Each "yes" is a control you owe before production.

Go deeper (primary source)

Context Engineering (Day 3), "Memory Retrieval" & "Inference with Memories" pp.56–64, and "Privacy and security risks" p.69; Model Armor for pre-commit sanitisation.

Notes

Milam, Gulli & Nawalgaria, Context Engineering: Sessions, Memory (Day 3), "Memory Retrieval", "Inference with Memories", "Procedural memories", "Testing and Evaluation", and "Privacy and security risks" pp.56–69.

Up next → Day 3 · Synthesis, reassemble the whole day (context loop → sessions → memory vs RAG → generation → retrieval/risk) from memory, and link it back to Days 1–2. Recall-only.