Day 4 · Lesson 6 — Agent Security & Evaluation

Evaluation: How to Measure It

No single method covers seven dimensions. Combine them, and mine production.

Lesson 5 named what to measure. This is how: a toolbox of methods, each fit for certain dimensions, plus the applied tricks that make evaluation work when there's no spec: the session prefix becomes the rubric, and every user correction becomes a labelled failure.

Recall first (spacing)

From Lesson 5: the hardest evaluation dimension is —

Intent satisfaction
Functional correctness
Token cost

Intent satisfaction, unstated and shifting. This lesson's cleverest tip is how to measure it at scale despite there being no written spec.

The methods (fit method to dimension)

No one method covers everything, so production pipelines combine several:¹

Method	Best for
Automated functional testing (pytest, eslint…)	Correctness (2), rule-checkable style (5) — cheapest signal
Security & safety eval (Snyk, Semgrep, red-team)	Safety (transversal) — scored alongside, not as a gate
LLM-as-judge / Agent-as-judge	Intent (1), style (5), trajectory (6) — where rules can't capture "right"
Browser-based testing (Playwright, screenshots)	Visual/behavioural correctness (3)
Trajectory inspection (OTel traces)	Trajectory (6), self-repair (7) — the internal dimensions
Human review	Intent (1) ground truth, nuanced safety — doesn't scale; calibrates the rest
Online evaluation (sample prod traffic)	All dimensions at sample rate

Observability is the prerequisite: without OTel traces (session / think / tool spans) the internal dimensions are invisible. Use tail-based sampling: keep the traces with errors and excessive self-repair, drop routine successes.

Benchmarks (Vibe Code Bench, SWE-bench Verified, LiveCodeBench, Kaggle SAE) calibrate cognitive capability against the field, but beware benchmark overfitting: a top SWE-bench score proves an agent can navigate a structured repo, not that it has the aesthetic judgment to vibe-code a consumer app. Use them for calibration, never as a replacement for evaluating real intent.

Four applied tips (this is the useful part)

Use the session prefix as the intent rubric. No spec exists, so treat the user's first one or two messages as the spec: auto-derive acceptance criteria from them, then score every later turn against those. The only scalable way to measure intent satisfaction.¹
Judge the rendered artifact, not the code. The user judges the output; a multimodal model looking at the rendered page catches broken mobile layout, low contrast, wrong button states, things code-level eval misses. Pair with Playwright for interactivity.
Evaluate session convergence, not turn accuracy. The question is less "was turn 4 right?" and more "did the user converge on what they wanted?" Sessions abandoned mid-flow are the most informative failures.
Mine user corrections as labelled failure data. Every "no, not like that" is a free labelled failure. Cluster them and the agent's systematic gaps surface faster than any synthetic benchmark.

The governance layer, on how

Governance overlay · Day 4 Lesson 6

Trajectory inspection is Evidence; online eval is continuous Validation

Trajectory inspection over OTel traces is the same E Evidence substrate as the security Vibe Trajectory (Lesson 4). One glass box serves both axes. Online evaluation on sampled production traffic is V Validation that never stops, the eval twin of continuous security monitoring. And mining corrections is the feedback loop that turns Evidence into improvement, the quality flywheel from Day 1.

Ladder read: one-off pre-launch eval is L2 Observed; continuous online eval + correction-mining that feeds back into the harness is the L3→L4 self-governing loop applied to quality, not just safety.

Through your three lenses

Individual (IC)

Steal the four tips: derive a rubric from the opening turns, judge the rendered artifact, watch convergence, and read your own "no, not like that" corrections as data. Cheap, and they work now.

Team

Combine methods deliberately (functional tests + a judge + trajectory inspection) and instrument OTel so the internal dimensions are even visible. Tail-sample to stay in budget.

Organisation

Run online evaluation on sampled production traffic against the same rubrics as offline, biased toward high-cost and abandoned sessions. Use benchmarks for calibration only, and guard against overfitting to them.

Check the reflex

Recall, don't re-read.

With no written spec, the practical intent rubric is —

The final rendered page
The user's first one or two messages
The passing test suite

The session prefix. Auto-derive acceptance criteria from the opening turns, then score every later turn against them, the only scalable way to measure intent satisfaction.

To evaluate a UI-producing agent, you should judge —

The diff line by line
The token count only
The rendered artifact

The user judges the output, not the code. A multimodal judge on the rendered page catches layout/contrast/state issues code-level eval misses; pair with Playwright for interactivity.

Relying only on standardized benchmarks risks —

Overfitting to the benchmark
Under-reporting token cost
Breaking the sandbox

A top SWE-bench score proves repo navigation, not the judgment to vibe-code a real app. Use benchmarks for cognitive calibration, never as a substitute for evaluating real intent.

Carry this into the week

Do one thing: pull your last ten agent sessions and cluster the "no, not like that" corrections. The clusters are your prioritized failure modes, a free, labelled eval set you already own, more honest than any synthetic benchmark.

Go deeper (primary source)

Vibe Coding Agent Security and Evaluation (Day 4), "How to evaluate", "Observability: The Prerequisite for Evaluation", and "Applied tips" pp.31–38.

Notes

Kartakis et al., Vibe Coding Agent Security and Evaluation (Day 4), "How to evaluate" pp.31–38 (methods; benchmarks & overfitting; observability prerequisite; the four applied tips).

Up next → Day 4 · Synthesis. Reassemble the whole day (two axes → 7 pillars → vibe-loop → identity → SecOps → evaluation) from memory, and link the 7 pillars back to VERDICT. Recall-only.