Day 4 · Lesson 5 — Agent Security & Evaluation

Evaluation: What to Measure

Security proved no harm. Evaluation proves it was worth shipping.

The second axis of trust. Evaluating a vibe-coding agent is its own problem, different from deterministic software and from a chatbot. Three things make it hard, and they shape the seven dimensions of quality you actually have to measure.

Recall first (spacing)

From Lesson 1: a secure agent can still fail because it —

Used too many tokens
Misread the developer's intent
Ran in a sandbox

Misread intent, inside the boundary but wrong. That's the gap evaluation exists to close, and intent satisfaction is its hardest dimension.

Why vibe-coding evaluation is different

Three constraints make it unlike any other evaluation problem:¹

The underspecification gap: there is no spec. "Make the dashboard load faster" is not a test case. The prompt is inherently underspecified; the agent fills the gap from latent knowledge. Job one of evaluation is checking whether it reconstructed the right unstated spec.
The user often can't validate the output. Non-technical users can't read 600 lines; experienced engineers can't in real time either. The gap between "the agent thinks it succeeded" and "it's actually correct" is wider here than anywhere.
The session is iterative; the codebase is state. Each turn edits real files; bad early decisions compound. You must evaluate the whole arc, not just one turn.

The seven dimensions

Quality splits into user-facing and internal dimensions, with safety cutting across all of them:¹

#	Dimension	The question
1	Intent satisfaction	Did it build what the user meant? (hardest; what they ultimately judge)
2	Functional correctness	Does it build, run, pass tests? (the floor — and gameable: tests can be deleted)
3	Visual / behavioural correctness	For UIs: does the rendered thing look and behave right?
4	Cost & efficiency	Tokens, latency, tool calls — and iteration count (1 turn vs 8 corrections)
5	Code quality & convention	Does it match the project's idioms? (passes tests but wrong style = a failure)
6	Trajectory quality	Sensible path? Read files first, right tool each step (bad reasoning = fragile success)
7	Self-repair behaviour	On failure, does it recover or compound the error?

Transversal: Safety & responsible AI (code vulnerabilities, refusal behaviour, content safety, IP exposure) is scored alongside each dimension, not as a separate gate.

They're not independent: better trajectory (6) tends to yield better functional correctness (2), which is a prerequisite for intent satisfaction (1). Reasoning quality flows up into outcome quality.

The governance layer, on evaluation

Governance overlay · Day 4 Lesson 5

Evaluation is the Validation pillar for non-deterministic value

Security's controls are V Validation of safety; the seven dimensions are V Validation of value, and dimension 2 alone is the naïve "it compiles" trap this whole day warns against. Note that functional correctness is gameable (delete the test, red goes green): that's why evaluation needs judges and trajectory inspection, not just a passing suite, the same reason Day 1 insisted on evals, not just tests. The transversal safety dimension is where evaluation and the 7 pillars fuse.

Ladder read: "we shipped because the demo worked" is L1; "we gate on measured intent satisfaction, trajectory, and safety" is the L3 evaluation discipline that makes autonomy defensible.

Through your three lenses

Individual (IC)

Stop treating "tests pass" as done; that's dimension 2, the floor. Ask whether it matched your intent (1), your conventions (5), and took a sane path (6). Those are where real quality lives.

Team

Track iteration count as a first-class metric: an agent that lands the diff in one turn is a different product from one needing eight corrections. Convergence is a team KPI.

Organisation

Require evaluation across the seven dimensions (with safety transversal) before an agent ships into a shared workflow, the eval equivalent of a security gate. "Set the bar at the eval, not the demo."

Check the reflex

Recall, don't re-read.

What makes vibe-coding evaluation fundamentally different?

The model is always deterministic
There is no spec to test against
Users read every line of code

The underspecification gap: an underspecified natural-language prompt, not a rigid spec. Users also often can't validate output, and the session is iterative on a living codebase.

The hardest — and ultimately decisive — evaluation dimension is —

Intent satisfaction
Functional correctness
Token cost

Intent satisfaction: did it build what the user meant? Unstated and shifting, it's what the user ultimately judges the agent on. Functional correctness is just the floor.

Why is "functional correctness" an insufficient measure of quality?

It's too expensive to run
It only works for UIs
It's the floor, and it's gameable

Tests can be deleted or mocked to turn red green without fixing anything. Passing tests is necessary, not sufficient, hence judges, trajectory inspection, and intent scoring.

Carry this into the week

For one agent, ask which of the seven dimensions you actually measure. Almost everyone measures dimension 2 (tests) and nothing else. Pick one more (intent satisfaction or iteration count) and start measuring it. That's the move from demo-trust to eval-trust.

Go deeper (primary source)

Vibe Coding Agent Security and Evaluation (Day 4), "Why evaluating vibe coding agents is different" & "What to Evaluate" pp.28–30.

Notes

Kartakis et al., Vibe Coding Agent Security and Evaluation (Day 4), "Evaluation" pp.28–30 (three constraints; the seven dimensions; transversal safety).

Up next → Day 4 · Lesson 6: Evaluation, How. The methods that measure each dimension (functional testing, LLM-as-judge, browser testing, trajectory inspection, benchmarks) and the applied tips that make it work at scale.