Aha! Agile Field audit

Aha! Agile · Capability audit · July 2026

Twelve projects.
One fleet.

A checklist did the rounds on X: build these twelve agentic AI projects and you're hired. Fair test. So we audited our production fleet against it, line by line. Nine shipped. Two partial. One honest no.

9Shipped
2Partial
1Not yet

First time here? The cast, in one minute.

The fleet

Seven AI agents run around the clock on one rented server. Each has a single job: one orchestrates a content pipeline, one drafts articles, one writes video scripts, one scouts research, one coaches a human through his health goals. None of them shares a login, a budget or a toolbox.

The gateway

Every AI call from every agent passes through one metered doorway. That doorway holds each agent's spending budget, its list of allowed models, and the ledger of what everything cost.

The pipeline

Ideas go in one end; published articles and videos come out the other. Between the two sit automated judges, quality gates and a human with the final say at every step that matters.

Why govern it

An unwatched agent fails silently, spends quietly and drifts from its instructions. Every mechanism on this page exists to make those failures loud, and cheap to undo.

01
Shipped

Structured output

The ask: enforce schemas on LLM output, validate, retry on parse errors, log failures.

The problem: a language model answers in prose, and software needs data. When an agent's answer feeds the next step of a pipeline it has to arrive in an exact shape, every time, and the system has to notice the moment it doesn't. Otherwise one garbled reply quietly corrupts everything downstream.

In the fleet

  • Draft-gate judges reply in JSON. Malformed replies are parsed, shape-checked, range-checked, and retried once before the judge is discarded.
  • The video judge runs a repair chain on almost-JSON (bad escapes, trailing commas) before giving up, then asserts exactly five score keys.
  • A contract gate on the article handoff raises on any missing or malformed field before the pipeline spends a cent.
  • Eval scorers check field, type and range on every scenario run; parse failures are logged with the reason.

Lives in

Draft-gate panelVideo engineFleet eval harness

Caveat: validation is hand-rolled rather than built on a standard schema library like Pydantic. The prose-level output contract is prompt-enforced, and only the stronger model tier honours it.

02
Shipped

RAG with citation grounding

The ask: retrieve context, cite sources, flag low confidence, fall back to search.

The problem: ask a model anything and it will answer, sourced or not. That is how confident nonsense gets published. The fix is retrieval with receipts: the agent looks things up in a real library first, cites what it found, and refuses to state what it cannot back.

In the fleet

  • Every wiki note carries source frontmatter. Drafts must cite inline, and a coverage gate hard-blocks publish prep on orphan citations.
  • The health coach grounds its advice in a cited knowledge vault and cites notes by path; vault entries carry an evidence-status taxonomy, including "emerging: handle with low confidence".
  • Claims with no source are never invented. They get flagged needs-source and handed to a research-scout agent whose whole job is to go and find one.

Lives in

Content pipelineHealth knowledge vaultResearch scout
03
Shipped

ReAct planning agent

The ask: an observe, think, act loop with iteration limits, self-critique and graceful degradation.

The problem: an agent works in a loop. It looks, thinks, acts, then looks again. Left uncapped, that loop can circle forever, burning money on a problem it cannot solve. A production agent needs a leash of a known length, and a plan for what happens when it reaches the end of it.

In the fleet

  • The agent loop is capped in config: 50 iterations, spawn depth 1, 600-second child timeouts, 3 concurrent children.
  • A failed video-script run retries once after five minutes, then parks: status failed, error posted to the task board, card moved to a failure state with a human assigned. A failure costs one retry, then it stops.
  • The draft gate stops auto-redrafting after three failed rounds and escalates to a human.
  • Drafting runs two cheap critic loops before anything reaches the judge panel.

Lives in

Agent framework configVideo pipelineDraft gate
04
Shipped

Multi-tool orchestrator

The ask: tool registry, capability-based routing, permission scoping, parallel execution.

The problem: one agent holding every tool and every permission is an accident waiting for a prompt. The safer shape is coordination: give each agent only the tools its role needs, block the dangerous ones outright, and let work fan out to helpers in parallel without the pieces treading on each other.

In the fleet

  • The content orchestrator routes each request to one of eight domain "spokes", and each spoke loads only its own tools and instructions.
  • Permissions are scoped per agent. The health coach has no terminal toolset at all: file access inside his own store, plus web for link capture, and nothing else.
  • Writes outside an allowlisted root are hard-denied in code. Cron sessions run deny-by-default, so dangerous commands are blocked with no approver present.
  • Delegation fans work out to parallel children with the risky tools (delegation, code execution, messaging) stripped from them.

Lives in

Content orchestratorHealth coachAgent gateway
05
Partial

Memory-enabled conversational agent

The ask: short-term buffer plus long-term vector recall, compression, cross-session sync.

The problem: a model forgets everything between conversations. A useful assistant cannot afford to. The engineering questions are what to write down, how to squeeze months of history into the small window a prompt allows, and how to carry decisions from one session into the next.

Done

  • The health coach keeps an append-only commitments ledger. It remembers what you promised last Tuesday, and asks about it.
  • Raw daily metrics are compressed into a trends rollup, so the coaching prompt gets a month of context in a few hundred tokens.
  • A wiki knowledge base serves as shared long-term memory across the fleet; session logs with wikilinks carry decisions between sessions.

Left

  • No vector store, no embedding-based relevance scoring. Memory is files and git, on purpose: at this corpus size grep beats embeddings and the audit trail comes free. Revisit when the corpus outgrows that.

Lives in

Health coachWiki knowledge baseSession-log vault
06
Shipped

Human-in-the-loop approval

The ask: detect uncertainty, pause, get human input, resume, keep a full audit trail.

The problem: some decisions should never be automated: publishing under your name, spending your money, touching your records. The pattern is simple to say and easy to fudge: the agent prepares, the human approves, and every step is written down so you can reconstruct who decided what, and when.

In the fleet

  • A written SOP defines gates as stops, not notifications. Angle confirmation, draft approval and publishing are human decisions, always.
  • The budget tool runs suggest, review, apply as separate phases. Nothing writes without per-row human approval, and every write lands in an append-only run log.
  • When the draft gate fails three times, the system stops guessing and hands the issue to a human with the full concern history attached.
  • Publishing surfaces (Substack, LinkedIn) hold no agent credential by design. A human is the gate.

Lives in

Content pipeline (SOP-03)YNAB budget toolingDraft-gate escalation
07
Partial

Cost-aware agent router

The ask: token budgets, routing by complexity and cost, cost-per-decision analytics.

The problem: agents spend real money on every call, and a stuck one can run up a bill overnight while you sleep. Cost control needs teeth: budgets that physically cut an agent off, cheap models for cheap work, and a ledger that can say what any given output actually cost to make.

Done

  • Every agent calls through a LiteLLM gateway on its own key, with a hard 30-day budget (breach returns HTTP 429) and a model allowlist (off-list returns 403).
  • Models are tiered by task: a cheap tier for chat and routing, a pro tier for anything that writes prose.
  • A costs console shows per-key spend for today, month-to-date and 30 days, budget headroom, and cost per produced asset. Telegram fires at 70% of any budget.

Left

  • Routing is assigned per task at design time. No runtime classifier picks a model per request, and there is no early exit on confidence.

Lives in

LiteLLM gatewayAgent Console (costs)Budget alert cron
08
Shipped

Event-triggered automation

The ask: webhooks and queues, idempotent execution, dead-letter handling, retry logic.

The problem: real automation is set off by events: a card moves on a board, a message arrives, a schedule ticks over. The hard parts are trusting that the trigger is genuine, surviving a rerun without doing the work twice, and giving failures a place to land where a human will actually see them.

In the fleet

  • Incoming webhooks carry a cryptographic signature that is verified before anything runs, so a forged event off the open internet is dropped at the door.
  • Sweeps can run twice without double-posting: already-processed items carry a marker and get skipped, so a rerun is harmless.
  • Failures land in a dedicated parking lane on the task board with a human assigned, instead of retrying forever.
  • Provider fallback chains catch upstream stalls, and the public webhook endpoint has its own uptime monitor.

Lives in

Task-board webhook handlerPost-publish promo sweepsVideo crons + uptime monitors
09
Shipped

Multi-agent debate system

The ask: agents propose, critics evaluate, voting and consensus, synthesis with confidence.

The problem: one model grading its own work is a rubber stamp. A panel of judges with different personalities, scoring blind and disagreeing on the record, catches what any single reviewer misses. The vote itself has to be arithmetic in code; leave it to the model and the judges will talk themselves into agreement.

In the fleet

  • Article drafts face a blind three-judge panel, each judge given a deliberately different persona (skeptical scroller, argument editor, brand-voice keeper) so they do not share one blind spot. The consensus rule is in code: unanimous pass, or it fails.
  • The video panel goes further: weighted per-criterion aggregation, a unanimous-veto floor, majority gates, and a contested verdict when judges diverge, instead of averaging the disagreement away.
  • Aggregation is deterministic and pytest-covered. The LLM never does the arithmetic.
  • An adversarial pair (logic auditor plus pre-mortem critic) red-teams slide decks between versions.

Lives in

Draft-gate panelAnimated-video panelVideo script judgingSlide-deck red team

Caveat: these are scoring panels over one artifact, not competing proposers debating alternatives. Same consensus machinery, narrower debate.

10
Shipped

Self-reflective agent with auto-eval

The ask: execute, evaluate via LLM-as-judge, critique, regenerate with constraints, log improvement.

The problem: first drafts are rarely shipped drafts, for agents as for people. The working loop: produce, get judged, revise against the specific objections, get judged again on a clean slate. And when you change the system itself, a regression suite should tell you what broke before production does.

In the fleet

  • Failed script verdicts feed only the named concerns back for a surgical revision, then get re-judged fresh. Verdicts are never reused across revisions.
  • Slide decks go through versioned audit, revise, re-audit cycles, with the review artifacts kept on disk.
  • A fleet eval harness gates every prompt or model change in CI, runs the full suite nightly, and pages Telegram on any red run.

Lives in

Video revision loopSlide-deck versioningFleet evals + eval alerts

Caveat: inside the eval harness itself, scoring stays deterministic; the LLM-as-judge hook there is a named, deliberately unbuilt gap. Judging lives in the pipelines.

11
Shipped

Production agent with observability

The ask: tracing, latency and cost dashboards, alerting, canary testing, rollback.

The problem: an agent that runs at 3am with nobody watching deserves the same discipline as any production service. That means a trace of every call, dashboards someone actually reads, alarms that reach a phone, a staging environment to break things in, and a practiced way back when a deploy goes wrong.

In the fleet

  • Every gateway call is traced to Arize Phoenix with token counts. The checklist names Arize; this is Arize Phoenix, self-hosted.
  • Eight uptime monitors page Telegram, alongside budget alerts, eval-failure alerts and a nightly registry-drift check.
  • Dashboards deploy to a staging environment before production, from separate pipelines.
  • Rollback has been rehearsed: pinned image digests per upgrade, backups before every container recreate, restore-drilled data backups, and sha256 stamps verifying deployed configs against the repo.

Lives in

Phoenix trace storeLiteLLM spend logsUptime KumaDashboard CI/CD
12
Not yet

Open source contribution

The ask: extend a framework, write docs and a demo, publish benchmarks, submit a PR.

The problem: everything above stands on open-source frameworks other people maintain. Using them is table stakes. Contributing back, whether patches, documentation or benchmarks, is the difference between using an ecosystem and belonging to it.

Closest artifacts

  • The agent gateway runs local overlay patches (webhook signature support, payload filters) that would make a reasonable upstream PR. The upstream ask was scoped, then consciously deferred in favour of shipping.
  • One tool ships as an installable plugin, but to a self-owned marketplace rather than a public registry.

To close it

  • Upstream the request-metadata patch to the Hermes agent framework, or publish the judge-panel pattern as a public plugin with docs and benchmarks.

Lives in

Nowhere yet.That is what an audit is for.

Method

Audited 21 July 2026 against the live fleet: eleven containers, seven agents, one LLM gateway, two dashboards, on one server. Verdicts cite shipped code, config and standard operating procedures, checked by SSH against the running system. Intentions did not count.

The checklist is Suraj Sharma's "12 Agentic AI projects" post on X, July 2026.

Agents appear here by role (content orchestrator, health coach, research scout) rather than by name.

This is version 1.1.0, rewritten in plain terms for readers new to agentic engineering. Earlier versions stay available: v1.0.0 (the original engineer-to-engineer cut, 21 July 2026).