Archived version 1.0.0 · current version
Aha! Agile Field audit

Aha! Agile · Capability audit · July 2026

Twelve projects.
One fleet.

A checklist did the rounds on X: build these twelve agentic AI projects and you're hired. Fair test. So we audited our production fleet against it, line by line. Nine shipped. Two partial. One honest no.

9Shipped
2Partial
1Not yet
01
Shipped

Structured output

The ask: enforce schemas on LLM output, validate, retry on parse errors, log failures.

In the fleet

  • Draft-gate judges reply in JSON. Malformed replies are parsed, shape-checked, range-checked, and retried once before the judge is discarded.
  • The video judge runs a repair chain on almost-JSON (bad escapes, trailing commas) before giving up, then asserts exactly five score keys.
  • A contract gate on the article handoff raises on any missing or malformed field before the pipeline spends a cent.
  • Eval scorers check field, type and range on every scenario run; parse failures are logged with the reason.

Lives in

Draft-gate panelTasveer video engineFleet eval harness

Caveat: validation is hand-rolled, not Pydantic. The prose-level output contract is prompt-enforced, and only the pro model tier honours it.

02
Shipped

RAG with citation grounding

The ask: retrieve context, cite sources, flag low confidence, fall back to search.

In the fleet

  • Every wiki note carries source frontmatter. Drafts must cite inline, and a coverage gate hard-blocks publish prep on orphan citations.
  • The health coach grounds its advice in a cited knowledge vault and cites notes by path; vault entries carry an evidence-status taxonomy, including "emerging: handle with low confidence".
  • Claims with no source are never invented. They get flagged needs-source and routed to Khoji, the research scout, to go and find one.

Lives in

Content pipelineHealth knowledge vaultKhoji (scout)
03
Shipped

ReAct planning agent

The ask: an observe, think, act loop with iteration limits, self-critique and graceful degradation.

In the fleet

  • The agent loop is capped in config: 50 iterations, spawn depth 1, 600-second child timeouts, 3 concurrent children.
  • A failed video-script run retries once after five minutes, then parks: status failed, error posted to Linear, card moved to a failure state with a human assigned. A failure costs one retry, then it stops.
  • The draft gate stops auto-redrafting after three failed rounds and escalates to a human.
  • Drafting runs two cheap critic loops before anything reaches the judge panel.

Lives in

Hermes fleet configVideo pipelineDraft gate
04
Shipped

Multi-tool orchestrator

The ask: tool registry, capability-based routing, permission scoping, parallel execution.

In the fleet

  • The orchestrator routes by domain through eight spoke contexts, each loading only its own tools and instructions.
  • Permissions are scoped per agent. The health coach has no terminal toolset at all: file access inside his own store, plus web for link capture, and nothing else.
  • Writes outside an allowlisted root are hard-denied in code. Cron sessions run deny-by-default, so dangerous commands are blocked with no approver present.
  • Delegation fans work out to parallel children with the risky tools (delegation, code execution, messaging) stripped from them.

Lives in

Mustansar (orchestrator)Hakeem (health coach)Hermes gateway
05
Partial

Memory-enabled conversational agent

The ask: short-term buffer plus long-term vector recall, compression, cross-session sync.

Done

  • The health coach keeps an append-only commitments ledger. It remembers what you promised last Tuesday, and asks about it.
  • Raw daily metrics are compressed into a trends rollup, so the coaching prompt gets a month of context in a few hundred tokens.
  • A wiki knowledge base serves as shared long-term memory across the fleet; session logs with wikilinks carry decisions between sessions.

Left

  • No vector store, no embedding-based relevance scoring. Memory is files and git, on purpose: at this corpus size grep beats embeddings and the audit trail comes free. Revisit when the corpus outgrows that.

Lives in

Hakeem (health coach)aa2brain wiki KBmyPKA session logs
06
Shipped

Human-in-the-loop approval

The ask: detect uncertainty, pause, get human input, resume, keep a full audit trail.

In the fleet

  • A written SOP defines gates as stops, not notifications. Angle confirmation, draft approval and publishing are human decisions, always.
  • The budget tool runs suggest, review, apply as separate phases. Nothing writes without per-row human approval, and every write lands in an append-only run log.
  • When the draft gate fails three times, the system stops guessing and hands the issue to a human with the full concern history attached.
  • Publishing surfaces (Substack, LinkedIn) hold no agent credential by design. A human is the gate.

Lives in

Content pipeline (SOP-03)YNAB budget toolingDraft-gate escalation
07
Partial

Cost-aware agent router

The ask: token budgets, routing by complexity and cost, cost-per-decision analytics.

Done

  • Every agent calls through a LiteLLM gateway on its own key, with a hard 30-day budget (breach returns HTTP 429) and a model allowlist (off-list returns 403).
  • Models are tiered by task: a cheap tier for chat and routing, a pro tier for anything that writes prose.
  • A costs console shows per-key spend for today, month-to-date and 30 days, budget headroom, and cost per produced asset. Telegram fires at 70% of any budget.

Left

  • Routing is assigned per task at design time. No runtime classifier picks a model per request, and there is no early exit on confidence.

Lives in

LiteLLM gatewayAgent Console (costs)Budget alert cron
08
Shipped

Event-triggered automation

The ask: webhooks and queues, idempotent execution, dead-letter handling, retry logic.

In the fleet

  • Linear webhooks are verified with HMAC signatures (constant-time compare) before anything runs.
  • Sweep jobs are idempotent. Already-processed issues are detected by marker comments and skipped, which makes every sweep safe to rerun.
  • Failures dead-letter into a dedicated Linear state with a human assignee, instead of retrying forever.
  • Provider fallback chains catch upstream stalls, and the public webhook endpoint has its own uptime monitor.

Lives in

Linear webhook handlerPost-publish promo sweepsVideo crons + Uptime Kuma
09
Shipped

Multi-agent debate system

The ask: agents propose, critics evaluate, voting and consensus, synthesis with confidence.

In the fleet

  • Article drafts face a blind three-judge panel with decorrelated personas: skeptical scroller, argument editor, brand-voice keeper. The consensus rule is in code: unanimous pass, or it fails.
  • The video panel goes further: weighted per-criterion aggregation, a unanimous-veto floor, majority gates, and a contested verdict when judges diverge, instead of averaging the disagreement away.
  • Aggregation is deterministic and pytest-covered. The LLM never does the arithmetic.
  • An adversarial pair (logic auditor plus pre-mortem critic) red-teams slide decks between versions.

Lives in

Draft-gate panelAnimated-video panelTasveer script judgingSlide-deck red team

Caveat: these are scoring panels over one artifact, not competing proposers debating alternatives. Same consensus machinery, narrower debate.

10
Shipped

Self-reflective agent with auto-eval

The ask: execute, evaluate via LLM-as-judge, critique, regenerate with constraints, log improvement.

In the fleet

  • Failed script verdicts feed only the named concerns back for a surgical revision, then get re-judged fresh. Verdicts are never reused across revisions.
  • Slide decks go through versioned audit, revise, re-audit cycles, with the review artifacts kept on disk.
  • A fleet eval harness gates every prompt or model change in CI, runs the full suite nightly, and pages Telegram on any red run.

Lives in

Tasveer revision loopSlide-deck versioningFleet evals + eval-alert

Caveat: inside the eval harness itself, scoring stays deterministic; the LLM-as-judge hook there is a named, deliberately unbuilt gap. Judging lives in the pipelines.

11
Shipped

Production agent with observability

The ask: tracing, latency and cost dashboards, alerting, canary testing, rollback.

In the fleet

  • Every gateway call is traced to Arize Phoenix with token counts. The checklist names Arize; this is Arize Phoenix, self-hosted.
  • Eight uptime monitors page Telegram, alongside budget alerts, eval-failure alerts and a nightly registry-drift check.
  • Dashboards deploy to a staging environment before production, from separate pipelines.
  • Rollback has been rehearsed: pinned image digests per upgrade, backups before every container recreate, restore-drilled data backups, and sha256 stamps verifying deployed configs against the repo.

Lives in

Phoenix trace storeLiteLLM spend logsUptime KumaDashboard CI/CD
12
Not yet

Open source contribution

The ask: extend a framework, write docs and a demo, publish benchmarks, submit a PR.

Closest artifacts

  • The agent gateway runs local overlay patches (webhook signature support, payload filters) that would make a reasonable upstream PR. The upstream ask was scoped, then consciously deferred in favour of shipping.
  • One tool ships as an installable plugin, but to a self-owned marketplace rather than a public registry.

To close it

  • Upstream the request-metadata patch to the Hermes agent framework, or publish the judge-panel pattern as a public plugin with docs and benchmarks.

Lives in

Nowhere yet.That is what an audit is for.

Method

Audited 21 July 2026 against the live fleet: eleven containers, seven agents, one LLM gateway, two dashboards, on one server. Verdicts cite shipped code, config and standard operating procedures, checked by SSH against the running system. Intentions did not count.

The checklist is Suraj Sharma's "12 Agentic AI projects" post on X, July 2026.