The cost of no governance · real-world failure · United States
US$340,000
billed in US dollars to a US health-tech team for a single weekend.
They shipped an update on a Friday evening. Over the weekend, one AI agent got stuck retrying malformed patient records for nearly 62 hours straight. There was no token ceiling, so the cost compounded silently until a US$340K invoice landed on Monday — by then the tokens were spent. A post-mortem traced the damage to four governance gaps. The eight-week rebuild brought monthly AI spend from US$412K down to US$71K.
Their post-mortem gap → our day-one control
1
No token budgets — a single faulty agent could spend endlessly
Every project carries a
budget_usd_month ceiling and tier-aware model
routing (ADR-014). A runaway agent hits a cap measured in dollars, not discovered in an invoice.2
No audit trail — compliance couldn't trace which AI outputs touched patient data
Every AI call and output is appended to
cc_audit_log with a SHA-256 hash
chain. "Which agent touched this record, with which model, when?" is a query, not an investigation.3
No human-in-the-loop — AI-generated patient messages were sent automatically
The CSEO veto gate plus the structural
ask-before-execute guardrails mean
sensitive or production actions require explicit sign-off. Nothing high-stakes goes out unattended.4
No evaluation pipeline — output quality had degraded 23% over four months, unnoticed
A dedicated QA Lead agent plus the evaluation monitor surface silent drift as a signal,
not as a quarterly surprise.
