An AI operating review is a monthly one-hour meeting where every production AI workflow is reviewed on four numbers: quality (eval score), cost per run, volume and incidents, each presented by its owner. It's the same ritual you already run for revenue and uptime, extended to the systems that now do real work.
The agenda (60 minutes)
- 1Scorecard scan (10 min): all workflows on one page, quality, cost per run, volume, incidents, versus last month.
- 2Exceptions (20 min): anything that regressed, spiked in cost, or had an incident, owner explains, group decides.
- 3Deep dive (15 min): one workflow examined properly per month, rotating.
- 4Pipeline (10 min): what's moving from pilot to production, and what pilot gets killed.
- 5Decisions (5 min): written down, with owners and dates. No decisions, no meeting next month.
The scorecard's four columns
- Quality: the eval score against its target, the OKR for the system.
- Cost per run: total spend divided by completed runs, watch the trend, not the absolute.
- Volume: runs per month, adoption is a health metric, unused automation is silent failure.
- Incidents: wrong outputs that reached a customer or a decision, with what changed since.
How the ritual dies (avoid these)
- Demo hour: showing new toys instead of reviewing running systems.
- No kill decisions: a review that never retires a workflow isn't reviewing.
- Metrics without owners: a dashboard nobody presents is a dashboard nobody fixes.
- Skipping quiet months, drift is quiet by definition.