Evals are the management tool for AI systems: a scored test set turns 'the AI seems fine' into a number a leader can set a target against, exactly like an OKR. If a system matters to the business, it should have an eval, a target, and an owner who reports on it, the same discipline you apply to revenue or uptime.
Why managers should care about evals
Without an eval, an AI system is managed by anecdote: the loudest complaint or the best demo wins. With one, you can answer the three questions leadership actually has, is it good enough, is it getting better, and did last week's change make it worse.
Writing an AI OKR
- Objective: the business outcome ('support drafts customers accept').
- Key result: the eval metric with a target ('draft acceptance ≥ 85% on the golden set by Q3').
- Baseline first: measure before you set the target, not after.
- Counter-metric: pair quality with cost or latency so you don't optimize one at the other's expense.
- Owner: one name accountable for the number, with authority to change the system.
The review cadence
- Weekly: owners scan scores and exception queues.
- Monthly: eval scores appear in the operating review next to revenue and churn.
- On every change: no prompt, model or retrieval change ships without a before/after eval run.
- Quarterly: refresh the test set so it still reflects real traffic.