Every engineering org has an incident response plan for the server going down: a page fires, someone acknowledges, a status page updates, a rollback happens, a postmortem gets written. Almost none of them have an equivalent plan for the failure mode that's actually more common in AI systems: nothing goes down, every health check stays green, and the system quietly produces wrong, biased, or harmful output at scale for days before anyone notices, usually because a customer complained rather than because a dashboard flagged it. Treating an AI incident like a server outage, wait for an alarm, is exactly why these failures run so long before detection. AI incidents need their own playbook, built around the fact that the failure is confident and plausible-looking rather than loud and obvious.
Why AI incidents are a different kind of problem
A traditional outage has a clear signature: something that was working stops working, an error rate spikes, a status code turns red, and the signal is unambiguous enough that automated alerting catches it in seconds. An AI incident rarely looks like that. The model keeps responding, latency stays normal, every traditional health check passes, and the actual failure is that the content of the response has quietly become wrong, biased, or unsafe, either suddenly (a bad prompt change went live, a vendor swapped a model version underneath a stable API name) or gradually (retrieval quality degraded as a document set drifted, a fine-tuned model's accuracy eroded as real traffic patterns shifted away from its training distribution). The absence of an error code is the whole problem: nothing in a traditional monitoring stack is built to notice that an answer is fluent, well-formatted, and wrong.
Detection: the triggers that actually catch this
Since traditional monitoring is structurally blind to AI-specific failures, detection has to come from a separate layer built specifically for it: scheduled evaluation runs against a ground-truth or rubric-scored sample of real traffic, drift monitoring on retrieval relevance and output characteristics, and, critically, a fast, low-friction path for humans, support agents, customers, or reviewers, to flag a bad output the moment they see one, since human-in-the-loop flagging catches failure patterns automated evals haven't been built to check for yet.
| Trigger type | What it catches | Typical cadence |
|---|---|---|
| Scheduled eval run against ground truth | Accuracy or hallucination-rate drift against a known-correct sample | Daily to weekly, tighter for high-stakes features |
| Retrieval relevance drift monitoring | A degrading knowledge base, a stale index, or a query pattern shift the KB doesn't cover | Continuous, alerted on threshold breach |
| Model/prompt version change logging | A silent vendor model update or an unreviewed prompt edit going live | Continuous, diffed against the last known-good version on every change |
| Human flagging pipeline | Novel failure patterns automated evals weren't built to catch yet | Real-time, low-friction (one click) for anyone who sees a bad output |
| Volume/pattern anomaly detection | A sudden shift in query types suggesting an upstream data or product change | Continuous, alerted on statistical deviation from baseline |
Severity classification built for AI, not copied from outages
A traditional severity matrix weighs user count affected and duration; that framing under-rates AI incidents that affect a small number of interactions but produce a genuinely harmful or reputationally damaging output, and over-rates ones that affect many low-stakes interactions with a minor, easily-tolerated quality dip. AI incident severity should weigh output visibility (did this reach a customer directly, or stay internal), reversibility (can the specific wrong outputs be identified and corrected after the fact, or did they already cause an irreversible action, a wrong refund, bad advice acted on), and harm category (factual error, biased or offensive output, safety-relevant misinformation), with any incident touching safety or bias treated as high severity regardless of how few interactions it affected.
- Critical: harmful, unsafe, or clearly biased output reached customers directly, or an AI-driven action caused an irreversible real-world consequence.
- High: confidently wrong output reached customers at meaningful volume, reversible but requiring active correction and likely customer communication.
- Medium: output quality degraded measurably against internal evaluation but hasn't yet been confirmed to reach customers at scale, or reached customers with low individual harm.
- Low: an internal-only quality regression caught by monitoring before any customer-facing impact, requiring a fix but no customer communication.
Rollback options: more levers than a code revert
Traditional incident response defaults to 'roll back the last deploy,' and AI systems have a broader, more specific set of rollback levers that need to be understood and rehearsed before an incident, not discovered during one. Model version pinning reverts to a known-good model version when a vendor's silent update caused the regression. Prompt reversion rolls back a specific prompt template change, which requires having versioned prompts in the first place. Retrieval source freezing reverts to a prior snapshot of the knowledge base or index when a data or document change caused degraded retrieval. Feature-flagged disablement turns off the AI feature entirely, falling back to a human or rules-based path, which is the blunt, always-available option when the specific cause isn't yet clear and the priority is stopping active harm rather than diagnosing.
- Model version pinning, reverting to a known-good model version; requires logging the exact version string on every request, not just the model name.
- Prompt template reversion, requires prompts to be versioned artifacts, not edited in place with no history.
- Retrieval index or knowledge-base snapshot rollback, requires the data pipeline to keep recoverable prior snapshots, not just the latest state.
- Feature-flagged full disablement with fallback to a human or rules-based path, the option to reach for when cause is unclear and stopping harm is the immediate priority.
Customer communication: the part most plans skip entirely
A traditional outage gets a status-page update because the failure is visible and undeniable; an AI incident often involves output that individual affected customers may not even realize was wrong, which makes the communication decision genuinely harder, not simpler. The default should not be silence. For incidents classified high or critical, affected customers, or a reasonable proxy set when exact identification isn't possible, should be proactively notified, with a clear, specific description of what was wrong (not a vague 'we had a technical issue') and what's being done about it, including manual review of decisions the AI made during the incident window if those decisions had real consequences. Silence discovered later, when a customer figures out on their own that they got a wrong answer and nobody told them, does more lasting damage to trust than a direct, specific acknowledgment does at the time.
A postmortem template specific to AI failures
A generic engineering postmortem template (timeline, impact, root cause, action items) is a reasonable starting skeleton, but the root-cause section needs AI-specific categories, because the fix and the prevention step differ completely depending on which of four buckets the failure actually falls into, and conflating them produces action items that don't address the real cause.
| Root cause category | What it looks like | Typical prevention step |
|---|---|---|
| Data drift | Real-world data or query patterns shifted away from what training or the knowledge base was built on | Scheduled retraining/re-indexing cadence, drift monitoring with alert thresholds |
| Model update (vendor-side) | A provider updated a model behind a stable API name, changing behavior without notice | Log and alert on exact model version string per request, not just the requested name |
| Prompt regression | A prompt template edit for one feature inadvertently changed behavior elsewhere | Versioned prompts, required eval-suite run before any prompt change ships |
| Edge case | A genuinely novel input type or scenario the system was never built or tested to handle | Add the case to the eval set permanently; treat every incident as a permanent eval-set contribution |
The single highest-leverage habit in AI incident postmortems is the last line of that table: every incident, regardless of category, should add at least one new case to the permanent evaluation set, so the same failure mode is caught automatically before it ships again. Teams that skip this step tend to relearn the same lesson from a repeat incident eighteen months later.
