The classic requirements document has a hidden assumption baked into every line: the system will do exactly what the spec says, every time. 'When the user clicks submit, the order is created.' For deterministic software that assumption holds, which is why 'the system shall' phrasing has survived decades. AI systems break the assumption at the root. A model doesn't do the right thing every time — it does the right thing some measurable fraction of the time, with a distribution of failure modes, and the interesting requirements are about that fraction, those failures, and what happens next. Teams that gather AI requirements with the classic toolkit end up with a spec that is simultaneously complete and useless: every line true of the demo, none of it enforceable in production. Here are the rules that change, and the interview technique that gets you requirements you can actually hold a system to.
Why fixed-spec thinking fails for probabilistic systems
Write 'the system shall correctly categorize incoming support tickets' into a requirements doc and you've created a requirement that is unfalsifiable and unmeetable at once. Unmeetable because no model categorizes correctly always; unfalsifiable because without a test set and a threshold, any accuracy level can be argued to satisfy it. This is the root failure, and it cascades: acceptance testing becomes a vibes exercise ('try a few and see'), sign-off becomes political (whoever tried the worst examples most recently wins), and production incidents become contract disputes ('the spec said it would work'). The deterministic parts of an AI project — the API, the UI, the pipeline plumbing — can and should keep classic requirements. It's specifically the model-mediated behavior that needs a different grammar: not 'shall do X' but 'shall do X at a measured rate on a defined population, and shall do Y — a specific, deterministic Y — on the cases where it doesn't.'
Behavioral acceptance criteria: the three-part spec
Every AI-mediated capability in the requirements doc should carry three clauses. If any is missing, the requirement isn't done. The pattern is worth internalizing because it applies from a classification model to a full agentic workflow.
| Clause | What it specifies | Example wording |
|---|---|---|
| Accuracy floor | Minimum measured performance on a named, frozen test set — plus the counter-metric that must not degrade | ≥92% top-1 accuracy on the 400-case acceptance set; false-escalation rate below 5% |
| Uncertainty behavior | The deterministic action when the model is unsure or the input is out of distribution: escalate, abstain, flag, or fall back | Below 0.7 confidence, route to the human queue with the top-3 candidates shown; never guess silently on invoices over €10k |
| Latency & cost budget | The p95 response time and per-unit cost ceiling at expected volume | p95 under 2s at 50 requests/min; inference cost under €0.004 per ticket at projected volume |
Acceptance ranges, not acceptance points
A single acceptance number is a dispute waiting to happen: the system lands at 91.4% against a 92% bar and the launch meeting becomes a negotiation about the test set. The more robust pattern is a range with three zones agreed in writing before anyone sees results. Above the ship line, it ships without a meeting. Below the kill line, it stops without a meeting — the team goes back to data or scope, not to lobbying. Between the two lines, a named group decides, and the requirements doc says who. This does two things: it removes the incentive to argue about the measurement after the fact, and it forces the real conversation — how good is good enough, and how bad is dead — to happen early, when it's a planning discussion rather than a verdict on someone's work. The same logic applies to the uncertainty clause: define the escalation-rate range you can staff for. An escalation behavior that routes 40% of cases to humans satisfies the letter of the spec and destroys the business case; the requirement should bound it.
Data requirements are requirements
Classic requirements treat data as an implementation concern — the spec says what the system does, and engineering figures out what it needs. For AI projects this is backwards: the data determines what the system can do, so data clauses belong in the requirements document itself, with the same sign-off weight as any functional requirement. If a data clause can't be satisfied, every behavioral requirement downstream of it is fiction, and it's far cheaper to discover that at requirements time than at model-training time.
- Availability clause: the named fields and sources the capability depends on, with measured (not assumed) population rates — 'requires customer-notes field, currently populated in 61% of rows'.
- Access clause: who grants access, under what permission regime, and by when — an unsigned data-access agreement is an unmet requirement, not an admin task.
- Freshness clause: how current the data must be at inference time, and what the system does when it's stale.
- Labeling clause: if supervised evaluation is needed, who produces labels, at what rate, at what agreement level — 'the AP lead reviews 50 cases/week' is a requirement with a named owner and a real cost.
- Retention and privacy clause: what the system may store, for how long, and what must never leave which boundary — written here, not discovered by legal in month four.
The interview technique: decisions and exceptions, not features
The standard elicitation question — 'what should the AI do?' — produces wishes: stakeholders describe an idealized employee and the requirements doc becomes fan fiction. The productive move is to never ask about the AI at all. Ask about the decision the human makes today, and then mine the exceptions, because the exceptions are where the behavioral requirements live. A useful sequence for each stakeholder interview: walk me through the last one you handled — not a typical one, the literal last one. Then: tell me about the last one that went wrong — what did wrong look like, who noticed, what did it cost. Then: which cases do you refuse to decide alone, and who do you pull in? That third question is gold — it's the human escalation policy, and it converts almost directly into the model's uncertainty clause. Finally: if you had to write one rule a smart new hire must never break in this task, what is it? That question surfaces the hard constraints ('never auto-approve anything from a new supplier') that stakeholders forget to mention because they're too obvious to say. Requirements gathered this way arrive pre-formatted as behavior: populations of cases, error costs, escalation rules and inviolable constraints — exactly the raw material the three-clause spec needs.
