Evaluating an AI vendor comes down to twelve questions across four areas: what happens to your data, what the product actually is under the demo, how quality is measured, and what it costs to leave. A vendor with good answers to all four areas is rare, which is exactly the point of asking.
The 12 questions
| Area | Question | Red flag answer |
|---|---|---|
| Data | Is our data used to train your models? | Vague or 'only to improve the service' |
| Data | Where is data processed and stored? | Can't name regions or subprocessors |
| Data | Can we delete our data and verify it? | Deletion 'on request' with no SLA |
| Product | What breaks if your model provider changes terms? | No answer; single hard dependency |
| Product | What's proprietary beyond the prompt layer? | Demo-ware: thin wrapper, no workflow depth |
| Product | Can we pilot on our own data? | Demo-only, or pilot needs full contract |
| Quality | How do you measure output quality? | No eval process, only testimonials |
| Quality | What's your handling of hallucinated output? | 'Our model doesn't hallucinate' |
| Quality | What human-review controls exist? | All-or-nothing automation |
| Exit | What does data export include? | Raw data only, no configurations or history |
| Exit | What are the contract terms if quality degrades? | No quality commitments at all |
| Exit | What does switching actually cost? | They can't describe an offboarding |
How to run the pilot
- Use your real, messy data, the demo data is curated by definition.
- Define pass/fail before the pilot starts, not after you've seen results.
- Include your hardest 20 cases, edge cases are where vendors differ.
- Have the people who'll use it daily run it, not the buying committee.
