Vet AI engineers in four stages: an evidence screen of shipped work, a realistic work sample, a systems-depth conversation, and a short team-fit trial. Run in that order, each stage filters cheaply before the next, and the whole loop fits inside a week without lowering the bar.
The four stages
- 1Evidence screen (30 min, on paper): shipped AI work in production, not demos; concrete numbers (accuracy, cost, latency, users); clear individual contribution. Pass: at least one system they owned end-to-end with results they can quantify.
- 2Work sample (2-4 hours, time-boxed): a realistic task shaped like your product, e.g. diagnose and improve a failing RAG pipeline with a small eval set. Score against a written rubric: diagnosis before fixes, measurement, code quality, communication. Pass: a defensible improvement plus evidence they measured it.
- 3Systems depth (60-90 min conversation): walk one of their past systems end to end, data, evals, failure modes, cost, what broke and what they changed. Push on trade-offs. Pass: specific, honest, numerate answers; they can say 'I don't know' and reason from there.
- 4Team-fit trial (half-day to 5 days): pair with the pod they would join on real work, review a PR, plan a small feature. Pass: the pod wants them back, communication is clear, and they leave things better documented than they found them.
Rules that keep the checklist honest
- Write pass criteria before you meet the candidate; never adjust them per person.
- Pay for anything beyond 4 hours of candidate time, unpaid week-long 'tests' select for the desperate, not the best.
- Have the working pod score the sample, not just the hiring manager.
- Decide within 48 hours of the last stage; a great vetting loop is wasted by a slow yes.
- Track outcomes: revisit the rubric quarterly against how hires actually performed.
