Hire a Head of AI against a written scorecard: three to five outcomes for the first year, the competencies that drive them, and explicit disqualifiers, agreed before you meet a single candidate. Without it, the role gets filled by whoever talks about AI most impressively, which is exactly how the mis-hires happen.
Outcomes: what year one must deliver
- Two or three AI capabilities in production with measured business impact.
- An AI roadmap the exec team actually uses to make decisions.
- A working eval and governance practice, quality and risk owned, not improvised.
- A hiring plan executed: the first AI engineers in seat and productive.
- A build-vs-buy posture that saved or focused real money.
Competencies and how to test them
- Technical judgment: walk through a system they shipped, decisions, trade-offs, failures. Depth is unfakeable in 45 minutes.
- Prioritization: give them your real candidate use cases and have them rank with reasoning.
- Translation: can they explain eval strategy to a CFO? Roleplay it.
- Hiring ability: who have they hired, where are those people now?
- Pragmatism: ask what they'd buy rather than build, zealots fail this.
Disqualifiers to enforce
- Never shipped AI to production, advised, evangelized or researched only.
- Can't name a project of theirs that failed and what they changed.
- Dismisses evals, safety or cost as 'details for later'.
- Talks exclusively in vendor and framework names rather than problems.
