The best way to interview AI engineers is a small, paid or time-boxed take-home that mirrors the real job, such as building and evaluating an LLM feature, because work samples predict performance far better than algorithm puzzles. Leetcode tests memorized patterns; AI work rewards judgment under ambiguity.
Designing the task
- Pick a slice of your real work: classify support tickets, extract fields from messy documents, build a small RAG answerer.
- Require an eval: a small labeled set and a metric, how they measure is the richest signal.
- Provide the boring parts (data, boilerplate) so hours go to judgment, not setup.
- Cap scope explicitly: 'we expect rough edges; document trade-offs instead of fixing everything'.
- Allow AI coding tools, you're hiring for how they work in 2026, not 2019.
Running the debrief
- 1Ask them to walk through decisions, not code line by line.
- 2Change a requirement live: 'latency budget just halved, what do you cut?'
- 3Ask what they'd do with two more days, prioritization reveals seniority.
- 4Probe the eval: why that metric, what cases does it miss?
Keeping it fair
- Same task, same rubric, same time-box for every candidate.
- Score anonymously where possible before the debrief.
- Compensate longer tasks or shrink them, senior candidates walk away from free 10-hour projects.
- Never use submissions as free work; keep tasks synthetic or on toy data.
