Every resume that crosses my desk this year says 'AI engineer.' Maybe one in six of them has actually shipped an AI system that survived contact with real users and real data. The other five learned the vocabulary from a course, built a chatbot over a weekend, and updated their title. That's not a moral failing, it's a rational response to a hot market, but it means the burden of filtering is entirely on you. The good news is the tells are consistent and checkable in a single conversation if you know what to listen for.
Vocabulary without specifics
Anyone can learn to say 'we used RAG with a vector database' in a weekend. What they can't fake is the next layer down: which retrieval strategy, why, what they tried first that didn't work, and how they measured whether the change actually helped. Ask one follow-up question past the buzzword and watch what happens. A real operator goes more specific. A title-inflated candidate goes more vague, or pivots to a different, equally generic claim.
- 'We fine-tuned a model' followed by no answer to 'on what data, and how did you know it worked?'
- 'We built a RAG pipeline' followed by no answer to 'what was your chunking strategy and why?'
- 'We used agents' followed by no answer to 'what happened when the agent took a wrong action?'
- Confident use of the year's trending terms with no ability to compare tradeoffs between two approaches.
No story about what broke
Production AI systems fail. They hallucinate, they drift when a model version changes underneath you, they get expensive at scale, they retrieve the wrong context. Anyone who has actually run one for more than a demo has a specific, slightly embarrassing story about something that didn't work. If a candidate describes every project as smooth and successful, either they haven't shipped anything real, or they're not being honest about their own work, both are disqualifying for different reasons.
Portfolio and resume tells worth checking
| What you see | Likely reality |
|---|---|
| GitHub full of forked tutorials, no original architecture decisions | Followed courses, hasn't designed a system independently |
| Every project description omits cost or latency numbers | Never had to make a production tradeoff |
| Resume says 'AI engineer' for 8 months after 5 years as a generic backend dev | Title updated, skill set may not have caught up yet, worth checking not disqualifying |
| Can name the eval metric they used and why it was the right one | Has actually had to defend a quality decision to a stakeholder |
| Describes a project that shipped, then got rolled back or rebuilt | Has lived through the part of the job that actually teaches judgment |
Three questions that cut through inflation fast
- 1"Walk me through the last time an AI feature you built was wrong in production, how did you find out, and what did you change?" Real operators answer in specifics within 30 seconds; others stall or generalize.
- 2"How did you decide the system was good enough to ship?" Listen for an evaluation methodology, a threshold, a comparison, not a vibe.
- 3"What did this cost to run, and did that ever become a problem?" Anyone who's operated a system past the demo stage has thought about unit economics; anyone who hasn't, hasn't been asked to.
