To hire a good LLM engineer, test for production judgment, evaluation, retrieval quality, cost and reliability, not for prompt tricks or framework name-dropping. The best predictor is a short work sample on a failing LLM feature: real LLM engineers reach for evals and data first, hobbyists reach for a bigger model.
The skills that actually matter
- Evaluation engineering: designing eval sets and metrics that catch regressions before users do.
- Retrieval and context: chunking, indexing, and knowing why RAG answers go wrong.
- Cost and latency literacy: model selection, caching and routing as engineering decisions, not afterthoughts.
- Failure-mode design: guardrails, fallbacks and graceful degradation for a probabilistic component.
- Plain software engineering: LLM features live inside real systems; strong engineers ship the whole feature.
Interview questions that reveal them
- 1'Our RAG feature answers confidently but wrongly about 10% of the time. Walk me through your first week.' Listen for: measure first, inspect retrieval, build an eval set, before touching models.
- 2'How would you know this feature got worse after a model upgrade?' Listen for: regression evals in CI, monitored quality metrics, not 'we'd notice'.
- 3'Inference cost just tripled with usage. What do you do?' Listen for: routing, caching, smaller models for easy cases, measuring quality impact of each.
- 4'Tell me about an LLM feature you shipped that failed. What did you change?' Listen for: specifics, data, and honesty, everyone who has shipped has failed.
Red flags to screen out
- Portfolio of demos and notebooks, nothing operated in production with real users.
- No numbers: cannot say what accuracy, cost or latency their past systems achieved.
- Model maximalism: every problem answered with 'use the latest model' rather than data and evals.
- Dismisses evaluation as slowing things down, this engineer ships regressions.
- Framework name-dropping in place of explaining what the framework does underneath.
