To hire an AI agents engineer, test for systems skills, not prompt tricks: tool design, state management, error recovery and evals are what separate engineers who ship reliable agents from those who demo them. Agentic systems fail in loops and edge cases, so the hiring bar has to live there too.
The skills that actually matter
- Tool and API design: clean, typed interfaces the model can call reliably.
- State and memory management across multi-step runs.
- Failure handling: retries, fallbacks, and knowing when to hand off to a human.
- Evals for agent behavior, not just single-response quality.
- Cost and latency budgeting, agents multiply token spend fast.
- Guardrails: permissions, sandboxing, and blast-radius thinking.
How to test for them
- 1Give a small take-home: an agent with two tools and a deliberately flaky API, ask them to make it robust.
- 2In the review call, ask how they'd eval it: what metrics, what traces, what failure taxonomy.
- 3Probe a real project: longest-running agent they shipped, what broke, what they changed.
- 4Ask about cost: how they'd cap spend on a runaway loop.
Red flags
- Talks frameworks and demos, can't describe a production failure they fixed.
- No answer for observability, can't say how they'd trace a bad run.
- Treats evals as optional polish rather than the core of the job.
- Assumes the model will 'figure it out' instead of designing constraints.
