An AI QA / eval engineer builds the measurement systems, test sets, metrics, regression suites, human-review loops, that tell you whether an AI feature actually works, and keeps working after every model or prompt change. Classic QA verifies deterministic behavior; eval engineering quantifies probabilistic behavior, and it's become the role that decides whether AI features are shippable.
What the role owns
- Golden test sets that represent real usage, including the ugly edge cases.
- Metrics that map to user outcomes, not just generic accuracy scores.
- Regression gates: no prompt or model change ships without passing evals.
- LLM-as-judge pipelines, calibrated against human ratings.
- Failure taxonomies so 'the AI was wrong' becomes actionable categories.
- Human review loops for the cases automation can't score.
How it differs from classic QA
Classic QA asks 'does the function return the right value?' Eval engineering asks 'across a thousand messy real inputs, how often is the output good, how bad are the failures, and did yesterday's change make it better or worse?' It's statistical, adversarial and never done, model updates alone can shift behavior with zero code changes.
When to hire one
- An AI feature is core to your product and quality complaints are anecdotal, not measured.
- Prompt changes ship untested because 'it looked fine'.
- Engineers spend serious time hand-checking outputs.
- A model upgrade silently broke something and you found out from users.
