Every non-technical founder we work with eventually asks the same question: how do I know if this person is actually good at the thing I'm paying them to be good at? The honest answer is you can get most of the way there without writing a line of code yourself, if you replace 'can I judge their code' with 'can I judge their process, their explanations, and their output against reality,' which are all things a careful non-technical founder can assess directly.
Reframe what you're actually testing
You're not qualified to judge whether their code follows best practices, and that's fine, that was never the highest-signal thing to check anyway. What you can judge, reliably, is whether they explain their decisions clearly, whether their claims about what they built hold up when you probe, and whether the actual output works on your real problem. Those three things predict success on your team better than a code review most non-technical founders couldn't meaningfully perform anyway.
The plain-language explanation test
Ask the candidate to explain, in a way you personally can follow, why they'd choose one approach over another for a specific problem you describe, and why a wrong answer from their system would happen and how they'd catch it. A strong AI engineer can do this without jargon-dumping on you; someone who can only answer in dense technical language, or who gets impatient at the question, is telling you something important about how they'll communicate with you for the next two years.
- Ask 'what would make this wrong, and how would you know?' about their own past project, not a hypothetical.
- Watch for jargon used to end the conversation rather than to answer the question.
- A good sign: they ask you clarifying questions about your business constraints before answering.
- A bad sign: every answer reduces to 'trust me, this is how it's done.'
Reference checks that actually work for non-technical founders
- Ask the reference: 'did what they built actually hold up in production six months later?' not 'were they a good employee.'
- Ask: 'what did they get wrong, and how did they handle finding out?' Everyone gets things wrong; how they respond is the signal.
- Ask: 'would you personally trust them to own a workflow without you checking their work daily?'
- If a reference is vague or only offers generic praise, ask directly whether they'd rehire this person, silence or hedging is itself an answer.
Use a structured take-home against your real data
| What you can check yourself | What to outsource or use a tool for |
|---|---|
| Did the output actually work on your real (anonymized) examples? | Whether the code itself is well-structured or efficient |
| Did they flag the cases where it failed, unprompted? | Whether their technical approach is state-of-the-art |
| Was their write-up clear enough for you to follow their reasoning? | Whether their architecture choices will scale technically |
| Did they ask good clarifying questions before diving in? | Deep security or infra review of what they submitted |
When to bring in outside technical judgment
For the final call, especially on a senior or expensive hire, it's worth paying a fractional technical advisor or an embedded evaluation service for a few hours to pressure-test the finalist's technical claims. This isn't an admission you vetted poorly, it's the same reason non-technical buyers get a mechanic to check a used car after they've already decided they like it. Use it as confirmation on your top choice, not as a substitute for the process above, which is what actually builds your own judgment for every hire after this one.