You don't need to read a line of Python to make a good AI hiring decision, but you do need a process that doesn't rely on your own technical judgment, because right now you have none to rely on and the candidate knows it. Non-technical founders lose this hire in one of two ways: they get dazzled by demo polish, or they overcorrect and hire whoever uses the most jargon. Neither test predicts who ships.
Why demos and resumes lie to you specifically
A charismatic candidate can build a demo that looks like magic in fifteen minutes, because demos are optimized for the happy path and you have no way to probe the unhappy one. Resumes are worse: 'built an AI-powered X' could mean they architected a production system end to end, or that they added one API call someone else wired up. As a non-technical founder, both artifacts are almost pure noise unless paired with a process that forces candidates to show their work under conditions you can actually judge.
- A demo shows the best five minutes; you're hiring for the worst five minutes, when the model returns garbage in front of a customer.
- Resume verbs ('built,' 'led,' 'shipped') are unverifiable without a specific artifact and a specific metric attached.
- Confidence and technical competence are uncorrelated; some of the best AI engineers are quiet and hedge their claims, which reads as weakness to an untrained ear.
The four-step framework
- 1Screen on shipped evidence, not claims: ask for one specific AI feature they built that is live today, and ask what it cost to run per month and what broke in the first month. A candidate who can't answer the cost question probably never owned the feature end to end.
- 2Run a small paid take-home or working session (2-4 hours, paid): give them a real, scoped problem from your own product, not a generic leetcode-style AI puzzle. Judge the questions they ask before they start, strong candidates ask about edge cases and data quality before writing anything.
- 3Bring in a technical proxy for one conversation: a fractional CTO, an advisor, or a vetted staffing partner, purely to review the take-home output and ask two or three technical follow-ups. Two hours of borrowed judgment is the highest-leverage money you'll spend in this hire.
- 4Check references on failure, not success: ask a past manager or teammate what went wrong on their last AI project and how the candidate handled it. Evasive or overly polished answers here are the clearest non-technical-readable red flag there is.
Red flags you can spot without reading code
- They can't explain the last time their model or system was wrong in production, everyone's has been.
- Every answer defaults to the newest model or framework name-drop instead of a tradeoff ('we used X because Y was too slow/expensive/unreliable for our case').
- No mention of evaluation or testing when you ask how they know something works, this is the single biggest tell of someone who prototypes but doesn't operate.
- They quote a cost or timeline with no caveats; real AI engineers hedge because model behavior and costs are genuinely uncertain.
- Portfolio is all personal projects with no discussion of what happened after launch, personal projects don't have angry users or a CFO asking about the API bill.
What a strong non-technical read looks like
The strongest signal you can pick up without technical depth is specificity paired with humility: a candidate who says 'it worked for 80% of cases, we caught the other 20% with a review queue, and here's the metric we tracked to know it was safe to remove that queue later' is telling you they've operated a real system through its failure modes. That sentence requires zero code literacy to evaluate and predicts performance better than almost anything else you can ask.