Star counts, demo polish and a list of impressive-sounding model names tell you almost nothing about whether someone can build an AI system that survives contact with real users. What tells you something is far less glamorous: did they measure quality before claiming it improved, did they make a deliberate tradeoff and explain why, did anything they built actually run in production long enough to break. A great portfolio reads more like an incident log than a highlight reel.
The signals that don't actually tell you much
It's easy to over-index on things that are visible and easy to compare, and that's exactly why they're weak signals: they're the easiest things for a candidate to optimize for appearance rather than substance.
- GitHub star counts, they correlate with marketing and timing more than engineering quality.
- A long list of model and framework names, name-dropping is free and doesn't demonstrate judgment.
- Polished demo videos, a demo is optimized to work once, on curated input, which is precisely what production never gives you.
- Number of projects, ten shallow tutorial clones say less than one project taken to production and maintained.
What to look for instead
- 1Evidence of measurement: did they define what 'good' meant before or after they built the thing? Look for an eval set, a metric, a before/after number, anything that shows quality was managed, not assumed.
- 2A production lifespan: has anything they built run long enough, with real users or real data, to have needed a fix after launch? A project that's still running six months later teaches you more about the builder than one that shipped and was never touched again.
- 3Tradeoff reasoning: can they explain why they chose one retrieval strategy, model size or architecture over an equally plausible alternative, and what it cost them? Judgment shows up in the roads not taken.
- 4A failure they can describe in detail: what broke, how they found out, what they changed. This is the single highest-signal thing you can ask for.
- 5Ownership of the unglamorous parts: logging, cost monitoring, a rollback plan. These rarely make it into a portfolio deck, which is exactly why asking about them directly is so effective.
How to actually run the portfolio review
Don't just read it, interrogate it. Pick the one project the candidate seems proudest of and spend fifteen minutes going deep rather than skimming five projects shallowly. The goal is to find the edge of their real understanding, which is usually one or two follow-up questions past whatever they volunteered first.
- "Show me the worst output this system ever produced, and what you did about it."
- "What would you build differently if you started this project again today?"
- "Who else used this, and how do you know whether it actually helped them?"
- "What did you decide not to build, and why?"
Weak portfolio vs strong portfolio, side by side
| Dimension | Weak portfolio | Strong portfolio |
|---|---|---|
| Evidence of quality | Claims it 'worked well' | Cites a specific eval score or before/after comparison |
| Production exposure | Demo, ran once, never touched again | Ran for real users, was maintained and fixed over time |
| Tradeoff awareness | One approach presented as obviously correct | Explains what else was considered and why it lost |
| Failure handling | No failures mentioned | A specific failure, root cause and fix |
| Cost/latency awareness | Never mentioned | Discussed as a real constraint that shaped decisions |
