Most hiring plans for an 'AI team' are copied from a generic engineering org chart with 'AI' pasted onto a few titles. That's not what companies actually shipping AI products look like inside. After watching dozens of teams go from first prototype to production revenue, the structure that works is flatter, more cross-functional, and arrives in a specific order, not the order most job reqs get posted in.
Stage one: one builder, not a team
At the prototype stage, the right structure is one senior generalist AI engineer, not a team. This person needs to be comfortable across the model layer, the data plumbing, and enough product judgment to know when a demo is actually a product. Hiring a 'team' here, a data scientist plus an ML engineer plus a PM, produces coordination overhead before there's anything to coordinate. The single builder should report directly to whoever owns the product decision, usually the founder, so the feedback loop from 'this doesn't work' to 'try this instead' stays under a day.
- One senior AI/full-stack engineer who has shipped an LLM-backed feature to production before, not just a notebook.
- Direct line to the product decision-maker, no PM layer yet, decisions move faster than a standup cadence.
- This person owns the eval, the prompt or model choice, and the data access question, all three, not a committee.
Stage two: the workflow triangle (3-5 people)
Once the first feature is in front of real users, the structure that reliably works is a triangle: one engineer who owns the AI/model layer, one who owns the surrounding product engineering (APIs, UI, data pipes), and one who owns evaluation and quality full-time. Teams that skip the dedicated eval owner ship regressions silently, because nobody's job is to notice when the model got worse. This is also the point where a part-time or fractional product lead starts paying for itself, someone has to keep the roadmap honest against what users actually do with the feature versus what the demo promised.
| Role | Owns | What breaks if it's nobody's job |
|---|---|---|
| AI/model engineer | Model choice, prompting or fine-tuning, latency/cost tradeoffs | Feature quality plateaus; nobody can say why |
| Product engineer | APIs, UI, integration with the rest of the product | AI feature stays a bolted-on tab forever |
| Eval/quality owner | Test sets, regression tracking, the 'is it actually better' answer | Silent quality regressions ship to production |
| Fractional PM | Roadmap vs. real usage, scope discipline | Team builds impressive demos users don't need |
Stage three: the split that scales (6-15 people)
Past five or six people, the triangle splits into two tracks that both feed a shared evaluation layer: a model/platform track (model selection, fine-tuning, infra, cost) and a product-integration track (features, UI, workflows). The eval and data-quality function graduates from one person's job to a small shared service both tracks pull from, this is the seam most teams miss, and it's why a team that scaled headcount 3x often ships slower, not faster. The fix isn't more engineers, it's making evaluation a first-class function with its own headcount before the two tracks diverge in what 'good' means.
- Model/platform track: model selection, cost and latency tuning, infra, sometimes fine-tuning.
- Product-integration track: the features users touch, workflow design, UI, adoption.
- Shared eval and data-quality function: the tiebreaker both tracks report their numbers to.
- A dedicated AI product manager (full-time by now) translating between the two tracks and the business.
Where the AI team should report
Reporting into a general engineering manager without AI product experience is the single most common structural mistake at this stage. AI features have a different failure mode than typical software, they degrade gradually and non-deterministically, so a manager who evaluates progress the way they'd evaluate a normal sprint (tickets closed, features shipped) will miss quality erosion until customers complain. The AI team should report to someone who can read an eval score the way a normal EM reads a burndown chart, whether that's a Head of AI, a technical co-founder, or an embedded senior AI lead brought in specifically for that judgment.
The hiring order that actually works
- 1One senior generalist AI engineer who can own a full workflow, model to UI, alone.
- 2A second engineer, deliberately picked to cover whichever half (model or product) the first person is weaker on.
- 3A dedicated eval/quality owner, even part-time or fractional, before headcount hits five.
- 4A product lead once the feature has real users and the roadmap needs defending against feature creep.
- 5The track split (model vs. product) only once eval is already a standing function, not before.
