Ask most teams why they picked a given model and the answer is some version of 'it was top of a leaderboard' or 'everyone else is using it.' Both are weak reasons, because leaderboards measure general capability on benchmark tasks that may have nothing to do with your task, and 'everyone else' optimizes for a different cost, latency and compliance profile than yours. Model selection needs to be a repeatable process with its own inputs and its own re-evaluation cadence, not a decision made once at kickoff and never revisited, because the model landscape changes meaningfully every few months and a choice that was right in January can be measurably wrong by July.
Why 'best on the leaderboard' is the wrong starting question
Public benchmarks (general knowledge tests, coding challenges, math problem sets) measure a model's capability across a broad, standardized task distribution. Your production use case is almost never that distribution: it's a narrow slice of tasks, on your specific data, under your specific latency and cost constraints, often in a domain the benchmark barely touches (internal contract language, a niche technical vocabulary, a customer support tone). A model that scores three points higher on a general reasoning benchmark can easily score worse on your actual task, because the benchmark gap has nothing to do with the specific weaknesses that matter for you. Leaderboard rank is a reasonable filter for 'is this model competent in general' but a poor basis for a final decision; the only benchmark that should decide a production model choice is one built from your own real examples.
The axes that actually matter
Enterprise model selection collapses to six variables, and teams that skip any one of them tend to discover it the expensive way, in production, months later.
| Axis | What to actually check | Common mistake |
|---|---|---|
| Accuracy on your task | Score against 50-200 real examples from your own data, not a public benchmark | Trusting a general leaderboard rank as a proxy for task fit |
| Cost at real volume | Model your actual monthly token volume, not the pricing page's per-1K-token number in isolation | Pricing decisions off a demo's token count instead of production-scale traffic |
| Latency under load | Test P50 and P95 latency under concurrent load, not a single warm request | Measuring latency once, in isolation, with no concurrent traffic |
| Context window fit | Check your actual input size (including retrieved context) against the window, with margin | Assuming a large window means no truncation risk as documents or chat history grow |
| Data residency and compliance | Confirm where inference actually runs and what the vendor's data-use and retention terms say, in writing | Assuming a vendor's general compliance certification covers your specific regulatory requirement |
| Vendor lock-in risk | Estimate the cost to migrate prompts, fine-tunes and integrations to a different provider | Building deeply against one vendor's proprietary features with no abstraction layer |
A scoring framework that survives scrutiny
The practical fix for gut-feel model selection is a weighted scorecard, decided before you start testing candidate models, not after you already have a favorite. List the six axes above, assign each a weight that reflects your actual business constraints (a regulated industry weights compliance and residency heavily; a consumer app at massive scale weights cost and latency heavily), score each candidate model 1-5 on each axis based on real testing, and let the weighted total, not a single standout number, drive the decision. This does two things: it forces you to actually test cost and latency at realistic scale instead of assuming, and it produces a decision you can defend later when someone asks 'why didn't we just use the model that topped last month's leaderboard.'
- Set the weights before testing begins, based on your actual constraints, not the model you already suspect you'll pick.
- Score accuracy against your own eval set of real, hard examples, never a generic benchmark.
- Test cost and latency at a volume and concurrency that resembles production, not a single demo call.
- Get compliance and data-residency terms confirmed in writing before scoring that axis, not assumed from a marketing page.
- Record the scorecard. It becomes the artifact you re-run at the next re-evaluation, not a decision you have to reconstruct from memory.
The two axes most teams underweight: compliance and lock-in
Data residency and compliance requirements get treated as a checkbox ('does the vendor have a SOC 2 report') when the real question is narrower and more specific: does this exact regulatory requirement, this data classification, this jurisdiction, actually get satisfied by this vendor's specific inference and storage setup, in writing. General compliance certifications are necessary but not sufficient; teams in regulated industries have been surprised late in a rollout to discover a vendor's certification didn't cover the specific data-handling clause their contract required. Vendor lock-in is the other underweighted axis, because it's invisible until you need to migrate: prompts tuned to one model's quirks, fine-tunes that don't transfer, integration code built against a provider's proprietary function-calling format. The insurance is cheap relative to the risk: build a thin abstraction layer between your application and the model provider from day one, so a provider switch is a configuration change, not a rewrite.
Re-evaluation cadence: model choice is not a one-time decision
The model landscape moves fast enough that a choice made a year ago, sometimes even a quarter ago, is worth re-checking against what's actually available now. The right cadence for most enterprise teams is a formal re-evaluation of the full scorecard every quarter, using the same eval set so results are comparable over time, plus an ad hoc check whenever a genuinely major new model releases from any provider you'd consider. 'Ad hoc' doesn't mean switching immediately, it means running your own eval set against the new model and comparing the score to your incumbent before deciding anything. Most quarters this confirms you should stay put; that confirmation is valuable too, because it means the next conversation about switching models starts from evidence instead of hype.
- Full scorecard re-run every quarter, using the same eval set each time for a comparable trend line.
- Ad hoc eval run whenever a major new model releases, before any decision to switch, not after.
- Track cost and latency drift too, providers change pricing and infrastructure without always announcing it loudly.
- Keep the abstraction layer current; a re-evaluation that reveals a better model is only useful if switching is actually cheap.
Who should own this decision
Model selection sits at the intersection of engineering, cost management and compliance, which is exactly why it tends to get made informally by whichever engineer built the first prototype with whatever API key they had. That's fine for a prototype; it's a liability once the choice is load-bearing for a production system with real cost and compliance exposure. Someone needs explicit ownership of the scorecard, the re-evaluation cadence, and the abstraction layer, whether that's a senior AI engineer, an ML platform lead, or a fractional AI advisor brought in specifically to build the evaluation discipline before the team scales past the point where 'we picked it because it seemed good' is an acceptable answer to give a customer or an auditor.