A model tops the leaderboard, you swap it into your product, and your users complain more, not less. This happens constantly, and it is not a mystery. The benchmark measured the model's ability to answer graduate physics questions. Your product summarizes support tickets for a plumbing company. The two have almost nothing to do with each other, and the leaderboard never claimed otherwise. If you want to know which model is best at your job, you have to measure your job. That means building your own eval, and it is far less work than it sounds.
Why public benchmarks mislead
Public benchmarks are useful for one thing: telling model builders whether they are making progress on broad, general capability. They are close to useless for telling you whether a model will do your specific task well. There are a few reasons for this, and they compound.
The tasks do not match yours. A high MMLU score says a model knows a lot of trivia. It says nothing about whether it can follow your formatting rules or refuse politely when a customer asks something out of scope. Worse, popular benchmarks leak into training data over time, so a strong number can partly reflect memorization rather than skill. And a single averaged score hides exactly the failures you care about. A model that is right ninety percent of the time can still be catastrophically wrong on the ten percent that is your core use case.
None of this means models are bad. It means the benchmark is answering a question you did not ask.
Building the thing
An eval is just a set of real inputs, a definition of what a good output looks like, and a way to score outputs at scale. You can start with thirty examples in a spreadsheet.
Collect real cases. Do not invent test inputs. Pull them from your actual logs, your support queue, the messy things users really send. Include the easy ones, but hunt specifically for the hard, weird, and embarrassing cases, because those are where models differ and where averages lie. Aim for enough variety that a model cannot pass by getting one category right.
Define pass and fail before you look at any output. This is the step people skip, and skipping it poisons everything downstream. For each case, write down what makes an answer acceptable. Sometimes it is an exact value. More often it is a checklist: did it cite the right ticket, did it stay under the word limit, did it avoid promising a refund. Vague criteria give you vague evals, which are worse than none because they feel rigorous while measuring nothing.
Automate the scoring. You will run this hundreds of times, so it cannot be manual. Three approaches, roughly in order of how much you should trust them:
- Exact or programmatic checks: string match, regex, JSON schema validation, a number within tolerance. Cheap, fast, and never lies. Use these wherever the answer is checkable.
- Model-as-judge: another model grades the output against your criteria. Useful for fuzzy qualities like tone, but the judge has its own biases and needs its own sanity checks.
- Human review: the gold standard and the bottleneck. Reserve it for the cases automation cannot handle, and for periodically auditing whether your automated scores still agree with human judgment.
Iterate. An eval is not a document you write once. Every time the model surprises you in production, that failure becomes a new test case. Over a few months your eval becomes a precise portrait of your task's hard edges, and the moment a new model comes out you can answer the only question that matters, which is whether it is better at your job, in an afternoon instead of a quarter.
The payoff you do not expect
The obvious benefit is that you can compare models honestly. The bigger benefit sneaks up on you: the act of defining pass and fail forces you to say, in plain terms, what your product is actually supposed to do. Most teams cannot do this cleanly, and it shows in their prompts. Writing an eval is the most productive argument your team will have about what good even means.
You will never top a public leaderboard, and you should not want to. You want to top the only leaderboard your users vote on, which has one entry, and it is you. Build the eval, keep it close, and let the people selling benchmark numbers argue about whose model knows more physics.