Read enough launch posts and you notice something strange: every new model is state of the art. All of them. Simultaneously. That cannot be true, and yet each chart is technically honest. Welcome to the benchmark wars, where nobody is exactly lying and almost everyone is being misled, gently, on purpose.
How everyone wins at once
The trick is simple. There are dozens of benchmarks. Pick the one your model tops, put it on the slide, and leave the rest off. Your rival does the same with a different benchmark. Both charts are accurate. Both are useless for the only question that matters, which is whether the model is good at your work. The arms race is not really about capability. It is about finding a number to be tallest on.
The quieter distortions
- Curated lineups. The competitors on the chart are chosen. The one that would have won is often just absent.
- Generous settings. Best-of-many attempts and special prompting produce scores you will never see in daily use.
- Contamination. Popular benchmarks leak into training data, so a high score can measure memory rather than skill.
- Tiny margins, huge claims. A two-point lead on a noisy test becomes "a new era" in the headline.
A leaderboard measures how a model did on someone else homework, graded by the model own teacher, on a test it may have already seen.
Why I still glance at them
Benchmarks are not worthless. As a rough first filter, they will tell you a model is roughly in the right class and not obviously broken. That is a real, if modest, use. The mistake is treating a filter as a verdict, letting a curated chart make a decision that only your own tasks can make honestly.
What to trust instead
Your own tiny, private benchmark: a handful of real tasks from your actual work, run through any model you are considering. It is unglamorous, it takes fifteen minutes, and it is worth more than every leaderboard combined, because it measures the one thing the marketing cannot spin, which is whether the thing works for you. Trust your tasks. Skim the charts. And treat any sentence containing the phrase state of the art as the beginning of a sales pitch, not the end of an argument.