Every model launch comes with a chart where the new model is tallest. The charts are technically true and practically useless, because they are marketing wearing a lab coat. Here is how to read a benchmark like a skeptic, so a leaderboard never again talks you into the wrong model for your actual work.
Ask what the test measures
A benchmark measures performance on a specific set of tasks, and that set may have nothing to do with yours. A model that tops a math or coding benchmark may be mediocre at the writing or analysis you actually do. The number is real. Its relevance to you is the open question, and it is the one the marketing hopes you skip.
Watch for the missing context
- Which competitors are shown, and which are quietly absent? The lineup is curated. The model that would have beaten theirs is often just not on the chart.
- What settings produced the number? Best-of-many attempts, special prompting, and generous configurations inflate scores you will never reproduce in normal use.
- How big is the gap, really? A two-point lead on a noisy benchmark is a rounding error dressed as a victory.
A benchmark tells you how a model did on someone else homework. Your homework is different, and it is the only one that matters.
Beware the contamination problem
Models are trained on enormous slices of the internet, and popular benchmarks leak into that training data. When a model has effectively seen the test before, its score measures memory, not ability. This is a real and under-discussed reason to distrust a suspiciously perfect number on a well-known benchmark.
Run your own tiny benchmark
The only test that counts is yours. Keep a small set of real tasks from your actual work, a handful of prompts you know well, and run any model you are considering through them. Fifteen minutes of that tells you more than every leaderboard combined, because it measures the one thing you care about: does it do my job well.
The takeaway
Treat published benchmarks as a rough filter, not a verdict. Ask what was measured, who was left out, and how the number was produced. Then ignore all of it and try the model on your own work. The chart is their argument. Your tasks are the evidence.