Benchmarks

A model tops the leaderboard, you swap it into your product, and your users complain more, not less. This happens constantly, and it is not a mystery. The benchmark measured the model's ability to answer graduate physics questions. Your product summarizes support tickets for a plumbing company. The two have almost nothing to do with each other, and the leaderboard never claimed otherwise. If you want to know which model is best at your job, you have to measure your job. That means building your own eval, and it is far less work than it sounds.

Why public benchmarks mislead

Public benchmarks are useful for one thing: telling model builders whether they are making progress on broad, general capability. They are close to useless for telling you whether a model will do your specific task well. There are a few reasons for this, and they compound.

The tasks do not match yours. A high MMLU score says a model knows a lot of trivia. It says nothing about whether it can follow your formatting rules or refuse politely when a customer asks something out of scope. Worse, popular benchmarks leak into training data over time, so a strong number can partly reflect memorization rather than skill. And a single averaged score hides exactly the failures you care about. A model that is right ninety percent of the time can still be catastrophically wrong on the ten percent that is your core use case.

None of this means models are bad. It means the benchmark is answering a question you did not ask.

Building the thing

An eval is just a set of real inputs, a definition of what a good output looks like, and a way to score outputs at scale. You can start with thirty examples in a spreadsheet.

Collect real cases. Do not invent test inputs. Pull them from your actual logs, your support queue, the messy things users really send. Include the easy ones, but hunt specifically for the hard, weird, and embarrassing cases, because those are where models differ and where averages lie. Aim for enough variety that a model cannot pass by getting one category right.

Define pass and fail before you look at any output. This is the step people skip, and skipping it poisons everything downstream. For each case, write down what makes an answer acceptable. Sometimes it is an exact value. More often it is a checklist: did it cite the right ticket, did it stay under the word limit, did it avoid promising a refund. Vague criteria give you vague evals, which are worse than none because they feel rigorous while measuring nothing.

Automate the scoring. You will run this hundreds of times, so it cannot be manual. Three approaches, roughly in order of how much you should trust them:

• Exact or programmatic checks: string match, regex, JSON schema validation, a number within tolerance. Cheap, fast, and never lies. Use these wherever the answer is checkable.
• Model-as-judge: another model grades the output against your criteria. Useful for fuzzy qualities like tone, but the judge has its own biases and needs its own sanity checks.
• Human review: the gold standard and the bottleneck. Reserve it for the cases automation cannot handle, and for periodically auditing whether your automated scores still agree with human judgment.

Iterate. An eval is not a document you write once. Every time the model surprises you in production, that failure becomes a new test case. Over a few months your eval becomes a precise portrait of your task's hard edges, and the moment a new model comes out you can answer the only question that matters, which is whether it is better at your job, in an afternoon instead of a quarter.

The payoff you do not expect

The obvious benefit is that you can compare models honestly. The bigger benefit sneaks up on you: the act of defining pass and fail forces you to say, in plain terms, what your product is actually supposed to do. Most teams cannot do this cleanly, and it shows in their prompts. Writing an eval is the most productive argument your team will have about what good even means.

You will never top a public leaderboard, and you should not want to. You want to top the only leaderboard your users vote on, which has one entry, and it is you. Build the eval, keep it close, and let the people selling benchmark numbers argue about whose model knows more physics.

People want one answer to "which AI is best," and there is not one. There is a best model for coding, a best for long documents, a best for cheap bulk work, and a best for running on your own hardware, and they are rarely the same model. Here is how I actually pick, in mid-2026, after using all four families on real work.

A context window is the amount of text a model can hold in its head at once, your prompt and its own reply combined, measured in tokens. For years that number was small enough to be a real constraint. Now vendors advertise a million tokens, sometimes more, roughly a stack of books. The pitch is that you can stop worrying about memory entirely. The reality is more interesting, and more annoying, than that.

What the window actually is

Everything the model knows in a given call has to fit in the window. There is no background memory, no notebook it flips back to. If a fact is not in the window, the model cannot use it, and if it is in the window, you paid for it. A million-token window means you can, in principle, drop an entire codebase or a year of email into a single request and ask a question about it. That part is real. You could not do it three years ago, and now you can.

What people hear, though, is "the model reads all million tokens as carefully as it reads a paragraph." It does not. Capacity and attention are different things, and the gap between them is where most of the disappointment lives.

Lost in the middle

The best-documented failure has a name: lost in the middle. Models pay the most attention to the start and the end of their context and get noticeably worse at anything buried in between. Plot the recall accuracy and you get a U-shape, strong at both ends, sagging in the middle. In needle-in-a-haystack tests, where a single fact is hidden in a long document, accuracy can drop by thirty points or more when that fact sits in the middle rather than near the edges.

It gets worse as the window fills. A Microsoft Research study found effective use of context falls to around 60 percent past 100,000 tokens. In plain terms, if you stuff 500,000 tokens into a prompt, the model is effectively ignoring or badly integrating a couple hundred thousand tokens' worth of it. The capacity is there on the spec sheet. The comprehension is not keeping pace.

So "put everything in the context and let the model sort it out" is a strategy that works right up until the answer depends on something in the murky middle. Then it fails quietly, which is the dangerous kind of failure, because the model does not announce that it skimmed.

The parts nobody puts on the slide

Then there is the bill and the clock. Every token in the window is a token you pay for and a token the model has to read before it says a word. A genuinely full million-token prompt can run north of ten dollars in input cost alone, and you may wait 30 seconds, sometimes past two minutes, before the first word of the answer appears. That prefill delay is not a bug, it is the model chewing through everything you gave it. For a background job, fine. For anything a person is waiting on, it is a dealbreaker.

This is the case against reaching for the giant window by default. If you can hand the model the ten relevant pages instead of the whole ten thousand, you get a faster answer, a cheaper answer, and often a more accurate one, because you have removed the haystack instead of asking the model to search it. This is a big part of why retrieval, pulling in just the passages that matter, has not been made obsolete by large windows the way some people predicted. Retrieval and a big window are tools for different jobs, not rivals.

When it earns its keep

Big windows genuinely shine in a few spots. Reasoning over a single long document where you cannot know in advance which part matters. First-pass exploration of an unfamiliar codebase. Analyzing one long transcript or contract end to end. Cases where the cost of missing a scattered detail is worse than the cost of latency and dollars. When the answer could depend on any part of a large whole, filling the window is the honest move.

The mistake is treating window size as a headline stat, the way phones once competed on megapixels. A million tokens is a real capability and a real convenience. It is not a memory upgrade that makes your data management problems disappear, and any pitch that implies otherwise is selling you the number, not the result. Use the whole window when the job actually needs the whole thing. The rest of the time, the smaller, sharper prompt wins, and it wins on every axis that shows up in your invoice.

Ask a developer if AI makes them faster and you get a confident yes. Measure it and the story gets complicated. That gap, between how fast people feel and how fast they actually are, is the most useful thing we have learned about coding assistants, and almost nobody talks about it because it flatters no one.

The study that ruined the vibe

In July 2025, METR ran the kind of trial this field badly needed. They took 16 experienced open-source developers, gave them 246 real tasks from mature repositories they already knew well, and randomly assigned each task to allow AI tools or forbid them. Not toy problems. Not a coding contest. Real issues in real codebases these people maintain.

The developers expected AI to make them about 24 percent faster. Afterward they believed it had made them about 20 percent faster. The stopwatch said they were 19 percent slower with the tools than without. They were slower and they walked away certain they had been quicker. Read that twice, because the perception gap is the real finding, not the headline number.

Why slower? On code you know cold, typing was never the bottleneck. The time went into reading the model's suggestions, judging them, correcting the ones that were subtly wrong, and steering it back onto a codebase with conventions it did not fully grasp. All of that felt productive. Reviewing generated code feels like work getting done. The clock disagreed.

I want to be fair to the tools here, because it is easy to swing too hard the other way. METR itself now flags the result as historical and says it does not necessarily describe today's models or workflows. The study used early-2025 tooling, mostly Cursor with Claude 3.5 and 3.7. It measured 16 senior people on their own turf, which is close to the worst case for an assistant. It is one careful study, not a law of nature. But it is real evidence, it was randomized, and it points the opposite direction from every vendor deck.

Where the surveys point

The big developer surveys tell a two-sided story that fits. Adoption is enormous and still climbing, most professional developers now use these tools regularly. But trust has been sliding. Stack Overflow's data has shown the share of developers who actually trust the accuracy of AI output falling even as usage rises, and a large chunk report that fixing almost-correct AI code is a real and recurring drain. People keep using it and keep getting burned by it. Both things are true at once.

Surveys measure feelings, and feelings are exactly what the METR study showed we cannot trust on this question. So hold them loosely. The signal worth keeping is the shape: heavy adoption, real enthusiasm, and a persistent undertow of "it looked right and was not."

Where it genuinely helps, and where it drags

Put the trial and the surveys next to your own week and a pattern shows up. The tools win when you know least and the work is most routine.

• Unfamiliar territory: a new language, a library you have never touched, boilerplate you would otherwise copy off the internet. Here the assistant is faster than you because your baseline is slow.
• Throwaway and first drafts: a script, a test scaffold, a regex, a quick prototype where being roughly right is enough and nobody has to maintain it.

It drags in the mirror image of those. Deep in a mature codebase you know well, where the constraints are subtle and the cost of a plausible-but-wrong line is high, the review tax outweighs the typing you saved. That is precisely the METR setup, and precisely where it lost.

The honest verdict

AI coding tools are real and they are useful, and they are not the raw speed multiplier the marketing implies, at least not the way most experienced people use them on code they own. The most reliable effect is not that they make you faster. It is that they make the work feel easier, lower friction, less staring at a blank file. That has genuine value. It is also exactly the feeling that fools you into thinking you sped up when you did not.

So use them where your baseline is weak and stay skeptical where it is strong. And when you feel fast, treat that feeling as a hunch, not a measurement. The one group that knows whether these tools are paying off is the group that actually timed it, and when they timed it, the answer surprised them. Until you time your own, you are guessing too.

Every model launch comes with a chart where the new model is tallest. The charts are technically true and practically useless, because they are marketing wearing a lab coat. Here is how to read a benchmark like a skeptic, so a leaderboard never again talks you into the wrong model for your actual work.

Turn on live captions during a quiet meeting with one clear speaker and you will think the problem is solved. Turn them on during a conference call with three people, a bad mic, and someone eating, and you will remember it is not. Speech-to-text has made a genuine leap in the last two years, and it has also hit a wall that no amount of marketing copy can talk past. Here is where the line actually sits, based on what these systems do outside the demo.

What works, and it is a lot

Clean audio in a common language is close to done. Whisper Large v3, the model most of the field is measured against, lands around a 2.7 percent word error rate on the clean LibriSpeech benchmark. In human terms that is roughly one wrong word in forty, which is better than most people type. If you are transcribing a well-recorded podcast, a single presenter with a decent microphone, or dictation in a quiet room, modern models will not be your problem.

Multilingual coverage is the other real win. Whisper handles ninety-nine languages, and while quality varies, the major ones are genuinely usable. A few years ago, transcription outside English was a research project. Now it is a checkbox. Batch processing, where the system has the whole recording and can take its time, is also strong. Deepgram's Nova-3 reports around 5.26 percent word error on batch audio, and for the common job of turning a recorded call into searchable text overnight, that is more than good enough.

Where it still falls apart

Now the wall. The headline numbers come from clean audio, and the real world is filthy.

One contact-center study told the whole story with three numbers from a single system: 92 percent accuracy on clean headset audio, dropping to 78 percent in a conference room, and collapsing to 65 percent on mobile calls with background noise. Same model, same language, same day. The only thing that changed was the room, and the room is exactly what you do not control. Background noise, crosstalk, and cheap microphones are not edge cases. For most real deployments they are the median case.

Accents remain a stubborn source of error, and it is a fairness problem as much as a technical one. Models trained mostly on standard American and British English degrade on strong regional accents and on second-language speakers, which means the people most likely to get a garbled transcript are often the ones already underserved by technology. This is getting better. It is not fixed.

Real-time is the other hard frontier. A model that scores beautifully in batch mode has to give up accuracy to run live, because it can no longer wait for the end of a sentence to decide what the beginning meant. Deepgram's own numbers show the gap, roughly 6.84 percent word error streaming against 5.26 percent batch. That spread is the price of being live, and for voice agents it is the difference between smooth and infuriating.

And then there is punctuation and speaker labeling, the unglamorous last mile. Getting the words right is one thing. Knowing where the sentence ends, who said it, and where the question mark goes is another, and it is where a technically accurate transcript still reads like a wall of mush.

What to actually pick

Match the tool to the job rather than the leaderboard.

• Offline transcription of recorded files, any budget: Whisper Large v3 is the sensible default, and if speed matters more than the last fraction of accuracy, Distil-Whisper runs about six times faster and stays within a point of it.
• Real-time voice agents and live captions: a streaming-first system like Deepgram Nova-3 will serve you better than forcing a batch model to go live.
• Noisy or multi-speaker audio: budget for cleanup and diarization as separate steps, and test on your actual recordings, not the vendor's samples.

The mistake I see most is trusting the benchmark number and skipping the test on your own audio. That gap between 92 and 65 percent is not hiding in a spec sheet. It is hiding in your particular room, your particular microphones, your particular speakers. Record ten real samples from the environment you actually care about, run them through two or three systems, and read the transcripts yourself. The winner is rarely the one with the prettiest published number, and you will know within an afternoon.

Read enough launch posts and you notice something strange: every new model is state of the art. All of them. Simultaneously. That cannot be true, and yet each chart is technically honest. Welcome to the benchmark wars, where nobody is exactly lying and almost everyone is being misled, gently, on purpose.

The reflex is to reach for the biggest model available and call it a day. It usually works, which is exactly why it is a trap. The frontier model is the safe default, not the right answer, and for a surprising share of real tasks a small model running on your own hardware does the job faster, cheaper, and without shipping your data anywhere. The interesting question is not whether small models can win. They can. It is how to know when yours will.

Where small actually wins

Four advantages, and they are concrete. Latency: a small model on a local GPU answers in tens of milliseconds with no network hop, which is the difference between an interface that feels alive and one that feels like waiting. Cost: the gap is not subtle. Analysts have put the swing at more than thirty times per token when you move a high-volume task off a frontier API onto a small model you host. At a million calls a day, that stops being a rounding error and starts being a line item someone asks about. Privacy: if the model runs on your machine, the data never leaves it, which for medical, legal, or internal work is not a nice-to-have but the whole ballgame. Focus: on a narrow, well-defined task, a small model tuned for it often matches or beats a giant generalist, because it is not spending capacity on the ten thousand things you will never ask.

That last point is the one people underrate. Microsoft's Phi-4, at 14 billion parameters, lands near models five to fifteen times its size on math and coding benchmarks. The old assumption that bigger simply means better broke somewhere in 2024, and it has not been true as a blanket rule since. A tightly scoped classifier, extractor, or router is precisely the shape of task where a small model shines.

How to actually test it

Do not trust the benchmark leaderboard and do not trust your gut. Both are measuring the wrong thing. The benchmark tests a generic task; you care about your task. Run the bake-off yourself, and it is less work than it sounds.

Build a small evaluation set from your real inputs. A hundred examples that look like production traffic beats ten thousand synthetic ones. For each, know what a good answer is, whether that is an exact label, an acceptable range, or a human judgment you write down in advance. Then run both models, the big one and the small candidate, over the identical set. Now you can compare the thing that matters: not "which is smarter" but "which is good enough for this," alongside cost per call and time per call.

Two habits make this honest. Use the big model as your reference for what "correct" looks like, but grade the small one against the task, not against the big one's exact wording. And look at the failures individually, not just the aggregate score. Small models tend to fail in clusters, on one input type or one edge case, and often you can fix a whole cluster with a better prompt or a few examples rather than reaching back for the giant.

When not to bother

I will save you some time. If the task is genuinely open-ended reasoning, long multi-step chains, hard code across a big context, nuanced writing that has to be right the first time, the frontier model still earns its price, and forcing a small model onto it is false economy. If your volume is low, a few hundred calls a day, the cost argument evaporates and the engineering time to host and maintain a local model is not worth recovering pennies. And if you do not have an evaluation set, you cannot tell whether the swap worked, so build that first or do not swap at all.

The right mental model is a portfolio, not a champion. Route the narrow, high-volume, latency-sensitive, or privacy-bound work to a small local model, and keep the frontier model for the hard cases where its extra capability actually shows up in your results. The teams getting real leverage out of this are not the ones who found the single best model. They are the ones who stopped asking that question and started matching the model to the job.