Benchmarks

People want one answer to "which AI is best," and there is not one. There is a best model for coding, a best for long documents, a best for cheap bulk work, and a best for running on your own hardware, and they are rarely the same model. Here is how I actually pick, in mid-2026, after using all four families on real work.

A context window is the amount of text a model can hold in its head at once, your prompt and its own reply combined, measured in tokens. For years that number was small enough to be a real constraint. Now vendors advertise a million tokens, sometimes more, roughly a stack of books. The pitch is that you can stop worrying about memory entirely. The reality is more interesting, and more annoying, than that.

What the window actually is

Everything the model knows in a given call has to fit in the window. There is no background memory, no notebook it flips back to. If a fact is not in the window, the model cannot use it, and if it is in the window, you paid for it. A million-token window means you can, in principle, drop an entire codebase or a year of email into a single request and ask a question about it. That part is real. You could not do it three years ago, and now you can.

What people hear, though, is "the model reads all million tokens as carefully as it reads a paragraph." It does not. Capacity and attention are different things, and the gap between them is where most of the disappointment lives.

Lost in the middle

The best-documented failure has a name: lost in the middle. Models pay the most attention to the start and the end of their context and get noticeably worse at anything buried in between. Plot the recall accuracy and you get a U-shape, strong at both ends, sagging in the middle. In needle-in-a-haystack tests, where a single fact is hidden in a long document, accuracy can drop by thirty points or more when that fact sits in the middle rather than near the edges.

It gets worse as the window fills. A Microsoft Research study found effective use of context falls to around 60 percent past 100,000 tokens. In plain terms, if you stuff 500,000 tokens into a prompt, the model is effectively ignoring or badly integrating a couple hundred thousand tokens' worth of it. The capacity is there on the spec sheet. The comprehension is not keeping pace.

So "put everything in the context and let the model sort it out" is a strategy that works right up until the answer depends on something in the murky middle. Then it fails quietly, which is the dangerous kind of failure, because the model does not announce that it skimmed.

The parts nobody puts on the slide

Then there is the bill and the clock. Every token in the window is a token you pay for and a token the model has to read before it says a word. A genuinely full million-token prompt can run north of ten dollars in input cost alone, and you may wait 30 seconds, sometimes past two minutes, before the first word of the answer appears. That prefill delay is not a bug, it is the model chewing through everything you gave it. For a background job, fine. For anything a person is waiting on, it is a dealbreaker.

This is the case against reaching for the giant window by default. If you can hand the model the ten relevant pages instead of the whole ten thousand, you get a faster answer, a cheaper answer, and often a more accurate one, because you have removed the haystack instead of asking the model to search it. This is a big part of why retrieval, pulling in just the passages that matter, has not been made obsolete by large windows the way some people predicted. Retrieval and a big window are tools for different jobs, not rivals.

When it earns its keep

Big windows genuinely shine in a few spots. Reasoning over a single long document where you cannot know in advance which part matters. First-pass exploration of an unfamiliar codebase. Analyzing one long transcript or contract end to end. Cases where the cost of missing a scattered detail is worse than the cost of latency and dollars. When the answer could depend on any part of a large whole, filling the window is the honest move.

The mistake is treating window size as a headline stat, the way phones once competed on megapixels. A million tokens is a real capability and a real convenience. It is not a memory upgrade that makes your data management problems disappear, and any pitch that implies otherwise is selling you the number, not the result. Use the whole window when the job actually needs the whole thing. The rest of the time, the smaller, sharper prompt wins, and it wins on every axis that shows up in your invoice.

Every model launch comes with a chart where the new model is tallest. The charts are technically true and practically useless, because they are marketing wearing a lab coat. Here is how to read a benchmark like a skeptic, so a leaderboard never again talks you into the wrong model for your actual work.

Read enough launch posts and you notice something strange: every new model is state of the art. All of them. Simultaneously. That cannot be true, and yet each chart is technically honest. Welcome to the benchmark wars, where nobody is exactly lying and almost everyone is being misled, gently, on purpose.

The reflex is to reach for the biggest model available and call it a day. It usually works, which is exactly why it is a trap. The frontier model is the safe default, not the right answer, and for a surprising share of real tasks a small model running on your own hardware does the job faster, cheaper, and without shipping your data anywhere. The interesting question is not whether small models can win. They can. It is how to know when yours will.

Where small actually wins

Four advantages, and they are concrete. Latency: a small model on a local GPU answers in tens of milliseconds with no network hop, which is the difference between an interface that feels alive and one that feels like waiting. Cost: the gap is not subtle. Analysts have put the swing at more than thirty times per token when you move a high-volume task off a frontier API onto a small model you host. At a million calls a day, that stops being a rounding error and starts being a line item someone asks about. Privacy: if the model runs on your machine, the data never leaves it, which for medical, legal, or internal work is not a nice-to-have but the whole ballgame. Focus: on a narrow, well-defined task, a small model tuned for it often matches or beats a giant generalist, because it is not spending capacity on the ten thousand things you will never ask.

That last point is the one people underrate. Microsoft's Phi-4, at 14 billion parameters, lands near models five to fifteen times its size on math and coding benchmarks. The old assumption that bigger simply means better broke somewhere in 2024, and it has not been true as a blanket rule since. A tightly scoped classifier, extractor, or router is precisely the shape of task where a small model shines.

How to actually test it

Do not trust the benchmark leaderboard and do not trust your gut. Both are measuring the wrong thing. The benchmark tests a generic task; you care about your task. Run the bake-off yourself, and it is less work than it sounds.

Build a small evaluation set from your real inputs. A hundred examples that look like production traffic beats ten thousand synthetic ones. For each, know what a good answer is, whether that is an exact label, an acceptable range, or a human judgment you write down in advance. Then run both models, the big one and the small candidate, over the identical set. Now you can compare the thing that matters: not "which is smarter" but "which is good enough for this," alongside cost per call and time per call.

Two habits make this honest. Use the big model as your reference for what "correct" looks like, but grade the small one against the task, not against the big one's exact wording. And look at the failures individually, not just the aggregate score. Small models tend to fail in clusters, on one input type or one edge case, and often you can fix a whole cluster with a better prompt or a few examples rather than reaching back for the giant.

When not to bother

I will save you some time. If the task is genuinely open-ended reasoning, long multi-step chains, hard code across a big context, nuanced writing that has to be right the first time, the frontier model still earns its price, and forcing a small model onto it is false economy. If your volume is low, a few hundred calls a day, the cost argument evaporates and the engineering time to host and maintain a local model is not worth recovering pennies. And if you do not have an evaluation set, you cannot tell whether the swap worked, so build that first or do not swap at all.

The right mental model is a portfolio, not a champion. Route the narrow, high-volume, latency-sensitive, or privacy-bound work to a small local model, and keep the frontier model for the hard cases where its extra capability actually shows up in your results. The teams getting real leverage out of this are not the ones who found the single best model. They are the ones who stopped asking that question and started matching the model to the job.