Reviews

Claude Fable 5 is the best model I have put on real work this year. It is also expensive enough to make you flinch, and it spent the second half of June switched off by a government order. All three of those things are true at once, which is the only honest way to review it.

Coding assistant comparisons usually happen on toy problems with clean answers, which is exactly why they are useless. So I did the opposite. I took one genuinely annoying real bug, the kind that spans a few files and does not announce itself, and pointed three assistants at it: Claude, GPT, and Microsoft new MAI coding model in Copilot. Here is how they actually did.

People want one answer to "which AI is best," and there is not one. There is a best model for coding, a best for long documents, a best for cheap bulk work, and a best for running on your own hardware, and they are rarely the same model. Here is how I actually pick, in mid-2026, after using all four families on real work.

Ask a developer if AI makes them faster and you get a confident yes. Measure it and the story gets complicated. That gap, between how fast people feel and how fast they actually are, is the most useful thing we have learned about coding assistants, and almost nobody talks about it because it flatters no one.

The study that ruined the vibe

In July 2025, METR ran the kind of trial this field badly needed. They took 16 experienced open-source developers, gave them 246 real tasks from mature repositories they already knew well, and randomly assigned each task to allow AI tools or forbid them. Not toy problems. Not a coding contest. Real issues in real codebases these people maintain.

The developers expected AI to make them about 24 percent faster. Afterward they believed it had made them about 20 percent faster. The stopwatch said they were 19 percent slower with the tools than without. They were slower and they walked away certain they had been quicker. Read that twice, because the perception gap is the real finding, not the headline number.

Why slower? On code you know cold, typing was never the bottleneck. The time went into reading the model's suggestions, judging them, correcting the ones that were subtly wrong, and steering it back onto a codebase with conventions it did not fully grasp. All of that felt productive. Reviewing generated code feels like work getting done. The clock disagreed.

I want to be fair to the tools here, because it is easy to swing too hard the other way. METR itself now flags the result as historical and says it does not necessarily describe today's models or workflows. The study used early-2025 tooling, mostly Cursor with Claude 3.5 and 3.7. It measured 16 senior people on their own turf, which is close to the worst case for an assistant. It is one careful study, not a law of nature. But it is real evidence, it was randomized, and it points the opposite direction from every vendor deck.

Where the surveys point

The big developer surveys tell a two-sided story that fits. Adoption is enormous and still climbing, most professional developers now use these tools regularly. But trust has been sliding. Stack Overflow's data has shown the share of developers who actually trust the accuracy of AI output falling even as usage rises, and a large chunk report that fixing almost-correct AI code is a real and recurring drain. People keep using it and keep getting burned by it. Both things are true at once.

Surveys measure feelings, and feelings are exactly what the METR study showed we cannot trust on this question. So hold them loosely. The signal worth keeping is the shape: heavy adoption, real enthusiasm, and a persistent undertow of "it looked right and was not."

Where it genuinely helps, and where it drags

Put the trial and the surveys next to your own week and a pattern shows up. The tools win when you know least and the work is most routine.

• Unfamiliar territory: a new language, a library you have never touched, boilerplate you would otherwise copy off the internet. Here the assistant is faster than you because your baseline is slow.
• Throwaway and first drafts: a script, a test scaffold, a regex, a quick prototype where being roughly right is enough and nobody has to maintain it.

It drags in the mirror image of those. Deep in a mature codebase you know well, where the constraints are subtle and the cost of a plausible-but-wrong line is high, the review tax outweighs the typing you saved. That is precisely the METR setup, and precisely where it lost.

The honest verdict

AI coding tools are real and they are useful, and they are not the raw speed multiplier the marketing implies, at least not the way most experienced people use them on code they own. The most reliable effect is not that they make you faster. It is that they make the work feel easier, lower friction, less staring at a blank file. That has genuine value. It is also exactly the feeling that fools you into thinking you sped up when you did not.

So use them where your baseline is weak and stay skeptical where it is strong. And when you feel fast, treat that feeling as a hunch, not a measurement. The one group that knows whether these tools are paying off is the group that actually timed it, and when they timed it, the answer surprised them. Until you time your own, you are guessing too.

Every AI demo lies a little. Not always on purpose, and not always a lot, but a demo is a performance, and performances are staged. The person on stage picked the example that works, ran it until it behaved, and cut the boring parts. That is not fraud, it is showmanship, and it is fine as long as you know you are watching a show and not a measurement. The trouble starts when you mistake one for the other and sign a contract on it.

I have sat through a lot of these, built a few, and been fooled by more than I would like to admit. The tricks are consistent enough that you can learn to spot them in real time.

The four moves to watch for

The most common one is the cherry-picked example. The prompt on stage is not a random input. It is the one that survived a hundred tries backstage, chosen precisely because the model nails it. A demo is one data point, presented as if it were the average. The task is not "can it do this once" but "how often does it do this," and the single triumphant run tells you nothing about the second number.

Then there is the hidden retry. On a prerecorded demo you are watching the take that worked, and you have no idea how many came before it. Even live, watch closely: a quiet reload, a rephrased prompt, a "let me try that again," and suddenly you are seeing attempt three sold as attempt one. If the presenter would not let you type your own prompt and accept whatever came out, ask yourself why.

The third is the latency edit. Real models pause. They think, they stream, they sometimes take fifteen seconds. Demo videos quietly cut that dead air, so the thing that takes ten seconds in your app looks instant on stage. It is one of the most common edits in a polished product video and one of the easiest to forget when you picture the model in your own workflow, where the wait is real and your users are watching a spinner.

The fourth is the staged input. The document is clean, the question is unambiguous, the data is perfectly formatted. Your Tuesday is a smudged scan, a half-finished sentence, and a spreadsheet with three header rows. Demos run on inputs shaped to fit the model. Production runs on whatever walks in the door, and the gap between those two is where most disappointment lives.

The questions that cut through it

You do not need to be a skeptic about the technology to be a skeptic about the demo. You just need to ask the questions the staging was designed to keep you from asking.

• Can I type my own prompt, right now, and we keep whatever comes out?
• How many times did you run this before it worked, and what is the success rate over, say, fifty tries?
• Is this real time, and if it is a video, was the timing edited?
• Show me it failing. What does a bad case look like, and how does the product handle it?

That last one is the tell. A team with a real product will happily show you a failure, because they have thought hard about the unhappy path and built for it. A team selling a demo will deflect, change the subject, or insist failures are rare, which is the answer of someone who has not looked. The willingness to show you the ugly case is the single strongest signal that there is something solid behind the polish.

None of this means demos are worthless. A good one shows you what is possible on a great day, and knowing the ceiling is genuinely useful. Just remember that you buy the floor, not the ceiling. The question that actually matters is not "how good is it when everything goes right" but "how good is it when everything goes normal," and no stage demo will ever answer that for you. You find that out by putting your own messy inputs in front of it, watching where it breaks, and deciding whether you can live with the breaks. Everything before that step is a trailer, and you would not review a movie off the trailer.

There is a specific moment where Gemini 3.1 Pro stops being one option and becomes the obvious one: the moment you drop a 400-page document or a video into the chat and ask it to make sense of the whole thing. For that job, nothing in my rotation touches it.

Gemma 4 12B is not going to top any leaderboard, and reviewing it against the frontier would miss the point entirely. The question that matters is different: how much useful AI can you now run entirely on your own laptop, with the internet unplugged? The answer, it turns out, is a surprising amount.

Reviewing GPT-5.6 is a bit like reviewing a reliable mid-size sedan. There is not much drama here, and that is sort of the point. It does almost everything competently, nothing spectacularly, and it will not embarrass you. For a huge number of people, that is exactly the right model.

Grok Imagine Video 1.5 is the kind of tool that looks incredible for the first ten seconds and complicated after the first hour. It generates video with native audio, roughly twice as fast as the last version, and it will happily produce something jaw-dropping right before it produces something faintly cursed.

Kimi K2 belongs to my favorite category of model: the one that makes you recheck the pricing page because the numbers look like a typo. Near-frontier reasoning, notably cheaper than the big names, and it gets to the answer using fewer tokens than its predecessor. Mostly, it holds up. Mostly.

Llama 4 Scout is easy to review badly. You quote the ten-million-token context window, call it revolutionary, and move on. Having actually run it, the honest review is more interesting, and more useful: Scout is a very good open model whose best feature is not the one on the box.

Microsoft did something quietly significant: it dropped its own MAI-Code-1-Flash model into GitHub Copilot, the coding assistant a lot of us use every day. This is a review of how it actually feels to code with, not how it looks in a keynote, because those are very different things.

The open-versus-closed argument gets treated like a team you join, complete with jerseys. It is not. It is a trade-off you make one job at a time, and the same person can sensibly land on opposite answers for two projects in the same week. What follows is a decision guide, not a leaderboard, because the leaderboard changes monthly and the trade-off underneath it does not.

What each side actually gives you

Open-weights models, the Llamas and DeepSeeks and Qwens of the world, hand you the actual model. You can download it, run it on your own hardware or a rented box, look at what it does, fine-tune it on your data, and keep the whole thing behind your own firewall. Nobody meters your calls. Nobody can change the model out from under you or deprecate it next quarter. If your data cannot legally or comfortably leave your walls, this is often the only real option.

Closed models, reached through an API, hand you a result. You send text, you get text back, and someone else owns the running of it. In exchange for giving up control you get the current frontier of quality, no infrastructure to babysit, and a model that quietly improves without you lifting a finger. For a lot of teams that is the entire pitch and it is a good one: you want the answer, not a second job running GPUs.

The costs hide in different places

People compare these on the wrong axis. They look at the per-token price of a closed API, see a number bigger than zero, and conclude that self-hosting an open model is cheaper. Sometimes. The closed price includes the hardware, the scaling, the uptime, and the salaries of people who keep it running. Open weights are free to download and very much not free to operate. You are now buying or renting GPUs, and someone on your team owns keeping the thing up at 3 a.m.

The honest version is about volume and steadiness. Spiky, low, or unpredictable traffic almost always favors the closed API, because you pay only for what you use and nothing when you are idle. Heavy, steady, round-the-clock traffic is where owning the hardware can win, because a machine you have already paid for does not care how many calls you push through it. The crossover is real, but it sits much further out than the sticker-price comparison suggests, and it moves every time GPU rental prices or token prices shift.

How to actually choose

Skip the identity and answer a few blunt questions about the specific job.

• Where is the data allowed to go? If it legally cannot leave your infrastructure, that decides it before any quality debate starts. Open weights, self-hosted, done.
• Do you need the absolute top of the quality range? For the hardest reasoning, the newest capabilities, the widest language coverage, the closed frontier models still tend to lead, and the gap is often worth paying for. For a well-scoped task, a mid-size open model may clear the bar with room to spare and cost far less to run.
• Does the model changing under you break things? If you have tuned prompts against exact behavior and cannot afford a silent update, a weights file you pin and control has an edge a hosted endpoint cannot match.
• Do you have people to run it? Serving a model in production is real, ongoing engineering. If that team does not exist, the API is not a compromise, it is the sane choice.

The gap keeps closing, the choice does not

The genuinely good news is that open weights have gotten shockingly close to the closed frontier on many everyday tasks. For summarizing, extraction, classification, ordinary chat, drafting, the practical difference is often small enough not to matter, and it keeps shrinking. That is a real shift and worth being cheerful about.

It does not, however, dissolve the trade-off. Control and privacy and the ops burden that comes with them sit on one side. Convenience and frontier quality and someone else's pager sit on the other. That tension is structural. It will still be here when today's model names are forgotten. The people who get the most out of AI are not the ones who picked a side and defended it. They are the ones who ask, for this specific job, which set of headaches they would rather have, and then pick accordingly.

Turn on live captions during a quiet meeting with one clear speaker and you will think the problem is solved. Turn them on during a conference call with three people, a bad mic, and someone eating, and you will remember it is not. Speech-to-text has made a genuine leap in the last two years, and it has also hit a wall that no amount of marketing copy can talk past. Here is where the line actually sits, based on what these systems do outside the demo.

What works, and it is a lot

Clean audio in a common language is close to done. Whisper Large v3, the model most of the field is measured against, lands around a 2.7 percent word error rate on the clean LibriSpeech benchmark. In human terms that is roughly one wrong word in forty, which is better than most people type. If you are transcribing a well-recorded podcast, a single presenter with a decent microphone, or dictation in a quiet room, modern models will not be your problem.

Multilingual coverage is the other real win. Whisper handles ninety-nine languages, and while quality varies, the major ones are genuinely usable. A few years ago, transcription outside English was a research project. Now it is a checkbox. Batch processing, where the system has the whole recording and can take its time, is also strong. Deepgram's Nova-3 reports around 5.26 percent word error on batch audio, and for the common job of turning a recorded call into searchable text overnight, that is more than good enough.

Where it still falls apart

Now the wall. The headline numbers come from clean audio, and the real world is filthy.

One contact-center study told the whole story with three numbers from a single system: 92 percent accuracy on clean headset audio, dropping to 78 percent in a conference room, and collapsing to 65 percent on mobile calls with background noise. Same model, same language, same day. The only thing that changed was the room, and the room is exactly what you do not control. Background noise, crosstalk, and cheap microphones are not edge cases. For most real deployments they are the median case.

Accents remain a stubborn source of error, and it is a fairness problem as much as a technical one. Models trained mostly on standard American and British English degrade on strong regional accents and on second-language speakers, which means the people most likely to get a garbled transcript are often the ones already underserved by technology. This is getting better. It is not fixed.

Real-time is the other hard frontier. A model that scores beautifully in batch mode has to give up accuracy to run live, because it can no longer wait for the end of a sentence to decide what the beginning meant. Deepgram's own numbers show the gap, roughly 6.84 percent word error streaming against 5.26 percent batch. That spread is the price of being live, and for voice agents it is the difference between smooth and infuriating.

And then there is punctuation and speaker labeling, the unglamorous last mile. Getting the words right is one thing. Knowing where the sentence ends, who said it, and where the question mark goes is another, and it is where a technically accurate transcript still reads like a wall of mush.

What to actually pick

Match the tool to the job rather than the leaderboard.

• Offline transcription of recorded files, any budget: Whisper Large v3 is the sensible default, and if speed matters more than the last fraction of accuracy, Distil-Whisper runs about six times faster and stays within a point of it.
• Real-time voice agents and live captions: a streaming-first system like Deepgram Nova-3 will serve you better than forcing a batch model to go live.
• Noisy or multi-speaker audio: budget for cleanup and diarization as separate steps, and test on your actual recordings, not the vendor's samples.

The mistake I see most is trusting the benchmark number and skipping the test on your own audio. That gap between 92 and 65 percent is not hiding in a spec sheet. It is hiding in your particular room, your particular microphones, your particular speakers. Record ten real samples from the environment you actually care about, run them through two or three systems, and read the transcripts yourself. The winner is rarely the one with the prettiest published number, and you will know within an afternoon.

Everyone reviews the frontier. Almost nobody carefully reviews the budget shelf, which is a shame, because that is where most real work should actually run. I spent time putting the cheap models through the same ordinary tasks I use every day. The winner was not the one I expected, and the losers were instructive.

The reflex is to reach for the biggest model available and call it a day. It usually works, which is exactly why it is a trap. The frontier model is the safe default, not the right answer, and for a surprising share of real tasks a small model running on your own hardware does the job faster, cheaper, and without shipping your data anywhere. The interesting question is not whether small models can win. They can. It is how to know when yours will.

Where small actually wins

Four advantages, and they are concrete. Latency: a small model on a local GPU answers in tens of milliseconds with no network hop, which is the difference between an interface that feels alive and one that feels like waiting. Cost: the gap is not subtle. Analysts have put the swing at more than thirty times per token when you move a high-volume task off a frontier API onto a small model you host. At a million calls a day, that stops being a rounding error and starts being a line item someone asks about. Privacy: if the model runs on your machine, the data never leaves it, which for medical, legal, or internal work is not a nice-to-have but the whole ballgame. Focus: on a narrow, well-defined task, a small model tuned for it often matches or beats a giant generalist, because it is not spending capacity on the ten thousand things you will never ask.

That last point is the one people underrate. Microsoft's Phi-4, at 14 billion parameters, lands near models five to fifteen times its size on math and coding benchmarks. The old assumption that bigger simply means better broke somewhere in 2024, and it has not been true as a blanket rule since. A tightly scoped classifier, extractor, or router is precisely the shape of task where a small model shines.

How to actually test it

Do not trust the benchmark leaderboard and do not trust your gut. Both are measuring the wrong thing. The benchmark tests a generic task; you care about your task. Run the bake-off yourself, and it is less work than it sounds.

Build a small evaluation set from your real inputs. A hundred examples that look like production traffic beats ten thousand synthetic ones. For each, know what a good answer is, whether that is an exact label, an acceptable range, or a human judgment you write down in advance. Then run both models, the big one and the small candidate, over the identical set. Now you can compare the thing that matters: not "which is smarter" but "which is good enough for this," alongside cost per call and time per call.

Two habits make this honest. Use the big model as your reference for what "correct" looks like, but grade the small one against the task, not against the big one's exact wording. And look at the failures individually, not just the aggregate score. Small models tend to fail in clusters, on one input type or one edge case, and often you can fix a whole cluster with a better prompt or a few examples rather than reaching back for the giant.

When not to bother

I will save you some time. If the task is genuinely open-ended reasoning, long multi-step chains, hard code across a big context, nuanced writing that has to be right the first time, the frontier model still earns its price, and forcing a small model onto it is false economy. If your volume is low, a few hundred calls a day, the cost argument evaporates and the engineering time to host and maintain a local model is not worth recovering pennies. And if you do not have an evaluation set, you cannot tell whether the swap worked, so build that first or do not swap at all.

The right mental model is a portfolio, not a champion. Route the narrow, high-volume, latency-sensitive, or privacy-bound work to a small local model, and keep the frontier model for the hard cases where its extra capability actually shows up in your results. The teams getting real leverage out of this are not the ones who found the single best model. They are the ones who stopped asking that question and started matching the model to the job.