AI Opinion & Analysis
Every AI demo lies a little. Not always on purpose, and not always a lot, but a demo is a performance, and performances are staged. The person on stage picked the example that works, ran it until it behaved, and cut the boring parts. That is not fraud, it is showmanship, and it is fine as long as you know you are watching a show and not a measurement. The trouble starts when you mistake one for the other and sign a contract on it.
I have sat through a lot of these, built a few, and been fooled by more than I would like to admit. The tricks are consistent enough that you can learn to spot them in real time.
The four moves to watch for
The most common one is the cherry-picked example. The prompt on stage is not a random input. It is the one that survived a hundred tries backstage, chosen precisely because the model nails it. A demo is one data point, presented as if it were the average. The task is not "can it do this once" but "how often does it do this," and the single triumphant run tells you nothing about the second number.
Then there is the hidden retry. On a prerecorded demo you are watching the take that worked, and you have no idea how many came before it. Even live, watch closely: a quiet reload, a rephrased prompt, a "let me try that again," and suddenly you are seeing attempt three sold as attempt one. If the presenter would not let you type your own prompt and accept whatever came out, ask yourself why.
The third is the latency edit. Real models pause. They think, they stream, they sometimes take fifteen seconds. Demo videos quietly cut that dead air, so the thing that takes ten seconds in your app looks instant on stage. It is one of the most common edits in a polished product video and one of the easiest to forget when you picture the model in your own workflow, where the wait is real and your users are watching a spinner.
The fourth is the staged input. The document is clean, the question is unambiguous, the data is perfectly formatted. Your Tuesday is a smudged scan, a half-finished sentence, and a spreadsheet with three header rows. Demos run on inputs shaped to fit the model. Production runs on whatever walks in the door, and the gap between those two is where most disappointment lives.
The questions that cut through it
You do not need to be a skeptic about the technology to be a skeptic about the demo. You just need to ask the questions the staging was designed to keep you from asking.
- Can I type my own prompt, right now, and we keep whatever comes out?
- How many times did you run this before it worked, and what is the success rate over, say, fifty tries?
- Is this real time, and if it is a video, was the timing edited?
- Show me it failing. What does a bad case look like, and how does the product handle it?
That last one is the tell. A team with a real product will happily show you a failure, because they have thought hard about the unhappy path and built for it. A team selling a demo will deflect, change the subject, or insist failures are rare, which is the answer of someone who has not looked. The willingness to show you the ugly case is the single strongest signal that there is something solid behind the polish.
None of this means demos are worthless. A good one shows you what is possible on a great day, and knowing the ceiling is genuinely useful. Just remember that you buy the floor, not the ceiling. The question that actually matters is not "how good is it when everything goes right" but "how good is it when everything goes normal," and no stage demo will ever answer that for you. You find that out by putting your own messy inputs in front of it, watching where it breaks, and deciding whether you can live with the breaks. Everything before that step is a trailer, and you would not review a movie off the trailer.
For about two years, "prompt engineer" was a job you could put on a resume and watch people's eyebrows go up. Six-figure listings, breathless threads about the magic phrase that cracked GPT open, whole courses selling the secret incantations. I want to be careful here, because the backlash overcorrected. Writing good prompts is a genuine skill and it will stay one. It is just not a career, and pretending otherwise did nobody any favors.
Here is the distinction that got flattened. A skill is something you use inside a job. A career is the job. Spreadsheet formulas are a skill. "Excel engineer" was never a role, even in the years when knowing Excel cold made you far more valuable than the person next to you. Prompting is heading down the same road, and quickly.
The tricks are decaying on purpose
Think about what the famous prompt hacks actually were. "Let us think step by step." "You are a world-class expert." "I will tip you 200 dollars." "Take a deep breath." These worked because early models needed a nudge to behave, and people found the nudges by trial and error. They were patches for the model's weaknesses.
But every one of those weaknesses is a bug the labs are actively fixing. Reasoning models now think step by step without being asked, because that behavior was trained in. Newer models shrug off the fake tip and the flattery because they were tuned to ignore that kind of manipulation. The clever phrase that earned you a screenshot last year does nothing this year, and that is not an accident. The whole direction of model development is to understand what you meant, so you have to say less. A skill built entirely on the model failing to understand you has a short shelf life by design.
I have watched people memorize prompt templates the way you would memorize keyboard shortcuts, and then act surprised when a model update makes half of them pointless. The templates were never the skill.
What actually survives
Strip away the incantations and look at what makes one person consistently get better results than another. It is not secret words. It is that they can state what they want. They know what a good answer looks like before they ask. They give the relevant context and leave out the noise. They notice when an output is subtly wrong and can say why. They break a fuzzy goal into parts a machine can actually chew on.
Notice that none of that is about the model. That is editing, specification, and clear thinking, the same muscles a good manager uses to brief a new hire or a good writer uses to structure an argument. Those skills were valuable before language models and they will be valuable after, because they are about knowing your own mind well enough to hand the intent to someone, or something, else.
The parts of prompting that do stay technical are drifting toward normal engineering anyway. Structuring context, wiring up tools, chaining calls, evaluating outputs at scale, managing what goes in the window. That work is real and growing, but it looks like software work, and the people doing it call themselves engineers, not prompters.
So what should you do with this
Learn to prompt well. Seriously. Anyone who uses these tools daily should get fluent, the same way anyone in an office should be able to write a clear email. Fluency compounds and it is cheap to acquire. Just do not build an identity on it, and do not pay a lot for a certificate in it.
If you want the durable version of this skill, practice saying exactly what you mean to a smart colleague who takes you completely literally and has no idea what is in your head. That is most of prompting, and it happens to be most of thinking clearly, which was always the thing worth getting good at. The models will keep meeting you halfway. Your job is to have something worth meeting them with.
Here is an uncomfortable result that deserves more airtime than it gets. In 2025 the research group METR ran a randomized trial with experienced open-source developers on real issues from repositories they already knew well. With AI tools allowed, they finished tasks 19 percent slower. Afterward, the same developers estimated that AI had made them about 20 percent faster. They were slower and felt faster, by nearly the same margin. That gap is the whole story, and it is worth sitting with.
Why fast can feel faster than it is
The researchers pointed at reduced cognitive effort. AI-assisted work felt easier, and we quietly file easier under faster even when the clock disagrees. Watching a model produce a wall of plausible code feels like progress in a way that staring at a blank editor does not, and the feeling is real even when the output is not saving you anything.
The catch is what the feeling hides. You did not write the code, so now you have to read it, and reading someone else's code closely enough to trust it is not free. If it is wrong in a subtle way, you pay twice: once to spot the problem and again to fix it, often after you have already convinced yourself it was fine. The generation was fast. The verification was not, and verification is the part that does not show up in the demo.
Before I am accused of doom: I should note METR themselves later flagged that their study design had problems, partly because the developers who benefit most from AI would not join a no-AI condition even at 50 dollars an hour. So do not read 19 percent as a law of nature. Read it as a real, measured case where the tool that everyone assumed was a speedup was not, and the users could not tell. That is the part that generalizes.
Where the time actually leaks
The productivity leaks are boring and specific, which is exactly why they are easy to miss.
- Reviewing confident nonsense. The failure mode that costs the most is a wrong answer delivered with total assurance: a made-up function that looks real, a plausible statute that does not exist, a config flag that was never a flag. Confident and wrong is more expensive than obviously broken, because obviously broken you catch in a second and confident-wrong you ship.
- Context-switching. Every trip out to the tool and back is a small tax on your attention, and enough small taxes add up to a workday where you were busy and moved nothing.
- The almost-right rabbit hole. The model gets you 80 percent of the way, and you spend longer chasing the last 20 percent through its logic than you would have spent writing the thing yourself. Nudging a nearly-correct output into a correct one can cost more than starting clean.
None of these feel like waste in the moment. They feel like work. That is the trap.
It still works, when you aim it right
I am not telling you to put the tools down. I use them daily and would be slower without them, which is precisely why I take the paradox seriously instead of waving it off. The tools genuinely help, but the win is conditional, and the conditions are learnable.
They shine when verification is cheap or the stakes are low. Boilerplate you can eyeball in seconds. A first draft you were going to heavily rewrite anyway. A language or API you half-remember, where the model jogs your memory faster than the docs. Throwaway scripts. Anything where being roughly right is good enough and checking is quick. In those spots the speedup is real and often large.
They quietly cost you when verification is expensive and correctness is non-negotiable. Subtle logic in unfamiliar code. Anything security-sensitive. Domains where you cannot easily tell right from wrong-but-plausible, which is exactly where the model's confidence is most dangerous, because you have no cheap way to check it.
The skill, and it is a skill, is noticing which situation you are in before you reach for the tool, not after. The developers in that study were not fools. They were experienced people who genuinely could not feel the slowdown while it was happening. That is the real lesson. The tool is not the problem and neither are you. The problem is that speed and the feeling of speed have come apart, and the only fix is to occasionally check the clock instead of trusting the vibe.
Every few months the internet holds another referendum on artificial general intelligence. Are we two years away, ten years away, or is the whole idea a mirage? People with real credentials line up on both sides and yell past each other. It is a fun argument. It is also, for almost everyone actually using these tools, a waste of a good afternoon.
Nobody agrees what the word means
Start with the obvious problem: AGI has no agreed definition. Ask ten researchers and you get ten thresholds. Human-level performance on most economically valuable work. The ability to learn any task a person can. Something that can do original science. A system that beats experts across every benchmark we can invent. These are not variations on one idea. They are different ideas wearing the same three letters.
An argument where the central term means something different to each participant is not a disagreement about the world. It is a disagreement about vocabulary, and those never resolve, because there is no fact that could settle them. You can watch a debate run for an hour and realize at the end that one person was talking about job displacement and the other was talking about consciousness, and they both thought they were talking about the same thing.
When a target keeps moving, hitting it stops meaning anything. Systems now do things that would have counted as AGI to a researcher in 2015: hold a coherent conversation, pass the bar exam, write working code from a description. We did not throw a party. We shrugged and moved the line. That reflex tells you the concept is doing rhetorical work, not descriptive work.
The questions that actually pay rent
Here is what the AGI argument crowds out. Is this tool useful for the thing I need done? Is it reliable enough that I do not have to check every output by hand? Can I afford to run it at the scale I need? Those three questions decide whether AI helps you this quarter, and none of them require anyone to agree on when or whether the machines wake up.
They are also answerable. You can measure whether a coding assistant saves your team time. You can count how often a support bot hands back a wrong answer. You can read a pricing page and do arithmetic. These are boring, concrete, checkable things, which is exactly why they get less airtime than the cosmic stuff. Nobody writes a viral thread about "we ran it for three weeks and it was fine, mostly."
The gap between capability and reliability is where all the real action sits, and AGI talk skips right over it. A model that can ace a graduate exam can also state a made-up statute with total confidence. Both facts are true at once. The exciting frontier and the boring failure live in the same box, and if you are deploying anything, the boring failure is the part that ends up in your incident report.
Skeptical is not the same as cynical
I want to be careful here, because dismissing AGI talk can slide into dismissing AI, and that is not the point. These tools are genuinely useful. I use them every day and would be annoyed to lose them. The point is narrower: whether they eventually become general in some philosophical sense has almost no bearing on whether they are worth your money and attention right now.
Treat the far-future question as a hobby, not a strategy. It is fine to find it fascinating. Read the essays, argue at dinner, place your bets. Just do not let it stand in for the work of evaluating what is actually in front of you, because the two have almost nothing to do with each other. A team that spends its meetings debating timelines is a team not testing the tool on its own data.
The honest position is unsatisfying and I will say it anyway: I do not know when or whether AGI arrives, and neither does anyone selling you a strong opinion about it. What I do know is whether the model in front of me got the invoice numbers right, and that is the question that pays my bills. When someone wants to argue about superintelligence, I am happy to listen. Then I go back to checking whether the thing works, because that is the only argument with an answer I can use.
Open almost any app you used happily a year ago and there is a new little sparkle icon in the corner, promising an AI assistant you did not ask for and probably will not use. The bolt-on chatbot is the defining product decision of the year, and I think it is quietly making a lot of software worse. I say this as someone who loves the underlying technology.
Read more: What we lose when every product bolts on a chatbot