Every AI demo lies a little. Not always on purpose, and not always a lot, but a demo is a performance, and performances are staged. The person on stage picked the example that works, ran it until it behaved, and cut the boring parts. That is not fraud, it is showmanship, and it is fine as long as you know you are watching a show and not a measurement. The trouble starts when you mistake one for the other and sign a contract on it.
I have sat through a lot of these, built a few, and been fooled by more than I would like to admit. The tricks are consistent enough that you can learn to spot them in real time.
The four moves to watch for
The most common one is the cherry-picked example. The prompt on stage is not a random input. It is the one that survived a hundred tries backstage, chosen precisely because the model nails it. A demo is one data point, presented as if it were the average. The task is not "can it do this once" but "how often does it do this," and the single triumphant run tells you nothing about the second number.
Then there is the hidden retry. On a prerecorded demo you are watching the take that worked, and you have no idea how many came before it. Even live, watch closely: a quiet reload, a rephrased prompt, a "let me try that again," and suddenly you are seeing attempt three sold as attempt one. If the presenter would not let you type your own prompt and accept whatever came out, ask yourself why.
The third is the latency edit. Real models pause. They think, they stream, they sometimes take fifteen seconds. Demo videos quietly cut that dead air, so the thing that takes ten seconds in your app looks instant on stage. It is one of the most common edits in a polished product video and one of the easiest to forget when you picture the model in your own workflow, where the wait is real and your users are watching a spinner.
The fourth is the staged input. The document is clean, the question is unambiguous, the data is perfectly formatted. Your Tuesday is a smudged scan, a half-finished sentence, and a spreadsheet with three header rows. Demos run on inputs shaped to fit the model. Production runs on whatever walks in the door, and the gap between those two is where most disappointment lives.
The questions that cut through it
You do not need to be a skeptic about the technology to be a skeptic about the demo. You just need to ask the questions the staging was designed to keep you from asking.
- Can I type my own prompt, right now, and we keep whatever comes out?
- How many times did you run this before it worked, and what is the success rate over, say, fifty tries?
- Is this real time, and if it is a video, was the timing edited?
- Show me it failing. What does a bad case look like, and how does the product handle it?
That last one is the tell. A team with a real product will happily show you a failure, because they have thought hard about the unhappy path and built for it. A team selling a demo will deflect, change the subject, or insist failures are rare, which is the answer of someone who has not looked. The willingness to show you the ugly case is the single strongest signal that there is something solid behind the polish.
None of this means demos are worthless. A good one shows you what is possible on a great day, and knowing the ceiling is genuinely useful. Just remember that you buy the floor, not the ceiling. The question that actually matters is not "how good is it when everything goes right" but "how good is it when everything goes normal," and no stage demo will ever answer that for you. You find that out by putting your own messy inputs in front of it, watching where it breaks, and deciding whether you can live with the breaks. Everything before that step is a trailer, and you would not review a movie off the trailer.