Ask a language model a question it has no business answering and watch what happens. It answers. Fluently, in complete sentences, with the same even tone it uses for the things it actually knows. The confidence does not move. That gap, between how sure the model sounds and how sure it should be, is the single most dangerous thing about the current generation of these tools.
People call this hallucination and treat it like a bug that a bigger model will fix. It is not a bug. It is baked into how the models are trained.
We taught them to guess
Think about how a model is graded. On most benchmarks a correct answer scores a point, a wrong answer scores zero, and "I do not know" also scores zero. Line those incentives up and the math is brutal: if abstaining pays the same as being wrong, you should always guess. A model that guesses on every uncertain question will beat an honest one that admits its gaps.
So that is the model we built. OpenAI researchers have made the point directly: under binary grading, systems are rewarded for guessing and penalized for saying they are unsure. We optimized for good test-takers, and good test-takers do not leave answers blank. The confident wrong answer is not the model failing. It is the model doing exactly what we rewarded.
Confidence is not knowledge
Here is the part that trips up smart people. We are wired to read fluency as competence. Someone who speaks in clear, structured, well-organized sentences usually knows their subject, because for humans that fluency was expensive to fake. Models make it free. A model produces the same polished prose whether it is reciting a fact or inventing one, and our instinct to trust the smooth talker fires anyway.
That is why "always check the output" is weaker advice than it sounds. Checking is exactly the effort the fluent answer discourages. The better the prose, the less you feel the urge to verify, and the more the one wrong sentence in twenty slips through.
Designing for a witness who never doubts
I do not think the answer is to distrust these tools. I use them every day and they are genuinely good. The answer is to stop treating a model like a source and start treating it like a very fast, very well-read assistant who is constitutionally incapable of saying "I am not sure." Once you accept that, the design choices get clearer.
Put the model where being wrong is cheap and visible: drafting, brainstorming, summarizing text you can see, writing code you are about to run and test. Keep it away from places where a confident fabrication is expensive and hard to catch, like a medical dose, a legal citation, a number that flows straight into a decision with no human in the loop. Retrieval helps, because grounding an answer in a document you can inspect turns "trust me" into "here is where it came from." And when you can, prefer systems that surface uncertainty at all, even a rough confidence score, over ones that render everything in the same calm voice.
There is real research now on training models to abstain, to earn credit for admitting a gap instead of being punished for it. I hope it works. Until it ships, the burden sits with us, and the honest move is to build workflows that assume the machine will never tell you when it is out of its depth. It will keep talking. The question worth sitting with is why we find that so much more comforting than a system that occasionally, usefully, went quiet.