AI Guides & Tutorials
An embedding is a list of numbers that stands in for a piece of text. Feed a sentence to an embedding model and it hands back a few hundred or a few thousand numbers. That list is the whole trick. Text that means similar things lands on similar lists, and text that means different things lands far apart. Once you accept that one idea, most of the useful applications fall out on their own.
The reason this matters is that computers are good at math and bad at meaning. Keyword search knows that "car" and "car" match. It has no idea that "car" and "automobile" are the same thing, or that "my sedan will not start" is closer to "vehicle trouble" than to "sedan chair." Embeddings give the machine a way to measure closeness in meaning instead of closeness in spelling.
How closeness actually works
Picture every sentence as a point in space. Not the three dimensions you live in, but a space with hundreds of them. You cannot draw that, and you do not need to. What you need is a way to ask how close two points are, and the usual answer is cosine similarity: it measures the angle between two of these lists of numbers and returns a score, roughly from minus one to one. Near one means the two texts point the same direction, which is to say they mean roughly the same thing. Near zero means they are unrelated.
You never look at the raw numbers. Nobody reads an embedding. You compute it, store it, and later ask the database which stored vectors sit nearest to a new one. That single operation, find the nearest points, is the engine under almost everything below.
The jobs embeddings are good at
Search is the obvious one, and the one most people meet first. You embed all your documents once. When a user types a question, you embed the question and pull back the passages whose vectors are nearest. The user searches by meaning, so "how do I cancel my plan" finds the paragraph titled "ending a subscription" even though they share almost no words.
Deduplication is quieter but pays for itself fast. If you have a support inbox, a product catalog, or a pile of scraped articles, near-duplicate items are everywhere and exact matching misses them. Embed everything, look for pairs whose similarity crosses a threshold you pick, and you catch the "same thing, slightly reworded" cases that a string comparison walks right past.
Clustering is what you reach for when you do not know your categories yet. Embed a few thousand customer messages, group the vectors that huddle together, and read a handful from each group. The themes come to you instead of you guessing them in advance. It is a genuinely nice way to find out what people are actually writing about.
And then there is retrieval for RAG, which is the same search step wearing a job title. Before a language model answers, you embed the question, fetch the most relevant chunks of your own documents, and hand them to the model as context. The retrieval half of retrieval-augmented generation is embeddings doing exactly what they do in search. Nothing more exotic than that.
Where it gets you into trouble
Embeddings measure similarity, and similarity is not the same as relevance, correctness, or intent. Two sentences can sit close in vector space and mean opposite things, because "the drug is safe" and "the drug is not safe" share almost every word. Negation, sarcasm, and small but load-bearing details are exactly where cosine similarity gets sloppy. The models improve on this every year, but do not assume the nearest neighbor is the right answer. It is the closest guess.
A few practical notes. The model you embed with matters more than the number of dimensions; a good small model beats a mediocre large one. You have to embed your query and your documents with the same model, or the comparison is meaningless. And thresholds are not universal. A similarity of 0.8 might mean "basically identical" for one model and "vaguely related" for another, so test against your own data before you trust a cutoff.
If you take one thing away, let it be this: embeddings are a ruler for meaning. Not a truth machine, not a reasoning engine, just a very good way to ask what is near what. Most of the time that is precisely the question you had.
Somewhere in every LLM tutorial there is a line that sets temperature to 0.7, and a remarkable number of people copy it, ship it, and never think about it again. Then the output goes weird and they have no idea which knob to turn. Temperature and top-p are the two settings that decide how the model picks each next word, they are genuinely simple once explained, and most people are using them on autopilot. Here is what they actually do.
The model is always guessing, these settings pick how
At every step, the model produces a ranked list of possible next tokens, each with a probability. It does not know the answer, it has a distribution over answers. Temperature and top-p are two different ways of deciding how to draw from that distribution.
Temperature reshapes the whole distribution. Low temperature, near zero, sharpens it so the single most likely token almost always wins, which makes the output predictable and repeatable. Crank it up toward one and beyond and you flatten the distribution, giving unlikely tokens a real shot and making the output more varied and, past a point, more unhinged. At temperature zero the model is effectively deterministic: same prompt, same answer, every time.
Top-p, also called nucleus sampling, works differently. Instead of reshaping probabilities, it draws a cutoff. Top-p of 0.9 means "consider only the smallest set of tokens whose probabilities add up to 90 percent, and ignore the long tail." The clever part is that it adapts to the model's confidence. When the model is sure, one token already holds most of the probability, so top-p samples from just a token or two. When the model is unsure and fifty tokens each hold a little, top-p keeps all fifty in play. It widens and narrows on its own.
Where people go wrong
The first mistake is turning both knobs at once. Temperature and top-p both constrain the same sampling step, and stacking them makes the effect of either impossible to reason about. The standard advice, which I agree with, is to move one and leave the other at its neutral setting. Pick temperature or pick top-p. Do not choreograph both and then wonder why the behavior is unpredictable.
The second mistake is believing high temperature equals creativity. It does not. High temperature buys you randomness, and randomness is not the same thing as good ideas. Past a certain point you are not getting a more imaginative model, you are getting a model that picks worse words more often. There is a real ceiling where interesting tips over into incoherent, and it arrives sooner than people expect. If you want surprising-but-good, a modest bump plus a better prompt beats cranking the dial to the roof.
The third mistake is silent and nasty: tuning these values on one backend and deploying on another. The OpenAI API, vLLM, and llama.cpp do not necessarily apply samplers in the same order, so the same numbers can produce different text on different stacks. Always tune against the exact backend you will actually run.
Sane defaults to start from
For anything where there is a correct answer, set temperature to 0. Code generation, data extraction, classification, factual questions, anything you would run twice and want the same result. Randomness here is pure downside: it introduces inconsistency across identical inputs and buys you nothing. This is the setting most people should be using far more than they do, because most real work has a right answer and they leave it at 0.7 out of habit.
For creative writing, brainstorming, generating varied options, raise it. Somewhere around 0.7 to 1.0 is a reasonable playground, and this is genuinely where a warmer setting earns its place: you want the model to wander a little, and repeating the exact same story every time defeats the point.
The default that ships with most APIs, around 0.7, is a compromise aimed at open-ended chat. It is a fine middle for conversation and a poor fit for either extreme. If you are extracting invoice fields at 0.7, you are inviting the occasional wrong number for literally no benefit.
The whole thing takes five minutes to understand and saves you a lot of confused debugging. When output feels too samey, you know which way to push. When it feels unreliable and you want the same answer twice, you know to pull temperature to the floor. That is the entire skill: not memorizing magic numbers, but knowing what the two knobs move so you can reach for the right one instead of copying 0.7 from a tutorial and hoping.
A context window is the amount of text a model can hold in its head at once, your prompt and its own reply combined, measured in tokens. For years that number was small enough to be a real constraint. Now vendors advertise a million tokens, sometimes more, roughly a stack of books. The pitch is that you can stop worrying about memory entirely. The reality is more interesting, and more annoying, than that.
What the window actually is
Everything the model knows in a given call has to fit in the window. There is no background memory, no notebook it flips back to. If a fact is not in the window, the model cannot use it, and if it is in the window, you paid for it. A million-token window means you can, in principle, drop an entire codebase or a year of email into a single request and ask a question about it. That part is real. You could not do it three years ago, and now you can.
What people hear, though, is "the model reads all million tokens as carefully as it reads a paragraph." It does not. Capacity and attention are different things, and the gap between them is where most of the disappointment lives.
Lost in the middle
The best-documented failure has a name: lost in the middle. Models pay the most attention to the start and the end of their context and get noticeably worse at anything buried in between. Plot the recall accuracy and you get a U-shape, strong at both ends, sagging in the middle. In needle-in-a-haystack tests, where a single fact is hidden in a long document, accuracy can drop by thirty points or more when that fact sits in the middle rather than near the edges.
It gets worse as the window fills. A Microsoft Research study found effective use of context falls to around 60 percent past 100,000 tokens. In plain terms, if you stuff 500,000 tokens into a prompt, the model is effectively ignoring or badly integrating a couple hundred thousand tokens' worth of it. The capacity is there on the spec sheet. The comprehension is not keeping pace.
So "put everything in the context and let the model sort it out" is a strategy that works right up until the answer depends on something in the murky middle. Then it fails quietly, which is the dangerous kind of failure, because the model does not announce that it skimmed.
The parts nobody puts on the slide
Then there is the bill and the clock. Every token in the window is a token you pay for and a token the model has to read before it says a word. A genuinely full million-token prompt can run north of ten dollars in input cost alone, and you may wait 30 seconds, sometimes past two minutes, before the first word of the answer appears. That prefill delay is not a bug, it is the model chewing through everything you gave it. For a background job, fine. For anything a person is waiting on, it is a dealbreaker.
This is the case against reaching for the giant window by default. If you can hand the model the ten relevant pages instead of the whole ten thousand, you get a faster answer, a cheaper answer, and often a more accurate one, because you have removed the haystack instead of asking the model to search it. This is a big part of why retrieval, pulling in just the passages that matter, has not been made obsolete by large windows the way some people predicted. Retrieval and a big window are tools for different jobs, not rivals.
When it earns its keep
Big windows genuinely shine in a few spots. Reasoning over a single long document where you cannot know in advance which part matters. First-pass exploration of an unfamiliar codebase. Analyzing one long transcript or contract end to end. Cases where the cost of missing a scattered detail is worse than the cost of latency and dollars. When the answer could depend on any part of a large whole, filling the window is the honest move.
The mistake is treating window size as a headline stat, the way phones once competed on megapixels. A million tokens is a real capability and a real convenience. It is not a memory upgrade that makes your data management problems disappear, and any pitch that implies otherwise is selling you the number, not the result. Use the whole window when the job actually needs the whole thing. The rest of the time, the smaller, sharper prompt wins, and it wins on every axis that shows up in your invoice.
A token is the thing you are actually paying for, and almost nobody thinks in tokens. They think in words, or messages, or vibes. Then the invoice arrives and the math does not line up. Here is the plain version of what a token is, how your text gets chopped into them, and why the model bills this way instead of the way you would expect.
Not a word, not a letter, something in between
When you send text to a model, the first thing that happens is that a piece of software called a tokenizer breaks your text into pieces. A token is one of those pieces. Sometimes a token is a whole word. Often it is a fragment. The word "sleeping" might become "sleep" plus "ing". Common words like "the" are usually a single token, while a rare or long word gets split into two or three.
The rough rule people quote is that one token is about four characters of English, or roughly three quarters of a word. OpenAI's own guidance says about 750 words works out to around 1000 tokens. That heuristic is fine for a quick estimate and wrong the moment your text stops looking like a plain paragraph. Code, punctuation, JSON, and tables all push the count up, because symbols and indentation do not compress into neat word-sized chunks. So does anything not in English. The tokenizers were trained mostly on English, so "strawberry" costs one or two tokens while the same word in Chinese or Arabic can cost several times more. Same meaning, bigger bill.
Why chop words up at all? Because the model has a fixed vocabulary, usually somewhere between 50,000 and 100,000 tokens. If every distinct word needed its own entry, the vocabulary would explode and the model would choke on any word it had never seen. Subword tokens are the compromise. The model learns a set of common fragments and reassembles anything, including words that did not exist when it was trained, out of pieces it already knows.
Why the meter runs on tokens
Providers charge per token because tokens are what the model literally processes, one step at a time. Every token in your prompt has to be read before the model writes anything, and every token it writes is another unit of compute. There is no natural billing unit called a "word" inside the machine. There are only tokens going in and tokens coming out.
That is also why input and output are usually priced differently, with output costing more. Generating a token is more expensive than reading one, because the model does a full forward pass for each new token it produces. When you see two prices on a pricing page, that is what the gap is about.
Where the bill bites
The practical consequences follow directly, and they are easy to miss until they cost you money.
- Long context is not free context. If you paste a 40-page document into every request, you pay for all of it on every request, whether or not the model needed most of it to answer. A chat that remembers the whole conversation is re-reading and re-charging that history each turn.
- Verbose system prompts are a recurring tax. That 800-token instruction block you wrote once gets billed on every single call.
- "Make it concise" saves output tokens, which are the expensive ones. Asking for a one-line answer instead of an essay is a real cost lever, not just a style choice.
None of this means you should obsess over trimming tokens like a coupon-clipper. Most single requests cost a fraction of a cent. It matters at volume. A prompt that is 30 percent heavier than it needs to be does not hurt when you run it twice. Run it two million times and it is a line item someone will ask you about.
Count before you commit
The one habit worth building: stop estimating in words when real money is involved. Run your actual prompts, the messy ones with the code blocks and the pasted logs and the non-English names, through a token counter before you ship. Most providers give you one, and the number is almost always higher than your gut said, because your gut counts words and the model counts something stranger.
The word is a unit humans invented for humans. The token is a unit the machine invented for itself. Your bill is written in the machine's units, and once you can see the text the way the tokenizer sees it, the invoice stops being a surprise and starts being something you can actually plan around.
Once you have the basics of prompting down, the next step is not a longer list of tricks. It is a small set of reliable patterns, reusable moves that keep working on real tasks, under pressure, across model versions. These are the ones I actually reach for, stripped of the hype.
Read more: Prompt patterns that survive contact with real work