AI agents

"Agent" is the most abused word in AI right now. Vendors slap it on anything that calls an API twice. So let us be plain: an AI agent is a model that can take actions in a loop, decide what to do next based on the result, and keep going until a goal is met or it gives up. That is it. The interesting question is not what they are. It is when you should let one loose.

Agentic AI is the phrase every vendor deck and every board meeting is chasing this year. It is also, for most of the companies rushing to deploy it, a mistake they have not noticed yet. I say this as someone who thinks agents are genuinely useful. That is exactly why the current stampede worries me.

"Memory" is one of the most oversold words in AI right now. Products promise an assistant that remembers you, learns your preferences, picks up where you left off. Then you use it for a week and notice it forgot the thing you told it on Tuesday. The gap between the pitch and the reality comes from a fact people gloss over: a language model has no memory at all.

The model is a function. Text goes in, text comes out, and the moment the response finishes it retains nothing. Everything we call memory is scaffolding we build around that stateless core, deciding what to put back into the input next time. There are only a few honest ways to do it, and each one breaks in its own way.

Stuffing the context window

The crudest approach: keep the whole conversation and paste it back in every turn. For a short chat this is perfect. The model "remembers" because you literally handed it the transcript.

It falls apart on two edges. Context windows are finite, so a long enough conversation eventually will not fit. And even when it fits, more context is not free. You pay for every token on every call, and models get measurably worse at finding the one relevant line when it is buried in tens of thousands of tokens of history. Stuffing is a real technique with a hard ceiling, not a memory system.

Summaries and their slow leak

So you compress. Every so often, ask the model to summarize the conversation so far and carry the summary forward instead of the raw text. This buys a lot of runway, and most chat products do some version of it.

The catch is that summarizing is lossy on purpose, and you do not get to choose what it loses. The detail that seemed irrelevant when the summary was written is exactly the one that matters three turns later. Summaries of summaries drift further each round, like a photocopy of a photocopy. Good for the gist, unreliable for the specific fact.

Retrieval, which is what "memory" usually means now

The approach that actually scales is to stop trying to hold everything in the prompt. Write facts and past exchanges to a store, and when a new message comes in, fetch only the handful of pieces relevant to it and drop those into the context. This is RAG pointed at your own history instead of a document library.

It is the best of the three and still not what people imagine. Retrieval memory only recalls what it thought to save and what the query happens to match. Phrase your question differently than the stored fact was written and the right memory may never surface. It cannot form the kind of connection a person makes across two things noticed months apart, because nothing is reasoning over the whole store, it is just fetching nearest matches. What you get is a good filing clerk, not a mind.

Here is the thing I wish more products said plainly. None of these give a model memory in the human sense, the kind that reshapes understanding over time. They are all variations on one move: choose what text to put in front of a stateless function on the next call. That is a real and useful engineering problem, and you can build genuinely helpful things by solving it well.

So when you design memory, do not chase the fantasy of an assistant that just knows you. Decide, deliberately, what is worth remembering, how it gets written down, and how it gets found again. Keep the recent turns raw, summarize the middle distance, retrieve the long tail, and accept that each layer forgets in a different way. The teams who ship memory that feels good are not the ones with a magic store. They are the ones who were honest about what a model is and engineered around it.

The demo worked because you were the only person hitting it. Then ten users showed up, a provider had a slow afternoon, and your app started returning blank screens and half-written answers. None of that is an AI problem. It is a plumbing problem, and the plumbing for AI apps is a little different because the calls are slow, expensive, and occasionally rejected on purpose.

Here is the honest version of what it takes to make a model-backed feature that does not fall over the first time the network has a bad day.

Rate limits are a promise, not an insult

Every provider caps how many requests and tokens you can send per minute. When you cross the line you get an HTTP 429. People treat that response like a failure, but it is closer to a traffic light. The provider is telling you exactly how to behave, and most of the time the answer is written in the headers. Anthropic and OpenAI both send a Retry-After value on many 429s. If it is there, wait that long. Do not invent your own number when the server already told you the truth.

The mistake I see most often is retrying immediately, in a tight loop, across every worker at once. That does not recover from a rate limit. It deepens it. You have now turned one rejected request into fifty, all landing at the same instant, guaranteeing the next fifty get rejected too. This is the thundering herd, and it is self-inflicted.

Retries: back off, and add jitter

The fix is exponential backoff. Wait one second, then two, then four, then eight, doubling each time up to a ceiling. That spaces your attempts out so the system has room to recover. But backoff alone is not enough, because if a hundred clients all failed at the same moment, they will all wait exactly one second and then all retry at exactly the same moment. You have synchronized your herd instead of scattering it.

So you add jitter: a random amount of slop on each wait. Instead of sleeping exactly two seconds, sleep somewhere between zero and two. This one trick, randomizing the delay, does more for stability than almost anything else you can do, and it is two lines of code.

A few rules that keep retries sane:

• Retry on 429 and 5xx and network timeouts. Do not retry a 400 or a 401, because a malformed request or a bad key will fail identically every time.
• Cap the total. Three to five attempts, then give up and surface an error. Infinite retries just hide the outage from you while your users feel every second of it.

Timeouts, and the streaming trap

Model calls are slow and the tail is long. A response that usually takes three seconds will sometimes take forty. If you have no timeout, a single stuck request can hold a connection open until something upstream kills it, and you find out when your whole pool is exhausted. Set an explicit timeout. Pick a number based on your real latency distribution, not on the median, because the median is not what hurts you.

Streaming changes the shape of this. When you stream tokens, the connection can be alive and healthy while the model has quietly stalled and stopped sending. A total request timeout will not catch that cleanly. What you want is an idle timeout: if no token has arrived in some number of seconds, treat the stream as dead and fall back. Watch the gaps between chunks, not just the clock on the whole call.

Idempotency and degrading on purpose

Here is the part people skip until it bites them. You send a request, the model does the work, and then the response gets lost on the way back to you. Your retry logic kicks in and sends it again. Now you have paid for the same completion twice, and if that call also wrote a row or charged a card, you have a duplicate in your data.

Send an idempotency key with anything that has a side effect. It is a unique string per logical operation. If the provider or your own service sees the same key twice, it returns the first result instead of doing the work again. Providers support this for exactly this reason. Use it, and a lost response becomes a non-event instead of a double charge.

Sometimes the model is just down, or slow enough that waiting is worse than not answering. Decide ahead of time what happens then. Maybe you drop to a smaller, cheaper model. Maybe you return a cached answer that is a little stale. Maybe you show the raw search results without the AI summary on top. The worst option is a spinner that never resolves, because that teaches people your product is broken.

None of this is glamorous and none of it shows up in a launch video. But it is the difference between a feature that survives contact with real traffic and one that only ever worked on your laptop. The models get the headlines. The retry loop with jitter is what keeps the thing standing.

I want to make a small, cranky request on behalf of clear thinking everywhere: stop calling everything an agent. The word has been stretched so far that it now means anything from a genuinely autonomous system to a chatbot that calls one API. When a word means everything, it means nothing, and the fuzziness is not an accident. It is marketing.

Here is an uncomfortable result that deserves more airtime than it gets. In 2025 the research group METR ran a randomized trial with experienced open-source developers on real issues from repositories they already knew well. With AI tools allowed, they finished tasks 19 percent slower. Afterward, the same developers estimated that AI had made them about 20 percent faster. They were slower and felt faster, by nearly the same margin. That gap is the whole story, and it is worth sitting with.

Why fast can feel faster than it is

The researchers pointed at reduced cognitive effort. AI-assisted work felt easier, and we quietly file easier under faster even when the clock disagrees. Watching a model produce a wall of plausible code feels like progress in a way that staring at a blank editor does not, and the feeling is real even when the output is not saving you anything.

The catch is what the feeling hides. You did not write the code, so now you have to read it, and reading someone else's code closely enough to trust it is not free. If it is wrong in a subtle way, you pay twice: once to spot the problem and again to fix it, often after you have already convinced yourself it was fine. The generation was fast. The verification was not, and verification is the part that does not show up in the demo.

Before I am accused of doom: I should note METR themselves later flagged that their study design had problems, partly because the developers who benefit most from AI would not join a no-AI condition even at 50 dollars an hour. So do not read 19 percent as a law of nature. Read it as a real, measured case where the tool that everyone assumed was a speedup was not, and the users could not tell. That is the part that generalizes.

Where the time actually leaks

The productivity leaks are boring and specific, which is exactly why they are easy to miss.

• Reviewing confident nonsense. The failure mode that costs the most is a wrong answer delivered with total assurance: a made-up function that looks real, a plausible statute that does not exist, a config flag that was never a flag. Confident and wrong is more expensive than obviously broken, because obviously broken you catch in a second and confident-wrong you ship.
• Context-switching. Every trip out to the tool and back is a small tax on your attention, and enough small taxes add up to a workday where you were busy and moved nothing.
• The almost-right rabbit hole. The model gets you 80 percent of the way, and you spend longer chasing the last 20 percent through its logic than you would have spent writing the thing yourself. Nudging a nearly-correct output into a correct one can cost more than starting clean.

None of these feel like waste in the moment. They feel like work. That is the trap.

It still works, when you aim it right

I am not telling you to put the tools down. I use them daily and would be slower without them, which is precisely why I take the paradox seriously instead of waving it off. The tools genuinely help, but the win is conditional, and the conditions are learnable.

They shine when verification is cheap or the stakes are low. Boilerplate you can eyeball in seconds. A first draft you were going to heavily rewrite anyway. A language or API you half-remember, where the model jogs your memory faster than the docs. Throwaway scripts. Anything where being roughly right is good enough and checking is quick. In those spots the speedup is real and often large.

They quietly cost you when verification is expensive and correctness is non-negotiable. Subtle logic in unfamiliar code. Anything security-sensitive. Domains where you cannot easily tell right from wrong-but-plausible, which is exactly where the model's confidence is most dangerous, because you have no cheap way to check it.

The skill, and it is a skill, is noticing which situation you are in before you reach for the tool, not after. The developers in that study were not fools. They were experienced people who genuinely could not feel the slowdown while it was happening. That is the real lesson. The tool is not the problem and neither are you. The problem is that speed and the feeling of speed have come apart, and the only fix is to occasionally check the clock instead of trusting the vibe.

Every executive survey says AI budgets are going up. A quieter set of numbers says most of that money is not paying off. Only about a quarter of enterprise AI initiatives deliver the ROI they promised. That gap is the most interesting, and least discussed, story in corporate AI, because the reason for it is almost never the thing everyone blames.