The moment someone decides to build anything with retrieval, a vector database appears in the plan. Pinecone, Weaviate, Qdrant, Chroma, pick your logo. It feels mandatory, like you cannot do this seriously without one. For a lot of projects that instinct is wrong, and acting on it costs you a monthly bill and a running service you did not need.
What a vector database actually does
Strip away the branding and a vector database does one job: given a query vector, find the closest vectors out of a large pile, fast. "Closest" usually means cosine similarity or dot product. The clever part is the "fast." Comparing your query against every stored vector is linear work: fine at ten thousand items, painful at ten million. Vector databases use approximate nearest neighbor indexes, HNSW being the popular one, to skip most of the comparisons and still find almost the right answers in milliseconds.
That word approximate matters. You trade a little accuracy for a lot of speed. At scale that trade is obviously worth it. At small scale you are paying for a solution to a problem you do not have.
When a plain file is genuinely enough
Say your corpus is a company handbook, one product's docs, or a few hundred support articles. Chunked, that might be two thousand vectors. Maybe twenty thousand. Here is the thing nobody selling a database says out loud: you can hold those in memory and compare against all of them on every query, and it will feel instant.
An exhaustive similarity search over twenty thousand embeddings is a single matrix multiply. NumPy does it in a few milliseconds. Store the vectors in a file, or a column of SQLite, load them once, and do the dot product yourself. No index to tune, no service to run, no approximation, no network hop. You get exact results and a system a new engineer understands in one sitting.
The rough line in my head:
- Under about 50,000 vectors, static or slow-changing: a file plus brute-force search is not just fine, it is better. Simpler, exact, free.
- Hundreds of thousands to millions, or heavy write traffic with metadata filtering: now you want a real index, and a vector database earns its keep.
Those numbers are soft. A beefy machine brute-forces further than you think, and a small but write-heavy workload can need real infrastructure sooner. The decision is about scale and change rate, not about seriousness.
The middle ground people skip
Between "a text file" and "a hosted vector database" sits a range that solves most real projects. SQLite with the sqlite-vec extension gives you vector search in a single local file. Postgres with pgvector adds vectors to the database you probably already run, so you get real filtering, transactions, and backups without adopting a new system. FAISS, the library from Facebook, gives you fast indexes embedded in your process, no server at all.
Reach for those before you reach for a managed vector service, and most teams never need the managed service. The ones that do tend to know it, because they have the scale, the query volume, or the ops appetite that justifies it.
My actual advice is annoyingly boring. Start with the smallest thing that works, usually vectors in a file or in the database you already have, and move up only when you can point at a real number that hurts: query latency, corpus size, write load. A vector database is a fine tool. It is just not the starting line, and treating it as one is how you end up maintaining infrastructure to search a document you could have fit in a spreadsheet.