Every explanation of embeddings I bounced off early on opened with a matrix, a mention of cosine similarity, and a 3D scatter plot rotating for no reason. None of that is what you need to use them. You can reason about embeddings perfectly well with a mental model you already carry around: a map.
This is the follow-on to what RAG actually is, where I called embeddings "one retrieval strategy among several" and moved on. Here's the strategy, explained for someone who builds software, not someone who's about to derive anything.
An embedding is a location
Picture a map of a country. Cities that are close together on the map tend to be similar in the ways that matter — climate, culture, who you can drive to for the weekend. The map took something complicated (a place) and turned it into two numbers, latitude and longitude, arranged so that distance means something.
An embedding does that to a piece of text. You feed in a sentence, a paragraph, a document, and you get back a list of numbers — its coordinates. The coordinates are arranged, by a model trained on a lot of language, so that text with similar meaning lands close together. "How do I rotate my API keys" and "credential rotation policy" end up near neighbors, even though they don't share a single important word.
That's the whole trick. Not two numbers like a real map — usually a few hundred or a couple thousand — but the idea survives the jump to more dimensions intact. Close together means similar in meaning. Far apart means unrelated.
Why you'd want that
Keyword search matches words. It's fast, it's cheap, and it fails the moment the person asking uses different words than the document. A user types "the app is slow on startup"; your runbook is titled "cold-start latency." Same problem, zero shared keywords, no match.
Embeddings match meaning, so they sail right over that gap. That's the one thing they do that plain text search can't, and it's the only reason to reach for them. If your questions and your documents reliably use the same vocabulary, embeddings are solving a problem you don't have — keyword search will be faster and just as good.
How retrieval actually works with them
The mechanics are less mysterious than the marketing:
- Once, ahead of time: run every chunk of your documents through the embedding model and store its coordinates. This is the part that costs something and the part you do offline.
- At query time: embed the user's question the same way, then find the stored chunks whose coordinates are nearest to it. Hand those to the model.
"Nearest" is measured by an angle — whether two sets of coordinates point in roughly the same direction. You don't need the formula; the takeaway is that it's a cheap arithmetic comparison, not another AI call.
And the vector database everyone treats as the headline? It's the thing that makes step 2 fast. Comparing your question against ten chunks is a loop. Comparing it against ten million is a problem, and a vector DB is a specialized index for "find me the nearest neighbors" at that scale. It is plumbing, not the point. For a few thousand chunks you can hold the vectors in memory and skip it entirely.
The catches nobody puts in the quickstart
This is where the rotating scatter plot would've served you better than the math.
"Similar" is not "relevant." Embeddings find text that's about the same thing as your question. That's not the same as text that answers it. Ask "is our auth flow secure" and you'll pull back every doc that talks about auth — the design doc, the postmortem, the onboarding guide — ranked by topical closeness, not by which one actually contains the answer. Nearness gets you in the right neighborhood; it doesn't knock on the right door.
Questions and answers don't always look alike. You're embedding a question and hoping it lands near the answer. But a question ("how do I configure retries?") and its answer (a YAML snippet with a max_attempts key) can be genuinely far apart in meaning-space, because they're different kinds of text. This one bites people constantly and feels like the model being dumb when it's really a shape mismatch.
The embedding model only knows what it was trained on. Your internal codename for a service, an acronym your team invented last quarter, a product that launched after the model's training cutoff — the embedder has no idea what those mean, so it places them more or less at random. Exactly the private vocabulary you most wanted to search well is the vocabulary embeddings handle worst.
So when do you actually use them
Reach for embeddings when vocabulary mismatch is your real failure mode — when you've watched real queries miss real documents purely because the words didn't line up. That's the problem they're built for, and they're very good at it.
Don't reach for them by default. Start with keyword search, because it's free and it's often enough. Add embeddings as a second retrieval pass when you can point at the searches keyword matching is losing. And whatever you add, measure it against a handful of real questions before you believe it — "we added embeddings" and "retrieval got better" are not the same sentence, and plenty of teams have shipped the first while assuming the second.
Embeddings are a map. Maps are wonderful for "what's near this place." They're useless for "is this the place I actually need." Knowing which question you're asking is most of the skill.