Embeddings & Vector-DB Efficiency

RAG efficiency treated retrieval as a black box that hands the model a few good chunks, and focused on the token cost of what you stuff into the prompt. This page opens that black box. Retrieval has its own bill — paid in embedding inference, in memory for the index, and in search latency — and at scale (millions or billions of vectors) it can rival the generation cost it was meant to save. The recurring question applies here too: what does this cost in latency, memory, and dollars, and how do we make it cheaper? For a vector database, the answers are unusually concrete.

Two cost centers: embedding and search

Retrieval spends in two places, at two different times:

   INDEX TIME (once per chunk)        QUERY TIME (every search)
   ─────────────────────────         ──────────────────────────
   embed each chunk → vector          embed the query → vector
   store + index the vectors          search the index for nearest vectors
        │                                  │
        ▼                                  ▼
   cost: embedding inference,         cost: search latency + the memory
         memory for the index               the index occupies in RAM

Embedding is cheap per token and, crucially, done once per chunk and reused across every future query — so it amortizes. The recurring cost is search: every query embeds and then searches. And the cost that never goes away is memory: the index sits resident in RAM, and at scale that is often the largest line item.

Embedding inference cost

An embedding model is just a (usually smaller) transformer that maps text to a single fixed-length vector. Its inference is far cheaper than generation — no autoregressive decode, one forward pass per input — which is why the RAG page urges spending on retrieval to save on generation. Two cost facts follow:

• Embed once, reuse forever. Re-embedding your whole corpus on every deploy is a silent, avoidable cost. Only new or changed chunks need re-embedding; only the live query needs embedding at runtime.
• Dimension size is a cost. An embedding’s dimensionality (say 768, 1536, 3072) sets how many floats each vector carries — which drives both storage/memory and the per-comparison cost of search. Bigger vectors can mean better recall, but every dimension is paid in memory across every vector you store.

ANN indexes: sublinear search

The RAG page’s “Under the hood” already made the key point: production search never does a brute-force scan, because comparing the query against every vector is linear in corpus size and would take seconds-to-minutes at millions of vectors. Instead, vector stores build an approximate nearest-neighbor (ANN) index. Two families dominate:

• HNSW (Hierarchical Navigable Small-World; Malkov & Yashunin, 2016) builds a layered graph you can traverse in roughly logarithmic time — a short walk through a navigation structure instead of a full scan. Fast and high-recall, but the graph lives in RAM and adds memory overhead on top of the raw vectors.
• IVF (Inverted File) partitions the vectors into clusters; a query only searches the few nearest clusters instead of everything. Cheaper memory than HNSW, with a recall knob (nprobe: how many clusters to check).

The word in the name is the trade: approximate. You accept a small chance of missing the true nearest neighbor in exchange for a thousands-fold speedup. Recall vs. latency vs. memory is the triangle every index parameter moves you around inside.

Vector quantization: shrink each vector

ANN cuts how many comparisons you do. Quantization cuts how expensive each vector is to store and compare — the same precision-reduction idea applied to embeddings instead of weights.

• Scalar quantization (int8). Store each dimension as an 8-bit integer instead of a 32-bit float: 4× less memory, with a small recall hit. Often the first, cheapest win.
• Binary embeddings. Reduce each dimension to a single bit (sign only): up to 32× less memory than fp32, and distances become hardware-cheap bit operations (Hamming distance). Recall drops more, so binary search is usually a fast first pass, then the top candidates are re-ranked with the full-precision vectors — search wide and cheap, refine narrow and accurate, the same shape as RAG’s retrieve-then-rerank.
• Product quantization (PQ). (Jégou et al., 2011.) Split each vector into sub-vectors, replace each sub-vector with the nearest entry from a small learned codebook, and store only compact codes. Dramatic compression of huge corpora, which is why PQ underpins billion-scale indexes.

By the numbers — what the index actually costs in RAM

Take 10 million chunks, each embedded to a 1,536-dimension vector.

fp32 raw vectors:  10,000,000 × 1,536 × 4 bytes  ≈  61 GB
int8 quantized:    10,000,000 × 1,536 × 1 byte   ≈  15 GB   (4× smaller)
binary (1 bit/dim):10,000,000 × 1,536 / 8 bytes  ≈  1.9 GB  (~32× smaller)

And that’s just the vectors — an HNSW graph adds more on top for its links. The lesson: at scale, the index is a memory problem first. Quantization is what keeps a billion vectors in RAM instead of spilling to disk (where search slows by orders of magnitude). You trade a sliver of recall for fitting in memory, and fitting in memory is most of the speed.

Matryoshka embeddings: truncatable dimensions

Normally an embedding’s dimensionality is fixed at training time — you store all 1,536 floats or you retrain. Matryoshka Representation Learning (Kusupati et al., 2022) changes that. It trains the embedding so that its most important information is packed into the earliest dimensions, like nested Russian dolls. That means you can truncate a 1,536-dim vector to its first 768 or 256 dimensions and still get a usable embedding — no retraining, just slice the vector.

   full vector:   [█████████████████████████]  1,536 dims  (best recall, most memory)
   truncated:     [████████████]                768 dims   (less memory, slight recall loss)
   truncated:     [████]                         256 dims   (smallest, fastest, lower recall)
                   ↑ the early dimensions carry the most signal

This makes dimensionality a runtime knob instead of a training decision. Store short vectors for cheap, fast first-pass search; keep the full vectors for re-ranking the top candidates. Newer embedding APIs (for example, OpenAI’s text-embedding-3 family, released January 2024) expose exactly this shortening, built on the Matryoshka idea.

The recall-vs-cost knob

Every technique here is the same dial under a different name. You are always trading recall (did I find the truly nearest vectors?) against cost (memory, latency, dollars):

Lever	Saves	Costs you
ANN index (HNSW/IVF)	search latency (linear → sub-linear)	RAM for the index + a little recall
int8 / binary quantization	memory (4×–32×)	recall (recover it with a re-rank pass)
Product quantization	memory at billion scale	recall + codebook complexity
Matryoshka truncation	memory + per-comparison cost	recall at smaller dimensions
Lower nprobe / smaller graph	latency	recall

The safe, repeated pattern across all of them: search cheap and wide for recall, then re-rank narrow and precise for accuracy. Quantized or truncated vectors find candidates fast; full-precision vectors confirm the winners. Measure recall as you tighten each knob, and stop when recall dips below what your answers need — not before.

It actually happened — Matryoshka embeddings reach production (2022 → 2024)

Matryoshka Representation Learning (Aditya Kusupati and colleagues, 2022) showed you could train a single embedding whose leading dimensions are independently usable, so one model serves many dimension budgets. For a while it was a neat research result. Then in January 2024, OpenAI’s text-embedding-3 models shipped with a dimensions parameter letting developers shorten embeddings on demand — a direct, widely-used application of the Matryoshka idea. It is a clean example of the frontier-to-default pipeline this part keeps pointing at: a 2022 paper became, two years later, a routine cost knob a developer sets in one line — trading a controlled bit of recall for real memory and latency savings.

The architect’s lens

Open RAG’s retrieval black box — at scale it’s a memory problem first:

• Why does it exist? Because retrieval has its own bill — embedding inference, index RAM, and search latency — and at millions-to-billions of vectors it can rival the generation cost it was meant to save.
• What problem does it solve? Keeping search sublinear and the index in RAM: ANN indexes (HNSW ~log time, IVF clusters) avoid the linear brute-force scan, while int8 (~4×) / binary (~32×) / product quantization and Matryoshka truncation shrink each vector.
• What are the trade-offs? Every lever is the same dial — recall vs cost: ANN trades a little recall for a thousands-fold speedup, quantization trades recall for memory (recover it with a full-precision re-rank), and the index sits resident in RAM, often larger than the raw vectors.
• When should I avoid it? At a tiny corpus size a brute-force exact scan is fine — ANN’s memory overhead and approximate recall aren’t worth it until linear search actually hurts.
• What breaks if I remove it? Without ANN, search goes linear (seconds-to-minutes at scale); without quantization/truncation, a billion vectors spill from RAM to disk, where search slows by orders of magnitude.

Check your understanding

1. Retrieval has two cost centers at two different times. Name them, and say which one amortizes and which recurs.
2. Why does production vector search use an ANN index like HNSW or IVF instead of comparing the query against every vector?
3. In the by-the-numbers example, why is the index fundamentally a memory problem at scale, and how does quantization help?
4. What does Matryoshka Representation Learning let you do at runtime that a normal fixed-dimension embedding does not?
5. State the single recall-vs-cost pattern that recurs across ANN, quantization, and Matryoshka truncation.

Show answers

1. Embedding inference (done once per chunk at index time, reused across all future queries — it amortizes) and search (embed the query and search the index on every query — it recurs); resident index memory is a third, ever-present cost.
2. Comparing against every vector is linear in corpus size and would take seconds-to-minutes at millions of vectors; an ANN index turns search into a sub-linear operation (roughly logarithmic for HNSW, or only-nearest-clusters for IVF), trading a small chance of missing the true nearest neighbor for a massive speedup.
3. The raw vectors (and the HNSW graph on top) must stay resident in RAM, and at tens of millions to billions of vectors that dominates the bill; quantization shrinks each vector (int8 ≈ 4×, binary ≈ 32×) so the index fits in RAM instead of spilling to disk, and fitting in memory is most of the speed.
4. It lets you truncate the embedding to fewer dimensions at runtime (because the most important information is packed into the earliest dimensions) and still get a usable vector — making dimensionality a runtime knob rather than a fixed training decision, with no retraining.
5. Search cheap and wide for recall (ANN over quantized or truncated vectors to find candidates fast), then re-rank narrow and precise for accuracy (confirm the winners with full-precision vectors) — always trading a controlled bit of recall for memory, latency, and dollars.