pgvector at Scale: Cut Index Size and Re-Embedding Costs

Every retrieval-augmented generation stack starts as a quality project and quietly becomes a cost project. The demo answers questions well. Six months later someone asks why the database instance is three sizes bigger, why the nightly re-index job runs for four hours, and why a provider’s model retirement notice just turned into an unbudgeted migration. None of that is a model problem. It is data infrastructure, and it is predictable.

Why the bill arrives after the demo works

Embeddings are the largest single object in most RAG databases. A 1,536-dimension embedding stored as a standard vector costs roughly 6 KB per row of pure float32 payload. A million chunks is around 6 GB before you add a single index, and an HNSW graph then adds its own memory-resident structure on top. None of that scaling is hidden: pgvector’s own scaling guidance names the two moves that shrink the working set — use halfvec instead of vector for tables, and use binary quantization for indexes with re-ranking at search time.

Lever one: store fewer bytes per vector

• Half precision. halfvec halves storage to 2 bytes per dimension and raises the dimension ceiling. pgvector’s plain vector type tops out at 2,000 dimensions, while halfvec handles up to 4,000 and bit up to 64,000. That matters the moment you adopt a model whose default output is 3,072 dimensions: it will not fit in a vector column at all.
• Binary quantization plus re-ranking. binary_quantize() reduces vectors to bits for the index, which is a large storage and traversal win — provided you re-rank the shortlist against the full-precision vectors. Skipping that re-ranking step is how teams ship a ten-times-faster index and a visibly worse product.
• Dimension truncation. Third-generation OpenAI embedding models accept a dimensions parameter, and newer models are increasingly trained so that truncated embeddings stay useful — the Matryoshka approach. Cutting 1,536 dimensions to 512 is roughly a 3x storage reduction. It is also a recall decision, which is why it belongs in an evaluation harness rather than a config file.

Lever two: match the index to the read/write pattern

pgvector offers two approximate index families, and they fail differently. HNSW gives better query performance for a given recall target but has slower build times and a larger memory footprint; it can be built on an empty table because it needs no training step. IVFFlat needs a training pass and is tuned through list and probe counts. Build time for either depends heavily on maintenance_work_mem: builds are significantly faster when the graph fits in memory, and pgvector emits a notice when it no longer does — for example, “hnsw graph no longer fits into maintenance_work_mem after 100000 tuples.” Treat that notice as a capacity alarm, not trivia. Query-time recall is tuned with hnsw.ef_search (40 by default) and with list and probe counts on IVFFlat.

If the index no longer fits comfortably in memory, the on-disk option deserves a benchmark. The pgvectorscale extension adds a StreamingDiskANN index that keeps the graph on disk. The maintainers’ published comparison on 50 million 768-dimension vectors reports 28x lower p95 latency and about 75% lower monthly cost than a comparable managed pod index at 99% recall, self-hosted on a roughly $835/month cloud instance. Vendor-run benchmarks deserve skepticism, but the methodology and harness are public — re-run it against your own corpus before you pay a specialty vendor to solve a problem you may not have.

Budget maintenance as well as queries. After bulk loads, REINDEX INDEX CONCURRENTLY before VACUUM keeps vacuuming on HNSW indexes from dragging for hours.

Lever three: plan for the re-embedding event

Vectors produced by two different embedding models are not comparable, so “just upgrade the model” is never a config change. It is a full-corpus rewrite, and providers schedule it for you. Published deprecation policy from one major provider commits to at least six months’ notice for generally available models, three months for specialized variants, and as little as two weeks for preview models. The practical rule that follows: never point a production index at a preview embedding model, and keep the re-embedding job written and tested before you need it.

The migration pattern that works:

1. Add a second embedding column and a model/version column alongside the existing one, and dual-write while queries still read the old column.
2. Backfill in batches. Because pgvector indexes can be built after the data is loaded, load first and index second — it is faster and it keeps write amplification off the hot table.
3. Evaluate against a frozen golden query set with known-correct chunks. Compare recall@k and p95 latency, not opinion.
4. Cut over, then drop the old column and its index. An unused HNSW index is pure memory cost with no benefit.

Store embedding_model, embedded_at, and a content hash per chunk. Those three columns buy you incremental refresh — re-embedding only the chunks that actually changed — instead of a full-corpus job every night. In most deployments that single change saves more than every index parameter combined.

The three numbers to watch

Track bytes per vector, index build and refresh duration, and recall@k on a fixed query set. When one of them moves, you know which lever to pull, and you know it before the bill arrives rather than after. Retrieval infrastructure has a long tail of exotic optimizations, but this short list is where the money and the latency actually live.

For teams running a knowledge base per client, that reduces to one question worth answering at onboarding: what triggers a re-embed, and who owns the refresh cadence? A stack with clear answers stays cheap for years. One without them gets rebuilt from scratch every time a provider ships a new model version — and pays full price for the privilege.