Retrieval infrastructure picked up a genuinely new option this month. On October 6, Google DeepMind released EmbeddingGemma 2, an Apache 2.0 embedding model that runs on the device rather than inside a vendor’s cloud. That single change shifts both the economics and the privacy story of every RAG pipeline built over client documents.
What shipped: an embedding model that lives on the hardware
EmbeddingGemma 2 is built on the Gemma 4 architecture and carries 740 million parameters, but its parts are modular: a 270M text stack, an optional 170M vision encoder, and a 300M audio encoder. Load text only and you carry roughly a quarter of the weights. It maps text, code, images, video frames, and audio into one shared 768-dimensional space, with an 8K token context window — enough for about 5.5 minutes of audio or 29 images in a single pass.
Two details matter if you pay for storage. First, quantized on a Pixel 11 Pro, the text-only configuration needs about 191MB of active memory; the full multimodal model lands near 567MB. Second, the model uses Matryoshka Representation Learning, so the native 768-dimension vectors can be truncated to 512, 256, or 128 dimensions and re-normalized — Google reports up to a 6x reduction in local vector storage. The weights are on Hugging Face, under a commercially permissive license.
The real change is where the embedding step runs
Until now the embedding step was usually a metered API call. Every chunk, every backfill, and every re-index cost tokens and round-trips, which quietly shaped architecture: you embedded once and avoided re-embedding, you cached aggressively, and you accepted that client source material left your premises.
A competent model this small removes the meter. Embedding becomes a local job: no per-token cost, no network hop, and the corpus never leaves the machine. For agencies handling contracts, regulated documents, or recorded calls, that is a compliance argument rather than a curiosity.
The trade-off is real. You now own model lifecycle: no managed upgrade path, no autoscaling endpoint, and a better model next quarter means you plan the re-embedding migration yourself. Benchmarks do not transfer either. A jump on a public coding-retrieval benchmark says nothing about whether the model separates your client’s product records, so quality has to be measured on your own corpus.
Retrieval quality is an architecture problem, not a model problem
The sharpest framing of the last month came from LlamaIndex’s head of engineering, George He, in the company’s session on building a retrieval harness for enterprise agents. His argument is that grep-versus-embeddings is a false binary, and scale decides. At 100 to 1,000 files, letting an agent traverse and read the corpus locally is usually good enough. At thousands to millions of documents it collapses: you cannot read every file on every query without burning tokens, so you need pre-indexed retrieval to point the agent in the right direction, plus grep-style tools for precision. He recommends hybrid retrieval — keyword and semantic results fused, then the top 100 re-ranked — and is candid that keeping a pre-indexed corpus fresh is the unsolved part of the design.
The lesson for anyone swapping in a local embedder: a better encoder does not fix retrieval. A smaller model with a real fusion and re-ranking layer beats a larger model with naive top-k. When hybrid search still returns the wrong chunks, the fault is usually in the query path rather than the encoder — which is why we treat quantization and BM25 fusion as one decision instead of two.
Where the vectors should live
For most client corpora the answer is still PostgreSQL. pgvector keeps embeddings in the same table as the metadata, permissions, and tenancy fields, so there is one query path, one backup, and no second system to keep in sync. Automating that ingest — chunk, embed, upsert, with a model version stamped on every row — is a solved pattern, and the details are in our walkthrough on vector embeddings in PostgreSQL for RAG.
Dimension choice is then a budget line. Truncating 768-dimension vectors to 256 cuts the vector payload to roughly a third, and the index graph shrinks and builds faster, at some cost in recall. Confirm that cost on your own queries before you commit; the trade is the same one covered in our notes on cutting index size and re-embedding costs.
One habit pays for itself: store the raw text, the model identifier, the vector dimension, and the chunking parameters alongside every row. That turns a future model swap from a rebuild into a backfill.
Permissions and freshness: where the pipelines actually break
Two failures show up in almost every client deployment. The first is permissioning. Multi-tenant document search needs access filters pushed into the index at ingest time, not applied after retrieval, or an agent will eventually cite another client’s file. The second is freshness. A pre-indexed corpus drifts away from the source system, and stale vectors are worse than missing ones because they look authoritative. Incremental sync, change detection, and re-embedding only what changed are the minimum viable answer.
Local embedding models make that second problem far cheaper to fix, because re-embedding a corpus no longer costs API tokens. It costs a scheduled job.
What we would do first
Choose by scale and sensitivity, not fashion. A small, private corpus can be embedded locally and searched straight out of PostgreSQL. A large, shared corpus needs hybrid retrieval, re-ranking, and an honest freshness pipeline. Then build a 50-query golden set and measure before you tune anything, and resist adding a dedicated vector store until query volume and tenancy genuinely demand one. In our experience the bottleneck is almost never the database — it is chunking, permissions, and sync.
If you want help designing retrieval that fits a client’s data — local or hosted, Postgres or a dedicated store, with the governance to match — start with our services page.