Vector Databases: The New Infrastructure for AI Applications

If you have shipped anything with large language models in the last two years, you have almost certainly run into the same wall: the model does not know your data. It does not know your product catalog, your internal documentation, or last quarter's support tickets. The dominant answer to this problem — retrieval-augmented generation (RAG) — depends on a piece of infrastructure that most engineering teams had never operated before 2022: the vector database. For engineering managers now responsible for AI features in production, understanding how vector databases work, how they fail, and how to size them is no longer optional. It is core infrastructure, on par with your relational database or message queue.

• Background / Why This Matters
• Core Concepts and Architecture
• Implementation Strategy
• Scaling and Operational Considerations
• Common Mistakes / What to Avoid
• Frequently Asked Questions
• Conclusion

Background / Why This Matters

Traditional databases are built for exact matching. You ask for the row where user_id = 4821 and you get it. This model breaks down the moment you want to search by meaning rather than by literal value. A customer typing "my payment keeps bouncing" should surface a knowledge base article titled "Resolving declined card transactions," even though the two share almost no keywords.

Vector databases solve this by storing embeddings — numerical representations of text, images, or audio produced by a model such as OpenAI's text-embedding-3-large or an open-source model like bge-large. Semantically similar items end up close together in a high-dimensional space (often 768 to 3,072 dimensions). Search becomes a geometry problem: find the vectors nearest to your query vector.

This matters for engineering managers for two concrete reasons. First, RAG is now the default architecture for grounding LLMs in private data, and it is far cheaper and faster to deploy than fine-tuning a model. Second, the vector database is frequently the bottleneck in production — for latency, for cost, and for accuracy. Getting it wrong shows up directly as bad answers and angry users.

Takeaway: Treat your vector database as a first-class production system with its own SLOs, not as a bolt-on cache for your LLM pipeline.

Core Concepts and Architecture

At the heart of every vector database is approximate nearest neighbor (ANN) search. Computing exact distances against millions of vectors on every query is too slow, so these systems build specialized indexes that trade a small amount of accuracy (recall) for a large gain in speed.

The indexing algorithms

• HNSW (Hierarchical Navigable Small World): A graph-based index used by Qdrant, Weaviate, and pgvector. Excellent recall and low latency, but memory-hungry.
• IVF (Inverted File Index): Partitions vectors into clusters and searches only the nearest few. Lower memory, slightly higher latency, common in FAISS-backed systems.
• Product Quantization (PQ): Compresses vectors to shrink memory footprint dramatically, at some cost to recall. Often layered on top of IVF.

Distance metrics

Your embedding model dictates the metric. Cosine similarity is the most common for text embeddings, while dot product and Euclidean (L2) distance appear elsewhere. Using the wrong metric silently degrades results — most OpenAI embeddings are normalized and work equally well with cosine or dot product, but always confirm against your model's documentation.

The data flow

1. Ingestion: Documents are chunked, each chunk is embedded, and the vector plus metadata (source, timestamp, permissions) is stored.
2. Query: The user query is embedded with the same model, then used to retrieve the top-k nearest chunks.
3. Generation: Retrieved chunks are inserted into the LLM prompt as context, and the model generates a grounded answer.

Takeaway: The single most common source of broken retrieval is embedding queries and documents with different models or versions. Pin your embedding model version explicitly.

Implementation Strategy

The first architectural decision is whether you need a dedicated vector database at all. If you already run PostgreSQL and your corpus is under a few million vectors, pgvector is often the pragmatic starting point — you get transactional consistency, familiar tooling, and no new system to operate. As scale and query volume grow, purpose-built engines like Pinecone, Qdrant, or Weaviate become worthwhile.

Option	Deployment	Best for	Trade-offs
pgvector	Self-hosted (Postgres extension)	Teams already on Postgres, <5M vectors	Scales less gracefully at very high volume; index tuning is manual
Pinecone	Fully managed SaaS	Teams wanting zero ops, fast time-to-market	Vendor lock-in; cost grows with scale; less control
Qdrant	Self-hosted or managed cloud	Teams needing rich filtering and open source control	You own operations if self-hosting
Weaviate	Self-hosted or managed cloud	Hybrid search and built-in vectorization modules	More moving parts; steeper learning curve

Chunking is where quality is won or lost

How you split documents matters more than which database you pick. Chunks that are too large dilute relevance and waste context tokens; chunks that are too small lose meaning. A common starting point is 300–800 tokens with a modest overlap (10–15%), but the right size depends entirely on your content. Legal contracts, code, and chat logs all behave differently. Test empirically.

Metadata and hybrid search

Pure semantic search struggles with exact terms — product SKUs, error codes, names. Hybrid search combines vector similarity with traditional keyword (BM25) scoring, and it consistently outperforms either approach alone for enterprise data. Weaviate and Qdrant support this natively; with pgvector you combine vector search and full-text search in SQL. Store rich metadata alongside every vector so you can filter by tenant, date, or access level before ranking.

When our team at Halkwinds designs RAG systems as part of our Data & Analytics engagements, hybrid search plus strict metadata filtering for permissions is almost always in the reference architecture — it is the difference between a demo and a system you can put in front of real customers.

Takeaway: Start with pgvector for a proof of concept, invest early in chunking experiments, and default to hybrid search for anything containing structured identifiers.

Scaling and Operational Considerations

Vector search behaves differently from relational workloads, and the surprises tend to appear at scale.

Memory is the dominant cost driver

HNSW indexes typically live in RAM for fast queries. Estimating footprint is straightforward: number of vectors × dimensions × 4 bytes, plus graph overhead. Ten million vectors at 1,536 dimensions is roughly 60 GB before overhead. This is why quantization matters — techniques like scalar or binary quantization can cut memory by 4x to 32x, with recall impact that varies by dataset. Test the recall trade-off before enabling it in production.

Recall versus latency tuning

Every ANN index exposes knobs — ef_search in HNSW, nprobe in IVF — that trade latency for recall. Do not accept defaults blindly. Build an evaluation set of representative queries with known-good answers and measure recall@k as you tune. Research and practitioner reports consistently suggest that untuned indexes leave significant recall on the table.

Keeping the index fresh

Data changes. Documents are edited, deleted, and added. Plan for:

• Incremental upserts so you are not rebuilding the whole index nightly.
• Soft deletes and re-embedding when your embedding model is upgraded — a model change requires re-embedding the entire corpus, which is an expensive and easily forgotten operation.
• Monitoring drift in retrieval quality over time as content grows.

Observability

Instrument retrieval quality, not just latency. Log the queries, the retrieved chunks, and whether the final answer was correct where you can measure it. Without this, silent degradation is invisible until customers complain.

Takeaway: Budget for RAM and re-embedding costs up front, build a recall evaluation harness before launch, and monitor retrieval quality as a product metric.

Common Mistakes / What to Avoid

• Treating retrieval quality as an afterthought. Teams obsess over prompt engineering while feeding the model irrelevant chunks. Fix retrieval first — a great prompt over bad context still produces bad answers.
• Ignoring the metadata/permissions layer. In multi-tenant systems, a vector database without strict pre-filtering can leak one customer's documents into another's answers. This is a security incident, not a bug.
• Over-provisioning a managed service. Pinecone and similar SaaS tools bill by capacity. Many teams pay for far more than they use because they never tuned index parameters or dimensionality.
• Using oversized embeddings. Higher-dimensional embeddings cost more to store and search. Newer models like text-embedding-3-small support dimension reduction with minimal quality loss — use it.
• No evaluation set. Shipping RAG without a golden dataset of questions and expected sources means you are flying blind on the metric that matters most.
• Re-embedding mismatch. Upgrading your embedding model on queries but not documents (or vice versa) quietly destroys relevance.

Takeaway: The failure modes here are mostly operational and organizational, not algorithmic. Put guardrails around permissions, evaluation, and model versioning.

Frequently Asked Questions

Do I really need a dedicated vector database, or can I use pgvector?

For many teams, pgvector is genuinely sufficient — especially under a few million vectors and moderate query rates. It keeps your architecture simple and avoids a new operational surface. Reach for Pinecone, Qdrant, or Weaviate when you need very high query throughput, advanced filtering,