Building AI-Powered Search: Semantic Search vs Keyword Search

Your users type "affordable running shoes for flat feet" into your search bar and get zero results — because none of your product descriptions contain those exact words. Meanwhile, a competitor's search returns the right products instantly. This is the gap between keyword search and semantic search, and it's one of the most common reasons engineering managers get pulled into "why is our search so bad?" conversations. The good news: modern embedding models and vector databases have made AI-powered semantic search practical to build, and a well-designed hybrid architecture often outperforms either approach alone. This article walks through how to design, build, and validate a search system that understands intent — without over-engineering it.

• Background / Why This Matters
• Prerequisites and Planning
• Step-by-Step Implementation
• Testing and Validation
• Common Mistakes / What to Avoid
• Frequently Asked Questions
• Conclusion

Background / Why This Matters

Traditional keyword search — powered by engines like Elasticsearch, OpenSearch, or a Postgres full-text index — matches documents based on the literal tokens a user types. It uses ranking functions like BM25 to score how well terms overlap. This works well when users know the exact vocabulary of your content. It fails badly when they don't.

Consider a support knowledge base. A user searches "app keeps crashing on startup." Your documentation title is "Resolving application initialization failures." There's almost no lexical overlap, so keyword search ranks it poorly or misses it entirely. Semantic search solves this by converting both the query and your documents into embeddings — dense numerical vectors that capture meaning — and measuring how close they are in vector space. "Crashing on startup" and "initialization failures" land near each other because they mean nearly the same thing.

But semantic search has its own weaknesses. It struggles with exact matches: product SKUs, part numbers, proper nouns, and rare acronyms. Ask a pure vector system for "error code E-4471" and it may confidently return semantically similar but wrong results. This is why the industry has largely converged on hybrid search — combining keyword (sparse) and semantic (dense) retrieval to get the strengths of both.

Where each approach wins

Scenario	Keyword (BM25)	Semantic (Vector)	Hybrid
Exact IDs, SKUs, codes	Excellent	Poor	Excellent
Natural-language questions	Weak	Strong	Strong
Synonyms & paraphrasing	Poor without tuning	Strong	Strong
Multilingual queries	Poor	Good (with right model)	Good
Infrastructure cost	Low	Moderate–High	Moderate–High
Explainability	High	Low	Moderate

Takeaway: Don't frame this as "keyword vs. semantic." Frame it as "which retrieval signals does my query distribution actually need?" Most real applications need both.

Prerequisites and Planning

Before writing any code, get clear on four things. Skipping this planning stage is where most search projects go over budget.

1. Understand your query distribution

Pull a few thousand real queries from your logs. Categorize a sample manually: How many are exact lookups? How many are natural-language questions? How many contain typos or code strings? If 80% of your traffic is exact product-code lookups, you may not need semantic search at all. If users are asking full questions, semantic retrieval will move the needle significantly.

2. Choose your embedding model

The embedding model turns text into vectors and is the single biggest quality lever. Common choices:

• OpenAI Embeddings (text-embedding-3-small / text-embedding-3-large) — strong general-purpose quality, easy API, pay-per-token. The "large" variant produces higher-dimensional vectors and better accuracy at higher cost.
• Cohere Embed (embed-v3) — excellent for retrieval, strong multilingual support, and a useful "input type" distinction between search queries and documents.
• Open-source models (e.g., the BGE or E5 families) — run them yourself for data-residency or cost reasons, at the price of managing GPU infrastructure.

A practical rule: start with a managed API model (OpenAI or Cohere) to validate the product, then evaluate self-hosting only if cost or compliance forces it.

3. Choose your vector store

Option	Best for	Notes
Pinecone	Fully managed, fast to ship	Serverless offering, minimal ops, native hybrid support
Weaviate	Teams wanting hybrid + flexibility	Built-in hybrid search and reranking modules, self-host or cloud
pgvector (Postgres)	Small–medium datasets already on Postgres	No new infrastructure; combine with existing full-text search
OpenSearch / Elasticsearch	Teams already invested in it	Supports dense vectors + BM25 in one engine

4. Define success metrics upfront

Decide how you'll measure "better" before building. Useful metrics include Recall@k (did the right result appear in the top k?), Mean Reciprocal Rank (MRR), and NDCG for ranked relevance. Build a small labeled evaluation set — even 100–200 query/relevant-document pairs — so you can compare approaches objectively rather than by vibes.

Takeaway: Pick your embedding model, vector store, and evaluation metrics before implementation. These three decisions determine 90% of your outcome.

Step-by-Step Implementation

Here's a pragmatic build sequence for a hybrid semantic search system. Each step is independently shippable, which lets you demonstrate value early.

Step 1: Prepare and chunk your content

Embeddings work best on coherent, appropriately-sized pieces of text. For documents, split into chunks of roughly 200–500 tokens with a small overlap (10–15%) so context isn't cut mid-thought. For structured records (products, tickets), build a concatenated text representation of the most searchable fields — title, description, key attributes. Store the original ID and metadata alongside every chunk so you can filter and display results later.

Step 2: Generate and store embeddings

Batch your content through your chosen embedding API. With OpenAI or Cohere, send documents in batches to reduce overhead and respect rate limits. Store each vector in Pinecone or Weaviate along with metadata (source ID, category, timestamp, and the raw text). Keep the model name and version in your metadata — when you upgrade models later, you'll need to re-embed everything, and versioning saves confusion.

Step 3: Build the keyword index in parallel

Index the same content in a BM25-capable engine — OpenSearch, or Postgres full-text if you're using pgvector. This is your sparse retrieval leg. Don't skip it; it's what catches exact IDs and rare terms the vector model fumbles.

Step 4: Implement hybrid retrieval

At query time, run both retrievers and fuse the results. Two common fusion strategies:

1. Reciprocal Rank Fusion (RRF): Combine result lists by rank position rather than raw scores. It's simple, robust, and doesn't require normalizing incompatible score scales. A great default.
2. Weighted score fusion: Normalize each system's scores and blend with a tunable alpha (e.g., 0.6 semantic, 0.4 keyword). More control, more tuning effort.

Weaviate and Pinecone both offer built-in hybrid modes that handle much of this for you, which is worth using before rolling your own fusion logic.

Step 5: Add a reranking stage

Retrieval gets you a candidate set of, say, the top 50 results. A reranker — like Cohere Rerank — then re-scores those candidates against the query using a more expensive cross-encoder model that reads query and document together. This dramatically improves the ordering of the final top 5–10 results shown to users. The pattern is: retrieve broadly and cheaply, then rerank precisely. This two-stage design is where a lot of quality comes from and it's frequently underused.

Step 6: Add filters and business logic

Real search needs metadata filtering — in-stock only, user's language, date ranges, access permissions. Apply these as pre-filters in your vector store when possible so you're only ranking eligible results. Layer business rules (promoted items, freshness boosts) after reranking, not before.

This end-to-end pipeline — chunking, embeddings, hybrid retrieval, reranking, and filtering — is exactly the kind of system the Halkwinds AI & ML team builds and integrates into existing product stacks, so teams don't have to assemble every component from scratch.

Takeaway: Ship in stages. Semantic retrieval alone is a visible win; hybrid + reranking is the production-grade finish.

Testing and Validation

Search quality is deceptively easy to feel good about and hard to actually measure. Use your labeled evaluation set from the planning phase to run offline comparisons: keyword-only, semantic-only, hybrid, and hybrid+rerank. Track Recall@10 and NDCG for each configuration so you can prove improvement rather than assert it.

Offline evaluation

Run every candidate configuration against the same query set and record metrics side by side. This is how you justify the added cost of a reranker or a larger embedding model — you show the recall and ranking gains numerically.

Online evaluation

Offline metrics don't capture everything. Once live, watch behavioral signals: click-through rate on top results, zero-result rate, query reformulation rate (users retyping suggests failed searches), and result dwell time. A/B test changes rather than deploying globally. Research and practitioner reports consistently suggest