Building Semantic Search

Semantic search helps people find relevant information even when their query does not use the same words as the content.

Vector search is one technique for making that possible. It compares numerical representations of meaning—called embeddings—instead of comparing only literal words.

The three ideas are related, but they are not identical:

• Semantic search is the product behavior: results are ranked by meaning and intent.
• An embedding is a numerical representation of an item, such as a sentence, document, image, or product.
• Vector search compares those representations and finds the closest ones.

A useful search experience may combine vectors with keyword matching, filters, permissions, business rules, and reranking. The embedding model and vector store are only parts of the system.

Start with a simple fruit example

High-level idea: embeddings place items with related meaning near one another in a multi-dimensional space.

Practical example

Imagine a catalog containing:

Apple — crisp, sweet, and often red
Strawberry — a small berry with a sweet flavor
Lemon — a yellow citrus fruit with a sharp taste
Wrench — a steel tool used to tighten bolts

A user searches for:

sweet red fruit

A literal search may find the apple because the words appear directly. It may rank the strawberry poorly because its description does not say red or fruit, even though a strawberry is relevant to the intent.

An embedding model converts the query and each item into vectors. A greatly simplified, imaginary representation could look like this:

apple       → [ 0.90,  0.72]
strawberry  → [ 0.86,  0.76]
lemon       → [ 0.68,  0.33]
wrench      → [-0.80, -0.35]
query       → [ 0.88,  0.74]

Real embeddings normally contain hundreds or thousands of values, not two. These numbers are illustrative only. They show the idea: the query is close to the apple and strawberry, farther from the lemon, and far from the wrench.

Vector search retrieves nearby candidates. Semantic search turns those candidates into a useful, authorized, and maintainable product experience.

Decide what meaning to embed

High-level idea: retrieval quality depends on the text representation sent to the embedding model. More fields do not automatically produce better meaning.

Useful inputs often include a title, concise description, body text, and descriptive labels. Internal identifiers, timestamps, display markup, and unrelated metadata may add noise.

Practical example

For a knowledge article, build one stable representation:

Title: Resetting a locked account
Summary: Recovery steps when sign-in attempts are blocked
Body: Confirm ownership, issue a recovery link, and expire old sessions
Labels: authentication, account recovery

Normalize whitespace and formatting before embedding it. Keep the same field order for every article. If the body is too large for the model or contains several independent topics, split it into meaningful chunks and retain the source identity for reconstruction.

The practical test is simple: if a field should not influence retrieval, do not include it merely because it exists.

Choose the embedding model for the product

High-level idea: embedding models differ in retrieval quality, language coverage, vector dimensions, input limits, latency, cost, privacy options, and deployment model.

Examples include:

• Alibaba’s Qwen3 embedding family;
• OpenAI’s text-embedding-3-small and text-embedding-3-large;
• Google’s gemini-embedding-001;
• open models that can run on infrastructure controlled by the product team.

I previously used alibaba/qwen3-embedding-0.6b through Vercel AI Gateway. At the time, it was an attractive fit because it was inexpensive and performed well for my use case. That is not a universal recommendation or a claim about current pricing.

Practical example

Evaluate two or three candidate models against the same labeled queries:

Query: "sweet red fruit"
Relevant: apple, strawberry
Not relevant: wrench

Query: "sharp yellow citrus"
Relevant: lemon
Not relevant: wrench

Query: "SKU-1042"
Relevant: exact SKU-1042 item

Compare retrieval quality, latency, and estimated request volume. Select the smallest model that meets the product’s relevance and operating constraints—not automatically the cheapest or largest model.

Store the model identifier and an explicit embedding-space version beside each vector. Change that version when the model revision, dimensions, task or instruction mode, chunking policy, or normalization rules change. An embedding-space change is a data migration: generate a new index, evaluate it, cut over deliberately, and retain a rollback path.

Store and retrieve vectors

High-level idea: a vector store keeps each embedding with the identity and metadata needed to retrieve safe candidates. Start with exact nearest-neighbor search when the corpus and latency allow it. Add approximate indexing only after measurement shows a need.

Practical example with PostgreSQL and pgvector

pgvector adds vector similarity search to PostgreSQL. A generic table might look like this:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE search_items (
  source_id uuid PRIMARY KEY,
  account_id uuid NOT NULL,
  searchable_text text NOT NULL,
  embedding_model text NOT NULL,
  embedding_space_version text NOT NULL,
  content_fingerprint char(64) NOT NULL,
  embedding vector(1024) NOT NULL,
  deleted_at timestamptz
);

CREATE TABLE search_item_access (
  source_id uuid NOT NULL REFERENCES search_items(source_id) ON DELETE CASCADE,
  principal_id uuid NOT NULL,
  PRIMARY KEY (source_id, principal_id)
);

The vector dimension must match the selected model. With a query embedding passed as $1, an authorized cosine-similarity search can be expressed as:

SELECT
  item.source_id,
  1 - (item.embedding <=> $1::vector) AS similarity
FROM search_items AS item
JOIN search_item_access AS access
  ON access.source_id = item.source_id
  AND access.principal_id = $3
WHERE item.account_id = $2
  AND item.deleted_at IS NULL
  AND item.embedding_space_version = $4
ORDER BY item.embedding <=> $1::vector
LIMIT $5;

The account boundary, authoritative access-control join, lifecycle state, and embedding-space version are inside candidate retrieval—before the limit is applied. A product where every account member can access every item may not need the separate access table; products with user-, group-, or document-level visibility need their real ACL or row-level-security predicate here.

For a larger corpus, an HNSW index can reduce retrieval latency:

CREATE INDEX search_items_embedding_hnsw
ON search_items
USING hnsw (embedding vector_cosine_ops);

Approximate search trades some recall for speed and adds tuning work. With pgvector’s approximate indexes, filtering is applied after the index scan. Restrictive account or ACL filters remain safe, but they can produce fewer authorized candidates than requested.

Starting with pgvector 0.8.0, iterative scans can continue through the index when filters remove candidates:

SET hnsw.iterative_scan = strict_order;

Depending on the data boundary, exact search over a filtered subset, partial indexes, or table partitioning may be a better fit. Benchmark approximate results against the exact baseline with representative permissions, filters, and data volume rather than enabling HNSW by default.

Keep keyword and exact search

High-level idea: semantic retrieval solves vocabulary mismatch; it does not make exact matching obsolete.

Keyword, literal, or structured retrieval remains useful for:

• product codes, error codes, names, and identifiers;
• exact or quoted phrases;
• prefixes and spelling-sensitive values;
• filters whose meaning should not be inferred;
• cases where users already know the terminology.

Practical example

For sweet red fruit, semantic retrieval may discover both apple and strawberry. For SKU-1042, an exact identifier lookup should normally win.

Run both retrieval paths over authorized content, then combine their rankings:

keyword_candidates = keyword_search(query, access_scope)
semantic_candidates = vector_search(query_embedding, access_scope)
results = reciprocal_rank_fusion(keyword_candidates, semantic_candidates)

Rank fusion uses position rather than adding raw keyword and vector scores that may have unrelated scales. When an item is absent from one list, its contribution from that list is zero.

This design also provides a natural fallback: keyword search still works when semantic retrieval is unavailable.

Avoid unnecessary embedding calls

High-level idea: embedding the same content or repeated query again wastes time and money. Hashing helps detect identical normalized input; caching lets the system reuse the resulting vector.

A hash does not replace semantic search. It is a change detector and cache key.

Practical example: skip unchanged content

Define an embedding-space version for the complete embedding configuration, then include it in the fingerprint:

space_version = "search-v3"
normalized_text = normalize(title + summary + body + labels)
fingerprint = SHA256(space_version + "\n" + normalized_text)

if stored_fingerprint == fingerprint:
    reuse the stored embedding
else:
    embedding = embed(normalized_text)
    upsert embedding, model_id, space_version, and fingerprint

search-v3 represents the exact model revision, dimensions, task or instruction mode, chunking policy, and normalization rules. Changing any of them creates a new version and prevents incompatible reuse. An edit to a timestamp or display-only field does not trigger a paid embedding call if it is excluded from normalized_text.

Practical example: cache repeated query embeddings

Normalize a query and key the cache by both embedding space and query:

normalized_query = normalize(query)
query_key = SHA256(space_version + "\n" + normalized_query)

if query_embedding_cache contains query_key:
    query_embedding = cached vector
else:
    query_embedding = embed(normalized_query)
    cache query_embedding with a bounded lifetime

Repeated searches such as sweet red fruit can reuse the same query vector. Cache failure should fall back to generating the embedding; it should not make search unavailable.

Caching full result lists is more sensitive than caching query vectors. A result-cache key must include the normalized-query fingerprint, authorization scope, filters, pagination, embedding-space version, ranking-configuration version, and a freshness or permission version. Otherwise, different queries can collide, or a fast cache can become an access-control or staleness bug.

Keep the index synchronized

High-level idea: the initial embedding import is only the beginning. Search quality drifts when the index no longer reflects source content, visibility, or deletion state.

Practical example

Define the lifecycle explicitly:

Create  → normalize content, embed it, and store the vector
Update  → compare fingerprints; re-embed only when meaningful text changed
Delete  → remove or invalidate the vector
Access change → update retrieval metadata immediately; do not wait for re-embedding
Backfill → process existing content in bounded, resumable batches
Repair → retry failed, missing, stale, or wrong-space embeddings

Synchronous indexing gives immediate consistency but adds model latency to writes. Asynchronous indexing keeps writes responsive but introduces a freshness window. Either can work if failures, backlog, and repair are observable.

Apply authorization during retrieval

High-level idea: semantic similarity is not authorization. Access rules must restrict candidates before ranking and limiting, not only after results are returned.

Embeddings are derived data, not anonymization. Review what is sent to a hosted model, how it is retained, and whether self-hosting is required.

Practical example

Suppose two companies each store a document called Incident response plan. A user’s query vector may be close to both. The vector query must filter by the user’s account and visibility rules before selecting the nearest results, as the pgvector query above does.

Test negative cases explicitly:

- private content never crosses account or user boundaries
- revoked access takes effect without re-embedding
- deleted or hidden content is excluded
- result caches cannot cross authorization scopes
- restrictive filters still retrieve enough candidates to rank

Retrieving globally, taking the top 20, and filtering afterward is both unsafe and incomplete: unauthorized candidates may consume the limit and hide relevant authorized results.

Design a graceful fallback

High-level idea: embedding providers, model servers, vector stores, and caches can fail. Search needs a defined degraded mode.

Practical example

Use a bounded flow:

1. Run authorized keyword retrieval.
2. Attempt query embedding and vector retrieval within a timeout.
3. Fuse both rankings when semantic retrieval succeeds.
4. Return keyword results when it fails.
5. Record the degraded mode without exposing internal errors to the user.

The product remains useful while the semantic path recovers. Measure fallback frequency so a silent degradation does not become the permanent operating mode.

Evaluate relevance before scaling

High-level idea: latency tests show whether search is fast; relevance tests show whether it is useful. Tune on one query set and verify on a separate holdout set.

Practical example

Create a small judged dataset that includes:

- natural language: "sweet red fruit"
- paraphrase: "recover access after too many sign-in attempts"
- exact identifier: "SKU-1042"
- ambiguous query: "apple"
- restrictive access filter
- query that should return no result

For each query, label relevant and irrelevant items. Compare keyword-only, semantic-only, and hybrid retrieval using:

• precision among the first results;
• recall of known relevant items;
• the rank of the first relevant result;
• ordering across strongly and weakly relevant items;
• correct rejection of low-confidence queries.

Measure operations as well: embedding and search latency, model errors, timeouts, fallback rate, indexing freshness, cache effectiveness, and estimated embedding volume avoided through reuse.

Offline evaluation does not prove that users succeed. Release behind a reversible flag or to a bounded cohort, define rollback criteria, and observe reformulation, abandonment, result selection, task completion, and direct feedback. Clicks alone do not establish relevance.

Know when not to add semantic search

High-level idea: semantic search is valuable when vocabulary mismatch is a real product problem. It is not a default requirement for every search box.

Practical example

Keep the simpler search path when a small catalog is dominated by exact SKUs, users already find what they need, or the team cannot maintain indexing and relevance evaluation.

Add semantic retrieval when repeated evidence shows queries and content express the same intent with different words—for example, users search for account locked while the support content is titled recover access after failed sign-ins.

A practical implementation sequence is:

1. Define representative queries and relevance expectations.
2. Measure the current keyword or exact-search baseline.
3. Design a stable content representation.
4. Evaluate candidate embedding models.
5. Store and retrieve authorized vectors—pgvector is one option.
6. Combine semantic and keyword results.
7. Add fingerprints, query-vector caching, synchronization, repair, and fallback.
8. Validate user outcomes through a reversible release.

Semantic search becomes useful when it is treated as a maintained product capability. The vector query is only one step; the surrounding decisions determine whether it stays relevant, affordable, secure, and available.