RAG is not just vector search

Retrieval-augmented generation lets a model answer from organizational knowledge: procedures, contracts, policies, specifications, and internal records. Many first versions stop at a simple pipeline. Embed the documents, store the vectors, fetch the nearest passages, and hand them to the model.

That can look strong in a prototype. In production, a similar passage is not always the authoritative one. A relevant document may be outdated, closed to this user, incomplete, or in conflict with another source. A correct passage can lose the sentence that qualifies it when it is cut into chunks. A model can still write a plausible answer the evidence does not support.

Enterprise RAG is a knowledge-access and evidence system. It has to coordinate ingestion, semantic and lexical retrieval, access control, ranking, provenance, freshness, context, generation, evaluation, and monitoring. The result we want is the right evidence, from the right source, for the right user, at the right time, and an answer someone can check.

Two paths, plus a control plane

Ingestion collects new and updated content, parses it, keeps its structure, chunks it, adds metadata, and indexes it for both embeddings and lexical search. The updated index is validated before it is served.

A query starts by checking identity and permissions. The system interprets the request, searches with semantic and keyword retrieval, narrows the results with metadata and access policy, ranks what remains, and gives the selected evidence to the model. Citations are checked where that is feasible, and the activity is recorded.

Evaluation sets, lineage, index freshness, access-policy sync, model and prompt versions, telemetry, security tests, and quality monitoring sit across both paths. Ingestion and query serving should be able to fail and scale separately. Reindexing a corpus should not take the query path down. A new embedding model should not be mixed silently into an index built with the old one.

Chunking keeps the meaning the answer depends on

A chunk that is too large brings in material the question does not need and spends the context window. A chunk that is too small can drop a definition, a table header, a warning, or the exception that changes the procedure.

Fixed-size chunks, with optional overlap, are a reasonable baseline for uniform text. Structure-aware chunks follow headings, paragraphs, tables, and sections, which matters for manuals and policies. Semantic chunks follow topic changes, at extra cost that evaluation has to justify. Hierarchical chunks keep passage, section, and document, so a precise hit can expand to its parent when the answer needs the surrounding rule.

For an incident manual, indexing the remediation steps without the severity table or the production exception can lead the model to recommend the wrong action. Keep headings, tables with their headers, procedures, warnings, and parent links. Attach document id, section path, version, and source location. Treat chunk size and overlap as settings you measure, against recall, precision, redundancy, citation accuracy, latency, and tokens.

Hybrid retrieval

Embeddings find conceptually related passages when the question and the document use different words. They miss a lot of enterprise queries that depend on an error code, a clause number, a firmware version, or a product identifier. Lexical search, such as BM25 or an inverted index, is strong on those exact terms.

Run both, then fuse the lists. Reciprocal rank fusion is one method: score each candidate by summing, across result lists, one divided by a constant plus that candidate’s rank. It combines positions without pretending the raw scores are comparable. Normalize identities, drop duplicates, enforce authorization, and keep which channel retrieved the hit so you can evaluate it.

Rewriting or splitting a question can raise recall. “Why did reconciliation fail after deployment?” may need searches for reconciliation errors, the deployment change, and transaction-state transitions. Expansion costs more and can add noise, so keep the original query and make every derived query traceable.

Metadata, reranking, and provenance

Metadata carries what similarity cannot: version, business unit, document type, effective date, approval status, tenant, jurisdiction, and whether the source is authoritative. An engineer asking for the approved production procedure should not receive the draft, the staging guide, or last year’s version just because the wording is close. Metadata itself needs owners and validation. A wrong effective date will hide the right document or promote the wrong one.

Filter before or during retrieval when you can, so unauthorized or ineligible candidates never enter the set. Filtering only afterward can wipe the top hits and leave nothing to replace them. Metadata filtering supports access control. It does not replace it.

Reranking looks harder at a smaller set. A cross-encoder, for example, scores the query and a passage together. Rank for answer-directness, exact identifiers, authority, freshness, completeness, and diversity. Three near-copies of one summary are weaker evidence than the policy statement, the exception, and the operational record. A high relevance score must never override a missing permission.

Every indexed passage should keep a stable source id, a location, a version, an ingestion time, a pipeline version, and an integrity reference. At query time, record what was eligible, what was selected, and which sources support the answer. Tie claims to passages when you can. A citation can point at a real document that does not support the sentence beside it, so citation quality is its own check. When evidence is missing or in conflict, say so, ask for clarification, or abstain.

Access and freshness

The serious failure is putting a restricted document into the model’s context because it ranked well. Authenticate the caller, resolve tenant and authorization, restrict retrieval to eligible content, and apply the same rules to citations, previews, caches, conversation memory, and later tool calls. The model may interpret the question. It does not decide who may read a document.

Watch stale permissions, shared caches, cross-tenant indexes, instructions hidden inside retrieved documents, sensitive data in traces, and slow revocation. Retrieved content is data. It does not become policy because the model read it.

A correct passage can still be the wrong answer if the source has moved on. Track source modification, last successful ingestion, last successful indexing, version, effective dates, deletion, and the index generation. Deleting a file in the source system is incomplete if a copy remains in a vector index, a lexical index, or a cache. For version-sensitive questions, retrieve the version that matches the context, such as the firmware actually installed. If you cannot establish that version, say so.

Freshness objectives are separate from query latency. A static handbook can lag longer than an incident feed or a compliance policy. Measure the delay from an authoritative update to retrieval, the delay to revoke access, and the share of indexed records that match the current source.

Evaluate the system, then operate it

A few impressive demo questions do not make the system ready. Build an evaluation set of realistic questions, expected evidence, versions, and authorization contexts. Include exact identifiers, paraphrases, multi-document questions, exceptions, conflicts, obsolete versions, questions that should not be answered, and users with different access.

Judge retrieval on its own, with recall, precision, reciprocal rank, and graded ranking where relevance has levels. Judge answers for correctness, faithfulness to the evidence, completeness, citation accuracy, abstention, and conflict handling. Automated judges need calibration against human review, especially where the decision matters.

Security tests are negative tests: another tenant’s data, a paraphrased request for a restricted document, a stale permission, a shared cache, a malicious instruction inside a document. An unauthorized disclosure is a critical failure. Average relevance does not offset it.

Also measure latency at the median and the tail, retrieval and reranking time, tokens, indexing failures, cache invalidation, and freshness. Keep evaluating after release with regression tests, sampled reviews, and controlled rollouts.

Treat indexes as versioned derived data. Record the parser, the chunking settings, the embedding model, and the index generation. If authorization cannot be established, do not fall back to open retrieval. If a reranker is down, a validated fallback may be acceptable for a low-risk informational query. If freshness cannot be verified for a time-sensitive policy question, abstaining can be the correct behavior.

Caches have to respect identity, tenant, authorization, document version, and freshness. A cached answer that was valid for one user, or at one moment, can be wrong for the next.

When the workload outgrows one retrieval pass

Complex requests can be split into smaller questions, with each sub-question keeping its own sources. Parent-child retrieval searches passages for precision and expands to the section when context is required. A knowledge graph helps when the question is about dependencies, ownership, or topology, and it is worth the modeling cost. Spreadsheets, tables, and transactional questions are often better served by a governed query or an API than by a vector store. An agent that retrieves in several steps needs a budget, a stopping rule, a trace, and an authorization check on every tool.

Possible business effects include less time spent finding the authoritative document, faster troubleshooting, more consistent use of the current procedure, less exposure to stale or unauthorized knowledge, and an audit trail. Those are things to measure against a baseline. They are not benefits to assume.

The architectural question is whether the system can retrieve the right evidence, show where it came from, confirm that it is current and authorized, and produce an answer whose quality can be measured.