Production RAG: a 50-point checklist for production

Why production RAG breaks in practice

RAG in a demo looks simple: you connect a vector index, add a prompt and get answers based on the documents. But the production environment is much more complicated. The system needs authentication, authorization, document access control, protection against prompt injection, handling of stale content, stable quality evaluation, observability, and control of latency and the token budget. Without that, a prototype can look good on the test set and work poorly on real traffic.

This checklist is intended for the teams that are taking RAG to production. It focuses on the operational risks, not on the presentation promises. Use it at an architecture review, at a security review, before the launch and in the regular operation.

The five areas below are not a formality: each one closes a specific class of failures — and it is exactly those failures that kill RAG on real traffic. Go through the list honestly, without closing an item with a “we have a policy for that” — and you get not a list of wishes but a map of the real gaps.

The five operational areas

Usually production RAG has to be checked in five areas:

1. Security and access control — the user sees only the data they are entitled to.
2. Evaluation and quality — the quality of the retrieval and of the answers is measured regularly and reproducibly.
3. Observability and debugging — every request can be traced from the input to the final answer.
4. Cost and performance control — latency, token spend and infrastructure costs stay under control.
5. Content lifecycle governance — ingestion, freshness, deletion, ownership and rollback are managed explicitly.

The checklist by area

1) Security and access control

• Authentication runs before the request reaches the RAG service.
• Authorization is checked at the document, row, tenant or ACL level, not only in the UI.
• Server-side retrieval filters cannot be bypassed through prompt manipulation.
• The protection against prompt injection works for both the user input and the retrieved content.
• Sensitive fields are redacted or masked before indexing, if that is required.
• Secrets, API keys and credentials do not end up in the prompt or in the logs.
• The data retention policies are defined for the chat transcripts, the traces and the embeddings.
• The external tools and the MCP connectors work only on an allowlist and with a limited scope.
• The tenant boundaries are checked with negative test cases.
• The security review covers jailbreaks, exfiltration and unsafe tool invocation.

Walkthrough: security trimming at the Claims level

In a project with the customer’s documentation — tens of thousands of documents, tens of gigabytes — the access differed at the level of the individual document, not of the folder and not of the role. Every user had their own set of claims, and the answer had to be assembled only from the chunks they were allowed to see.

The naive way — add a filter on the rights directly to the vector index query — gives a quiet degradation: narrowing the candidate sample lowers the recall, and the quality drops without a single error in the logs. On that scale it is especially noticeable: cutting off half of the corpus inside the query turns the index search into an almost full scan. The working scheme is post-retrieval trimming: the candidates are fetched with headroom, trimmed by the claims and only then passed to the generation.

On average 10–20% of the candidates were trimmed out: the rights mostly matched the corpus, so the main work went into the indexing, not into the filtering. At a volume of tens of thousands of documents the embeddings are computed offline as a batch, and the user gets a search over the ready index — and it is exactly that difference between offline and runtime that decides whether the system answers in time.

public sealed class ClaimTrimmingRetriever(
    IRetriever retriever,
    ILogger<ClaimTrimmingRetriever> logger)
{
    // The headroom is needed precisely because of the trimming: with k = 8 and 20 %
    // of the documents trimmed out, about six relevant chunks reach the generation.
    private const int OverFetchFactor = 4;

    public async Task<IReadOnlyList<Chunk>> RetrieveAsync(
        string query, ClaimsPrincipal user, int topK, CancellationToken ct)
    {
        var candidates = await retriever.SearchAsync(query, topK * OverFetchFactor, ct);

        var allowed = candidates
            .Where(c => user.HasPermission(c.Acl))
            .ToList();

        logger.LogInformation(
            "trimmed {Trimmed} of {Fetched} chunks for {User}",
            candidates.Count - allowed.Count, candidates.Count, user.Identity?.Name);

        return allowed.Take(topK).ToList();
    }
}

Three rules follow from that, and they are the ones most often violated:

1. The trimming happens strictly between the retrieval and the generation. A model that has seen an extra chunk has already received the data. A check “after the generation” is not a protection, it is a confession of a leak.
2. The share of the trimmed is a metric, not a log line. If it grows, the ACL and the corpus have diverged: the documents are there, but there are no rights to them. That is a signal to look into the quality of the source, not to turn the threshold.
3. The access scope is part of the cache key. Otherwise an answer assembled for one set of rights is handed to another user.

// The access scope is part of the key, not a check after the read.
var cacheKey = $"{scope.TenantId}:{scope.PermissionSetHash}:{query}";

2) Evaluation and quality

• There is a gold set of typical questions and the expected results.
• The retrieval metrics are tracked, for example recall@k or MRR.
• The answer quality is evaluated separately from the retrieval quality.
• The tests for hallucination and groundedness are run before the release.
• The empty-answer, partial-answer and wrong-answer rates are measured.
• Human review is enabled for the high-risk requests.
• The regression tests cover the chunking, the embeddings, the prompts and the rerankers.
• The online feedback is collected in a useful form, not only as thumbs up/down.
• The evaluation is stratified by language, tenant, content type and request type.
• The release criteria are documented and mandatory for the rollout.

3) Observability and debugging

• Every request has a trace ID that links the input, the retrieval, the reranking and the generation.
• The logs show the model version, the embedding version, the index version and the prompt template version.
• The retrieval candidates and the scores are available for debugging.
• The tool calls and the external calls are logged together with the latency and the status.
• The errors distinguish a retrieval miss, a timeout, a guardrail block and a model error.
• The dashboards show the latency percentiles, the error rates, the token usage and the cache hit rates.
• The sampling is configured so that the sensitive content does not end up in the logs more than necessary.
• There are alerts on a latency regression, a cost spike and a quality drop.
• The postmortem allows you to reconstruct the exact path of the answer.
• The on-call team knows where to look for the cause of a retrieval quality drop.

4) Cost and performance control

• The token budgets are defined per request, per tenant and per workflow.
• The context window is minimally sufficient for the task.
• The chunking strategy reduces the wasteful retrieval and the duplication of the context.
• The reranking is enabled only where it really improves the quality.
• Caching is used for the embeddings, the retrieval results and the safe answer templates.
• The ingestion, the reindexing and the evaluation are run as batch jobs if the latency is not critical.
• There is a fallback for a slow, unavailable or too expensive model.
• The model choice is based on the criticality of the task, not on the maximum size.
• The search latency and the generation latency are measured separately.
• The cost per answer and the cost per resolved query are reviewed regularly.

5) Content lifecycle governance

• A source of record is defined for every corpus.
• The ingestion pipeline validates the schema, the encoding and the deduplication.
• The stale content is reindexed or expires on a clear schedule.
• The deletion requests propagate to the source data, the indexes, the caches and the logs, if that is required.
• Every corpus has an owner and an escalation path.
• There is versioning for the documents, the embeddings, the prompts and the policies.
• The rollback procedures are tested for bad uploads and bad index builds.
• The corpus drift is tracked as the content changes.
• The localization rules and the language-specific content are defined.
• Outdated documents must not silently influence the answers.

Decision matrix

Diagram description

Make a simple left-to-right diagram of five blocks: User request → Retrieval with ACLs → Rerank/guardrails → LLM generation → Logged response and feedback. Under the retrieval show the index, the metadata filters and the policy checks. Under the generation show the token budget and the citation checks.

FAQ

1. Do all five areas have to be closed before the launch?
For production — yes. For a narrow pilot exceptions are possible, but every exception must have an owner, a deadline and a mitigation plan.

2. Is vector search enough for RAG quality?
Usually not. More often you need metadata filters, reranking, prompt discipline and an evaluation cycle.

3. How do you know the hallucinations are really going down?
Look at the groundedness, the citation accuracy and the human-rated correctness on a fixed gold set and on samples of the real traffic.

4. Where should the access control live?
At the service boundary and inside the retrieval. Checks in the UI alone are not enough.

5. How often does the system have to be re-evaluated?
After significant changes to the data, the prompts, the embeddings, the rerankers or the model version — and on a regular schedule to track the drift.

Related materials

• Enterprise AI architecture: four decisions that decide everything
• RAG on .NET: an architecture that survives production
• .NET and Azure for AI systems: a reference architecture
• Taking an AI PoC to production: 90 days by phases

CTA

Ready to strengthen your RAG stack?

If you are taking RAG from prototype to production, use this checklist as a launch gate and as a template for the regular review.

A technical audit will help you run the checklist against your own system and get a breakdown with priorities — we will return a list of the real gaps, not a general list of wishes.