RAG evaluation on .NET: Recall@5, MRR and the release gate

A pilot that works proves nothing

A pilot on a dozen documents almost always looks good. The answers are meaningful, the citations are in place, the client is happy. The problem starts when there are tens of thousands of documents and the questions come not from the demo scenario but from real support.

The real problem is not even the quality of the answer — it is that the team has no way to find out whether it got worse. Changed the chunker — better or slightly worse? Swapped the model — will the wording fall apart? Added a reranker on top of the hybrid search — will it help or hurt?

All of these changes feel the same: “seems to work”. The difference between them is visible only on a fixed set of questions with known correct answers — a Golden Set. Until you have such a set, any change in the pipeline remains a bet.

Eval turns a bet into a check: a fixed set, fixed metrics, a “before and after” comparison. Then it comes to the concrete pipeline: what is in the set, how the four metrics are computed and where the run blocks the release.

Golden Set: 50 questions, not 5

A Golden Set is a set of questions from real support, where for each of them you already know the chunk (or chunks) that contains the correct answer. Not “test questions”, but the ones users actually ask.

The first rule: the questions come from real support, they are not written by developers. A synthetic question is formulated in the terms of the document: it is written by the team that already understands that document. A user asks about their own task and trips up where the words of the document and the words of the question do not match.

The second rule is the size. Five questions do not give a metric: one failed case is minus 20 percentage points, and the Recall@5 = 0.85 threshold stops meaning anything. Fifty questions turn the same miss into two percentage points — that is a signal, not noise. The working range is 50–100 questions: the exact number depends on the task at hand, the composition of the content and the direction of the company, so the range sets the order of magnitude rather than a measured constant. The point is that the set covers dozens of scenarios, not single ones.

Coverage matters more than quantity. The set has to contain different types of requests:

• a fact from a specific document;
• an exact identifier — a contract number, an article code, an error code; full-text search finds those more reliably;
• a question whose answer is assembled from two or three chunks of different documents;
• a question that has no answer in the corpus: the correct behavior is “not found”, not an invention; such a case has no reference chunk, so it does not go into Recall@5 and MRR and is checked separately.

A case is a structure, not a line in a list. The set lives in JSONL, one line — one case:

{"id":"sup-0142","question":"What SLA does the client have under contract 44-A/2024?","expectedChunkIds":["doc-441/ch-06"],"category":"contract","source":"ticket-88213"}

The fields: id — a stable case identifier, question — the question verbatim, expectedChunkIds — the reference chunks, category — the class of the question for the slices, source — a link to the original ticket.

Here is the first technical decision that takes the manual work out of every re-chunking: the reference id is bound to the structure of the document, not to an offset inside it.

bad:   doc-441#offset-5120..5632   — breaks on every re-chunking
good:  doc-441/ch-06                — survives the re-chunking as long as the structure is the same

The format is the document id, the path of the headings and the number of the section. When the chunker changes, the boundaries and the section numbers change, but the path stays, and the old reference ids keep pointing at the right place. What the architecture bound to the set looks like is in RAG on .NET: an architecture that survives production.

The set is not rebuilt silently. The triggers are a new corpus, a change in the chunking schema, a swap of the embedding model, a fix to the prompt. The old ids are mapped to the new ones by the “document + heading path” pair, and a case whose reference chunk has disappeared is marked needs-review and is not deleted: if you delete it, the set quietly shrinks, and the metric gets better simply because the questions are gone. The new ids are fixed in a separate commit, so that the diff of the set is visible in the review.

The set is versioned as a pair: the golden.v3.jsonl file and a manifest next to it — the date, the checksum, who rebuilt it, the versions of the index, the embeddings and the chunking. The old versions are not deleted: they show what has changed.

Metrics and thresholds

Four metrics, four thresholds. The first two are about the retrieval, the second two about the generation:

Why four metrics and not six: Precision@k and nDCG, which often come up next to them in a RAG evaluation, answer the question of the share of relevant items in the results and of the distribution of the positions — it is meaningful when a question has many equally valid correct answers, while in our gate what matters is that the needed chunk lands in the first five and where it lands.

A threshold is not a goal, it is a tolerance boundary. Below Recall@5 = 0.85 the retrieval is broken regardless of how brilliantly the answer is phrased: the model will not save a context that is not in the results. Faithfulness with a threshold of 0.90 is the strictest of the four: an invented clause of a contract is worse than an honest “not found”.

The thresholds are calibrated on your own set and your own judge: you cannot carry them over from another project — both the rubric and the composition of the set change the metric values.

The computation of the retrieval metrics is deterministic — the code is below. The generation metrics are computed by the judge, and it does not see the reference answer: faithfulness checks the grounding in the context, and the reference is needed for the manual labeling.

How it is computed

// The golden set: a question from real support and the id of the chunk with the correct answer.
public sealed record GoldenCase(
    string Id,
    string Question,
    IReadOnlyList<string> ExpectedChunkIds,
    string Category);

// The result of one run: what the results returned and in what order.
public sealed record CaseRun(
    string CaseId,
    IReadOnlyList<string> RetrievedChunkIds,
    double LatencyMs);

GoldenCase is the input of a run, CaseRun is the output of the retrieval for one case. The order in RetrievedChunkIds matters: the position is computed from it, not just the fact of a hit.

public sealed record RetrievalMetrics(double RecallAtK, double Mrr, int Cases);

public static class RetrievalEvaluator
{
    public static RetrievalMetrics Evaluate(
        IReadOnlyList<GoldenCase> goldenSet,
        IReadOnlyList<CaseRun> runs,
        int k = 5)
    {
        var byCase = runs.ToDictionary(r => r.CaseId);

        double recallSum = 0;
        double reciprocalRankSum = 0;

        foreach (var testCase in goldenSet)
        {
            if (testCase.ExpectedChunkIds.Count == 0)
                throw new InvalidOperationException(
                    $"Case {testCase.Id} without reference chunks");

            if (!byCase.TryGetValue(testCase.Id, out var run))
                throw new InvalidOperationException(
                    $"No run for the case {testCase.Id}");

            var top = run.RetrievedChunkIds.Take(k).ToHashSet();

            // Recall@k: the share of the reference chunks that made it into the first k.
            // If there is a single reference chunk, it is 0 or 1 —
            // that is, Recall@k is the same as the hit rate.
            var found = testCase.ExpectedChunkIds.Count(id => top.Contains(id));
            recallSum += (double)found / testCase.ExpectedChunkIds.Count;

            // MRR: 1 / the position of the first reference chunk in the results.
            // Zero means "the reference was not found": the default tuple is empty.
            var rank = run.RetrievedChunkIds
                .Select((id, index) => (id, rank: index + 1))
                .FirstOrDefault(x => testCase.ExpectedChunkIds.Contains(x.id))
                .rank;

            reciprocalRankSum += rank > 0 ? 1.0 / rank : 0.0;
        }

        return new RetrievalMetrics(
            RecallAtK: recallSum / goldenSet.Count,
            Mrr: reciprocalRankSum / goldenSet.Count,
            Cases: goldenSet.Count);
    }
}

The average is computed over the cases, not over the documents: on a small set the metric jumps around, and the gate fires on noise.

Faithfulness is checked in two layers, and the tools of the layers differ. The first layer is deterministic code: it catches two classes of defects — a claim without a reference to a chunk from the passed context, and a reference to a chunk that does not exist. Both are an automatic failure, and the layer costs zero tokens.

// The deterministic part of the evaluation: every claim of the answer must
// reference a chunk that was really passed in the context.
public sealed record GroundingCheck(int Claims, int Cited, int ForeignCitations)
{
    // The share of claims with a correct reference to a chunk from the context.
    // This is not a semantic grounding: the content of the claim is not checked.
    public double CitedRatio =>
        Claims == 0 ? 0 : (double)Cited / Claims;
}

public static class GroundingChecker
{
    public static GroundingCheck Check(
        string answer,
        IReadOnlyList<string> contextChunkIds)
    {
        var known = contextChunkIds.ToHashSet(StringComparer.Ordinal);

        // A paragraph is one claim. The line with the source list is not counted.
        var claims = answer
            .Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)
            .Where(line => !line.StartsWith("Sources:", StringComparison.Ordinal))
            .ToList();

        var cited = claims.Count(claim =>
        {
            var ids = SourceIds(claim).ToList();
            return ids.Count > 0 && ids.All(known.Contains);
        });

        // A reference to a chunk that was not in the context is an invented source.
        var foreign = claims.SelectMany(SourceIds).Count(id => !known.Contains(id));

        return new GroundingCheck(claims.Count, cited, foreign);
    }

    // The reference format in the answer: [chunk-id].
    private static IEnumerable<string> SourceIds(string claim)
    {
        var start = claim.IndexOf('[');
        while (start >= 0)
        {
            var end = claim.IndexOf(']', start + 1);
            if (end < 0) yield break;

            yield return claim[(start + 1)..end].Trim();
            start = claim.IndexOf('[', end + 1);
        }
    }
}

The second layer is the judge. It evaluates not the answer as a whole, but the individual claims: a mixed verdict across the parts is normal, a “partially true” answer is a bad signal for the gate. The rubric is claim-level: the verdict is given only by the passed context — the knowledge of the model does not count as a confirmation. A claim for which the context has no supporting wording is unconfirmed. A contradiction of the context is always an unconfirmed claim, even if it is plausible.

// The judge evaluates not the answer as a whole, but the individual claims:
// a "partially true" answer is a bad signal for the gate.
public sealed record ClaimJudgement(string ClaimId, bool Supported, string Reason);

// The answer is evaluated on a 1..5 scale along two axes.
// The completeness is computed for information and does not go into the metric.
public sealed record AnswerJudgement(
    int RespondsToQuestion,
    int OnTopic,
    int Complete,
    string Reason);

public sealed record JudgeVerdict(
    IReadOnlyList<ClaimJudgement> Claims,
    AnswerJudgement Answer)
{
    // Faithfulness: the share of claims confirmed by the passed context.
    public double Faithfulness =>
        Claims.Count == 0 ? 0 : Claims.Count(c => c.Supported) / (double)Claims.Count;

    // Answer relevancy: the average of the two axes, brought to 0..1.
    // A score of 1 → 0.0, a score of 5 → 1.0.
    public double AnswerRelevancy =>
        ((Answer.RespondsToQuestion + Answer.OnTopic) / 2.0 - 1.0) / 4.0;
}

Answer relevancy is computed by the same judge call, but along another axis of the rubric: RespondsToQuestion — whether the answer answers the question, OnTopic — whether it drifted into an adjacent topic. The third axis, the completeness, does not go into the metric.

The key difference: the deterministic code catches a reference, but not the content. A claim that references the right chunk but does not follow from it passes the deterministic check — only the judge can catch it. Counting one thing as another is a mistake: the metric is green and the answers are garbage.

Eval as a release gate

An eval run is not a report that somebody will look at. It is a step of the Azure DevOps pipeline with an exit code: broke the threshold — the release did not go out.

The order is not decorative. Step 2 goes before step 3 for the economics: if the needed chunk is not in the results, the answer is wrong anyway. Step 4 goes before step 5: a cheap filter for zero tokens.

public sealed record GateRule(string Metric, double Minimum, double? Baseline = null);

// The summary for one case: did the reference make it into the results and which rule was broken.
public sealed record CaseResult(
    string CaseId,
    bool Passed,
    string? FailedRule);

public sealed record EvalReport(
    IReadOnlyDictionary<string, double> Metrics,
    IReadOnlyList<CaseResult> Cases);

public static class ReleaseGate
{
    // The gate is not "the metric is above the threshold", but "above the threshold and not worse than the baseline".
    // A metric can stay above the threshold and still drop — the threshold does not catch that.
    public static bool Passes(EvalReport report, IReadOnlyList<GateRule> rules)
    {
        foreach (var rule in rules)
        {
            if (!report.Metrics.TryGetValue(rule.Metric, out var value))
                return false;

            if (value < rule.Minimum)
                return false;

            if (rule.Baseline is { } baseline && value < baseline)
                return false;
        }

        return true;
    }
}

The gate has two conditions: a threshold and a baseline. The threshold catches a failure, but it does not catch a degradation inside the tolerance — a metric can stay above the threshold and still drop. That is why a rule has a baseline, and the comparison is only possible within one version of the set and the index.

// The thresholds of the gate live in one place.
public static class Rules
{
    public static readonly IReadOnlyList<GateRule> All =
    [
        new("recall_at_5", 0.85),
        new("mrr", 0.75),
        new("faithfulness", 0.90),
        new("answer_relevancy", 0.85)
    ];
}

// The breakdown by case: the case from the set, the results and the verdicts of the judge.
public sealed record CaseDetail(
    GoldenCase TestCase,
    IReadOnlyList<string> RetrievedChunkIds,
    string Answer,
    IReadOnlyList<ClaimJudgement> Verdicts);

// The build artifact: the aggregates plus the breakdown by case.
public sealed record EvalArtifact(
    EvalReport Report,
    IReadOnlyList<CaseDetail> Cases);

// The eval run in the Azure DevOps pipeline: one step, one exit code.
public static async Task<int> RunEvalAsync(EvalArtifact artifact)
{
    await File.WriteAllTextAsync(
        "artifacts/eval-report.json",
        JsonSerializer.Serialize(artifact));

    // 0 — the gate passed, 1 — the threshold was broken or there is a regression against the baseline.
    return ReleaseGate.Passes(artifact.Report, Rules.All) ? 0 : 1;
}

Every run records five versions: IndexVersion, EmbeddingVersion, PromptVersion, ChunkingVersion, GoldenSetVersion. Without them the delta between the runs is unreadable: it is not clear what exactly has changed.

The report is a build artifact, and it shows which case broke. EvalReport carries the aggregates and, per case, the passed / failed result with the broken rule — that is enough for the gate. The breakdown is collected separately in EvalArtifact: the question, the category, the reference and retrieved chunk ids, the text of the answer and the verdict of the judge for each claim with the reason. Plus the diff against the baseline: which cases dropped, which ones were added. Without that artifact the team only sees “a red run”.

The minimal mandatory option is one full gate before the release, steps 0–7. A practical extension is two levels: on every pull request run only the retrieval part (steps 0–2 and 6–7), without the LLM and the judge — it is cheap, and the full gate stays on the release candidate.

Offline and online: what is measured where

The offline run on the Golden Set is reproducible: it goes before the release and catches the regressions in the chunking, the embeddings, the prompts, the reranker and the context assembly. The cost and the latency before the release are computed there as well.

The eval run goes with the cache disabled: the cache on the query vector closes 30–50% of the requests, and if the run goes through it, the metric measures the cache, not the pipeline.

But the offline run does not see the requests that are not in the set: it covers the typical scenarios and stays silent about everything else. That is why production needs a second loop — online monitoring. It gives what the Golden Set does not have:

• cache hit rate: the share of the requests closed by the cache on the query vector; in the loop where we did this, 30–50% of the requests never reached the generation;
• cost per request: the routing by complexity reduces it three to four times;
• p95 latency: the latency of the answer, not the cost, so it is read separately from the cost;
• the share of the candidates trimmed by the rights — 10–20%; if the share grows, the rights and the corpus have diverged and it is time to look into the quality of the source, not to turn the threshold;
• the share of the requests without an answer — the first signal that the retrieval has stopped finding things;
• the share of the requests that went to a human: if it grows, the system has stopped closing the question on its own;
• drift: the distribution of the categories of the live questions against the composition of the golden set — a divergence shows which classes of questions are still not covered.

One thing deserves to be said separately: the judge is needed not only offline. In production it runs on a sample of the answers and on all the answers without citations: offline gives the metric on the labeled data, online catches the drift on the unlabeled data.

The key difference that is worth stating plainly: the online metrics have no labels. A live request has no reference, so there is nothing to compute Recall@5 from in production — online monitoring is a proxy, sampling and feedback. Offline is the only place where the quality is really measured.

Where we got it wrong

Three decisions that we reconsidered after the system went to production.

The filter by rights inside the vector index query → post-retrieval trimming. At first the access was filtered directly in the index query. The failure mode is quiet: narrowing the candidate sample lowers the recall, and there is not a single error in the logs — it just “got a bit worse”. If the rights do not match the corpus and, say, half of the candidates are trimmed out, the index search in the limit turns into an almost full scan. Now the candidates are fetched with headroom and cut by the rights already after the retrieval. 10–20% of the candidates are trimmed out: the rights mostly match the corpus, so the main work went into the indexing, not into the filtering.

The embeddings at runtime → an offline batch. On tens of thousands of documents and tens of GB, recomputing the embeddings on the fly does not fit into the response time. The embeddings are computed as an offline batch, the user gets a search over the ready index — and the response time stops depending on the recomputation of the corpus.

A fixed window of 512 tokens → chunking by structure. A document cut by length loses its meaning at the seams: half of the answers did not find the context that was in the neighboring chunk. Now we cut by structure — for contracts the parties, the subject, the deadlines, the liability, the appendices; for the technical documentation the sections and subsections, the code blocks whole. The choice of the chunker affects the quality of the answer by roughly 80% — that is the main lever. That is exactly why the reference ids in the Golden Set are bound to the structure of the document and not to the offsets: otherwise every change of the chunker would break the set.

What we kept: hybrid search with RRF in Azure AI Search (K = 60) on top of the vector and full-text results, routing by complexity and the semantic cache on the query vector — a measurable effect without a rewrite. More about the pipeline itself — Building a RAG pipeline on .NET: a practical walkthrough.

Checklist before production

• A Golden Set of 50–100 questions from real support, with a version, the reference chunks and the category for each case
• The reference ids are bound to the structure of the document, not to the offsets
• Recall@5 above 0.85 on the set
• MRR above 0.75 on the set
• Faithfulness above 0.90 on the set
• Answer relevancy above 0.85 on the set
• The eval run executes in the Azure DevOps pipeline and blocks the release when a threshold drops
• The gate checks the baseline, not only the threshold
• The report of the run is saved as a build artifact, with the breakdown by case
• The eval run goes with the cache disabled
• Chunking by structure, not by a fixed window
• The cases with a disappeared reference are marked needs-review and are not deleted
• In production the cache hit rate, the cost per request, the share of the trimmed candidates and the p95 latency are visible

The expanded checklist for production — Production RAG: a 50-point checklist for production.

Related Articles

• Building a RAG pipeline on .NET: a practical walkthrough
• RAG on .NET: an architecture that survives production
• Production RAG: a 50-point checklist for production
• Enterprise AI architecture: four decisions that decide everything
• .NET and Azure for AI systems: a reference architecture

CTA

If your RAG works but the decisions are made by eye, start with a single step: pick 50 questions from real support, fix the reference chunks and compute Recall@5 and MRR. After that every change in the pipeline becomes verifiable instead of a bet.

To break down your own pipeline on real data — write to us. We will show you exactly where it breaks: in the chunking, in the retrieval, in the generation or in the very way you measure the quality.