# .NET and Azure for AI systems: a reference architecture

How to assemble a production-ready AI system on .NET and Azure: async pipeline, RAG, Polly, observability and cost control — with working code.

> For the index of this section of the site, see [llms.txt](https://anetlab.com/llms.txt)

October 13, 202612 min read

# .NET and Azure for AI systems: a reference architecture

How to assemble a production-ready AI system on .NET and Azure: async pipeline, RAG, Polly, observability and cost control — with working code.

[Victor Buzin](https://anetlab.com/about.md) · Founder & Principal Engineer, AnetLab

16+ years in enterprise development: .NET, AI/LLM systems, RAG, MCP. Author of the material and the practices behind it.

## Why this architecture is needed

AI-enabled systems in production are not just an LLM call from a controller. In real operation you need identity, authorization, asynchronous execution, retrieval, retry budgets, observability, cost management and data protection. .NET and Azure fit this scenario well, because they offer a mature platform, clear cloud integration and a convenient path to operational readiness.

This reference architecture describes a practical enterprise approach: a .NET application takes the request, verifies identity and policy, pushes the work into an async flow where that makes sense, mixes in the context through retrieval, calls the model endpoint, writes telemetry and returns the result through an observable and protected pipeline.

The key choice in this architecture is not the model but the boundary between synchronous and asynchronous. Everything that fits into 800–1500 ms should live in the HTTP request; everything longer belongs in a background queue with an explicit status and the possibility of cancellation. A mistake in this choice costs more than anything else: it either kills the SLA or hides all the real failures in the noise of HTTP errors.

One of our setups is an assistant on top of a customer’s corporate documentation: the documents are heterogeneous, the rights differ at file level, and the repeated questions are the same ones. That setup showed two things: where exactly it pays off to cut the cost of a request in the .NET stack, and what cannot be put in the cache without looking at access rights first. We break it down at code level below.

## The basic .NET AI stack

A practical .NET AI stack usually includes these layers:

*   **ASP.NET Core** for the API, auth, streaming and orchestration.
*   **Microsoft.Extensions.AI** and related abstractions, if you need a model-agnostic approach.
*   **Semantic Kernel** for planners, tool calling, prompt composition and higher-level orchestration.
*   **Azure OpenAI** for chat, embeddings and function calling.
*   **Azure AI Search** or another vector-capable retrieval layer for RAG.
*   **Entity Framework Core** or other data access libraries for transactional state.
*   **Azure Service Bus / Storage Queues** for async workflows.
*   **Azure Cache for Redis** for hot context and short-lived state.
*   **OpenTelemetry** with Application Insights / Azure Monitor for traces, metrics and logs.
*   **Polly** for retries, circuit breakers and timeout policies.
*   **IHostedService / BackgroundService** for background processing, indexing and long-running tasks.

The main idea is not the number of packages but the separation of responsibilities: request handling, orchestration, retrieval, generation and persistence must be spread across clear boundaries.

## The reference architecture at a glance

A production-ready .NET + Azure AI architecture usually has these logical components:

1.  **Client layer**: web app, mobile app or internal tool.
2.  **API gateway / edge**: Azure API Management or a reverse proxy for auth, quotas and validation.
3.  **.NET application layer**: ASP.NET Core API, minimal APIs or worker services.
4.  **Orchestration layer**: prompt builder, policy engine, tool invocation and response shaping.
5.  **Retrieval layer**: Azure AI Search, vector indexes, SQL/NoSQL metadata, document chunk store.
6.  **Model layer**: Azure OpenAI or another LLM endpoint, embeddings endpoint, moderation endpoint.
7.  **Async processing layer**: Service Bus, queues, Durable Functions or worker services.
8.  **Observability layer**: OpenTelemetry, logs, traces, metrics, dashboards, alerts.
9.  **Governance and security layer**: Managed Identity, Key Vault, content filters, DLP, RBAC, network isolation.

The key rule: synchronous requests must be short and predictable, and everything expensive, long or unstable has to be moved into asynchronous workflows.

## Diagram description

**Brief for the reference architecture diagram**:

*   Left: users and enterprise apps that reach the ASP.NET Core API through API Management.
*   Top center: a .NET orchestration service with prompt builder, policy checks, tool invocation and response formatter.
*   Center: retrieval services connected to Azure AI Search, the document store and the embeddings index.
*   Bottom center: async worker services that read messages from Service Bus for indexing, summarization and long-running tasks.
*   Right: Azure OpenAI and the supporting model endpoints.
*   Bottom layer: a telemetry pipeline into OpenTelemetry, Application Insights, Log Analytics, dashboards and alerts.
*   Side rails: Managed Identity, Key Vault, private endpoints, rate limits and cost budgets.

The diagram must show solid arrows for the synchronous request/response and dashed arrows for queue-based async processing.

## Async architecture and the execution flow

A mature AI system must not block the HTTP request during chunking, embedding, indexing or summarization of large documents. Instead:

*   the API takes the user request.
*   it runs the identity, authorization, request size and tenant policy checks.
*   it writes a job record or a command message.
*   a worker service handles the retrieval and the model interaction, if the task is long.
*   the API returns a correlation ID or a lightweight initial response.
*   the client gets the final result through polling, a webhook or a websocket.

This approach lowers the latency, makes the behavior predictable and limits the radius of failure when the model misbehaves.

Typical scenarios for async:

*   document ingestion and chunking
*   generation of embeddings and reindexing
*   summarizing long conversations
*   batch classification and extraction
*   tool-call fan-out
*   human-in-the-loop review queues

For production it is important to add idempotency keys and deduplication, otherwise retries can create duplicates.

## Resilience patterns

The behavior of AI services differs from ordinary REST calls: there are timeouts, throttling, moderation denials, transient network errors and service overload. So resilience has to be explicit.

Recommended approaches:

*   **Timeouts** for every outbound model and retrieval call.
*   **Retries with jitter** only for transient failures.
*   **Circuit breakers** for degrading dependencies.
*   **Bulkheads** to isolate prompt generation, embeddings and search traffic.
*   **Fallbacks** to the cache, to smaller models or to a retrieval-only mode.
*   **Queue-based buffering** for peaks.
*   **Idempotency** to protect against repeated side effects.

Polly is convenient to use together with the HttpClient pipeline, so that the policies stay uniform.

## Cost controls

Cost control is an architecture task, not an after-the-fact optimization. AI spending grows because of token volume, a long context window, repeated retrieval and endless loops.

Practical measures:

*   **Token budgets per request**.
*   **Prompt trimming** and shortening the history.
*   **Retrieval top-k limits**.
*   **Model tiering** for simple tasks.
*   **Caching** of embeddings, retrieval results and deterministic outputs.
*   **Batching** of offline summarization and embedding jobs.
*   **Rate limiting** per user, tenant or feature.
*   **Budget alerts** on abnormal spend.

Improve the quality of the context first, and only then increase the model size.

### Case: a semantic cache keyed by the query vector

In a customer’s corporate setup the users ask the same questions dozens of times a day: about regulations, about the team composition, about deadlines. An exact-match cache does not work on them — the wording differs from query to query. A cache keyed by the query vector does work: in the setup where we implemented it, 30–50% of the requests hit the cache — a third to a half of the requests never reached the generation stage.

```
public sealed class SemanticCache(
    IEmbeddingService embedder,
    ISemanticCacheStore store,
    ILogger<SemanticCache> logger)
{
    // The threshold is tuned on the golden set, not "by eye": with a threshold
    // that is too low the cache starts returning the answer to another question.
    private const double MatchThreshold = 0.94;
    private const int ProbeLimit = 5;

    public async Task<CachedAnswer?> TryGetAsync(
        string query, ClaimsPrincipal user, CancellationToken ct)
    {
        var scope = AccessScope.For(user);
        var embedding = await embedder.EmbedAsync(query, ct);

        var candidates = await store.SearchAsync(
            embedding,
            scope,                    // access scope — mandatory filter
            limit: ProbeLimit,
            ct);

        var best = candidates.FirstOrDefault();
        if (best is null || best.Similarity < MatchThreshold) return null;

        logger.LogInformation(
            "semantic cache hit {Similarity:F3} for {User}",
            best.Similarity, user.Identity?.Name);

        return best.Answer;
    }
}
```

Three things decide everything in an implementation like this:

*   **The access scope is part of the search, not a check afterwards.** An answer assembled for a different set of rights must not appear as a candidate even when the question is very similar.
*   **The threshold is tuned on the golden set.** With a threshold that is too low the cache returns the answer to a similar but different question — and it looks like “the model hallucinates”, while the cache is to blame.
*   **The match is recorded in the logs.** Without `Similarity` in the telemetry you will not see what share of the real requests the cache closes, and you will not notice when it stops working.

The saving does not come from one setting, but from the fact that the cache removes the most expensive part — the generation with a long context. The embeddings, meanwhile, are cached separately and always: they depend only on the query text, not on the access rights.

## Observability and token-aware logging

In AI systems, observability is not just the HTTP status. You need to know which prompt version was used, which retrieval sources made it into the answer, how many tokens were spent, which tools were called and where the latency came from.

Track:

*   request ID / correlation ID
*   tenant and user context with privacy in mind
*   prompt template version
*   retrieval query and source IDs
*   model name and deployment
*   input/output token estimates
*   latency by stage: auth, retrieval, generation, post-processing
*   retries, fallback mode, moderation outcome

Do not log secrets, the full prompt content or documents unless necessary. Use structured logging with redaction and sampling.

### Code example: ILogger token logging

```
using Microsoft.Extensions.Logging;

public sealed class AiRequestLogger
{
    private readonly ILogger<AiRequestLogger> _logger;

    public AiRequestLogger(ILogger<AiRequestLogger> logger)
    {
        _logger = logger;
    }

    public void LogPromptSummary(string correlationId, string model, int inputTokens, int outputTokens, TimeSpan latency)
    {
        _logger.LogInformation(
            "AI request completed. CorrelationId={CorrelationId} Model={Model} InputTokens={InputTokens} OutputTokens={OutputTokens} LatencyMs={LatencyMs}",
            correlationId,
            model,
            inputTokens,
            outputTokens,
            latency.TotalMilliseconds);
    }
}
```

Logging like this helps you analyze cost and performance without exposing the prompt bodies.

## RAG integration

Retrieval-Augmented Generation is often what separates a demo from a genuinely useful system. RAG raises factuality, because the model is grounded on internal sources: policies, manuals, tickets, knowledge base articles.

A typical RAG flow in .NET:

1.  Ingest documents.
2.  Chunk along semantic boundaries.
3.  Generate embeddings.
4.  Store vectors and metadata in the retrieval engine.
5.  On a query, normalize the user query.
6.  Get the top-k candidates with filters on tenant, language, freshness and permissions.
7.  Assemble a grounded prompt with citations.
8.  Send it to the model.
9.  Return the answer with source references.

Important:

*   store document-level and chunk-level metadata
*   use a stable citation format
*   filter by permissions before the retrieval or immediately after it
*   re-rank the chunks if needed
*   keep offline indexing separate from online query handling

In RAG, a well-selected context works better than a long but noisy prompt.

## Decision matrix

| Scenario | Best pattern | Why |
| --- | --- | --- |
| A fast answer from a single knowledge base | Synchronous RAG in ASP.NET Core | Low latency and a simple UX |
| Document ingestion or reindexing | Async worker through a queue | No timeout in the request path |
| Multi-step tool use between systems | Orchestrated workflow | Clear boundaries and observability |
| Frequent expensive prompts | Cache + prompt versioning | Reduces the token spend |
| Tenant-specific policies and quotas | API gateway + app policy checks | Defense in depth |
| Bulk background summarization | IHostedService or a worker pool | Efficient and scalable |
| High availability requirements | Fallbacks + circuit breakers + async buffering | Reduces the blast radius |

## Code examples

### 1) Polly + HttpClient pipeline

```
using System.Net.Http.Headers;
using Microsoft.Extensions.DependencyInjection;
using Polly;
using Polly.Extensions.Http;

var services = new ServiceCollection();

services.AddHttpClient("AzureOpenAI", client =>
{
    client.BaseAddress = new Uri("https://your-resource.openai.azure.com/");
    client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
    client.Timeout = TimeSpan.FromSeconds(30);
})
.AddPolicyHandler(GetRetryPolicy())
.AddPolicyHandler(GetCircuitBreakerPolicy());

static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrResult(msg => (int)msg.StatusCode == 429)
        .WaitAndRetryAsync(
            retryCount: 3,
            sleepDurationProvider: attempt => TimeSpan.FromMilliseconds(200 * Math.Pow(2, attempt))
                                       + TimeSpan.FromMilliseconds(Random.Shared.Next(0, 100)));
}

static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy()
{
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .OrResult(msg => (int)msg.StatusCode == 429)
        .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30));
}
```

### 2) ILogger token logging

```
using Microsoft.Extensions.Logging;

public sealed class GenerationTelemetry
{
    private readonly ILogger<GenerationTelemetry> _logger;

    public GenerationTelemetry(ILogger<GenerationTelemetry> logger)
    {
        _logger = logger;
    }

    public void LogGeneration(string requestId, string deployment, int promptTokens, int completionTokens, bool usedRag)
    {
        _logger.LogInformation(
            "Generation telemetry RequestId={RequestId} Deployment={Deployment} PromptTokens={PromptTokens} CompletionTokens={CompletionTokens} UsedRag={UsedRag}",
            requestId,
            deployment,
            promptTokens,
            completionTokens,
            usedRag);
    }
}
```

### 3) IHostedService background processing

```
using System.Threading.Channels;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;

public record AiJob(string JobId, string Payload);

public sealed class AiJobQueue
{
    private readonly Channel<AiJob> _channel = Channel.CreateUnbounded<AiJob>();

    public ValueTask EnqueueAsync(AiJob job) => _channel.Writer.WriteAsync(job);
    public IAsyncEnumerable<AiJob> DequeueAllAsync(CancellationToken cancellationToken) => _channel.Reader.ReadAllAsync(cancellationToken);
}

public sealed class AiBackgroundWorker : BackgroundService
{
    private readonly AiJobQueue _queue;
    private readonly ILogger<AiBackgroundWorker> _logger;

    public AiBackgroundWorker(AiJobQueue queue, ILogger<AiBackgroundWorker> logger)
    {
        _queue = queue;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var job in _queue.DequeueAllAsync(stoppingToken))
        {
            try
            {
                _logger.LogInformation("Processing AI job {JobId}", job.JobId);
                await ProcessJobAsync(job, stoppingToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "AI job failed {JobId}", job.JobId);
            }
        }
    }

    private static Task ProcessJobAsync(AiJob job, CancellationToken cancellationToken)
    {
        // Chunk, retrieve, call model, persist results.
        return Task.CompletedTask;
    }
}
```

## FAQ

### Is .NET suitable for AI-enabled systems?

Yes. .NET is especially strong where enterprise integration, security, observability and a maintainable service architecture around the model calls matter.

### Do all LLM calls have to be synchronous?

No. Fast user-facing calls can stay synchronous. Everything long, batch-oriented or failure-prone is better moved into async processing.

### Do you need Semantic Kernel in every project?

No. Use it where its abstractions really help the orchestration, not by default.

### How do you avoid the cost of prompts growing?

Limit token budgets, shorten the history, cache deterministic outputs and use cheaper models for simple tasks.

### What is the most common mistake in RAG?

Indexing too much low-quality content and returning too many chunks. RAG requires relevance and governance.

### How do you log AI requests safely?

Log structured metadata, token counts, latency, model version and correlation IDs. Redact prompt bodies and secrets.

## Related Articles

*   [RAG on .NET: an architecture that survives production](https://anetlab.com/blog/rag-na-dotnet.md)
*   [Modernising legacy .NET: WCF, AppDomains and the AI bridge](https://anetlab.com/blog/legacy-dotnet-modernisation.md)
*   [Production RAG: a 50-point checklist for production](https://anetlab.com/blog/production-rag-checklist.md)
*   [Enterprise AI architecture: four decisions that decide everything](https://anetlab.com/blog/enterprise-ai-architecture.md)

## CTA

If you are designing an AI-enabled platform on .NET and Azure, start with the boundaries: synchronous vs asynchronous, retrieval vs generation, online vs offline processing. Then add resilience, logging and cost controls before you scale the functionality. That order lowers the risk and makes the architecture easier to evolve.

If the system already runs and you need to understand where exactly the budget leaks and where the quality breaks, [write to us](https://anetlab.com/contact.md). We will break down the architecture and hand over a plan with priorities.

Read also

October 21, 2026AI/LLM

### [Contextum: the document pipeline to Markdown for AI agents](https://anetlab.com/blog/contextum-document-pipeline.md)

AnetLab Contextum: a five-stage pipeline between an agent and an LLM — Markdown conversion, context compression, idempotent delivery and an honest loss registry.

[Read the deep dive](https://anetlab.com/blog/contextum-document-pipeline.md)

[.NET](https://anetlab.com/blog/?tag=.NET)[RAG](https://anetlab.com/blog/?tag=RAG.md)

October 20, 2026AI/LLM

### [RAG evaluation on .NET: Recall@5, MRR and the release gate](https://anetlab.com/blog/rag-evaluation-dotnet.md)

How to measure RAG quality on .NET: a Golden Set of 50–100 questions, computing Recall@5 and MRR, measuring faithfulness and eval-as-a-release-gate in CI.

[Read the deep dive](https://anetlab.com/blog/rag-evaluation-dotnet.md)

[.NET](https://anetlab.com/blog/?tag=.NET)[RAG](https://anetlab.com/blog/?tag=RAG.md)[Evaluation](https://anetlab.com/blog/?tag=Evaluation.md)