# Taking an AI PoC to production: 90 days by phases Why AI pilots do not reach production and how to fix it: Golden Set, eval pipeline, integration with the corporate context, LLMOps and a readiness checklist. > For the index of this section of the site, see [llms.txt](https://anetlab.com/llms.txt) October 16, 202614 min read # Taking an AI PoC to production: 90 days by phases Why AI pilots do not reach production and how to fix it: Golden Set, eval pipeline, integration with the corporate context, LLMOps and a readiness checklist. [Victor Buzin](https://anetlab.com/about.md) · Founder & Principal Engineer, AnetLab 16+ years in enterprise development: .NET, AI/LLM systems, RAG, MCP. Author of the material and the practices behind it. ## The gap between a demo and a deployment You have a working demo. The LLM answers questions about your internal documentation, the stakeholders are impressed, and management wants to see the system “in the field” by the next quarter. This is the most dangerous moment for any AI project. The move from a Proof of Concept (PoC) to an industrial-grade system is where most AI initiatives die. And it is rarely because “the model is not smart enough”. Almost always the reason is that a system built for a demo is a fragile prototype for sterile conditions, while production is a chaotic, aggressive environment with high stakes. In a PoC an error rate of 5% is considered “promising”. In production, 5% errors is a PR disaster or a multi-million risk. **Thesis:** the gap between a working AI demo and a production system is not a technical problem, it is a delivery problem. The process has a repeatable structure that does not depend on the technology in use — be it GPT-4, Llama 3 or a custom transformer. Success requires moving from “prompting” to “engineering” and from “manual testing” to “automated evaluation”. This playbook describes a 90-day path across that gap. ## Why AI PoCs break in production: four root causes To solve the delivery problem you have to understand why it appears. Most teams try to apply the traditional SDLC (Software Development Life Cycle) to nondeterministic AI models. That leads to four typical failure scenarios. ### 1\. The Happy Path fallacy * **How it shows up:** during the PoC, the team (and the loyal alpha testers) unwittingly feed “clean” input. They ask clear questions within the context. In the real world users type nonsense, use slang, paste huge chunks of unrelated text, ask about competitors or actively try to jailbreak the system. * **Root cause:** the PoC was built to prove *possibility* (can the system do X?), not *reliability* (can it do X for everyone, always and under pressure?). The “long tail” of user behavior was never modeled. * **Prevention:** adopt the “Red Team” mindset from day one. Use adversarial prompting to find the failure points before the users do. ### 2\. Evaluation blindness * **How it shows up:** a developer changes a prompt to improve accuracy on one specific case. In manual tests everything looks great. However, that change quietly breaks three other cases. The team finds out only three weeks later, after a customer complaint. * **Root cause:** the absence of a “Golden Set” or an automated evaluation pipeline. The team relies on “vibe checks” — eyeballing 5-10 answers — instead of a statistical, repeatable evaluation. AI is nondeterministic; it cannot be “tested” like a piece of code. It has to be “evaluated” like a statistical model. * **Prevention:** build the evaluation system before you start polishing prompts. If you cannot measure the impact of a change on the whole dataset, you are just guessing. ### 3\. Integration friction (the laptop trap) * **How it shows up:** the PoC lives in a Python notebook or a script on a local machine. Moving it into the corporate infrastructure (Azure, AWS, CI/CD, VPC) creates enormous complexity. Questions about Managed Identities, firewall rules, network latency between the app and the LLM API and strict privacy rules suddenly appear. * **Root cause:** the technical environment of the PoC was detached from the constraints of the production environment. What works in a Jupyter notebook often fails in a microservice under load. * **Prevention:** start Phase 1 with a “Skeleton Deployment”. Deploy a “Hello World” version into a real VPC right away, to surface the infrastructure blockers early. ### 4\. Operational debt (the day two problem) * **How it shows up:** the system is running, but the costs are getting out of control. The team runs into the provider’s rate limits, so 50% of requests fail. Developers spend 40 hours a week manually digging through logs, trying to understand why the model “behaves strangely”. * **Root cause:** ignoring the “Ops” half of LLMOps. Traditional monitoring looks for 200 vs 500 errors; AI monitoring has to track toxicity, cost per request, semantic drift and retrieval quality. * **Prevention:** ship semantic caching, solid observability (OpenTelemetry) and cost limits by the end of Phase 2. The first rule of the whole 90-day program grows out of the same thing: **every phase ends with a working measure, not a report**. If by the end of Phase 1 you have no before/after number for quality and cost, you are not measuring progress further along — you are retelling impressions. In a pilot that reached production in 90 days this rule played out literally. Most of the time went not into the prompt, not into the model choice and not into eval — but into connecting to the corporate context: who the user is, which documents they may see, how access is revoked and what that means for retrieval and for the cache. A demo has no such layer: the user has one set of documents, rights never change, and the question “who can see this” simply never comes up. * * * ## The 90-day delivery framework Success is not a destination, it is a 90-day sequence of risk reduction. We split the process into three phases. ## Phase 1: Stabilization (Days 1–30) — The engineering foundation The first 30 days are about removing the “magic” from the PoC and replacing it with engineering rigor. We do not add new features, we stabilize the core. **Goal:** create a measurable baseline and a reproducible development environment. ### Weeks 1-2: Building the Golden Set You cannot optimize what you cannot measure. The Golden Set is a collection of 50-100 diverse “Query” and “Expected answer” (ground truth) pairs. * **Diversity is critical:** include short queries, long texts, edge cases and questions that are “out of scope”. * **Human-in-the-loop:** the expected answers must be checked by experts (SME — Subject Matter Experts). This set becomes your yardstick for the next 90 days. ### Week 3: Infrastructure and tooling We move the code out of the notebooks. * **Tracing:** implement tracing through OpenTelemetry or tools like LangSmith. You have to see which context was extracted, what the final prompt looked like and how long each step took (retrieval, embedding, generation). * **Skeleton Deployment:** deploy a basic API version into the production cloud. Make sure it can reach the database and the LLM API through the corporate firewall. ### Week 4: The evaluation loop Set up an automated pipeline that runs the Golden Set through the system on every prompt or retrieval parameter change. Use these metrics: * **Exact Match (EM):** for structured answers. * **ROUGE/METEOR:** for text summarization. * **LLM-as-a-judge:** use a stronger model (for example GPT-4o) to score the production model against a rubric (accuracy, tone, usefulness). **Milestone:** by day 30 you should have a dashboard that tells you exactly how “good” your system is today. * * * ## Phase 2: Integration (Days 31–60) — The enterprise context Once the core is stable, it has to be integrated into the company’s ecosystem. Here we deal with the “nondeterministic” nature of the LLM and with the complexity of real data. **Goal:** turn the AI from an isolated service into a reliable part of the product. ### Weeks 5-6: Hardening the API The LLM is a remote, third-party and unstable service. Your code has to account for the possibility of failure. * **Retries and timeouts:** use libraries like **Polly** (.NET) or **Tenacity** (Python) to handle transient failures and limits. * **Fallback strategies:** what if the LLM is unavailable? Return a cached answer, a “the system is busy” message, or switch to a local compact model. * **Streaming UX:** if the latency is high, implement streaming of the answer so the user sees progress immediately. ### Week 7: RAG (Retrieval-Augmented Generation) tuning If you use RAG, the retrieval stage is where the system fails most often. * **Chunking strategy:** optimize the document split. Pieces that are too small lose the context, pieces that are too large dilute the answer. * **Hybrid search:** combine semantic (vector) search with keyword search (BM25). That guarantees that specific terms or product IDs are found. * **Re-ranking:** use a reranker model (for example from Cohere) to pick the top-20 documents and sort them by relevance before handing them to the LLM. ### Week 8: The governance and security layer * **PII masking:** a layer that removes personal data before it reaches the provider’s API. * **Moderation API:** blocking toxic requests and answers. * **Hallucination checks:** basic checks that the LLM answer is actually supported by the context that was found. **Milestone:** by day 60 the system is integrated into the UI, handles errors correctly and respects data boundaries. * * * ## Phase 3: Operations (Days 61–90) — Operational excellence The last 30 days are about long-term survival. A system without monitoring either degrades or bankrupts you. **Goal:** ensure scalability and a path of continuous improvement. ### Weeks 9-10: LLMOps and observability * **Cost monitoring:** a breakdown of the spend by users, features and departments. Set up automatic “kill switches” on abnormal cost spikes. * **Semantic caching:** store question-answer pairs in a cache (for example Redis). This cuts costs by 30–50% and reduces latency to milliseconds. * **Performance drift:** the Golden Set is not static. Track how the answer quality changes as your data changes. ### Week 11: Deployment pipelines Shipping a new prompt or a model has to be as rigorous as shipping code. * **A/B testing:** send 10% of the traffic to the new prompt and compare the scores and the feedback. * **Shadow deployments:** run the new prompt in the background on real traffic, without showing the result to users. Compare its “shadow” work with the live system. ### Week 12: The feedback loop * **Implicit feedback:** watch the user behavior. Did they copy the answer? Did they share the link? * **Explicit feedback:** thumbs up/down. Collect them and automatically add the “bad” answers to the Golden Set for further training and calibration. **Milestone:** by day 90 the system improves on its own, is protected against overspending and is transparent operationally. By this point the delivery team has changed: evaluation became part of the regular CI, and the “unpredictability of the model” became a managed risk with fixed boundaries. That is exactly what separates a system you can hand over to operations from a demo that is fun to show. From this follows a correction to the plan itself: Phase 2 “Integration” cannot be planned with the same length as Phase 1 just because “integration is connect”. In our pilot it was exactly the phase that took most of the schedule, and it was exactly where the work on document rights, the cache key and access revocation grew from. If you postpone that to Phase 3, in the last moment you have to redo both the retrieval and the cache — and do it after the architecture has already been declared working. * * * ## Building the delivery team An AI delivery team is not a classic IT team. It needs a balance between “scientific” search and “industrial” execution. * **1\. Delivery Lead (tech lead for delivery)** * *Role:* owns the 90-day plan. Manages stakeholder expectations (calms things down when it looks “slow” and raises the alarm when someone wants to skip the evaluation stage). * *Metric:* Readiness Score (readiness against the checklist). * **2\. AI Engineer (prompts and evaluation)** * *Role:* the link between Data Science and Software Engineering. Writes prompts, creates the evaluation rubrics, picks models and manages the Golden Set. * *Metric:* accuracy on the Golden Set. * **3\. Platform/DevOps Engineer (platform and scaling)** * *Role:* builds the containers, sets up the VPC, manages the vector database and the observability stack. * *Metric:* P95 latency and uptime. * **4\. AI Product Manager (product expert)** * *Role:* defines what “good” means. Works with the experts on the Golden Set. Designs the feedback loops in the interface. * *Metric:* user satisfaction and ROI per token. * * * ## Typical mistakes by phase **Phase 1 mistake: the trap of premature optimization** Teams spend weeks on fine-tuning or prompt polishing before the evaluation system even exists. That is like tuning an engine without a speedometer. * *Fix:* no changes to the system prompts until the Golden Set and the eval loop are ready. **Phase 2 mistake: integration as a black box** Treating the LLM as a simple REST API. Unlike code, an LLM answer can change in length, format and accuracy because of one extra character in the prompt. * *Fix:* instrument every step. Use tracing to “see” the model from the inside. **Phase 3 mistake: “set it and forget it”** Believing that if the AI works on day 90 it will still work on day 180. The context changes, providers update the model weights, user expectations grow. * *Fix:* automatic weekly drift checks against the Golden Set. * * * ## Artifact: the 90-day production readiness checklist | Phase | Item | Go/No-Go condition | Owner | | --- | --- | --- | --- | | **P1** | **Golden Set** | 50+ cases reviewed by experts (SME) | AI PM | | **P1** | **Eval Pipeline** | Every prompt change has an automated evaluation | AI Engineer | | **P1** | **Tracing** | The prompt, context and result of every step are visible | DevOps | | **P1** | **Baseline Latency** | P95 is documented and acceptable for the UX | AI Engineer | | **P1** | **Skeleton Prod** | The application is running in the production VPC | DevOps | | **P1** | **Security Audit** | An audit of access rights and privacy has been done | Security | | **P1** | **Red Teaming** | The abuse tests found no critical leaks | AI Engineer | | **P1** | **Cost model** | The cost per 1k users is calculated and approved | Delivery Lead | | **P2** | **Error handling** | Retries/timeouts/fallbacks are implemented | SW Engineer | | **P2** | **RAG tuning** | Chunking and indexing are optimized for speed | Data Engineer | | **P2** | **PII filtering** | Personal data is removed before it goes to the API | Security | | **P2** | **Citations in the UI** | Answers reference the source documents | PM / UX | | **P2** | **Feedback loop** | Thumbs up/down are stored in the logs | SW Engineer | | **P2** | **Hybrid search** | Vector + keyword results are merged | Data Engineer | | **P2** | **Token budget** | Hard limits on the max tokens per request | AI Engineer | | **P2** | **SLA defined** | The uptime and speed targets are fixed | Delivery Lead | | **P3** | **Semantic cache** | The cache layer is active to save cost and time | SW Engineer | | **P3** | **Cost Guardrails** | Budget limits and auto-shutdown are active | Delivery Lead | | **P3** | **Drift monitoring** | Weekly automated run against the Golden Set | AI Engineer | | **P3** | **HITL Dashboard** | An interface for the manual triage of AI errors | AI PM | | **P3** | **A/B Testing** | A framework for testing prompts on live traffic | DevOps | | **P3** | **Legal approval** | Final approval on security and toxicity | Legal | | **P3** | **Model updates** | The process of migrating to new LLM versions is described | AI Engineer | | **P3** | **Readiness** | Signed off by the stakeholders for the full rollout | Delivery Lead | * * * ## FAQ **1\. Are 50 examples in the Golden Set enough?** For a start — yes. It is better to have 50 high-quality, expert-reviewed examples than 500 noisy ones. As you find bugs in production, be sure to add them to the set. By the end of the first year the set usually grows to 500+ items. **2\. Should I start with a cheaper model to save money?** No. Start with the most powerful one (for example GPT-4o). Your goal is to establish the “upper bound” of quality. Once you reach a high score, try replacing the model with GPT-4o-mini or Llama 3 8B and see whether the quality holds. **3\. What is the main risk when delivering an AI product?** The absence of a “Stop” button. Teams get so absorbed by the “intelligence” of the system that they forget the safeguards. If you cannot switch the AI off separately from the rest of the product, you are not ready for production. **4\. How do you deal with nondeterminism?** You do not fight it, you accept it. Use evaluation metrics (for example “92% accuracy on the test set”) and communicate that to the stakeholders. Use UX patterns that tell the user the answer was generated by AI and needs to be checked. * * * ## Related Articles * [Production RAG: a 50-point checklist for production](https://anetlab.com/blog/production-rag-checklist.md) * [Enterprise AI architecture: four decisions that decide everything](https://anetlab.com/blog/enterprise-ai-architecture.md) * [.NET and Azure for AI systems: a reference architecture](https://anetlab.com/blog/dotnet-azure-ai-architecture.md) * [Engineering leadership in AI products: evaluation, roles, delivery](https://anetlab.com/blog/engineering-leadership-ai.md) ## Ready to go to production? Taking an AI PoC to production takes more than a lucky prompt. It takes a repeatable delivery machine. **Victor Buzin** — a technologist specializing in integrating AI into the corporate sector. To get an audit of your AI readiness roadmap, [get in touch](https://anetlab.com/contact.md). Read also October 20, 2026AI/LLM ### [RAG evaluation on .NET: Recall@5, MRR and the release gate](https://anetlab.com/blog/rag-evaluation-dotnet.md) How to measure RAG quality on .NET: a Golden Set of 50–100 questions, computing Recall@5 and MRR, measuring faithfulness and eval-as-a-release-gate in CI. [Read the deep dive](https://anetlab.com/blog/rag-evaluation-dotnet.md) [.NET](https://anetlab.com/blog/?tag=.NET)[RAG](https://anetlab.com/blog/?tag=RAG.md)[Evaluation](https://anetlab.com/blog/?tag=Evaluation.md) October 18, 2026AI/LLM ### [Enterprise AI architecture: four decisions that decide everything](https://anetlab.com/blog/enterprise-ai-architecture.md) A practical guide to the architecture of corporate AI systems: RAG vs fine-tuning, agents vs workflows, quality evaluation and governance. [Read the deep dive](https://anetlab.com/blog/enterprise-ai-architecture.md) [Architecture](https://anetlab.com/blog/?tag=Architecture.md)[Evaluation](https://anetlab.com/blog/?tag=Evaluation.md)