Steps to Develop Golden Test Scenarios

How we build reliable LLM products: the Scenario-Driven Data Flywheel Most teams start with prompts. But the best LLM applications start with scenarios. Here’s the process we follow — inspired by behavior-driven development and formalized in application-centric evaluation frameworks: 1. Start with a product requirement Define the specific user behavior or output your model must support. Example: “Rewrite casual requests into formal tone.” 2. Convert it into a scenario Write a concrete testable spec. Name the dimensions: tone, intent, complexity, etc. 3. Generate synthetic examples (or pull real data) Use those dimensions to create realistic inputs that reflect user language. 4. Write your first prompt Implement your first draft of the system (prompt, RAG, tool calls, etc.) 5. Run the model, get outputs Evaluate against your scenario using LLM-as-Judge or reference-based scoring. 6. Do structured error analysis Label 50–100 traces. Identify failure modes like tone mismatch, hallucination, or drift. 7. Improve the prompt or system logic Fix what’s broken: underspecified prompts, missing retrieval, or poor tool design. 8. Generate new examples for the same scenario Re-test until the failure mode is fixed. 9. Repeat for the next scenario Expand your eval set and build confidence scenario by scenario. This process becomes a data flywheel: Each failure teaches you how to generate better examples, write better prompts, and improve reliability without guessing. Stop thinking in terms of prompts and answers. Start thinking in terms of scenarios and behaviors. That’s how you build products that work — and keep working.

McKinsey surveyed 2,000 companies in 2025. A whopping 51% said AI backfired on them. Top reason? Inaccuracy. From what I can tell, most of these systems weren't broken. They were unreliable. And unreliable is wayyyy worse because you can't predict when it fails. So I got Ashalesh Tilawat (the Mr. Miyagi of teaching AI) from Gauntlet AI to walk me through the solution. Here's his 2026 framework for evaluating if your AI is trustworthy, reliable, and production-ready: 1. build your golden set Identify 30–50 core requests your AI must handle correctly. The stuff that, if broken, makes the whole system useless. And sit with the person whose job this AI is doing/automating/replacing/helping with. 2. test the weird stuff Your golden set covers common requests. But in production, users don't only ask common requests. So build a matrix of categories (topic x complexity) and fill the gaps. Every gap is a corner where failures can hide behind. 3. build a replay harness Record the exact state of every interaction so you can test prompt changes without burning API calls. Think of it like game film... you don't put players back on the field just to review the play. 4. create your rubric Use an LLM to grade outputs on accuracy, completeness, and tone. But calibrate it first -> run 50–100 examples through human and LLM scoring, find disagreements, fix the rubric, repeat until they match. 5. run experiments New model? Prompt rewrite? Run your eval suite against both versions. Ship if the golden set passes, no regressions, and the cost is acceptable. The teams still running production AI on vibes will be f***** in 2026. But the teams building eval libraries are compounding an advantage that gets harder to catch every month. Competitors can copy your product. They can't copy your test cases. h/t Austen Allred for helping put this together. Full playbook + vid below 👇

Your LLM app isn't broken because of the model. It's broken because you never measured it. AI Evals!! Most teams do the same thing: → Build it → Test it on 5 examples → Demo goes perfectly → Ship it → Pray Then 3 weeks in, a user screenshots your chatbot confidently hallucinating your own product pricing. Here's the eval stack that actually works: 1/ Golden dataset first. Even 20 hand-crafted examples with validated answers are enough to start. Quality over quantity. This is your source of truth. 2/ Two types of evaluators — both are required. LLM-as-judge for subjective signals (hallucination, relevance, tone). Code-based eval for structural checks (did the JSON parse? is the number in range?). One without the other is incomplete. 3/ Never use 1–10 scores. LLMs can't score consistently at that granularity across runs. Use binary (correct/incorrect) or multi-class (relevant/partially relevant/irrelevant). You can average those. You can't trust a score of 7.2. 4/ Wire evals to CI/CD. Every prompt change, model swap, or retrieval tweak runs against your golden dataset before it ships. This is your gate. LLM evaluations are your new unit tests. 5/ Add guardrails last, not first. Don't block everything. Over-indexing on guards kills user intent. Start with PII removal, jailbreak detection, and hallucination prevention. Add more when production tells you to. Your app can degrade with zero code changes. Model updates and input drift happen silently. Run your evals on a schedule, not just on deploys. Measure it. Or be surprised by it. What's your current eval setup? Drop it in the comments. Read the full blog and follow me Priyanka for more ↓ https://lnkd.in/gsjnbubY #LLMOps #AIEngineering #MachineLearning #GenerativeAI #MLOps #SoftwareEngineering #AIProductDevelopment #evals #aievals

Starting with Eval: If you’re starting fresh with evals for AI agents, the first thing to do is define your criteria clearly. Don’t jump into metrics or tooling until you know exactly what you’re measuring. Ask yourself: Is success accuracy? Is it safety? Is it response efficiency? Or maybe reliability and explainability? Whatever you choose, it has to map directly to how the agent is expected to perform in the real world. Build Your Golden Dataset Next comes the golden dataset. Think of this as your foundation, a small set of annotated examples that set the benchmark for what good looks like. This is where human feedback is critical. Start small, label a handful of traces, and refine until your evaluator consistently agrees with human judgment. This dataset becomes your single source of truth. Align the Judge With criteria and golden data in place, the next step is aligning a LLM judge prompt. The evaluator prompt is not just a template it’s the lens through which everything is judged. If it’s vague, you’ll get misleading results. If it’s precise and tuned to your golden set, you’ll get evaluations that reflect reality. Finally, treat evaluation as a continuous loop, not a one-time task. Gather agent traces, run evaluations, compare results to your golden data, and refine the evaluator. Each cycle gets you closer to an evaluator that measures what actually matters, not just vanity metrics. Over time, this loop turns messy outputs into a reliable, production-ready evaluation framework. Evals aren’t hard to run. The challenge is aligning them to the agent’s purpose. When your evals mirror business outcomes and user expectations, they stop being demos and start being value drivers. That’s when you know you’ve built an eval framework that actually matters.