Braintrust

Agent design has evolved through six distinct generations as models have grown smarter and more capable. From simple prompts to modern AI harnesses, each generation broke old assumptions and created new failure modes that require different eval strategies. Read more → https://lnkd.in/gvucz3Ri

AI observability has shifted from the traditional pillars of metrics, logs, and traces to a new set of challenges: traces, evals, and annotation. Traces reconstruct the full decision path across model calls and tools. Evals quantify performance both in production and dev. Annotation creates corrective signals for continuous improvement. Read more → https://lnkd.in/gBntrWaJ

We tested whether "bash is all you need" for AI agents by building an eval harness that compared SQL, bash, and filesystem approaches on the same dataset. SQL hit 100% accuracy while bash achieved 53%. The hybrid approach won by using both tools and self-verifying results. Read more → https://lnkd.in/gFTG4cdX

Streamline dashboard management by copying charts and entire dashboard views across projects and organizations. Export raw chart data for external analysis or import proven monitoring setups to new environments. Read more → https://lnkd.in/gJgBvCmy

Five hard-learned lessons from teams running thousands of evals daily: - Good evals enable 24‑hour model swaps, feed on real user bugs, and validate features pre‑launch. - Engineer your data pipelines and scorers with the same rigor as production code. - Context (tools, formats, flows) often matters more than the prompt itself. - New models can upend your roadmap. Stay ready with continuous evals and a provider‑agnostic proxy. - Optimize the full loop (data + prompt + scorers), not just single lines of text. Read more → https://lnkd.in/gTU8cMpw

Organize experiments by what matters to your workflow with filterable tags in the dataset runs panel. Compare runs across a specific model version, prompt variant, or release candidate. Read more → https://lnkd.in/gPiSNRSM

Single-turn evals can't tell you if your chatbot asked for the same information twice or kept customers in polite loops without solving anything. These failures only surface when you score entire conversations. Learn how to eval multi-turn conversations with both single-turn and conversation-level scoring in Braintrust.

Running evals locally ties up your machine and makes it hard to collaborate with teammates. Braintrust's Sandboxes feature lets you package your agent's entire runtime environment (database, dependencies, eval code) and run it in the cloud via Modal or AWS Lambda. Learn how production AI teams use sandboxes to build evals that scale with their applications. Watch the full session → https://lnkd.in/grQBEzFS

Evals course module fourteen: the eval improvement loop. Learn how to complete the full eval-driven improvement cycle in Braintrust. Find problems in production, sample them into a dataset, run a baseline, test a fix, and verify the results. More here → https://lnkd.in/gXXK6v6P

Going from prototype to production is more challenging than ever. With AI products, teams need to manage multi-step agents, tool use, and the unpredictability of real users. Learn how to ship production AI applications in this workshop from AI Engineer Europe with Braintrust and Trainline.