Freeplay

We've been thinking a lot about a workflow problem that every serious AI team faces now: How do you use agents to improve agents efficiently? Right now, when a coding agent needs to debug an AI product, the default in most cases is to try to pull down thousands of production traces and pattern-match its way to a root cause. That's slow and expensive at best. Kind of like handing someone a phone book and asking them to find the interesting people... Our new Insights agent was built to fix that. Rather than wait for a request, Freeplay's agent runs in the background, looking at production evaluator scores and reasoning traces on a schedule. It then clusters patterns, ranks issues by impact, and links each one to specific traces that show the problem. This is the kind of infrastructure coding agents need to work efficiently -- pre-computed analysis and signal so they can debug fast, not on-demand queries. Now when you ask Claude Code "what's wrong with our agent?", the Freeplay MCP server doesn't hand back 10,000 traces. It returns the top insights that matter this week, with the 50 traces that demonstrate them. The agent can know immediately what to look for before it reads a single log line. The old way: dig through lots of logs, hope you spot something. New way: start with the diagnosis, then dig deeper to decide how to fix it. This same workflow also helps every human user who logs into Freeplay. Dashboards and metrics help, but Insights tell you where to look much faster. We wrote up how it works and where it fits in the broader data flywheel to continuously improve an agent. Check out the video (shoutout to Jeremy Silva), and the link in the comments.

New case study out today about eval and AI optimization workflows with Cisco Duo Security, and this one's a bit different: A big part of their success involves Design. 👀 Most of the conversation about AI quality focuses on engineering. Duo deeply involves designers into AI quality work, alongside engineering and product. They shared what their process looks like, and why they do things this way. For the past year, the Duo AI Assistant team has been running what they call a "communal" quality practice. Every week, designers, PMs, data scientists, and engineers each review 15-20 real assistant conversations. They calibrate together on Fridays. They've turned cross-functional collaboration into cadence. What stood out to me: the Design team doesn't see this as a chore. They see it as part of their user research process. Jillian Haller, Design Manager: "This is the next best thing to a contextual inquiry... We're actually able to see how the interaction unfolds." A year in, the results speak for themselves: Duo is expanding their AI Assistant to global customers, their team and leadership have clear visibility into quality, and the team has built the operational muscle to keep improving week over week. Huge thanks to Brianna Penney, Jillian Haller, Laura Cole, and Shakeel Ahamed for sharing their story with us. They built this practice without a blueprint, and now it's become a model for other teams to learn from. We're proud Freeplay is the shared surface where their cross-functional team collaborates on AI quality. Full case study in the comments. 👇

IT'S TIME AGAIN! Colorado AI Builders kicks off for the new year on February 11 in Boulder. Now officially the start of our 4th year doing this. 🏔️🧠 RSVP link in the comments. The last Boulder meetup had nearly 300 people. We're excited to get back together again and show off some of the latest cool AI projects being built in CO. We've had a ton of demand to do demos at these -- way more than we have space for. If you're interested to demo, please tell us in the RSVP form and share a quick Loom. We're excited to keep building this community together in 2026. Thanks as always to SVB, Technical Integrity, Ombud, Matchstick Ventures, Drive Capital and Freeplay for making this happen.

If you're building production AI agents, you've probably learned the secret to AI quality is to "look at lots of data." You don't want to spend all your time searching for data to look at. We build Freeplay's new Automations feature so you can trigger evals, update datasets, curate review queues, or send Slack notifications based on custom filters for your logs. The basic idea: Build custom filters for production logs ⏩ pick an action ⏩ set a schedule. Then... * Review queues populate themselves for human reviewers. * Complex eval logic triggers for the edge cases or issues you care about most. * Test datasets stay fresh with production examples. * Slack pings your team when something's off. Our team's been focused on speeding up the Ops workflow for AI engineering teams, and this is a small (but very cool!) new tool to help. Shoutout to Aditya Pandey Rob Rhyne Nico Tonozzi especially for bringing this to life!

We're excited to share our latest customer case study with Chime. It talks about how they've scaled production "AI engineering" by empowering domain experts and engineers to collaborate. Chime is the #1 most-loved banking app, serving millions of members with a mission to deliver helpful, transparent, and fair financial services. They went public last year as CHYM. Going back to 2024, their AI team set a clear goal: scale production AI use cases across the company without compromising quality. This was especially important in a regulated environment where trust and accuracy aren't negotiable. Most teams hit a wall here. Domain experts know what "good" looks like, but all changes go through engineering. Iteration often slows to a crawl as a result. Chime broke through by rethinking who owns what. Engineering builds the pipelines, and domain experts own prompt performance, evals, and ground truth. The two groups work in parallel, not in sequence. And as a result, they've been able to do more with AI to better serve their members. Chime has now scaled production AI across a range of use cases. In just one financial crimes example described in the case study, they achieved: ✅ 40% efficiency gains ✅ 99%+ quality scores, outperforming human agents ✅ Millions of $$ in OpEx savings ✅ Domain experts running evals and shipping prompt improvements independently Huge credit to the Chime team for building a true cross-functional AI practice -- and for showing others how it can be done. We're proud that Freeplay serves as the ops layer where their Engineering, Product, and Operations teams collaborate. If you're scaling AI in a complex domain and want to see what's possible when you empower domain experts alongside engineering, the full case study is in the comments. 👇