Engineering Software Usage Metrics

 GenAI productivity claims fall apart fast. Measure it the way leaders do. Measuring GenAI productivity is challenging when tools remain fragmented. Some expose APIs, others hide them, and many provide no access at all. This makes it difficult to correlate their impact across the development lifecycle. The real value comes from connecting data across the stack - Git repositories, product management systems, HR data, and GenAI usage. By consolidating these sources, leaders gain a unified view of how AI influences engineering work. With this foundation, unique metrics emerge. Cycle Time, from first commit to merge, becomes more meaningful when filtered by AI involvement. Comparing PRs influenced by Copilot, Cursor, or Windsurf reveals how each tool affects development speed. In one case, PRs assisted by Windsurf showed significantly lower Cycle Times than other vendors. Milestone connects these dots by attributing AI impact down to the commit level. This clarity allows organizations to see where GenAI truly accelerates delivery and where it falls short.

Last month, our AI tool adoption rate reached 62.5% among 40 engineers. But that number only tells part of the story. When I shared our change management approach and experimentation framework in previous posts, many of you asked: "How do you actually measure success?" The answer? We have built a comprehensive tracking system that focuses on encouragement, rather than enforcement. 1. Make it visible everywhere. We keep AI adoption front-of-mind through: Bi-weekly NPS surveys (54.5 current score) Monthly Community of Practice meetings Active Slack channel for sharing wins and learnings Real-time usage dashboards are shared team-wide The key insight: visibility drives curiosity, which in turn drives adoption. 2. Track both tools AND outcomes. We monitor two distinct categories: - Agentic Development tools (Copilot, Claude, Cursor) - Conversational AI (ChatGPT, Gemini, Claude) But here's what most teams miss—we also track work outcomes by tagging Jira tickets as "agentic_success" or "agentic_failure." This connects tool usage to actual impact. 3. Focus on insights, not enforcement. Our bi-weekly surveys don't just ask "did you use AI?" They capture: - Which specific tools do teams prefer - Key insights from their experiments - Barriers preventing adoption - Success stories worth sharing The result? 4.8M+ tokens used, 678% growth month-over-month, and most importantly—engineers actively sharing what works. Remember: this isn't about forcing adoption through metrics. It's about creating transparency that encourages experimentation. The dashboard becomes a conversation starter, not a performance review. What metrics have you found most valuable for tracking innovation adoption in your teams? P.S. Links to the change management and experimentation posts in the comments for those catching up on the series. #AIAdoption #EngineeringLeadership #TechTransformation #AgileMetrics

Over the past six months, our team has taken significant steps to integrate coding agents into software engineering workflows. A recurring question I hear from teams is: “What is the impact of coding agents on SWE Productivity?” Impact can be assessed through multiple lenses: Code Quality: Does Copilot-generated code reduce bugs and minimize change-related outages? Velocity: Are engineers able to deliver pull requests at a faster pace? Developer Experience: Are engineers feeling more energized and empowered? Beyond these traditional metrics, I’ve been exploring a personal approach—measuring weekly time savings based on accepted Copilot-generated code. To do this, I built a VS Code plugin that locally tracks sessions with Copilot Chat and evaluates: COMPLEXITY: Based on language and structural patterns QUALITY: Adherence to coding guidelines defined in the Copilot instructions file VOLUME: Lines of code accepted At the end of each week, the plugin generates a report showing how much time Copilot has saved me—often adding several extra hours to my schedule. I’d love to hear what methods others are using to evaluate the productivity impact of coding agents.

What metrics should we use to measure AI’s impact? Laura Tacho, CTO at DX, recommends 3 key dimensions to track: 1. Utilization Are developers using the tools? Which tools and how often? - Weekly and daily active users - Percentage of PRs that are AI-assisted - Percentage of merged code that is AI-generated 2. Impact Are the tools delivering real value? Are they improving workflows or just adding overhead? - Direct impact: AI-driven time savings, developer satisfaction - Indirect impact: Improvements in metrics like PR throughput, Developer Experience Index, or Perceived Rate of Delivery 3. Cost Is the organisation getting a positive return on its investment? What high-value use cases exist that we should be replicating? - AI spend (total, and per developer) - Net time gain per developer - Agent hourly rate (HEH / AI spend) To learn more, read the latest article that I did in collaboration with Laura, called "How to Measure AI Impact in Engineering Teams" here: https://lnkd.in/e9FyxJyz

𝗠𝗮𝗻𝘆 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 𝗳𝗼𝗰𝘂𝘀 𝗼𝗻 𝘄𝗿𝗶𝘁𝗶𝗻𝗴 𝗰𝗼𝗱𝗲. But top engineers measure what matters I used to think writing clean code and following best practices was enough until I encountered real-world systems where slow response times, system failures, and scalability issues hurt user trust and business growth. If you want to break into top companies, build reliable, high-performing systems, and thrive as a modern software engineer, start focusing on the performance metrics that truly matter. 𝗛𝗲𝗿𝗲’𝘀 𝘁𝗵𝗲 𝘀𝗸𝗶𝗹𝗹 𝘀𝗲𝘁 𝘁𝗵𝗮𝘁 𝘁𝗼𝗽 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 𝗳𝗼𝗰𝘂𝘀 𝗼𝗻: 🔹 System & Application Performance – Monitors speed, load, and system efficiency. → TTFB, Response Time, Throughput, Resource Utilization 🔹 Reliability & Resilience – Tracks system stability and failure recovery. → Error Rate, MTTR, Crash Rate 🔹 Network & Infrastructure – Evaluates connectivity, speed, and reliability. → Network Latency, Bandwidth, DNS Resolution Time 🔹 Business-Centric Metrics – Connects technical metrics to user outcomes. → Conversion Rate, Drop-off Rate, Session Duration 🔹 Development & CI/CD – Optimizes code quality and delivery speed. → Build Time, Deployment Frequency, Test Pass Rate 🔹 User Experience (UX) – Improves speed, interactivity, and usability. → FCP, LCP, CLS, INP, TTI You don’t have to master everything at once. Pick one category, apply it to a project, and start leveling up. Which performance metrics are you currently stuck on? 📌 Save this. 🔔 Follow Tauseef Fayyaz for more software and career growth content. #softwareengineering #kpi #systemdesign #uxmetrics #softwaremetrics #devtools #cicd #testing #webdev #uxdesign #productengineering

𝐃𝐞𝐯𝐎𝐩𝐬 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 𝐓𝐡𝐚𝐭 𝐀𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐌𝐚𝐭𝐭𝐞𝐫 Most teams measure the wrong things. They track commits per day, lines of code, hours spent deploying. These are vanity metrics—they show activity, not impact. 𝐇𝐞𝐫𝐞'𝐬 𝐰𝐡𝐚𝐭 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐦𝐚𝐭𝐭𝐞𝐫𝐬: 𝐃𝐎𝐑𝐀 𝐌𝐞𝐭𝐫𝐢𝐜𝐬 (The only 4 metrics proven to predict software delivery performance) --- 𝐓𝐇𝐄 𝟒 𝐃𝐎𝐑𝐀 𝐌𝐄𝐓𝐑𝐈𝐂𝐒: 𝟏. 𝐃𝐄𝐏𝐋𝐎𝐘𝐌𝐄𝐍𝐓 𝐅𝐑𝐄𝐐𝐔𝐄𝐍𝐂𝐘 How often you deploy to production ✅ High frequency = faster feedback loops ✅ Indicates automation maturity Elite teams: Multiple times per day Low performers: Once per month 𝟐. 𝐋𝐄𝐀𝐃 𝐓𝐈𝐌𝐄 𝐅𝐎𝐑 𝐂𝐇𝐀𝐍𝐆𝐄𝐒 Time from commit → production ✅ Shorter lead time = faster value delivery ✅ Shows pipeline efficiency Elite teams: Less than 1 hour Low performers: 1-6 months 𝟑. 𝐂𝐇𝐀𝐍𝐆𝐄 𝐅𝐀𝐈𝐋𝐔𝐑𝐄 𝐑𝐀𝐓𝐄 % of deployments causing incidents ✅ Low failure rate = quality releases ✅ Stability over speed Elite teams: 0-15% Low performers: 46-60% 𝟒. 𝐌𝐄𝐀𝐍 𝐓𝐈𝐌𝐄 𝐓𝐎 𝐑𝐄𝐂𝐎𝐕𝐄𝐑𝐘 How fast you recover from failure ✅ Fast recovery > zero failures ✅ Resilience matters Elite teams: Less than 1 hour Low performers: 1 week to 1 month --- 𝐇𝐎𝐖 𝐓𝐎 𝐈𝐌𝐏𝐋𝐄𝐌𝐄𝐍𝐓 𝐃𝐎𝐑𝐀 𝐌𝐄𝐓𝐑𝐈𝐂𝐒 Week 1: Measure Current State → Calculate your baseline DORA metrics → Identify your biggest bottleneck → Set improvement targets Week 2-4: Automate → CI/CD pipeline (reduce lead time) → Automated testing (reduce failure rate) → Monitoring & alerts (reduce MTTR) Month 2+: Optimize → Increase deployment frequency gradually → Reduce batch sizes → Improve observability → Build blameless post-mortem culture --- What DORA metric is your team struggling with most? Drop a comment—let's discuss how to improve it. ♻️ Repost if you found it valuable ➕ Follow Jaswindder for more insights #DevOps #DORAMetrics #CloudEngineering #SoftwareDelivery #ContinuousDeployment

Measuring the Success of Scaled CAE Teams #CAE #engineering #metrics #performance Scaling CAE teams is a critical step in maximizing the impact of simulation on product development and business performance. But how do you know if your scaled CAE operations are truly successful? In this post, we'll explore key metrics and best practices for measuring the effectiveness and efficiency of CAE teams at scale. Simulation Throughput and Cycle Time • Track the number of simulations completed per time period (e.g., week, month) to assess productivity • Monitor the time from simulation request to delivery of results to identify bottlenecks and improvement opportunities • Benchmark simulation throughput and cycle time against industry peers and best-in-class performers Model Accuracy and Validation • Measure the correlation between simulation predictions and physical test results to assess model accuracy • Track the percentage of models that meet validation criteria and identify areas for improvement • Monitor the impact of model accuracy on product quality, performance, and development time Return on Investment (ROI) • Quantify the financial impact of CAE activities, such as cost savings from reduced physical testing or warranty claims • Track the ROI of CAE investments, including software, hardware, and personnel costs • Communicate the value of CAE to senior leadership and cross-functional stakeholders using ROI metrics Stakeholder Satisfaction and Engagement • Conduct surveys to assess the satisfaction of CAE customers, such as design engineers and product managers • Monitor the adoption and utilization of CAE tools and processes across the organization • Foster a culture of continuous feedback and improvement to ensure CAE is meeting the needs of all stakeholders Talent Development and Retention • Track the skills and certifications of CAE team members to ensure continuous growth and development • Monitor employee engagement and turnover rates to identify opportunities for improvement • Benchmark compensation and benefits against industry peers to attract and retain top CAE talent By regularly measuring and monitoring these key metrics, CAE leaders can gain valuable insights into the performance and impact of their teams. Armed with this data, they can make informed decisions to optimize processes, invest in new technologies, and drive continuous improvement. What metrics do you use to measure the success of your CAE teams? How do you ensure these metrics are aligned with business goals and stakeholder needs? Share your thoughts and experiences in the comments. #KPIs #ROI #talentdevelopment #continuousimprovement #stakeholderengagement #benchmarking

In studies on tech productivity, the best teams typically outperform the worst by a 10:1 ratio. A CEO friend asked me: what should I measure to help my organisation become more of a 10x organisation? Here is 𝗣𝗔𝗥𝗧 𝗧𝗪𝗢 of the answer I gave him. The best starting point to measure productivity in software are the 4 key metrics of Accelerate: - change lead time; - deployment frequency; - mean time to restore (MTTR); - and change fail percentage. They are based on extensive research and have passed the test of time. But to make sure tech productivity is aligned with actual value and these metrics are used to learn and improve, rather than micro-manage, I embed the 4 key metrics inside the broader #LeanTech frame. [Part 2 out of 2: connecting Accelerate to the Lean body of knowledge] 𝗥𝗶𝗴𝗵𝘁-𝗙𝗶𝗿𝘀𝘁-𝗧𝗶𝗺𝗲 In Lean wisdom, improving productivity starts with quality. Ensuring every team produces high quality reduces the amount of rework and the frustrating back-and-forth with QA down the line. In #LeanTech, we measure quality by tracking the number of defects, categorised by stages of detection. We call this approach "dantotsu". Accelerate proposes two metrics that are simpler to adopt and still very effective: - Change failure rate: typically the rate of deployments that are followed by a rollback/fix deployment - Mean time to recovery (MTTR): time between the deployment that failed and the deployment that fixed it 𝗝𝘂𝘀𝘁-𝗜𝗻-𝗧𝗶𝗺𝗲 We are finally addressing the more traditional understanding of productivity: how fast teams are working. In #LeanTech we recommend measuring "feature lead-time", the calendar time between a decision to invest in a new feature and the moment it is finally in the hands of the users. Daniel Terhorst-North calls it "time to thank you". Accelerate proposes again two metrics that are simpler to adopt than lead-time but still very effective: - deployment frequency, which is easier to measure than lead-time and strongly correlated to it (see Little's Law). The trade-off is the lack of granularity in the data - lead-time from commit to production, which measures the technical time needed to deploy a change. For Lean experts, this reminds us of SMED. 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗢𝗿𝗴𝗮𝗻𝗶𝘀𝗮𝘁𝗶𝗼𝗻 The final aspect is ensuring these efforts are sustained, by investing in human capital. Examples of how to measure this: - a skills matrix, to measure the progression levels of each team-member on the different skills they need to succeed - the number of "dantotsu" defect root-cause analyses, to track how seriously the organisation is problem-solving and learning from its mistakes. Embedding Accelerate's 4 key metrics within the Lean Tech system provides an important systemic understanding of productivity. Because productivity is only good if aligned to value and sustainable if obtained by investing in human capital. Curious to hear about other experiences or challenges on this topic?

The real story of AI in software engineering isn’t “30% of code is AI‑written.” It’s what that actually means for speed, quality, and business value. Here are some of the AI metrics our customers are using to cut through the noise: 1. Cohort-based adoption: Who are your light, moderate, and heavy AI users? 2. Before-and-after trend analysis: How do key metrics change when devs actually embrace these tools? 3. PR throughput: Is AI helping teams deliver faster? 4. PR revert rates: Are we trading speed for rework and defects? 5. Cycle time: Is AI-assisted work making reviews and delivery smoother, or slower? This is how leading orgs are moving past vanity metrics to measure real impact. #dx #aimeasurement #genAI #devex