Developer Productivity Metrics

You're a #CTO. Your board asks: "What's our ROI on AI coding tools?" Your answer: "40% of our code is AI-generated!" They respond: "So what? Are we shipping faster? Are customers happier?" Most CTOs are measuring AI impact completely wrong. Here's what some are tracking: - Percentage of AI-generated code - Developer hours saved per week - Lines of code produced - AI tool adoption rates These metrics are like measuring how fast your assembly line workers attach parts while ignoring whether your cars actually start. Here's what you SHOULD measure instead: 1. Delivered business value 2. Customer cycle time 3. Development throughput 4. Quality and reliability 5. Total cost of delivery (not just development) 6. Team satisfaction Software development isn't a typing competition—it's a complex system. If AI makes your developers 30% faster but your deployment takes 2 weeks and QA adds another week, your customer delivery improves by maybe 7%. You've speed up the wrong part. The solution: A/B test your teams. Give half your teams AI tools, measure business outcomes over 2-3 release cycles. Track what customers actually experience, not how much developers produce. Companies that measure business impact from AI will pull ahead. Those measuring vanity metrics will wonder why their expensive tools aren't moving the needle. Stop measuring how much code AI generates. Start measuring how much faster you deliver value to customers. What are you actually measuring? And is it moving your business forward? -> Follow me for more about building great tech organizations at scale. More insights in my book "All Hands on Tech"

In many Chinese schools, students pause class for 1–3 minutes and move together — inside the classroom. Are you taking breaks during your office hours? Not a dance. Not military. System design. It’s called 广播体操 (Radio Calisthenics) and it’s been used nationally for decades to reset posture, circulation, and attention. • Prolonged sitting reduces cognitive performance after 30–40 minutes • Short movement breaks improve focus and working memory by 10–15% • Light physical activity increases blood flow to the brain by up to 20% • Even 2 minutes of movement measurably reduces mental fatigue Now apply this to tech and business. Knowledge workers sit 9–11 hours/day, live in back-to-back video calls, and are expected to make high-quality decisions at speed. That’s not a productivity issue. It’s a human-system mismatch. As AI scales execution, human attention becomes the bottleneck. The next performance upgrade may not be more software — but movement designed into workflows. China implemented it at national scale. Optimize the human. Then optimize the system. #FutureOfWork #AI #Productivity #Leadership #HumanPerformance #Neuroscience #TechLeadership #DigitalTransformation #WorkplaceDesign #CognitivePerformance

LLMs are optimized for next turn response. This results in poor Human-AI collaboration, as it doesn't help users achieve their goals or clarify intent. A new model CollabLLM is optimized for long-term collaboration. The paper "CollabLLM: From Passive Responders to Active Collaborators" by Stanford University and Microsoft researchers tests this approach to improving outcomes from LLM interaction. (link in comments) 💡 CollabLLM transforms AI from passive responders to active collaborators. Traditional LLMs focus on single-turn responses, often missing user intent and leading to inefficient conversations. CollabLLM introduces a :"Multiturn-aware reward" system, apply reinforcement fine-tuning on these rewards. This enables AI to engage in deeper, more interactive exchanges by actively uncovering user intent and guiding users toward their goals. 🔄 Multiturn-aware rewards optimize long-term collaboration. Unlike standard reinforcement learning that prioritizes immediate responses, CollabLLM uses forward sampling - simulating potential conversations - to estimate the long-term value of interactions. This approach improves interactivity by 46.3% and enhances task performance by 18.5%, making conversations more productive and user-centered. 📊 CollabLLM outperforms traditional models in complex tasks. In document editing, coding assistance, and math problem-solving, CollabLLM increases user satisfaction by 17.6% and reduces time spent by 10.4%. It ensures that AI-generated content aligns with user expectations through dynamic feedback loops. 🤝 Proactive intent discovery leads to better responses. Unlike standard LLMs that assume user needs, CollabLLM asks clarifying questions before responding, leading to more accurate and relevant answers. This results in higher-quality output and a smoother user experience. 🚀 CollabLLM generalizes well across different domains. Tested on the Abg-CoQA conversational QA benchmark, CollabLLM proactively asked clarifying questions 52.8% of the time, compared to just 15.4% for GPT-4o. This demonstrates its ability to handle ambiguous queries effectively, making it more adaptable to real-world scenarios. 🔬 Real-world studies confirm efficiency and engagement gains. A 201-person user study showed that CollabLLM-generated documents received higher quality ratings (8.50/10) and sustained higher engagement over multiple turns, unlike baseline models, which saw declining satisfaction in longer conversations. It is time to move beyond the single-step LLM responses that we have been used to, to interactions that lead to where we want to go. This is a useful advance to better human-AI collaboration. It's a critical topic, I'll be sharing a lot more on how we can get there.

Probably the simplest most-effective way to improve productivity is to reduce your work in progress (things you work on simultaneously) to 1. Think about a situation where you must work with a "platform team." Your team is bopping along until it comes across something it needs to do that the platform can't handle. It then stops work and hands off to the platform team. Rather than being idle while it waits, the first team now starts working on a second thing until it needs a database change, which it hands off to the database team. Not wanting to be idle, it starts working on a third thing. Weinberg points out that every "thing" you work on reduces productivity by about 20%. So, if you have three 5-day tasks. Working on two of them at once adds 20% to each task, so it will take 12 days to do 10 days of work. Add a third task and we're adding 2 days to each task, so it now will take 21 days to do 15 days of work. This isn't even considering what happens if the other team gets it wrong and you need to resubmit the request or the fact that it now takes up to four times longer (21 days rather than 5) to get something useful into your customer's hands. So, to work on only one thing at a time, we need to eliminate the dependencies. Our single product team needs to be able to make platform and database changes (safe ones, at least, to avoid collisions with other teams). They need to align with the other teams when they make those changes so that they don't break anything, but I find that an occasional chapter/guild meeting to deal with consistency issues takes way less time than the time you lose to WIP>1.

As a client project manager, I consistently found that offshore software development teams from major providers like Infosys, Accenture, IBM, and others delivered software that failed 1/3rd of our UAT tests after the provider's independent dedicated QA teams passed it. And when we got a fix back, it failed at the same rate, meaning some features cycled through Dev/QA/UAT ten times before they worked. I got to know some of the onshore technical leaders from these companies well enough for them to tell me confidentially that we were getting such poor quality because the offshore teams were full of junior developers who didn't know what they were doing and didn't use any modern software engineering practices like Test Driven Development. And their dedicated QA teams couldn't prevent these quality issues because they were full of junior testers who didn't know what they were doing, didn't automate tests and were ordered to test and pass everything quickly to avoid falling behind schedule. So, poor quality development and QA practices were built into the system development process, and independent QA teams didn't fix it. Independent dedicated QA teams are an outdated and costly approach to quality. It's like a car factory that consistently produces defect-ridden vehicles only to disassemble and fix them later. Instead of testing and fixing features at the end, we should build quality into the process from the start. Modern engineering teams do this by working in cross-functional teams. Teams that use test-driven development approaches to define testable requirements and continuously review, test, and integrate their work. This allows them to catch and address issues early, resulting in faster, more efficient, and higher-quality development. In modern engineering teams, QA specialists are quality champions. Their expertise strengthens the team’s ability to build robust systems, ensuring quality is integral to how the product is built from the outset. The old model, where testing is done after development, belongs in the past. Today, quality is everyone’s responsibility—not through role dilution but through shared accountability, collaboration, and modern engineering practices.

Has Amazon cracked the code on developer productivity with its cost to serve software (CTS-SW) metric? Amazon applied its well-known "working backwards" methodology to developer productivity. "Working backwards" in this case starting with the outcome: concrete returns for the business. This is measured by looking at the rate of customer-facing changes delivered by developers, i.e. "what the team deems valuable enough to review, merge, deploy, and support for customers", in the words of the blog post by Jim Haughwout https://lnkd.in/eqvW5wbi . This metric is different from other measures of developer productivity which look only at velocity or time saved. Instead, "CTS-SW directly links investments in the developer experience to those outcomes by assessing how frequently we deliver new or better experiences. Some organizations fall into the anti-pattern of calculating minutes saved to measure value, but that approach isn’t customer-centered and doesn’t prove value creation." This aligns with Gartner's own research on developer productivity. In our 2024 Software Engineering survey, we asked what productivity metric organizations are using to measure their developers. We also asked about a basket of ten success metrics, including software usability, retention of top performers, and meeting security standards. This allowed us to find out which productivity metric was associated most with success. What we found in our survey was that *rate of customer-facing changes* is the metric most associated with success. Some other productivity metrics were actually *negative associated* with success. But *rate of customer-facing changes* is what organizations should focus on. Sadly, our survey found that few organizations (just 22%) use this metric. I presented this data at our #GartnerApps summit [and the next summit is coming up in September: https://lnkd.in/ey2kpc2 ] Every metrics gets gamed. So I always recommend "gaming the gaming". A developer might game the CTS-SW metric by focusing more on customer-facing changes. But... this is actually a good thing. You're gaming the gaming. We will be watching closely how this metric gets adopted alongside DORA, SPACE, and other metrics in the industry.

Stanford released the first systematic study of local AI efficiency - and the results seem really interesting! 🔥 Their main insight is this intelligence/watt metric, which measures the efficiency of an LLM model as a function of: Task accuracy ÷ power consumption. Simple, yet captures both what your model can DO and how much energy it burns doing it. They looked at 20+ local models (≤20B params) and tested across 1M real-world queries from WildChat, Natural Reasoning, MMLU Pro, and SuperGPQA. Hardware spanning Apple M4 Max, RTX Quadro, NVIDIA H200/B200, AMD MI300X. Full telemetry: accuracy, latency, energy, throughput, everything. (essentially datasets of tasks that measure things like world knowledge, ability to reason, ability to chat and so on...) Two cool trends observed: 📈 Local model capability: 3.1× improvement from 2023 until 2025  - 2023: 23.2% win/tie rate vs frontier models - 2024: 48.7% - 2025: 71.3% Local models went from handling ~1 in 4 queries to ~3 in 4 queries in just two years! ⚡ Intelligence efficiency: 5.3× improvement - 2023: 7.92e-4 acc/W (Mixtral-8x7B on RTX 6000) - 2024: 1.80e-3 acc/W (Llama-3.1-8B on RTX 6000 Ada)   - 2025: 4.18e-3 acc/W (GPT-OSS-120B on M4 Max) That's 3.1× from better models + 1.7× from better accelerators = compounding gains! 88.7% of single-turn queries can run locally NOW. With smart routing between local + cloud models, you get 60-80% savings on energy/compute/cost while maintaining quality. Even at 80% routing accuracy (totally realistic), you capture most theoretical gains. What I like is that this infrastructure shift from centralized cloud to distributed local+cloud is happening RIGHT NOW, and these are the metrics that prove it's viable. B) (link to paper in the comments) #AI #LocalAI #EfficientAI #LLMs

𝐓𝐡𝐞 "𝐁𝐮𝐬𝐲" 𝐓𝐫𝐚𝐩: 𝐖𝐡𝐲 𝐘𝐨𝐮𝐫 𝐁𝐞𝐬𝐭 𝐖𝐨𝐫𝐤 𝐑𝐞𝐪𝐮𝐢𝐫𝐞𝐬 𝐘𝐨𝐮𝐫 𝐀𝐛𝐬𝐞𝐧𝐜𝐞 For the longest time, I viewed "taking a break" as a sign of slowing down. I thought if I wasn't constantly tethered to my notifications or strategizing the next framework, I was losing momentum. We tend to ignore the quiet whispers of burnout until they become a roar. We tell ourselves "just one more week" or "after this project," not realizing that a tired mind cannot innovate; it can only replicate. Last week, I finally put the "Out of Office" on and traded my screen for the shoreline. There is something transformative about the ocean. Watching the tide reminded me that life has a natural rhythm of receding and returning. I spent my days disconnected from the digital world and reconnected with the physical one—the warmth of the sand, the sound of the waves, and the clarity of silence. The result? My energy didn't just return; it doubled. I arrived back at my desk this morning, and the timing couldn't have been more intense. A high-stakes, incredibly challenging project was waiting for me on day one. Six days ago, that project might have felt overwhelming. Today? My headspace is clear. My perspective is fresh. I’m not just ready to tackle the challenge; I’m excited to lead it. The Lesson: Productivity isn't about how many hours you sit at your desk; it’s about the quality of the energy you bring to those hours. If you’re waiting for the "right time" to take a breath—this is your sign. Go find your version of the beach. Your work (and your well-being) will thank you when you get back. #Leadership #WorkLifeBalance #MentalHealth #BurnoutPrevention #Productivity #PeopleFirst

GenAI is delivering productivity gains of up to 20% to the software development lifecycle, and Deloitte’s latest research dives into how GenAI is driving this transformation. Faruk Muratovic, Diana Kearns-Manolatos (she/her), and Ahmed Alibage, CMS®, Ph.D. recently published an insightful report in the IEEE Computer Society’s journal [https://deloi.tt/3TtkCC6]. Their findings highlight not only productivity gains, but also the importance of trust and transparency.     Building trust in GenAI starts with thoughtful human oversight. The report recommends keeping humans-in-the-loop (HITL) to ensure code quality, manage risk, and provide transparency. These key actions stand out:    •Promote design transparency and explainability: By fostering open, iterative design, teams can balance innovation with consistent, high-quality results.  •Strengthen code accuracy with clear metrics: Leveraging repeatable measures like defect density and time-to-delivery helps maintain quality and build confidence in GenAI-driven solutions.  •Create a culture of continuous learning and improvement. As GenAI evolves, teams will stay resilient and innovative.    By taking these actions, tech leaders can help build a future where technology and human expertise go hand in hand—delivering real value, safely and responsibly.   

Step-by-Step Guide to Measuring & Enhancing GCC Productivity - Define it, measure it, improve it, and scale it. Most companies set up Global Capability Centers (GCCs) for efficiency, speed, and innovation—but few have a clear playbook to measure and improve productivity. Here’s a 7-step framework to get you started: 1. Define Productivity for Your GCC Productivity means different things across industries. Is it faster delivery, cost reduction, innovation, or business impact? Pro tip: Avoid vanity metrics. Focus on outcomes aligned with enterprise goals. Example: A retail GCC might define productivity as “software features that boost e-commerce conversion by 10%.” 2. Select the Right Metrics Use frameworks like DORA and SPACE. A mix of speed, quality, and satisfaction metrics works best. Core metrics to consider: • Deployment Frequency • Lead Time for Change • Change Failure Rate • Time to Restore Service • Developer Satisfaction • Business Impact Metrics Tip: Tools like GitHub, Jira, and OpsLevel can automate data collection. 3. Establish a Baseline Track metrics over 2–3 months. Don’t rush to judge performance—account for ramp-up time. Benchmark against industry standards (e.g., DORA elite performers deploy daily with <1% failure). 4. Identify & Fix Roadblocks Use data + developer feedback. Common issues include slow CI/CD, knowledge silos, and low morale. Fixes: • Automate pipelines • Create shared documentation • Protect developer “focus time” 5. Leverage Technology & AI Tools like GitHub Copilot, generative AI for testing, and cloud platforms can cut dev time and boost quality. Example: Using AI in code reviews can reduce cycles by 20%. 6. Foster a Culture of Continuous Improvement This isn’t a one-time initiative. Review metrics monthly. Celebrate wins. Encourage experimentation. Involve devs in decision-making. Align incentives with outcomes. 7. Scale Across All Locations Standardize what works. Share best practices. Adapt for local strengths. Example: Replicate a high-performing CI/CD pipeline across locations for consistent deployment frequency. Bottom line: Productivity is not just about output. It’s about value. Zinnov Dipanwita Ghosh Namita Adavi ieswariya k Karthik Padmanabhan Amita Goyal Amaresh N. Sagar Kulkarni Hani Mukhey Komal Shah Rohit Nair Mohammed Faraz Khan