Measuring Developer Productivity—In Defense of “Developer Intelligence”

The debate is raging right now in development circles. Sure, McKinsey might have sparked the latest flare up, but it’s certainly not a new argument—should we measure the productivity of software developers? And how to go about it.

I, for one, think it’s high time we hashed it out and put it to bed.

Developers aren’t automatons…

It seems most people deeply involved in software development agree that you can’t measure developer productivity strictly based on output. The impact of generative AI’s surprisingly effective ability to produce code quickly has placed this issue in the spotlight. Now, developers tasked with producing relatively simple code—often junior team members with less experience—can as much as double their output. But, clearly, that doesn’t mean they’re doubling the business value being produced, or that they’re actually outperforming senior team members whose responsibilities involve far less straight coding and more of the creative, analytical, and mentoring tasks that contribute so much to the overall quality, consistency, and competitive differentiation required for businesses to succeed.

They’re not automatons or workers on an assembly line churning out code like widgets. There’s as much art as science to what these talented professionals accomplish, and it leans more on the art side of the equation the more skilled and experienced they become. So, applying some formulaic combination of productivity metrics to an entire team of developers based on lines of code written or bugs squashed makes no sense.

…but leaders need metrics to lead

At the same time, I’ve been working with clients for over twenty years now, and they’ve always wanted a way to measure the efficacy of their software teams. It just makes good business sense: we studiously measure, evaluate, and seek to improve every other aspect of our businesses; why would we not do so for what has become an actual core competency in most organizations? Especially considering technology workers represent the biggest or second biggest item on their P&L.

Executives rightly want and need better visibility into how teams and individuals are performing, and they want accountability that will support improvement. They need to be able to translate engineering work into business value and drive alignment at a strategic level.

As a CEO, I get that. It would be foolish not to want all those things.

So, is there a more nuanced take on this seemingly unresolvable debate? I think there is, but it requires a more pragmatic, holistic, and nuanced approach than many in the industry have considered.

Measuring Value or Productivity—instead of OR, can it be an AND?

Moving beyond the simplicity of the ‘value or productivity’ debate, it’s important to explore how these concepts can coexist and complement each other in a balanced Agile approach. Of course, this is not a fundamental problem with Agile itself. And it’s not a problem with the concept of measuring what developers do. It’s really a cultural problem.

Agilists say it’s all about the value being created—and they’re right. Value is definitely the outcome we’re all driving for. But here’s the thing—value is a lagging indicator. Plus, it’s not the tech team’s fault if they rapidly deliver all the stories the business has prioritized, but it turns out those stories don’t move the dial on business value.

Measuring value is vital, but elusive and complex. What constitutes value is nuanced, so it’s slightly different in every organization and every product. Identifying and applying one all-encompassing formula to divining it is an exercise in futility.

Overcoming our reluctance to measure productivity is key, but it’s more than just tracking tasks like coding, testing, or deployments. Productivity metrics are vital not just as leading indicators of value but as tools to gauge the efficiency and direction of our systems. They inform us if we’re delivering value timely and effectively. It’s about ensuring that when an idea enters our system, it emerges as a predictable, reliable product. Understanding and respecting the time aspect of delivery, alongside quality and capability, is what truly bridges gaps, eliminates bottlenecks, and optimizes resources.

It’s not an either/or proposition. Productivity and value go hand-in-hand. Without getting visibility into one, we’ll never truly understand the other, and the transparency, accountability, and continual improvement Agile promises will never come to fruition.

Productivity and value go hand-in-hand. Without getting visibility into one, we’ll never truly understand the other, and the transparency, accountability, and continual improvement Agile promises will never come to fruition.

So, how do we fix it?

This isn’t a one size fits all, but here’s what the experts at Cprime know from our centuries of collective experience:

1. You need to get past the cultural aversion to measuring developer productivity.
2. You need to experiment to establish a set of productivity metrics that offer fair and consistent feedback that makes sense for your unique business goals.
3. You need to accept and account for the fact that high-performing teams are not necessarily filled with individuals who all equally produce tremendous amounts of code.
4. You need to establish feedback loops that marry productivity and value metrics so you can really see where you’re going and how you’re getting there.
5. And you need to set up these systems to be as streamlined, automated, and widely accessible as possible. (We’ve found tools like Jira, Jira Align, Gitlab, and Apptio do wonders in this regard; and a tool like Allstacks can bring them all together for across-the-board reporting and analysis.)

Towards a more effective measurement approach

What you measure is what you get, so it’s crucial to set metrics that push you towards your goals. It’s all about context—tailor your metrics to what your business needs right now, knowing they might change down the road. Let’s use developer productivity as an example:

If customer churn is spiking due to buggy software, it’s time to hit the brakes. Slow down, focus on thorough testing and code reviews, and aim to catch more issues pre-launch. Key metrics? 

Think:

• Velocity
• Pull request cycle time
• Test coverage
• Found defects
• Escaped defects
• Mean time between failures
• Mean time to recovery
• Change failure rate

In this case, contrary to the norm in an Agile environment, we actually want velocity to go down so that test coverage, PR cycle time and found defects all go up, and escaped defects goes down. It’s a balancing act—shifting gears in some areas to ramp up quality and customer satisfaction.

Here are some more key areas where our measurement approach can evolve for better alignment of output and value:

• Measuring Agile performance: A useful metric to consider is the commitment accuracy, from the original story to what is actually delivered, to quantify the team’s ability to understand and meet project requirements.Mean time to pivot is another critical measure of agility and responsiveness.
• Code reviews and retrospectives: Code reviews and retrospectives are fundamental for continuous improvement in Agile methodologies, and foster a culture of collective learning and accountability.
• Simplifying development processes: The ability to develop solutions with less complexity, exemplified by reducing the amount of code without compromising functionality, is a clear indicator of effectiveness.
• Measuring continual learning: Measuring Developer Productivity—In Defense of “Developer Intelligence”
• Speed and reliability in getting products to market: Speed and reliability in delivering features not only reflect the team’s efficiency but also their alignment with market demands and business value.

Let’s keep talking—and evolving

We’re at a crossroads: either we embrace linking measurement directly to the value we create and using it to champion our teams’ needs, or we keep clinging to outdated gripes about single metrics and how executives don’t understand the engineering craft. The latter is a one-way ticket to restrictive governance and off-target metrics. I believe it’s clear that we have to embrace the right measures for the health of our teams, employees and success of our organizations.

I hosted a Webinar panel (watch it here) on December 13th, with experts from Allstacks, Atlassian, Agile Alliance, and Cprime, Inc, where we’ll dive deep into these productivity metrics from various perspectives. And yes, we’ll tackle how GenAI is rewriting the rulebook on software development and measurement. But this conversation is as much yours as it is ours. Drop your thoughts below—let’s collectively define how we measure, value, and advocate for our work in this ever-evolving landscape.

And finally, at Cprime we developed and have been using the PRIME approach for incrementally defining and validating value-based business metrics. Look forward to a future article where we’ll discuss how developer productivity fits into this holistic approach.