The Best Metric for Your Product Team

Sometimes metrics can get in the way of a good conversation.

A few weeks back, I had a conversation with a peer coach who was introduced to me. As we chatted, we discussed several aspects of coaching. At one point, he asked me a question that has proven difficult for me to answer.

Well, to clarify, I gave an immediate answer, but I have since changed my mind about it. The question has been weighing heavy on my mind for the past few weeks. I keep coming back to it. After pondering on it, I have a much better answer.

The other coach asked, “If you could track only two metrics on a product team, what would they be?”

As he was asking the question, I became triggered as soon as he uttered the word “metrics.” My experience with metrics has been poor, to say the least. They are far too often used as a weapon versus input for improvement. Metrics tend to make teams feel like a laboratory experiment versus human beings.

But nonetheless, I answered the question. And I answered with the first thing that came to my mind. I am not proud of this initial answer I gave. If I were to give an answer today, it would be much more thought-out.

In response to the question, I said, “Well, it is obvious. You would want to track metrics for user adoption and lead time.”

And the words spilled out far too easy. I fell into the trap everyone falls into when it comes to metrics. In my rush to give an answer, I forgot something. I missed the one thing more important than any outcome or time-to-market indicator.

I’ll get back later to the one thing I missed. But for now, let’s unpack this by first outlining some of the perils of metrics. And then, I will tell you how I would respond today if asked the same question.

The Perils of Metrics

Metrics can go down the wrong path fast. While we all start down the metric journey with good intentions, it is easy to take things in a poor direction. When we focus on the numbers, we often ignore the people making the product or using the product. We become seduced by numbers.

Let’s explore three pitfalls when it comes to metrics.

№1: Succumbing to Metric Overkill

Let’s track everything. After all, why have one indicator of performance when you can have many? More is better, right? Well, no, it is not when it comes to teams developing a product.

Let’s say a team is being tracked on these metrics:

• User adoption by feature
• User loss over time
• New users by region
• Average velocity over time
• Cycle time
• Lead time
• Cumulative flow
• Work-in-progress
• Mean backlog item age
• Sprint burn-down
• Release burn-up
• Mean time to recover
• Escaped defect rate
• Percent code coverage for unit tests
  

Which metric should the team focus on today? This is not an easy answer, and it depends on an ever-shifting context. And improving one metric might impact another one in a negative manner. Usually, it boils down to which metric management cares about the most today.

“It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”

—W. Edwards Deming

And the more metrics you have, the more effort and people it takes to gather that data. To make it worse, we don’t take the time to assess if the measurement is worth tracking. In spite of all this, we are fast to take data tracking to the extreme.

We often go so far as to form entire teams around tracking the data, analyzing the data, and reporting the data. Unfortunately, these analytics teams are often isolated from the delivery teams and the end-users. Interpreting the data becomes akin to a laboratory analyzing samples or specimens. There is no human contact and no context to gain empathy.

So less is more when it comes to measurements. But it is not where our instincts take us.

№2: Comparing Teams

Once you compare teams against each other using a metric, the metric becomes useless. Teams become concerned with their measurement value in comparison to their peers. This creates an unhealthy focus on the target value of the metric.

Per Goodhart’s law, when you measure teams against a target value, they optimize behavior to it. In other words, they begin to find workarounds to achieve the target value. You lose the original intent of tracking the metric.

“Tell me how you will measure me, and then I will tell you how I will behave.”

—Eli Goldratt

And people do behave differently when they think they are being watched. This is known as the Hawthorne effect.

Let’s take a metric that receives reliable abuse in this manner—Velocity. For a team, Velocity sums the number of Story Points for Stories the team delivered to a “Done” state in a Sprint. Velocity is a planning tool for teams and nothing more. Teams calculate an average Velocity over the past three or so Sprints. And they use the average to forecast how much Story effort they can deliver in future Sprints.

Unfortunately, this is not how those interested in a team’s performance use Velocity. Often, stakeholders will use Velocity as a way of comparing a team’s performance to another team. This changes what should be a great forecasting tool into a performance target. And it creates unhealthy competition between the teams.

The end result is the manipulation of Velocity in a way that makes it meaningless. Here are some of the more common ways I have seen Velocity gamed:

• Large Stories have more points, so the team decides not to split the Story smaller
• The team estimates Stories to be larger than they are so they count for more points
• To consider a story “Done” in the Sprint, the team cuts corners on quality
• The team reduces transparency and minimizes issues it finds in favor of counting the story as “Done”
  

The results of these behaviors act as a drag to team productivity. And we have lost the original intent of Velocity as a planning mechanism.

If this can happen with Velocity, it can happen with any metric that becomes a target.

№3: Overemphasizing Objective Data

Schools teach us to be objective from a young age. Bias has no place in decision making. The business world propagates this thinking. As a result, we have an obsession with objective, quantitative data. And we often ignore or reduce the value of qualitative data.

When our teams do not have psychological safety, a reduction in transparency occurs. And reduced transparency makes qualitative data even more suspect and undesirable as it is not trusted as being complete.

“Whenever there is fear, you will get wrong figures.”

—W. Edwards Deming

So we dive deep into numbers, graphs, charts, and other performance indicators. Talking to the team becomes less and less commonplace. And this does nothing to improve the team’s feeling of safety in its environment. In fact, it often reduces safety further.

I remember a time when a company I coached decided to track team performance in an automated manner. They analyzed logs from Jira, the continuous delivery pipeline, and the support team. Great effort went into automating the tracking of this data without human intervention.

This was all done because they did not trust information coming from the teams. They were trying to avoid teams gaming the numbers. As a result, communication with teams reduced. And the core problem of psychological safety was not addressed. In the end, the company reduced safety further since they consulted the teams less.

The One Metric You Need

By this point, I am sure you are thinking I don’t like data. Let me set the record straight. My argument is not to convey that data is bad. Data is necessary. Evidence is the cornerstone of solid decision making.

“Without data, you’re just another person with an opinion.”

—W. Edwards Deming

But the three pitfalls—overkill, comparison, and objectivity—block effective use of data. So what are we to do?

I have been considering the best answer to this over the past several weeks. I have concluded the answer lies with Team Engagement.

The Team Engagement Metric

The best metric to ensure product success is Team Engagement. You evaluate a team’s engagement on its level of autonomy, mastery, and purpose¹.

Engaged teams will be happier teams. These teams are in control of how they organize to achieve their goal. Engaged teams master their craft and learn ways to deliver better and operate better. And these teams have a purpose they are passionate about; one that drives and grounds them.

All the other possible tracking metrics remain valuable. But they need to stay within the boundaries of the team. An autonomous team will track the metrics appropriate for its context. Through mastery, the team tunes its behavior and product direction to achieve better results. And the team’s purpose will drive it to track the right metrics relevant to its context.

So does this mean we should crank up a Team Engagement tracking system? Should we get to work on a team happiness dashboard? Absolutely not. If we did this, we would be right back in the three pitfalls described earlier. Then, what should we do?

Talk to your teams.

Get out of the office and have a conversation with your teams at the place where the work happens. Go to the team rooms. Go to the Sprint Reviews. Engage in Backlog Refinement sessions. See where they need help. Query their feeling of engagement in their work.

We talk with words and not numbers and charts for a reason. The power of a conversation beats a metric or chart any day. It is rich, and it elevates understanding. Metrics and charts can enhance a conversation but do not substitute for it.

Let your teams track and respond to their own metrics. An engaged team with autonomy, mastery, and purpose should deeply care about the value it creates. And the team knows how best to measure the value it produces. 

Visit the team areas and talk to your teams on a regular basis. And support them when they need help. A funny thing will happen. Safety will start to improve. And Team Engagement levels will skyrocket. No tracking is necessary.

My Revised Answer

We started this post discussing my conversation with the other coach. He asked me about the best two metrics to track.

After my time to think, my more measured response would be as follows:

“Let the teams decide. Visit and talk with them often. And make sure they feel engaged and supported.”

This thing we call agility focuses on people working together and getting better at it. It is not about numbers and charts and graphs.

It is time we focus on enhancing Team Engagement by fostering team autonomy, mastery, and purpose. And this includes putting the metric tracking in the hands of our teams. Then, we can focus on giving them the environment and support they need to get the job done.

Also published in Serious Scrum on Medium.

---

---

Related Posts

Read posts related to this one below:

Three “Right” Ways to Develop Your Product

If Your Scrum Is Not Fun, You Are Doing It Wrong

We Need Managers to Become Agile Leaders

References

1. Drive – The Surprising Truth About What Motivates Us, Daniel H. Pink, 2011 ↩