Principles of Metrics Driven Management

It is certainly in vogue to use metrics to drive management decisions.  This is especially true in a distributed workforce.  There can be no pretending my team is working as I stroll by their desks late into the night.  In fact, my personal preference is to be remote from my employees to ensure I do not judge them by which of us leaves the office later at night.

Metrics can be pretty difficult to use. So, before you get any further into this, if you are unfamiliar with Accelerate: The Science of Lean software and Startups.  You should almost certainly familiarize yourself with their data driven approach and come back to this anecdotal piece after.

If you are still here then thanks.  My struggle with metrics for years in software is that most metrics appear objective and in fact are quite subjective because the target is subjective. Look at a simple metric like: Did this feature get delivered on time? Well, navigating the minefield of the quality, the actual vs expected team size, the actual vs. expected scope delivered.  All open multiple cans of worms for something that on its face seems simple. 

So, instead of giving up, I have a few principles that I like to apply to metrics used by my team.  Then in the future we can walk through some of the metrics I use on top of those defined by Accelerate.

1. Requirements Driven

This may seem self evident. However, I have rarely seen clear documentation on exactly how one specifies a requirement before it is driving team member behavior. For example, escaped defects is a metric teams like to use.  However, different executives care if those tickets are tied to support tickets or not. i.e. if the customer hasn’t found the ticket, then perhaps it’s not that severe.  Perhaps a defect only counts if it is within a key workflow path.  With a bit of brainstorming there are probably even more facets we should at least consider.

None of these specific facets are complicated to implement, yet if we do not think through them we will not be speaking the same language.  This is extremely important in all teams and even more so in a distributed environment.  Silveira in Building and Managing High-Performance Distributed Teams: Navigating the Future of Work talks about an explicit North Star so that team members can make decisions alone because they understand the direction the company is taking.  This metric example may seem too small, but not writing down the exact specification for a metric can either lead your team astray or just as bad accidentally lead you in the right direction which will reinforce this incorrect behavior.

We expect detailed docs to write code for our customers, it is reasonable to request this of the metrics we are measured by. In fact, opportunities arise when we treat these at first order features.

2. Produced via code

We are measuring software engineers, engineers generally write code to make money.  Our metrics should be built via the tools to which we are accustomed.  If I want to measure deployments or MTTR, then I should be able to view a report in Excel or Google Docs, or my favorite Airtable and see real time data about how things are going. 

Here, we are standing on the shoulders of principle 1.  If there are not requirements for how to calculate a metric, then we will be unable to manifest that requirement in code.  Further, if we find that even with good requirements the data doesn’t exist cleanly to produce the metric.  Then we may need to think more deeply about our processes because there is likely a misalignment.  This is great news.  Finding out in December I missed my bonus because of bad or missing data is way less fun than finding this out in January when goal setting is occurring.  Then we can determine if spending the time and energy on a new metric is worth it, because these aren’t free and will likely displace some customer work.

Like we ended section 1, this is not meant to deride data or sound like this is insurmountable.  But wrong data is probably worse than no data and respecting that engineers and managers all have only so much time in a week goes a long way towards building a culture of sustainably delivering high quality business value.

3. Balanced

Balancing metrics happens across 2 dimensions.  The first is attempting to measure countervailing forces to ensure we are not swinging too hard to the side.  For example if features are all being delivered on time, but Customer Satisfaction (CSAT) is down and defects are up relative to a baseline… Then perhaps we are not sufficiently focused on delivering a whole package of value to our customers:  Software that is both regularly updated and high quality.

The second dimension is considering both absolute and relative changes.  Let’s presume for a minute that I look over 2019 and 2020 escaped defect counts and I see that there is a 50% increase in escaped defects.  All things being equal that could be an indication of a problem.  However, if we also consider that the team size grew by 100% over the same time period, this could actually be a great number, as we are seeing new developers entering the system reducing our rate of defects per engineer. 

Balancing metrics is definitely more art than science.  It also has a tendency to push us towards many different metrics.  I would suggest that a general overview can probably be done with a single digit number of metrics, this can reasonably be kept in everyone’s heads and can capture quite a bit of context.

4. Visible to all

Metrics should be placed where those they are measuring can see them. In fact, not only should the dashboards exist in an appropriate location. (Not necessarily a TV hanging over everyone’s heads, but at least on confluence or somewhere clear) But referring back to principle 2, the code should be open to the team as well.  These are highly paid knowledge workers and as you start coding up metrics, you may find, as I have on many occasions that bugs can be introduced.   This openness also aligns team members to the North Star, and does it speaking the programmers’ language: code.

Which leads us to the last principle.

5. Continuously Improved.

Metrics set about to codify the goals of the organization.  Those goals can change, the measures we set out to hold ourselves accountable to may not actually drive the behavior we desire.  They can even be buggy.  We should expect the people to whom these goals are being applied should be able to suggest updates, make those updates, and see how their hard work is in fact ‘moving the numbers up and to the right.’ 

Closing Out

Metrics are an opportunity to drive trust and empowerment within our organizations.  Often times they are used to limit bonuses or shepherd employees onto performance improvement plans.  Just because they are often used poorly does not mean we as leaders cannot turn the ship a bit towards success.  It is very likely the case that your organization is predominantly doing a great job.  Do your numbers expose this?  Do they provide opportunities to give praise to your teams and team members doing the great work that is keeping you employed?  Further, are you able to foster feedback from your team that some metric is leading us away from our North star?  When that happens you will know that you have fostered the trust and collaboration that not only drives grow with in an organization, but serves a key countervailing force: It doesn’t have to suck at work.

Take care.

Twitter Discussion

Homework #1 – Management has Consequences

It’s not a lot to say one is a fan of podcasts, so with the homework series we can travel together though some of my favorite podcasts and perhaps from this we can leave a trail of something novel on the internet.

The first assignment is to look into Reply All’s episode on Bon Appetite.  The podcast is #172 The Test Kitchen.

When I first heard this it hit close to home, here is a set of white guys at the top of an org with little to no appreciation for the turbulence they were causing in their org due to them not appreciating the effects their role power and biases were causing.

Today I am responsible for 40 engineers, and their performance impacts the performance of quite a few additional people and customers.  I have often found myself leading the team astray or causing issues because of mistakes, misunderstandings, and poor assumptions on my part. 

In fact I developed one of my most important principles: Nate doesn’t change the plan in standup, as a remediation for my behavior in the past. We’ll have to revisit this in a later article, but the short version is that the blinking sign above my head as a manager can often accidentally be used to redirect a train that was humming along just fine without me butting in. Back to the podcast.

What’s profound about this podcast, is not only did it try to highlight this issue with management at the magazine, it uncovered this same behavior within the Reply All podcast team itself. So, now we have not only a case study of this magazine, but in real time we can learn from an organization coming to appreciate how hurtful and disruptive this behavior can be in any organization if we are not actively examining our impact.

Your homework, should you choose to accept it, is to listen to this case study (this will take a few podcasts) and see how you as a leader are perpetuating these same issues within your org.  Then take it one step further and see if you can illicit from your team honest feedback about where you are leading the team towards this same cliff. 

Just be aware as we (managers) ask for feedback, take our time and know that if we haven’t laid the groundwork of trust and felt safety, we will get nothing.  If we get nothing, it’s unlikely that is because nothing is wrong.  Further, it is not our team’s responsibility to give us feedback, traditionally feedback rolls downhill, we have to use our relational power to dim the blinking manager sign above our heads.  And ask with a penitent heart for the opportunity to help grow together as a team.

10 2-hour blocks per week

Individual Contributors in an organization need time to get work done.

This feels axiomatic, yet it is often the case that any number of meetings can gather on an individual’s calendar such that there actually isn’t enough time to sit quietly and think the deep thoughts one needs to deliver business value. Today is not the day to wax nostalgic about how terrible meetings are and how we should have no meeting Wednesdays. Meetings are not intrinsically bad and neither is their absence.

However, as a leader it is my job to ensure my team has time to think. To do this, I ask them to look at their schedule at the start of the week and before standup every day and ensure there are at least 2 – 2 hour blocks every day and 10 – 2 hour blocks each week that are unallocated. This can seem like a pretty low bar in an eight hour day, but I would suggest that this is a place to start that is measurable and actionable.

So, on any given day that an IC doesn’t have these two free windows of time. That counts as a blocker in standup. This is really what standups are for; status updates are great but removing blockers is the real money maker. As the manager on the team you just had a person tell you they cannot get their work done. And perhaps even with enough time for you to act

Now it’s time for the manager to step in. Can some meetings be skipped or rescheduled? Or are we writing off this day as one lacking full productivity assigned to the in progress work. It will almost always be a hard decision, but if you don’t know about it. If there is no stake in the ground, you cannot hope to train your team to help them help you identify this concern.

Bonus points: This can be systematically measured. Outlook and Gmail both have APIs that can help you query your team members calendars so that a metric can be built around what percentage of your team has unblocked time. 

Lastly, not all Specialties in an engineering organization get the same amount of free time. As a team lead one might only get 4-6 free blocks a week depending on how much planning is happening. A manager or director could have 2-5. So, as you are digging into your directs and skip level calendars be cognizant of their particular role and how focused on coordination it is. (See High Output Management where Andy Grove talks specifically about how he is in meetings all day, because he is their to coordinate and scaffold decision making)

Take care,

Nathan

See also:

Twitter link for discussion

Empowerment through process change

I spent quite a long time in my career hating processes and actively fighting against it. This is a fairly acceptable approach with a small team and as someone who likes talking and saying the same thing over and over again. This is me. I love to hear my own voice and I used this as an opportunity to engage with my team members. 

This worked great until two things happened: 1. my organization grew to three teams and 2. I was lucky enough to have a boss that said I can only enact processes that I wrote down. I fought this pretty hard until I realized my team wasn’t getting a single message and I was engendering conflict as different people received different messages from me. 

It is extremely frustrating to find out I was the cause of contention in the org I’m trying so hard to build into a caring, constructive, high performing team.

So, here’s my suggestion for building a more sustainable, empowered, and better aligned culture through writing.

Let’s take a detour to empowerment through process. One issue I’ve fought with process changes is the feeling it is handed down from on high with nary a consideration of the lowly individual contributor whose life is actually impacted by this change. This too needs to be addressed.

Step 1: Write down the change you want to make before you take it anywhere near your team. I don’t mean to preclude brainstorming and musing. But keep it away from a wider audience until you can get your thoughts formed enough to write down. Spend the time here to think through the language and grammar of this change. This is a chance to model the behavior we are building to throughout the organization.

Bonus points if your process includes some measurable metric that can tell you whether or not this was actually a good idea. This can be harder than it seems, so don’t make it gating.

Now we test how empowered a culture you are a part of.

Step 2: Put the doc in a place that is visible and editable by your team and prefix the doc and title with ‘Proposal:’ Make sure everybody knows the doc is at the rough draft portion of this endeavor.

Bonus point: Did you detail how you roll out this process change? What day does this start, what docs will be updated? When will employees be expected to conform?

For me that’s putting the doc confluence and dropping it into a chat room #squad-process-changes

Using a wiki or slack is really just tactics. The key is there is one source of truth that can be commented on publicly and even changed by those who are affected. So, give the team a few days, coach them that they should raise any concerns (give a timeline for feedback) and ask for sign off. A manager’s output is only the sum of the team they run so getting buy-in is powerful.

So, in a perfect world, you’ve got a process written down, vetted by your team and it’ll either now be the new way the world works or you will know without a doubt that implementing this is going to be an uphill battle.

All of this is much better than a surprise email your team receives from you about another crazy change they didn’t get a vote on. Oh, and the next time you find your team members arguing about the process they can point at a doc and even change it if it’s due to ambiguity. (But that’s another process change)

Some further reading:

Discussion

A little about me

I am a few things of note for the purposes of this site:

  • A manager of software development teams
  • A person with a passion to care for the people who work for me

Hopefully, you’ll find something here of note that will help you as this has helped me.