Be Careful with What You Measure
Following the famous saying of “you can’t manage what you can’t measure,” we, Project Managers, set metrics to help monitor and improve productivity across the stages of the software development life cycle (SDLC). Usually, when you want to measure something, people tend to pay attention to it. So if you want your team to pay attention to the cycle time of the tickets they deliver, start measuring it.
Many team leads or managers are lost in the process of measurements and lose sight of the end goal. Thus it is of utmost importance to set goals: e.g. your cycle time is 20 days with 50% probability and 29 days with 75% probability and you set the goal to achieve 15 days with 50% probability and 22 days with 75% probability. This is a kind of target the team and Project Manager can set depending on the business need and team’s capabilities.
Every time, I moved to a new company I was asked to go deeper into the team’s effectiveness metrics, understand where they can improve and how we can help teams be more efficient. While, previously, I thought that having a “high velocity” and increasing it over time is a good indicator of a healthy team, now I know it’s NOT, as NO metric is perfect if it’s used solely. Aiming at understanding the team’s velocity and taking it as the main measurement, can lead to velocity growth, but at the cost of team’s burnout and employee high turnover rate. That is why setting correct metrics is extremely important for us, otherwise, the wrong ones can easily lead us off target.
So, we should start asking ourselves questions “WHY” and “HOW?”
Why do we need to measure something? How are we going to do that? What metrics should be used?
The data itself should not be the ultimate goal, but, instead, a tool to track our team’s journey, understand where we need to invest and develop, provide feedback, add resources, etc.
Another thing I noticed during my expertise is that many Project Managers or Scrum Masters are more prone to using Scrum metrics since those are easier to obtain and interpret (via Jira tools) as opposed to Kanban metrics which require more time and analysis.
Traditional Agile metrics and analytic tools give little to no idea of what to do when things go wrong. “Estimate Better,” “Plan Better,” “Hope,” “Work Harder,” are not feasible or sustainable strategies, while Kanban metrics suggest specific interventions that one can make to get back on track.
I have picked 2 useful metrics that can help you learn more about your team and give hints on where you can improve or at least get a better understanding of your current workflows and bottlenecks.
- Cycle time
- Cumulative flow diagram
Ready? Let’s dive deeper!
Cycle time
Cycle time is one of the best metrics that provides us with enormous information. But more importantly, it gives the following benefits: a good predictor of cost (the longer the cycle, the more money is spent on feature development), the amount of time it takes to get customer feedback (no one wants to work on a product that is being built forever) and the overall picture of your process health.
Cycle time is defined as the amount of elapsed time that an item spends as Work In Progress (WIP). To calculate your WIP limit, refer to this article that explains the calculation method very thoroughly.
So you start with picking 2 data points: Start and End points. This way you can measure how much it takes the ticket to go from the left side to the right (every project or company defines its data points). More precisely, how long it takes to provide the work item to the customer, or Quality assurance team depending on your chosen endpoints that you want to measure. If you want to understand where your team’s bottleneck is, measure the time the item passes through the workflows (E.g. how long it takes to provide the item to the QA team, etc). Low-cycle time means your team is working efficiently, high-cycle time shows that something is stalling your process. Keeping your cycle times low keeps your lead times down, and so leads to faster time to market.
Usually, cycle time is visualized either via scatterplot (Control charts in Jira) or histograms (in this article I will refer only to scatterplot).
Even though the chart looks confusing at first glance, it can give you probabilistic forecasts about future performance. How? The dots represent the task/story that is completed in a given time with a given cycle time. When you get the dots for a defined timeframe, all you have to do is draw a horizontal line that will separate 50% of your dots above and 50% below the line. This will mean that there is a 50% chance to deliver a work item in N days. If you want to get a more precise estimate in terms of percentage, draw another line of 85% or 95%. Note: don’t worry you don’t have to draw the lines by yourself: there are Jira plugins that will do that for you.
From this particular scatterplot, you can see that when a work item enters our process, it has a 50% chance of finishing in 5 days or a 95% chance of finishing in 27 days. You can go deeper to filter and get more specific data (e.g what is the cycle of time of bugs, see per person cycle time, in which status the task gets stuck, and analyze the reasons). Depending on your business or customer requirements you can decide that 27 days of delivery with a 95% chance is a long time to market and analyze what can be done to reduce it to 22 days.
So this chart can be an amazing addition to your metrics list, allowing you to get a transparent picture of the cycle time of tickets that pass through your team’s Kanban board. You’ll also be able to use this tool as an additional forecasting tool on how long it would take for future assignments to be finished.
Cumulative Flow Diagram (CFD)
Our daily job usually starts with understanding the bottlenecks the team is facing, be it a daily blocker or flow inefficiency. CFD provides clues on what must be looked at. Through the chart, it’s possible to get information about 3 important metrics of your flow: Cycle time, WIP, Throughput.
Each colored band shows how many tasks are present in each state of your process (in all your “In Progress” sections) at a given time. The Cumulative Flow Diagram displays the distribution of each task in each process state.
Let’s take an example and analyze it.
The mechanism is very simple: on the vertical axis, we see the number of tasks (by tasks we mean every kind of issue-type: story, task, bug, etc.) in the workflow during different times. The horizontal axis shows the time frame for which the chart is visualizing the data. The colored bands always go up or sideways in accordance with the number of tasks that go through your process.
The top line of each color band on the CFD shows the entry point of tasks in the particular stage of your Kanban board, while the bottom one represents when it leaves the stage. For example, a rise in the color for the “Done” band means your team is delivering work faster. On the other hand, a sharp increase of the “In Progress” band shows that a bottleneck is beginning to develop. So when you see signs of bottlenecks, then it’s time to have a closer look at the problems and understand why it is occurring; is it an external blocker that makes your work slow down at some point? Has your WIP crossed the limit and you missed it? Do you have a need for resources to unblock all those tickets that are stuck under a specific stage? So by having a more thorough look at blocker issues, you can eliminate them before they become real problems.
On the chart presented above, we have 75 tickets in progress with 7 days of cycle time.
The first thing you notice is that the team might have a bottleneck related to the Testing phase on the mentioned date (February 16th). The red-colored band (“In testing”) narrows down, seems like it’s disappearing having only 1 ticket “In testing,” while 13 tickets are already waiting for the Quality Assurance team to pick them (Ready for QA- 13 tickets).
Is it because we need more resources or there is another reason?
This should be taken in more detail to understand the case better, because it depends on the number of variables, such as the number of QA engineers in the team, reasons for not picking QA-ready tickets, etc.
Cumulative Flow Diagrams are more interesting in terms of learning if the team has troubles and then analyzing to identify real blockers behind the problem.
I have shared an example of one of the teams I know. Remember that having a nice CFD is a rare case, in most cases, the color bands are overlapping with at least a few colors. But, don’t fear! Once you understand how to read the chart you will come up with good ideas to start your investigation and give proper solutions that your team will benefit from.
These were the two metrics I’ve decided to elaborate on. There are others that will also guide you to analyze and furthermore increase team(s) delivery rate. I might write about them in my future articles or if you want to ask me questions, feel free to DM me on LinkedIn.
