DevOps Metrics to Monitor Software Development

Presentation Abstract

DevOps solutions are tying together increasingly complex tools, environments and solutions that can be challenging to manage and monitor. To check on the health of your processes you need to be constantly tuned into your source code, artifact management, continuous integration, delivery and deployment, static code analysis, security analysis, monitoring health, infrastructure, and test automation, just to name a few.

In addition to monitoring the health of your release pipeline, you also need big picture metrics that can measure the value you are delivering to the business and highlight where your current bottlenecks are. If you don’t have your finger on the pulse of your organization, you could already be in trouble.

Two types of metrics dashboards are needed. One to view the real-time health of your delivery and operations pipeline, and another to go beyond measuring deployment lead time and start measuring end-to-end lead time.

Come see how to aggregate your view of the DevOps world in practice.

Top 3 things you will learn on this webinar:
**Which dashboards you need for optimizing CI/CD and value delivery
**Best practices adopted by the Fortune 100 to achieve these metrics (Which metrics does Silicon Valley use to measure success?)
**Successfully combining metrics of varying granularity to provide a full, up-to-date picture of your development pipeline

Presentation Transcript

Myriam – So once again, good morning, good afternoon, good evening to all. We’re really glad to have you all join us today for our Webinar. My name is Myriam Sanie and I’m with the marketing team here at cPrime and we’re very happy to be joining forces with our partner Tasktop to provide you with really an added value presentation. So, joining me today, we have Naomi Lurie, Director of Product at Tasktop. She’s really passionate about making businesses successful through effective customer-centric communication. She has over 15 years of B2B product management and marketing experience. She specializes in large enterprises and their digital transformations. So, thank you so much for joining us today. And also joining me is Brandon Cipes, VP of DevOps at cPrime. Brandon is an innovative and results driven professional with experience, overhauling and refactoring teams, system software and processes to bring the best solutions possible through a variety of technical and business challenges. He has a proven track record of conceiving and delivering large scale projects on time using the latest in cloud technology and Agile and DevOps methodologies. So, Brandon thank you so much for joining us today.

Brandon – Thank you, Myriam. So, today we’re here to really talk about software delivery and DevOps metrics. It’s a hefty subject with a lot of things to cover and consider, but first a little bit of background. cPrime is a company about 12 years old located in San Francisco, that specializes in consulting with companies to help them improve software process in all the areas that encapsulate Agile and DevOps. I think there are three areas that are really central to most organizations. You’ve got a bunch of processes of how the methodologies you pursue, the flows that you worked through and those that really govern how your day to day operations function. But of course, the technology you use really helps guide a lot of what those processes are and the tools that you use, how you bring data together between them. All those other kinds of things are really critical. But, two of the subjects, processes and technology are kind of almost nothing compared to the people because people are everything and so the people help guide the technology and the processes and getting the right people in place. So, the people are critical and what really makes it. So that’s a little bit about cPrime, but I think we’re ready to kick off real fast

So the big thing we talked about is people and how we’re trying to figure out how to make software delivery better, faster and stronger. And I’d like to take a step back and first say, let’s just talk about the most basic elements of what software delivery requires. And when I think about it in that respect, I sort of see it as this cycle. You don’t build software and then you’re done. You iterate it, you’ve changed, it evolves. And so, what I think of the most basic element of that cycle, you got to figure out the definition of what it is you want to build. Then you start planning and figure out who’s going to do what and when and you actually get into the nitty gritty of building it, the development, and then there’s this whole operational side which is, you know, it’s up and running.

Is it working the way you expect? But as I said, it doesn’t stop there. It keeps going because you have bugs and feature enhancements and customer requests and all these things. And you know, irony of ironies, the better your team does and being able to do those things and meet those needs, the more work that probably piles up on your group, the more things you have to do, and I’d probably venture to guess that pretty much everybody on this call has more things on their plate that they are responsible for now than five years ago just because there’s more stuff out there. And so, this cycle is a catch 22. It’s a great thing when it’s running smoothly, but it’s also, it can be very complex, very fast and if it breaks, I can throw a lot of things out of whack, but you know, when you take this very basic interpretation of it and people started to think, okay, well, fine, we’re talking about the simple elements, but I want to get into the methodologies and how you apply it.

Really the nice thing is the first part to find a plan that’s agile, that’s what we’ve actually been trying to do for 15 years and has a lot of good ways to start and attack this problem. And as you can kind of suspect, develop and operate, that’s your DevOps piece. And so, these two things really together are what can help deliver software development and software development practices that we see to be healthy, sustainable and accelerated. And it’s interesting as well because when you go, and you break up agile and DevOps and you look at their original foundational documents, like the Agile manifesto or for anybody who’s read the Phoenix Project, I’m the DevOps side. Their core drivers are actually the same even though they’re coming from different ends of the aisle. They both want to get into it.

If they want to go quick, fast and nimble, all those kinds of elements and then they wrap it in, hey, let’s be contextually aware. Let’s make sure we’re communicating openly and so all technical elements aside; the methodologies are actually very aligned and that’s why we always bring them together and I think it’s an important thing to keep in mind for anybody who’s going down the software development and management path. That one without the other is going to be a little limited. You really need to think of software delivery as a holistic project management and technical endeavor that the combination of groups need to be lock step to be most successful, but since we’re talking about more of the DevOps stuff, let’s unpack this a little bit and so we’ll. We’ll break up our little circle and we’ll kind of skip the define and plan stuff.

So, Agile has been around for a while but DevOps is the newer piece out there that a lot of people are trying to learn more about and when we think of DevOps, we break it into two halves. The first is sort of that methodological cultural thing, right? You’ve got to be more collaborative. Smaller groups, cross functional teams, but when you think about just the technology, there’s six pillars that we really see driving us. The first one being continuous integration, pretty straightforward. This is really concept of, Hey, don’t write a couple hundred lines of code with 10 different changes and then commit it all at once. Commit each little granular elements. Get that all in there as a small chunk as you can because it’s going to make it a lot easier downstream to under understand the effects of every change. And then it’s really continuous deployment, which is the idea that every time you get one of those changes, put it somewhere, so you can actually see what it does and understand the effect that it has. If you do a release and you released 10 different changes and something breaks, you don’t necessarily know which of those 10 or what combination of the 10 led to your problem. Whereas if you do 10 iterative changes, it’s much easier to isolate what’s going on and also as a point of calling out because we get this question a lot, people say, well, is it a continuous delivery or continuous deployment? And there’s a subtle difference, but it’s a very important one, which is just to say that when you’re delivering something, you’re actually delivering it to a client, it’s going to an end state. It’s hitting production, but deployment, it can be deployment, anything, right? It could be deployment to a test server to a staging server. And so, a lot of companies will have very, very high rates of deployment. You go look into Netflix and they will have continuous integration supporting hundreds and even thousands of changes and I think given day, and they might be deploying those changes to all myriad of environments, but they’re actually not delivering that, so the customer, except for a few times a day, certain windows maybe if they’re hot fixes, things like that, but so it’s just a distinction to keep in mind. Either way. The marriage of CI/CD is all about get small, get quick. It makes it a lot easier to understand what’s going on. The next step that comes in is, and it has a lot of different things. It’s infrastructure as a service. It’s the cloud. It’s virtualization, but the idea is when you’re releasing your updates much more frequently per what CI/CD enables, it’s not going to work for you. If you’ve got a six week procurement cycle to get a new Dell or HP server in your office, even if it’s on your own premise and you have a large stack or a set of blades that are virtualized so that you can spin environments up in that quickly or you’re sitting on AWS or Azure, you have to be able to spin these environments up and down to meet the time sensitive needs of passing all the software, much less the cost efficiencies of, Hey, when I’m done passing it, I don’t need it anymore, so let me turn it off and save that money. So that’s a really, really critical element that we see within DevOps. It’s just being able to enable the speed. Now the next two are also very similarly attached at the hip test. Automation is exactly what you think it is. Everybody’s doing testing. If we’re doing that testing manually, you’re never going to keep up. It just an impossibility. We have clients that run hundreds of thousands of test cases with their releases and it all has to be automated through your functional, your unit, your integration of all the different kinds of testing elements that you’ll have and you really need to be able to do it very quickly or development operations to keep pace. And the other side of testing is security testing or DevSecOps, which is to say don’t just pass functionality. You don’t just need to know that if I click on the button to turn from blue to green, but can any users but the authorized ones haven’t trait that system or my ports all locked down, if I’m using public libraries, have I checked them for vulnerabilities to make sure that I don’t have some liability because we’re not on a patch version or these known problems that exist that are just very important thing to stay in front of it. So, security is definitely something that you need to be more proactive than reactive about, but less than at least is the monitoring. And as you can imagine, if you’re doing all of these things and you had such an accelerated pace, not just of how often within one product you’re doing all this work, but the fact that people are now supporting many more software lines than they did five years ago, you have to have automated systems that are capturing the data for all these different elements and saying like, Hey, is this working the way we’d expect? So, as we start to jump into the details and really the metrics behind these things, if you can’t measure how these are performing, you’re going to have a very hard time managing them, and the thing we always say is you’ve got to measure the metrics that matter. So, continuous integration, how often are you committing? These are within these various repositories and projects. Are there individuals that are coming more often than not? When you start to think about deployment, are those builds working or are they failing? How often are they able to go out? You know, are you still in the old waterfall days to release every three months or are you releasing three times a day? As you start to think about infrastructure that now is this living, breathing thing that expands and shrinks you have your basic APM stats, right? Resource utilization, free memory, disk IO, network latency. But you also want to know how big and small the infrastructure is, how many servers are running at a given point. So, these things start to become really critical, especially from a cost perspective. Then you get into test automation. Coverage is huge. You want to make sure that you have test cases for a large enough portion of what you’re doing and just simply how many problems are being discovered. Then we get into that security bit again, which is really about are you identifying how many followers the vulnerabilities are out there? Are you making sure that you’re covering all the different assets that are in play, but all these different data points. And this is really just a small snippet of the hundreds that you can imagine exists. Monitoring is all about collecting. You have to be able to bring this data together to understand how things are performing because the end game is you want to take this data and make decisions. You want to see, not necessarily dashboards for the sake of dashboards, a collection of information. It gives you some visual understanding of trends that also calls out the anomalies, the problems, and so you want to be able to understand how many times are things failing, how quickly is that working, how efficient is my testing regimen, what is my infrastructure look like over time, all of these things are really, really key to being able to then say, how am I going to get better?

There’s a collection of KPIs that one might consider most pressing or important, some basic things like how often are you committing your code, how often are you releasing it, how many bugs are you finding and how stable it is. So, when you talk about one to three metrics, of course it depends on what those one to three metrics are that you have, but those are the most basic elements that you want to start with. Those couple areas I outlined. And from there I think you want to progress into that, you know, maybe two tiers of metrics. I think there’s probably five to 10 that really are your leading indicators of health. But then to be able to iterate and improve, a lot of times you need to go to that second layer of detail where you might have 20 or 30 metrics that you go in when you need to do deep dives to understand how am I going to improve this particular area of my business? What detail can I sort of a ferret out that will help me understand where I can make a change? That’s going to make difference.

And here is where we jump to the other side of this.

Naomi – Thanks Brandon. So, hey everyone, this is Naomi and I’m from Tasktop and I’m going to be picking up right where Brandon left off. But first let me briefly introduce Tasktop to you. So Tasktop was founded in Vancouver in 2007 and we serve the needs of large enterprises for software delivery, value stream management, which I’ll talk to you about today, including 43 of the fortune 111 of the world’s top banks and the US’ largest insurance providers. So basically, we know the challenges that face companies, agencies large and small, and we have four product lines that integrate with 58 software delivery tools and span over 300 versions. So, Brandon gave us a great introduction to the technical metrics that organizations collect from their release pipeline or from their CI/CD process. And so many organizations have embarked on their DevOps journey with the understanding that deployment is their bottleneck.

So now I want to ask you, what if deployment is no longer your bottleneck? Let’s say you’ve done a really great job, you know, rolling out your DevOps and some, have you ever tested that? You’re just starting on that journey and some of you have started and are facing some challenges so you’re working on it. And in the meantime, while you’re automating and streamlining your release pipeline, your customers are coming to you and whether they’re internal or external and they’re still complaining that it is too slow, right? I’m sure you’ve heard that. So, what are you doing to optimize end to end, to shorten lead times from the initial customer request till it’s delivered and not just from code commit to deploy? If you had to, could you find your current bottleneck in our Agile and DevOps enough to allow you to do that?

Because it’s a known fact that if you optimize the process, that’s not your bottleneck, you are not going to improve. Software delivery has become a growth engine for so many organizations, right, for every organization on the planet, it is no longer like this necessary evil, but it’s an absolute must for revenue generation, so software delivery exists to create value for customers and I want to talk to you about the business metrics that software delivery organizations need to measure in order to be able to generate more and more value for the customers and basically to also justify all the investments that we’re putting in to automate more and more of the software delivery process like we are with CI/CD. So, software delivery is a value stream. The value stream is the sequence of the activities that an organization undertakes to deliver on a customer need.

So, every software delivery organization has at least one value stream. You have a value stream for product or application or service and what you’re seeking to optimize is the system, meaning the total time it takes to deliver value to the customer, which is the beginning and the end of everything. So similar to Brandon’s diagram as you can see here. I do have an illustration that’s broken up into four main parts and it’s a little overlapping but a little different. So, when I look at the value stream, identify four major segments, we have Idea, where you imagine the feature, you plan it, you design it, create is where you implement it, release is where you make it available to your customers and operate is where you ensure that it’s running properly and support customers that are using it. So, each of those segments is made up of these core activities that are performed by some very specialized roles using tools that are very specific to the type of work that they do.

And if you want to optimize the system, you need to have the ability to measure the value creation process through this entire process. And you have to learn how to manipulate it and manage it to get the kind of business outcomes you desire. And that means being able to monitor and measure how work progresses through the value stream. So, let’s take an example. You know, the reality is that organizations, have lots of tools that support the value stream. In this example, a customer’s requests are logged in Salesforce. Then they’re transferred over to the requirements management tool or, whatever planning tool that’s used by the product owners, product managers, business analysts and there they gather and articulate the requirements and then they pass those on to the developers, where they’re created as epics and stories and the developers’ agile planning tools. And then of course you have your test to teams that need to have early access to those stories in order to design a proper test coverage for anything new that’s coming and then the actual development kicks off and that involves SEM or you have your builds kicking up in Jenkins and your defects are being blogged in test management and you’ve got security scans that are identifying vulnerabilities and everything is feeding back defects to the developers to fix what’s broken before the release.

And then finally comes the day or the moment when you have a good build which you want to release into production. And then once it’s there, it continues to be managed through the ITSM tool. Customers start using it, they’re opening tickets, finding defects, or they’re asking for new features and the whole process would kick off again and again over and over, so what you end up having is this true network of activities and artifacts and tools and network that you want to have complete visibility into. You want to know how the work is flowing between all the people and their activities and that’s the only way you can identify bottlenecks and accelerate the time to value and based on all your investments in agile and DevOps. So now let’s see what kind of business metrics you want to monitor and measure. Brandon spoke a lot about the interesting DevOps metrics that are collected from code commit to code deploy.

So, I’m going to talk to you about two more metrics at this point. The first one is flow time. That’s a metric that measures the speed of delivery from the moment the work was accepted way upstream and the second one is lead time. That’s a different metric that measures the speed of delivery from the moment the customer request was initially raised. So those are the metrics that the business cares about because they are directly tied to pass fast. They can introduce value to their customers, value that the customer is pull from the system.

So, to find your bottlenecks, you really need first of all to have a top down view and then drill down capabilities into your data. So, let’s first talk about what kind of value to customers pull from a software delivery value stream. They basically pull for things they call features, New functionality that will delight them, resolve defects that improve the quality of their experience. And unbeknownst to them they are also pulling a paid down technical debt and reduced risk. Both are extremely important and sometimes transparent, usually transparent, to the customer, but very significant for their outcomes. So, what you need to do is have these ends to end metrics that can correlate performance of the value stream as it pertains to these four types of value against business outcomes.

Business outcomes could be revenue generation, number of subscriptions, usage of a product, reduction in cost, improvement in quality so you have certain business outcomes. So, what you’re looking is to see how work is flowing through your value stream and how is the performance of the value stream contributing or correlating with business results so that you can manipulate it and manage it and change things to get more of the desired outcome that you’re looking for. And this comes to like when you’re having those conversations with your business stakeholder, the technical metrics don’t mean as much to them, right? So, we kind of have to abstract and level up the conversation when we’re talking to our business stakeholders to things that are, are a little bit broader, right? And that’s why we talk about flow, of value, and how things are flowing through the value stream.

So, this is an example of an executive level dashboard where you can look at the flow of value through your value stream and evaluate certain things flow velocity, so you’d want to be looking at how many features are customers getting per quarter or per month, whatever your period is, to see are we getting a lot of features to our customers? The one that you can see, which has like kind of the bar charts as they’re distributed, that one is a distribution that one allows you to see how are you allocating your resources, are you putting the right capacity on features versus defects versus tech debt versus risks? And that really depends on what you’re optimizing for and where you are in the product lifecycle. If you look at the one that’s called slow time, that’s basically how fast, how long does it take from request to release, um, for you all to release a feature or any of these other flow items. Then there’s efficiency, meaning how much wait time do you have inside the process? That’s slowing you down if work gets worked on for two hours, but then waits for 30 days for the next person to pick it up, your flow efficiency is very low and then you have a flow load which is about how much load the teams, because teams can only handle a certain amount of work in progress and if you overload them then their delivery goes down. They will deliver less things and it will take them longer. So, when you look at these metrics degrading over time, you can drill down deeper to find your bottleneck and you can use this to prove that you have a bottleneck. It for those of use you who said that you have an idea and what this does is that prevents you from fixing the wrong problem, like adding developers when what you really need is a having more UX designers on your team for example.

So then comes the question, how would you go about collecting this type of end to end data? Because it lives in so many tools. It’s not really elementary how to collect it. So, the first thing you have to do, you need there to be a seamless flow between the tools, meaning all this automation that you’re putting in place from code commit to code deploy. You want to bring that upstream. You want to put it in place so that all the activities and the handovers that exists from the moment the request is logged or the work is accepted, you can trace that as it makes its progress through the whole value stream. So if you think about traffic optimization as an analogy, you need to have the roads and the bridges between the tools to make the value flow faster. And then the work that’s taking place in each of the tools is basically your digital footprint of value creation and they can sync signal their location and their status in real time.

Just like mobile phones do. When you’re using a traffic navigation app like Waze or Google maps and then you need something in place to collect the data and extract it up to those meaningful metrics that can help you find your roadblocks and bottlenecks and work around them. But together with your business stakeholders, because if you remember what I said at the beginning, you might be optimizing in one place, but your customers are coming back and saying, well, it is too slow. I cannot compete with these nimble startups that are nipping at our heels or these digital giants that are trying to move into our space. So, you really need a very rapid value creation process and you know, you have to take in the end to end flow if you’re going to achieve that.

So this is what that might look like. You connect your tools and you flow artifacts between them, but you normalize the data as it flows through some kind of abstraction. And then you have all that funneling into a reporting database where you can generate those types of metrics and dashboards and insights and use that to kind of create a joint visibility is shared visibility to see the health of your software delivery processes and how making changes correlates with achieving the business outcomes that you want. So, before we move onto your questions, what we wanted to do was to just leave you with some practical advice of like next steps that you guys could take in your journey to monitor the health of your DevOps implementation and your software delivery value streams. So, Brandon, do you want to kick us off with the first one?

So, we’ve talked about those six different areas and check them out, see and make sure that you’ve got some kind of operation and check happening within each of those six, and you’re starting to collect some kind of data. The second would be just implementing solutions right now, are you starting to gather that information to figure out what you’re going to do with what I’m going to tell you?

I’m going to add that you should lay down the infrastructure to measure end to end metrics and value creation. So, don’t stop when you fixed CI/CD because your bottlenecks are going to move somewhere else. So, in the meantime, lay down the infrastructure so that you can optimize the entire system, monitor it, measure it, and optimize it, and then my other one would be a recommendation that to try to create metrics that speak the language of the business, speak the language of your CIO and the conversation that the CIO is having with his or her business counterparts and that is all about value and business outcomes.

Ok, thank you Brandon and Naomi. So, our first question will actually be a comment. Peter says if you have no questions about your metrics, you can be confident and that is why they collect as much as possible. Which kind of leads into a question that Amir from Chicago poses is what metrics should I really start with?

And you know, I would also again preface and I sort of alluded to it earlier, but I think there’s two tiers of metrics. So, just to get to that point, the comments initially that, you know, hey, if you know the metrics, you aren’t sort of flying blind and it’s a very, very validating critical point, especially for whoever your boss is. Because when you can point to these things, I think it makes it very, very defensible. And actually, in a proactive sense, to be able to say like, look, we’re doing this, look how the metrics have improved over time. A metric inn and of itself actually means very little without the trend associated with it and the history associated with it to try to get some directional understanding of it. So, you know, so what those high-level metrics should be in that first phase of it? Again, I think it’s basic elements within those six areas and it being the idea of how often are you committing code, how often are you deploying it and what kind of quality these are coming through, right? So, number of commits, number of builds or successful build I should say, and number of bugs identified. Those are your most foundational pieces to focus on. Now there’s, there’s a myriad of other ones is going to come up within each of those six pillars. I mean, you, you will find a ton of information, but there is a little bit of a, a, a red flag or a warning, but all throughout which is just say there can be too much data and, and, and many of you probably heard about this, you know, in the, in the age of big data, it’s get everything figured out what’s going on. Oh my God, you have all the data, petabytes of stuff going on, but what happens unfortunately more often than not as a lot of that data isn’t necessarily meaningful or certainly not meaningful in isolation.

And so, it’s really critical to, yes, it’s always good to grab as much data as you want, but as far as what you manage to and how you measure health and success and things like that is pretty important that you have a little bit of a curated selection of what you regularly are going to. Because data in isolation can, can eventually get to being a little misleading. It’s understanding how it relates to other things. It becomes most important. So, I’d say it’s better to have a handful of metrics, but you really understand then just a whole dumping ground of hundreds and thousands of things. If you’ve got fiveish top tier metrics and twentyish kind of middle level, that’s a really, really great place to be. And then you evolve and iterate against broader sense.

I’m going to add to that if you had to choose like a metric, let’s say a business metric to also start with, I would recommend that you try to measure flow time and see are your technical metric improvements moving the needle on value delivery. So, if you’re improving in specific things in those six categories that Brandon identified, are you also, are they correlated in some way to something that’s bringing more value to the business? So, is that increasing? Is that I’m shortening your time to value, for example. I would say don’t only measure the process. Like, you know, when organizations start rolling out agile, so they would have measurements like how many people have been trained on agile? Well, that’s measuring the process, the process of retraining and implementing, but it’s not measuring an outcome. The outcome is are we delivering features faster? So, I would always advise to try to get, gather at least one to start with at least one of those kind of big picture flow metrics and see if there’s a correlation between that and your technical metrics and also between that and achieving the desired business outcomes for your business stakeholders.

Thanks Naomi. We have another question. Some people in my organization don’t seem to really understand the value of those metrics. So, they’re asking how can they get buy in to actually obtain these metrics?

I think the way to get buy in from managers, you know, even middle management, moving up to directors and VPs and they are very, very much measured at the end of the day on financials. So, I think if you can show them that these metrics could result in improved financials, that that resonates with them and they are also being called now to a higher duty, which is really helping the CIO through this digital transformation and helping the organization transition it from a cost center to a profit center. So, I think, you know, even though a lot of us are technical people and our roles are technical, anyone who has a kind of a more business oriented approach usually succeeds to relate to your managers, right? And your director is because all these people are, you know, often they have MBAs and you know,, they’re, you board meeting and everything is always around financials any company that you’re working at. So, if you can start relaying things in terms of business value, and you know as perceived by customers, I think you will achieve getting more buy in.

I couldn’t agree more as much as these metrics are important. The truth of it is it’s always coming back to you’re not doing DevOps because it’s fun. You’re doing DevOps to try to find some efficiency and improve what you do and that ultimately is did the same number of people produce more results, more features, or generate less bugs that require, it’s where you saved they’re dot faster. That’s really critical.

Myriam – Okay. Thank you so much for that answer. And then we have people kind of asking the same, a similar question. David and Mandy are kind of talking about the same thing of fluid flow. Efficiency is notoriously difficult to calculate. So, they’re kind of asking what approaches or how would you measure flow time? And Mandy also asked, in addition, how would you measure lead time?

So, I mean, I can also comment on flow efficiency, right? One thing is clear, there’s no magic, right? Our data lives in so many tools, right? And each tool has its own kind of schema and its own universe of the way data is organized. So, what’s really necessary together, any of those end to end metrics if you are working with you know, a best of breed DevOps tool chain is to be able to trace work as it moves from tool to tool, right? And to be able to map activities that take place in all these different tools into kind of like an abstraction and what of those activities is a wait time on what of those activities is an active time and then be able to calculate those numbers and it is difficult, right?

But it can be achieved if you create this kind of end to end integration between your tools and couple that with modeling and abstraction a which then gives you, you those results. So, but let’s put the question of flow efficiency aside for a second. And they were asking about slow time and lead time. So, if you want to do time flow time, you could start measuring from the moment your product, a manager, let’s say, or the product owner accepts a request as something we are going to work on. So, it’s kind of that and this, every organization kind of has to identify for themselves what exactly is the specific, you know, status or triggering event that, that starts the clock, but it’s basically, you know, you might receive a thousand requests from customers and you know, you’re only going to do 100 of them. Okay, so don’t start the clock and I’ll be a thousand, but the one, the 100 that you’ve decided you’re going to do start the clock on that and then you need to really find to be able to almost have this thread of traceability from the original requirement or the feature as it was divided, does that design by the product owner and two when it turns into an epic and the epic to the story and from the story, you know, to the bill until it’s running in production.

So that’s possible. I mean, that’s something that can be done as you synchronize artifacts from tool to tool lead time would require going one step upstream. Right? And having a way to log the receipt of the customer requests and, and pull the string over there from the, from that system into the, the tool that the product owners are using. So, then you really have the ability to tie from the original sales, let’s say the original request all the way to the end. I can tell you that it tested. We do that. So, like the example that I showed you all was kind of loosely based on what we do. We don’t necessarily use that exact tool chain, but we do have customer requests that are coming in from a, let’s say, captured in Salesforce. We flow them automatically into target process, which is the tool that’s used by our product owners to triage and prioritize and then develop, you know, their, their side of the requirements and the features. And then we flow those into Jira where our developers work. And then we, you know, we push those out into our releases and then into our ITSM. So, we have actually connected the end to end flow. So, we do have all of those metrics possible, uh, today, uh, flow time, lead time, like basically the dashboard that I was showing you was, was based on a dashboard that we have for ourselves.

Myriam – Thank you so much. Naomi. The next question from Adam and be involved in selecting much metrics and then also who else can benefit from transparency with regard to these metrics at the enterprise level?

Brandon – We always actually get very disciplined around who collects metrics because what you want to do is you want to escape the problem where for different people, like four different metrics all at the same thing. They all call it releases, but they actually have subtly different definitions for the releases and that, that can be a host of problems. So definitely try to organize and make sure that not necessarily distinct team, but one group that’s in charge of one metric has a definition for that metric and it has some automated function where they’re, they’re grabbing the logs or whatever else so they can bring that day to day and consistently and maintaining a consistent definition of what that data represents.

Myriam -Okay, thank you. Carmen brings up an interesting point if you know how confident that once you address the current bottleneck, be able to identify the next one and how do you make sure that you have a system in place to do that?

I mean the truth is, is that you solve one bottleneck, right? You get another one, right? Because that’s the nature of the beast and value stream management is not a process that has a beginning and an end. It’s something that you have to do all the time to continually optimize and you know, there’ll be new methodologies and new tools coming into the market all the time to help you get better and better. Um, so basically once you’re measuring a and you’re capable of tracing all your tools and all your, all the tools, all the artifacts in them, all the relationships between them and all your products, you know, uh, you will be able to always find the next bottleneck, right, because you’ll be able to see where your performance is degrading and drill down to identify the root cause. Um, but indeed to do that, I mean, it’s like an ongoing practice so you don’t solve the bottleneck can go away. You, you solve one and you watch what happens to the rest of the system. Okay? And you watch where now your bottleneck has, has migrated to and then you, you tackled that one. But every time you’re going to address a bottleneck, you’re going to do it with the intention of improving one of your, of improving a specific, metric that is correlated to the business outcome that you want.

Myriam – Okay, great. Thanks so much, Naomi and Brandon for a really great presentation today demonstrating the importance of metrics in DevOps. Thanks to all of you for joining us today. We hope to see you on one of our other webinars. Have a great day, everyone!