How to Use the Assessment of DevOps Capabilities (ADOC)

The Assessment of DevOps Capabilities (ADOC) is crowdsourced, vendor-neutral and designed for individuals, teams and organizations who want to baseline their current DevOps state, identify the next target state, gain insights into how to improve their organization and team capabilities and measure and accelerate continuous improvement during their DevOps journey.

The assessment models five DevOps dimensions – the human aspects, process and frameworks, functional composition, intelligent automation and technology ecosystems – to help you gauge your DevOps maturity and identify areas for improvement. In this TechStrongTV interview, Accelerated Strategies Group CEO Mitch Ashley talks with Ulises Gonzalez and Leonardo Murillo, two DevOps Institute Ambassadors, about how to apply the ADOC to get the most out of any DevOps journey. The video is below, followed by a complete transcript.

Announcer: This is Digital Anarchist.

Mitch Ashley: Well, I have the pleasure today of being joined by two very interesting gentlemen, we have a great topic to talk about. The first gentleman is Ulises Gonzalez, and also Leonardo Murillo. Very nice to meet you both. Congrats—good to be getting together with you. Congratulations on getting together, [Laughter] to have a TechStrong TV talk.

Leonardo Murillo: [Laughter] Thank you. Thank you, Mitch.

Ashley: Ulises, would you introduce yourself? Tell us a little about you and who you represent.

Ulises Gonzalez: Oh, absolutely. I’m Ulises Gonzalez, I’m a DevOps Ambassador for Panama. I’m a Product Owner on Agile611. We are a company of way of working located in Spain and Panama. I’m glad to be here.

Ashley: Great to have you. And Leonardo?

Murillo: Alright, thanks for having us here, Mitch. So, my name is Leonardo Murillo, and I’m CTO at Qwinix, we’re a product engineering company in Denver, Colorado, and I’m also Founder of Cloud Native Architects, which is a consulting firm specialized in continuous delivery, security, and reliability, doing consulting for enterprises, and I’m based out of Costa Rica.

Ashley: Excellent, excellent. Well, good to be with you both, we have a fun topic. I know you both are DevOps Institute Ambassadors. For people that don’t know what DevOps Institute is or maybe what the Ambassador program is—does one of you wanna take a shot at kinda giving a quick overview?

Murillo: Absolutely, I can give it a shot. So, there’s a huge component to DevOps that is human centered, right? It’s about culture, it’s about practices and kinda mindset. So, the DevOps Institute is an organization that looks to provide a means for DevOps practitioners to skill up, to gain knowledge, to network. It’s at DevOpsInstitute.com, so I really encourage everybody to go and find out some more. It’s a great, great organization.

Ashley: A lot of sharing, a lot of collaboration. A lot of folks who are going through the same journey, maybe at different speeds or different places with their career or with their organizations, so what better place to work with each other, learn from each other, and help each other? So, that’s a great thing. Very cool organization, the DevOps Institute.

Well, we’re here to talk about this new assessment program, the short name of it is ADOC, but it’s the DevOps Institute Assessment of DevOps Capabilities—thus, ADOC.

So, does one of you wanna tell us a little bit about what the assessment is for? Why would someone want to use this as an assessment tool?

Murillo: Why don’t you go first, Ulises? [Cross talk]

Gonzalez: We need to remember that adopting DevOps is not a one-day process. Adopting DevOps requires a lot of effort, a lot of co-working, a lot of understanding on where we are. So, understanding your DevOps capabilities is vital to assess your current practices. And then, it helps to know how to devise or develop an improvement plan, it’s very important.

In my experience, when we don’t know how to do DevOps implementation in a proper way, we can do more damage, do more harm that receive the benefits. That’s why DevOps capability assessment is a very wise decision to make to start or to understand where we are. Where we are to make good decisions, to get that deep understanding about our practices, and of course, to make a plan, to make a—how to define the next steps to be close to our goal or our vision to accomplish. That’s why DevOps Capability Assessment is a more than a tool, it’s a very wise way of starting of gaining understanding on what we are doing in our company.

Ashley: Mm-hmm. Good overview. I know you both are with partner organizations that would actually help people go through this assessment, correct?

Murillo: Correct.

Gonzalez: Yeah.

Ashley: Great. Leonardo, why don’t you say a little bit about what you think that process would look like? And by the way, it’s not an unusual thing to hire someone to come in and do an assessment and get an external perspective, and then you kind of evaluate where you are and decide where you wanna go or what things would help you get to the next place.

This really gives you a standardized way, something that you’re at least kind of—not quite a benchmark, but something kinda like that so that you have a consistent assessment of how you’re looking at organizations so you can benefit from what others have learned about going from where you are to whatever the next step is. Does that make sense?

Murillo: That makes sense, and I think you basically are talking about some really relevant points in terms of kinda like the standardization of the tool to assess, right? Because as Ulises pointed out, it is a long time effort. It doesn’t happen overnight. And it requires the involvement and ownership of many people across the organization. It’s not just technology alone, right, that will enable this transformation. And I think the assessment is a tool that needs to be leveraged and revisited often, right?

Because, as you perform initiatives or as you deliver initiatives towards elevating a certain capability, the target is gonna continuously move, right? And you wanna make sure that, as you progress, you have a common framework that allows you to really gauge progress, right? Because you wanna be able to compare your previous state to your current state as you work towards that desired state, right? And in the context of capabilities, for instance, there is something relevant as to thinking of capabilities in terms of dimensions as kinda like the DevOps capability assessment approaches it, and not as maturity.

Because capabilities are subject to continuous improvement. There is a certain sense of kinda like a ceiling to maturity, right? You’re mature now, there’s nothing else to do. But there is an ongoing process of continuous improvement which, I think—well, I think we’re all into DevOps, right? We know continuous improvement is one of the core fundamentals, right, of DevOps adoption. And that’s a little bit—and also talking about how, I think, this should be approached and how, at least my organization, approaches it. It’s not a one-time assessment, and a one-time picture that tells you how to get to the end, it’s an ongoing process, and this is just a way for you to quantify the current state. And you will need to continue doing that as you progress, and deliver initiatives in the dimensions where more value can be extracted. Does that make sense?

Ashley: Yeah, very much so, and I’m glad you brought up the maturity. I mean, there are methodologies, methods of using, a maturity matrix. Those tend to be, you know, do you have a process, do you follow it, do you improve it, or whatever that might look like as your adoption. DevOps is so customized to organizations, where they are, how they create software, their own sort of life cycle of how they work, which is one of the great things about it, but there’s no one way to do it to assess that, “Are you mature?”

But I think that point in time look at it makes a lot more sense of how to think about where we are now on our journey. You may want to course correct, right? You may wanna say, “We’ve learned a lot, and we’ve actually just learned that this actually is really something that’s even a more beneficial part of DevOps, let’s focus on there, next.”

Am I on the right path, here, Ulises?

Gonzalez: Oh, absolutely. To me, the process of continuous improvement is literally endless. Organizations and companies are immersed right now in a competitive environment that is constant and required a lot of frequent changes—in every sector, in every area, in every country. That’s why quality process improvement becomes literally an imperative. For that reason, the purpose of offering products demands and services demands us low cost and meet customer expectation.

To me, as I said, I’m very aligned with Leonardo. It’s very important to get insights about our current transformation state. Identify what our strengths are, and understand and accept our weaknesses to make decisions. Because the decision latency is critical to reduce, because we need to inspect and adapt about our outcomes. DevOps, it’s not only about tools, even though that kind of tool, like the assessment, helps us a lot to, as I said, gain insights from different points of view. In my case, a Product Owner, a Product Development leader, we need to inspect constantly about the results that we get. Because it’s not about what I want as a business, it’s what a market needs. That’s why this kind of assessment helps us a lot through this kind of process, to realize how steps we need to make to get close to our goal or our business objectives for this month or this quarter.

Murillo: And let me riff off of that a little bit of what you just said.

Ashley: Yeah—please do.

Leonardo Murillo: I think you raise a very valid point in terms of prioritizing, right? And I think that’s very valuable of this type of assessment. When organizations approach the adoption of DevOps, attempting to follow a pre-defined path, a pre-defined immutable process, they’re really not evaluating what is most meaningful for the organization at that particular point in time. Like you were just saying, right, when you’re building a product, you wanna deliver those features that are most meaningful, that are gonna provide the most value to the users of that product.

And I think a capabilities assessment gives us a little bit of that in the context of an organization. There are many, many different dimensions and sub-dimensions of what an organization can improve towards DevOps maturity. But what is more relevant to your organization at this point in time is not gonna b the same for other organizations.

Ashley: Mm-hmm.

Murillo: And this type of assessment gives you the ability to identify where to focus your efforts. Because you don’t wanna try and boil the ocean; you don’t wanna try and fix everything at the same time. Because then, you likely will struggle, right?

So, this concept of identifying, granularly, which areas allow for improvement and providing a quantifiable means to assess which ones are more relevant or your organization at this point in time is a very effective mechanism to direct, after all, scarce resources. There isn’t endless money, there isn’t endless time, and there isn’t an endless capacity of people to change. So, you need to very effectively apply initiatives to those areas that make the most sense for your current state.

Ashley: Mm-hmm. Spot on—very important point. [Cross talk] How is this administered? How does someone go through the assessment? Is it a series of interviews, do they fill out an online survey? Lots of ways of doing these things—how do we do this with ADOC?

Murillo: You wanna talk about that Ulises?

Gonzalez: Yeah, a little bit, it’s ________, it’s a mix between online surveys and, of course, interviews with several leaders or persons of importance of the company, because we need to understand or get that 360 degree understanding about what is really happening. That’s why the instrument or that kind of assessment, it was common understanding of what we need to know about we need to understand what we are.

That’s why we can use it in several ways. As you said, we can use it from an online survey, so everyone can participate independently of the part of the value stream where you are, because this is a very important topic. And then, when you get inside, you can interview the people who are already responsible, already part of the time to start to make decisions. That kind of point is very important to me as well, because this instrument allows us, we can gain a capability to make decisions as well. We need to wait a long period of time. We don’t need to wait a lot to make decisions quickly in a smart way, data-driven. We can make a data-driven hypothesis, we can put in our roadmap a lot of, as Leonardo said, prioritizing steps to get what the company wants. That’s why this is an easy way to get a lot of understanding on what’s happening right now in our company.

Ashley: Mm-hmm.

Murillo: And let me share some ________ as to how I think this also needs to be applied. Because as Ulises mentions, right, the tool itself is an online delivered Likert style assessment, right—multiple questions, different dimensions, and that is used to define the scores to your capabilities.

But as we’ve mentioned multiple times in this conversation, ever organization is different. And I think this is also where the DevOps Capability Assessment is being partner led. There are consultants out there that are delivering this and applying this in organizations.

I think this is also why it’s very important to partner with somebody. Because it’s relevant to have an external view to understand where there might be bottlenecks to tie this together with other practices when you think kinda in terms of value stream management. You wanna make this tool a component of a comprehensive intervention, and that intervention is going to be defined not just by the tools that you’re using, but also by the structure of the organization.

You will want to understand exactly what your structure looks like—both in terms of paper, what the organizational structure looks like, as well as in terms of team dynamics. And that will give you a sense of who you really need to talk to, who are the ones that are influencing, who are the ones that can give you further insights? Extend the tool. I think that’s very important in terms of applying this, which was kinda your original question, right, Mitch?

Ashley: Mm-hmm.

Murillo: And this is just one tool in an arsenal. It’s a tool that satisfies a lot of precision in terms of gauging a state, and it needs to be executed in a way that is consistent with the organization, just as the process of maturing into different levels of skill, right, will vary depending on your unique characteristics. I think that also applies in the context of delivering this type of assessment. There is a discovery opportunity for partners to really get to understand the culture of the organization, the culture of the teams.

And I think another component that is relevant, and in the book Accelerate, that I’m sure everybody is familiar with by now, there’s this concept of organizations trying to apply DevOps practices to one group and then just trying to replicate that across the organization and now that really doesn’t work too well.

As you expand your DevOps adoption to more groups in the organization, you’re gonna have to repeat a little bit of the process to really understand how each one team will evolve, and what is the current state of each different organization as you expand. This is not something that you do once, you transform a business unit once and then you copy paste. That’s kinda all the thoughts that come to mind when you mention how this actually looks like in practice, being applied within an organization.

Ashley: Well, and it seems extremely helpful to have the data that’s collected through the online portion and then really, those interviews also add, as you mentioned, talking about things like culture, structure of the organization—it’s that context, right?

So, you may be wherever you are on your answer to certain parts of the assessment tool, but understanding how that fits into your environment, your company, your culture, how you work—I like to describe thigs as, you know, if you are a data driven, no decision gets made without data, that’s a different kinda culture than one who goes by, let’s say, these are the inspiring leaders that help make the important decisions, right, just to pick two very different examples. But that context matters. You’re gonna take different actions based on the environment that you’re doing DevOps in. [Cross talk]

Let’s talk about how someone would use this—I’m sorry, Ulises, I cut you off there, but we really wanna hear, too. So, you go through this process with your help. What does someone get from it? Do they get a document or a readout? And then how do you go about helping them with the next phase of using some of that information to their benefit? Go ahead.

Murillo: Yeah. Go ahead, Ulises.

Gonzalez: Okay, everybody. [Laughter] Okay, to me, of course, there are different outcomes you can get, absolutely. You can, after you complete that kind of assessment, you gain insight of your current information, you get the ability to benchmark your teams and your organization across different industries on your actual level of practices. With our help, you can visualize the road ahead from one baseline to another future state. Of course, the management gets the investment areas, so we need to make decisions. They’ve got very normalized indicators to understand what we need to do on the next iteration or the next spring or the next process.

Of course, the operation teams part gets understanding on most critical areas you need to improve. So, as I said, the assessment brings us a lot of information with data, normalized data, in a controlled environment to make decisions, to make smart decisions, to gain common ground, to create consistency. Because, as Leonardo said previously, to me, it’s about consistency and a holistic view in a way that enables everybody in the company to understand independently of the level that we are, we understand the same parts of DevOps that we need to achieve, because it’s a business outcome we need to support.

So, that provides us with a high impact of our software, our service, our productivity. So, more than an instrument that provides us number, it gives us very helpful insights, as I said, to make wise decisions.

Ashley: [Laughter] I appreciate your perspective on it, Leonardo, because what you’re describing, Ulises, very much follows that Plan Do Check Act.

Gonzalez: Yeah.

Ashley: You know, take those actions based on that continuous improvement process and cycle again so you can use this as a tool more than once, right? You can go back and assess yourself maybe in a year or so, see if you—did you end up where you thought you were going to head, or did, actually, the journey migrate a little bit, move a little bit on a different track, now let’s adjust to that and move forward?

Murillo: Exactly, exactly. And I think the Act component is where there’s a lot of room for creativity and innovation, right? Once you have your, as Ulises points out, your normalized metrics, right, your normalized numbers, you will need to identify what are the better mechanisms to influence change, to produce change? They might be technical, they might be technically implemented, they might be technically led, and they might not.

So, here’s where training—and I’m shamelessly plugging, her, the DevOps Institute, this is where a lot of the skill up type of initiatives that organizations like the DevOps Institute also promote and enable. There is skill up, there is education, there is cultural changes, there is many, many ways in which you can act. And, as you point out, Mitch, which is really, really spot on, you will need—you want to constantly measure. And in terms of constantly measuring the impact of these changes, you need to reassess, and it’s a constant, ongoing effort, whether it’s every year, whether it’s every six months, whether it’s more often, I think it depends on the state of the organization and kinda what they’re experiencing at this point, what they’re going through at this point.

But I think it should never stop, because it’s continuous improvement, so you want to constantly renew this perspective of where you are.

Ashley: Well, let’s talk a little bit about—so, how would one go about doing this? Do we visit the DevOps Institute site and that kinda helps us get the process started, maybe we have a partner, one of your organizations already, or maybe we need help in selecting somebody to help us go through the process? I assume there’s resources available on DevOpsInstitute.com, is that correct?

Murillo: I think that’s correct. I think you can go and see all the different partners that are offering this tool, and you can basically look them up. That’s what I would do, look them up, see what their specific areas of expertise are in addition to the tool, where they’re located, and get a sense of who might work best for your organization. It’s a tool that, I think a lot of the value of the tool revolves around it being introduced by a partner that can lead the way. It’s sometimes difficult to gauge when you’re too close, right? So, go to DevOps Institute and look for a partner.

Ashley: And it’s a global activity, right, as represented by both of you.

Gonzalez: Yeah.

Ashley: We were talking U.S. and Panama, and I forgot where you were.

Murillo: I’m based in Costa Rica, yeah. My [Cross talk] is actually not based in Costa Rica, so we sell to the U.S. market.

Ashley: I see.

Murillo: But I’m in Costa Rica and I speak Spanish, so if you want some Spanish support, you’ve got two people here that can give you that guidance.

Ashley: Exactly. Well, very good. We’re running out of time, unfortunately. We could talk about this all day, I’m sure there’s some really interesting things and, as you do this with organizations, you know, you carry that knowledge with you to the next customer, too, so there’s benefit in working with you as you go through this process, because it—

Murillo: Oh, and Mitch, you bring something up—real quick, before we go.

Ashley: Mm-hmm.

Murillo: There’s something unique about the Assessment for DevOps Capabilities, and that’s that the body of knowledge that was used to produce this tool is directly extracted from the real world experience of, I don’t remember the number, but it’s dozens of actual DevOps practitioners, the DevOps Institute’s own Ambassadors and other partners, I think there’s tremendous value in that. It gives a collective perspective, a joined wisdom that I think is encapsulated in the tool.

Ashley: Mm-hmm, yeah. It’s far from an academic exercise, right?

Murillo: Mm-hmm.

Gonzalez: Yeah.

Ashley: Developed by DevOps people for DevOps people.

Murillo: Exactly.

Gonzalez: Exactly.

Ashley: Well, it’s been a great pleasure talking with both of you, Leonardo and Ulises, and I wish you the best as you work with organizations and use the ADOC tool, and we’re looking forward to seeing some great impact and help that you can help bring to their organizations. Thank you for joining me today. Great to talk with you both.

Murillo: Thank you, Mitch.

Gonzalez: Thank you so much, bye bye. Ciao, Leonardo.

[End of Audio]