The Assessment of DevOps Capabilities (ADOC) is crowdsourced, vendor-neutral and designed for individuals, teams and organizations who want to baseline their current DevOps state, identify the next target state, gain insights into how to improve their organization and team capabilities and measure and accelerate continuous improvement during their DevOps journey.

The assessment models five DevOps dimensions – the human aspects, process and frameworks, functional composition, intelligent automation and technology ecosystems – to help you gauge your DevOps maturity and identify areas for improvement. In this TechStrongTV interview, Alan Shimel talks with Garima Bajpai, Louise Cermak and Brendan O’Reilly about reshaping culture to get the most out of any DevOps journey. The video is below, followed by a complete transcript.

Announcer: This is Digital Anarchist.

Alan Shimel: Hi, everyone. Thanks for joining us on another segment here for TechStrong TV. This is a special segment panel for TechStrong that I’m gonna be conducting, here. I’m happy to have you join us. I’m also really happy to have three interesting people join us. Let me introduce you to them. And if I mispronounce people’s names, I apologize; we do the best we can here on TechStrong TV, right?

But we’re gonna start with Garima, is it Bajpai? Fair? Good? Okay. Garima, why don’t you introduce yourself?

Garima Bajpai: Thank you, Alan. I’m Garima Bajpai from Ottawa, Canada. I am the Founder for the DevOps Community of practice here in Canada, which has three chapters, Toronto, Edmonton, and Ottawa. I’m also the organizer for the DevOps Summit which happens in Canada November 8th and 9th, and I am Ambassador with DevOps Institute, as well as an Ambassador with the CD Foundation. So, today, we are going to have a very interesting dialogue on DevOps assessments. So, I can’t wait to get started, right?

Shimel: Yeah, thank you, thank you. Thank you for all you’re doing for the DevOps community. Next up, let me introduce you to Brendan O’Reilly. Brendan, welcome—a little bit about yourself?

Brendan O’Reilly: Sure. So, my company is called Daysha DevOps. We’ve been working with the DevOps Institute now for about five or six years, predominantly delivering their training. And I myself am a consultant, and like the other speakers here today, really looking forward and excited about what’s happening with ADOC, because I see a big opportunity for our customers to get deeper into their DevOps journeys, to progress those journeys.

Shimel: Fantastic, and thank you. And then last, but certainly not least is, we have Louise Cermak—Louise, did I get that right?

Louise Cermak: Almost, yeah.

Shimel: Well, say it right, because you’re entitled to have your name said right, it’s my fault. So, please, how do we pronounce your name?

Cermak: Thanks, Alan, it’s “Chermock.”

Shimel: “Chermock,” okay.

Cermak: Yeah, so, I’m Louise Cermak. I’m a Principal Consultant with Catapult CX. We build software ourselves, and we also consult in agile and DevOps and have been doing so for sort of 10, 15 years. And we’re excited for ADOC, because actually, having the combination of being engineers as well, we can really understand the capability and how to apply it.

Shimel: Absolutely. Well, all three of you, thanks for joining us. So, a couple of you have mentioned ADOC, right, and this is something that our friends at the DevOps Institute announced, I guess it was last week or the week before now. And ADOC, for our audience who may not be familiar, stands for Assessment of DevOps Capabilities. And this is a new assessment model that the folks at DevOps Institute have been working on.

Look, you know, I know they’ve been working on it for over a year already. It’s been primarily driven by Helen Beal, who is a—Helen is a very well-known DevOps person in the community, but she’s also the Founder of the DevOps Institute Ambassadors, and has been a DevOps consultant herself for many, many years, as long as I’m doing DevOps.com.

And then Helen was really helped with this by Eveline Oehrlich, who’s a DevOps Research Director at DevOps Institute, a former Forrester VP. And then I also know that she’s been helped a lot with people like the three of you, who are real world DevOps consultants who are helping organizations every single day with their DevOps transformations and capabilities.

So, before we jump into ADOC, though, let’s take it up a notch and talk about DevOps assessments in general, right? Are they—why are they important? Why are they important, and if anyone wants to give us sort of a brief history of evolution or time around these assessments and how they’re used, let’s say, by consultants and/or end user organizations.

Garima, I know you have a lot of experience, so if you don’t mind, I’m gonna ask you to kick it off, and then have the rest of the panel chime in.

Bajpai: Right, and I actually term DevOps transformations as a journey towards remaking culture. So, it has been largely a complex journey, because of two factors, actually. The first factor is that the complexity brings along with, you know, a lot of focus on how you can upskill people, how can you actually transform your knowledge based and skill set to a journey which is basically based on experimentation, on learning, and learning from failures, right?

And the second aspect is that there is no such literature available. You know, there is a lot of talk about DevOps transformation, but the literature is in the making. So, there is limited academic support as well as support from a best practices perspective how to do this transformation, right? So, these assessments come very—it is very crucial, because it will help you assess where you are, and every organization will have unique challenges, right?

So, it is important to assess where you are in your journey and what you can do. You know, it’s not like a traditional transformation journey of three to five years where you are relying on a lot of targets ________ models which were already defined. It’s continuously evolving, right? So, you will have to have a view on what you can do today and how you can incrementally, in short cycles, transform.

So, what happens in this process is that, you know, assessment becomes more crucial and essential, because there are discrete efforts. There are efforts in short cycles, and this assessment and measurement will help you course correct, will help you understand where you are heading to in your journey and what is needed as next steps to ensure that you incrementally build value, right?

So, that’s why, I think, these assessments are extremely important. We will talk along this conversation how ADOC can actually help you on this journey as well as what are the major impediments and challenges organizations have and what should they look out for.

Shimel: Fair.

Cermak: I think, over the last few years where we’ve been helping organizations, what we’ve noticed is that many of them are trying to do their own assessments, and I’ve been into so many organizations where people have got Excel spreadsheets and they’re trying to measure how they’re performing.

Because, you know, Garima’s right—there’s a lot of learning and experimentation. A lot of it is self-learning, which is what we encourage, right, from a DevOps culture perspective. But people wanna know that they’re going in the right direction, so they start with their own spreadsheets, fill them in, try and measure their capabilities. And a tool like ADOC helps organizations have a much more structured way of doing that, and obviously, it’s more scalable than an Excel spreadsheet, but people wanna know that they’re doing the right things.

Shimel: Agreed.

O’Reilly: Just a little bit about the history, perhaps Alan—you know, ADOC is brand new, but there was a progenitor in the form of DORA, if you recall.

Shimel: Sure, sure.

O’Reilly: Yeah, so, that was with Jez Humble and Nicole Forsgren. So, when they produced their State of DevOps report, it was a means by which organizations could form views about what good looked like, so you have your release and then so on.

And it definitely raised people’s curiosity about—can we do this, can we measure? And of course, one of the principles with any DevOps transformation is to be data-driven. And so, the 180-odd questions that are included in the enterprise edition are very much about trying to capture and turn subjective opinion into something quantitative, a good first guess. And I think DORA has been successful, it’s now part of the Google organization, I believe, but this is a nice independent offer from the DevOps Institute.

We’re really looking forward, here at Daysha, to taking that to our clients. I see some really interesting use cases. Some of the people that I’m talking to right now are just starting in a CTO role, and they want to baseline what they’re doing.

Shimel: Absolutely.

O’Reilly: So, they can look—you know, six months from now, they can say, “How much have we done?” They wanna get an understanding of what have they got in-house right now and then look at it again to see if they’ve been effective in those changes that Garima was—really well-made point by Garima—nice, short, sharp, Plan Do Check Act type changes that are gonna take place over the next six months.

Shimel: I mean, for me [Cross talk]—

Bajpai: Right, I mean, I also want to focus on two specific aspects of ADOC, which stands out. So, a lot of people have actually understood that you cannot remake culture without people-centricity, right? So, that’s a given. And we also understand that emerging technology can be a tool in your toolbox when you’re trying to kind of transition from X state to Y state. And a lot of focus has been placed on processes and frameworks.

What ADOC has been standing out for me is, they have actually reflected to other different things, which are equally essential. The first thing is functional silos. And if you look at large enterprises, what they struggle with, like, what has been in the past is that we have large, functional organizations and units with their own business goals and objectives. And when these people are trying to [Audio skips] towards ________-centricity, towards DevOps capabilities. That’s a large challenge, because you know, it’s not easy to actually break down the functional silos, burn all the ships so that there is no, you know, way back. It has to be carefully done so that we have a possibility of course correction. That’s the first important point.

The second important point which stands out for me is automation. And when I look at automation, I see that data driven approach towards DevOps and how that will ensure that we automate good processes, we automate the capabilities which are creating value, not only end to end, but also within the practices, right? So, that’s—those two things, I think, put together with people, technology, and process would actually make a kind of, a full house for, you know, doing this assessment in one way rather than looking at it in pockets.

Shimel: Agreed. In my mind, you know, Brendan, you mentioned the DORA folks, and of course, Nicole has moved on from Google, she’s over at GitHub, and I don’t know if Jez is even with Google anymore, frankly. But really, with DORA, it wasn’t necessarily a consultant’s tool, though consultants used it.

I think one of the interesting things with ADOC and what I know about how it came about is, it really was designed for consultants to use as part of their consulting practice to help you collectively help your clients get a baseline, a roadmap to show where you’re strong, where you’re weak and going forward so that that consultant can be more helpful, more impactful and, frankly, more efficient, which translates to less dollars, right, that clients waste in kind of circling around to figure out where the ground zero is.

So, I would just point out that, now, that’s one use of ADOC is as a consultant’s tool. There’s also the community version for individual organizations and large enterprises can do it on their own, they don’t necessarily need a consultant. But it really was designed with that in mind, and I think it’s gonna be interesting to see, you know, DevOps especially, it was one of the knocks on DevOps, right? And I’m DevOps.com, I’ve heard this from day one—it’s squishy, there’s no manifesto, like in agile. There’s no real definition. It’s, how do you put your finger on it? You know, it seems like a unicorn or a mythical beast or something.

Well, with ADOC, a lesson I learned early in my career in technology was, everything can be measured, and we measure everything, right? And ADOC kinda gives us a little bit of that certainty, at least as opposed to where you are in DevOps. I’m wondering, have you guys actually had a chance yet to try ADOC out with real world clients and what some of the results or findings were?

Cermak: Yeah, we’re doing that right now with a client and we found that it was much easier to prioritize the activities that they needed to do, so we could align the activities with their roadmap and make sur that they were incrementing in the right direction to address some of the challenges that they had with their own customers.

So, we took a very customer-based approach to prioritization of the activities, and they could really see a difference in the way they were able to engage the customer, deliver the features much faster, get the features agreed properly and through improving their continuous delivery pipeline, for example, and their lead time to change would improve and stability and reliability. So, we were able to prioritize exactly the things that were important right then to them with their customer base.

Shimel: That’s great. Anybody else?

O’Reilly: Well, just to echo, I personally haven’t had experience with the product just yet, but I have had that experience of walking the corridors with a spreadsheet, meeting people, capturing data—yeah, and that’s a really slow, inefficient way to capture a picture that evolves very, very quickly. So, I’m excited to do it this way, which is just way more efficient and lower cost to the client.

And I think in terms of the consulting add-on, it’s that experience of having been through several transformations that allows us to analyze the results once they’re produced. And then, exactly like Louis has pointed out—prioritize, right? So, what changes can we make that will fit in with the natural rhythm of this company if they’ve got, if they’re not yet delivering on a regular basis? So, that kinda stuff I think is where the consulting value add is.

Shimel: Interesting.

Cermak: And the other thing that’s quite nice is, you can get the marginal gains as well, because you don’t have to tackle everything at once. You can improve one metric and then go and do the next one and the next one and continually improve, which I think the teams that we’re working with have really appreciated that, because they can see an uptick quite quickly, they get motivated, they wanna change more, they want to get better, and they keep going and keep going and getting those marginal gains.

Bajpai: Right, and this is actually an extremely good point, because it gives us a perspective and this is something which I actually have experienced myself. I went to experience ADOC myself and, you know, I wanted to bring my team in as well. But what has happened in this journey, I have realized that organizations have overestimated the potential of short-term gains. And when I say that, often, people do not realize that whatever transition or transformation from a DevOps perspective they are doing, they are looking at short-term gains and probably losing out on the long-term potential of, you know, larger, creating larger value, long-term value, right?

So, this is also a reflection that when you take this kind of an assessment, you would potentially have two aspects—what you can do in short-term, and what you achieve through several initiatives put together incrementally and how that outcome would look like in long-term, right? So, that is where these assessments would become very crucial for organizations, at especially big organizations.

Shimel: Agreed, agreed. With all three of you being involved in consulting, I’m gonna ask you to take off your consultant hats for a second and for end users, end user organizations out here watching this, is ADOC of use to them without a consultant?

Cermak: Oh, definitely, yeah. You know, we do build software, and we use those metrics ourselves, and we’re constantly monitoring those dials and—okay, we do some consulting around this stuff as well, but actually, we’ve got customers that we need to deliver to, and for us, being on top of all of that all the time is really helpful.

So, anyone taking the assessment would be able to identify changes that they could make in their own engineering. That’s straightforward. It’s really set out very easily to understand.

Shimel: Very good.

O’Reilly: Alan—

Shimel: Go ahead, Brendan.

O’Reilly: Yeah, so, there’s a friend of mine who’s a lawyer, and pretty much any time I ask him a question, the answer is, “It depends.” [Laughter] So, in answer to your question about whether end using organizations can benefit from this or that consultant, I would say it depends. It depends on the mindset in the organization, the appetite for learning, the appetite for change, and so forth.

But for sure, even just to kind of get started and to understand where they’re at, there’s huge value here without a consultant just to get a picture of where you’re at.

Shimel: Agreed. And I think that’s part of this, too, and I think that’s also where there’s the—I forget if they call it the enterprise edition, which is the full, I forget how many questions there are already, hundreds of questions, versus the community edition, which I think is just, like, 30 questions or something like that. But even the community edition, you know, I think brings value to an end user organization who may just want to—you know, let me stick a flag in the ground as a marker and then see where we go from there.

Guys, we’re about out of time, but I want to thank you, all three of you, for joining us. I should mention that if you’re interested in ADOC, right, Assessment of DevOps Capabilities, you can find out a lot more about it at the DevOps Institute site, which is DevOpsInstitute.com, and I know that there’s licenses for consultants, end user organizations. We’ve also launched the membership association, so as a member of the association, I think you get some use of it as well.

But for now—Louis, Brendan, Garima—thank you so much for joining us on TechStrong TV. Good luck, and keep us posted with ADOC.

Cermak: Thanks, Alan—pleasure.

Bajpai: Thank you.

O’Reilly: Thanks, Alan.

Shimel: Thank you. Alright, we’re gonna take a break and we’ll be right back with our next guest.

[End of Audio]