DevOps Chats: DevSecOps and OpenShift, with Red Hat

In this DevOps Chats, we had a talk with Kirsten Newcomer, senior principal product manager at Red Hat. It is a great discussion on the state of DevSecOps and how the Red Hat OpenShift team is trying to make it easier for developers, DevOps and cyber folks to work together to create more secure applications.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Alan Shimel: Hey, everyone, it’s Alan Shimel, DevOps.com, Container Journal, Security Boulevard—you’re listening to another DevOps Chat. Welcome, and I’d like to welcome to our DevOps Chat Kirsten Newcomer of Red Hat. Kirsten, welcome.

Kirsten Newcomer: Thank you, Alan. Great to be here. So, I’m Kirsten Newcomer as you said, Red Hat. My focus at Red Hat is DevSecOps for cloud-native applications, been working at Red Hat for about four years now. It’s really kind of fun. I’ve managed to leverage kind of my—all sorts of different elements of my background over the years. I spent time at Rational Software, nine years, working on developer tools.

Shimel: Sure.

Newcomer: Yeah. Spent some time with BMC BladeLogic on the operations side of the world, seven years at Black Duck Software focusing on solutions that help companies manage vulnerabilities in open source software, and that led to the move to Red Hat where I get to focus on OpenShift security and DevSecOps and sort of everything comes together. So, DevOps—

Shimel: It really does.

Newcomer: Yeah.

Shimel: I mean, following your career, I think I know just about all of those companies intimately and know a lot of people—I’m sure we even have a lot of people in common that we’ve worked with.

Newcomer: I bet we do.

Shimel: It’s funny. So, you were at Rational, left Rational only to return to IBM Rational.

Newcomer: I know! [Laughter] That is true.

Shimel: You know, it’s the circle of life right there, man. [Laughter]

Newcomer: It is the circle of life, yep.

Shimel: Welcome. So, Kirsten, look, DevSecOps is of course something near and dear to us. We put on every year at the RSA conference our DevSecOps Days or DevOps Connect/DevSecOps Days.

Newcomer: Uh-huh.

Shimel: We actually have the virtual event of that live on Thursday, I think it’s the 26th, March 26th.

Newcomer: Nice.

Shimel: So, you know, it’s something, quite frankly, this whole security thing is the reason I got into DevOps is, I thought—what a great opportunity for security to cure a lot of original sin.

Newcomer: Absolutely. [Laughter]

Shimel: But anyway, you know, when we look at the DevSecOps and the DevOps world, Kirsten, you can’t help—you must take notice of the impact that Kubernetes and the whole cloud-native kind of movement has had on this.

Newcomer: Okay.

Shimel: And a great—

Newcomer: Absolutely, no—big changes, yeah.

Shimel: And, you know—and it’s really at the core of so much of what is being done right now from an infrastructure point of view. And oftentimes, I don’t know if people—like everything else, right? I remember when the cloud came out and we moved from client-server and all this. Security, there’s always that lag, right, where, “Oh, yeah, we gotta do security with this.”

Newcomer: Yep.

Shimel: And we had a little bit of that with Kubernetes, I think, where—

Newcomer: No, I think that’s fair.

Shimel: Yeah? You know, people were rushing to do Kubernetes and experiment with it, deploy it, and then someone started saying, “Wait a second, what about security, here? What about compliance? What about this or that?”

Newcomer: Yeah.

Shimel: And now, I will tell you, and I’m interested in your opinion—I thought the lag time was much shorter this time than what we’ve seen in the past.

Newcomer: Oh, I absolutely think so. And I think it’s really interesting. A couple of things that I’ve observed. One is that I really think containers and Kubernetes have created a terrific opportunity to shift security left, and I’ll kinda be more explicit about that in a couple ways. But I, in fact, have worked with a chief of cyber defense in the public sector who believes and evangelizes that containers improve security, and I’ve stolen his line, because I just actually think that that’s really valid.

Because of the changes in the model, with the OS dependencies and the run time, everything’s kinda packaged together and you have that immutable container image that you’re always deploying from, you really can manage things in a new way and not every team is ready for this, and not every team, not every security vendor is ready for this, either.

So, I think the opportunity has been twofold. One is to kind of adjust the thinking about security so that it no longer becomes something that happens at the end, but that’s still a process, that’s still a lot of work. And the other thing is, the security tooling space itself has historically been pretty siloed, right? There’s been, you know, the OS layer security, the network layer, there’s been kinda—you know, my SIEM and my SOC that I’m working with, and my cert management and my vaults.

And none of them really were designed to work with the level of automation that Kubernetes and containers enable, which led to this really interesting opening for smaller companies, a lot of startups that now have been around for a little while, actually, to start breaking down those silos and delivering security capabilities in a new way that kind of really do fit the DevSecOps model, because you get kinda one vendor bringing a set of capabilities that ranges from CI/CD integration to run time security to network security, really fascinating space to see how it’s been evolving.

Shimel: It really has. The other thing that I will tell you, Kirsten, that I was again encouraged by was when I first started seeing people talk about security through containers and Kubernetes, it really was like—let me scan your container for vulnerabilities, right?

Newcomer:  Yeah. [Laughter]

Shimel: And I’m thinking to myself—I don’t care whether my code’s in a container or in a hypervisor or on bare metal. A vulnerability scan is a vulnerability scan. There’s gotta be more to this than just a more vulnerability scan kind of a thing.

Newcomer: Absolutely. And that’s, I think, where, as you mentioned earlier, that, you know, Kubernetes really has come a long way in that space. So, there are things like, if we think about—so, absolutely. Vulnerability scans are fundamental and kinda, people always understood those. They tended—historically, they’ve happened a little too late in the life cycle, making challenges for the AppDev team and such.

We’re seeing that shift left a lot, but then there’s all these other range of things that you should be doing and can be doing to build security into the platform. And so, when we saw pod security policies in Kubernetes, for example, that’s a way that the Kube admin can take advantage of the Linux features that enable container isolation at the Kubernetes layer and enforce things like ensure that a container doesn’t run with unnecessary privileges.

Shimel: Sure.

Newcomer:  And so, we are seeing more capabilities—now, that’s still beta, so the community still has some work to do there. But as you say, we’re seeing more emphasis and, you know, the open source Kubernetes security audit sponsored by the CNCF is a great example of that.

Shimel: Oh, absolutely. And look, CNCF has done yeoman’s work here. They’ve really, really done a nice job with the whole Kubernetes project, specifically security.

You know, but one of the reasons, like—we did a webinar yesterday for instance and it was on, you know, the different cloud providers versions of Kubernetes and—

Newcomer: Uh-huh.

Shimel: [Cross talk] environments. And, you know—fantastic, we had 700 and something people sign up for this webinar.

Newcomer: Great.

Shimel: A lot of great questions, even on the YouTube afterwards, a lot of great questions. One of the things that became clear to me in listening to people talk is that—you know what? Just going to, whether it’s AWS or Azure or GCP or whatever and setting up my Kube environment may not be the best thing for me because I’d rather get one of the package distributions, if you will, or a more complete solution, of which Kubernetes is a piece, like an OpenShift, but some others were mentioned—Rancher and so forth.

Newcomer: Sure.

Shimel: But OpenShift was the most popular one mentioned. And I think one of the reasons is, people want to know that security has been kinda built into it.

Newcomer: Yeah.

Shimel: Beyond just the Kube release, whether it’s 1.4, 1.5, what have you, 1.6—you know, what else has been wrapped around that just K8—

Newcomer: Yeah.

Shimel: —to make it easier?

Newcomer: Absolutely, and that’s been a big focus for us at Red Hat, really, from the beginning. You know, an enterprise solution—and this is a place where, so, we’ve talked about changes in the security tooling that needed to be made to adjust to the technology, but by and large, the principles still apply. You know, you still need audit logs, right? You still need, you know, to log data from the cluster itself. You need log data from the applications. You need monitoring tools. And so—and you need to think about host arrest security as well.

And so, we’ve kind of really looked at the stack as kind of spanning all of those things, and we want to make it as easy as possible to manage that solution as a solution, right? And so, that if you—and plenty of folks do this, you know, and get value from it. But if you’re in a situation where you can afford to kind of do a DIY Kubernetes and add in a logging stack and add in kind of all the different pieces that you really need for the enterprise solution, then you have to maintain them each separately and you’re maintaining the host OS separately as well.

And so, we’ve really focused recently, especially with OpenShift 4, on automating everything, which includes the host operating system. So, Rel CoreOS, Fedora CoreOS available as an open source project. It’s a container optimized operating system that we manage as part of the full platform. It makes it much easier to apply OS updates across your cluster. You can do that in a way that has zero downtime for your well-behaving apps. You can take a node out of service in the way you would, you know, that Kubernetes manages it. Update that with a patch to OpenShift or its components or the operating system itself, put it back in service.

And so, kind of a combination of looking at the whole stack holistically, what is an enterprise need in order to ensure that they have a secure environment, a stable environment, a highly available environment and making sure that we kind of leverage the terrific declarative and automation capabilities of Kubernetes kind of throughout the whole stack.

Shimel: Absolutely. You know, I didn’t realize just really all of—you know, until you just said it, I never kinda put together all the pieces of what you guys are doing with that. It’s pretty amazing, isn’t it?

Newcomer: It’s really cool. And one of the other fun things that we get to do is, we actually—so, back to, if we think about the host OS layer for a bit, right? If you need to scale your clusters, that means adding a new host and you’ve got the work involved to be sure that you deploy that infrastructure and that you secure that infrastructure according to your standards and then you put Kubernetes on top of it and add it to the cluster and you’re ready to put workloads on it.

And we’re using the concept of Kubernetes operators to automate the components of OpenShift, including the host OS. So, we have a machine config operator, so if you’re deploying OpenShift in a cloud where you can automate the addition of infrastructure easily, you can use the machine config operator and the declarative nature of Kubernetes, right? It’s a container optimized OS, it’s delivered primarily as container images, the user space is read-only, you’ve got that declaration of what a host should look like and you can just use the machine config operator to spin up new hosts and add them to your cluster.

Shimel: Excellent. You know, we would be derelict in our duties, I guess, if we didn’t mention the whole COVID-19 and how that’s affecting things.

Newcomer: Fair enough, yes.

Shimel: In my mind—so, first of all, I don’t think a lot of what we’re seeing right now and its effects are necessarily three-week or six-week or, you know, a lot of this stuff is gonna stick.

Newcomer:  Yeah.

Shimel: I think one of the things that are gonna stick are, people are gonna want more automation, right?

Newcomer: Absolutely.

Shimel: In their deployments, in their operations, in their infrastructure.

Newcomer: Yep.

Shimel: And automation, from a security point of view, is kind of a double-edged sword, right? At least we, from—when things are automated from a security point of view, we think we know exactly what’s gonna happen, and so we can plan around that. On the other hand, security people tend to get a little fritzy, you know, when things come out of our cold, dead hands, here.

Newcomer: Yeah.

Shimel: What do you think? How—what do you think?

Newcomer: I know; I agree with you. And I think one of the things that, one of the ways that Kubernetes is kinda pushing the boundaries with security teams. And I talk to a lot of different security teams over time. So, it’s kinda like—you know, they’re a little bit more comfortable with the idea of DevOps for the application layer at this point, right? For those containerized apps that are gonna be deployed, it’s like—okay, they kinda have an idea of, “Yep, security static analysis tools, integrated vulnerability scanners integrated.” There’s some cool new stuff coming out that I’m seeing on the market, they’re tools that are starting to evaluate the configs and the Dockerfiles, too, that’s really great.

But—and then as we continue to move into the ops space, right, with Kubernetes and the SDN, which is required for any Kubernetes cluster, and now we have service mesh added into the mix, or Istio for microservice based communications. The network security teams are uncomfortable with who’s gonna manage network policies at the Kubernetes layer and now, I’ve got service mesh or Istio policies as well, and who’s responsible for those?

And in the end, it’s really about visibility. So, if we circle back to automation, we’re really looking at infrastructure as code, you know, GitHub’s model that applies again to the Kubernetes layer, to the configuration of the Kube cluster as well, and we need to put in place, you need the automation to get the best results, but you need to do it in a way that gives the security team visibility into the kinds of things they’re used to seeing.

And so, I think this is another way where we’re starting to shift left in new ways, and I’m also seeing solutions that help automatically generate network policies based on evaluating an environment or an automatic generating service mesh policies based on evaluating the config for the apps.

But all of that, you know, the network security guys aren’t necessarily gonna log into the cluster to look at that. So, how do they get the view they need to get, and how do they help the AppDev teams understand that whole risk management and their security perspective? And so, this is where it’s not just about tools, it’s about people and processes, right? And I think there’s so much richness here, but you’re absolutely right—we can’t do it without automation. And so, we’re gonna have to help these people become comfortable with this shift while ensuring that they get the visibility they need to know that things are being done in a way that meets their risk guidelines, their regulatory requirements, and the security posture they want.

Shimel: So, to me, one of the fundamental things with DevSecOps, one of the fundamental shifts in DevSecOps is, first of all, security people recognizing that they’re not the only ones who care about security.

Newcomer: Mm-hmm.

Shimel: Believe it or not, the developers and the DevOps folks, they do care about security. They may not be security trained people, but they care about security.

I think the second thing with that is, the developer people have to realize, and the DevOps teams have to realize that the security people aren’t there just to be the roadblock.

Newcomer: Right.

Shimel: And they aren’t always gonna be the people who say no.

Newcomer: Right.

Shimel: And what we’re—so, this has nothing to do with technology, per se. This is a people to people issue.

Newcomer: Absolutely.

Shimel: And I think we’re making real progress in that, and as a result, what we’re seeing is security people paying for and approving security tools that developers in DevOps teams are using—

Newcomer: Yeah.

Shimel: —on the Kubernetes platform—

Newcomer: Yep.

Shimel: —you know, because that’s kinda the platform of choice now, and that, to me, that’s what keeps me optimistic about this.

Newcomer: Yeah. No, I’d agree, and also, when we think about the broader landscape, I mean, there’s, the security threats are just ever-evolving, ever-changing. And there aren’t enough security trained professionals to go round, right? There aren’t enough cybersecurity folks, et cetera.

And so, we have to figure out, how do we help teams scale, and collaboration is the way to do that. And you’re right, I see the same thing you do, right, that the security team is approving those tools that are container native, Kube native tools that help to ensure. And in some ways, I don’t mean this in a negative way, but in some ways they kind of are pushed into that because they’re traditional tools that used to be used kinda at the back end of things aren’t effective in the container and Kubernetes world.

And so, they really do have to go with this shift that we’re seeing in the vendor landscape, and it just opens up this terrific opportunity, though, for that collaboration and that shared knowledge which, you know, is an adjustment for people sometimes.

Shimel: Yeah, agreed, agreed. Kirsten, when we first started, I said, you know, the time goes really quick, I’m only gonna keep you here 15 minutes—well, that was 21 minutes ago.

Newcomer: [Laughter] Okay.

Shimel: We—yeah, we did it. Alright. We’re gonna have to call a wrap on this one. Maybe we’ll have you come back another time and we—

Newcomer: It’s a plan!

Shimel: —can continue the conversation there, good?

Newcomer: Great. It was great fun—thank you so much, Alan.

Shimel: Thank you.

Newcomer: Alright.

Shimel: Kirsten Newcomer, Red Hat, our guest here. This is Alan Shimel for DevOps.com, Container Journal and Security Boulevard. You’ve just listened to another DevOps Chat. Stay well, everyone.

— Alan Shimel