DevOps Chats: Plugins Extend Spinnaker for the Enterprise

Spinnaker Summit 2019 Preview: Secrets management is vital to any vibrant DevOps and GitOps toolchain. As part of its open source maturation and thanks to the contribution of our DevOps Chat guest, Spinnaker now has a secure secrets management capability enabling GitOps, taking non-application secrets out of plaintext in hal config files.

Cameron Motevasselani, software engineer at Armory, is giving his talk “Spinnaker Plugins: Extending Spinnaker for the Enterprise” on Sunday, Nov. 17, at 10:45 AM PT.

In addition to his contributions to Spinnaker’s new secrets management facility, Cameron shares the work in progress to create an extensible Spinnaker plugin system. The current work is an early MVP to be followed by the implementation of PF4J and concrete extension points as part of Spinnaker stages.

As usual, the streaming audio is immediately below, followed by the transcript of our conversation.

Transcript

Mitch Ashley: Hi, everyone. This is Mitch Ashley with DevOps.com and you’re listening to another DevOps Chat podcast. Today, I’m joined by Cameron Motevasselani—Motevasselani, thank you [Laughter]—a software engineer. He’s also a speaker at Armory. He has a couple of talks that are at the upcoming Spinnaker Summit 2019 in San Diego. First one—I guess this first one is on the plugin system and the second one is on secrets management.

So, Cameron—welcome to DevOps Chat.

Cameron Motevasselani: Hi. Thanks for having me.

Ashley: Glad to have you on. Well, tell us a little bit about yourself. Introduce yourself and tell us a little bit about what you do at Armory and what Armory does.

Motevasselani: Oh, totally. Yeah. So, I’m Cameron Motevasselani. Thanks for the quick introduction. Let’s see, I’ve been at Armory for about a year now. I joined and hopped feet first and started working on this secrets management project. So, that was pretty cool. And then, now, recently, I’ve been working on this plug-ins system. Yeah, that, I think, about sums it up.

Ashley: Great. What does Armory do? Yeah.

Motevasselani: Yeah, Armory, we basically have an enterprise offering for Spinnaker. So, we have some extensions that we’ve written on top of Spinnaker, so we don’t fork and have to maintain all that, we just—we extend Spinnaker and provide some extra features that way.

Ashley: Very nice. Now, the way you phrased that, you’ve been working on secret management and the plugin. Does that mean you’re actually contributing, writing code and contributing it to the open source, or are you primarily an implementer of what’s already been built?

Motevasselani: Oh, no, that’s a great question. Yeah, for the secrets management as well as plugins, we both, we’re contributing directly to open source for both of those projects.

Ashley: Awesome. Excellent. Well, let’s start with secrets management. I mean, I think we all kinda have the idea of where we’re gonna keep our keys, our passwords, certificates, all that good kinda stuff. We don’t want that heading into the source code management system or somewhere in the cloud or in someone’s software. Tell us a little bit about the secrets management part of Spinnaker and what you’ve been doing, what you’re gonna talk about.

Motevasselani: Yeah, totally. So, secrets management really came up as we started talking with customers about their Spinnaker setups and how they deploy and manage the configurations for Spinnaker.

As you know, Spinnaker has the keys to the kingdom, essentially. It has your cluster keys and your credentials in order to deploy cloud resources that cost you money, right? And those are running—it runs the code that your organization arrives on. You can’t have that in source control. That is not, not good to have.

Ashley: No-no—a big no-no, though a common mistake. [Laughter]

Motevasselani: That’s right. Right. [Laughter] I think GitHub even yells at you now, if you have keys—

Ashley: Oh, does it? [Laughter] Nice.

Motevasselani: – in there. So, they’ve been adding nice little security features here and there. People would like to move towards this idea of GitOps so that you can manage your configuration via pull requests. It’s a great idea, and a lot of people can’t do that, because the keys are quite literally in the Spinnaker configurations.

So, we pull those keys out and are essentially referring to them with our secrets management solution. So, you can use different backing stores such as S3 buckets or Vault and store your secrets there and then refer to them in the Spinnaker configuration that way.

Ashley: Mm-hmm. Excellent, great. So, in your talk, you’re gonna be walking through what you’ve been doing, the functionality of it, how to implement it via GitOps?

Motevasselani: Yeah, so—that’s right. The functionality, how it differs from, there are other solutions coming into open source now that tackle a slightly different use case. So, discussing the differences there as well as the future, essentially, of secrets management.

Ashley: Okay. Very good. Great. Anything else to tell us about that before we move into the plug-in system?

Motevasselani: Yeah. You know, secrets management, it is about the secrets for Spinnaker in particular. It’s not actually tackling the application secrets themselves.

Ashley: Oh, okay.

Motevasselani: So, I just wanted to make that clarification. And that’s at the current time. It’s possible that we might have a story for that in the future, but the talk that I’ll be giving is around Spinnaker secrets, not application secrets.

Ashley: Okay. So, this is just what’s required to run, operate, manage the Spinnaker environment, not your application secrets?

Motevasselani: That’s correct.

Ashley: Okay, great. Well, that’s an important clarification. Great. Well, let’s talk about the new plug-in system. What is new about it, other than being new code?

Motevasselani: Yeah. So, the plug-in system is, we’ve been developing it as a very—it’s been a very alpha, MVP type feature. So, right now in open source, we do have a method for users to create custom stages in Orca. There is quite a bit of knowledge and expertise that you have to have in order to do that, so I don’t quite recommend doing it right now. But we’re building on this plug-in system and we’ll be testing it out with users hopefully in the near future.

So, right now, there’s currently no UI, but we’re working on a UI and then that should be getting open sourced as well in the future.

Ashley: Now, that sounds like kind of a tough assignment. How do you create a plug-in system that anybody can do anything they want with, because that’s probably one of the requirements is, you want a lot of flexibility. How do you go about architecting and designing that so what you’re building can be extensible and you’re not gonna have to re-do it in a year?

Motevasselani: Yeah, that’s such a great question. Because we are going with this MVP model of just getting something working and then getting feedback on it to iterate, we actually are already planning on deprecating this first, let’s say, method of loading plugins into Spinnaker. We’ve already been doing some internal revs on it and we’ve been working with the community, Google and Netflix in particular, on this project. So, we are going to be moving away from our current implementation into another full-fledged plug-in framework, and we’ve chosen PF4J for that.

Ashley: Okay. Now, I’m not familiar with that framework. Can you tell us a little bit about that?

Motevasselani: Yeah, so the way we currently did it—and hopefully, this isn’t giving too much of my talk away [Laughter]—but the way we’re doing it right now is essentially shoving JARs into a ClassLoader for Orca. Well, via Quark. So, these are different Spinnaker services and libraries that make up Spinnaker.

Ashley: Mm-hmm.

Motevasselani: Right now, it’s a pretty dumb way of doing it, just adding JARs onto a Classpath in order to load it onto the Spinnaker spring application context.

Ashley: Makes sense. Pretty easy way to do it.

Motevasselani: Yeah, exactly. But like you’re saying, there’s no—it’s very open ended right now. We are going to, as we move to PF4J, define concrete extension points within Spinnaker so that there’s actually not a sprawl of extension points, there’s clearly defined extension points throughout the code.

Ashley: Mm-hmm.

Motevasselani: That will help us iterate on this feature and make sure it’s a good solution as we continue developing it.

Ashley: Okay, and this is the PF4J that’s on GitHub? I think it’s an Apache license, right?

Motevasselani: Yeah, that’s correct. I believe it’s Apache. That’s correct. And so, moving into PF4J will give us a lot of really nice features that our current implementation doesn’t have, such as using a different ClassLoader for every plug-in so that it can have its own dependencies, et cetera.

Ashley: That makes a lot of sense. Are there any unique challenges for implementing a plug-in system in an environment like Spinnaker? You know, this is software to create software, software to deploy software. You know, it’s not an application in and of itself, it’s something that you’re gonna make a key part of your kinda tool chain, your workflow pipeline. I wonder if there’s any unique challenges that come about in trying to build this.

Motevasselani: Yeah, it’s interesting, because we have been looking at, you know, as you develop something new, you look at prior art in order to determine how to build what you’re building, I guess. [Laughter]

Ashley: Mm-hmm.

Motevasselani: For lack of a better word.

Ashley: Technical speak, I know. [Laughter]

Motevasselani: That’s correct. [Laughter] So, looking at something like Jenkins or Eclipse that both have plug-in systems, same with WordPress, those are all kind of monoliths, right?

Ashley: Mm-hmm.

Motevasselani: Spinnaker is definitely many different microservices composed into one application. Essentially, we’re just kind of turning it into a platform, kind of, by making it extensible and pluggable.

But that is, to answer your question, a major challenge is, you have all these different microservices. You know, depending on how you deploy and update your Spinnaker installation, they might be on different—I don’t wanna say different versions, but if you’re deploying master all the time, you might have a different set of microservices in production compared to if you’re all running the same release version.

So, deploying plug-ins into that environment is not an easy task and how you get the versioning done is important, as well as rollbacks.

Ashley: You kinda have a recursive problem here, right? Because you have to version control the plug-in manager, which then version controls of what plug-ins can work with it, c?

Motevasselani: Yeah, exactly. Plugins might rely on some functionality that is in one version of a microservice that’s not in another version, and if you need to do a rollback, how does that work, et cetera. That’s correct.

Ashley: Well, you kinda took my breath away for a moment. You scared me when you said, you mentioned the WordPress plug-in system.

Motevasselani: [Laughter]

Ashley: Please, don’t do that! [Laughter]

Motevasselani: I do think, though, that WordPress is an example of a project that has done very well—

Ashley: They have.

Motevasselani:—because it has such as good ecosystem of plugins.

Ashley: It’s come a long way. It has come a long way. It’s much more, I don’t know, not self-sustaining, but kinda auto updating much better than it used to be. [Laughter]

Motevasselani: Right, right. And we do hope to take learnings, right, from past projects, let’s say, as well. [Laughter]

Ashley: Absolutely. I don’t mean to dis WordPress, you know? You gotta every once in a while. Well, tell us a little bit about more about your talk. What are you gonna cover, then? Is this more how to use it, or you’re gonna lay out kinda where you’re going in this next version and you’re looking for feedback? What are you trying to accomplish or get out of it? What would you like to have happen in the talk?

Motevasselani: Yeah, you know, that’s a really good question. I think there’s a lot of stuff to cover, all the way from our journey running and creating this project, working with the community and those efforts, to showing users, “Hey, this is actually a nice and easy way of modifying or extending Spinnaker’s functionality.”

The idea here is to—there’s kind of a few ideas here with this project, because we really want to lower the barrier of entry for users, particularly in enterprises, to be able to create extensions for Spinnaker. The method we chose to do this at first was a way that most users interact with Spinnaker, and that’s creating pipelines and defining stages.

So, let’s say an enterprise has some custom infrastructure at their business running their data centers, running their cloud. If they want to, let’s say, interact with that in a native—in a Spinnaker native way, you would want to have a stage that their users can just choose. So, rather than having to learn how Spinnaker works on a very deep level, you can implement this nice, simple interface and get it going.

So, at the end of the talk, it would be great for the audience to have an understanding of how to create and implement their plugin on their own.

Ashley: Oh, excellent. Well, then, hopefully, you can, you’re creating an audience and a community to go out and try your new plug-in system and the variations of it.

Now, you talked about using plug-ins as extensions to Spinnaker. Do you also see this plug-in system being kind of an integration interface to third party tools, maybe other open source, other commercial tools, or is it more things that just extend the functionality of Spinnaker?

Motevasselani: Oh, that’s a really good question. So, there is a push to move Spinnaker to be a lean core and an expansive ecosystem, I believe, is the term being used, and this goes directly towards that goal. There’s certain functionality in Spinnaker, the core, Spinnaker core, that doesn’t quite make sense.

For instance, there’s some stages that are specific to companies. Don’t wanna call Gremlin out, I think they do great work. There’s a Gremlin stage in Spinnaker that should probably be a plug-in. So, hey, Gremlin guys, hit me up. [Laughter] But not quite yet, maybe for the end plug-in system.

Ashley: It’s that evolution of software, right?

Motevasselani: Exactly.

Ashley: I mean, that’s how you get to this point where you need to do that, you need to spin things out, so it’s part of its growth.

Motevasselani: That’s right, that’s right. Yes. It would be really cool if you could have a Spinnaker installation where you can essentially pick and choose which cloud providers you want to have installed and running. If you only—you know, if you interact with, let’s say, both Google and Amazon’s cloud, maybe you don’t need Oracle’s cloud software to be on your cloud driver. Or maybe you want to have a cloud driver in each of those data centers—sorry, in each of those cloud accounts that only has the code necessary to deploy to that cloud environment.

Ashley: Mm-hmm.

Motevasselani: So, then you have a smaller attack surface, just leaner and smaller binaries, et cetera, et cetera.

Ashley: Great. Well, I think, unless there’s anything you wanna add, I think you’ve given a really comprehensive look to kinda what that talk, both talks are gonna be about.

Motevasselani: Yeah, that’s—you know, that’s hopefully, just getting the community excited and get some ideas flowing, I think, would be great with those talks.

Ashley: Well, and I also wanted to just throw in my little thanks just to appreciate what you do and all the other contributors to the projects in software. And I always encourage people who are just using open source, and there are folks at the Spinnaker Summit who are implementing it, not writing code for Spinnaker, but talking about it at a summit like this is another way to contribute to the community. So, great to talk with you who are actually writing code as well.

Motevasselani: Yeah, thanks. Nice talking with you, too, Mitch.

Ashley: Great. Well, it’s been my pleasure. You’ve listened to another DevOps Chat podcast. I’d like to thank today’s guest, it’s Cameron Motevasselani, software engineer at Armory, and he is speaking at Spinnaker Summit 2019. The dates for the conference are November 15th through 19th in San Diego, and Cameron has two talks. His talk on the plugins is Saturday, November 16th, at 2:30 p.m. And then he’s also talking bright and early about secrets management—boy, you’re really gonna wake ‘em up here—Sunday morning at 9 a.m., Cameron, so. [Laughter]

Motevasselani: [Laughter]

Ashley: So, you know, maybe you serve mimosas at the morning talk. [Laughter] So, it’s been my pleasure to have Cameron on, and thank you to all of our listeners today for joining us. This is Mitch Ashley with DevOps.com. You’ve listened to another DevOps Chat podcast. Be careful out there.

— Mitchell Ashley