DevOps Chat: From 0 to 1,000 Deploys Per Month with Redbox

Transcript

Mitch Ashley: Hi, everyone. This is Mitch Ashley with DevOps.com, and you’re listening to another DevOps Chat podcast. Today, I’m joined by Joel Vasallo who is manager of Cloud DevOps at Redbox. We know Redbox is the movies, the kiosk, the online service—great.

Our topic today is going from zero to thousands of deploys a month. Boy, if you’re not doing a thousand a month already, that sounds like a great place to be. This is actually a preview of a talk Joel is doing at the Spinnaker Summit on “Transforming Software Delivery Using Spinnaker.” It’s 12:30 p.m. Sunday, November 17th in San Diego. Joe, welcome to DevOps Chat.

Joel Vasallo: Hey, thanks so much for having me, Mitch. I’m glad to be here.

Ashley: Glad to have you here. Well, tell us a little bit about yourself, what you do at Redbox and also, hopefully people know Redbox, but just a bit about Redbox.

Vasallo: Yeah, no, my name is Joel Vasallo, I’m the Manager of Cloud DevOps at Redbox involved with software delivery to the cloud. In my spare time, I’m a GDG organizer, so a Google Developers Group here in Chicago. But about Redbox, I mean, you said a lot of people know what Redbox is. We’re the kiosk that’s usually found in your grocery store. So, we do DVDs and Blu-rays, but we also, a little known fact is, we do have an on demand platform, so video on demand, and that’s kind of our new thing that we’re doing right now. And there’s actually a lot of motivation around building fast deployments, and that’s kind of gonna be my drive for the talk.

Ashley: Awesome. Now, just a question for you for cloud DevOps—does cloud actually include software that’s being delivered or information being delivered out to those kiosks, too, or is this really for the on demand platform in the cloud that you—

Vasallo: Yeah, there’s some aspects of it as well. I mean, there’s a lot of APIs in use across the company, so there are some that do kind of traverse the cloud in that regard—yeah.

Ashley: The only thing better than APIs are common APIs, right?

Vasallo: [Laughter] Exactly.

Ashley: Reuse.

Vasallo: Yeah, especially if you have a solid delivery platform—of course.

Ashley: Absolutely delivery platforms. Okay, so, let’s jump into it. We’re giving a preview of your talk, so there’ll be a lot more content, of course, in your actual talk. Tell us a little bit about, if you’re already achieved this thousands of deploys, you’ve been at this for a while. Tell us a little bit about how you got started on this journey.

Vasallo: Yeah. You know, to preface a bit, I mean, I’ve been in the Spinnaker community since, like, late 2015, when it was still kind of an early release out of the open—in the open source community. And at a previous role at Gogo, I used to work at Gogo and we implemented Spinnaker basically from day one. And prior to that, we were using some other open source—Netflix open source tooling as well.

But in that journey, we were going from a traditional monolithic, you know, to a microservices architecture, and the goal was to essentially build a way to deploy to the cloud as fast as possible, right? And just taking and building on those practices, you know, applying all the things, the same—the 12 factor concepts, immutable infrastructure, routing, all that stuff.

It kind of transformed into some open source tooling that we’ve written in my history, and there’s also some practices when using Spinnaker to make such a big change in an organization, right? I mean, the whole topic of DevOps in an organization, there’s so many great talks out there, right?

Ashley: Mm-hmm.

Vasallo: And that’s definitely one of the—I guess, arguably, the hardest bit is, “Well, what does the DevOps mean for you?” Right? That’s the typical phrase you hear when people say, “Well, we’re DevOps.” “Well, what does that mean?” Right?

Ashley: Well, now, you’re talking, obviously, at the Spinnaker Summit, and it sounds like you’re pretty committed to the product. This isn’t a podcast about that specifically, but you sound pretty dedicated to it. Why Spinnaker?

Vasallo: You know, it’s—the core fact is, it kind of extends a little bit past just deployments. It has—it’s really, the biggest thing is, it’s so extensible. Right out of the box, you’re just coming out swinging. The fact that Netflix, Google, Microsoft, a lot of big names—Amazon as well, how could I forget?—are contributing and helping build this product up, you have some pretty big names out there, right? And why not lean on the shoulders of giants. They’ve seen, I would argue, probably one of the bigger challenges in terms of software delivery, especially at an in demand space such as, you know, for example, Netflix, right?

Ashley: Mm-hmm.

Vasallo: And building on that success, you know, you basically get those batteries included, right? You get some battle tested software, and I think that’s one of the biggest things.

But in addition to that, I mean, it’s extensible. It covers Amazon very well, it covers Kubernetes, it has support for various deployment types and various customizations as well, some of which are customizations you need to build, and some customizations are actually built into the platform. And the best part, it’s open source, so that’s my favorite thing.

Ashley: One of the key words I didn’t mention that I know your talk’s gonna be about is automated deploys, right? You’re not doing hundreds of thousands with a lot of manual process. Talk a little bit about the key of automation and you kind of assume a development team is gonna be all about—yeah, let’s automate this and make our lives easier, but maybe it’s not that easy to get from point A to B where you do have that automated. Talk about that part of the journey.

Vasallo: Yeah, no, I agree with you. And I mean, it’s an unfortunate thing in the landscape right now where a lot of companies are rushing into the DevOps landscape, right, and they just say Jenkins, right—Travis CI, GitLab, whatever. And, you know, you’re essentially flooding tools on the market, right? And it’s very tough.

But having a lot of tools doesn’t mean you’re DevOps, right? There’s some fundamental business changes that have to occur in order to achieve that delivery that we’re talking about, right?

Ashley: Mm-hmm, yeah.

Vasallo: So, with that, it’s kind of taking—how do I call it?—the little infinity loop of DevOps, right? Where that’s great and, you know, the plan, code, build, release—you know, that little cycle, it’s great, but you can’t use that as a guiding principle. There’s not enough detail, right? And then you can have the extreme on the other side, where it’s like the Candyland, I like to call it, where it’s like the Candyland of ITIL process where you’re just essentially passing Go and then passing down a ladder, if conditions. That’s too complex, and that’s hard to explain. So, there has to be something kind of in between.

And that’s kind of what I’m gonna be focusing on in my talk and really asking the questions of how do you lay out an effective delivery process? What are some steps—what are the steps at your organization to get changes out the door? Because it’s a different answer for every company, right? There’s no one truth out there, unfortunate as it sounds.

Ashley: Mm-hmm. Well, and everybody’s environment is unique. Maybe some of the problems are the same, but there’s also, you know, what is unique about your organization, whether it’s the business technology or just people, the team you’re working with has strengths in certain areas.

Vasallo: Yeah, of course. And that’s something that I kind of build on in the talk is about building minimum viable products, right? And the argument is, many minimum viable pipelines, right? Really laying out, end to end, what needs to be done to get software delivery to occur. And the best way to start is really to look at what’s kinda currently happening, right? Lay out, you know, change has to come in, goes through phase two, phase three, phase four, and then ultimately phase five—profit, right?

Ashley: Mm-hmm.

Vasallo: Well, what are those phases and how much investment should be done in each spot, right? You can spend your time automating, I don’t know, some aspects of your pipeline, but what’s the return on investment, right? Would it have been better served in, for example, automating quality versus automating security scans, right? What’s gonna be the most immediate value in shaving time and getting you back from, you know, these long, drawn out pipelines to potentially fast and efficient ones, right?

Ashley: It’s interesting. What was sort of the starting problem that took you down this path? Was the business asking you, “Hey, we need stuff faster, faster, faster”?

Vasallo: Yeah.

Ashley: Or was it just your own, “We don’t have resources, so let’s figure out how to automate it”? I mean, what kinda got you going on this?

Vasallo: Yeah. So, a little bit about myself, I joined Redbox back in 2018 with the mission of building a software delivery pipeline to get software—me and my team to get software into the cloud. And with that, there’s also the transformation aspect of, “Well, what is the cloud? How do we integrate with the cloud?” And a lot of questions kinda came up.

Fortunately, we took a big investment in doing a cloud native approach, you know, from day one. Which means no lift and shift, which is awesome. It’s arguably the hardest path, [Laughter] as we probably all know, but this means essentially from the ground up, building a software delivery platform, right, that’s integrated from the, all the way down to the development layer and all the way up to the automation layer. So, we’ve been very fortunate to work on something like that.

That’s kind of where we started, and that was the motivation of building a truly cloud native pipeline, so, yeah, working on that stuff.

Ashley: Now, you mentioned you’ve developed some tools, too, and with Spinnaker being open source, have you contributed a lot back to the community yourself, or are you more of a consumer of it? You know, everybody plays a different role, and not everybody’s, you know, having an open source project.

Vasallo: I’ll selfishly say that I’m more of a consumer of it.

Ashley: That’s okay.

Vasallo: I would love to contribute more to it. I’ve done a few contributions in my past, but in recent times, I haven’t.

With that being said, though, my team is empowered to make change and drive changes, so we do have one or two changes in flight, and one of them actually got merged this past week. So, we’re looking to help grow and not just be consumers. We’re hoping to also build and solve the same problems we’re seeing, maybe, for other people as well.

Ashley: Well, good. That’s great if you get a chance to do that. Keep in mind, you know, exercising it, especially at the volume you’re talking about, that is super valuable, too. That’s what helps improve software, whether it’s open source or paid for. So, that’s another—

Vasallo: Well, yeah, and I mean, it’s kinda like what you’re saying. If you’re not paying for the software, maybe you can pay for it within support in the sense that Spinnaker’s community is an open community. There’s a Slack channel out there and you can join—anyone’s more than welcome to join, ask questions, and get answers from potentially some high up people. And Netflix, even people in the community in general, right?

We’re all on the same journey, and I mean, that’s arguably what kinda kept me involved in the Spinnaker community is just seeing such an open and collaborative group of individuals just willing to help each other—special interest groups every week or so meeting and just saying, “Hey, what’s the direction of the project? What else can we do? What are some challenges we’re seeing?” It’s actually been pretty fun.

Ashley: What are some of the objections you typically see or maybe that you even ran into about automating and doing deploys at that volume?

Vasallo: Yeah. So, it’s not like we’re doing 1,000 more changes a month, right? It’s arguably, running fast can initially be seen as a reckless approach, right? You know, if you said, “Hey, we’re averaging 10 deployments a month”—maybe that’s a standard organization—“to 1,000,” you’re gonna definitely open up some eyeballs, right? People will be like, “Wait, wait, wait. What do we need that for?”

Ashley: “What about stability? What about up time? What about”—

Vasallo: Exactly, exactly. But the argument is, it could be said that smaller changes typically result in less down time, right? If you can break down a purchase path change in some company, right, and maybe saying, “Hey, we’re gonna make a UI change and then we’re gonna do a back end change that’s backwards compatible and we’re gonna do, then, an API tier change to support that change such that everything hooks together,” you then broke up one large change of three major components into three separate small, individual pieces that can be rolled out safely, right? And tested and embedded as well. That’s the most important part. Because you’re only as good as your automation and testing as well.

Ashley: Now, did you do much shift left in terms of doing more security testing or vulnerability testing earlier in the Dev cycle, too, as part of this?

Vasallo: Well, so, that’s the beauty of it. I mean, essentially, the pipeline doesn’t, at least for us, doesn’t start at Jenkins or the build time. It starts all the way at the commit. So, we have a lot of tooling, at least at Redbox, that’s built to empower developers. The thing to also note about those 1,000 deployments—those deployments aren’t, you know, some Ops team clicking an OK button. They are developers promoting their own code, and they are empowered to write and deploy code to production when they feel it’s appropriate to release, right?

But in doing that, we don’t just give them the keys to the city, right? We build a nice, safe, and reliable pipeline—essentially, a paved road, if you will. That’s the kind of term I’ve been hearing it described where, you know, you can ride the paved road and have fun. You can get off road and it’ll be bumpy, but the majority of the time, the paved road will get you to production as fast as possible.

Ashley: Right. Yeah, that paved road concept comes from the Netflix paved road talk that they gave. I think that was OSCON or something.

Vasallo: Yeah, there are definitely a lot of great things from the engineers there, a lot of inspirational things kinda came out of it. I mean, again, it’s building on the shoulders of these giants, right? They’ve seen, arguable, large percents of Internet traffic, right? And at that level of success, you know that some of those concepts definitely can be applied elsewhere, right?

Ashley: Mm-hmm. Well, you know, one of the things that’s maybe a little bit misleading in saying automated deploys is, it isn’t just the deployment process that’s automated, is it? Talk about what all the parts that are automated to be able to deploy at that volume.

Vasallo: Yeah, no—definitely. I mean, it’s essentially part of the—part of the talk is also gonna, the little secret of it is, it’s not like you can roll in Spinnaker and have an automated deploy and you’ll be good, right? There’s a lot of aspects of a pipeline, and as I was alluding to from a commit layer, right, whether you do your static analysis for security early on, whether you do your unit tests at build time, whether you have an automated way of storing in archiving artifacts, right? You need to know what is making it out to your environment stage prod.

In addition to that, you need to build that infrastructure automatically, right? So, we have things—you know, you can use things like Terraform and CloudFormation as well, right?

Ashley: Mm-hmm.

Vasallo: But your load balancers, your security groups—again, we’re talking within the context of AWS—how do you reliably and repeatedly build your infrastructure in such a way that it can support that rapid deployment, right? You can’t just have an automated pipeline where it’s like, “Okay, hold the lights, let’s go call up somebody and say, ‘Hey, we need a load balancer. Can we get that done?’ ‘Sure, give me five minutes.’” Even five minutes is too slow, right? We need to have an automated way of creating and also getting fast feedback to our developers when these changes work and when they don’t work, right, so they can help triage and figure out what’s going on.

Ashley: It’s almost like I’ve said before—DevOps is an overnight success 35 years in the making. [Laughter]

Vasallo: [Laughter] That’s a good one, that’s really good.

Ashley: Where you kinda get there one step at a time, but it is, and I think that’ll be something interesting to anticipate about your talk is really thinking about it holistically and what that entire process—not that you’re gonna go into every bit of detail in that amount of time, but it is about all of that and connecting the toolchain, the work flow, and the process, people process part of it into something that can be automated and happen at high scale.

Vasallo: Yeah, that’s definitely something to be said there. I mean, a lot of—it’s generally a soft topic, but it’s arguably one of the more important ones is, there has to be a cultural transformation and understanding if you truly wanna be as successful as some of these giants, right?

Building the concepts of open culture, right, working together, cross functional teams—there’s no magical fix out there in the world to fix a broken process in a broken culture, right? And that’s the toughest thing. And then that’s something, definitely, we touch on as well. And I alluded to it earlier with the concepts of, for us, that’s empowering our developers. So, building compelling tools for our developers to use every day they come to work.

Ashley: Well, you mentioned keys to the kingdom for the developers. I can’t imagine you did all this without some form of senior leadership backing. How did you get that and where did you get it from to go down this path?

Vasallo: Yeah. So, it definitely started very grassroots, right? Say the challenge to get to the cloud, right? That’s an easy challenge and I think a lot of—sorry, that’s an easy statement to say, but what does that mean? And for every organization, it’s different, right? For some, lift and shift is the most appropriate solution if you’re not willing to invest in the cultural changes, right?

But for me, at least in my experience, it’s always been finding the hero in the regards of not necessarily a hero mentality, but finding that one champion who can help drive that, you know? That team who is under a tight deadline and you’re like, “Hey, you know, you’re a relatively new project. You’ve never really seen—you’ve never really gotten deployed. How about we just take an experiment and see if we can automate your app end to end, right?”

Ashley: Mm-hmm, mm-hmm.

Vasallo: The good news is, it’s a pretty low risk, it’s a new feature. The timeline, obviously, is always a concern, right, in terms of meeting business deliverables. But in doing that, working with those individuals, you not only can get fast feedback as to the proposal of the pipeline, you can also build a relationship that essentially is like DevOps and Devs working together to kind of create this delivery solution, if you will.

Ashley: Hmm. Well, I feel like we just scratched the surface, which is probably a good thing, because it’s the preview of your talk. I appreciate you coming on the podcast today, Joel.

Vasallo: No—no worries. Hey, thanks so much for having me. Again, I hope to see everyone at the Spinnaker Summit. It’s gonna be a great time, so I hope to see everyone there.

Ashley: It is. Well, definitely, I’d like to thank Joel Vasallo, Manager of Cloud DevOps at Redbox for joining us today. Again, Joel’s talk at Spinnaker Summit, which is at 12:30 p.m. Sunday, November 17th in San Francisco, it’s titled “Transforming Software Delivery using Spinnaker,” so be sure and check that out if you’re making it. Now you have a reason to go to San Diego in the fall coming on the winter.

Vasallo: Other than the great tacos, yeah. And everyone, join us on the Spinnaker Slack, we’re happy to help out and answer any questions as well. We’re more than a welcoming community, so hope to see more folks there.

Ashley: That’s a great point. It’s a very vibrant community. So, we wish you all the best in your talk, and everyone, you’ve listened to another DevOps Chat podcast. I’d like to thank you, our listeners, for joining us today. This is Mitch Ashley with DevOps.com and you’ve listened to another DevOps Chat. Be careful out there.

— Mitchell Ashley