DevOps Chats: Solving Complexity in Continuous Testing, with Sauce Labs

Transcript

Alan Shimel: Hey, everyone, it’s Alan Shimel for DevOps.com, and you’re listening to another DevOps Chat. Today’s DevOps Chat is meant to give you a little bit of a flavor, a preview for the upcoming SauceCon put on by Sauce Labs, which will be at the end of April. So, about a month and a half away, almost two months—well, a little less than two months away. It promises to be another great event from the folks at Sauce, and they have, really, just a packed lineup of speakers from a wide range of enterprises and startups and tech companies, manufacturing companies. I mean, just a whole gamut.

Anyway, speaking with me today about SauceCon, about maybe some new developments and trends in the testing market is Matt Wyman. Matt is chief product officer at Sauce. And Matt, welcome.

Matt Wyman: Thank you. Glad to be here.

Shimel: My pleasure. So, Matt, let’s get this out of the way first. You recently joined Sauce in, like, January, correct?

Wyman: I did, January 6th. I am a recent addition to Sauce Labs, and very excited to be here. I’ve been in the DevOps community for some time, worked as the head of product for CircleCI, so definitely a space that I know well.

Shimel: Sure

Wyman: But I see this massive opportunity and impact around test, and this fragmented space, and hopefully, we’ll get into this and talk about complexity a little bit. But it just really seems like a huge opportunity and a company that I love and a great reputation, so I’m excited to be here.

Shimel: Very cool. You know what’s interesting? I mean, of course, CircleCI is in and of itself a great company, too. We’ve—you know, I’ve done a couple podcasts with Rob, the CTO over there.

Wyman: Yep.

Shimel: And, you know, they recently, they raised a bunch of money, they’re really becoming a player in the CI/CD space. But it’s interesting that you moved from CI and CD to testing here, and you know, when we look at, let’s say, CloudBees, right, and Jenkins, the Jenkins founder, KK, you know—I’m not even try to pronounce his name; let’s just call him KK. But anyway, KK recently left CloudBees and the Jenkins community to start a new company also focused in the testing space.

So, I think your move here, KK’s move—I think it shows that, as the whole CI/CD, you know, software delivery pipeline type of industry and community matures, people are recognizing that testing is kinda where it’s at right now, right? That’s the new battleground. And maybe it is, as you mentioned, Matt, because you know, the mission has grown so much more complex than, you know, let’s say pre-DevOps, right?

Wyman: Absolutely.

Shimel: So, why don’t you—if you don’t mind, give me or give our audience an idea of where you see or how you see testing has become, you know, more complex.

Wyman: Absolutely. So, speaking to the CI/CD platforms just for a moment, orchestration in CI/CD has become a very complex problem, and that’s reflective of changes in the software landscape and how people build software, which itself is a reaction to where we started.

So, we had all these monoliths, these, you know, large systems, we’ve all talked about it, they still exist out there in the world. Most every system that says that they do microservices actually has a monolith somewhere, like, micro-monoliths all around.

When you’re in the monolith world, testing is all about end-to-end testing. You’re trying to put all the pieces together, validate that everything works top to bottom, and that’s really where quality and test have their roots. And as much as possible, it’s all about trying to automate those difficult processes and make it easier and more efficient to move through all your different test cases.

But as those entities split up, and as we started to recognize that each component of a larger application needed different types of tests, testing itself became more complicated. And furthermore, it became more than just clicking buttons. At one point, testing was about validation, that if you click a button, you go to the correct page. If you click a button, the right business logic executed.

But these days, it’s a lot more than that. It’s also validating that the CSS is correct, that the experience is performant and is wonderful and delightful. Companies that have bad user interfaces and bad user experiences don’t win.

And so, you have this testing world that, on the one hand, is trying to move faster, dealing with a more complex software landscape, and on the other hand, is now broadening in the types of tests that are being run, not just functional testing, but also, if you will, delight testing and trying to validate that the user experience, the customer experience is going to be an excellent experience at the end of the day

Shimel: Yeah.

Wyman: And so, that’s how I see this space splitting up.

Shimel: Absolutely. I don’t disagree with you there. You know, Matt, I almost wanna put myself in the shoes of the test professional and the QA person who, you know, maybe six or seven years ago was hearing that DevOps and this move, you know, agile to DevOps and beyond was gonna somehow make them obsolete. And then other people were saying, “Oh, no, it’s not gonna make you obsolete, it’s gonna free you up to do higher level kind of tasks that are more worthy of your talent and skill set.”

And I think that has come true, certainly. I think what we’ve seen with test automation has freed up testers to do higher level stuff, but I also think it shifted the—not the burden, but the emphasis on people who know how to set up automated testing, right? You’ve gotta know maybe a little bit of coding, a little bit of scripting, you gotta know some of these open source tests.

Wyman: There’s a bit of all of the above, don’t you think?

Shimel: Yeah, it is. That’s exactly it.

Wyman: The—and it’s further complicated by the fact that the number of devices and environments we’re testing on has also fragmented and increased. And you can’t run the same type of test in every environment, so most software applications have some sort of mobile component to them.

Shimel: Yeah.

Wyman: And yet, mobile testing is not automated in the same way that you would automate browser testing. And in fact, in some cases with mobile testing, it’s more effective to be manual and to go through it piece by piece, or to use visual testing to go through it.

And so, you have to pull together all these different techniques. I think there’s often this perception that, with any new technology or any new change, that it’s going to somehow upset the apple cart and suddenly everyone who has a job doesn’t. And it just doesn’t typically work out that way. I think what we’ve seen is just this increasing complexity, and then also the opportunity to use the right type of testing in the right place. Meanwhile, a lot of the people out there doing quality testing continue to do so, because the number of applications has only multiplied.

You know, the fact this whole digital transformation and all the buzz words that you hear about that—fundamentally, it’s true. Every business is now a software business, and one of my favorite examples recently is, where I live, there’s a highway called 101, a pretty big highway, and there’s a big billboard on the side that has a, it’s from Wells Fargo, and it has a mobile app on it. And the billboard is not advocating or talking about Wells Fargo’s core business, they’re not talking about loans or lending or savings accounts, it’s talking about their mobile app and the experience of their mobile app. And you better believe that they care about testing it, right?

And so, it’s more than just—again, I covered this earlier, it’s more than just feature function right now. You have—you need more people who need to look across all the different touch points and then utilize the right type of testing in each area. And so, there isn’t one test solution that solves all of it. You have to use different techniques, because ultimately, you’re managing business risk, and you have to do it all the way from the moment a developer checks in code all the way through to production and validate, if you will, a virtual user. Someone has to go validate that it’s actually working as expected when a customer reaches out to interact with the tool.

Shimel: Yeah, yeah. And, you know, Matt, you hit on a couple—you hit on web testing, mobile testing. I think one of the things, you know, thinking back to when I was running software companies, testing was always something that meant that we were a day late and a dollar short with, you know?

Wyman: Absolutely.

Shimel: It seemed like we never had enough budget to do all the testing we thought we should do, and we never had enough time to finish the testing at least we were going to do. But when I look at the kinds of testing and the amount of testing that goes in today, and all of it pre-deployment and everything—yeah, it’s grown more complex, but I think just beyond pure complexity, I mean, would you say we’re testing more now than we did before.

Wyman: Well, I think part of the answer there is to dig in on what does it mean to test. So, these days, it’s about risk management. And so, if we’re going to have developers able to check in code and 10 minutes later update production, then we’ll need mechanisms to ensure the production is safe. And typically, that’s done with feature flagging, it’s done with canaries. There’s all sorts of ways to ensure that each change doesn’t blow out production.

But along that path, it’s also about automating as many of the different possible negative outcomes as you can. And that is what we would typically call testing. So, there’s visual testing, there’s browser testing, there’s unit testing, there’s end to end testing, integration testing. All these are necessary, and you’re trying to compress them all into that 10 minute window, which—you know, most organizations can’t hit that kind of number, but that’s the gold mark if you can hit that 10 minutes from change to production.

And so, I believe that it’s a matter of thinking about risk and saying it’s not about quote-unquote test coverage with a go/no go meeting where you look at the test coverage—and frankly, there’s so much peer pressure to go that typically organizations do. It’s more about saying, “What is the possible negative outcome, and how do I ensure that I have appropriate coverage for each of those negative possibilities?” And in some cases, you’re going to build tests to simply block and fail that particular change, and in other cases, you’re going to provide monitoring and observability to ensure that, if something negative happens, you can pull it back quickly.

And if I go very meta for a moment and go well above even what Sauce Labs is doing as a platform, I would suggest that all of that is a part of the overall test or risk management perspective. And ultimately, I think our goal is to then create visibility across the whole thing, so you can manage your risk, manage your test all from one place.

Shimel: Got it. Makes sense. Now, Scott—Matt, excuse me—let’s talk a little bit about SauceCon. How is all of this manifesting itself, you think, at SauceCon?

Wyman: Well, one of the fun things for me is, this is gonna be my first SauceCon. So, I haven’t had the opportunity to see—

Shimel: Oh, you never went? Okay.

Wyman: I’ve never been, so, this is gonna be my first one. I mean, I’ve seen the lineup, it looks fantastic. I’m hoping to see not only a whole series of really good conversations within our existing community, but also, there are quite a few people that are coming who are Selenium, who are Appium users, developers and whatnot, to share in some of their experiences.

I know, as a product person, one of the conversations I’m very eager to have is in how we help people shift left. And so, it’s more than just running Selenium tests, it’s about the headless tests, it’s about getting earlier upstream so that we can provide a little bit more of a developer oriented experience. And so, I’m very excited to go have those conversations, and I think the lineup looks good to have it.

Shimel: I agree as well. So, I’ll tell you, my experience is, you know, if you’re looking for conferences around testing and how—like, what’s pushing the frontiers there, right, and also pulling in the open source testing community, SauceCon is one of the best. And I’ll tell you what I like about what Sauce Labs has done there is, you will see other testing companies represented, mentioned, participating, right? I remember, you know, CA who had bought BlazeMeter, for instance, was involved in SauceCon. I don’t know if BlazeMeter will be there this year or not, but they were there. I know the Tricentis folks also like to participate.

And part of the reason is because testing is such a big world with so many different facets today, you know, everyone kinda does—you know, they’re not necessarily head on competitors. A lot of this is sort of complementary. And so, it’s really nice to see the community come together. You see it a lot—and maybe this is something that is specific to DevOps in general, I don’t know, but you see it a lot at these DevOps events where, yeah, it’s Sauce Labs throwing the party, but the whole testing community partakes. And that, I think, to me is one of the things that makes SauceCon really special.

The other thing is, is just the speakers. I mean, you know, like in any vendor/user type conference, you’re gonna get the CEO, the CTO, and I’m sure Matt, you’ll do some presentation. But when you look at the speaker lineup here, you know, you have everything from, like, Google kind of engineers to startups to large enterprises, you’re really getting a good 360 degree view.

Wyman: What excites me is that the way in which we think about this event, we don’t see it as a sales event. We’re not sending salespeople there.

Shimel: Right.

Wyman: A number of my product folks will go, because we’re interested in having conversations. The test market overall is fragmented, but it’s fragmented for a good reason, which is, there are a lot of different use cases, a lot of different problems that people are trying to solve, and quite a few of those folks are going to be there, and so, it’s a good chance to have a conversation if you wanna do, if you wanna know more about visual testing, if you wanna know more about how to use headless effectively or deal with the grid.

So, there’s this opportunity to share that information and share that knowledge, and I think that’s really what we’re aiming for is an environment that’s a practitioner’s environment. It’s [Cross talk].

Shimel: Yeah, definitely, in years past—in years past, you know, SauceCon, that’s exactly what it’s been, and I think Sauce Labs has done a great job fostering that, right? It’s good stuff, all around.

We don’t have a lot of time left, but you know, you just joined in January, you’re fairly new, your first SauceCon, you’re getting a handle around product. But do you feel up to giving us a little projection, maybe, of what we’ll see coming out from Sauce Labs over the next, you know, six months?

Wyman: Well, I don’t wanna give away too much, but we will have something that I expect to be quite exciting, a new beta labs experience that I think people will be able to go get their hands on and play with, something that moves, let’s call it shift left, a bit.

And we’ll continue to talk a bit about our analytics and in particular, there’s a new report that we are going to be showing. I think some folks may be aware of it already, but it’s relatively new, it’s called Failure Analytics, where we’re able to aggregate all of the different log outputs and provide intelligence and cohorting of different common errors so that, instead of having to hunt through hundreds, potentially, of failures, we will highlight the common problems, which gives developers an easier time of going and fixing things. Also, it makes it a lot easier to debug across large automated test suites.

So, we’ve got quite a bit that we’ll be showing from a product standpoint, and like I said, there is gonna be a pretty exciting unveil around a beta project that we’re moving forward with.

Shimel: I think that’s enough of a tease right there. Hey, Matt, you may not know this, but I’ll mention it for our audience—anyone interested in going to SauceCon, it’s actually SauceCon, S-A-U-C-E-C-O-N dot com, and you can go on there and you can still register to attend. I should also mention that our MediaOps, DevOps.com team will be there under our Digital Anarchist platform. We will be broadcasting live interviews from SauceCon.

So, if anyone listening is going to be in Austin for the event—hey, come find the Digital Anarchist broadcast booth or me, and we’d love to hear what you have to say and maybe interview you. You never know. So, yeah, please do that, but check out SauceCon.com for more information.

Matt, before we wrap up, anything else you wanna add?

Wyman: I think this is great. I appreciate the opportunity to have the conversation.

Shimel: Well, now that you’re part of Sauce, you know, we’d like—we do these DevOps Chats pretty regularly, so maybe we can have, maybe we’ll do another one even before SauceCon itself.

I’d like to—you know, I didn’t wanna put you on the spot today, [Laughter] but maybe in the future, I’d like to hear, what are your top three sessions at SauceCon that you’re looking forward to. You don’t have to answer now, though, but I’m gonna give you a chance to do a little digging, but I’m gonna invite you back on in about two weeks and let’s talk about it.

Wyman: That sounds great. I think it’d be a good idea to, I assume that we’re at the end, here?

Shimel: Yeah, we’re wrapping up, and you know what, for our next one, Matt, I’m gonna pick three, too. So, folks, there you have it—stay tuned for our next chat with Matt where, between us, we’re gonna highlight his three and my three—

Wyman: That sounds like fun.

Shimel: Favorite talks for SauceCon that we’re looking forward to. Sound good?

Wyman: Alright, sounds great. I look forward to it.

Shimel: Alright! Hey, Matt, welcome aboard to Sauce Labs, I think—and you already know it, but you joined a great outfit, a great bunch of people there, and looking forward to seeing you in person at SauceCon.

Wyman: And likewise—thank you.

Shimel: Okay, and we’re looking forward to seeing you, our listeners, there at SauceCon, but if you can’t make it, do log onto digitalanarchist.live and you can follow along with us from there.

Until then, this is Alan Shimel for DevOps.com, and you’ve just listened to another DevOps Chats. Have a great day, everyone.

— Alan Shimel