Before the new year, I had a chance to sit down and chat with Huw Price, Vice President, Application Delivery, CA Technologies. Given Huw’s title what do you think we spoke about? You guessed it, automation. Actually there is more to it than that. Automation for automation’s sake alone is not necessarily a good thing. As Huw points out, smart automation is key.
Huw and I will be participating in a full webinar on the topic of Smart Automation and how to build it on January 24th. You can register for this free webinar here.
As usual with our DevOps Chats, below is a streaming audio player with our interview and below that is a transcript of our conversation. Enjoy!
Alan Shimel: Hi, everyone, it’s Alan Shimel, Editor in Chief at DevOps.com here for another DevOps chat. Today’s guest on DevOps Chat is Huw Price of CA Technologies. Huw, welcome to DevOps Chat.
Huw Price: Hi, Alan, nice to be here.
Alan Shimel: Nice to have you here, Huw. Huw, if you wouldn’t mind, if you could give our audience a little bit – I said you worked at CA, but what’s your title and what are you responsible for over at CA?
Huw Price: Yeah, I’m a VP at CA. I was the managing director of Grid Tools, which venture you may have heard of, where we specialized in all things test data, and laterally we moved into all things test case optimization and into test automation. My role at CA – I’m kind of an evangelist QA specialist practitioner, and it’s actually fitting quite well with the DevOps land of trying to improve quality whilst improving speed at the same time, which is a bit of a magic trick. But it can be done and it needs some new thinking.
I would say that over the years, it used to be cost was the top driver for businesses, but now due to the pressure of business, it’s all about competition and it’s all about getting stuff to market quickly, and you can’t have poor quality code going out to customers because they’ll just move to someone else. And I think that’s really what’s kind of driving a lot of our thinking at CA at the moment.
Alan Shimel: Absolutely, and it is somewhat counterintuitive when they tell you you could go faster but have higher quality. And I’ll even throw one more improbably into it – and be more secure by doing that.
Huw Price: Yeah, that’s an interesting area. I think actually what I’m finding now is that I’m a big fan of doing everything in parallel, the days of sequentially waiting for one team to hand stuff off to the other, and what we’re finding is things like performance testing and security testing are kind of job one, and you need to build those in almost as functional requirements as you’re designing your software really. That sort of hopping from one piece of functionality or requirement to another seems to have vanished. It’s all hands to the pump I think at the moment is what I’m finding out there.
Alan Shimel: It is a bit of a Chinese fire drill, as they say. But out of chaos comes order, and that’s kind of the good news. Working with you and your team we’re actually doing a webinar after the New Year, in January, and it’s titled “How To Build the Right Automation.” And you and I were talking a little bit offline, of course, and the idea that you just automate for automation’s sake, maybe that was a couple years ago, but now we’re trying to be smarter about designing what’s to be automated and how we do it. Can you share with our audience a little bit of maybe your insight into that?
Huw Price: Yeah, I think so. Automation’s got a bit of a bad press, and to be honest, I think rightly so, it’s considered very fragile. You only need a small change, a pop-up appearing on a screen, some kind of slight delay in a network and some of the automation breaks. And then everyone looks for the results and you spend days and days actually finding out that the program is fine or the functionality is fine, but it was actually the automation script that was wrong. That’s the first problem.
And the other one is the time it takes to build the automation in that it’s not uncommon for automation to be built three months after a sprint is finished, which of course means you’re automating something that doesn’t exist any more, which is not very continuous. And the value of what you’re actually doing is pretty poor. I think that is kind of the backdrop of where people have been.
They’ve moved towards test-driven development, behavior-driven development etcetera, but they still have the problem that actually maintaining the automation scripts, even if they’re key word or Gherkin, it’s still an issue. Because, you know, requirements don’t exist any more, just change requests exist, and a change request means you’ve got to go back through all of your existing automation and say, “Do I have to change all these automation scripts, which ones do I change, which ones are duplicates, which ones make no sense?” And that is a massive amount of work.
I’ll just give one example. There was a bank in Canada, and basically for every single requirement change, they would go through every automation script by hand to see if it needed to be changed. So one change request took one tester all day to see if it impacted any existing automation scripts. So that’s just one change request. So how many testers do you have, how many automation guys do you need, how many change requests can you fit through that very narrow pipe. I think that’s where people, even with some of the new techniques – and I’ve got to say that the automation frameworks are becoming better. They’re becoming more structured; they’re becoming more malleable key word based. But they still don’t really address the problem of if I make a change to a requirement how quickly can I update my automation framework.
Alan Shimel: Excellent. Huw, I want to talk to you a little bit about the go to market with your group. You’re obviously probably dealing with mostly large enterprises, how does a large enterprise realize they have this problem? Is the problem one where, “Geez, we need help automating,” and they haven’t even cut across the threshold of “we need to be smarter about automation” because they’re still at just the “we need automation.” Or is it organizations who have already, let’s say, dabbled with automation and are now realizing, “Wait a second, this is just a giant cluster waste of time.”
Huw Price: Yeah.
Alan Shimel: “And we need help being smarter about automation.”
Huw Price: I think the answer is yes to all of them. Even within the same organization – and I can think of one we’re dealing with at the moment – you get such a sort of disparate set of skill sets, capabilities, maturity models, and quite often we’ll go into a large company and introduce teams to each other and say, “Do you realize you have a really nice sort of key word driven framework sitting over here with a very clear object library, which you’ve got all this work and you’re going off and building yet another selenium framework, maybe you guys should talk to each other.” I just find that we do a lot of sort of introducing companies to other people in the same company.
Alan Shimel: Sort of like matchmaking, huh? Matchmaking all that stuff. Interesting.
Huw Price: I’d like to add on that, certainly what I do and members of my team and other people in the business are doing, we are genuinely hunting for best practice here. This is not about us selling software any more, the market is moving so quickly and this ability to be a sort of lightening rod of best practice amongst other companies, genuinely it’s quite fun and we’re enjoying it, and we do share around what we’re finding out in the market in terms of what’s good, what people are struggling with, and hopefully we’re bringing not just software to the market, it’s more some best practice thinking as well, and I think that’s quite interesting.
I’ve got to say that chatting with you when we met in Vegas, I learned a couple of things from you. And I find that when I talk to pretty much everyone. And I’m trying to feed that back, whether that’s by webinars or with the direction of how we build the software. We’re trying to just keep pushing the boundaries, keep communicating, keep thinking. It’s kind of fun. We’re really enjoying it at the moment.
Alan Shimel: Absolutely. I always tell people these are interesting times to be in the IT world, there’s just so much change happening – and change I think in a good way. New tools and trends and ways of doing things, it is fun, it really is fun.
Huw Price: I’m going to take a slight tangent here. I was listening to an agile coach talking and I agreed with 80 percent of what he said, and disagreed with 20 percent. But then I challenged him and said, “Can you tell quickly whether an agile team is good or bad?” And he said, “If they’re having fun, that’s generally good.” And I know everyone is into hero culture at the moment, charging around trying to get the release out the door at the last minute. That’s what we’re trying to focus on is move away from the hero culture and actually getting that fun back into – you know – solving challenging problems and automating as much as possible and joining stuff up and getting feedback loops going and all that kind of fun things. So it is, as you say, very interesting times.
Alan Shimel: Sure. Steve, let me ask you a – and I’m not being facetious but a real question – can we have too much automation?
Huw Price: Actually, that’s interesting. I did a presentation at Star West and I talked about risk based testing. It’s actually kind of touched on an interesting subject here. The reality is, if you have a very stable system and you’ve got a bunch of very experienced programmers, you know, you run it through a code quality engine and it’s very solid and stable, and they also understand all of the data that flows in and out of it – the reality is that actually you might need a very minimal amount of automation to test that.
If you’ve got something which is highly, highly active – I’ll mention an Indian company, they’ve got one API which brings in 150,000 new customers a day. That API, I mean, that is staggering. 150,000. This is to do with the 4G rollout across India. That API, you know, I would be automating the hell out of that. I would have so many automation scripts around that. I would have docker images spun up in every different direction, I would have visualization wrapped around it, I would have negative testing up the wazoo, so that if you make a change – performance testing, blaze meet up, whatever it takes, to make sure that that baby, when you make the slightest change, you don’t mess with it — because if it’s down for ten minutes, that’s 5,000 customers. So I think people maybe should start thinking a little more sort of holistically and higher up in terms of the different levels of risk associated with it. I think that’s one thing.
So I think that’s one thing, and then the other one is the speed at which you can automate. The reality is that what we do, we will generate – we’re very much into building a model of what the system should do. You could potentially build all your automation scripts before you’ve actually built the application. I was doing some work with Eggplant the other day and we actually generated a whole bunch of Eggplant scripts based off a Microsoft PowerPoint of what the system should do. So in other words, before the developers started, we’ve actually created the automation framework.
Now, if that takes 15 minutes, great. If it took 3 days, then it’s a little bit of a tradeoff. So I think this balancing things up based on risk, and also the technology that you have to hand, is probably the two criteria you should start thinking about.
Alan Shimel: Agreed, agreed. Steve, as I had told you offline, the time here goes incredibly quick and we’re coming up on our 15-minute outside limit. I just wanted to bring you back home to the webinar in wrapping up. I believe our webinar is scheduled for in the latter half of January, and it is called “How To Build the Right Automation.” For people who are listening right now, give me two or three points why they really should make it their business to attend the webinar.
Huw Price: I think firstly they’ve got to understand what they’re automating. If you look at most automation scripts, you’re probably only covering about five percent of the functionality. We would look to automatically generate 85 to 90 percent of the functionality of the system in either the same time or less time, so I think that’s firstly. To really understand coverage, I think that’s important.
The other one then is to think about that actually automation is not just about the scripts themselves, it’s about getting the right test data and the right – I’m going to use the phrase visualization, whether that’s service visualization or message visualization or sort of network visualization or visualization environments. So trying to create an environment such that your automation is actually testing the bit that you really want to test. If you think of an end to end test, to get to something like a share trade close where you might have 200 functional logic gates I need to test, is actually really difficult, because you’ve got to have all of the right bits around your automation to make sure that you’re testing the right stuff.
So better automation in terms of the design and also the assets around the automation to make sure that you’re actually testing the correct stuff. I think that’s probably what I’m going to focus on during the webinar.
Alan Shimel: Got it, got it. Well, obviously on DevOps.com we’ll be publishing links to the registration page for people who want to join us on that webinar. It sounds like it could be – it sounds like it could be a great one if folks are interested.
Huw Price: And it’s lots of demos. I’ll do a couple of slides, but I prefer to do demos.
Alan Shimel: Oh, that’s greatest.
Alan Shimel: Yeah, most techies love to see stuff.
Huw Price: Actually, we’ll drive right in here.
Alan Shimel: They’ll give you a slide, but anything beyond that they’re just like, “Show me stuff, show me stuff.”
Huw Price: All right, so let’s stay tuned to DevOps.com. I would imagine we’ll probably have the landing page up within the next week or two as we head into the holidays.
Alan Shimel: Well, Huw price, I’m going to ask you one last question – not a curveball or anything, but I usually ask a lot of our guests this – for our audience listening, one book that you would highly, highly recommend they read.
Huw Price: Actually there’s a book called Model Based Testing. What we’re discovering is that people are now moving pretty much – they’re realizing that without a model it’s very difficult to proceed. And there’s a book called, it’s actually an adjunct book to the Model Based Testing Course. We’re massing this out to our customers now to kind of reset their thinking. Everything we do in life is a model, everyone’s making decision gates, logic gates, risk based profiles, and this is going down really well. People are really taking the time to read it. What I’ll do is during the webinar I’ll put up the pages so people can go and get it. I can’t remember the exact title. It’s called Model Based Testing – The Course Guide, or something like that. But I highly recommend that.
Alan Shimel: Excellent. Huw Price, thank you so much for being our guest on this episode of DevOps Chats.
Huw Price: Oh, it’s a pleasure.
Alan Shimel: Thank you, it was our pleasure and we look forward to the upcoming webinar, “How to Build the Right Automation,” including demos, and we’ll have more information on that. And we hope to have you back on DevOps Chat soon, Huw.
Huw Price: Great, all the best. Thanks so much.
Alan Shimel: All right. This is Alan Shimel for DevOps.com and DevOps Chat. Thanks very much, everyone, and we’ll see you on the next DevOps Chats.
