Continuous Testing Success at ExxonMobil

Safety, security and controls are the foundations upon which 135-year-old ExxonMobil is built. As such, quality has become a key component in delivering software. The company is currently undergoing a shift from traditional project management organizations and support organizations to product teams, which has highlighted the need for testing early in the development cycle and to combine new development with ongoing maintenance.

Testing, the team “in the middle,” has become an integral component of the ExxonMobil development life cycle as “Quality Engineering.”

To do this, ExxonMobil started with the basic idea—quality control manages risk—and then explored what is tracked today and whether quality is being measured. Another key activity was to remove the roadblock to continuous testing.

Ann Lewis from ExxonMobil discusses in this video how the Quality Engineering team has created the expectation that testing is a part of everyone’s mindset, as well as the company’s view that continuous testing and deployment can only be accomplished through model-based testing and that managing quality reduces overall cost of ownership.

The full transcript of her discussion is below:

“At ExxonMobil, we have presence in six continents, so that means we get to deal with a lot of people. There’s over 72,000 employees plus another 20 percent contract staff, and that’s a problem. Guess what they want to do? Manual testing. Yeah, because they’re not gonna give it up. We have been on an adventure this last year that I’m gonna talk about and explain to you how we help train these 72,000 plus people that there’s a better way to do it.

“In addition to the numbers of people that we have, the upstream, I always … Well, someone referred to it about sticking a straw in the ground. Yeah, it was actually somebody from Tricentis. Yeah. I was just like, ‘There’s a little bit more to it than that.’ No, we do produce all of the product from under the ground. We get it out, and then we send it downstream to our refineries so they can do something with it like sell it. We also have a chemical company, that … I think right now are second or third to DuPont. We have a lot of product and a lot of things to do with it. That’s a problem, too. Because when it comes to the applications that test, that need to be tested, we have something like 5,000 applications that run our business all technologies. We’ll talk a little bit about that as we go through, too.

“The title is The Middle Child Rises. Why did I choose this? Because in the world of Dev and Ops, test is in the middle.

“We have a few other friends in the middle, and they are a little bit like a middle child, so we’ll talk about that. For today, your takeaways are that our organizational design has shifted. We’re going to explain why and how that happened. We will explain that Tosca is strategic for ExxonMobil and we will explain why that is.

“Tosca is an enabler to speedy delivery.

“You’ve heard about that this morning from many of our Tricentis friends, and it is true and I’m here to share that story. We now have within ExxonMobil, a quality philosophy, yeah, and it’s a lot more than just testing. Yes indeed, it is. We’re going to take a look at what we did and have a lot of fun stories in between.

“Being the middle child, you really didn’t get as much attention. How many are middle children, by the way? A few, okay. Do you have babies in here? Yeah, yeah, you kind of get away with everything, don’t you? The oldest, I have a favorite, I know. Oh, there’s another one out there of my fave, okay, cool.

“It’s very interesting in family relationships how these children get along. The oldest kind of sets the stage for everybody. That middle one comes along and it’s like, ‘Oh, it’s the second kid.’ The third one comes along and it’s like, ‘Oh, it’s the baby, it’s the baby,’ and the middle child just kind of sneaks through, okay? They don’t get quite as much attention. We’re going to talk about the middle child as they relate to testing.

“Typically, it’s the middle child that ended up with rules. That oldest child was all about, if you just tell them once, they usually would stop doing it. That’s the case I had. Then the second one was born, that turned out to be the baby, but I swear he’s got components of being the baby and the middle child. It was like, ‘Adam, we have rules in this household. Yes, we have rules.’ He was like, ‘I just don’t get the same respect as that older sister, Jessica.’

“Now within the corporate world, these middle children, you can think of them as being security and controls, data migration, data conversion. It’s like, ‘Oh, yeah, we forgot about you.’ Last but not least, testing. How many times has it been when suddenly that project shows up, ‘Oh, yeah, I think I might need to engage your team to help me organize my testing.’ Yes, yes, those of us that live in the testing world.

“The big deal right now, though, is that these middle children, our testing organizations, our quality assurance organizations, are becoming the key enabler, like we heard this morning, 50 percent of our problem is with testing. Oh, I could do that but I can’t get testers. I could do that but, ugh, we’ve got to do the testing. It takes so long.

“According to the World Quality Report, the reliance on manual testing is the top technical challenge in app development.

“How many of you knew about the World Quality Report? Yes? Yes? Good, good, good. We’re all the same kind of people, yeah. I like that.

“Businesses see test automation as a way to address one of their top challenges to delivering better software faster, by adopting Agile, DevOps and continuous delivery practices to shorten delivery cycles.” – Forrester

“By 2020, DevOps initiatives will cause 50% of enterprises to implement continuous testing using frameworks.” – Gartner

“The need to support faster time to market with higher quality is driving the demand for effective functional test automation tools. This is why we’re here. Tricentis Tosca, it is a framework. Yay! It’s not scripting. Boo!

“Tricentis Tosca will address that top technical challenge of delivering software faster.

“The need to support faster time to market with higher quality. It’s higher quality in the form of risk coverage. That is what we do with Tricentis Tosca. That is what we do with Tricentis Tosca.

From a framework perspective, the last year and a half we have been on a mission to explain the difference on automation tools. We have gone from record and replay, to test automation from a framework perspective using scripting, such as with the Hewlett-Packard products and Selenium. It is programming, it takes developer time to do automation when you’re talking about some of these historical tools. We have now moved into model-based, and I hope everyone in here has been to at least some of the Tricentis’ demos on what model-based is. I’m not going to go into detail here, but I will tell you that it has taken our automation from a 40-hour event down to a 8 to 16-hour event. No joke.

“I have been able to take Tricentis Tosca and put it into non-developer, normal people’s hands.

“I actually sat down with one of the Tricentis’ architects and did automation myself. I said, ‘Oh my gosh, I haven’t touched a keyboard to do anything technical in so long,’ and I actually did it. That is key, that is significant. That has been the story that has been going on with ExxonMobil for the past year and a half.

“It has become strategic for us. It’s automated screen scanning, it’s lower maintenance, it’s targeted testing. I’m doing it in half of the time, if not less. It’s faster, it’s easier to support, it’s talking about prevention versus detection. We have created a road map.

“What you’re seeing in our colorful TFS diagrams on the top, yes, we are doing our monitoring in TFS. You’re seeing a migration and better put a re-engineering of our existing scripts into Tosca. Re-engineering because for the last two years that I’ve been in this job, people kept telling me, ‘Ann, did you know that those existing Hewlett-Packard scripts really don’t test anything?’ ‘Well, gee, I don’t think I knew that.’ ‘Well, gee willikers, let’s see if it’s really true.’ Lord, have mercy. Twenty percent of those Hewlett-Packard scripts that were driven by the business to automate didn’t test anything. Well okay, they tested something but it wasn’t worthwhile.

“All of these little blue dots on your right beyond the solid blue vertical line are things that don’t really matter because we didn’t do a risk analysis. When I talk about re-engineering, we’re not just taking what we had and push it into Tosca, which you can do, by the way, there’s a tool for that. It’s up to you to use it, I’m not going to say anything. All I’m telling you is all our experience has been it made no sense.

“What we have found is when we sit down with our subject matter experts and we had this really cool dialogue like, ‘What do you do all day long?’ ‘Oh, I use VA-01 and NM-16 all day long.’ I said, ‘Cool. And so what happens when one of them doesn’t work? Dead in the water. You’re in customer service, aren’t you?’ ‘Yeah.’ ‘How many sales? Millions of dollars of sales do you do every day?’ ‘Well … ‘ This is the user, I mean they are totally into explaining to us what they do. Through this dialogue, this interview, we find out that they’re going to cost us $1 million a day, an hour, I don’t know what the big number is but it’s a big number, when it comes to customer-facing transactions. Therefore, those transactions are the ones that we want to automate.

“Now if I find out, no offense to anybody in controllers, that at the end of the month they have a report that they need to generate, ‘I’ve got some time to fix that one, okay?’ ‘How often do you do this? Monthly? Okay, that’s a one. What happens if it doesn’t work?’ ‘Well, I just did last month’s report and … ‘ ‘That’s a one.’ Low priority, move on.

“We have been re-engineering all of our existing tests.

“This word crypt is an ugly word. I actually I know the person who named our existing automations. Critical Integrated Process Testing. We don’t say that word anymore, by the way.

“The middle child has risen. This is my team. When I joined the group in February of 2015, we had about 25 people. About 25 people. Within ExxonMobil, I think there might have been seven of us. Then, we had some offshore group that did all of those monthly executions of those Hewlett-Packard scripts.

“Today, I have Vulcar over automation, he’s based in Hamburg. I have Cathy in Houston for functional testing and Santa Juan for functional testing, basically split between the two regions. They’re what we call test leads. They do a lot of the coordination of our testing activities. I have Aaron in Houston for performance testing. I have Jay, he is Doctor Jay, he is the man that knows everything about testing and everybody loves Doctor Jay. We have Suzanne right there, one of my oldest children. I didn’t say that, did I? I meant oldest child. Anyway, the other thing that was really funny about these pictures, they kept saying, ‘Oh, I want mine to be skinny.’ I want mine to be really upfront. We had a whole dialog on how the pictures turned out. It was pretty fun. Guess what? There’s close to 150 people now. That’s a signal that over this past year and a half, two years of getting this team put together, the emphasis that ExxonMobil has put on quality. One of the ways to get to quality was through testing.

“We are everywhere. I’m going to go into a little bit more detail now about organization because it’s good to understand where you are located. I was talking about being on the six continents. We have now placed people in region, so that when a person has a question on, ‘I want to automate, should I automate?’ We’ve got people in those locations to help with that. You will see we are using architects.

“Within the Tricentis nomenclature, you have architects, automation specialists and automation engineers. It is a tiering of skillset.

“We have found that the architects become those go-to people, when it comes to, ‘Should I be automating this or should I not and how do I go about it?’ Yes, we are using some offshore resources in India. Yes, we have people in Auckland. Yes, Bangkok, and that’s actually one of our ExxonMobil business support centers in Bangkok. Then, you can see Houston, Vienna, New York and Curitiba, Brazil.

“Organizationally, what we have found is that when it comes to implementing automation and doing it in a … I’ll call it conscientious and disciplined way, you’ve got to have a structure. Don’t think that you can just, I’ll say, run into doing automation. There needs to be a structure upfront to help you manage this work, because once one customer comes, you’re going to get five more. The enthusiasm level that we have experienced is unbelievable. We’re almost on tilt. But through this concerted effort of having an organization to help support it through a governance and then program management. Again, you can see this partnership that we have with Tricentis. Highly recommended when it’s time to get started.

“Successful team composition. It’s typically one architect to one engineer to 10 specialists and then you ramp up or ramp down, based upon if you’re working on systems of engagement. The shiny coin fun stuff, mobile, gooey, or your systems of record. They’re a little bit of a worn coin, all right? I’m sure some of you have systems that are quite old. We have ones that are really old so our financial system is still in IMS in PO1. Yes, rock on! That’s what I grew up with. Yeah, give me some JCL, too.

“See, geek, we’re all geeks, I know. It’s like once you join the world of IT, you’re just like into it. Nobody really understands what you talk about, unless you’re with people that are like you. Uh-huh (affirmative), all in the family, totally all in the family.

“Okay. We also have an immersion program. We have, again, the process of getting somebody into you’re approved, we think you can handle the work, all right? Going through automation specialists and all the way up to graduation. Again, working with Tricentis, we have come up with again, a disciplined process. You need this when you get started because they will come in out of the woodwork, I want to automate. I talked to you two years ago about automating those processes at the refinery and you said that no machine could do what you could do. I don’t say that, but that’s what I’m thinking when they come to me two years later with, ‘I think I want to do automation.’ I’m like, ‘Cool, let’s get into it.’

“We have a quality manifesto. Yes, we do. We will create forward-looking, value for our customers and for our organizations through quality approaches. Quality engineering is a team responsibility. I’ve got a great team, but we’re just here. We’re the messengers, we’re the trainers, we’re the coaches. Everybody that we work to, we talk about quality.

“How did you get to work today? Did you think about a safe way to get to work today with all of the rain? Yes. That’s a quality decision right there. Little things that we teach when we’re starting to push the quality manifesto out. You have to understand business function risk. Again, what do you do all day? What happens if it doesn’t work? Oh, really bad things happen, or maybe not so bad and a month in close, we’ve got a whole month to worry about it. Yeah, we can always close the books this month of last months’ numbers. I actually have a degree in accounting, I can talk about them.

“Including quality checks throughout the delivery life cycle. Quality gates. That’s another item that we have built into our system delivery life cycle. Starting to talk about at the moment requirements are written down, let’s start talking about what you’re going to test, based upon those requirements. We have built these quality gates, these questions throughout system delivery life cycle.

“What has happened? Our organizational design has shifted.

“Tosca is strategic for ExxonMobil. Tosca is an enabler to speedy delivery.

“Test automation brings value. We’re actually connecting the automation into total cost of ownership, as a way to reduce application total cost of ownership. We have a quality philosophy now and we’re just pretty darn excited about what we did. It took the rise of the middle child in order to make it happen. Thank you so much.”

— Jessica Wanivenhaus