The ability to fail and recover quickly and frequently roll out new features are DevOps benefits any development team can appreciate. But when the team is building a security application that must protect users in real time, the ability to update and upgrade the product quickly isn’t just a nice-to-have—it’s a requirement. Security applications require rapid development to respond to security issues and must be tested continuously to ensure quality.
By design, Cyxtera’s AppGate SDP is a highly technical product. A comprehensive software-defined perimeter solution, AppGate SDP is designed to secure any application, on any platform, in any location. The product dynamically creates a one-to-one network connection between the user and the resources they need. These sophisticated capabilities make development and testing a challenge.
Testing AppGate requires a complex infrastructure that includes the network, user desktops, the application itself and a variety of scenarios. To ensure product quality, everything must be tested, including all upgrade processes on all versions of clients.
Previously, the AppGate  team used an iterative approach to development, but the cycles were long. In some cases, developers had to wait up to two weeks for feedback on their build, making it difficult to maintain quality.
In an effort to reduce the time to feedback, the AppGate team adopted a DevOps approach. By automating testing, we sought to eliminate the barrier between developers and test results. Developers are most productive when you give them instant feedback. They are more eager to change their code when they’re excited about a feature they’ve just developed and want to see it succeed.
In addition to automating tests, we removed the QA team and created a single team of 20 developers who are responsible for coding, testing and implementing the feedback from tests and from customers. This organizational change created some cultural challenges.
Our developers needed a different attitude, and QA needed a different skill set to go from manual testing to development. That’s not a change you can do overnight, and sometimes it’s not even possible if a QA engineer has no ambitions to become a developer.
DevOps offers many advantages but can be challenging for organizations to embrace. For us, it took about a year of coding outside the product before developers began coding in the product itself. You need to invest time at the beginning to automate procedures, and that’s not always feasible in a normal project because development is all about quick gains and getting the product to market rapidly.
After about a year, our AppGate SDP team started to see speed and quality improvements in an order of magnitudes. The team of 20 has performed last year more than 2.5 million automated end-to-end product tests and created 560,000 virtual machines for testing alone. Those numbers are only possible with automation. If a security issue or a bug is identified in the field, the DevOps team can address it immediately. The team’s agility empowers them to respond to inbound customer requests as well.
An unforeseen benefit of moving to a DevOps model is the additional creativity the team enjoys.  Test automation requires strict rules and processes, but instead of being perceived as a restriction, developers believe the structure gives them more freedom. Developers can write anything they want as long as the tests, code reviews, security checks, etc., are passed. If the tests weren’t automated, we’d have procedures in place for sign-offs, etc., to make sure everything is done properly. Now developers feel like they have more freedom, and that drives creativity and productivity.
The new model also provides a safety net, allowing the team to be creative without fear of failure because they know they can recover quickly. They get instant feedback on product quality and feedback for rapid failure. It’s better to fail fast instead of over several weeks, when it’s much harder to fix.
The AppGate team has the luxury of using the product they build on a daily basis. When developers request access to an application or system, AppGate looks at the user and device profile to enable secure, context-based access. This serves as another layer of testing, and basically makes every developer and employee a test user. It brings a lot of knowledge to the team on how the overall product works (so it goes way beyond the code they have developed), providing a better knowledge base for future improvements.
Automating tests enables you to scale at an unseen level, but you’re always testing the same use case repeatedly, and that’s not realistic. Using our own product all the time provides a lot of exploratory testing feedback. We’re testing a variety of traffic patterns and different ways of using the product, and we have immediate feedback. This feedback, combined with rapid development, empowers us to deliver a high-quality product.
— Kurt Glazemakers
