Performance testing historically has been a challenge for most IT organizations. It is highly misunderstood, from executives down to the few resources allocated to perform an extremely difficult task. Most companies consider performance testing expensive and unnecessary and simply refuse to acknowledge it is a critical component of the software development life cycle. Have we forgotten about major performance failures such as Healthcare.gov? It is time companies start to take performance engineering seriously.
IT organizations using the Waterfall methodology relied on a few performance testers to develop performance scripts and run them after most of the testing had been completed and the application was a few weeks from a production deadline. If the performance team was lucky, they would get in one or two successful tests, typically working off-hours and executing performance tests on small, patched-together ancient hardware. The problem with this approach is when issues are found, there is not enough time to fix them adequately and performance degrades over time.
Developing a Performance Engineering Strategy
Here are some items to consider when incorporating performance engineering into DevOps:
- Learn—It is important to learn what performance engineering is and how it can be incorporated into your organization. If you don’t have the right skills within your organization, go find someone who can help. There are many organizations that can help with an initial assessment of where things stand.
- Develop Your Strategy—This is perhaps were most teams fail. Upfront planning is critical and it will save you in the end. It is important to develop a performance engineering road map. This is not something that will be built in a few weeks; a solid performance road map could take 18-24 months, so be patient and build it right.
- Build Your Baseline—It is important that you build a baseline of your performance tests so that you can determine if you have successful results. Performance testing is different from functional or automation because you have to consider the performance run as a whole. A few high transactions usually are acceptable. It is a good idea to build a history of runs to see how the application performance is trending.
- Identify Key Business Workflows—It is too time-consuming to build hundreds of performance tests, so focus on those that are the most critical to your business. Typically those workflows will account for the majority of your performance load.
- Test Upfront—With DevOps you need to start early and you will need to run performance tests frequently. The more you can run upfront, the more performance issues will be avoided in production.
- Tools and Hardware—Performance engineering requires tools and hardware. It is important that your assessment evaluates what you currently have and what you will need to do it properly. There are many open-source tools on the market today that can help and other tools that are making a big difference in the performance space.
- Budget—Developing a world-class performance engineering practice is going to take money. When you develop your performance road map, you might need two to three budget cycles to get the tools and infrastructure you need.
- Load Performance Environment—Yes, you will need to have a performance environment that is scaled to match your production environment. If there isn’t enough budget, then you will have to know how much smaller the environment is so that you can be confident in your performance results.
- Data—It is important to have data that is similar size to your production environment. This will help you understand how your system will handle performance within the production environment.
Incorporating performance engineering within DevOps will require work, but the rewards will certainly put the organizations that embrace it at the top of the list. DevOps enables teams between Dev and Ops to produce a stable application that performs exceptionally well under load. Here are some benefits of DevOps performance engineering:
- Predictable production performance results
- Production deployment confidence
- Eliminate catastrophic performance failures
- Increased performance efficiency
DevOps gives organizations a real opportunity to get performance engineering right; unfortunately, most organizations will carry over the same mistakes into DevOps. I hope you will take the opportunity to embrace the performance engineering challenge!
About the Author / Ron Wilson
Ron Wilson is a passionate champion of Agile, DevOps and Quality Engineering. He has managed large global Quality Engineering teams across multiple industries. He is currently the director of Quality Engineering at Arch MI. He regularly shares insights through social media where you can find him at www.qarevolution.com and on LinkedIn.