How much time do you spend optimizing your delivery pipeline? My guess is none. Many teams have figured out great ways to automate their releases, and some even automate testing and monitoring. But they don’t often take the next step and learn from those results to refine the delivery processes.
Automated load testing and application monitoring are all the rage these days. I’m sure that this is the right path to better software, faster, and I am happy to see that many organizations are embracing automation tools. But many are not leveraging the huge opportunity these tools provide. They can do more than improve the quality of releases; the feedback they supply can improve the whole release process.
Since organizations usually do not consider their end-to-end process when they begin with delivery automation tools, they do not build their tool-chains to supply test results that will drive any meaningful change. And even if they spend some time with the results, it is usually not much more than looking at pretty graphs and pointing at warning flags. Don’t get me wrong; this is better then nothing, but it’s missing out on a huge opportunity. And if the organizations is ready to spend the time to install the agents, load tracking scripts, and run tests, they should be prepared to think about how this time can help improve the process, and ultimately, the product.
Application monitoring and load testing should not be considered the end of a release process. It is just the beginning. Over time, the results from multiple tests and releases in production will show how well your overall deployment process is working. And if it’s constructed thoughtfully, the results will start showing where the chinks might be.
Soon, teams can discover trends in releases and relate those trends to activities during the release cycle. For example, a really fast load test combined with application monitoring saying that certain pages are returning 404 means that you did not just dramatically improve the performance of your application— you actually just serve up error pages very fast. Or you might find that you are much less efficient, even when the product team increased the size of the backlog proportional to the new head count. Intuitive or counter-intuitive, these insights will be evidence based and easy to apply within your testing organization.
Thinking beyond individual releases
It’s hard not to fall into reified Release thinking, where entire teams live by the ebb and flow of individual release cycles. It might even be argued that with this mentality, a team is actually doing a really fast waterfall approach—not Agile, and for sure not DevOps.
While organizations leverage great technology to measure the quality of their applications in production, they also do so on a release-by-release basis. This does a lot to help the individual release, but does nothing to inform the team about the efficiency of the overall process.
DevOps attempts to not only improve the speed and quality of each release, but also build in relationships between all releases. Its kind of like meta-release management. And if an organization effectively watches the relationships between releases they can optimize their deployment pipeline, iterate over it, and improve it just like a piece of code.
The only way to effectively do this is by having a great collection of data, automated testing at the end of each release, and correlating all of the previous releases. But it is much more than pretty graphics and error page warnings. If you look at the data holistically you have a health of your processes as well as the individual releases.
Oh yeah … It takes planning
I know it is not easy. One of the biggest challenges of the fragmented development tool market is getting two tools to work together like one. It’s rare that a best of breed tool is built with other best of breed tools in mind. So it’s often up to the implementing organization to plan for their comprehensive view.
In the best case, it should be possible to track a cohort of all your releases for an entire year, and by the end of the year know exactly when your team/pipeline was the most effective at releasing software. Hopefully the trend shows improvement throughout the year, but it’s more likely that you will find some seasonality, correlation to the size of the backlog, or relationships to the types of functionality being released.
It is not something that can happen haphazardly; it needs to be thought out. More tools and more “DevOps” headcount won’t solve it alone. In fact, they may add more problems. For example, combining the results of multiple releases’ load tests might be easy within the single load test application. But it’s much harder once a second tool enters: e.g. it’s hard to correlate multiple load tests with time-series data from an application monitored in production. And, as is often the case, more headcount means more division of labor: one person takes responsibility for load testing, and another “owns” application monitoring. Even with great people and great tools, complexity can come out ahead of comprehensive and concise planning.
At the end of the day it’s important to also get smarter about the entire pipeline above and beyond individual releases. The results of application monitoring and load testing are a great place to start.
One element is totally out of your control. And that is how well the ISVs products decide to play together. Most ISVs are focused on their individual use cases, and only on integration with other tools as a business development effort. But it is possible.
You can do it today, just think ahead
For example a cool mobile back end load-testing tool called Blaze Meter is a very easy place to setup load tests, run them, and review their results. The output of the tests are typically associated with specific releases, and each tests results individualized. So you can go in and get a great idea about each specific test, but not all tests together.
Combine that with a popular application monitoring tool like the new Insights product from NewRelic. It takes their popular monitoring features further, allowing users to write application tests like they are questions in near natural language. Feeding BlazeMeter results into NewRelic Insights combines all the load tests with all the application monitoring in one interface.
I picked this particular example because I know it works. You can actually very easily take the results from tests in Blaze Meter, and inject them into Insights, and build a dashboard on the results. But also because both of these tools are typically considered tools at the end of your applications release processes and moving into operations and monitoring.
In planning stages, the development, QA, and operations team needs to have some idea of the types of questions and analysis they want to be able to do. The entire team should have access to the visualization of the quality of the release pipeline, and be able to pinpoint successes and weak points. That way, they can go through their tool selection and integration with a target.
The net result is a real DevOps process. The team will not only iterate very quickly on application fixes, but also on the processes around the application, providing highly visible continuous improvement— the what and the why—of the pipeline.