So, here we were, at one of the bigger financial corporations in New Zealand, working on an elaborate infrastructure built on Docker containers. It had configurations, SSL certificates, DNS entries and all the other condiments required to spice up a tasty infrastructure curry. Though we were able to stand up our complete infrastructure environments from scratch using automated configuration management tools, we weren’t really sure if it was indeed working correctly, or rather the way it was supposed to. The question staring at us was, Is there a way we can declare the desired state of our infrastructure and then validate our infrastructure against it?
Inspec to the Rescue
The open source tool Inspec has been used with considerable success over the last few years to achieve security compliance for infrastructure. The syntax/constructs happens to be unbelievably simple. For instance, let’s consider the code snippet below. It doesn’t really require a lot of programming knowledge to make sense of it. Isn’t it?
This made us wonder, what if we use Inspec to validate our infrastructure setup? What if we map our infrastructure configurations in an easy-to-read declarative syntax? What if this actually solves our problem?
In line with the best practices of DevOps, we started with a simple experiment; an experiment that could solve a tangible problem. We had a Docker host which was supposed to have a particular docker container running inside it, but it used to fail quite often, for some reason. We decided to write a test in Inspec to check whether this container was running on this host:
… and tried executing the test.
This was just what we wanted!
Excited with this success, we started writing a battery of tests that validated each and every component of our infrastructure. Within a single day, we churned out 192 tests. The number of tests wasn’t as important as the scope and breadth of the test coverage. Once the tests were there, we plugged them on to our infrastructure pipeline (which was built using Jenkins) to ensure the components align with the desired state. We called them ‘component-readiness’ tests.
So far so good.
Now that our components match the desired state, are we still confident that our platform would work the way we intended? Perhaps we need some “platform readiness tests” as well. Can we use Inspec for that, too? The answer: Yes!
Now that the platform readiness tests were ready, we extended our infrastructure pipeline to incorporate them as well.
Setting our component and platform readiness tests using Inspec had one more fantastic offering: live documentation for infrastructure. Inspec allows the audit results to be displayed in a variety of report formats, including JUnit, JSON, HTML and others. We used the JUnit report format and displayed them using the Jenkins JUnit Plugin. With this, we were able to have human-readable JUnit reports that doubled as live documentation for our infrastructure.
Needless to say, this infrastructure testing with Inspec provided that (much needed) confidence on our setup and readiness. Our pipeline is designed in such a way that whenever any infrastructure component changes in our ecosystem, the pipeline is triggered which reconfigures and (if required) rebuilds the ecosystem. These Inspec tests are triggered with this infrastructure pipeline and validates the newly configured infrastructure setup. Thanks to this, we are never in a situation in which the client application running on this infrastructure complains or aborts because of incorrect configurations. Moreover, as described above, the living documentation helps document the expected configuration of the environment.
So far, the Inspec implementation I have referred to is using the freely available Inspec tool and the Jenkins platform. Inspec can also be combined with Chef Compliance to produce more descriptive reports and charts. Chef Compliance also comes with out-of-the-box CIS profiles that can be used to jump start an Inspec implementation for security compliance and audits.
Inspec is an easy tool to master with simple and intuitive constructs and one could get up and working quite easily. However, this simplicity has a downside. It’s very easy to speed ahead and gloss over the documentation while missing out on the best practices and tips/techniques to implement the solution in a better way. One way to master this is to have a crack at the newly released Auditing with Inspec certification, which provides the impetus to go through the documentation in detail while preparing for the certification. And, you also get to show off a shiny badge for your efforts!
In Summary
Infrastructure validation is quite an important step in any DevOps implementation. Apart from being very useful for security compliance testing, Inspec can be used to validate infrastructure using a simple and easy-to-understand declarative syntax. Moreover, it has a small footprint, a vibrant community looking after it and up-to-date and clear documentation. It definitely is a tool with the potential to provide quick and effective results.