Information security conversations often start with the question “What is your threat model?” This blog asks: “What is your trust model?”. Trust is a complex subject and an integral part of managing DevOps-oriented organizations and highly automated IT infrastructures. Who (developers, ops) or what (code, process) is trusted to accomplish specific tasks in the infrastructure can sometimes be difficult to characterize to management, auditors, or in operational/incident reviews. The goal is to describe relationships among people and systems at work in the DevOps world and discuss approaches to making systems more transparent (to coders and non-coders!).
The Trusted Image
Organizations like Amazon, Canonical, and Red Hat provide virtual machine images of popular operating system distributions. It’s common to launch these images directly “as-is” and install software onto them after launch.
However, there are advantages to specializing one (or several) of these images and using it instead of the base OS image. A base image which has been supplemented with configuration and installed packages that suit the needs of your application is referred to as a “Trusted Image”. And there is definitely value to putting your trust in a purpose-built and well-characterized image.
Why Make a Trusted Image?
The reasons are several:
Reliability: The more downloads that a VM requires in order to provision and configure, the more likely that something will go wrong. Network errors, outages at package repositories, and random download errors can render your VM unusable or unstable. Building base packages into a trusted image minimizes problems.
Secure Configuration: If there are configuration options that you want to ensure on every VM, you can build these settings into your trusted image. For example, you can configure firewall (iptables) in a trusted image to protect every VM that you launch.
Secure Communication: If you want to use SSL or TLS inside your application, you can establish trust in a root certificate via a trusted image.
Built-in Service “Wiring”: Do you want to ensure that the syslog of every VM you launch is sent to your logging server? Or to ensure that a specific version of Python, Ruby, Java, Node.js, etc is pre-installed on every VM that you launch? How about client libraries? A configuration management agent? Build these into your trusted image.
Speed: No more waiting for package installations; build them into your trusted image, and launch the end product faster.
VM Identity and Authorization: You can build trusted images that will create an identity for each VM, and help the VM to get permissions to operate accordingly in your infrastructure. Bootstrap credentials can be used to access resources that the VM needs to join the application, such as database credentials, system passwords, API tokens, and other sensitive information.
The use of trusted images can have drawbacks, for example:
Security Patching: Security updates from vendors are frequent. Any new image that you build will have all the latest security patches built into it; but your images will quickly get out of date as patches are released by your OS vendor. One approach is to configure VMs to patch themselves on a schedule, for example, nightly during a “quiet period”. [ Note: Patching is tough. Please share your experiences with this problem! ]
Management: Once you have built a trusted image, you will need a way to manage them. Image tags and other metadata features provided by your cloud can be helpful here. Treat your images like any other product of your build system: version them strictly and keep track of the provenance (history). Lastly, be sure that the system you use to build trusted images is itself trustworthy!
Weighing the pros and cons
Overall, for many organizations the security, performance and reliability benefits of the trusted images outweigh the drawbacks. Would love to hear more experiences from those of you who use trusted images and who’ve considered trusted images but taken another path.