When you’re creating work that gets deployed into production, there is typically a repository involved. It could be a code base, a deployment hub, a definitive software library, a library of VM or application templates, etc.
Often, you may not think twice about what’s in these repositories or whether you can trust their content – after all, you and your teams built these repositories, right? That may be true, but what if your trust is misplaced?
The recent Target breach illustrated (among other things) that you may be incorrect in assuming the “inbox” of your deployment process is trustworthy. In the Target example, there was a queue of patches ready for deployment but someone added additional, malicious code to that queue without being noticed. The automated deployment process grabbed both the authorized and malicious code and distributed it to every point of sale device in the network. Not a good outcome.
Their workflow included a trustworthy process to test code that before it was checked into the deployment queue, but there was also a flaw in the process: there were no controls in place to detect when anyone went around the process and made arbitrary changes to the repository. In other words, even if you put trusted packages into the queue, the packages could be modified and additional code could be added without those alterations being noticed.
What we can learn from this kind of situation is the value of the tried & true principle of “trust but verify:” don’t assume you can trust your repositories without validating what’s in there. With that notion in mind, think about your environment – are there repositories that people could alter without being noticed? If so, that could present a problem both for security and reliability in your organization.
To correct this, we need to treat our repositories like bank accounts – account for everything that goes in or out, and monitor the contents of the repository continuously to look for any undocumented or unauthorized “transactions” (adds, removes, or changes). We must also add validation checks to the deployment process to make sure everything is accounted for before the deploy occurs. Yes, this will likely mean you need to do some extra work on your deployment workflow, add technology for instrumentation, etc. but it will more than pay for itself through better security, accountability, and availability.
By the way – this same principle applies to the scripts and automation tools you rely on, as well. If someone were to alter these tools without your knowledge, you may get results you don’t like. You should also consider that your automated deployment repositories aren’t the only vector for untrusted code to enter production – continuous monitoring of the production environment is also a crucial activity.
Protect and defend your repositories – it’s important.