Who hasn’t heard the classic statements “It works on my machine!” or, “Man that would have been easier with real data!”? In either case, had the development happened in production, problems would have been avoided! Fortunately, in a DevOps world, the walls between production and non-production get broken down making the reality of developing and testing in production more tangible. In no particular order, here are some of the things I’ve done to breakdown those walls.
Develop on and deploy from a single code line.
Many times we find ourselves in complex processes around deployments and production bug fixes. For example, the code is branched for release candidates and deployments are done from the branches. Branching before release by itself isn’t complicated, however, when there is a production bug, engineers find themselves repeating or merging the fix into at least two, if not several, different places. Repeat that a few times over and, uh oh, now there is a problem because there are many moving parts and questions about what exactly is in production. Unfortunately, the differences are most likely unintentional.
Instead, simplify the process and have everyone work off the same code line and have that same code line deployed directly to production. Now there is only one code base, therefore no question about what code is running in production. Production isn’t static, so change will still happen after deployment. Notation is required indicating what version was deployed, so you can still reproduce production.
Let testing happen on the production database.
Ok, even I’m not that crazy, but let developers use a copy of the production database. There are two very important aspects to this. First, refresh it often. I suggest daily. I have had very good luck using EBS snapshots to get a copy of the database on a different server, but if your database is small, good old SQL dumps also will work. Second, make it safe. Put some safe guards in place to make sure a change doesn’t accidently get made to the real production database. Keep your ACLs tight, change the passwords and practice proper database user management. Lastly, a bonus outcome is that you are constantly testing your database backups and recovery procedures.
Minimize the steps between development and production.
Each step, especially the manual ones, between the developer’s machine and production is another point where unexpected changes can be introduced. It sounds obvious, but the gates code must pass through before entering production must be purposeful and prove their usefulness. For example, if code must go through a regression, but no regressions are being found, then that step is not useful and just increases the probability of unintended change. I have been a part of a process where code had to be deployed to “staging” where it was then hooked up to the live production database, but was only to be exercised in a read-only manner. After doing that for many, many releases and very rarely finding defects before production, we decided to just get better at deploying to production and eliminated the deployment to staging. In the end, we got faster and with no impacts to quality. Less can be more.
Depending on your situation, database size, regulatory compliance, etc. you might to be able to do all of them, but breaking down those walls will yield better code, fewer bugs, happier engineers and fewer sleepless nights to name a few. So, grab that wrecking ball and break down a few walls.
