Reading, “The Goal” was a pivotal event in my career. More than a decade ago a friend of mine who was an operations professor at Georgia Tech told me it was his favorite book. We were discussing the challenges of getting work done in a corporate environment. I was grumbling about leading a web dev team in a company that did not embrace the software development discipline. Already a fan of Tom DeMarco and Timothy Lister’s, “Peopleware”, and being very vocal about the lack of ability to focus whether due to environmental distractions or the constantly changing priorities and schedules, I was receptive to the message in “The Goal”. It gave me a vocabulary and a roadmap for improving our throughput issues.
Our throughput problems were a textbook case for development disaster. When I joined the project, we had three source code repositories, from none of which could we build production and when we did produce a build it would work in pre-production environments but not production (this just slightly predates the widespread acceptance of continuous integration). We also had the classic inconsistent release cycles that would lengthen based upon insistence by the business owners to include more features. Looking back, it is not surprising in the least that we were not successful.
Not surprisingly, these issues led to a lack of trust with the business. Overcoming the technical problems was straightforward using brute-force tactics. We addressed the source code problem by recruiting a senior developer and empowering him to make improvements. We overcame the build quality issue by using a primitive form of agile and enforcing knowledge transfer; the operations engineer sat between two developers during the build and release process and he was the only one allowed touch the keyboard. Our tactics were basic, and they worked.
The bigger challenge facing us was getting our business owners to accept a different SDLC model. We were saddled with a classic waterfall process, each phase being dependent upon the last with a lot of idle time built-in at each phase gate. The shortcomings of the waterfall process were amplified because the business lacked confidence about when, if ever, they would get a feature released to production. This lack of confidence led to a constant rearranging of the priorities to get the most important feature du jour in the next release. The constantly changing priorities created even more idle-time and a lot of work-in-progress (WIP) which Goldratt’s Theory of Constraints (TOC) tells us is at the root of inefficiency.
Using TOC as a guide, we redefined our SDLC from requirements through QA. We focused on developing a process that de-coupled the phases, reduced idle-time and work-in-progress (WIP). De-coupling the phases meant the work could queue between each phase allowing the teams, (requirements, development, QA, and operations) to vary independently; they didn’t have to wait on the team before or after to complete their work which reduced idle-time. We then focused on improving the slowest areas, which seemed to be requirements and deployments.
We convinced our business owners to trial a monthly release cycle (this was a long time ago, even before continuous integration was widely accepted!). The monthly release cycle helped give the business confidence that they would see their features in production at an expected pace and it allowed requirements to cycle on schedule independent of development and QA. Some features would get done in a month and some would span several months. Over a period of three months we got into a rhythm that worked for about 14 months until organizational focus changed and we focused elsewhere.
Reading “The Phoenix Project” reminded me about those days and the challenges Goldratt helped us overcome. I really liked “The Phoenix Project” and many of the situations rang true. I have since moved from development to operations (twice!) which gives me a unique perspective as I see both sides. I see DevOps as the embodiment of Goldratt’s ideas in IT; reducing work-in-progress, reducing cycle-time and always focusing on the most constrained processes. Add a consistent release schedule and it looks a lot like DevOps. Everyone has their own ideas about what is DevOps and how best to implement, I suggest we follow Goldratt’s advice and focus our efforts where they will have the most benefit.
