Expert, like Master, can be a Dangerous Word

William Sealy Gosset was a mathematician that published while working at Guinness. The company was concerned about trade secrets being leaked so all brewers had to write under a pseudonym. Given he was publishing strictly for academic purposes, he decided to write under the name “Student”. If you have ever taken a statistics course you’ve probably used his Student’s t-test. I couldn’t find a completely consistent account of why he selected the name, but Student seems to have been chosen in the spirit of continuous learning.

I often confuse the L in the CALMS model for Learning, but it’s actually Lean. At first glance it may appear that continuous learning and lean practices are in opposition of each other, but in fact continuous learning is an operational expense of implementing and maintaining lean practices along with devOps as a whole.

The Old Way of Learning

Up until the 1900s brewing was a trade that operated off of an apprenticeship system. I’ve known a few people in apprenticeship positions and they were treated horribly for the most part. If they swept enough floors and cleaned enough toilets they may graciously be given the privilege of getting tenuously involved with some side work. Often the hazing got pushed to the point that the apprentice quit before they were given any work that wasn’t custodial in nature. Only a couple of times did the expert realize the apprentice left because they weren’t being respected. Most of the time it was chalked up to an inadequacy of some sort.

There was some of this mentality at Guinness. Newly hired positions were instructed that if they were lucky enough to meet a brewer in the hallways they should not identify them but lower their eyes until they had passed.

If you’re in an toxic environment you should quit and go work somewhere else if at all possible. I’ve heard it’s easier than you think from many walks of life, but Katherine Daniels (conveniently known as @beerOps) was a recent guest for a podcast on developing careerOps with @SoberBuildEng, @eciramella, @sacha_d, and @SonOfGarr on the The Ship Show. You should check it out for additional information on finding the best fit position and employer for yourself.

Transitioning to a New Way

I’m guessing this brewer centric culture at Guinness had caused enough problems for Cecil Guinness to get sick of it. With the assistance of the managing director at the time, a Mr. Christopher Digges La Touche, Guinness started hiring STEM graduates from Cambridge and Oxford and making them brewers after a two year apprenticeship. They had decided to make the brewing of beer scientific.

The culture and environment sound like they belong to a start up. These newly hired brewers lived together in a house at St. James’ Gate and spent a lot of free time on team outings. A group of really passionate people with similar interests will start to brainstorm and collaborate if they spend enough time together. They started to figure out ways to measure properties of the raw materials used in brewing that had always been qualitative in the apprentice program, such as subtleties in the grains that had to be learned from experience under the tutelage of a master brewer.

I’ve never brewed any beer and I have no idea what a brewer would mean by the rub of the hops or the texture of the barley, especially when described as milky or steely. The real problem with subtle nuances that are driven from intuition instead of data is the difficulty in comparing results. I certainly couldn’t explain how one barley was milkier than another barley. By identifying a method of measuring these properties consistently and accurately the brewers were then able to compare different sourcing options empirically instead of off their gut. It also helped the brewery identify how to store their materials until needed while maintaining the quality of the ingredients.

Op-ex Well Spent

The discovery process wasn’t cheap or easy. Facilities and laboratories were built. Tours of fields and networking events where brewers could meet and converse with the farmers and maltsters where arranged. Outside of finding the right hops to grow they studied the cost associated with producing them. I suspect it was worth the cost of growing their hops if they could control the quality consistently. Guinness eventually bought enough farms for these young ‘brewers’ to experiment with that Guinness became one of the largest hop growers in Ireland and England.

Not only was this expensive, but it was culturally difficult. Getting brewers and farmers to change traditional practices that had been handed down over generations is not easy. Especially when your reasoning depends on science and mathematics that aren’t always easily understood.

Good Experts Collaborate

Each of these young ‘brewers’ did become a resident expert that could help the other ‘brewers’ when they encountered a problem in a specific area. Not only was Gosset good at listening to their concerns but he made sure he helped them find a best fit solution they understood. He even built tools to help solve their problems when needed, the t-test being the most famous among them.

A maltster named E.S. Beaven was a bit of a Ron Swanson and had no interest in statistics or statisticians. Gosset was so fascinated with his skill in cross breeding barley though that he eventually overcame the social hurdle of being a mathematician and became close friends with Beaven. Gosset was then able to help him set up several nursery experiments, including a group of experiments where farmers in different areas swapped seed to see how the yield varied with agricultural methods and environment.

These young ‘brewers’ Gosset was working with were good experts. They were always learning more about brewing by taking measurements and sharing their findings, even if they had to do it anonymously. This was all done in an effort to eliminate waste and improve quality. I’m not sure how focused they were on automating everything, but this was just before the time Henry Ford’s assembly line started rolling. Automation probably came earlier at Guinness. A Brewery, unlike many other factories, can power automation using only gravity instead of steam or electricity. Scaling one up is easy to do compared to something like the Ford factory.

The First DevOps Engineers

I see the CALMS model being followed here. I see a systems perspective being embraced, feedback loops being completed, and a culture that understands the connection between experimentation, failure, and eventual improvement. This is consistent with the Three Ways. Was Guinness an early (albeit unaware) adopter of devOps? In my opinion yes. Not only was Guinness transcending industrial engineering by running one of the first devOps incubators in the 1900s, but Gosset and his group of young ‘brewers’ would be more aptly titled DevOps Engineers.

DevOps requires production. It is a lot like industrial engineering, but the current revival of these practices under the banner of devOps is a result of the rapid changes in technology we’ve seen. Viable applications can be provisioned at scale using these techniques, but this isn’t a job just for software companies any more. If you produce or sell something and aren’t optimizing from a systems perspective, leveraging technology to automate and measure where ever you can, and keeping your employees and customers satisfied in the process then your business could probably be making more money.

There Are Bad Experts Too

I have walked through the debate on hiring devOps engineers with a few folks. Being a devOps engineer myself, I obviously think there’s a place for it and it can have merit. Growing a devOps advocate internally can be really difficult because of history, baggage, and biases that are already in place. A devOps engineer, much like Gosset and his buddies, really needs to have a fresh set of eyes on the work flow and where the bottlenecks, waste, and quality concerns are located. Guinness was smart to hire more than one of them though. If you’re going to have experts or masters in your organization you should have more than one expert in any single subject.

I’ve encountered a lot of resident experts in enterprise sized companies that have become the single source of truth on a particular subject. When brought a problem that needs a solution it’s very easy for them to declare it “as designed” or a job for someone else because it’s not quite specific enough for their level of expertise. If there’s no one to give a second opinion on the matter how can they ever know when they’re wrong?

I had a boss that used to tell people, “A Master electrician can still change a light bulb” when they complained someone was doing work outside their area. If people are showing up with problems and someone can easily help them they should help them. If your expert isn’t getting their job done then their customers are going to take the demand somewhere it can be met. A good expert should have the awareness to collaborate with someone who can help them manage their workload instead of expecting people to wait or just deal with support issues.

Which is why Gosset and Beaven could become friends. Beaven was truly a master maltster and an expert in cross breeding barley. Gosset respected this and wanted to learn from him. If Beaven had been mediocre at his craft I doubt Gosset would have cared what he thought of statistics, but when Beaven saw the mathematics was a tool Gosset used to understand and confirm why he was good at cross breeding barley he then became interested in it. A demonstration of utility is the best sales pitch you can make, as seeing is believing.

Collaboration Makes the Difference

A good devOps Engineer must become an expert in many things. It helps us help each other, it fosters empathy, it keeps our career options open, and it keeps us honest. If we get bored or frustrated with a job where there’s no traction then we can take our diverse set of skills and do something else. Sometimes this can even be done in the same company. Most companies would rather re-purpose a talented individual if they can move them into a role they can perform with some passion again than see them leave. There’s strength in diversity and it ensures continued survival, even when talking about what you do day to day in the line of your work.

If you’ve landed in a position of absolute authority you run the risk of degenerating into a bad expert that’s mostly sitting around, punching a clock, and telling people to get lost instead of helping them. It’s not that you’re a bad person, but absolute authority corrupts absolutely and you’re still human. If there’s any question whether you might run this risk yourself then find a colleague you can collaborate with that can help you learn as much as you teach in complementary subjects. Find a Gosset to your Beaven (or a Beaven to your Gosset). Cross training will help manage your work flow but it’s a balancing act between optimizing hand-offs and individual task transition. Still, if you diversify your skill set it will definitely be good for your own self-preservation in your current role and in your next one.

SOURCES/ADDITIONAL READING