In Internet slang, a troll (//, //) is a person who sows discord … by starting arguments or upsetting people, by posting inflammatory, extraneous, or off-topic messages in a… community …with the deliberate intent of provoking readers into an emotional response or of otherwise disrupting normal on-topic discussion.
When I stopped by the Chef booth at Velocity Santa Clara this year Nathen Harvey was out of XL t-shirts. I happened to have a Puppet shirt on at the time too, but it was just a lucky coincidence. Nathen’s always advocated using something (anything) to automate your infrastructure, as the ‘A’ in the CALMS model for devOps stands for automation, but the look on his face when I crossed his line of sight was as delightfully awkward as it could have been. Nathen told me he’d get me a Chef shirt if I got him my address back in Atlanta. I had been meaning to do it before I made it out to PuppetConf, but a tweet from Mandi Walls ultimately ended up prodding me into action.
To Nathen’s word, he got me a shirt in under a week, and in plenty of time for #PuppetConf.
I think “Stirring up delight” was the right shirt to send me too. I was half worried I’d get a shirt that was much more confrontational, putting me at risk of getting jumped and having my lunch money stolen. Lucky for me pretty much everybody that noticed the shirt thought it was funny. At least anyone who said anything to me about it. A proper troll is going to get at least one person a little bit angry though, right? Anyone that recognized I was in a Chef shirt happened to create an opportunity for me to ask a few questions, so I did.
- Have you used Chef, Puppet, CFEngine, or any other configuration management tools?
- Which do you prefer?
- Why do you prefer that particular tool?
- Why does the competition between these tools help or hurt our industry?
- Where do you see these tools in five years?
I got pretty consistent answers just talking to people, and the Q&A often evolved into a natural conversation, which I prefer. (Another devOps principle I like to practice is shut the hell up and listen). I also put up fliers with a link to a survey while at the conference in an attempt to gather some empirical data. Truth be told, the survey was the real troll.
One of the best talks I saw at PuppetConf was by Lindsay Holmwood, call The DevOps Field Guide to Cognitive Biases. I wasn’t familiar with the term before the conference but I think I had the idea. I also know I still don’t fully understand it. Watch the video and check out the books Lindsay suggests at the end. It’s pretty deep stuff. I really like his advice on having engineers manage and managers engineer to help break these biases down, as it creates a lot of empathy between the two groups.
Given my own cognitive biases, I wouldn’t have filled out the survey if I had seen one of the flyers I posted. The flyer said nothing about a home school project for twin girls, which would have had a much better chance of getting my attention. Instead, it just asked folks to fill out the configuration management survey with the promise of cookies for their trouble. I’m pleased to say the tech community will respond to surveys for cookies even when crudely marketed. While I have enough responses to do sound statistical analysis, I know my data isn’t sufficiently stratified and probably doesn’t represent the larger population of the configuration management community. In other words my data lacks diversity.
My daughters’ home school project is to start a business, and the twins decided to launch Gemini Snacks. Their company is pretty much an online bake sale to allow them to give back to all the hospitals and organizations that have helped them and cheer up the kids that can’t go home yet.
The deal I made with my daughters was simple. I would print out flyers for PuppetConf to get survey responses to help me write this article. I would thank as many people as we could for filling out the survey by sending them some cookies, and I would hire their company to bake and ship those cookies. If people filled out the survey my kids would be in business.
There was a strong enough trend in the responses that I can discuss those now with confidence despite a lack of diversity in the data. Of course I got a lot of responses from Puppet users, but I also heard from people using CFEngine, Chef, Ansible, Salt, and bconfig2. A lot of people weren’t using anything and just recently picked up Puppet. I ran into a few folks that had used three or more tools at one point or another and could give comparisons which were really interesting. Here are the main conclusions I’ve been able to draw so far.
A Rising Tide Floats All Boats
Most organizations are still sshing into a lot of hosts to handle things, sometimes they do it with scripts. Configuration management (CM) tools are still becoming a fully adopted standard. I have heard users of every CM application in the market say it really doesn’t matter what tool you’re using, automate your infrastructure with something. As a result this is still very much a growing sector in the tech industry.
Don’t get into a Configuration Management Bake-Off
Bake-Off is the word that was used, and more than one person put it that way specifically. I don’t know what you’d call setting up test environments to try different CM tools before making a decision, but I’d call it a bike shed of yak shaving myself. Bake-Off is probably more familiar people anyway. My opinion is you should either be using Puppet or Chef, but if you’re not using any configuration management tool at all it’s advised that you start.If you can’t help but take more than one for a spin before committing then keep it short and sweet. You’ll be better off taking the time to commit to one and put the work into making the transition quickly than trying to figure out the easiest way to do it. This automation will take some effort. The key thing to understand is the right tool depends on your environment.
There’s a Lot of Respect and Appreciation for CFEngine
CFEngine is a trailblazing academic project that gained a lot of popularity in the early days of CM. The adoption of the software in the open source community really proved the viability of CM as we know it. While the application isn’t seeing continued growth now that there’s competition with newer tools in this space, several people went out of their way to emphasize the immense respect they have for the product and its founder, Mark Burgess, as CM wouldn’t be as evolved without them.
Puppet: Preferred by Ops Folks because it’s Easy to Configure
The main thing the ops crowd seems to prefer about Puppet is that it’s easy to configure. I had a book recommended to me at the conference called, “Switch: How to Change Things when Change is Hard”, by Chip and Dan Heath. The book describes a “Project Mood Chart” used by IDEO, a product design firm, to brace their clients for adversity.
When a new project starts we’re optimistic and hope for the best. We inevitably face adversity during the course of our work, and while this adversity is always frustrating it does lead us to insight. Empowered by our new perspective on the problem we can proceed to a viable solution with confidence.
I believe this is how Puppet evolved out of CFEngine. Luke Kanies, the founder and CEO of Puppet, was an early contributor to CFEngine. I can see Luke following this curve, gaining insight on the configuration difficulties the CFEngine community was experiencing, and going on to found Puppet, confident he had a product better adapted to industry since it was easy to configure.
Chef: Preferred by Devs because it Allows them to Treat their Systems Like Objects
I can see this same curve being followed as Chef came out of Puppet as well. With ease of use comes a lack of robustness. Puppet’s advantage over CFEngine is it’s easier to configure, but there seems to have been a point of frustration within the development community that lead to the creation of Chef. Developers like to treat things like objects, which should be obvious if you have any experience with object oriented programming. Abstracting the complexity of the software we write so it can be configured by end users that are not developers is challenging. Since Chef allows its users to treat their systems like objects, the difficulty surrounding the abstraction of any complexity into a declarative model is circumvented.
Which is the main reason ops folk don’t like Chef as much, because they’re not programmers. If you’re in an organization with a mature operations department that’s not so interested in learning to code, Puppet ‘s declarative model for describing systems is going to be a better fit. However, if you have developers doing most of the operations work out of necessity in a small shop or an operations group that’s doing more software development for internal tools, especially if they like Ruby, Chef may be a better fit for your organization.
Ansible: Symbiotic Orchestration
The best things anyone had to say about Ansible is that it’s great for orchestration. It seems to be competing more directly with mcollective, a framework Puppet Labs supports for orchestration tasks. Mcollective can be used with both Chef and Puppet, and the same appears to be true for Ansible. Ansible is also agentless, managing nodes over SSH. Depending who you talk to this can be a good or a bad thing. By not running an agent the nodes aren’t dealing with the overhead that would come with it, but you’ve also lost the ability to monitor your systems as tightly as you could if they were running an agent. I’m under the impression CM is trying to move us away from SSH loops. While I don’t think it’s fair to write Ansible off as an overblown SSH loop, I am of the opinion that the advantages provided by agents running on each node in your environment are well worth the overhead. It will be interesting to see if Ansible’s relationship with other configuration management systems continues to evolve into a symbiotic orchestration provider or if another product, like mcollective, continues on to fill that void.
Salt Isn’t Really Configuration Management Software, but it’s Fast
Salt is a remote execution platform build on zeroMQ that was designed with speed in mind. Everything salt’s being used for, including CM, has been an afterthought to the fast remote execution platform at the heart of the project. The constant connection the nodes (called minions) maintain with their master definitely contribute to the speed of the system. This would lead me to believe Ansible is the slowest of the four technologies discussed here, since it doesn’t run an agent at all, and Chef and Puppet would fall somewhere in the middle since they run an agent on their nodes but it isn’t on constant communication. (This is an educated guess on my part. Please let me know if you can speak to the speed concern from personal experience.)
I have run into a few businesses that deal with high speed financial systems that use salt for a lot of things, including CM, but only because it’s fast. If you have systems that demand split second responses to failure and change, salt may help solve a lot of problems your organization is currently facing. I’m not convinced it’s that easy to use.
Salt seems popular with groups that want to avoid Ruby like it’s the plague or folks that are so horribly comfortable in a CLI that using a GUI just seems unnatural. I think these are terrible reasons to select a tool, since it’s directly avoiding the ‘adapt or die’ mentality that the devOps community embraces. Still, depending on your environment and the adversity you’re dealing with, these may be entirely reasonable grounds to use salt instead of another technology. I’d be less concerned if you told me you picked salt because it’s fast though.
The Survey is Still Open
I hope you like what you read here but I’m sure somebody doesn’t agree with me at all. The good news is you can still fill out the survey and let me know. When my data looks sufficiently diverse I’ll publish another update, and there’s a decent chance you’ll get some cookies for you time too. I’d also like to thank everyone that took the time to discuss this with me. Gene Kim, John Willis, Chris Webber, Charles Johnson, Dawn Foster, Kara Sowles, everyone at Puppet Labs, and of course Nathen Harvey and Mandi Walls, all helped to make PuppetConf a huge success and a memorable experience for me. I’ve learned way more than I expected from the experience and I’m excited to see how the competition in this space plays out in years to come. Competition is a good thing. As different challenges arise and as projects go through mood swings I believe we’ll see two things happen: The communities for these tools will adapt them to adjust to the environment they’re operating in, and at some point enough adaptation will lead us to a new set of tools all together. I for one am really excited to see what happens.