Red Hat is readily recognized for its accomplishments with Linux and software design. The Red Hat DevOps culture itself started by design as a program built on training developers in operations skills and increasing operations’ faith in these developers. Red Hat’s IT Platform Operations Manager, Anderson Silva tells DevOps.com how internal teams initiated the cultural change inside this Linux development icon.
Get Back In Line!
The Platform Operations team in IT manages and executes a lot of the deployments of IT-managed applications into these environments, says Silva. On the road to establishing Red Hat’s internal DevOps dynasty and prior to running the Platform Operations team, Silva captained a small band of operations folk in a department known as Production Control. There Silva and crew established controls to ensure that any new deployments would not and could not break the production environment.
“Sometimes this meant we had to halt the whole of a deployment because a developer forgot to add one link to our release documentation, putting them back in line with everyone else who was requesting permission to deploy,” says Silva. Feeling that Production Control as such was more the bottleneck than a deployment solution, Silva saw this as a challenge to Red Hat that would be increasingly important and increasingly difficult to meet.
In those days before DevOps was cool, a Red Hat development manager came to the rescue with a program to turn that bottleneck into a solution that could speed up the deployment process. According to Silva, this first initiative was the root of what became known at Red Hat as the Lieutenant program, which supports developers willing to gain the Red Hat Certified Engineer certification so they could enjoy production and pre-production access and manage each their own deployments as frequently as they liked. Red Hat eventually added a “corporals” component as an additional and similar program. These developers in “corporals” gained QA access and did not have to earn the RHCE certification to get it.
“These two programs added significant value over the years. By building trust between developers and operations, we addressed many technical issues that we had historically viewed as ‘another person’s problem’,” says Silva, who also holds the RHCE and RHCA certifications and actively maintains a Fedora Package. Red Hat continues to foster a flexible DevOps culture and to nurture the flexibility that sprouts from added trust between engineers and developers.
That flexibility is paying dividends. Where as it once took Red Hat a week to develop and deploy software for about 20 servers to its staging environment, today the Linux titan can complete such a release a few times a day, says Silva. “I wouldn’t be surprised if in the near future we are able to do it a few times an hour when necessary,” Silva says.
And by keeping the faith between operations and development, Red Hat can limit the risk that might otherwise come with its migration to its hybrid cloud. This DevOps-relationship-based culture will also benefit Red Hat’s new image-based deployments, which count on tools that are common among members of this culture such as the Red Hat Enterprise Linux OpenStack Platform, OpenShift, Kubernetes (a data center tool), and Docker.
“As an open source company, we use open source solutions where possible and we have written our own tools to help us automate deployments,” says Silva. As Red Hat’s Platform Operations team is about to retire some of these tools in favor of others, it is also reaching out to other internal IT departments to spread the use of image-based deployments, which also leverage Jenkins, AWS, Puppet, Ansible, and Git.
The DevOps culture that a few managers seeded within departments has borne fruit in the form of a top-down leadership example that rewards and encourages developers who innovate. Red Hat draws temporary ad hoc teams including sys admins and developers together to address nagging development issues. The key to success, says Silva, is in working on these challenges together instead of finding someone to blame.
Silva advises that enterprise leaders who want to wade into fostering a DevOps culture focus on their people before whatever looks new and shiny among the many choices for tackling DevOps approaches. “Trust your people and let them do their jobs. When mistakes happen, don’t look to blame; look for a solution. Then take a look at what’s new and shiny,” Silva says.