DevOps.com

  • Latest
    • Articles
    • Features
    • Most Read
    • News
    • News Releases
  • Topics
    • AI
    • Continuous Delivery
    • Continuous Testing
    • Cloud
    • Culture
    • DataOps
    • DevSecOps
    • Enterprise DevOps
    • Leadership Suite
    • DevOps Practice
    • ROELBOB
    • DevOps Toolbox
    • IT as Code
  • Videos/Podcasts
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
    • DevOps Unbound
  • Webinars
    • Upcoming
    • Calendar View
    • On-Demand Webinars
  • Library
  • Events
    • Upcoming Events
    • Calendar View
    • On-Demand Events
  • Sponsored Content
  • Related Sites
    • Techstrong Group
    • Cloud Native Now
    • Security Boulevard
    • Techstrong Research
    • DevOps Chat
    • DevOps Dozen
    • DevOps TV
    • Techstrong TV
    • Techstrong.tv Podcast
    • Techstrong.tv - Twitch
  • Media Kit
  • About
  • Sponsor
  • AI
  • Cloud
  • CI/CD
  • Continuous Testing
  • DataOps
  • DevSecOps
  • DevOps Onramp
  • Platform Engineering
  • Sustainability
  • Low-Code/No-Code
  • IT as Code
  • More
    • Application Performance Management/Monitoring
    • Culture
    • Enterprise DevOps
    • ROELBOB
Hot Topics
  • Cisco Acquires Splunk to Create Observability Powerhouse
  • Nobl9 Unfurls Reliability Center for Managing SLOs
  • Harness Launches Open Source Gitness Platform
  • Documentation as Code: A Game Changer for DevOps Teams?
  • Innersourcing Open Source Principles in the Enterprise

Features DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow

DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow

Avatar photoBy: John J Balena on May 16, 2014 1 Comment

In organizations trying to achieve agility with stability and quality, silos (of teams, tools, and processes) create major stumbling blocks to managing configuration drift between pre-production and production systems. Breaking down those silos, improving visibility into configuration drift and facilitating collaboration for improved flow are the keys to overcoming this challenge.

Recent Posts By John J Balena
  • Merging creation with operations: facilitated collaboration enables continuous delivery
Avatar photo More from John J Balena
Related Posts
  • DevOps Best Practices: Break Down Silos, Avoid Drift and Optimize for Flow
  • Merging creation with operations: facilitated collaboration enables continuous delivery
  • DevOps: Shift Left to Reduce Failure
    Related Categories
  • Features
    Related Topics
  • best practices
  • drift
  • flow
  • process
  • silos
Show more
Show less

The challenge of staying in sync when both sides are moving

Production environments aren’t static. Whether you are applying patches for improved security, optimizing settings for availability and performance, or making changes for a technology refresh, production systems are continuously evolving.  As new releases are being developed, tested, and staged, pre-production environments are continuously evolving too. Ideally you want to develop with production in mind, but that gets harder and harder when both sides are changing and different teams are responsible for different environments and live in their “own worlds.” (e.g. the common split between QA teams, release teams and production operations).

AWS Builder Community Hub

Even within pre-production, staying in sync is a significant challenge. For example, because applications ride on middleware and middleware settings are unique to each application, keeping systems in sync from QA to staging can be a real challenge.

QA and staging environments are regularly reconfigured to support multiple releases that are in process at the same time.  The costly way, often employed to minimize changes, is to “protect” QA environments for a certain application or release leading to low utilization and bottlenecks.  The rapid proliferation of mobile applications to front end legacy systems magnifies the challenge by adding another dependency layer.

Disconnected teams, information, and perspectives (those pesky silos)

Why are these challenges so hard to overcome if we know the issues?  Many IT teams operate in their “own worlds” with tailor-made but disconnected tools and information resources.  Silos have evolved because they permit the specialization required to deal with the immense, growing complexity IT deals with everyday, but they have now become the biggest impediment to agility and quality.

For example, the operations team has access to systems management alerts that most others don’t, engineers have access to build systems and scripts that others don’t, developers have access to functional requirements and stories that others don’t, and change approvals are stored in a system of record that only certain people (often managers) have access to – just to name a few silos.

Coordinating work and sharing information across these silos of teams often equates to lots of meetings, conference calls, and email threads.  This is not the most efficient way to get work done and it’s not particularly effective in sharing information to keep environments in sync.

“Didn’t you see my email from yesterday with the attachment listing all the middleware settings required for today’s test?”  “Weren’t you in the meeting three weeks ago when I said a Unix server was probably going to be needed in addition to a Windows machine?”  “I told so and so in an instant message on Friday to run this script before he went out on PTO, but he must have forgotten. Who else has root permission to run it while he’s out?”

Sadly, these kinds of statements are the norm, not the exception, in most IT shops.

The reality is there is inefficient coordination to manage all the steps and handoffs between application owners, development, QA, release and production functions.  Instead, I recommend thinking about the software development and deployment lifecycle as an “uber change” and “uber collaboration” process that must be viewed and managed from end to end; from business requirements, to development, to QA to staging to production.  All of the different silos of teams must have a way to work together in a unified way with access to critical information from upstream and downstream teams to ensure a smooth lifecycle end to end.

Automation ≠ DevOps

The heart of DevOps is collaboration and managing change, yet the conversation often seems to start with automation.  The fallacy here is the belief that we can eliminate the need to collaborate and manage change by removing people from the process and relying on the machine.  This out of balance perspective between the machine and people creates more problems than it solves.

Automation tools, while important and necessary, are not all that’s needed to truly solve these synchronization and deployment issues. In addition, to have environments only changed by automation tools and never by people during the firefighting of incidents is unrealistic.  You need your people armed with intelligence and tribal knowledge to drive your automations, but all too often we rely on individuals with disconnected perspectives and limited awareness so things get missed or overlooked.

Automation does what it’s supposed to do, but it makes a critical assumption; what’s being changed is configured how it’s supposed to be configured.  If drift has occurred because critical settings were modified to remediate an incident or compliance issue, without the automation engineer aware of the change, the automated deployment will result in you hearing a loud “boom!”

As one popular DevOps Twitter handle has said, “To make a configuration error is human – to do it across a thousand servers is DevOps.”  While that may seem like an extreme statement designed for comedic effect, it isn’t extreme to say that thousands of deployments go south every day because systems aren’t configured the way we expect them to be given that the complex software stacks that multi-tiered applications now depend on are touched in so many ways.  Automation alone doesn’t solve the DevOps challenge.

Negative consequences to business

Of course, the issues I’m raising here aren’t just IT issues; they are real business issues.  Time to market is typically the number one impacted area. For example, if a two-week development sprint is used to get a business critical need addressed, but then it takes weeks and weeks to get the sprint through the quality cycle and into production due to resource constraints and configuration issues, what have we achieved?  

 Stability and quality are the next big area of impact.  When change is rapidly introduced through new releases and unexpected drift occurs in environments, performance and availability impacts occur.  It is one thing for IT to be perceived as slow and unresponsive to the business, but it is quite another when they are perceived as slow and quality is poor resulting in business impact.

Just when IT thinks it can’t get any worse, the business asks a presumably simple question, “What happened?”  IT’s typical response, “Let me get back to you on that.”  And then begins a scramble of hours, days or weeks as IT’s best people desperately try to figure out what went wrong.  This is obviously not the best confidence builder for the business.

Compliance is typically the other casualty as there is often very little awareness of operational policies on the development side of the house so deployment packages and instructions aren’t sensitive to compliance issues.  Building and testing an application without full awareness of policy governance across the whole, multi-tier stack is bound to introduce delays and errors when attempting to deploy in production.

Visibility, configuration transparency and collaboration; a better way!

  1. Understand drift for the entire application service. Rather than everyone having their own disjointed perspectives on databases, app middleware, network or firewall, and server configurations, provide team members with an understanding from the business service (or application) perspective with direct visibility into the differences between how things should be configured and how they are actually configured for the entire multi-tier application stack.  This enables you to create smarter automations and initiate change processes to bring configurations back in line when necessary rather than waiting for a failed deployment to occur, identifying the root cause, and correcting it while the business waits.
  2. Optimize for flow not individual processes.  As stated in the previous example, making one process faster (e.g. development) without considering its impact on QA, staging and production will generate new constraints downstream.  Instead, view DevOps as one continuous process from goals and requirements all the way through to deployment into production and define success for the team in those terms. When you optimize for flow instead of silos, you will be amazed at how teams share more and collaborate better.

The benefits of breaking down silos through better visibility, understanding drift and collaboration are significant:

  • faster time to market;
  • quicker ability to react to competitive pressures;
  • deployment with fewer errors;
  • better compliance with policies; and
  • greater productivity for everyone.

 To sum it up, if you break down silos, identify and resolve drift, and manage for flow (while avoiding islands of automation, information, and people) you can unlock agility, maintain stability and ensure the quality your business demands.

Filed Under: Features Tagged With: best practices, drift, flow, process, silos

« How to turn executives into DevOps champions
Avoiding Parrot Processes »

Techstrong TV – Live

Click full-screen to enable volume control
Watch latest episodes and shows

Upcoming Webinars

Cloud Security Turbocharged: A Wild Ride of Innovation, Threats and Staying Ahead
Friday, September 22, 2023 - 11:00 am EDT
Infosys Zero Cost Mainframe Transformations
Monday, September 25, 2023 - 11:00 am EDT
How PRINCE2 Improves Cybersecurity
Tuesday, September 26, 2023 - 11:00 am EDT

GET THE TOP STORIES OF THE WEEK

Sponsored Content

JFrog’s swampUP 2023: Ready for Next 

September 1, 2023 | Natan Solomon

DevOps World: Time to Bring the Community Together Again

August 8, 2023 | Saskia Sawyerr

PlatformCon 2023: This Year’s Hottest Platform Engineering Event

May 30, 2023 | Karolina Junčytė

The Google Cloud DevOps Awards: Apply Now!

January 10, 2023 | Brenna Washington

Codenotary Extends Dynamic SBOM Reach to Serverless Computing Platforms

December 9, 2022 | Mike Vizard

Latest from DevOps.com

Cisco Acquires Splunk to Create Observability Powerhouse
September 21, 2023 | Mike Vizard
Nobl9 Unfurls Reliability Center for Managing SLOs
September 21, 2023 | Mike Vizard
Harness Launches Open Source Gitness Platform
September 21, 2023 | Mike Vizard
Documentation as Code: A Game Changer for DevOps Teams?
September 21, 2023 | Gilad David Maayan
Innersourcing Open Source Principles in the Enterprise
September 21, 2023 | Bill Doerrfeld

TSTV Podcast

On-Demand Webinars

DevOps.com Webinar ReplaysDevOps.com Webinar Replays

Most Read on DevOps.com

Why Enterprises Should Embrace Data-Driven Software Management
September 15, 2023 | Alex Circei
Should You Measure Developer Productivity?
September 18, 2023 | Bill Doerrfeld
DevOps is Making Gains on Mainframe Platforms
September 15, 2023 | Mike Vizard
JFrog swampUP: Addressing the Advent of AI
September 18, 2023 | William Willis
Buildkite Acquires Packagecloud to Streamline DevOps Workflows
September 19, 2023 | Mike Vizard
  • Home
  • About DevOps.com
  • Meet our Authors
  • Write for DevOps.com
  • Media Kit
  • Sponsor Info
  • Copyright
  • TOS
  • Privacy Policy

Powered by Techstrong Group, Inc.

© 2023 ·Techstrong Group, Inc.All rights reserved.