scream

DevOps does not equal “Developers managing Production”

I’ve had a few conversations lately, mainly with smaller start-ups or development houses, who tell me “yes, we work in a DevOps model”.

What they really mean is “We pretty much have no Operations capability at all, and we rely on the Developers to build, deploy and manage all of the environments from Development to Test to Production. Mostly by hand. Badly”.

As someone from a predominately Operations background I find this quite frustrating!

Operations is a discipline, with its own patterns & practices, methodologies, models, tools, technology etc. Just because modern cloud hosting makes it easier to deploy servers without having to know one end of a SCSI cable from another doesn’t mean you “know” how to do Operations (just like my knowledge of SQL is enough to find out the information I need to know monitor and manage the environment but a long way from what’s required to develop a complex, high-performing website).

DevOps means Development and Operations working together collaboratively to put the Operations requirements about stability, reliability, performance into the Development practices, whilst at the same time bringing Development in the management of the Production environment (e.g. by putting them on-call, or by leveraging their development skills to help automate key processes).

It doesn’t mean a return to the laissez-faire “anything goes” model where developers have unfettered access to the Production environment 24x7x365 and can change things as and when they like.

Change control was invented for a reason, and whilst change control has becomes its own “cottage industry” involving ever more bureaucratic layers of “management by form filling” the basic discipline remains sound – think about what you want to change, automate it if you can, test it, understand what to do if it screws up (roll back plan), document the change, make sure everyone knows when, where and how you are making the change, and make sure the business owner approves.

When I took over the Operations of a high-volume UK website about 8 years ago I spend the first 3 weekend working fighting fires and troubleshooting Production issues.

My first action after that baptism of fire was to revoke access to production for all developers (over howls of protests). Availability and stability immediately went up. Deafening silence from Development – Invitations to beers from the Business owners.

Next step was to hire a build manager to take over the Build and Deployment automation, and a Release Manager to coordinate with the Business what was going into each release, when etc. End result – 99.98% availability, with more releases, being deployed within business hours without impacting the users, and a lower TCO. The Business was much happier, and so was the Development Manager, as he was losing far fewer developer-hours to fire-fighting Production issues, and hence the overall development velocity improved considerably. Win-Win.

Was that a DevOps anti-pattern? Did I create more silos? Probably… but in a fire-fight a battlefield commander doesn’t sit everyone down for a sharing circle on how they are going to address the mutual challenge of killing the other guy before he kills you. Sometimes a command & control model is the right one for the challenge you face (like getting some supressing fire on the target while you radio in for some air support or artillery!).

That said, once we had developed a measure of stability we did move partway to a more DevOps pattern – we had developers on-call 24×7 as 3rd line support, we virtualised our environment(s) and gave Developers more control over them, and we increased our use of automation.

Organisationally we remained siloed however – we were incentivised in different ways (Operations emphasising availability, Development emphasising feature delivery), we remained in essentially a waterfall delivery model and Ops VS Dev was a constant struggle for manpower & resources. All the usual problems that the DevOps movement is trying to address.

In summary, what I am trying to get at is please don’t devalue the “DevOps” concept by saying you do DevOps when you don’t.

Unless you currently do both Development AND Operations separately, and do them well, AND you’re now trying to synthesise a better, more agile, more cloud-oriented way of working that takes the best part of BOTH disciplines… you aren’t doing DevOps!

– TheOpsMgr

About these ads

5 thoughts on “DevOps does not equal “Developers managing Production””

  1. Heya! I know this is kind of off-topic but I needed to ask.
    Does operating a well-established blog such as yours require a massive amount work?

    I am completely new to writing a blog but I do
    write in my journal everyday. I’d like to start a blog so I can easily share my personal experience and thoughts online.
    Please let me know if you have any kind of ideas or tips for brand new aspiring bloggers.
    Thankyou!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s