Tag Archives: ITIL

What does #DevOps mean to the roles of Change & Release managers?

What does #DevOps mean to the roles of Change & Release managers?

One of our team raised this question in our internal #DevOps Slack channel this week and it sparked off an interesting discussion that we thought was worth sharing with a wider audience.

Firstly, let’s start with one of my favourite definitions of DevOps:

“DevOps is just ITIL with 90% of stuff moved to ‘Standard Change’ because we automated the crap out of it” – TheOpsMgr

Now that’s a bit tongue in cheek, obviously, as the scope of DevOps in a CALMS model world is probably wider than just that but it’s not a bad way to start explaining it to someone from a long-term ITIL background.

They (should) know that a Standard Change is:

“Standard Changes are pre-approved changes that are considered relatively low risk, are performed frequently, and follow a documented (and Change Management approved) process.
Think standard, as in, ‘done according to the approved, standard processes.”
http://itsmtransition.com/2014/03/name-difference-standard-normal-changes-itil/

If we break that down into the 3 main parts we get:

  • relatively low risk” – DevOps reduces the risks via automation, test-driven development (of application AND infrastructure code), rapid detection of issues via enhanced monitoring and robust rollback.
  • are performed frequently” – this is a key tenet of DevOps… if it’s painful, do it more often, learn to do it better and stop trying to hide the pain away!
  • follow a documented (and Change Management approved) process” – DevOps is about build robust digital supply chains. This is your highly automated, end-to-end process for software development, testing, deployment and support and as part of that we are building in the necessary checks and balances required for compliance to change management processes. It’s just instead of having reams of documentation the automated workflow *IS* the documented process.

This starts to point us in the direction of how the role of the Change & Release Manager will change over time.

In the same way that the role of IT Operations will become more like Process Engineering in a factory the role of Change & Release Management will start to focus on more on the digital supply chain & software production line rather than trying to inspect every single package that flows along that pipeline.

In this way they become more like quality control engineers in that their job becomes to work with the process engineers to design the pipeline in such a way that what comes out the other end meets the quality, risk and compliance criteria to be considered a “Standard Change”.

They also need to move towards more sampling-based, audit-style approach for ensuring that people aren’t skipping steps and gaming the systems.

For example, if the approved process says that all changes must pass automated regression testing they might periodically pick one release and review that the regression suite was actually testing meaningful cases and hadn’t been replaced with a return:true; because no-one could be bothered to keep it up do date.

Similarly, if the process mandates “separation of duties” they could check so see that the person who initiates a change (via the pipeline) isn’t also in the same role required to approve the Jira ticket that Jenkin’s checks to see if that release can be deployed to Live.

The overall goal here is to move towards a “High Trust Culture” but keeping in mind the mantra of “Trust, but Verify” they need to work with DevOps to ensure the appropriate checks and balances.

That, and never, ever invite me to a CAB meeting, ever again. :-)

-TheOpsMgr

DevOps, Antifragility and the Borg Collective

Whilst researching how to reconcile ITIL with DevOps I came across this interesting blog post from the IT Skeptic entitled “Kamu: a unified theory of IT management – reconciling DevOps and ITSM/ITIL”. This lead me to Jez Humble’s post on “On Antifragility in Systems and Organizational Architecture” referencing Nicholas Taleb’s book “Antifragile” and generally lead to a lot of intense cogitation on fragility versus robustness versus antifragility.

The IT Skeptic (Rob England) expands on his thoughts in this presentation which introduces this diagram below

Kamu: reconciling DevOps and ITSM/ITIL

However I struggled to mentally conceptualise the differences between the 3 points of the triangle until I came up with the following analogies (and please bear with me while I explain my thoughts behind them!):

  • Fragile = Humpty Dumpty
  • Robust = A medieval castle
  • Anti-fragile = The Borg collective

Fragile

“Fragile” systems are those (often legacy) systems that you really, really don’t want to touch if you don’t have to! Like Humpty Dumpty, one good push and all the King’s horses and all the King’s Men and a 24 hour round-the-clock marathon from the Ops team won’t get that pile of crap system up and running again.

It’s a snowflake – not documented properly, there are dependencies you can’t trace, the hardware’s out of warranty, the platform is 3 versions behind and can’t be upgraded because of some customisation that no-one understands, the code is spaghetti and the guy that wrote it retired last year.

We all know what fragile looks like!

Robust

“Robust” systems are those that have been through the ITIL life-cycle and for most of us they are probably our pride & joy. Monitored, instrumented and well documented with their own run book and wiki pages they are highly available with redundancy at every level we “know” they can withstand the slings and arrows of outrageous fortune.

Just like a medieval castle they are impregnable. The very essence of robust!

And then comes along the “black swan” event… something we haven’t anticipated, a failure mode we can’t have foreseen, a cascade of errors that we did not plan for.

Just as our predecessor, the medieval castle owner, didn’t foresee the invention of gunpowder and cannons that reduce his impregnable castle to rubble. Just like the builders of the Maginot Line didn’t anticipate the invention of Blitzkreig and mechanised warfare nor the defenders of the dams of the Ruhr Valley a bomb that bounces.

This is the key message of Taleb’s book and Jez’s post – that the “robustness” mindset often leads to a resistance to change. As Jez explains in the context of organisations:

“The problem with robust organizations is that they resist change. They aren’t quickly killed by changes to their environment, but they don’t adapt to them either – they die slowly.” – Jez Humble

A castle is robust… but it’s fixed, immobile, and its very robustness to “normal” assaults reduces the incentives to change and adapt.

Anti-fragile

Contrast these to the “anti-fragile” system (or organisation) typified by the Borg Collective. The Borg seek out new life and new civilisations to assimilate into the Collective in order to improve.

With each change and adaptation the system (the Collective) becomes more resilient – it improves as the result of the external stress (the essence of an adaptive, evolutionary system).

Anti-fragile organisations seek out and embrace change – they are inherently “outward-focused” and seek to be continually learning, adapting and assimilating (not hiding behind the walls of their castle, content in their robust impregnability).

Likewise, DevOps seeks to be “anti-fragile” by embracing change (and disorder a la Chaos Monkey) whilst incorporating feedback mechanisms (the “3rd Way” of DevOps) to ensure that learning is correctly assimilated.

The DevOps mindset encourages continual learning; through experimentation and collaboration in order to seek to improve the current system as opposed to a codified mindset of a fixed position of “one way of doing things” implied in a formulaic, rigid ITIL worldview.

In this way DevOps encourages what Schumpeter called “creative destruction” – clearing out the old to make way for the new (and hopefully improved) system.

Summary

I’ve summarised these 3 points of the triangle into the following table;

 

Fragile

Robust

Anti-Fragile

Icon

Humpty Dumpty

Medieval Castle

The Borg

Methodology

“Spaghetti”

ITIL

DevOps

Attitude to change

Fear Change

Resist Change

Embrace Change

Response to change

Break

Repel

Adapt

Rate of Change

Ideally never!

Slow

Rapid

Change initiated by

Needs CEO approval

Change Management Board

User-initiated
(via automation)

Focuses on

Survival

Process*

Business Value

* Yes, I know that ITIL v3 in particular *IS* in theory very focused on business value and benefits realisation BUT in my experience the end result of an “ITIL implementation” is often the triumph process over outcome.

If anyone has any ideas for more rows to add to the table please let us know on the comments!

-TheOpsMgr