One of our favourite books here at DevOpsGuys is Donald Reinertsen’s “Principles of Product Development Flow”. It’s a compelling argument about the importance of “flow” – small batch sizes, frequent releases, rapid feedback, all the key tenets of continuous delivery and DevOps.
It is, sadly, not the sort of easily consumed management book that might appeal to your ADD-afflicted manager (should you be able to tear his/her attention away from his/her iPhone or iPad).
@TheDevMgr and myself (@TheOpsMgr) were discussing this last week as we landed at Munich airport to attend the @Jetbrains partner conference (we’re huge fans of TeamCity for continuous integration).
As we went thru the process of disembarking from the flight we realised we were in the middle of a real-world analogy for benefits of flow – an analogy that might be readily understandable to those senior managers we mentioned earlier.
Let’s walk thru it.
The plane lands and reaches the stand. Immediately everyone in the aisles seats un-clicks and stands up, congesting the aisle. Once the aisle is congested (=high utilisation) the ability of people to collect luggage from the overhead lockers is significantly reduced – everything becomes less efficient.
At high rates of utilisation of any resource waiting times, thrashing between tasks and the potential for disruption are likely to go up. So that’s Useful lesson #1.
All this activity and jamming of the aisle is, ultimately, futile because no-one is going anywhere until the stairs get there and the cabin doors are opened. This is the next constraint in the pipeline.
Useful lesson #2 – until you understand the constraints you might be just rushing headlong at the next bottleneck.
Eventually they arrive and we all start shuffling off like sheep (or zombies) and walk down the stairs… to the waiting buses on the tarmac.
Useful lesson #3 – if you’re trying to optimise flow you need to look beyond the immediate constraint.
In this, case the cabin door & stairs, and look at the entire system (this is the essential message of systems thinking and the “1st way of DevOps”).
The buses were fairly large and held about 60+ people each (=large batch size), and everyone tries to cram onto the first bus… which then waits until the second bus is full before both buses head across the tarmac. When we reach the terminal the buses park up… and the second bus is actually closer to the door we need to enter.
Useful lesson #4 – don’t assume that the batch size is what you think it is (i.e. 1 bus load). It might be more (2 buses!). Also, just because something looks FIFO doesn’t mean it is…
Once we enter the terminal we then hit another constraint – clearing Immigration control. Luckily we were able to go thru the EU Resident’s queue, which flowed at a fairly constant rate due to the minimal border control. But looking at the non-EU Residents queue that the flow was turbulent – some passengers went thru quickly but others took much longer to process due to their different nationality, visa requirements or whatever had caught the Border Control officer’s attention. Unfortunately, the people who could have been processed faster were stuck in the queue behind the “complex” processing.
Useful lesson #5 – If you can break your “unit of work” down to a relatively standard size the overall flow through the system is likely to improve. This is why most Scrum teams insist that a given user story shouldn’t’ take more than 2-3 days to complete, and if it would it then it should be split up into separate stories until it does.
Luckily we avoided the queue for checked luggage as we only had carry-on and we were able to get in the queue for the taxis instead… so that’s the end of our analogy for now.
So let’s think of some theoretical optimisations to improve the flow.
Firstly, why not only let the people on ONE side of the aisle stand to collect their overhead luggage and prepare to disembark, thereby avoiding the congestion in the aisle? You can then alternate sides until everyone’s off.
Secondly, why not see if you can get a second set of stairs so you can disembark from the forward AND rear cabin doors, and alleviate that bottleneck?
Thirdly, why not have smaller buses, and dispatch them immediately they are full, and thereby reduce the batch size that arrives at Immigration?
Fourthly, why not have more agents at Border Control to alleviate that bottleneck, or create different queue classes to try to optimise flow e.g. EU Residents, “Other countries we vaguely trust and generally wave through” and “countries we generally give a hard time to”. You could even have a special queue for “dodgy looking people from whatever nationality that are about to spend some quality time with a rubber glove”. Or why not create totally new categories like “those we hand luggage” versus “those with checked luggage who are only going to have to wait at the luggage carousel anyway so you might as well wait here anyway”.
These proposed optimisations might be simplistic. For example the reason the two buses leave together is probably because ground traffic at airports is strictly controlled (in fact there is a “ground traffic controller” just the same as an “air traffic controller”). So there are often constraints we might not be aware of until we do more investigation BUT the goal of any DevOps organisation should be to try and identify the constraints, and experiment with different ways to alleviate that constraint. Try something, learn from it, iterate around again.
DevOps … is a movement, a philosophy, a way of thinking.
DevOps … is a person who can perform both Dev and Ops roles.
DevOps … means cross skilling people.
DevOps … is continuous delivery.
DevOps … is a team of developers and operation staff.
DevOps …is a culture movement.
DevOps … is monitoring.
So… what’s the “minimum viable product” (MVP) for DevOps? What core things should you be doing before you can truly say you are “doing DevOps”?
(1) “ Emphasize the performance of the entire system” – a holistic viewpoint from requirements all the way through to Operations
(2) “Creating feedback loops” – to ensure that corrections can continually be made. A TQM philosophy, basically.
(3) “Creating a culture that fosters continual experimentation and understanding that repetition and practice are the pre-requisites to mastery”
These are excellent guidelines at a high level, but we’d like to see a more operational definition. So we’ve made up our own list!
As a starter – we propose that;
You must have identified executive sponsors / stake holders who you are actively working with to promote the DevOps approach.
You must have developed a clear understanding of your organisation’s “value chain” and how value is created (or destroyed) along that chain.
You must have organizationally re-structured your development and operations teams to create an integrated team – otherwise you’re still in Silos.
You must have changed your team incentives (e.g. bonus incentives) to reinforce that re-alignment – without shared Goals you’re still in Silos.
You must be seeking repeatable standardized processes for all key activities along the value chain (the “pre-requisite to mastery”)
You must be leveraging automation where possible – including continuous integration, automated deployments and “infrastructure as code”
You must be adopting robust processes to measure key metrics – PuppetLab’s report focuses on improvement in 4 key metrics – Change Frequency, Change Lead Time, Change Failure Rate and MTTR. We suggest Availability, Performance and MTBF should be in there too.
You must have identified well-defined feedback mechanisms to create continuous improvement.
As mentioned above, this is just a starter list – feel free to agree/disagree in the comments and suggest additions or alterations.
We’ll be writing more about “DevOps Incentives” in an upcoming post, and we’ll revisit the “Are you doing DevOps?” topic once we’ve consolidated your feedback.
In my first ever development role I was luckily enough to work in a completely virtualised environment.
I could, with a single script, build a complete development environment just for me. I could compile my code and deploy into my test environment automatically, and when I was testing – I had tools I could use to trace execution flow down every path in my code. If I needed any assistance, there was a dedicated Operations Guy who was an expert in Automation who sat 3 desks away and could answer all my problems.
DevOps Heaven, right?
This was in 1990.
The operating system was IBM MVS/ESA running on VM/370 (which pretty much invented the term “hypervisor”), and the “Operations Guy” was called a “systems programmer” (or “SysProg” for short). At the time we were writing NetView which I guess was a kind of Nagios equivalent for VTAM/SNA networks, although it also had automation capabilities a la ScienceLogic. Whilst we were at it we also wrote an in-memory, object-oriented, non-relational (noSQL) database called RODM (Resource Object Data Manager) thereby pre-empting “innovations” like memcached, Couchbase and the current noSQL movement by about 20 years… but I digress!
“What has been is what will be, and what has been done is what will be done, and there is nothing new under the sun.” (Ecclesiastes 1:9).
Control within organisations tends to moves in cycles from centralised command/control mindsets to distributed autonomous teams and back again, in the same way computing resources have moved from centralised mainframes to distributed PC’s and now are moving back centrally again as “The Cloud”.
Make no mistake, I am not denigrating in any way the goals of the DevOps movement (or the Cloud for that matter) by pointing out that some of it might have been done before! Each expansion and contraction, dispersion and centralisation, of computing power needs to be matched with the correct management and organisational models.
I firmly believe that removing the silos within IT Departments and building cross-functional autonomous teams with end-to-end responsibility is the correct model to match the incredible flexibility we now possess with “on-demand” Cloud Computing (although I might mention that the term “on-demand” was an IBM slogan in 2002…).
“To err is human” – automation can reduce human error that leads to expensive downtime (“Downtime caused by incorrect manual configuration estimated at £48K/hr or $72K/hr in 2012”) but nothing, and I mean nothing, seriously screws things up like getting your automation wrong. Want every application to suddenly stop working simultaneously? Yup, that change-password script didn’t quite work the way you planned.
Back in the day we had a job role called an “Operator” – which was a soul destroying shift job basically watching batch, automated processes run. So for everyone whose “empowered” by DevOps make sure you aren’t creating a under-class at the same time!
Aligning reward and recognition – just because you’ve created your awesome cross-functional DevOps team doesn’t mean the organisation’s HR function has kept pace. Nothing destroys team cohesion faster that misaligned reward and recognition. It’s worth noting that lessons from anthropology show us that primates have an innate sense of fairness/justice… so it’s not enough to just ensure that the “good” get rewarded, it’s equally important that the “bad” get punished (or sacked). If management doesn’t control this the “group” will by means of in-group formation and ostracism. Both of which are anathema to high-performance teams.
Make room for exceptions. Not everyone will want to work in an Agile DevOps world. Back in the day we had a short, bitingly sarcastic, New Yorker on our team. He wouldn’t be my first pick for a multi-disciplinary DevOps team… but man that boy could code! I’m talking the kinda code that made something hideously complex look blindingly obvious and had everyone slapping themselves upside that head saying “gee, why didn’t I think of that?”. Well, probably cos he was just better than you, numbnuts! So, make some space in the corner for misfits that don’t play nice with others and work out a way to incorporate their strengths into your DevOps model while patching over their weaknesses.
Make environment building part of the Development pattern – I saw a tweet the other day that said “Dev: “It works on my machine, just not on the server.” Me: “Ok, backup your mail. We’re putting your laptop into production.” – For me, this is one of the great messages of DevOps – get the Operational requirements for the run-time environment written into the Development process as user stories and ensure that it DOES work (in your virtualised, “infrastructure as code” environment) before you say it’s ready. Gene Kim calls this his “DevOps favourite pattern #3”. That’s what the SysProg did for us, back in the day, and it works.
What lessons do you have from “the good old days” that might be relevant to the DevOps movement?
Continuous Delivery is a set of practices and principles aimed at building, testing and releasing software faster and more frequently.
As more teams embrace the DevOps methodology and implement continuous delivery, more platforms and services will move toward a self-service model that encourages this collaboration. But what are the barriers to adoption? For the next month, we’re running a poll to try and understand, what you think prevents companies from adopting continuous delivery.
The Maturity Model is a useful assessment tool for understanding your organizations level of Continuous Delivery adoption. Many organizations today have achieved what is needed to move from Level-1 (Regressive) to Level-0 (Repeatable), which is a significant accomplishment and as a reader of this blog post, you’re either about to start your journey of improvement or are already underway.
The advice for organizations wanting to adopt Continuous Delivery is ever more abundant, but for organizations that started adoption some time ago, the guidance on how to mature the process is still sparse. In this article, we explore one continuous improvement methodology that may help your organization mature its’ Continuous Delivery process.
Humble and Farley outline maturity Level 0 (Repeatable) – as one having process which is documented and partly automated.1 For this to be true, an organization must have first classified its’ software delivery maturity, identified areas for improvement, implemented some change and measured the effect. As Humble observes;
The deployment pipeline is the key pattern that enables continuous delivery.2
Humble also identifies that Deming’s Cycle is a good process to apply to initial adoption. 1 The process, according to Deming, should then be repeated so that further improvements can be planned and implemented; having the advantage the data and experience from previous cycles is available. This process of continuous improvement is the first step to maturing the continuous delivery process.
Continuous Delivery and Kaizen
Continuous Delivery is a set of practices and principles aimed at building, testing and releasing software faster and more frequently.3 One key philosophy at the heart of Continuous Delivery, is Kaizen. The Japanese word, Kaizen, means continuous improvement. By improving standardized activities and processes, Kaizen aims to eliminate waste. Within Kaizen are three major principles;
The 5S principle characterizes a continuous practice for creating and maintaining an organized and high-performance delivery environment. As 5S has the focus of waste reduction, it appears prudent to explore the 5s methodology as an approach to optimizing the Continuous Delivery pipeline.
What is 5S?
5S is a lean engineering principle focused on waste reduction and removal. The methodology is recursive and continuous and it’s primary goal is to make problems visible so that they can be addressed.4
There are five primary 5S phases, derived from five Japanese words that translate to; sort (seiri), straighten (seiton), shine (seiso), standardize (seiketsu) and sustain (shitsuke).5
The 5s principle is more commonly applied to workplace organization or housekeeping. Since the delivery pipeline is a key component of the process, it is therefore a fundamental place where work will take place. The pipeline exhibits elements that are common with any physical work environment including;
Tools, process, paperwork.
Storage areas for artifacts, such as files.
A requirement that components must be, physically or virtually, located in the correct location, and that they are connected by process.
Tools and process need maintenance and cleaning. E.g. retention policies.
Systems and procedures can be standardized and can benefit from standardization.
When applied to the delivery pipeline, the five primary 5S phases can be defined as;
Phase 1 – sort (seiri)
Analysis of processes and tools used in the delivery pipeline. Tools and processes not adding value should be removed.
Phase 2 – straighten (seiton)
Organization of processes and tools to promote pipeline efficiency.
Phase 3 – shine (seiso)
Inspection and pro-active/preventative maintenance to ensure clean operation throughout the delivery pipeline.
Phase 4 – standardize (seiketsu)
Standardization of working practice and operating procedure to ensure consistent delivery throughout pipeline.
Phase 5 – sustain (shitsuke)
Commitment to maintain standards and to practice the first 4S. Once the 4S’s have been established, they become the new way to operate.
Implementing 5s in the Delivery Pipeline
Experience has shown that there are several key factors that should be considered in the adoption of 5S.
Begin with small changes. Remember that 5s is a continuous and recursive process.
Involvement is crucial.6 Encompass everyone involved in the deployment pipeline.
Create Champions – find leaders who love & live the philosophy and encourage them to promote it.
5s implementation involves working through each phase (or S) in a methodical way. This is commonly achieved by Kaizen events, which are focused activities, designed to quickly identify and remove wasteful process from the value stream and are one of the prevalent approaches to continuous improvement.
Phase 1 – sort (seiri)
The first step of 5s is to sort. The purpose is to identify what is not needed in the delivery pipeline and remove it.
Additional process and practice can easily creep into the delivery mechanism, especially in the early phases of Continuous Delivery adoption, when you are experimenting.
The removal of items can either mean total elimination of a tool or process, since it is redundant, or the flagging, known as “Red Tagging”, of a tool or process which needs to be evaluated before it is disposed of. The review of red tagged tools and process should evaluate their importance and determine if they are truly required.
1. Evaluate all tools and process in the delivery pipeline and determine what is mandatory.
2. Ascertain what is not needed. Assess what it’s elimination status is, e.g. immediately or Red Flag.
3. Remove unnecessary or wasteful tools and process.
Phase 2 – straighten (seiton)
The second step requires items in the delivery pipeline to be set in order. This requires the pipeline to be arranged so that maximum efficiency is achieved for delivery.
To achieve seiton, the key components of the automated pipeline should be analysed, and should include the following;
Automated build process.
Automated unit, acceptance tests including code analysis.
Automated deployment process.
It is suggested that for the pipeline to be truly efficient the most commonly used items are the easiest and quickest to locate, such as;
Build status reports and metrics including KPI’s such as Mean-time-to-release (MTTR)
Automated test coverage, error and defect reports
In this step, actions can also be taken to visually label elements of the pipeline so that demarcation of responsibility is clear. This may mean categorizing tools, process and target locations E.g. servers into groups such as production and pre-production or development and test.
1. Analyse the value stream including tools and process.
2. Ensure the necessary information is available quickly to those who need it.
3. Visually distinguish or categorize key pipeline components.
Phase 3 – shine (seiso)
Once the waste has been removed and only what is required has been kept and ordered, the next phase inspects the delivery pipeline to ensure that it’s operation is as clean as possible.
During pipeline execution large amounts of data can be produced. If this information is not dealt with efficiently it can become a waste product, detracting from the value of the delivery process.
Inspection of this data is vital. Data generated in the pipeline is likely to include instrumentation, metrics and logging information, as well as build artifacts and report data.
In context of the delivery pipeline, cleaning can be defined as archiving, roll-up or removal of delivery pipeline data. House keeping routines should be evaluated to ensure that data retention rules are applied and that waste, in the form of build artifacts, metric and report data and instrumentation data are correctly archived or removed as required.
Implementation of this phase requires assignment of cleaning responsibilities and scheduling of cleaning process.
1. Inspect data to understand what should be rolled-up, archived or deleted.
2. Ensure that responsibilities for data management are clearly assigned within the team.
3. Create a scheduled, automated cleaning processes.
Phase 4 – standardize (seiketsu)
The primary focus of step four is to ensure that the working practice, training and operating procedures remain consistent. Standardization allows for continuous improvement and each of the first 3S’s should be regulated.
A critical element of seiketsu is to ensure that the process is reviewed on a regular basis or cycle. This ensures that best practices are maintained and areas where standards have slipped are quickly identified.
A settled and regular process or practice becomes one that is hard to give up. Therefore, if standardization, is implemented correctly, it can support culture change.
1. Ensure that the team have clear responsibilities for the tasks that need to be completed.
2. Create ownership and pride in the process.
3. Ensure champions provide guidance and support changes to the 5s process.
Phase 5 – sustain (shitsuke)
The purpose of the final phase is to ensure that the 5s process is not a one-time event. To achieve this, the focus of the fifth phase centers on culture. To sustain 5s organizations must engage in support, commitment and gainful communication, with the aim of keeping the team devoted to continuous improvement.
There are a number of elements which influence cultural support of 5s . Which elements take precedence varies within every organization and is heavily dependent on the existing culture of that organization. Key points are:
Communication – aims & objectives need to be clear for all team members.
Education – 5s methodology, concepts and practices need to be communicated.
Recognition – team members need to feel that their efforts are recognized.
It is interesting to note that the literal translation of shitsuke is discipline. The final phase of 5s can also be one of the hardest to implement.
1. Establish an open channel for discussion and feedback, to facilitate continual learning.
2. Adopt the habit of auditing regularly and reward the best teams.
3. Ensure that all problems are addressed quickly make sure that root cause analysis is completed to discover the cause.
The adoption of Kaizen events to implement 5s within the Continuous Delivery pipeline can provide significant support to maturing the model, by providing a continuous improvement process within a well-defined and standardized structure.
Lean manufacturing has been using the 5S pratice as a common method for lean implementation for years. Nearly 70% of of lean manufacturing companies use this approach 7, but the use of 5S as an adoption method for lean in software development would appear less common, as it has not be commonly applied to development practices.
Lean software development principles provide the promise of companies becoming more competitive by delivering software faster, increasing quality, and reducing cost leading to greater customer satisfaction, all of which align closely to the key principles of Continuous Delivery.
1. Humble, J. Farley D. (2010). Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Indiana: Addison Wesley. 419.