I’ve had a few conversations lately, mainly with smaller start-ups or development houses, who tell me “yes, we work in a DevOps model”.
What they really mean is “We pretty much have no Operations capability at all, and we rely on the Developers to build, deploy and manage all of the environments from Development to Test to Production. Mostly by hand. Badly”.
As someone from a predominately Operations background I find this quite frustrating!
Operations is a discipline, with its own patterns & practices, methodologies, models, tools, technology etc. Just because modern cloud hosting makes it easier to deploy servers without having to know one end of a SCSI cable from another doesn’t mean you “know” how to do Operations (just like my knowledge of SQL is enough to find out the information I need to know monitor and manage the environment but a long way from what’s required to develop a complex, high-performing website).
DevOps means Development and Operations working together collaboratively to put the Operations requirements about stability, reliability, performance into the Development practices, whilst at the same time bringing Development in the management of the Production environment (e.g. by putting them on-call, or by leveraging their development skills to help automate key processes).
It doesn’t mean a return to the laissez-faire “anything goes” model where developers have unfettered access to the Production environment 24x7x365 and can change things as and when they like.
Change control was invented for a reason, and whilst change control has becomes its own “cottage industry” involving ever more bureaucratic layers of “management by form filling” the basic discipline remains sound – think about what you want to change, automate it if you can, test it, understand what to do if it screws up (roll back plan), document the change, make sure everyone knows when, where and how you are making the change, and make sure the business owner approves.
When I took over the Operations of a high-volume UK website about 8 years ago I spend the first 3 weekend working fighting fires and troubleshooting Production issues.
My first action after that baptism of fire was to revoke access to production for all developers (over howls of protests). Availability and stability immediately went up. Deafening silence from Development – Invitations to beers from the Business owners.
Next step was to hire a build manager to take over the Build and Deployment automation, and a Release Manager to coordinate with the Business what was going into each release, when etc. End result – 99.98% availability, with more releases, being deployed within business hours without impacting the users, and a lower TCO. The Business was much happier, and so was the Development Manager, as he was losing far fewer developer-hours to fire-fighting Production issues, and hence the overall development velocity improved considerably. Win-Win.
Was that a DevOps anti-pattern? Did I create more silos? Probably… but in a fire-fight a battlefield commander doesn’t sit everyone down for a sharing circle on how they are going to address the mutual challenge of killing the other guy before he kills you. Sometimes a command & control model is the right one for the challenge you face (like getting some supressing fire on the target while you radio in for some air support or artillery!).
That said, once we had developed a measure of stability we did move partway to a more DevOps pattern – we had developers on-call 24×7 as 3rd line support, we virtualised our environment(s) and gave Developers more control over them, and we increased our use of automation.
Organisationally we remained siloed however – we were incentivised in different ways (Operations emphasising availability, Development emphasising feature delivery), we remained in essentially a waterfall delivery model and Ops VS Dev was a constant struggle for manpower & resources. All the usual problems that the DevOps movement is trying to address.
In summary, what I am trying to get at is please don’t devalue the “DevOps” concept by saying you do DevOps when you don’t.
Unless you currently do both Development AND Operations separately, and do them well, AND you’re now trying to synthesise a better, more agile, more cloud-oriented way of working that takes the best part of BOTH disciplines… you aren’t doing DevOps!
The Maturity Model is a useful assessment tool for understanding your organizations level of Continuous Delivery adoption. Many organizations today have achieved what is needed to move from Level-1 (Regressive) to Level-0 (Repeatable), which is a significant accomplishment and as a reader of this blog post, you’re either about to start your journey of improvement or are already underway.
The advice for organizations wanting to adopt Continuous Delivery is ever more abundant, but for organizations that started adoption some time ago, the guidance on how to mature the process is still sparse. In this article, we explore one continuous improvement methodology that may help your organization mature its’ Continuous Delivery process.
Humble and Farley outline maturity Level 0 (Repeatable) – as one having process which is documented and partly automated.1 For this to be true, an organization must have first classified its’ software delivery maturity, identified areas for improvement, implemented some change and measured the effect. As Humble observes;
The deployment pipeline is the key pattern that enables continuous delivery.2
Humble also identifies that Deming’s Cycle is a good process to apply to initial adoption. 1 The process, according to Deming, should then be repeated so that further improvements can be planned and implemented; having the advantage the data and experience from previous cycles is available. This process of continuous improvement is the first step to maturing the continuous delivery process.
Continuous Delivery and Kaizen
Continuous Delivery is a set of practices and principles aimed at building, testing and releasing software faster and more frequently.3 One key philosophy at the heart of Continuous Delivery, is Kaizen. The Japanese word, Kaizen, means continuous improvement. By improving standardized activities and processes, Kaizen aims to eliminate waste. Within Kaizen are three major principles;
The 5S principle characterizes a continuous practice for creating and maintaining an organized and high-performance delivery environment. As 5S has the focus of waste reduction, it appears prudent to explore the 5s methodology as an approach to optimizing the Continuous Delivery pipeline.
What is 5S?
5S is a lean engineering principle focused on waste reduction and removal. The methodology is recursive and continuous and it’s primary goal is to make problems visible so that they can be addressed.4
There are five primary 5S phases, derived from five Japanese words that translate to; sort (seiri), straighten (seiton), shine (seiso), standardize (seiketsu) and sustain (shitsuke).5
The 5s principle is more commonly applied to workplace organization or housekeeping. Since the delivery pipeline is a key component of the process, it is therefore a fundamental place where work will take place. The pipeline exhibits elements that are common with any physical work environment including;
Tools, process, paperwork.
Storage areas for artifacts, such as files.
A requirement that components must be, physically or virtually, located in the correct location, and that they are connected by process.
Tools and process need maintenance and cleaning. E.g. retention policies.
Systems and procedures can be standardized and can benefit from standardization.
When applied to the delivery pipeline, the five primary 5S phases can be defined as;
Phase 1 – sort (seiri)
Analysis of processes and tools used in the delivery pipeline. Tools and processes not adding value should be removed.
Phase 2 - straighten (seiton)
Organization of processes and tools to promote pipeline efficiency.
Phase 3 – shine (seiso)
Inspection and pro-active/preventative maintenance to ensure clean operation throughout the delivery pipeline.
Phase 4 – standardize (seiketsu)
Standardization of working practice and operating procedure to ensure consistent delivery throughout pipeline.
Phase 5 – sustain (shitsuke)
Commitment to maintain standards and to practice the first 4S. Once the 4S’s have been established, they become the new way to operate.
Implementing 5s in the Delivery Pipeline
Experience has shown that there are several key factors that should be considered in the adoption of 5S.
Begin with small changes. Remember that 5s is a continuous and recursive process.
Involvement is crucial.6 Encompass everyone involved in the deployment pipeline.
Create Champions – find leaders who love & live the philosophy and encourage them to promote it.
5s implementation involves working through each phase (or S) in a methodical way. This is commonly achieved by Kaizen events, which are focused activities, designed to quickly identify and remove wasteful process from the value stream and are one of the prevalent approaches to continuous improvement.
Phase 1 – sort (seiri)
The first step of 5s is to sort. The purpose is to identify what is not needed in the delivery pipeline and remove it.
Additional process and practice can easily creep into the delivery mechanism, especially in the early phases of Continuous Delivery adoption, when you are experimenting.
The removal of items can either mean total elimination of a tool or process, since it is redundant, or the flagging, known as “Red Tagging”, of a tool or process which needs to be evaluated before it is disposed of. The review of red tagged tools and process should evaluate their importance and determine if they are truly required.
1. Evaluate all tools and process in the delivery pipeline and determine what is mandatory.
2. Ascertain what is not needed. Assess what it’s elimination status is, e.g. immediately or Red Flag.
3. Remove unnecessary or wasteful tools and process.
Phase 2 - straighten (seiton)
The second step requires items in the delivery pipeline to be set in order. This requires the pipeline to be arranged so that maximum efficiency is achieved for delivery.
To achieve seiton, the key components of the automated pipeline should be analysed, and should include the following;
Automated build process.
Automated unit, acceptance tests including code analysis.
Automated deployment process.
It is suggested that for the pipeline to be truly efficient the most commonly used items are the easiest and quickest to locate, such as;
Build status reports and metrics including KPI’s such as Mean-time-to-release (MTTR)
Automated test coverage, error and defect reports
In this step, actions can also be taken to visually label elements of the pipeline so that demarcation of responsibility is clear. This may mean categorizing tools, process and target locations E.g. servers into groups such as production and pre-production or development and test.
1. Analyse the value stream including tools and process.
2. Ensure the necessary information is available quickly to those who need it.
3. Visually distinguish or categorize key pipeline components.
Phase 3 – shine (seiso)
Once the waste has been removed and only what is required has been kept and ordered, the next phase inspects the delivery pipeline to ensure that it’s operation is as clean as possible.
During pipeline execution large amounts of data can be produced. If this information is not dealt with efficiently it can become a waste product, detracting from the value of the delivery process.
Inspection of this data is vital. Data generated in the pipeline is likely to include instrumentation, metrics and logging information, as well as build artifacts and report data.
In context of the delivery pipeline, cleaning can be defined as archiving, roll-up or removal of delivery pipeline data. House keeping routines should be evaluated to ensure that data retention rules are applied and that waste, in the form of build artifacts, metric and report data and instrumentation data are correctly archived or removed as required.
Implementation of this phase requires assignment of cleaning responsibilities and scheduling of cleaning process.
1. Inspect data to understand what should be rolled-up, archived or deleted.
2. Ensure that responsibilities for data management are clearly assigned within the team.
3. Create a scheduled, automated cleaning processes.
Phase 4 – standardize (seiketsu)
The primary focus of step four is to ensure that the working practice, training and operating procedures remain consistent. Standardization allows for continuous improvement and each of the first 3S’s should be regulated.
A critical element of seiketsu is to ensure that the process is reviewed on a regular basis or cycle. This ensures that best practices are maintained and areas where standards have slipped are quickly identified.
A settled and regular process or practice becomes one that is hard to give up. Therefore, if standardization, is implemented correctly, it can support culture change.
1. Ensure that the team have clear responsibilities for the tasks that need to be completed.
2. Create ownership and pride in the process.
3. Ensure champions provide guidance and support changes to the 5s process.
Phase 5 – sustain (shitsuke)
The purpose of the final phase is to ensure that the 5s process is not a one-time event. To achieve this, the focus of the fifth phase centers on culture. To sustain 5s organizations must engage in support, commitment and gainful communication, with the aim of keeping the team devoted to continuous improvement.
There are a number of elements which influence cultural support of 5s . Which elements take precedence varies within every organization and is heavily dependent on the existing culture of that organization. Key points are:
Communication – aims & objectives need to be clear for all team members.
Education - 5s methodology, concepts and practices need to be communicated.
Recognition - team members need to feel that their efforts are recognized.
It is interesting to note that the literal translation of shitsuke is discipline. The final phase of 5s can also be one of the hardest to implement.
1. Establish an open channel for discussion and feedback, to facilitate continual learning.
2. Adopt the habit of auditing regularly and reward the best teams.
3. Ensure that all problems are addressed quickly make sure that root cause analysis is completed to discover the cause.
The adoption of Kaizen events to implement 5s within the Continuous Delivery pipeline can provide significant support to maturing the model, by providing a continuous improvement process within a well-defined and standardized structure.
Lean manufacturing has been using the 5S pratice as a common method for lean implementation for years. Nearly 70% of of lean manufacturing companies use this approach 7, but the use of 5S as an adoption method for lean in software development would appear less common, as it has not be commonly applied to development practices.
Lean software development principles provide the promise of companies becoming more competitive by delivering software faster, increasing quality, and reducing cost leading to greater customer satisfaction, all of which align closely to the key principles of Continuous Delivery.
1. Humble, J. Farley D. (2010). Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Indiana: Addison Wesley. 419.