Tag Archives: Flow

The word flow made out of pebbles

5 useful #Devops lessons about flow you can learn while getting off a plane.

One of our favourite books here at DevOpsGuys is Donald Reinertsen’s “Principles of Product Development Flow”. It’s a compelling argument about the importance of “flow” – small batch sizes, frequent releases, rapid feedback, all the key tenets of continuous delivery and DevOps.

It is, sadly, not the sort of easily consumed management book that might appeal to your ADD-afflicted manager (should you be able to tear his/her attention away from his/her iPhone or iPad).

@TheDevMgr and myself (@TheOpsMgr) were discussing this last week as we landed at Munich airport to attend the @Jetbrains partner conference (we’re huge fans of TeamCity for continuous integration).

As we went thru the process of disembarking from the flight we realised we were in the middle of a real-world analogy for benefits of flow – an analogy that might be readily understandable to those senior managers we mentioned earlier.

Let’s walk thru it.

The plane lands and reaches the stand. Immediately everyone in the aisles seats un-clicks and stands up, congesting the aisle. Once the aisle is congested (=high utilisation) the ability of people to collect luggage from the overhead lockers is significantly reduced – everything becomes less efficient.

At high rates of utilisation of any resource waiting times, thrashing between tasks and the potential for disruption are likely to go up. So that’s Useful lesson #1.

All this activity and jamming of the aisle is, ultimately, futile because no-one is going anywhere until the stairs get there and the cabin doors are opened. This is the next constraint in the pipeline.

Useful lesson #2 – until you understand the constraints you might be just rushing headlong at the next bottleneck.

Eventually they arrive and we all start shuffling off like sheep (or zombies) and walk down the stairs… to the waiting buses on the tarmac.

Useful lesson #3 – if you’re trying to optimise flow you need to look beyond the immediate constraint.

In this, case the cabin door & stairs, and look at the entire system (this is the essential message of systems thinking and the “1st way of DevOps”).

The buses were fairly large and held about 60+ people each (=large batch size), and everyone tries to cram onto the first bus… which then waits until the second bus is full before both buses head across the tarmac. When we reach the terminal the buses park up… and the second bus is actually closer to the door we need to enter.

Useful lesson #4 – don’t assume that the batch size is what you think it is (i.e. 1 bus load). It might be more (2 buses!). Also, just because something looks FIFO doesn’t mean it is…

Once we enter the terminal we then hit another constraint – clearing Immigration control. Luckily we were able to go thru the EU Resident’s queue, which flowed at a fairly constant rate due to the minimal border control. But looking at the non-EU Residents queue that the flow was turbulent – some passengers went thru quickly but others took much longer to process due to their different nationality, visa requirements or whatever had caught the Border Control officer’s attention. Unfortunately, the people who could have been processed faster were stuck in the queue behind the “complex” processing.

Useful lesson #5 – If you can break your “unit of work” down to a relatively standard size the overall flow through the system is likely to improve. This is why most Scrum teams insist that a given user story shouldn’t’ take more than 2-3 days to complete, and if it would it then it should be split up into separate stories until it does.

Luckily we avoided the queue for checked luggage as we only had carry-on and we were able to get in the queue for the taxis instead… so that’s the end of our analogy for now.

So let’s think of some theoretical optimisations to improve the flow.

Firstly, why not only let the people on ONE side of the aisle stand to collect their overhead luggage and prepare to disembark, thereby avoiding the congestion in the aisle? You can then alternate sides until everyone’s off.

Secondly, why not see if you can get a second set of stairs so you can disembark from the forward AND rear cabin doors, and alleviate that bottleneck?

Thirdly, why not have smaller buses, and dispatch them immediately they are full, and thereby reduce the batch size that arrives at Immigration?

Fourthly, why not have more agents at Border Control to alleviate that bottleneck, or create different queue classes to try to optimise flow e.g. EU Residents, “Other countries we vaguely trust and generally wave through” and “countries we generally give a hard time to”. You could even have a special queue for “dodgy looking people from whatever nationality that are about to spend some quality time with a rubber glove”. Or why not create totally new categories like “those we hand luggage” versus “those with checked luggage who are only going to have to wait at the luggage carousel anyway so you might as well wait here anyway”.

These proposed optimisations might be simplistic. For example the reason the two buses leave together is probably because ground traffic at airports is strictly controlled (in fact there is a “ground traffic controller” just the same as an “air traffic controller”). So there are often constraints we might not be aware of until we do more investigation BUT the goal of any DevOps organisation should be to try and identify the constraints, and experiment with different ways to alleviate that constraint. Try something, learn from it, iterate around again.

Hopefully by using a common, real-world analogy for product development flow you’ll be able to convince your Boss to let you apply these principles to your DevOps delivery pipeline and improve flow within your organisation!
Photo credit: aselundblad / Foter / Creative Commons Attribution-NonCommercial-ShareAlike 2.0 Generic (CC BY-NC-SA 2.0)

AdamSmith

DevOps, Adam Smith and the legend of the Generalist

Is DevOps at risk of being misinterpreted as “everyone should be able to do everything”? 

That somehow DevOps means you should be able to switch seamlessly between Development and Operations tasks, writing some Java application code in the morning, re-configuring the SAN in the afternoon and finishing off with a bit of light firewall maintenance in the evening ?

Take this great quote;

I’ll tell you EXACTLY what DevOps means.

DevOps means giving a shit about your job enough to not pass the buck. DevOps means giving a s**t about your job enough to want to learn all the parts and not just your little world.

Developers need to understand infrastructure. Operations people need to understand code. People need to f***ing work with each other and not just occupy space next to each other. -  John E. Vincent (@Lusis)

Whilst we whole heartedly agree with the sentiments in the quote I think that there is a risk that the phrase “need to know” about Development or Operations will be misinterpreted by some as “we need generalist staff who can do it all”.

Sadly, this would be a tragic mistake for several reasons but first and foremost because division of labour exists for a reason.

There is literally hundreds of years of economic theory, dating back to Adam Smith’s “Wealth of Nations” in 1776 and beyond, that shows how & why division of labour increases overall productivity.

Whether it’s making pins, cars or software applications the correctly co-ordinated activity between a sequence of specialists will massively increase overall productivity. In the context of making pins Smith quoted an increase in productivity of 24,000% (in comparison to one person performing all of the tasks required to make a pin).

Note that there is a critical caveat in the paragraph above – “correctly coordinated”.

Poorly coordinated activities between specialists impedes productivity and creates bottlenecks, mistakes and re-work. Reinert’s ideas of “flow” in product development, Goldratt’s Theory of Constraints (TOC) and the whole Lean movement are about ensuring that activities in the “value chain” are correctly aligned and smoothly flow from one stage (or specialism) to another to ensure that maximal productivity and value are achieved.

So what does this mean in terms of DevOps and the quote above?

Coordination requires effective communication so I think that the quote above could be re-stated to say “need to know enough about Infrastructure/software development in order to be able to communicate effectively with other specialists”.

For example, Developer’s need to know enough about infrastructure and web operations to understand why a scalable user session state management solution is critical when running a distributed system because “just running it all on one box like we do in my dev environment” isn’t a good solution. Conversely Operations need to know enough about the language stack they support to be able to read a stack trace as part of their troubleshooting workflows to help identify the root cause of a problem.

So whilst hopefully agreeing that DevOps doesn’t mean the end of specialists performing specialized skills and ergo the creation of hybrid generalists it’s worth mentioning that Adam Smith did have some warnings on the downsides of over-specialization that have some resonance for DevOps:

“The man whose whole life is spent in performing a few simple operations, of which the effects are perhaps always the same, or very nearly the same, has no occasion to exert his understanding or to exercise his invention in finding out expedients for removing difficulties which never occur. He naturally loses, therefore, the habit of such exertion, and generally becomes as stupid and ignorant as it is possible for a human creature to become. The torpor of his mind renders him not only incapable of relishing or bearing a part in any rational conversation, but of conceiving any generous, noble, or tender sentiment, and consequently of forming any just judgment concerning many even of the ordinary duties of private life” – Adam Smith (Wealth of Nations)

Alexis de Tocqueville agreed with Smith:

“Nothing tends to materialize man, and to deprive his work of the faintest trace of mind, more than extreme division of labour.” Alexis de Tocqueville

Smith warned that workers “become ignorant and insular as their working lives are confined to a single repetitive task” which is the silo-oriented mental state that @Lusis rails against in the opening quote when he says DevOps “means giving a shit about your job enough to not pass the buck”.

DevOps “First Way” emphasises “Systems Thinking” – looking at the larger system as a whole not just your narrow silo view – so the risks raised by Smith and De Tocqueville are valid concerns for DevOps.

So there is a tension here… we need the productivity of specialisation but we don’t want the insular worldview of “extreme division of labour”.

Scrum in Agile software development addresses similar concerns by emphasising cross-functional teams, “Individuals and interactions over processes and tools” and generally focussing on high-bandwidth communication as a way to tear down silos.

DevOps can (and should) emphasise the same solutions to resolve this tension.