Tag Archives: Development

horizon

DevOps and the Knowledge Horizon

One of the things I love about SF novels is that they are always full of fascinating ideas that sometimes spark resonances in my own thinking (in this case, on DevOps).

Alastair Reynolds’ new novel “On the Steel Breeze” has this passage when discussing the rotation of Hyperion (one of Saturn’s moons):

“There’s a value in chaos theory, a number called the Lyapunov exponent, which tells you how to predict a chaotic systems boundary – it’s knowledge horizon, if you like. Hyperion’s Lyapunov exponent is just forty day – we can’t predict this moon’s motion beyond the next forty days. That’s the maximum limit of our foreknowledge! If my life depended on this moon’s motion, I would still not be able to say a word about its state beyond forty days”

My first thought when I read this was “Yep, sound like every IT project I’ve ever worked on… pretty much turns to chaos after about 40 days…”.

Now, the mathematics of the Lyapunov Exponent are way beyond my long-forgotten basics of high school calculus so I can’t comment on whether the author’s usage of the exponent is valid but the concept – that there is a limit on our foreknowledge beyond some sort of chaos event horizon – really stuck with me.

So what’s this all got to do with DevOps?

Well, “classical” project management takes a rather linear, deterministic view of the world – that we can accurately define a set of initial conditions that are complete (the mythical “requirements spec”) and that we can define a series of steps (the “waterfall process”) that will result in us achieving a desired end state (a working system that matches the specification AND the customer’s intent).

But what happens if software development is “non-linear”, chaotic and subject to “the butterfly effect”?

If “small changes to the initial conditions” (i.e. the spec is a tiny bit wrong, or more likely the customer changes their mind) can have a major impact on the end state (not to mention any of the other factors that might turn a project plan to “chaos” in the common usage of the term) does any given software project have a “Lyapunov Exponent” that limits our ability to predict the future?

I’d argue that it might… and this is intuitively what DevOps seeks to address in the “2nd Way of DevOps”

The Second Way is about creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.

By shortening the feedback cycle we, consciously or unconsciously, are seeking to bring back our project cycles within our “Lyapunov Exponent”. If we can carry out our planning cycles (Sprints in an Agile development world) within that “limit of our foreknowledge” we create a system where we can correct and tweak the “initial conditions” of our next cycle to seek to keep our project “on track”.

What I also find interesting is the idea that different projects have different “Lyapunov Exponents” depending on their degree of “non-linearity” and sensitivity to initial conditions etc.

Perhaps this is why different projects can have success (or failure) with different sprint durations.

Just because a one, two or four week sprints work for project X doesn’t mean that’s the best duration for Project Y because it has a different exponent. Part of our feedback cycle also involves finding the optimal value for our “knowledge horizon” – not so far out we descend into “chaos” but not so short we spend more time re-planning than we might need to otherwise do.

Sprint Duration < “Lyapunov Exponent” = DevOps Success?

Sadly I don’t think that there is any theoretical way to determine a project (or organisation’s) hypothetical “Lyapunov Exponent” when it comes to project planning… it has to be empirical based on trial and error, but I’d be curious to know if, even anecdotally, that for a given organisation or type of project the likelihood of a “successful project” falls off a cliff beyond some planning horizon. If there was some way to determine the “Lyapunov Expontent” for a project and optimise an organisation’s feedback cycle (or sprint duration) below that limit then I think that would be of significant benefit.

-TheOpsMgr

p.s. yes, I know I’ve played fairly fast and loose with the mathematical usage of the word “chaos” compared to its vernacular usage. I apologise to any mathematicians reading this blog but I hope you’ll forgive me in the spirit of exploring an interesting concept!

image source: - CC  -  donkeyhotey via Flickr -http://farm7.staticflickr.com/6077/6144165108_8758c2a5c5_d.jpg
AdamSmith

DevOps, Adam Smith and the legend of the Generalist

Is DevOps at risk of being misinterpreted as “everyone should be able to do everything”? 

That somehow DevOps means you should be able to switch seamlessly between Development and Operations tasks, writing some Java application code in the morning, re-configuring the SAN in the afternoon and finishing off with a bit of light firewall maintenance in the evening ?

Take this great quote;

I’ll tell you EXACTLY what DevOps means.

DevOps means giving a shit about your job enough to not pass the buck. DevOps means giving a s**t about your job enough to want to learn all the parts and not just your little world.

Developers need to understand infrastructure. Operations people need to understand code. People need to f***ing work with each other and not just occupy space next to each other. –  John E. Vincent (@Lusis)

Whilst we whole heartedly agree with the sentiments in the quote I think that there is a risk that the phrase “need to know” about Development or Operations will be misinterpreted by some as “we need generalist staff who can do it all”.

Sadly, this would be a tragic mistake for several reasons but first and foremost because division of labour exists for a reason.

There is literally hundreds of years of economic theory, dating back to Adam Smith’s “Wealth of Nations” in 1776 and beyond, that shows how & why division of labour increases overall productivity.

Whether it’s making pins, cars or software applications the correctly co-ordinated activity between a sequence of specialists will massively increase overall productivity. In the context of making pins Smith quoted an increase in productivity of 24,000% (in comparison to one person performing all of the tasks required to make a pin).

Note that there is a critical caveat in the paragraph above – “correctly coordinated”.

Poorly coordinated activities between specialists impedes productivity and creates bottlenecks, mistakes and re-work. Reinert’s ideas of “flow” in product development, Goldratt’s Theory of Constraints (TOC) and the whole Lean movement are about ensuring that activities in the “value chain” are correctly aligned and smoothly flow from one stage (or specialism) to another to ensure that maximal productivity and value are achieved.

So what does this mean in terms of DevOps and the quote above?

Coordination requires effective communication so I think that the quote above could be re-stated to say “need to know enough about Infrastructure/software development in order to be able to communicate effectively with other specialists”.

For example, Developer’s need to know enough about infrastructure and web operations to understand why a scalable user session state management solution is critical when running a distributed system because “just running it all on one box like we do in my dev environment” isn’t a good solution. Conversely Operations need to know enough about the language stack they support to be able to read a stack trace as part of their troubleshooting workflows to help identify the root cause of a problem.

So whilst hopefully agreeing that DevOps doesn’t mean the end of specialists performing specialized skills and ergo the creation of hybrid generalists it’s worth mentioning that Adam Smith did have some warnings on the downsides of over-specialization that have some resonance for DevOps:

“The man whose whole life is spent in performing a few simple operations, of which the effects are perhaps always the same, or very nearly the same, has no occasion to exert his understanding or to exercise his invention in finding out expedients for removing difficulties which never occur. He naturally loses, therefore, the habit of such exertion, and generally becomes as stupid and ignorant as it is possible for a human creature to become. The torpor of his mind renders him not only incapable of relishing or bearing a part in any rational conversation, but of conceiving any generous, noble, or tender sentiment, and consequently of forming any just judgment concerning many even of the ordinary duties of private life” – Adam Smith (Wealth of Nations)

Alexis de Tocqueville agreed with Smith:

“Nothing tends to materialize man, and to deprive his work of the faintest trace of mind, more than extreme division of labour.” Alexis de Tocqueville

Smith warned that workers “become ignorant and insular as their working lives are confined to a single repetitive task” which is the silo-oriented mental state that @Lusis rails against in the opening quote when he says DevOps “means giving a shit about your job enough to not pass the buck”.

DevOps “First Way” emphasises “Systems Thinking” – looking at the larger system as a whole not just your narrow silo view – so the risks raised by Smith and De Tocqueville are valid concerns for DevOps.

So there is a tension here… we need the productivity of specialisation but we don’t want the insular worldview of “extreme division of labour”.

Scrum in Agile software development addresses similar concerns by emphasising cross-functional teams, “Individuals and interactions over processes and tools” and generally focussing on high-bandwidth communication as a way to tear down silos.

DevOps can (and should) emphasise the same solutions to resolve this tension.

bad-idea-sign

Twelve DevOps Anti-Patterns

So you wanna do DevOps? Okay, but before you start, let’s have a look at some of the things you shouldn’t do.

In the good old days, we just called them “bad ideas”, but along came diplomacy and political correctness, out went the “brain storm” and in came the “idea shower” and with it came the “anti-pattern”.

If the “pattern” is the right way, then inherently the “anti-pattern” is the wrong – and so to stop you going wrong, we’ve compiled this list (with a little help from the DevOps community).

1. DevOps is a process

Not exactly. It’s a philosophy. It’s a way of thinking. DevOps is supported by process and tools.

DevOps according to Gene Kim, is underpinned by 3 core principles known as the “Three Ways”

The First Way emphasizes the performance of the entire system – the value stream.

The Second Way is about shorting and amplifying feedback loops.

The Third Way is about creating a culture that fosters continual learning and understanding.

http://itrevolution.com/pdf/Top11ThingsToKnowAboutDevOps.pdf

2. Agile equals DevOps?

If you’re asking this question, then you’re probably running some agile process. That’s good. You’ve got a software development process that compliments DevOps, but Agile doesn’t mean you’ve adopted DevOps.

DevOps is an agile enabler allowing operations to collaborate supporting a more continuous flow of work into IT Operations and out into production where customers can realize its value.

3. Rebrand your ops/dev/any team as the DevOps

CIO: “I want to embrace DevOps over the coming year.”

MGR: “Already ready done, we changed the department signage this morning. We are so awesome we now have 2 DevOps teams.”

Yeah great. And I bet you now have lots of “DevOps” engineers walking round too. If you’re lucky they may sit next to each other at lunch.

4. Start a separate DevOps group

Go on. I dare you. Done it? Well done. You’ve implemented DevOps. Actually what you just did is create yet another silo. Now you’ve got yourself another team you’ve got to try and integrate. Another team with walls to breakdown. Maybe you could go back and rebrand (see AP: x) and create 3 DevOps teams then you’d be super awesome.

DevOps is not about cherry picking some developers and some IT Operations people and silo’ing them off. You’ve got to embrace and embed. Collapse the development team into the ops team or vice-versa. You need to fully break down the barriers / walls / guards between the teams and mould them into a single unit with shared goals and responsibilities.

http://www.slideshare.net/realgenekim/2012-velocity-london-devops-patterns-distilled

5. The hostile takeover

DevOps. So that’s a word that starts with “Dev”. That means development lead, because development comes first…… Problem?

DevMgr – “Er, we’re now doing DevOps. My guys need to learn the production systems.”

OpsMgr – “Er….ok. So who’s going to be developing the code?”

The word DevOps is clever. It’s a portmanteau. This means the combination of two words, to form a new word, which gives a new meaning. It even delivers some efficiency. It doesn’t mean we took the word operations and replaced it with the word development. So why would you try and adopt DevOps in that manner?

DevOps requires both groups to recognise their key skills. Share what needs to be shared to collaborate. Learn what needs to be learnt to improve. It does not mean retraining. It does not mean cross-skilling (however, this may be a welcome side-effect). It does mean providing feedback and visibility to improve.

http://www.slideshare.net/realgenekim/2012-velocity-london-devops-patterns-distilled

6. DevOps is a buzz word

If you think DevOps is a buzz word, then you’ve probably been using “The Cloud” as a misnomer too. DevOps is a word, you got that right. Actually, it’s a portmanteau of Development and IT Operations (I’ll collect my gold star from teacher later).

DevOps is more than just a cool buzz word. It’s a state of mind. You must embrace its values, you must help others embrace its values and you must continually improve yourself and help others to improve for it to be successful. Once you throw away the BS and start collaborating you might get people to think your catchy new word “DevOps” might actually be cool.

http://www.slideshare.net/realgenekim/2012-velocity-london-devops-patterns-distilled

7. Sell DevOps as a silver bullet

DevOps is voodoo. You basically get your Development team and your IT Operations team together. They smoke some peyote and then sacrifice a chicken. Once you’ve done that your organisation will be revolutionised. You’ll be able to ship software faster than ever before. Configuration will be self-managing. Your deployment tools will become self-aware. Your development and IT Operations teams will have a new found harmony.

Get this….. DevOps is hard work! For most it requires Culture Change! That’s one of the hardest things you’ll ever attempt. For seasoned development and IT Operations teams you’re about to try and turn their world up-side down. Don’t try and do it overnight or you will fail.

8. DevOps means Developers Managing Production

No. Hell No and No again. I’m so incensed you’ll just have to read this….

9. DevOps is Development-driven release management

Let me get 2 things clear;

  1. DevOps is not development driven.
  2. DevOps is not IT Operations driven.

If you want a developer driven environment, fine, go create one. Just don’t call it DevOps. It’s not.

Take a look at this article for example.

http://www.computerweekly.com/cgi-bin/mt-search.cgi?IncludeBlogs=113&tag=release%20management&limit=20

“Within DevOps, programmers are programmers.” – Right on!

“Equally, within DevOps, operations staff are operations staff.” – We’re cooking now!

“Traditionally, getting software out to production can be either the responsibility of operations, or of the development team.”

Hang on……

“IT operations teams will have established and trusted deployment strategies in place that minimise downtime and ensure stability at the expense of agility and speed of response.”

Yep, we’re back on track…… and then bam!

“Development-driven release management” – WTF? It gets worse

“Development-driven release management goes the other way and looks at how deployment can be carried out as often and easily as possible. However, these deployments aren’t necessarily into production……………From a process standpoint, continuous delivery has two big requirements: first, the process itself has to be solid beyond development. This means that it has to be as solid as any process that a traditional IT operations team might put into place.”

No. Non. Nej. Na. Nee. Nein.

Development-driven anything may be a process. It’s just not DevOps. Replacing your IT Operations team with an automated deployment process is nonsense. Please don’t try this at home folks!

10. We can’t do DevOps – We’re Unique

Yes you are, you little beauty you! But you’re not special enough that you can’t adopt DevOps. I bet you’re the best developer out there; you code quicker than lighting, and deliver the sort of code that makes grown men cry with joy. No? Okay, so you’re the most awesome Ops Guy on planet. If Chuck Norris were an IT Operations Engineer he’d be want to be you. However, you and your organisation don’t have some unique factor that won’t allow you to adopt DevOps. So give it a go!

Jesse Robbins from OpsCode has some good advice for getting started;

  • Start Small – Build Trust
  • Create Champions
  • Build Confidence
  • Celebrate Success
  • Exploit Circumstance

http://www.slideshare.net/jesserobbins/cloud-expo-jesserobbinsopscode20130129b

https://www.ibm.com/developerworks/mydeveloperworks/blogs/ambler/entry/adoption_antipattern_we_re_special?lang=en

11. We can’t do DevOps – We’ve got the wrong people

Well why did you hire them? That’s right – they’re awesome! If you don’t think that, then you need to take a long hard look at yourself, then go and discover the real hidden talents in your team.

Someone told me recently that they couldn’t do DevOps because “they have the wrong developers or the wrong ops people…”. So they have developers who can’t code? I thought to myself, “my organisation has the wrong developers, people who can’t code, they run HR and Marketing!”

DevOps fosters a collaborative working relationship between Development and IT Operations. This collaboration can extend right through the organisation further enhancing working relationships between teams.

You don’t have the wrong people. You have the wrong thought process. Deal with it.

12. Collaboration when the S**T hits the fan

Ok genius. You f**ked up. So what? We all do it. But now you want your IT Operations guys out of bed at 2am to clean-up something they know nothing about. They are IT Operations engineers – not the “fixer” like Michael Clayton. Waiting until an error occurs during a deployment for Development and IT Operations to collaborate sucks.

It’s too late for this problem….. but maybe not for the next. You have your Development team and your IT Operations team talking (or swearing at 2am) with each other, but at least they are talking. Keep the dialog going. Get a retrospective review of what happened and how you can fix it going forward. If you have encountered this situation, then try and keep the dialog going with between your teams. Open the communication channels with Development and IT Operations early. There’s hope for you yet!

http://cdn.dzone.com/sites/all/files/refcardz/rc145-010d-continuousdelivery_0.pdf

– TheDevMgr