DevOps, Death Marches and the Hero Hacker Myth

I’ve been thinking a lot about DevOps culture over the last few weeks which is probably why my ears pricked up when I overheard someone telling a story about an IT project they’d worked on where they did “10 weeks straight, 7 days a week” in order to get the project in on time.

These types of stories are a key part of any organisations culture – they form part of the mythology that defines the cultural norms and “the way things are done” within the organisation and can tell you a lot about its values.

In this case the speaker, judging by his tone of voice, was very proud of the effort they’d put in and the fact they’d managed to “pull it out of the bag” with such a super-human effort.

After all, the project was a success, right?

Death Marchs

My perception, from the outside, was a bit different. To me it sounded like a “death march project”, which Ed Yourdon defines as:

“A death march project is one for which an unbiased, objective risk assessment (which includes an assessment of technical risks, personnel risks, legal risks, political risks, etc.) determines that the likelihood of failure is ~ 50 percent.” – Edward Yourdon

Wikipedia goes on to say:

Often, the death march will involve desperate attempts to right the course of the project by asking team members to work especially grueling hours (14-hour days, 7-day weeks, etc) or by attempting to “throw (enough) bodies at the problem”, often causing burnout.
- Wikipedia “Death march (project management)

Sound familiar?

Hero Hacker Myths

Closely related to the “Death March” is the “Hero Hacker Myth”, neatly described here by Rob Mee:

“The myth of the hero hacker is one of the most pervasive pathologies to be found in Silicon Valley start-ups: the idea that a lone programmer, fuelled by pizza and caffeine, swaddled in headphones, works all hours of the night to build a complex system, all by himself. Time out. Software development, it turns out, is a team sport. All start-ups grow, if they experience any meaningful success. What works for a lone programmer will not work in a company of 10. And what’s worse, encouraging the hero mentality leads to corrosive dysfunction in software teams. Invariably the developers who do a yeoman’s 9-to-5, week after week, cranking out solid features that the business is built on, lose out to the grasping egomaniacs who stay up all night (usually just one night) looking to garner lavish praise. Rather than reward the hero, it’s better to cultivate a true esprit de corps.” – Rob Mee, Pivotal Labs

The bold and red highlighting in the quote above is mine because that sentence is central to the message of this post – if your organisational culture mythologises “Death March”-type projects or the “Hero Hacker” then you have a dysfunctional culture that needs to be fixed.

Once the Death March mythology is engrained into your organisational culture EVERY project will end up being a Death March. Why? Because if they’ve “succeeded” in the past (because of super-human effort on behalf of the project team) with a project that was only allocated 50% of the time, resources or budget it needed to have a realistic chance of success then that 50% becomes the new norm.

What manager is going to scope the next project, of equivalent size, with TWICE the time, resources or budget of their last Death March? It would probably be career suicide.

Of course, this is only possible because the hidden costs of Death March projects like staff turnover, sickness, burn-out, divorce, poor quality / huge technical debt, etc. don’t appear on the project’s balance sheet. The narrow ROI view of the project budget doesn’t include these intangibles which are normally shouldered by the wider IT Department budget, or borne solely by the individuals whose health, marriages and families are harmed in the process.

Similarly the Hero Hacker Myth is an enabler for the Death March by mythologizing the long hours and hero mentality that any Death March project requires to have any chance of success.

In addition, as in the highlighted section in Rob Mee’s post, the Hero Hacker Myth is corrosive to the idea of teamwork and shared responsibility that methodologies like Agile and DevOps seek to engender. Almost by definition the “Hero Hacker” is anti-social in his work practices and is likely to horde his knowledge in a way that hurts the team’s success.

DevOps

So what does this have to do with DevOps?

Well, if we view the Death March and the Hero Hacker through the lens of the “Three ways of DevOps” we can see that DevOps should stop Death Marches after a few miles and show the Hero Hacker the door.

The “First Way” emphasises “Systems Thinking” – seeing the big picture about how value is delivered to the organisation and the customer.

Systems thinking should encompass the intangibles discussed above and see that you can’t deliver sustainable long-term value on a foundation of projects that have a 50% chance of failure and that require super-human effort to have any chance to succeed.

It’s like tossing a coin and expecting it to come down heads time and time again – eventually it will come down tails and you’ll have a massive project failure that will probably destroy any value generated by your earlier “successful” Death March projects.

The “Second Way” says “Amplify Feedback Loops” – and the key way to do this is to shorten the feedback cycle by iterating faster. This is the essence of the “fail fast, fail often” mantra which is the antithesis of the long, destructive Death March. With short feedback loops the message that the goal is likely to be unachievable should be received faster.

If course, just because you get rapid feedback that your project is turning into a Death March (or that your Hero Hacker is on his 3rd all-nighter and is running out of Jolt Cola) doesn’t necessarily mean anyone will head the message and take corrective action.

That’s where the Third Way of DevOps comes in – foster a “Culture of Continual Experimentation and Learning”. DevOps should foster a culture of learning from your mistakes, whether it’s your previous Death March or the fast feedback on your current project that you receive from amplifying your feedback loops.

“Those who do not learn from history are doomed to repeat it” and the essence of the Third way is to create the “virtuous spiral” by taking the learnings and lessons from the previous iteration and applying them to the next cycle.

Similarly a culture of learning is ipso facto a culture of sharing which means the knowledge hoarding of the Hero Hacker should be anathema to a DevOps culture.

In summary, listen to the stories your organisation mythologises and see if those stories are taking your down the path of DevOps learning or yet another destructive Death March waiting for a Hero Hacker to save it…