DevOps – How do you measure team success?

In an earlier blog post we talked about the importance of re-structuring your Dev & Ops organisation to remove silos and in another post we touched on how incentives and metrics influence how the teams work together.

Organisationally we remained siloed however – we were incentivised in different ways (Operations emphasising availability, Development emphasising feature delivery), we remained in essentially a waterfall delivery model and Ops VS Dev was a constant struggle for manpower & resources. All the usual problems that the DevOps movement is trying to address.

So how do we create alignment in incentives across our merged DevOps team and what Metrics should we be tracking to measure success?

Jesse Robbins from OpsCode wants us to “make more awesome” and proposes a measure of “time to value” as a measure of DevOps success.

Rich Steer gets excited about the “time to value” concept here – – but I’ve yet to see a truly operational definition of how it works.

As someone points out in the comments on Rich’s article when do you start/stop the “Time to Value” clock?

  • Do you start the clock when the business first has the idea (timing being vague and nebulous) or when it becomes a Story in the backlog (more quantifiable but late)?
  • How do you measure “value” (which is intangible) and how long after “deployment” do you decide “value” has been achieved, if it ever does? Or does “deployment=value”?
  • Some changes will have more “value” than others so how do you weight the metric to account for that?

So if “time to value” is a bit nebulous what other metrics are people proposing?

PuppetLab’s State of DevOps report focuses on improvement in 4 key metrics – Change Frequency, Change Lead Time, Change Failure Rate and Mean Time To Recover (MTTR). We suggest that classic metrics like Availability, Performance (page load time) and Mean Time Between Failure (MTBF) should be in there too in order to give a better rounded measure of the overall site performance.

Thoughtwork’s have re-worked the old “function point” or Agile “story point” concept by adding a business spin and came up with “Business Value Points”. (As an aside why the points-based approach might be a bad idea you should read this article here on Agile Story Points and why they might be harming your velocity, not helping).

I’m not convinced that we can find a single metric that can effectively incentivise or measure a DevOps teams performance – I suspect that we’ll have to create a weighted equation of some type to derive a synthetic metric or use a “balanced scorecard” approach to weigh different metrics from different perspectives.

Regardless of the metrics we pick we have to ensure that they incentivise the behaviour we want to reward because as we know from the Met Police and the NHS the wrong incentives result in the wrong behaviour.

This is where an interesting piece of management theory comes into play – it’s called “Vroom’s Expectancy Theory”, and it outlines 3 factors I always like to keep in mind when I am setting team incentive metrics

Basically it boils down to 3 simple things.

“I am only going to be motivated to work harder IF…”:

 (1)    Expectancy = My effort will really make a difference to the overall performance (conversely, “why bust a gut if no matter how hard I work it doesn’t make a difference to the overall performance”, hence why getting the weighting in the overall equation is critical)

(2)    Instrumentality = If we achieve our performance goals I will get the reward/bonus (i.e. do I trust my Boss enough to deliver on the bonus and not weasel out with some BS about the “state of the economy”).

(3)    Valence = The reward is something I care about enough to make the incremental effort worthwhile. (e.g. it might be an order of magnitude harder to go from 99.9 to 99.99% availability but if it is only an extra 1% in my bonus is that really worth all that effort?)

 The real key here is finding metrics that EVERYONE in the DevOps team can contribute to, and that overall the metric balances out the different contributions from the different roles within the Team.

If you get the “balance” wrong it can lead to frustration for the team members

For example, a systems administrator might think “no matter how hard I work (e.g. say on infrastructure automation) the crappy code quality will still make us miss out targets, but I’m not a developer and I have few (if any) ways to improve that…”.

Conversely from a developer perspective “I can write awesome code but if the ops guys keep messing up the server configuration and causing downtime then what’s the point?”?

As a thought experiment I tried to think of what sort of “equation” might be the basis for a meaningful synthetic metrics.

What I came up with was this:

DevOps_Incentive_Bonus_Metric = “Development Velocity ((100availability percentage”)*100) + (Apdex(3) * 100) + NetPromoterScore ?

If we take my sample equation let’s see how some of that works if we turn this into “user stories” for our DevOps Team:

As an “Operations Person”
I want “to create automated test environment build scripts using Chef”
So that “Developers can work faster and increase their Development Velocity”

As a “Development Person”
I want “to incorporate page load time non-functional requirements into my user stories
So that “we only ship code that meets the 3 second page load time objective”

As an “Operations Person”
I want “to create a load-balanced production environment”
So that “we can remove any single point of failures and improve availability”

As a “Developer Person”
I want “to remove any dependency on per-server session state”
So that “my code works better in a load-balanced environment and improves availability”

As a “Developer Person”
I want “ship awesome feature XYZ”
So that “customers are delighted and increase our Net Promoter Score”

These “team user stories” seems to work pretty well  to me (but I am sure that many of you could come up with something better!).

So the questions for the DevOps community at large are:

  1. What Metrics do you use to motivate and measure the successes of your DevOps Team?
  2. How are those metrics shared across the different roles within the Team?
  3. How well are those metrics working for your DevOps team?

I am sure lots of people would like to know so please give us your thoughts in the comments or link over to your blogs etc.


Are you really doing DevOps? 8 prerequisites you must consider

A recent “2013 State of DevOps” report from PuppetLabs indicated that “52% of [over 4,000] respondents said that they’ve been doing DevOps for > 12 months”. Our own, less formal, survey found very much the same – 60% of respondents indicated that they were currently doing DevOps”. Take part in our survey, by clicking here.

PuppetLabs Survey Results
PuppetLabs Survey Results

The question is Are you really doing DevOps?

As we pointed out in our earlier DevOpsDays wrap-up blog there are at least 7 different definitions of what DevOps is, judging by the presentations given on the day:

DevOps … is a movement, a philosophy, a way of thinking.
DevOps … is a person who can perform both Dev and Ops roles.
DevOps … means cross skilling people.
DevOps … is continuous delivery.
DevOps … is a team of developers and operation staff.
DevOps …is a culture movement.
DevOps … is monitoring.

So… what’s the “minimum viable product” (MVP) for DevOps? What core things should you be doing before you can truly say you are “doing DevOps”?

In the whitepaper “The Top 11 Things You Need To Know About DevOps” Gene Kim emphasizes the “3 Ways”:

(1) “ Emphasize the performance of the entire system” – a holistic viewpoint from requirements all the way through to Operations
(2) “Creating feedback loops” – to ensure that corrections can continually be made. A TQM philosophy, basically.
(3) “Creating a culture that fosters continual experimentation and understanding that repetition and practice are the pre-requisites to mastery”

These are excellent guidelines at a high level, but we’d like to see a more operational definition. So we’ve made up our own list!

As a starter – we propose that;

  1. You must have identified executive sponsors / stake holders who you are actively working with to promote the DevOps approach.
  2. You must have developed a clear understanding of your organisation’s “value chain” and how value is created (or destroyed) along that chain.
  3. You must have organizationally re-structured your development and operations teams to create an integrated team – otherwise you’re still in Silos.
  4. You must have changed your team incentives (e.g. bonus incentives) to reinforce that re-alignment – without shared Goals you’re still in Silos.
  5. You must be seeking repeatable standardized processes for all key activities along the value chain (the “pre-requisite to mastery”)
  6. You must be leveraging automation where possible – including continuous integration, automated deployments and “infrastructure as code”
  7. You must be adopting robust processes to measure key metrics – PuppetLab’s report focuses on improvement in 4 key metrics – Change Frequency, Change Lead Time, Change Failure Rate and MTTR. We suggest Availability, Performance and MTBF should be in there too.
  8. You must have identified well-defined feedback mechanisms to create continuous improvement.

As mentioned above, this is just a starter list – feel free to agree/disagree in the comments and suggest additions or alterations.

We’ll be writing more about “DevOps Incentives” in an upcoming post, and we’ll revisit the “Are you doing DevOps?” topic once we’ve consolidated your feedback.


This week in DevOps #4

State of the Union

This week saw the publication of the “2013 State of DevOps” report from PuppetLabs. It’s a pretty in-depth survey of over 4,000 IT Operations professionals and Developers, which highlights an understanding of the state of DevOps today.

Key Findings

  • 63% of organizations implementing DevOps saw improved quality of software deployment, with 50% seeing fewer failures in production – Software bugs in production have always been expensive in-comparison to early stages of the SDLC, but today bugs are becoming increasingly more expensive to fix and their impact greater with the adoption of Cloud. Adopting a DevOps model, where Development and Operations combine to take greater ownership of the software delivery process and software quality is already starting to show benefits to organisations who have adopted it. It’s also important to note, that the research shows organizations implementing DevOps are likely to be able to fix production problems 12 times faster than organizations that don’t.
  • Organizations implementing DevOps are 5 times more likely to be high-performing businesses – The report defines high-performing organizations as those with the capability to deploy critical applications quickly without disrupting service. This is further categorized by organizations that deploy frequently with less change lead time and failure and have a lower mean-time to recover. Adopting a DevOps model was shown to improve all of these characteristics significantly and showed continued improvement over time.
  • 75% increase in demand for DevOps skills in the last year – The report found that DevOps skills were broadly in 4 categories; Coding/Scripting, People Skills, Process Re-Engineering and specific experience with “DevOps” tools. e.g. infrastructure automation. The overall head-line figure is a good one. More companies are looking for more people with skills to help them implement DevOps.  However, the report highlights the growing debasement of the term.  “DevOps” as a skill adverts on LinkedIn increased by 50 percent in the same time period. Recruiters are certainly jumping on the buzz word band wagon. We asked Gene Kim for his thoughts on this, and were surprised by his response;
  • “I’m ok w/ DevOps in job postings & LinkedIn profiles. It’s a great signal to match buyers/sellers of a needed skill” – @RealGeneKim

  • 48% indicated the value of DevOps wasn’t understood outside of their group – This reason was believed to be the biggest difficulty in implementing DevOps. Cultural factors are still seen as the biggest barriers to DevOps adoption. The report identifies improved communication as one of the best methods for overcoming DevOps adoption barriers.  We believe that there are 4 further key elements that can help to break down the barriers;  Build Trust, Create Champions, Build Confidence and Celebrate Success.

What we found this week

  • DevOps for Windows Azure – a comprehensive set of resources for DevOps teams wanting to deploy and support applications in Windows Azure.
  • DevOps: What’s All the Hype Really About? – A concise 60 second summary (unless your a slow reader) of DevOps with some interesting points for discussion raised. Two questions to ask yourself, So Everyone LOVES DevOps Right? and is DevOps Worth Pushing in My Organization?
  • DevOps and Cloud Management –  RightScale’s Cloud Evangelist Uri Budnik and Blackhawk Network’s Arindam Mukherjee give a real-world use case on using DevOps and RightScale to cut the time it takes to provision a spec environment by over 80%.

Tweets of the Week