Category Archives: DevOps

DevOps Cardiff Meetup is Launched!

If you want to deliver better software faster, and live in or close to Cardiff, Wales – then this group is for you!

A chance to exchange some of the latest ideas and technologies for implementing Continuous Delivery, Automation and DevOps practices through short presentations by invited speakers, and a chance to chat it after over drinks/snacks with new friends.

This event was created with the intention of uncovering betters ways of interacting between development and operations, by doing it and helping others do it.

Our aims are simple;

1. Demonstrate tools, techniques and practices we use to bridge the gap between software development and IT groups they rely upon.

2. Facilitate discussion on DevOps successes and challenges.

3. Explore, experiment and learn new and better ways of practicing DevOps.

Come and join us and help create more awesome!

DevOps Cardiff

Cardiff, GB
19 Members

If you’re a Developer or Operations person in the Cardiff Area and wanting to deliver better software faster, then this group is for you!A chance to exchange some of the late…

Next Meetup

Inaugural DevOps Meetup

Wednesday, May 7, 2014, 6:30 PM
11 Attending

Check out this Meetup Group →

#DevOps. Big in Japan. Apparently. The rise of #DevOps

DevOps is “Big in Japan“, apparently, or so it would seem from Google Trends!

I was reading an article on Mashable on “cool Google tools” and decided to have a play on Google Trends researching DevOps versus other trends (namely “Agile Development” and “continuous delivery”).

The graph below shows some interesting trends:

  1. The DevOps search term has exploded since 2010 to almost be as popular search term as “agile development”
  2. The popularity literally nearly went vertical in early 2013
  3. The rise is probably slightly steeper that the rise for “agile development” in the 2005-2006 period but generally shows some similarities in virality (based upon my Mark 1 eyeball interpretation of the graphs, if there are any statisticians out there that can be more accurate feel free!)
  4. “continuous delivery” is a laggard in comparison?
  5. DevOps is most popular in… Japan! (DevOpsGuys blog was actually being translated into Japanese at one point, not sure if that’s still happening or not).

Graph showing rise of DevOps world wide in Google Trends

DevOps is big in Japan?

CALMS Model image

The secret of #DevOps success isn’t in the IT literature (yet)!

Can you find “DevOps Success” by reading only IT literature?

The answer is “mostly No, but a little bit Yes”, for a number of reasons.

The main reason is that many of the blogs, whitepapers and webinars around DevOps are ultimately about technology and toolchains. Whilst they might reference the DevOps C.A.L.M.S model in passing the conversation is generally focussed on the A for Automation and the M for Metrics.

CALMS Model image
Culture Automation Lean Metrics Sharing

The seminal Phoenix Project did talk about organisational culture as did Mandi Wall’s O’Reilly ebook on Building a DevOps Culture.

But what both of those books have in common is that they drew extensively from non-IT business literature.

Goldratt’s “The Goal” & TOC, Systems Thinking, Lean manufacturing, Deming and Kanban being major influences on the Phoenix project, and Mandi’s e-book drawing on business-centric cultural/organisational change literature.

“Searching on the Harvard Business Review website for “cultural change” will get you 60+ publications going back nearly 30 years” – Mandi Walls

“Lean Manufacturing”, “Strategic Alignment”, “Organisational Change”, “Culture”, “Business Transformation” and many other topics have been staples of the MBA curriculum in business schools for many years and there is a wealth of resources available online to explore. Reading beyond the IT literature and exploring the wider business context for your DevOps Transformation will, we believe, significantly increase your chances of getting business buy-in and having a successful outcome to your DevOps change programme.

In order to make this easier for you we (@DevOpsGuys) will be publishing a weekly blog post exploring an area of business literature and how it can be used in DevOps.

We’re call it the #DevOpsMBA :-)

So please subscribe to our blog (link on right), follow us on Twitter or search for the #DevOpsMBA hashtag on Twitter to keep informed!

What does DevOps look like in UK job ads

#DevOps – Organic versus Transformational DevOps

Despite what some people seem to think there is more to DevOps than just Continuous Delivery and Infrastructure Automation with Puppet, Chef or Ansible.

To me, DevOps is “an alternative model for the creation of business value from the software development life-cycle that encompasses a product-centric view across the entire product life-cycle (from inception to retirement) and recognises the value in close collaboration, experimentation and rapid feedback”.

Moving from one model of value creation can either be an organic process or a transformational one – you can “grow into” the new model or you can plan a strategy of change to transform your organisation from one to the other.

It’s in this “Organic DevOps” versus “Transformational DevOps” that I see a growing disconnect between different sectors of the DevOps community, particularly between “DevOps for Start-ups” and “DevOps for Enterprise”.

IMHO, “Start-Ups DevOps” normally follows the “organic DevOps” path – you’re often starting from a relatively “greenfields” approach based on a cloud infrastructure. You probably already have a very close, collaborative culture because there’s only 20 of you anyway and you all work in the same office and you spend 18hrs a day there. Automation is part of your DNA because you’ve never had the staffing levels to do it manually.

“Enterprise DevOps” is normally “Transformational DevOps” – you have large, distributed IT teams that cross geographic locations, time-zones and probably organisational boundaries (due to outsourcing). You have extensive legacy applications and infrastructure estates (JVM 1.4 on Tomcat 5 anyone?) and you’re likely to have well developed Prince2/ITIL/SixSigma delivery models rigidly enforced by a centralised command&control mindset, backed by an army of highly-paid consultant from the Big 5 telling your CEO, CIO and CTO the best way to manage their IT budget.

Moving an enterprise to DevOps via a transformation programme is a very different challenge to introducing DevOps concepts into a receptive start-up and watching them grow organically, and the DevOps community needs to make sure that when it’s evangelising DevOps to the world that it’s aware of the differences and challenges inherent in each approach.

If you want to debate this idea of “Start-up Organic versus Enterprise Transformational DevOps” we’re taking part in a Webinar tonight with the great folks over at ScriptRock that’s focussing on Enterprise DevOps. It’s at 1900 BST,  11:00am PT / 2:00pm ET (60 minutes).

We’d really like to get your thoughts on this by asking a question on the webinar or by leaving a comment below as these concepts are still experimental and, just like DevOps itself, the faster we get feedback and the more we iterate around the concept the stronger it will be!

http://info.scriptrock.com/devops_webinar-2

Enterprise DevOps Webinar
Enterprise DevOps Webinar

 

 

Why companies are investing in DevOps and Continuous Delivery

A thought-provoking infographic – with some interesting data points – shows how companies are reaping real rewards from investing in agile software delivery processes. Check out the graphic – from Zend – for more on how DevOps and Continuous Delivery are bridging the speed and innovation gap between business demand and IT.

Continuous Delivery Infographic by Zend Technologies.

Continuous Delivery Infographic

A scientific basis for #DevOps success?

A fascinating TED talk from “Predictably Irrational” author Dan Ariely has some interesting pointers to some of the underlying psychological mechanisms that make the DevOps model a better way to structure work within IT departments.

“we care much more about a product if we’ve participated from start to finish rather than producing a single part over and over.”

The accompanying article lists 7 insights into our motivations when performing tasks (and the research that supports those insights).

  1. Seeing the fruits of our labour may make us more productive
  2. The less appreciated we feel our work is, the more money we want to do it
  3. The harder a project is, the prouder we feel of it
  4. Knowing that our work helps others may increase our unconscious motivation
  5. The promise of helping others makes us more likely to follow rules
  6. Positive reinforcement about our abilities may increase performance
  7. Images that trigger positive emotions may actually help us focus

Many of these tie directly back to key DevOps principles.

For example the “First Way of DevOps” encourages “systems thinking” which relates directly to #1 above – if we are looking at the entire system (not just our small part) we will inherently be looking at the “fruits of our labour”.

Similarly fostering a team-based DevOps culture where we can see “how our work impacts on others” is closely aligned with #4.

For me, #2 and #6 tie directly back to “leadership” (as opposed to “management”). Good leaders know that praise (either private 1:1 praise with individuals or public praise in front of the team) can have a huge impact on morale, with a subsequent impact on productivity and quality.

It’s fascinating to see how behavioural science is increasing our understanding of human motivation. The challenge for us in the DevOps movement is to take these science-based insights and see how we can apply them with our teams to create a better way of working.

DevOps 101 for Recruiters

We put together this “DevOps Intro” to the recruitment team at a London recruitment consultancy to help them understand the DevOps Market place.

The goal was to help them understand both the WHY and the WHAT of DevOps and what that might mean for recruiters.

Hopefully you will find this interesting as well!

requirements_rotated

The Top Ten DevOps “Operational Requirements”

Join us for “The Top 10 DevOps Operational Requirements” –  http://www.brighttalk.com/webcast/534/98059 via @BrightTALK

One of the key tenets in DevOps is to involve the Operations teams in the full software development life cycle (SDLC) and in particular to ensure that “operational requirements” (“OR’s”, formerly known as “non-functional requirements”, “NFR’s”) are incorporated into the design&build phases.

In order to make your life easier the DevOpsGuys have scoured the internet to compile this list of the Top Ten DevOps Operational Requirements (ok, it was really just chatting with some of the guys down the pub BUT we’ve been doing this a long time and we’re pretty sure that if you deliver on these your Ops people will be very happy indeed!).

#10 – Instrumentation

Would you drive a car with a blacked out windscreen and no speedo? No, didn’t think so, but often Operations are expected to run applications in Production in pretty much the same fashion.

Instrumenting your application with metrics and performance counters gives the Operations people a way to know what’s happening before the application drives off a cliff.

Some basic counters include things like “transactions per second” (useful for capacity) and “transaction time” (useful for performance).

#9 – Keep track of the Dependencies!

“Oh yeah, I forgot to mention that it needs [dependency XYZ] installed first” or “Yes, the system relies on [some 3rd party web service] can you just open up firewall port 666 right away”.

Look, we all understand that modern web apps rely on lots of 3rd party controls and web services – why re-invent the wheel if someone’s already done it, right? But please keep track of the dependencies and make sure that they are clearly documented (and ideally checked into source control along with your code where possible). Nothing derails live deployments like some dependency that wasn’t documented and that has to be installed/configured/whatever at the last moment. It’s a recipe for disaster.

#8 – Code defensively & degrade gracefully

Related to #9 above – don’t always assume the dependencies are present, particularly when dealing with network resources like databases or web services and even more so in Cloud environments where entire servers are known to vanish in the blink of an Amazon’s eye.

Make sure the system copes with missing dependences, logs the error and degrades gracefully should the situation arise!

#7 – Backward/Forward Compatibility

Existing code base with new database schema or stored procedure?

New code base with existing database schema or stored procedures?

Either way, forwards or backwards, it should work just fine because if it doesn’t you introduce “chicken and the egg” dependencies. What this mean for Operations is that we have to take one part of the system offline in order to upgrade the other part… and that can mean an impact on our customers and probably reams of paperwork to get it all approved.

#6 – Configurability

I once worked on a system where the database connection string was stored in a compiled resource DLL.

Every time we wanted to make a change to that connection string we had to get a developer to compile that DLL and then we had to deploy it… as opposed to simply just editing a text configuration file and re-starting the service. It was, quite frankly, a PITA.

Where possible avoid hard-coding values into the code;  they should be in external configuration files that you load (and cache) at system initialisation. This is particularly important as we move the application between environments (Dev, Test, Staging etc) and need to configure the application for each environment.

That said, I’ve seen systems that had literally thousands of configuration options and settings, most of which weren’t documented and certainly were rarely, if ever, changed. An “overly configurable” system can also create a support nightmare as tracking down which one of those settings has been misconfigured can be extremely painful!

#5 – “Feature Flags”

A special case of configurability that deserves its own rule – “feature flags”.

We freakin’ love feature flags.

Why?

Because they give us a lot of control over how the application works that we can use to (1) easily back out something that isn’t working without having to roll-back the entire code base and (2) we can use it to help control performance and scalability.

#4 – Horizontal Scalability (for all tiers).

We all want the Product to be a success with customers BUT we don’t want to waste money by over-provisioning the infrastructure upfront (we also want to be able to scale up/down if we have a spiky traffic profile).

For that we need the application to support “horizontal scalability” and for that we need you to think about this when designing the application.

3 quick “For Examples”:

  1. Don’t tie user/session state to a particular web/application server (use a shared session state mechanism).
  2. Support for read-only replicas of the database (e.g. a separate connection string for “read” versus “write”)
  3. Support for multi-master or peer-to-peer replication (to avoid a bottleneck on a single “master” server if the application is likely to scale beyond a reasonable server specification). Think very carefully about how the data could be partitioned across servers, use of IDENTITY/@Auto_Increment columns etc.

#3 –Automation and “scriptability”

One of the key tenets in the CALMS DevOps Model is A for Automation (Culture-Automation-Lean-Metrics-Sharing if you want to know the others).

We want to automate the release process as much as possible, for example by packaging the application into versionable released or the “infrastructure-as-code” approach using tools like Puppet & Chef for the underlying “hardware”.

But this means that things need to be scriptable!

I can remember being reduced to using keystroke macros to automate the (GUI) installer of a 3rd party dependency that didn’t have any support for silent/unattended installation. It was a painful experience and a fragile solution.

When designing the solution (and choosing your dependencies) constantly ask yourself the question “Can these easily be automated for installation and configuration”? Bonus points if you can, in very large scale environments (1,000 of servers) build in “auto-discovery” mechanisms where servers automatically get assigned roles, service auto-discovery (e.g. http://curator.apache.org/curator-x-discovery/index.htm) etc.

#2 – Robust Regression Test suite

Another think we love, almost as much as “feature flags” is a decent set of regression test scripts that we can run “on-demand” to help check/verify/validate everything is running correctly in Production.

We understand that maintaining automated test scripts can be onerous and painful BUT automated testing is vital to an automation strategy – we need to be able to verify that an application has been deployed correctly, either as part of a software release or “scaling out” onto new servers, in a way that doesn’t involve laborious manual testing. Manual testing doesn’t scale!

The ideal test suite will exercise all the key parts of the application and provide helpful diagnostic messaging if something isn’t working correctly. We can combine this with the instrumentation (remember #10 above), synthetic monitoring, Application Performance Management (APM) tools (e.g. AppDynamics), infrastructure monitoring (e.g. SolarWinds) etc to create a comprehensive alerting and monitoring suite for the whole system. The goal is to ensure that we know something is wrong before the customer!

#1 – Documentation

Contrary to popular belief we (Operations people) are quite happy to RTFM.

All we ask is that you WTFM (that’s W as in WRITE!) J

Ideally we’d collaborate on the product-centric documentation using a Wiki platform like Atlassian Confluence as we think that this gives everyone the easiest and best way to create – and maintain – documentation that’s relevant to everyone.

As a minimum we want to see:

  1. A high-level overview of the system (the “big picture”) probably in a diagram
  2. Details on every dependency
  3. Details on every error message
  4. Details on every configuration option/switch/flag/key etc
  5. Instrumentation hooks, expected values
  6. Assumptions, default values, etc

Hopefully this “Top Ten” list will give you a place to start when thinking about your DevOps “Operational Requirements” but it’s by no means comprehensive or exhaustive. We’d love to get your thoughts on what you think are the key OR’s for your applications!

Our experienced DevOps team provides a fully-managed “application-centric” website support service to your business and  your customers. Contact us to today, to find out how we can help.

image source: - CC  - c_knaus via Flickr - http://www.flickr.com/photos/soda37/6496536471