Tag Archives: DevOps

#DevOps and automating the repayment of technical debt

One of the common things we find in Enterprise organisations looking to move to a DevOps model is high levels of technical debt.

To be more accurate, they are caught in the “vicious cycle of technical debt” to the point that trying to ship anything in a rapid, agile way is nearly impossible. It’s the Greek Debt Crisis level of technical debt.

In many cases layers and layers of process and management have been added into the software development lifecycle in order to try and fix the symptoms of the problem (low quality releases, bugs in production, unstable environments, poor performance etc) but they are just Band-Aids on the underlying issues.

http://technical-debt.org/cycle.png

So how do we get out of this death spiral before the organisation can’t compete anymore and a disruptive innovator comes along and eats their lunch?

Well, one trend we see is people looking to DevOps automation to try to create the breathing room they need to break out of this vicious cycle and try to create a virtuous circle instead.

.

If we can automate some of the routine, error-prone and time-intensive tasks we can leverage that productivity gain and invest that time we’ve freed up into re-paying technical debt.

As we pay back technical debt we get a higher quality, more stable and more agile application, we can then re-invest in more automation to start the next cycle of improvement.

We know that this works because we’ve seen it and done it with clients BUT it comes with a caveat (or two).

Firstly, you need to get commitment from the Product Owner etc that the productivity gains will be spent on paying down technical debt and not on an endless cycle of feature bloat (that was probably one of the causes of the problem in the first place).

I’m not going to wave a magic wand and say that this is easy – it’s not – but if you can find the right analogy (“Technical debt is like walking through quicksand” or “Technical debt is like trying to run a marathon with an 80lb backpack on” or “Technical debt is what start-ups don’t have…”) then you might have a chance.

Secondly, DevOps is more than just A for Automation – it’s Culture-Automation-Lean-Metrics-Sharing (CALMS) – so ideally you’d doing more than just “automate some stuff” and begin to start looking at being “product-centric”, coaching your product owner to understand operational requirements and moving away from the Finance-driven project-centric model.

I’d love to hear some stories from our blog readers about how you’ve paid down technical debt, what your “tech debt repayment model” looks like so please leave some comments or links to your own blog thoughts in the comments below

-TheOpsMgr

What does the future of IT Operations look like (in a #DevOps world)?

What does the future of IT Operations look like? As more businesses rely on virtualisation, containers, cloud, Infrastructure as a Service and Microservices is there still a role for IT Operations? How do these teams change to continue to deliver value when supporting Agile Operations techniques?

Is there still a role for IT Operations? Absolutely 100% (we believe that so much we started a company to offer application-centric cloud operations!).

We blogged about this back in 2013 when we said that “Devops Does Not Equal “Developers Managing Production”. We said then:

“Operations is a discipline, with its own patterns & practices, methodologies, models, tools, technology etc. Just because modern cloud hosting makes it easier to deploy servers without having to know one end of a SCSI cable from another doesn’t mean you “know” how to do Operations (just like my knowledge of SQL is enough to find out the information I need to know monitor and manage the environment but a long way from what’s required to develop a complex, high-performing website).” – @TheOpsMgr

This still holds true today.

That said, the role of Operations is changing – Ops has to become more “Application-Centric” and understand the applications that are running on the platforms they provide. It’s not enough anymore to take a narrow view that says “my servers are OK, it’s not my fault the application doesn’t work”. Well, it might not be your “fault” but you share the responsibility for making sure the application is available for your customers. Stop passing the buck!

Operations people almost certainly need to learn to code, since we are heading towards a code-driven, API enabled world.  If you can’t code (or at least have solid scripting skills) you risk being left behind will be left behind.

More importantly, the Operations Engineer/Developer of the future will be filling a role more akin to that of a “process engineer” in a physical factory or logistical supply chain.

A process engineer designs a process and production line that transforms raw materials into a finished product.

The Operations Engineer/Developer of the future will be building Digital Supply Chains and Digital Production Lines.

These Digital Supply Chains will transform raw materials (source code) via continuous integration, test automation, packaging, release automation, infrastructure-as-code etc into applications running in cloud-hosted environments.

The rate of changes flowing along the Digital Supply Chain will far exceed “old school” Change and Release methodologies – you can’t have a weekly CAB (Change Advisory Board) meeting if you’re doing multiple deployments per day (or every 11.6 seconds à la Amazon).

So, just like a physical production line includes statistical sampling, automated testing etc., so will the Digital Supply Chain of the future. We already do this with TDD/BDD, automated testing with tools like Selenium etc but it will become the Operations Engineer/Developer job to ensure that the digital production line delivers release packages of sufficient quality to ensure the stability of the application (and the organisation’s revenue that depends on it!).

Modern supply chains are complex and have many interdependencies on 3rd parties, particularly if you’re operating a “Just-In-Time” (JIT) model. Modern software applications have the ultimate in JIT dependencies due to their integrations with 3rd party SaaS API’s like payment gateways, recommendation engines, authentication gateways, cloud providers etc. Modern Operations Engineers will need to ensure that they design the digital supply chain that can cope with failures in these interdependencies, or at least ensure that they select the right 3rd party partners who can offer the right levels of performance and availability needed for their applications.

In summary, will the Operations Engineer/Developer of the future be “just managing (virtual) servers”? No, almost certainly not.

What they will be doing is designing and building complex digital supply chains with complex interdependencies both internally and externally to the organisation, digital supply chains designed to meet the needs of applications that are designed to meet the needs of their customers, safely, securely and cost-effectively.

The Q&A above is part of material prepared as our contribution to an CA ebook on “Agile Operations”. We wrote our thoughts on 6 questions, of which 4 will be used in the ebook, scheduled to come out in August 2015. You can read the earlier Q&A here – http://blog.devopsguys.com/2015/06/23/what-is-important-for-an-it-ops-to-team-more-effectively-with-preproduction-teams-devops/  

What is important for an IT Ops to team more effectively with preproduction teams? #DevOps

“DevOps can present IT Operations teams with new ‘customers’ in development and test. What traditional or new tools and technologies are most likely to be important for IT Ops to team more effectively with preproduction teams? What information does IT Ops need to pass right to left and which tools are most likely to aid in that?”

The short answer is “A whiteboard marker, a pad of Post-It notes and a couple of pizzas” :-)

That answer is a bit tongue-in-cheek, but there is a serious side to it; whilst new tools can be an important part of DevOps (particularly in Automation) you can get started in changing your Culture and improving your Sharing with very simple tools i.e. the aforementioned whiteboard marker, Post-It notes and pizza.

Start to break down the silos by getting key people in a room with some blank walls and whiteboards and start sharing information, mapping out your value stream and trying to find out, collaboratively, where the bottlenecks in your existing processes are. Once you’ve identified your key constraints then fire up Google and start searching for the tools to solve your problems (or visit a site like DevOpsBookmarks).

DevOpsGuys, like most organisations, have our own “Opinionated Stack” – we like the Atlassian Toolset for managing our Agile workflow, TeamCity or Jenkins as our CI tool of choice, Ansible as our configuration management tool for Linux, Powershell DSC for Windows, AppDynamics as our APM tool, Redgate for our Database Lifecycle Management (DLM) and so on. We partner with many of these companies now because we’ve “dogfooded” the products internally and with our customers and they’ve worked well for our use cases. We always “try before we buy” and we “try before we partner” too because, as they say, “your mileage may vary” (YMMV).

This comes back to fostering a culture of experimentation – give something a try and see what works for you. We started off using Atlassian HipChat as our chat tool and we really liked it. Then we tried Slack and we liked that one more, so we switched. YMMV.

One additional point worth mentioning – the premise of the question is flawed!

They aren’t customers they’re colleagues.

There isn’t a silo of “Us” (IT Ops=supplier) versus “Them” (Everyone Else=customer).

We are supposed to be breaking down these silos to create cross-functional, multi-disciplinary, product-based teams. Development, Test, IT Security, Networks shouldn’t be silos any more – they are people in our team, sitting over the desk from us, attending our daily standups, eating our pizza :-)

The Q&A above is part of material prepared as our contribution to an CA ebook on “Agile Operations”. We wrote our thoughts on 6 questions, of which 4 will be used in the ebook, scheduled to come out in August 2015. We’ll post the remaining 2 questions with our answers onto the blog over the next 2 weeks. 

DevOps and the Digital Supply Chain

What is the “Digital Supply Chain” and why is it important to your organisation and to DevOps as a practice?

The concept of the “Digital Supply Chain” is a different way of looking at the SDLC and the Continuous Delivery “pipeline” that we feel makes it easier for traditional organisations to understand the criticality of software delivery (and by extension DevOps) in the modern world.

Any organisation that deals with physical goods understands the concept of the supply chain. They are intimately familiar with ideas like supply chain management, supply chain optimisation and, most importantly, they understand the economics of inventory in the supply chain e.g the carrying cost of inventory.

So what is the “Digital Supply Chain”?

The current definitions of the digital supply chain are anchored in the “New Media” sector and focus on digital assets like music, video etc

“The “digital supply chain” is a “new media” term which encompasses the process of the delivery of digital media, be it music or video, by electronic means, from the point of origin (content provider) to destination (consumer).” – Wikipedia

The Wikipedia article references above breaks it down into a number of discrete steps as shown in Figure 1 below.

image of New Media Digital Supply Chain

If we contrast this with our SDLC Continuous Delivery Pipeline (Figure 2) we can see that many of the steps are directly analogous – we are creating digital assets (code) which we then “compress” (i.e. the Build/Integrate process), which we then subject to Quality Control (Test), we store in a Digital Asset Management system (e.g. like Nexus or Artifactory), we tag it with metadata (e.g. what release/version we’re deploying) and when then deploy it out to servers, CDN’s, the AppStore or wherever.

image of SDLC Digital Supply Chain

Once your customers grasp the idea that software is a digital asset and that carrying excess inventory and delays in moving these digital assets along the supply chain is costing them money it can be a lightbulb moment for many organisations.

Software assets can depreciate over time. Indeed “technical debt” can be looked at as the “cost of deprecation” of your software assets over time.

Code that is “stuck” in your Digital Supply Chain waiting for your next release (as source code in Git, or as artefacts in an artefact repository) represents a capital investment in “digital assets” held as “digital inventory” and having it sat on the digital shelf in your digital warehouse is costing you money is analogous to the carrying cost of inventory for physical inventory.

Sure, the warehousing costs of a digital asset – your latest idea transformed into software code – is fairly trivial compared to the costs of physical warehousing BUT the “opportunity cost” is very real.

Each digital software asset represents a significant investment in time & money by your designers, developers, testers, project managers etc and it doesn’t start generating a return on that investment until it gets to the end of your digital supply chain and into the hands of your customers.

DevOps then becomes a way to optimise your digital supply chain to ensure that we:

  1. only build the right things (reducing waste and optimising our digital inventory),
  2. Supplier management (by improving the relationships between Dev, Test, Ops etc we ensure that we are getting the best from all of the “suppliers” in our digital supply chain)
  3. improving our logistics to get our digital assets in the hand of our customers (by automating testing, release and deployment to accelerate the movement of the digital assets from left to right)
  4. Constantly seeking “flow” across the supply chain (the 1st way of DevOps!)
  5. Gathering metrics along the supply chain to give us insight into the bottlenecks (the M in the C.A.L.M.S model)

So next time you’re talking with people in the business try out the “Digital Supply Chain” analogy and see if it works for you – we’d love to hear your feedback!

-TheOpsMgr

5 Definitions of #DevOps

In my talk at QCon last week I was lucky enough to be the first speaker on the DevOps track so I took the opportunity to poll my fellow speakers for THEIR definition of DevOps and the results where fascinating.

What I ended up with was different 5 Definitions of DevOps:

DaveDefinitionOfDevOps

DevOps - Matthew

DevOps - Anna

DevOps - Amy

DevOps - Steve

So what are the key take away messages from these DevOps definitions?

Firstly – there is no single, definitive version of what DevOps “is” as you can see from the responses above!

DevOps clearly means different things to different people and we (as the DevOps community) have to ask ourselves is this a good or bad thing? Good in that it’s promoting a diversity of ideas that contribute to the DevOps movement, but “bad” in that is causes confusion amongst people trying to learn more about DevOps.

Secondly – there is a clear message around communication, collaboration and culture being a key part of the DevOps message, which is excellent” #NoMoreSilos! J

Thirdly – sadly the name “DevOps” might be seen as being exclusionary in itself and not embracing critical parts of the SDLC community – specifically in Amy’s case the Test community. Amy expands on this in her blog post “DevOps with Testers”.

A number of people would say that DevOps should be called “DevTestOps” or “BizDevOps” but at QCON Dave Farley was arguing that the correct term should be “Continuous Delivery” as DevOps is just a subset of CD. We discussed this earlier in our post on Continuous Obsolescence.

  • So what does DevOps, DevTestOps or BizDevOps means to you?
  • Has the term DevOps had it’s day and should we just extend the definition of Continuous Delivery to embrace the full application lifecycle?
  • Do we need to rename our company? :-)

Your thoughts in the comments please?

-TheOpsMgr

#DevOps = #ContinuousDelivery + #ContinuousObsolescence?

During my talk at QCon last week we had started an interesting debate about whether DevOps was a subset of Continuous Delivery or vice versa.

We were lucky enough to have Dave Farley in the audience and it was clear from his definition of DevOps versus Continuous Delivery that he felt that CD was the superset and DevOps a component of that set… but he admitted that he if he was writing the Continuous Delivery Book again in 2015 would change the definition of “finished” to embrace the full product lifecycle from inception to retirement.

This aligns nicely with his co-author Jez Humble’s thoughts on being “Product-Centric” that we discussed in an earlier blog post.

“Finished” should mean “no longer delivering value to the client and hence retired from Production”.

DaveDefinitionOfDevOps

DaveOnCD

That was the we joined a new term (live on stage as captured by Chris Massey’s tweet below) – “Continuous Obsolescence”

 MasseyTweet

So what is #ContinuousObsolescence?

Continuous Obsolescence is the process of continuously grooming your product/feature portfolio and asking yourself “is this product/feature still delivering value to the customer?” because if it isn’t then it’s “muda” (waste), it’s costing you money and should be retired.

Continuous Obsolescence is the process of continuously grooming your product/feature portfolio and asking yourself “is this product/feature still delivering value to the customer?” because if it isn’t then it’s “muda” (waste), it’s costing you money and should be retired.

Every product/feature that’s not delivering value costs you by:

  • Wasted resources – every build, every release, you’re compiling and releasing code that doesn’t need to be shipped, and consuming server resources once deployed.
  • Overhead of Regression testing – it still needs to be tested every release
  • Unnecessary complexity – every product/feature you don’t need just adds complexity to your environment which complicates both the software development process and the production support process. Complex systems are harder to change, harder to debug and harder to support. If it’s not needed, simplify things and get rid of it!

I’d argue that DevOps could even be defined as
“#DevOps = #ContinuousDelivery + #ContinuousObsolescence”

If you’re doing DevOps and you’re a believer in the CALMS model then you should be practising “Continuous Obsolescence”… in fact I’d argue that DevOps could even be defined as “#DevOps = #ContinuousDelivery + #ContinuousObsolescence” – what do you think?

Send us your comments or join the debate on Twitter @DevOpsguys

-TheOpsMgr

The DevOps Revolution

“We transformed from a team of employees to a team of owners”– Jim Stoneham, Opsmatic

In a fascinating interview with Opsmatic’s Jim Stoneham Gene Kim revisits Flickr’s innovative 2009 ’10 deploys per day’ presentation.

Comparing different approaches to tackling the rapidly evolving platforms within Yahoo! Stoneham highlights the benefits of integrated working across departments, taking risks in order to learn and involving everyone, at every level with every element of the deployment process.

Working quickly, deploying frequently and developing on the fly means that teams learn faster from mistakes and are able to remain ahead of competitors, as well as responding intuitively to the needs of their audience. Teams are also more involved in the process – collaborating across departments means that everyone learns more and the product belongs to the team as a whole,  because they are all involved in every element of its creation, deployment and improvement.

Sharing knowledge and experience across teams means huge pools of resources at your fingertips. You can monitor your audience and give them what they are asking for as they need it. Working slowly to avoid risks will just lead to missed opportunities and an overall slower development. Frequent deploys and rapid responses to audience needs are vital:

When you move at that speed, and are looking at the numbers and the results daily, your investment level radically changes. This just can’t happen in teams that release quarterly, and it’s difficult even with monthly cycles.” Jim Stoneham

Check out the full interview; a  genuinely interesting insight into the start of the DevOps revolution.