Tag Archives: Deployment

BamiWvqCIAAfBgw

Upgrading to Octopus 2.0

On 20th December 2013, Paul Stovell and the Octopus Team released Octopus Deploy 2.0. We’ve had a chance to play with the product for sometime, and it’s a great next version.

Here’s 5 great reasons to upgrade;

  1. Configurable Dashboards
  2. Sensitive variables
  3. Polling Tentacles (Pull or Push)
  4. IIS website and application pool configuration
  5. Rolling deployments

So let’s take a look at how the upgrade process works.

Step 1 – Backup your existing database

Things to check before you upgrade;

  1. Your administrator username and password (especially, if you are not using Active Directory)
  2. Before attempting to migrate, make sure that you don’t have any projects, environments, or machines with duplicated names (this is no longer allowed in Octopus 2.0, and the migration wizard will report an error if it finds duplicates).
  3. IIS Configuration. Are you using ARR or rewrite rules of any kind.
  4. SMTP Server Settings (these are not kept in upgrade)

The first step is to ensure that you have a  recent database backup that you can restore in case anything goes wrong.

1 - Database Backup

Confirm the location of your backup file. You’ll need this file at a later stage in the upgrade process (and if anything goes wrong)

2-Database Backup Location

Step 2 – Install Octopus 2.0

Download the latest installer MSI from the Octopus Website at http://octopusdeploy.com/downloads

Next follow the installer instructions and the Octopus Manager process. This process is really straight forward, and if you’ve installed Octopus before you’ll be familiar with most of the process. There’s a great video here on how to install Octopus http://docs.octopusdeploy.com/display/OD/Installing+Octopus

Step 3 – Import Previous Octopus Database

Once you’ve installed Octopus 2.0, you’ll most likely want to import your old v1.6 database with all your existing configuration. To do this, select the “import from Octopus 1.6″ option and choose the database back-up from the process you followed in step 1 and follow the instructions.

3 - Import Octopus Database

Step 4 – Post Upgrade Steps (server)

When you have completed the installation it’s essential to task a copy of the master key for backups. Instructions on how to backup the key can be found here http://docs.octopusdeploy.com/display/OD/Security+and+encryption

Once you have the server installed there’s a couple of items you’ll need to configure again.

  1. SMTP Server Settings are not kept, so these will need to be re-entered.

Step 5 – Upgrading Tentacles

If you are upgrading from version 1.6 – Octopus 2.0 server can no longer communicate with Tentacle 1.6. So in addition to upgrading Octopus, you’ll also need to upgrade any Tentacles manually. There are two ways to achieve this;

1. Download and Install the new Octopus Deploy Tentacle MSI on each target server

2. Use  a PowerShell script to download the latest Tentacle MSI, install it, import the X.509 certificate used for Tentacle 1.6, and configure it in listening mode. The powershell script is as follows;

function Upgrade-Tentacle
{ 
  Write-Output "Beginning Tentacle installation"
  Write-Output "Downloading Octopus Tentacle MSI..."
  $downloader = new-object System.Net.WebClient
  $downloader.DownloadFile("http://download.octopusdeploy.com/octopus/Octopus.Tentacle.2.0.5.933.msi", "Tentacle.msi")
  
  Write-Output "Installing MSI"
  $msiExitCode = (Start-Process -FilePath "msiexec.exe" -ArgumentList "/i Tentacle.msi /quiet" -Wait -Passthru).ExitCode
  Write-Output "Tentacle MSI installer returned exit code $msiExitCode"
  if ($msiExitCode -ne 0) {
    throw "Installation aborted"
  }
  
  Write-Output "Configuring and registering Tentacle"
   
  cd "${env:ProgramFiles(x86)}\Octopus Tentacle 2.0\Agent"
  Write-Output "Stopping the 1.0 Tentacle"
  
  & .\tentacle.exe create-instance --instance "Tentacle" --config "${env:SystemDrive}\Octopus\Tentacle\Tentacle.config" --console
  & .\tentacle.exe configure --instance "Tentacle" --home "${env:SystemDrive}\Octopus" --console
  & .\tentacle.exe configure --instance "Tentacle" --app "${env:SystemDrive}\Octopus\Applications" --console
  & .\tentacle.exe configure --instance "Tentacle" --port "10933" --console
  & .\tentacle.exe import-certificate --instance "Tentacle" --from-registry  --console
  Write-Output "Stopping the 1.0 Tentacle"
  Stop-Service "Tentacle"
  Write-Output "Starting the 2.0 Tentacle"
  & .\tentacle.exe service --instance "Tentacle" --install --start --console
  
  Write-Output "Tentacle commands complete"
}
  
Upgrade-Tentacle

Step 6 – Upgrade TeamCity Integration

The JetBrains TeamCity Plugin has also been released and requires an upgrade. To do this

  1. Download the plugin from the Octopus Website http://octopusdeploy.com/downloads.
  2. Shutdown the TeamCity server.
  3. Copy the zip archive (Octopus.TeamCity.zip) with the plugin into the <TeamCity Data Directory>/plugins directory.
  4. Start the TeamCity server: the plugin files will be unpacked and processed automatically. The plugin will be available in the Plugins List in the Administration area.

Please Note: The API Key’s for the users will have changed and will need to be re-entered into the Octopus Deploy Runner Configuration within the Build Step

Summary

We think that overall the Octopus Team have done a great job making the upgrade process easy. There’s still a few little issues to iron out, but overall the process is robust and simple. It’s a shame that automatic upgrade of the tentacles couldn’t have been supported as this may cause pain for customers with a large server install base, but Powershell automation makes for a great workaround.

As we learn more about Octopus 2.0, we’ll continue to let you know what we find. Good Luck with Upgrading — it’s definitely worth it!

DevOpsGuys use Octopus Deploy to deliver automated deployment solutions – ensuring frequent, low-risk software releases into multiple environments. Our engineers have deployed thousands of software releases using Octopus Deploy and we can help you do the same. Contact us today – we’d be more than happy to help.

slide2

This week in DevOps #2

#1: The Rise of a New Kind of Administrator

Luke can see the growing trend of admin empowerment within customers and partners from his view at PuppetLabs. His outlook on tools that support admins in complex environments has always been clear, creating tools which allow “you to focus more on the business value of your work and spend a less of your brainpower on implementation details.”

This article looks at the changing role of administrators within organisations as “tools better enable people to build teams to focus on their customer instead of their technology.”

Read more: http://insights.wired.com/profiles/blogs/the-rise-of-a-new-kind-of-administrator

#2: 8 Lessons in Deployment Tooling Lessons Learned

Another great article on learning’s from the UrbanCode guys. This one gives us a history lesson on AnthillPro and what the team learned from creating and using the product. In summary the 8 lessons are;

  1. Builds are about quality and checking quality.
  2. Deployment is a serious challenge.
  3. Developers know jack about audit and security.
  4. Deployment starts with production.
  5. Deployment requires co-ordination.
  6. Deployment starts with high-availability and scalability.
  7. “Release” is more than just deployment.
  8. Multiple integrated tools are not optional.

Read more:  http://architects.dzone.com/articles/8-lessons-deployment-tooling

#3: Playing Pitfall with DevOps and CMDB’s

One paragraph in and Sam delivers a stark warning that  “there are some underlying shortcomings; some so major they might render DevOps redundant in the near future.”

In summary, he outlines the following challenges;

  1. DevOps goes against organizational culture; which advocates for the separation of responsibilities.
  2. Departure from IT Service Management Process
  3. Configuration Management Database redundancy
  4. Inadequate Automation

And so it is, that Sam leaves us with one final point to ponder “Be prepared to say goodbye to DevOps and CMDB, as they are sure to be abandoned in favor of new and improved counterparts in the face of a future IT revolution.”

In response, we ask you to consider Gene Kim’s 3 ways as the underpinning to the DevOps philosophy;

  1. The First Way emphasizes the performance of the entire system – the value stream.
  2. The Second Way is about shorting and amplifying feedback loops.
  3. The Third Way is about creating a culture that fosters continual learning and understanding.

If these truly are the essence of DevOps, are they principles that can ever be abandoned?

Read more: https://www.scriptrock.com/articles/playing-pitfall-devops-cmdbs/?utm_source=twitterfeed&utm_medium=twitter

#4: DevOps Complete Certification Kit

Only 18 Hours and you could be certified! Sounds too good to be true, well it is. As Patrick Debois commented, “they are not getting it!” The geniuses who put this material have really missed the point. We’d love to be certified “DevOps” guys, but instead we’ve focused on recognizing what the practice requires to be implemented. Don’t get us wrong, education on the background to Agile process and the challenges facing Development and Operations teams is vital, but don’t think that 18 hours of study are going to equip you with all you need to become a “DevOps” engineer (a term in itself which is entirely flawed). Remember this…DevOps is a philosophy, not a qualification.

Read more: http://archive.aweber.com/theartofservice/JxFjA/h/DevOps_Complete.htm

#5: Agile Alliance has a DevOps Track

DevOps seeks to address the challenges organizations face when applying change to high velocity environments. Speedy software development handing work over to risk-averse operations need not result in bottlenecks of pessimism. The rise of PaaS, SaaS, IaaS combined with improved understanding of systemic dependencies and business risk is changing how collaboration, automation and operations work in today’s modern enterprises. Mental models are shifting from protection mode to enabling-change mode. The track is going to cover the processes, tools and culture shifts shaping the DevOps movement. We look forward to seeing the output from this. Get submitting!

Read more: http://agile2013.agilealliance.org/program/tracks/devops/

#6: Tweet of the Week

Jesse Keating ‏@iamjkeating

#DevOps lesson: things go well when you’re environment is in an expected state. Not so well when suffering from technical debt. #Fixit

slide2

This week in DevOps #1

TWIDO

We hate acronyms, but this one’s kinda catchy. This Week in DevOps (TWIDO) [[[ yeah, yeah, we promise not to use it again]]], is going to be a Friday round up of stuff we found interesting this week.

We’d love you to contribute, if you have articles, research, tweets on any other stuff you’d like to share, please get involved.

#1:  Q&A From Building a DevOps Team That Isn’t Evil

Q: How have auditors been ok with a devops model where devs have access to prod? e.g., continuous testing, automation, version control, logging, etc?  is one single dev allowed to push a code chg thru the entire process?

Few public companies allow a developer to go from change to production, or access production databases directly. There may still be roles with responsibility for building new features and others that operate on production. The goal is enhance the flow of knowledge between the people in those roles, and for there to be some joint responsibility. For instance, logs may be exposed to developers without giving access to the production box. I think the team at Stackify is trying to make a business around exactly that.

http://architects.dzone.com/articles/qa-building-devops-team-isnt 

Commentary

DevOps does not equal developers managing production. It’s aimed at promoting understand and collaboration between groups.  Thanks to Eric Minick from DevOps Zone and UrbanCode for supporting this.

#2: IT stability and business innovation are not enemies.

Q: DevOps promises a more responsive, more collaborative IT department that can realize business ideas faster. So what is holding back its widespread adoption? What’s the challenge or downside?

DE: There was a movie called “Charlie Wilson’s War” that had a great line between Tom Hanks, playing a U.S. Congressman, and Philip Seymour Hoffman, playing a CIA agent. Hoffman asks, “Why is Congress saying one thing and doing nothing?” Hanks replies, “Well, tradition mostly.”

All jokes aside, tradition is a powerful thing and hard to break. Tradition, or “what we’ve always done,” in IT is no different. There was a thread on Slashdot just this past month that asked whether developers should be allowed to deploy their own applications. You should have seen the outcry. The sheer number of commenters who shot down the idea as pure heresy was shocking. And the richest part of all of their denunciation was that the mob said over and over that the idea would “never work at a real company with real revenue at stake.” I thoroughly enjoyed sending that to John Allspaw, who runs all of technology at Etsy, and Jesse Robbins, who was in charge of risk and disaster planning for operations at Amazon. Etsy does over $600 million of transactions per year, and Amazon does about $50 billion in revenue. In both companies, developers are the ones who deploy and own the uptime for their own code. John’s reaction to the thread was a simple yet priceless one: “OMG.”

http://dev2ops.org/2013/02/

Commentary

Empowering development, operations and even QA teams to take ownership of code deployment is crucial for DevOps teams. It ensures that developers work with operations to understand the impact of change and take accountability/ownership for the “uptime of thier own code”. It also ensures that operations understand what changes are coming and how best to support the change. Uptime is a shared responsibility  with the environment the code executes within and the code that is executing both factors that must be managed correctly. 

#3 – StartOps: Growing an ops team from 1 founder

Bootstrapped startups don’t have the luxury of a full team of ops engineers available to respond to issues 24/7, so how can you survive on your own? This talk will tell the story of how to run your infrastructure as a single founder through to growing that into a team of on call engineers. It will include some interesting war stories as well as tips and suggestions for how to run ops at a startup.

http://devopsdays.org/events/2013-london/

Commentary

We are really looking forward to this track at DevOps Days London. DevOpsGuys are attending. Would be great to meet some of you.

#4: Engineering running the production environment

http://devopsreactions.tumblr.com/post/43882100705/engineering-running-the-production-environment

Commentary

Both funny and awesome! Nuff said!

#5:  Top 10 Practices for Effective DevOps

Practice 1: Active Stakeholder Participation

A fundamental philosophy of DevOps is that developers, operations staff, and support people must work closely together on a regular basis. An implication is that they must see one other as important stakeholders and actively seek to work together. A common practice within the agile community is “onsite customer,” adopted from Extreme Programming (XP), which motivates agile developers to work closely with the business. Disciplined agilists take this one step further with the practice of active stakeholder participation, which says that developers should work closely with all of their stakeholders, including operations and support staff–not just business stakeholders. This is a two-way street: Operations and support staff must also be willing to work closely with developers.

Practice 6: Integrated Deployment Planning

From the point of view of development teams, deployment planning has always required interaction with an organization’s operations staff; in some cases, via liaison specialists within operations typically called release engineers. Experienced development teams will do such planning continuously throughout construction with active stakeholder participation from development, operations, and support groups. When you adopt a DevOps strategy, you quickly realize the need to take a cross-team approach to deployment planning due to the need for operations staff to work with all of your development teams. This isn’t news to operations staff, but it can be a surprise to development teams accustomed to working in their own siloed environments. If your team is not doing this already, you will need to start vying for release slots in the overall organizational deployment schedule. Furthermore, to support continuous deployment, release engineers will need to increase the number of release slots available to agile teams that are disciplined enough to continuously and consistently meet the quality requirements for release.

Practice 7: Continuous Deployment

Continuous deployment extends the practice of continuous integration. With continuous deployment, when your integration is successful in one sandbox, your changes are automatically promoted to the next sandbox, and integration is automatically started there. This automatic promotion continues until the point where any changes must be verified by a person, typically at the transition point between development and operations.

Continuous deployment enables development teams to reduce the time between a new feature being identified and being deployed into production. It enables the business to be more responsive. However, continuous deployment increases operational risk by increasing the potential for defects to be introduced into production when development teams aren’t sufficiently disciplined. Successful continuous deployment in an enterprise environment requires all the practices described earlier.

Commentary

Overall we found this article rather in conflict with itself. It’s interesting how the author views Operations and Development as separate stakeholders within a process. The principles of DevOps are to break this view and to combine Development and Operations as a single entity and in this context they should be considered a single StakeHolder.

We fully agree that Operations should be managers of production with development participation, as Developers should be mangers of development with Operations participation, but on deployment we differ with the author. It’s easy in our view. If operations are a bottleneck to deployment, expand the role of “release engineers” to encapsulate the entire team. Anyone should be able to take accountability for and action deployment if the correct principles and practices have been applied.

The author also battles with Continuous Delivery principles. “release engineers will need to increase the number of release slots available to agile teams”. If you adopt a continuous delivery process your actually going to reduce the number of release slots, to one. One continuous release slot. Hence Continuous Delivery.

http://www.drdobbs.com/architecture-and-design/top-10-practices-for-effective-devops/240149363

#5: Tweet of the Week

Every #ITmanager, no matter how strong he is, lies to himself about his #DevOps initiative. I will find your lie. I will break you

bad-idea-sign

Twelve DevOps Anti-Patterns

So you wanna do DevOps? Okay, but before you start, let’s have a look at some of the things you shouldn’t do.

In the good old days, we just called them “bad ideas”, but along came diplomacy and political correctness, out went the “brain storm” and in came the “idea shower” and with it came the “anti-pattern”.

If the “pattern” is the right way, then inherently the “anti-pattern” is the wrong – and so to stop you going wrong, we’ve compiled this list (with a little help from the DevOps community).

1. DevOps is a process

Not exactly. It’s a philosophy. It’s a way of thinking. DevOps is supported by process and tools.

DevOps according to Gene Kim, is underpinned by 3 core principles known as the “Three Ways”

The First Way emphasizes the performance of the entire system – the value stream.

The Second Way is about shorting and amplifying feedback loops.

The Third Way is about creating a culture that fosters continual learning and understanding.

http://itrevolution.com/pdf/Top11ThingsToKnowAboutDevOps.pdf

2. Agile equals DevOps?

If you’re asking this question, then you’re probably running some agile process. That’s good. You’ve got a software development process that compliments DevOps, but Agile doesn’t mean you’ve adopted DevOps.

DevOps is an agile enabler allowing operations to collaborate supporting a more continuous flow of work into IT Operations and out into production where customers can realize its value.

3. Rebrand your ops/dev/any team as the DevOps

CIO: “I want to embrace DevOps over the coming year.”

MGR: “Already ready done, we changed the department signage this morning. We are so awesome we now have 2 DevOps teams.”

Yeah great. And I bet you now have lots of “DevOps” engineers walking round too. If you’re lucky they may sit next to each other at lunch.

4. Start a separate DevOps group

Go on. I dare you. Done it? Well done. You’ve implemented DevOps. Actually what you just did is create yet another silo. Now you’ve got yourself another team you’ve got to try and integrate. Another team with walls to breakdown. Maybe you could go back and rebrand (see AP: x) and create 3 DevOps teams then you’d be super awesome.

DevOps is not about cherry picking some developers and some IT Operations people and silo’ing them off. You’ve got to embrace and embed. Collapse the development team into the ops team or vice-versa. You need to fully break down the barriers / walls / guards between the teams and mould them into a single unit with shared goals and responsibilities.

http://www.slideshare.net/realgenekim/2012-velocity-london-devops-patterns-distilled

5. The hostile takeover

DevOps. So that’s a word that starts with “Dev”. That means development lead, because development comes first…… Problem?

DevMgr – “Er, we’re now doing DevOps. My guys need to learn the production systems.”

OpsMgr – “Er….ok. So who’s going to be developing the code?”

The word DevOps is clever. It’s a portmanteau. This means the combination of two words, to form a new word, which gives a new meaning. It even delivers some efficiency. It doesn’t mean we took the word operations and replaced it with the word development. So why would you try and adopt DevOps in that manner?

DevOps requires both groups to recognise their key skills. Share what needs to be shared to collaborate. Learn what needs to be learnt to improve. It does not mean retraining. It does not mean cross-skilling (however, this may be a welcome side-effect). It does mean providing feedback and visibility to improve.

http://www.slideshare.net/realgenekim/2012-velocity-london-devops-patterns-distilled

6. DevOps is a buzz word

If you think DevOps is a buzz word, then you’ve probably been using “The Cloud” as a misnomer too. DevOps is a word, you got that right. Actually, it’s a portmanteau of Development and IT Operations (I’ll collect my gold star from teacher later).

DevOps is more than just a cool buzz word. It’s a state of mind. You must embrace its values, you must help others embrace its values and you must continually improve yourself and help others to improve for it to be successful. Once you throw away the BS and start collaborating you might get people to think your catchy new word “DevOps” might actually be cool.

http://www.slideshare.net/realgenekim/2012-velocity-london-devops-patterns-distilled

7. Sell DevOps as a silver bullet

DevOps is voodoo. You basically get your Development team and your IT Operations team together. They smoke some peyote and then sacrifice a chicken. Once you’ve done that your organisation will be revolutionised. You’ll be able to ship software faster than ever before. Configuration will be self-managing. Your deployment tools will become self-aware. Your development and IT Operations teams will have a new found harmony.

Get this….. DevOps is hard work! For most it requires Culture Change! That’s one of the hardest things you’ll ever attempt. For seasoned development and IT Operations teams you’re about to try and turn their world up-side down. Don’t try and do it overnight or you will fail.

8. DevOps means Developers Managing Production

No. Hell No and No again. I’m so incensed you’ll just have to read this….

9. DevOps is Development-driven release management

Let me get 2 things clear;

  1. DevOps is not development driven.
  2. DevOps is not IT Operations driven.

If you want a developer driven environment, fine, go create one. Just don’t call it DevOps. It’s not.

Take a look at this article for example.

http://www.computerweekly.com/cgi-bin/mt-search.cgi?IncludeBlogs=113&tag=release%20management&limit=20

“Within DevOps, programmers are programmers.” – Right on!

“Equally, within DevOps, operations staff are operations staff.” – We’re cooking now!

“Traditionally, getting software out to production can be either the responsibility of operations, or of the development team.”

Hang on……

“IT operations teams will have established and trusted deployment strategies in place that minimise downtime and ensure stability at the expense of agility and speed of response.”

Yep, we’re back on track…… and then bam!

“Development-driven release management” – WTF? It gets worse

“Development-driven release management goes the other way and looks at how deployment can be carried out as often and easily as possible. However, these deployments aren’t necessarily into production……………From a process standpoint, continuous delivery has two big requirements: first, the process itself has to be solid beyond development. This means that it has to be as solid as any process that a traditional IT operations team might put into place.”

No. Non. Nej. Na. Nee. Nein.

Development-driven anything may be a process. It’s just not DevOps. Replacing your IT Operations team with an automated deployment process is nonsense. Please don’t try this at home folks!

10. We can’t do DevOps – We’re Unique

Yes you are, you little beauty you! But you’re not special enough that you can’t adopt DevOps. I bet you’re the best developer out there; you code quicker than lighting, and deliver the sort of code that makes grown men cry with joy. No? Okay, so you’re the most awesome Ops Guy on planet. If Chuck Norris were an IT Operations Engineer he’d be want to be you. However, you and your organisation don’t have some unique factor that won’t allow you to adopt DevOps. So give it a go!

Jesse Robbins from OpsCode has some good advice for getting started;

  • Start Small – Build Trust
  • Create Champions
  • Build Confidence
  • Celebrate Success
  • Exploit Circumstance

http://www.slideshare.net/jesserobbins/cloud-expo-jesserobbinsopscode20130129b

https://www.ibm.com/developerworks/mydeveloperworks/blogs/ambler/entry/adoption_antipattern_we_re_special?lang=en

11. We can’t do DevOps – We’ve got the wrong people

Well why did you hire them? That’s right – they’re awesome! If you don’t think that, then you need to take a long hard look at yourself, then go and discover the real hidden talents in your team.

Someone told me recently that they couldn’t do DevOps because “they have the wrong developers or the wrong ops people…”. So they have developers who can’t code? I thought to myself, “my organisation has the wrong developers, people who can’t code, they run HR and Marketing!”

DevOps fosters a collaborative working relationship between Development and IT Operations. This collaboration can extend right through the organisation further enhancing working relationships between teams.

You don’t have the wrong people. You have the wrong thought process. Deal with it.

12. Collaboration when the S**T hits the fan

Ok genius. You f**ked up. So what? We all do it. But now you want your IT Operations guys out of bed at 2am to clean-up something they know nothing about. They are IT Operations engineers – not the “fixer” like Michael Clayton. Waiting until an error occurs during a deployment for Development and IT Operations to collaborate sucks.

It’s too late for this problem….. but maybe not for the next. You have your Development team and your IT Operations team talking (or swearing at 2am) with each other, but at least they are talking. Keep the dialog going. Get a retrospective review of what happened and how you can fix it going forward. If you have encountered this situation, then try and keep the dialog going with between your teams. Open the communication channels with Development and IT Operations early. There’s hope for you yet!

http://cdn.dzone.com/sites/all/files/refcardz/rc145-010d-continuousdelivery_0.pdf

– TheDevMgr

scream

DevOps does not equal “Developers managing Production”

I’ve had a few conversations lately, mainly with smaller start-ups or development houses, who tell me “yes, we work in a DevOps model”.

What they really mean is “We pretty much have no Operations capability at all, and we rely on the Developers to build, deploy and manage all of the environments from Development to Test to Production. Mostly by hand. Badly”.

As someone from a predominately Operations background I find this quite frustrating!

Operations is a discipline, with its own patterns & practices, methodologies, models, tools, technology etc. Just because modern cloud hosting makes it easier to deploy servers without having to know one end of a SCSI cable from another doesn’t mean you “know” how to do Operations (just like my knowledge of SQL is enough to find out the information I need to know monitor and manage the environment but a long way from what’s required to develop a complex, high-performing website).

DevOps means Development and Operations working together collaboratively to put the Operations requirements about stability, reliability, performance into the Development practices, whilst at the same time bringing Development in the management of the Production environment (e.g. by putting them on-call, or by leveraging their development skills to help automate key processes).

It doesn’t mean a return to the laissez-faire “anything goes” model where developers have unfettered access to the Production environment 24x7x365 and can change things as and when they like.

Change control was invented for a reason, and whilst change control has becomes its own “cottage industry” involving ever more bureaucratic layers of “management by form filling” the basic discipline remains sound – think about what you want to change, automate it if you can, test it, understand what to do if it screws up (roll back plan), document the change, make sure everyone knows when, where and how you are making the change, and make sure the business owner approves.

When I took over the Operations of a high-volume UK website about 8 years ago I spend the first 3 weekend working fighting fires and troubleshooting Production issues.

My first action after that baptism of fire was to revoke access to production for all developers (over howls of protests). Availability and stability immediately went up. Deafening silence from Development – Invitations to beers from the Business owners.

Next step was to hire a build manager to take over the Build and Deployment automation, and a Release Manager to coordinate with the Business what was going into each release, when etc. End result – 99.98% availability, with more releases, being deployed within business hours without impacting the users, and a lower TCO. The Business was much happier, and so was the Development Manager, as he was losing far fewer developer-hours to fire-fighting Production issues, and hence the overall development velocity improved considerably. Win-Win.

Was that a DevOps anti-pattern? Did I create more silos? Probably… but in a fire-fight a battlefield commander doesn’t sit everyone down for a sharing circle on how they are going to address the mutual challenge of killing the other guy before he kills you. Sometimes a command & control model is the right one for the challenge you face (like getting some supressing fire on the target while you radio in for some air support or artillery!).

That said, once we had developed a measure of stability we did move partway to a more DevOps pattern – we had developers on-call 24×7 as 3rd line support, we virtualised our environment(s) and gave Developers more control over them, and we increased our use of automation.

Organisationally we remained siloed however – we were incentivised in different ways (Operations emphasising availability, Development emphasising feature delivery), we remained in essentially a waterfall delivery model and Ops VS Dev was a constant struggle for manpower & resources. All the usual problems that the DevOps movement is trying to address.

In summary, what I am trying to get at is please don’t devalue the “DevOps” concept by saying you do DevOps when you don’t.

Unless you currently do both Development AND Operations separately, and do them well, AND you’re now trying to synthesise a better, more agile, more cloud-oriented way of working that takes the best part of BOTH disciplines… you aren’t doing DevOps!

– TheOpsMgr

Maturing the Continuous Delivery Pipeline

The Maturity Model is a useful assessment tool for understanding your organizations level of Continuous Delivery adoption. Many organizations today have achieved what is needed to move from Level-1 (Regressive) to Level-0 (Repeatable), which is a significant accomplishment and as a reader of this blog post, you’re either about to start your journey of improvement or are already underway.

Continuous Delivery Maturity Model
Continuous Delivery Maturity Model

The Maturity Model

The advice for organizations wanting to adopt Continuous Delivery is ever more abundant, but for organizations that started adoption some time ago, the guidance on how to mature the process is still sparse. In this article, we explore one continuous improvement methodology that may help your organization mature its’ Continuous Delivery process.

Humble and Farley outline maturity Level 0 (Repeatable) – as one having process which is documented and partly automated.1 For this to be true, an organization must have first classified its’ software delivery maturity, identified areas for improvement, implemented some change and measured the effect. As Humble observes;

The deployment pipeline is the key pattern that enables continuous delivery.2

Humble also identifies that Deming’s Cycle is a good process to apply to initial adoption. 1 The process, according to Deming, should then be repeated so that further improvements can be planned and implemented; having the advantage the  data and experience from previous cycles is available. This process of continuous improvement is the first step to maturing the continuous delivery process.

Continuous Delivery and Kaizen

Continuous Delivery is a set of practices and principles aimed at building, testing and releasing software faster and more frequently.3 One key philosophy at the heart of Continuous Delivery, is Kaizen. The Japanese word, Kaizen, means continuous improvement. By improving standardized activities and processes, Kaizen aims to eliminate waste.  Within Kaizen are three major principles;

  1. 5S
  2. Standardization
  3. The elimination of waste; known as muda.

The 5S principle characterizes a continuous practice for creating and maintaining an organized and high-performance delivery environment. As 5S has the focus of waste reduction, it appears prudent to explore the 5s methodology as an approach to optimizing the Continuous Delivery pipeline.

What is 5S?

5S is a lean engineering principle focused on waste reduction and removal. The methodology is recursive and continuous and it’s primary goal is to make problems visible  so that they can be addressed.4

There are five primary 5S phases, derived from five Japanese words that translate to; sort (seiri), straighten (seiton), shine (seiso), standardize (seiketsu) and sustain (shitsuke).5

The 5s principle is more commonly applied to workplace organization or housekeeping. Since the delivery pipeline is a key component of the process, it is therefore a fundamental place where work will take place. The pipeline exhibits elements that are common with any physical work environment including;

  1. Tools, process, paperwork.
  2. Storage areas for artifacts, such as files.
  3. A requirement that components must be, physically or virtually, located in the correct location, and that they are connected by process.
  4. Tools and process need maintenance and cleaning. E.g. retention policies.
  5. Systems and procedures can be standardized and can benefit from standardization.

When applied to the delivery pipeline, the five primary 5S phases can be defined as;

  • Phase 1 – sort (seiri)
    • Analysis of processes and tools used in the delivery pipeline. Tools and processes not adding value should be removed.
  • Phase 2 – straighten (seiton)
    • Organization of processes and tools to promote pipeline efficiency.
  • Phase 3 – shine (seiso)
    • Inspection and pro-active/preventative maintenance to ensure clean operation throughout the delivery pipeline.
  • Phase 4 – standardize (seiketsu)
    • Standardization of working practice and operating procedure to ensure consistent delivery throughout pipeline.
  • Phase 5 – sustain (shitsuke)
    • Commitment to maintain standards and to practice the first 4S. Once the 4S’s have been established, they become the new way to operate.

Implementing 5s in the Delivery Pipeline

Experience has shown that there are several key factors that should be considered in the adoption of 5S.

  1. Begin with small changes. Remember that 5s is a continuous and recursive process.
  2. Involvement is crucial.6 Encompass everyone involved in the deployment pipeline.
  3. Create Champions – find leaders who love &  live the philosophy and encourage them to promote it.

5s implementation involves working through each phase (or S) in a methodical way. This is commonly achieved by Kaizen events, which are focused activities, designed to quickly identify and remove wasteful process from the value stream and are one of the prevalent approaches to continuous improvement.

Phase 1 – sort (seiri)

The first step of 5s is to sort. The purpose is to identify what is not needed in the delivery pipeline and remove it.

Additional process and practice can easily creep into the delivery mechanism, especially in the early phases of Continuous Delivery adoption, when you are experimenting.

The removal of items can either mean total elimination of a tool or process, since it is redundant, or the flagging, known as “Red Tagging”, of a tool or process which needs to be evaluated before it is disposed of. The review of red tagged tools and process should evaluate their importance and determine if they are truly required.

Suggested Steps

1. Evaluate all tools and process in the delivery pipeline and determine what is mandatory.

2. Ascertain what is not needed. Assess what it’s elimination status is, e.g. immediately or Red Flag.

3. Remove unnecessary or wasteful tools and  process.

Phase 2 – straighten (seiton)

The second step requires items in the delivery pipeline to be set in order. This requires the pipeline to be arranged so that  maximum efficiency is achieved for delivery.

To achieve seiton, the key components of the automated pipeline should be analysed, and should include the following;

  • Automated build process.
  • Automated unit, acceptance tests including code analysis.
  • Automated deployment process.

It is suggested that for the pipeline to be truly efficient the most commonly used items are the easiest and quickest to locate, such as;

  • Build status reports and metrics including KPI’s such as Mean-time-to-release (MTTR)
  • Automated test coverage, error and defect reports

In this step, actions can also be taken to visually label elements of the pipeline so that demarcation of responsibility  is clear. This may mean categorizing tools, process and target locations E.g. servers into  groups such as production and pre-production or development and test.

Suggested Steps

1. Analyse the value stream including tools and process.

2. Ensure the necessary information is available quickly to those who need it.

3. Visually distinguish or categorize key pipeline components.

Phase 3 – shine (seiso) 

Once the waste has been removed and only what is required has been kept and ordered, the next phase inspects the delivery pipeline to ensure that it’s operation is as clean as possible.

During pipeline execution large amounts of data can be produced. If this information is not dealt with efficiently it can become a waste product, detracting from the value of the delivery process.

Inspection of this data is vital. Data generated in the pipeline is likely to include instrumentation, metrics and logging information, as well as build artifacts and report data.

In context of the delivery pipeline, cleaning can be defined as archiving, roll-up or removal of delivery pipeline data. House keeping routines should be evaluated to ensure that data retention rules are applied and that waste, in the form of build artifacts, metric and report data and instrumentation data are correctly archived or removed as required.

Implementation of this phase requires assignment of cleaning responsibilities and scheduling of cleaning process.

Suggested Steps
1. Inspect data to understand what should be rolled-up, archived or deleted.

2. Ensure that responsibilities for data management are clearly assigned within the team.

3. Create a scheduled, automated cleaning processes.

Phase 4 – standardize (seiketsu)

The primary focus of step four is to ensure that  the working practice, training and operating procedures remain consistent. Standardization allows for continuous improvement and each of the first 3S’s should be regulated.

A critical element of seiketsu is to ensure that the process is reviewed on a regular basis or cycle. This ensures that best practices are maintained and areas  where standards have slipped are quickly identified.

A settled and regular process or practice becomes one that is hard to give up. Therefore,  if standardization, is implemented correctly, it can support culture change.

Suggested Steps

1. Ensure that the team have clear responsibilities for the tasks that need to be completed.

2. Create ownership and pride in the process.

3. Ensure champions provide guidance and support changes to the 5s process.

Phase 5 – sustain (shitsuke)

The purpose of the final phase is to ensure that the 5s process is not a one-time event. To achieve this,  the focus of the fifth  phase centers on culture. To sustain 5s organizations must engage in support, commitment and gainful communication, with the aim of keeping the team devoted to continuous improvement.

There are a number of elements which influence cultural support of 5s . Which elements take precedence varies within every organization and is heavily dependent on the existing culture of that organization. Key points are:

  1. Communication – aims & objectives need to be clear for all  team members.
  2. Education –  5s methodology, concepts and practices need to be communicated.
  3. Recognition –  team members need to feel that their efforts are recognized.

It is interesting to note that the literal translation of shitsuke is discipline. The final phase of 5s can also be one of the hardest to implement.

Suggested Steps

1. Establish an open channel for discussion and feedback, to facilitate continual learning.

2. Adopt the habit of auditing regularly and reward the best teams.

3. Ensure that all  problems are addressed quickly make sure that root cause analysis is completed to discover the cause.

Summary

The adoption of Kaizen events to implement 5s within the Continuous Delivery pipeline can provide significant support to maturing the model, by providing a continuous improvement process within a well-defined and standardized structure.

Lean manufacturing has been using the 5S pratice as a common method for lean implementation for years. Nearly 70% of of lean manufacturing companies use this approach 7, but the use of 5S as an adoption method for lean in software development would appear less common, as it has not be commonly applied to development practices.

Lean software development principles provide the promise of companies becoming more competitive by delivering software faster, increasing quality, and reducing cost leading to greater customer satisfaction, all of which align closely to the key principles of  Continuous Delivery.

– TheDevMgr

References

1. Humble, J. Farley D. (2010). Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Indiana: Addison Wesley. 419.

2.  Humble, J. Farley D. Continuous Delivery: Anatomy of the Deployment Pipeline.  [September 7, 2010.]  InformIT. http://www.informit.com/articles/article.aspx?p=1621865

3. McGarr, Mike. Continuous Delivery. SlideShare.net [June 17, 2011.] http://www.slideshare.net/jmcgarr/continuous-delivery-8341276

4. Liker, Jeffery K. The Toyota Way. New York City : McGraw Hill, 2004.

5. MLG UK. 5S – The Housekeeping Approach Within Lean. MLG UK. [Cited: February 5th 2013] http://www.mlg.uk.com/html/5s.htm

6. Maready, Brian. Transparency = speed . Lean Lessons. [May 27, 2010] http://leanbuilt.blogspot.com/2010/05/transparency-speed.html.

7. Compdata Surveys. Lean Manufacturing and Safety Help Manufacturers Survive Tough Times.  Compdata Surveys. [December 13, 2010.] http://www.compdatasurveys.com/2010/12/13/lean-manufacturing-and-safety-help-manufacturers-survive-tough-times/

Automated release management for .NET with OctopusDeploy

Introducing OctopusDeploy

We are big fans of OctopusDeploy and have been using it since its beta release in early 2012. If you’re not familiar with OctopusDeploy, here’s the background;

1. It’s an automated release management tool for Microsoft .NET, comprising of a centralized server and agents, known as Tentacles, which are installed on each target server.

2. It uses the NuGet package format,  an open source package manager for the Microsoft .NET Framework.

3. Tentacle agents listen on TCP port 10933 by default and HTTP requests are encrypted using a pair of X509 certificates.

The following diagram shows the release management components;

Deployment Components
Deployment Components

Release Management On-premise or Off-premise

One of the highlights of using OctopusDeploy is its’ ability to deploy software to servers located in any environment. The highlight reel goes a little something like this;

Whether your servers live in the cloud, in a data centre, or under your desk , Octopus can securely upload, install and configure your applications.

Just install a tiny service agent called Tentacle on your servers, and then use our easy installation wizard to set up a secure connection between the Octopus and Tentacle based on public/private key encryption technology. Your server is ready for deployments!

We’ve installed OctopusDeploy many times and have used some of the following installation scenarios to deploy to on-premise and off-premise data centres.

Scenario 1: On-Premise Installation with On-Premise Deploy

One of the more common ways of installing Octopus is one where the NuGet Server and Octopus Server are located on your own network,  along-side Continuous Integration servers or source control systems.

In this installation, OctopusDeploy will pull NuGet packages from a NuGet Server (such as MyGetJetBrains TeamCity, a NuGet Feed Server or just a local file path) and push the deployment out to target servers located on a local network.

Since all servers are located on the same network, this configuration is straight-forward and requires not much more than Windows Firewall configuration to be completed,  where TCP traffic over port 10933 (presuming the default ports) is allowed between the OctopusDeploy server and the target servers.

On-Premise Installation with On-Premise Deploy
On-Premise Installation with On-Premise Deploy

Scenario 2 :On-Premise Configuration with Off-Premise Deploy

This installation presumes that your target servers are not within your own network. E.g. hosted in an off-premise data centre E.g. a cloud environment. Configuration is the same as Scenario 1, and Windows Firewall will still need to be configured.

However, in this scenario it is likely that additional layers of security will be in place E.g. a hardware firewall. Network teams will need to ensure that TCP traffic on the OctopusDeploy port (10933 by default) is allowed into each server.

This deployment scenario has some complications however. Each target server needs to be publicly addressable. For load balanced server farms this is going to add a level of complexity, as firewalls and content switches/load balancers will need to be configured to allow each target server to be individually and publicly available to an external caller.

More common, is the fact that servers located in the off-premise data centre might not be publicly addressable with good reason. E.g. Web Servers in a DMZ can receive traffic originating from public networks, whereas application and database servers (and firewalls) are configured only to receive traffic generated within the off-premise data centre.

If you do decide to expose servers publicly to allow Octopus to deploy from an on-premise to an off-premise location, you can take some solace from the fact that the Paul Stovell from OctopusDeploy has ensured deployment is secured with HTTP requests that are encrypted using a pair of X509 certificates.

On-Premise Configuration with Off-Premise Deploy
On-Premise Configuration with Off-Premise Deploy

Scenario 3 :Off-Premise Installation with Off-Premise Deploy

Finally, you might want to consider a scenario where the Octopus Server is hosted within the off-premise data centre that software will be deployed into.

The NuGet server can remain on-premise in this scenario. Since Octopus will generate outbound HTTP (TCP:80) traffic to a known destination ( in our experience ), security guys (not all) are cool (don’t complain complain less) with this approach.

Advantages over Scenario 2

1. Octopus and target servers are now located within the same network, or are on trusted networks.

2. Servers do not have to be made publicly addressable. Non public servers can be deployed to.

3. Load balanced challenges in Scenario 2 are overcome. Octopus can now individually address all servers.

4. Deployment time can be reduced, as NuGet packages are transferred once from NuGet source and are deployed across local or trusted networks.

Disadvantages

1. Additional server infrastructure is required in data centre to host OctopusDeploy Server.

Off-Premise Installation with Off-Premise Deploy
Off-Premise Installation with Off-Premise Deploy

We hope that’s given you a flavor of the installation scenarios available to Octopus Deploy! More information on OctopusDeploy can be found at http://octopusdeploy.com

DevOpsGuys use Octopus Deploy to deliver automated deployment solutions – ensuring frequent, low-risk software releases into multiple environments. Our engineers have deployed thousands of software releases using Octopus Deploy and we can help you do the same. Contact us today – we’d be more than happy to help.