This week in DevOps #3

Hey, it’s that time again and this week I start with an apology as we missed a week due to the awesome DevOpsDays London and a not so awesome bout of Tonsillitis.

DevOpsDays London

DevOpsDays London – easily the best conference/unconference I’ve ever attended. Great people, great speakers, OpenSpaces was awesome, food was good and to top it all off – free beer. Sadly, I missed the Saturday due to illness (not caused by the free beer, or the Wales Vs England rugby game).

Highlights;

  • @niekbartho gave a solid opening presentation, monitoring his heart rate as he went. Some good real world advice and a story from the trenches is always a nice touch. I felt the story was a little more focused on continuous delivery, rather than DevOps. Niek did clarify the term “Snowflake Server” very nicely as being “a server having configuration as unique as a snowflake, and no one has a clue how it was built!” Thanks Niek.
  • @johnC_bristol  gave us an insight into key cultural aspects of DevOps and how they monitored them at Nokia. It was a really interesting insight into how culture affects DevOps with some real information to backup theory. John completes a study with involvement of over 50 employees and presented some interesting findings.
  • @DavidMytton gave a great insight into bootstrapping a startup. Some good advice on life hacks relating to Startups relating to mobile usage, travelling, etc. His advice was wide ranging, but our favourite item was on task automation. “Do it once, Learn, automate!” — Thanks David. The ServerDensity team have produced some content DevOps Dayers which is worth a read.
  • In the final presentation before the ignite sessions, Deri Jones enlightened us on using metrics to calculate lost sales. His content focused on ensuring that all the great metrics captured by DevOps should be translated back to business measurements.
  • The ignite sessions brought some great fast paced presentations. I really enjoyed @simonvc on “How schemaless databases fit into the continuous delivery world”. His thoughts;

My final thoughts on DevOpsDays; awesome people, awesome event format but even at a DevOps event, it was surprising how many different ways people interpreted the terms DevOps. Here a just a few;

  • DevOps  … is a movement, a philosophy, a way of thinking.
  • DevOps  … is a person who can perform both Dev and Ops roles.
  • DevOps … means cross skilling people.
  • DevOps … is continuous delivery.
  • DevOps … is a team of developers and operation staff.
  • DevOps …is a culture movement.
  • DevOps … is monitoring.

I have my views on what DevOps is and more importantly is not. It seems that there is still some teaching to be done!

Thanks to all those at DevOpsLondon 2013. It was great to meet you.

What we found this week

  • Citrix outfits NetScaler with troubleshooting tools to help sysadmins troubleshoot the cause of poor application performance from multiple data centres, cloud services and networks. NetScaler can emit performance data in the AppFlow format, which third-party diagnosis tools such as Splunk.
  • The benefits of DevOps are numerous. First and foremost, there is more collaboration and trust within an organization. Where are you on the timeline? Where do you want to be?

Tweets of the Week

  • @patrickdebois: Mgmt wants us to replace shell scripts to become devops, /me Do they work now? Yes /me I suggest you find your real bottleneck.
  • @rposbo: Loving that there’s no DevOps Manifesto; just get everyone on the same goal, automate what you can, don’t be evil!
  • @withneedle: Pro tip: Changing everyone’s title to “#devops engineer” will not magically make it so.
  • @DevOpsBorat: In startup we are only use technology if is cover in blog of expert devops on benchmark is run on own laptop.
  • Interesting Idea. The Twelve Factor App is a methodology for building software-as-a-service apps  that can be applied to apps written in any programming language, and which use any combination of backing services.

Continuous Delivery Adoption Barriers

For about a month, DevOpsGuys has been running a survey to understand what barriers exist for Continuous Delivery adoption. We’ve analysed the results and published our findings below. We’ve love to hear your feedback, so let us know what you think through the comments section.

barriers to cd

1. Organisation Culture (41%)

Organisational culture is probably the most important aspect to consider when adopting sustainable Continuous Delivery principles. 41% of respondents said that organisational culture was seen as the primary barrier to Continuous Delivery adoption.

Organizational culture is capable of blunting or significantly altering the intended impact of even well-thought-out changes in an organization.

Organisations seeking to adopt Continuous Delivery will primarily focus on evaluating and improving working practice, but to truly embrace Continuous Delivery, organisations must ensure that their employees encompass its primary principles. Continuous Delivery adoption is usually impossible unless management is bought in and at least some of the engineers are willing to change the way they work.

As organizations mature their Continuous Delivery adoption they must seek to embrace a culture which;

  • Is open, honest and transparent.
  • Encourages collaboration.
  • Promotes innovation, accountability and responsibility.
  • Builds and Rewards trust across organizational boundaries.
  • Enhances visibility of change and risk.

Organizational change is hard; it’s well known, theorised and documented and culture is the most difficult organisational attribute to change. For Continuous Delivery to be truly successful, the entire organisation must adopt its philosophy; to provide high-quality, valuable software as quickly as possible.

2. Lack of integrated development and operations (DevOps) (19%)

DevOps does provide an implementation strategy for Continuous Delivery principles, but neither philosophy is truly dependent on the other.  Continuous Delivery and DevOps are working towards common goals by providing business value through software delivery, within a culture that enables collaboration and understanding between the functional groups that deliver IT services where everyone is responsible for delivery.

Within the DevOps movement, Silo’s between teams are being removed so that software can be delivered quicker. That is, business value can be delivered quicker.

Developers are learning how to create production-ready software that can always be deployed, thus delivering high throughput and Operations are learning that Agile principles can enable effective and low risk change management, thus protecting stability.

recent study by Forrester Consulting and ThoughtWorks found that;

  • Companies are looking to prioritize innovation through developing software services.
  • Software development providers can’t deliver new services at the rate business leaders want.
  • Corporate culture and development process immaturity impede communication and slow service delivery.
  • Few IT organizations regularly perform advanced continuous delivery practices.

Therefore, it is not surprising that 19% of respondents consider a lack integrated development and operations capability as an adoption barrier for Continuous Delivery.

3. Technical hurdles (15%)

While Organizational Culture might be the primary adoption barrier for Continuous Delivery, there are also a number of technical hurdles to overcome. The technical challenges fall into 4 broad categories.

  • Configuration Management
  • Continuous Integration
  • Automated Testing
  • Automated Deployment

In each of these areas tools and associated best practice are becoming more advanced. Infrastructure-as-Code as promoted by OpsCode and Delivery Pipeline Management tools such as ThoughtWorks Go are being combined with an ever growing set of Cloud enabled services and platforms to make Continuous Delivery adoption easier from a technical viewpoint. However 15% of respondents believe that technical hurdles are significant in preventing adoption of Continuous Delivery.

4. Lack of understanding (15%)

Education of Continuous Delivery principles was seen as a big adoption barrier by 15% of respondents. Lack of understanding spans each of the possible responses in our survey from organisational culture through technical knowledge.

But there are more basic misunderstandings of the terminology, principles and practice of Continuous Delivery. As Paul Stack points out Continuous Delivery is not continuous deployment. Paul correctly defines Continuous Delivery as the process of having shippable software after each check in to source control, whereas Continuous Deployment is the process of shipping your product after every check in to source control. The difference is subtle but significant.

Another misconception is that Continuous Delivery and DevOps are basically the same thing. As Stephen Smith observes, Continuous Delivery and DevOps are interdependent, not equivalent. Some of this confusion may have arisen from the  misinterpretation of the DevOps principles leading to differing opinions of the relationship  between Continuous Delivery and DevOps, especially as “People often talk about DevOps and Continuous Delivery in the same breath.” as Jeff Sussna notes.

5. Business readiness to accept change at a fast pace (10%)

In many organizations, change can often feel constant. However as Isaac Asimov said “The only constant is change”. In general, people don’t like change, so methodology that promotes the acceleration of change is likely to face opposition. Only 10% of respondents saw business readiness to accept change at a quicker pace as a barrier to Continuous Delivery adoption. However, this may be influenced by the mainly technical community completing the survey.

Reducing the Mean-time-to-deploy can have significant business advantages. Continuous Delivery aims to provide high-quality and valuable software as quickly as possible. That is, to deliver business value is less time, whilst protecting the value quality. Given, respondents saw this reason as the least important factor is preventing Continuous Delivery adoption – could mean that organizations are aware of the benefits of getting to market faster.

For organizations aware of the benefits of delivering value at a quicker pace, and for those willing to embrace cultural change, Continuous Delivery provides a set of principles, which if implemented can provide significant benefit.

This week in DevOps #2

#1: The Rise of a New Kind of Administrator

Luke can see the growing trend of admin empowerment within customers and partners from his view at PuppetLabs. His outlook on tools that support admins in complex environments has always been clear, creating tools which allow “you to focus more on the business value of your work and spend a less of your brainpower on implementation details.”

This article looks at the changing role of administrators within organisations as “tools better enable people to build teams to focus on their customer instead of their technology.”

Read more: http://insights.wired.com/profiles/blogs/the-rise-of-a-new-kind-of-administrator

#2: 8 Lessons in Deployment Tooling Lessons Learned

Another great article on learning’s from the UrbanCode guys. This one gives us a history lesson on AnthillPro and what the team learned from creating and using the product. In summary the 8 lessons are;

  1. Builds are about quality and checking quality.
  2. Deployment is a serious challenge.
  3. Developers know jack about audit and security.
  4. Deployment starts with production.
  5. Deployment requires co-ordination.
  6. Deployment starts with high-availability and scalability.
  7. “Release” is more than just deployment.
  8. Multiple integrated tools are not optional.

Read more:  http://architects.dzone.com/articles/8-lessons-deployment-tooling

#3: Playing Pitfall with DevOps and CMDB’s

One paragraph in and Sam delivers a stark warning that  “there are some underlying shortcomings; some so major they might render DevOps redundant in the near future.”

In summary, he outlines the following challenges;

  1. DevOps goes against organizational culture; which advocates for the separation of responsibilities.
  2. Departure from IT Service Management Process
  3. Configuration Management Database redundancy
  4. Inadequate Automation

And so it is, that Sam leaves us with one final point to ponder “Be prepared to say goodbye to DevOps and CMDB, as they are sure to be abandoned in favor of new and improved counterparts in the face of a future IT revolution.”

In response, we ask you to consider Gene Kim’s 3 ways as the underpinning to the DevOps philosophy;

  1. The First Way emphasizes the performance of the entire system – the value stream.
  2. The Second Way is about shorting and amplifying feedback loops.
  3. The Third Way is about creating a culture that fosters continual learning and understanding.

If these truly are the essence of DevOps, are they principles that can ever be abandoned?

Read more: https://www.scriptrock.com/articles/playing-pitfall-devops-cmdbs/?utm_source=twitterfeed&utm_medium=twitter

#4: DevOps Complete Certification Kit

Only 18 Hours and you could be certified! Sounds too good to be true, well it is. As Patrick Debois commented, “they are not getting it!” The geniuses who put this material have really missed the point. We’d love to be certified “DevOps” guys, but instead we’ve focused on recognizing what the practice requires to be implemented. Don’t get us wrong, education on the background to Agile process and the challenges facing Development and Operations teams is vital, but don’t think that 18 hours of study are going to equip you with all you need to become a “DevOps” engineer (a term in itself which is entirely flawed). Remember this…DevOps is a philosophy, not a qualification.

Read more: http://archive.aweber.com/theartofservice/JxFjA/h/DevOps_Complete.htm

#5: Agile Alliance has a DevOps Track

DevOps seeks to address the challenges organizations face when applying change to high velocity environments. Speedy software development handing work over to risk-averse operations need not result in bottlenecks of pessimism. The rise of PaaS, SaaS, IaaS combined with improved understanding of systemic dependencies and business risk is changing how collaboration, automation and operations work in today’s modern enterprises. Mental models are shifting from protection mode to enabling-change mode. The track is going to cover the processes, tools and culture shifts shaping the DevOps movement. We look forward to seeing the output from this. Get submitting!

Read more: http://agile2013.agilealliance.org/program/tracks/devops/

#6: Tweet of the Week

Jesse Keating ‏@iamjkeating

#DevOps lesson: things go well when you’re environment is in an expected state. Not so well when suffering from technical debt. #Fixit

This week in DevOps #1

TWIDO

We hate acronyms, but this one’s kinda catchy. This Week in DevOps (TWIDO) [[[ yeah, yeah, we promise not to use it again]]], is going to be a Friday round up of stuff we found interesting this week.

We’d love you to contribute, if you have articles, research, tweets on any other stuff you’d like to share, please get involved.

#1:  Q&A From Building a DevOps Team That Isn’t Evil

Q: How have auditors been ok with a devops model where devs have access to prod? e.g., continuous testing, automation, version control, logging, etc?  is one single dev allowed to push a code chg thru the entire process?

Few public companies allow a developer to go from change to production, or access production databases directly. There may still be roles with responsibility for building new features and others that operate on production. The goal is enhance the flow of knowledge between the people in those roles, and for there to be some joint responsibility. For instance, logs may be exposed to developers without giving access to the production box. I think the team at Stackify is trying to make a business around exactly that.

http://architects.dzone.com/articles/qa-building-devops-team-isnt 

Commentary

DevOps does not equal developers managing production. It’s aimed at promoting understand and collaboration between groups.  Thanks to Eric Minick from DevOps Zone and UrbanCode for supporting this.

#2: IT stability and business innovation are not enemies.

Q: DevOps promises a more responsive, more collaborative IT department that can realize business ideas faster. So what is holding back its widespread adoption? What’s the challenge or downside?

DE: There was a movie called “Charlie Wilson’s War” that had a great line between Tom Hanks, playing a U.S. Congressman, and Philip Seymour Hoffman, playing a CIA agent. Hoffman asks, “Why is Congress saying one thing and doing nothing?” Hanks replies, “Well, tradition mostly.”

All jokes aside, tradition is a powerful thing and hard to break. Tradition, or “what we’ve always done,” in IT is no different. There was a thread on Slashdot just this past month that asked whether developers should be allowed to deploy their own applications. You should have seen the outcry. The sheer number of commenters who shot down the idea as pure heresy was shocking. And the richest part of all of their denunciation was that the mob said over and over that the idea would “never work at a real company with real revenue at stake.” I thoroughly enjoyed sending that to John Allspaw, who runs all of technology at Etsy, and Jesse Robbins, who was in charge of risk and disaster planning for operations at Amazon. Etsy does over $600 million of transactions per year, and Amazon does about $50 billion in revenue. In both companies, developers are the ones who deploy and own the uptime for their own code. John’s reaction to the thread was a simple yet priceless one: “OMG.”

http://dev2ops.org/2013/02/

Commentary

Empowering development, operations and even QA teams to take ownership of code deployment is crucial for DevOps teams. It ensures that developers work with operations to understand the impact of change and take accountability/ownership for the “uptime of thier own code”. It also ensures that operations understand what changes are coming and how best to support the change. Uptime is a shared responsibility  with the environment the code executes within and the code that is executing both factors that must be managed correctly. 

#3 – StartOps: Growing an ops team from 1 founder

Bootstrapped startups don’t have the luxury of a full team of ops engineers available to respond to issues 24/7, so how can you survive on your own? This talk will tell the story of how to run your infrastructure as a single founder through to growing that into a team of on call engineers. It will include some interesting war stories as well as tips and suggestions for how to run ops at a startup.

http://devopsdays.org/events/2013-london/

Commentary

We are really looking forward to this track at DevOps Days London. DevOpsGuys are attending. Would be great to meet some of you.

#4: Engineering running the production environment

http://devopsreactions.tumblr.com/post/43882100705/engineering-running-the-production-environment

Commentary

Both funny and awesome! Nuff said!

#5:  Top 10 Practices for Effective DevOps

Practice 1: Active Stakeholder Participation

A fundamental philosophy of DevOps is that developers, operations staff, and support people must work closely together on a regular basis. An implication is that they must see one other as important stakeholders and actively seek to work together. A common practice within the agile community is “onsite customer,” adopted from Extreme Programming (XP), which motivates agile developers to work closely with the business. Disciplined agilists take this one step further with the practice of active stakeholder participation, which says that developers should work closely with all of their stakeholders, including operations and support staff–not just business stakeholders. This is a two-way street: Operations and support staff must also be willing to work closely with developers.

Practice 6: Integrated Deployment Planning

From the point of view of development teams, deployment planning has always required interaction with an organization’s operations staff; in some cases, via liaison specialists within operations typically called release engineers. Experienced development teams will do such planning continuously throughout construction with active stakeholder participation from development, operations, and support groups. When you adopt a DevOps strategy, you quickly realize the need to take a cross-team approach to deployment planning due to the need for operations staff to work with all of your development teams. This isn’t news to operations staff, but it can be a surprise to development teams accustomed to working in their own siloed environments. If your team is not doing this already, you will need to start vying for release slots in the overall organizational deployment schedule. Furthermore, to support continuous deployment, release engineers will need to increase the number of release slots available to agile teams that are disciplined enough to continuously and consistently meet the quality requirements for release.

Practice 7: Continuous Deployment

Continuous deployment extends the practice of continuous integration. With continuous deployment, when your integration is successful in one sandbox, your changes are automatically promoted to the next sandbox, and integration is automatically started there. This automatic promotion continues until the point where any changes must be verified by a person, typically at the transition point between development and operations.

Continuous deployment enables development teams to reduce the time between a new feature being identified and being deployed into production. It enables the business to be more responsive. However, continuous deployment increases operational risk by increasing the potential for defects to be introduced into production when development teams aren’t sufficiently disciplined. Successful continuous deployment in an enterprise environment requires all the practices described earlier.

Commentary

Overall we found this article rather in conflict with itself. It’s interesting how the author views Operations and Development as separate stakeholders within a process. The principles of DevOps are to break this view and to combine Development and Operations as a single entity and in this context they should be considered a single StakeHolder.

We fully agree that Operations should be managers of production with development participation, as Developers should be mangers of development with Operations participation, but on deployment we differ with the author. It’s easy in our view. If operations are a bottleneck to deployment, expand the role of “release engineers” to encapsulate the entire team. Anyone should be able to take accountability for and action deployment if the correct principles and practices have been applied.

The author also battles with Continuous Delivery principles. “release engineers will need to increase the number of release slots available to agile teams”. If you adopt a continuous delivery process your actually going to reduce the number of release slots, to one. One continuous release slot. Hence Continuous Delivery.

http://www.drdobbs.com/architecture-and-design/top-10-practices-for-effective-devops/240149363

#5: Tweet of the Week

Every #ITmanager, no matter how strong he is, lies to himself about his #DevOps initiative. I will find your lie. I will break you