DevOps

There are many buzzwords in technology and especially corporate IT. Sometimes they mean different things to different people, sometimes they are thrown around to show that a business is keeping up with the trends, sometimes to attract talent and probably all these use cases are not wrong. However, the value of these buzzwords really depends on how they are implemented, how it fits the business and its way of working and collaborating, to make the most of why an organization should even bother considering them.

DevOps is a concept that has been going for some years now and although it's not as fresh as AI is currently, it's now deeply rooted in many organizations and has brought many benefits to IT teams around the world. Although I have been practicing DevOps for 10+ years now, and doing deployment pipelines while listening to Thoughtworks guides and reading Martin Fowler's complex blog articles, I never thought about it more than the technicality behind it and the process and benefits it brings to an engineer, until I listened to 2 audio books: The DevOps Handbook and The Phoenix Project. It's those books that enlightened me on the true value that DevOps has at an organizational level, the strength of the visualization of work moving from left to right and how change management and technology leadership decisions can be made with much more insights and understanding of what the whole IT organization is doing. Those books, and especially The Phoenix Project, presented also the chaos that comes with managing IT projects in lack of strong processes, the impact it has on the organization and team relationships.

This article is not intended to be a summary of those books but I recommend the read. I am also not intending to provide a recipe of how to do DevOps the right way, because the recipe is different for each team and organization and evolves over time. In this article, I merely want to share some of my learnings and principles while implementing DevOps in various places.

In one of my roles, it was thought that DevOps only works for software development teams, and cannot be implemented for infrastructure or analytics or cyber teams. The principle behind this was that if there isn't any code written, or code that is very niche, then there is no opportunity to create a deployment pipeline. But that perspective was only focused on the concrete artefact of pipelines, whereas DevOps is much more than this. In the end, I managed to convince my reporting managers that DevOps works for any team, regardless of the IT function it carries and that not all practices need to be adopted by all teams, but some principles cross team or technology boundaries. And I managed to convince my teams about this, by having them read The Phoenix Project.

The main change we introduced was that we need boards to see the flow of work, document user stories or requirements on those board rather than Word documents flying around over emails, place work items into buckets, and, in a Kanban style, limit the WIP for the teams so that things start moving, rather than 'clogging up' the pipeline with too many in progress items, which results in a lot of context switching, reduced velocity, increased likelihood of production incidents and generally an unhappy IT team (because of too much context switching) and unhappy business (because of a perception that IT is too slow). Whenever it came to production releases, guess what, we had a board for those as well, with cross functional teams, and twice a week meetings with the leads for each area to review pending requests before releasing to production and impacts it can have across the whole set of services or what we were missing. While DevOps says that one can do multiple deployments per day and that this is a non-event, in some environments, like the ones I worked in, a deployment to production was still an event and we put more governance around it. We made 1,300 deployments to production in one year, and that was a metric we never could show to the board in the past, so we were still turning things quickly, despite the increased governance. Of course we had deployment pipelines, and multiple environments and we could roll back those deployments, but that was just technical and it came naturally.

We also ended up improving our game as part of this DevOps implementation, as Infrastructure teams moved to Infrastructure as Code, as then they could define and deploy infrastructure in code too, just like software teams.

We upped the game for the Analytics team, to stop deploying changes manually and looked at ETL pipelines that could do orchestration and could reside in source control.

We introduced a Change Advisory Board facing the business, to show that IT is making progress and DevOps boards allowed us to put a visual thing in front of General Managers for various business units, have frank conversations backed up by metrics on lead time to delivery, size of backlog, priority in backlog, effort/cost to release something, and IT became a more popular department because it could articulate the work it was doing much better.

The teams were happier because there was less change in priorities and context switching and everyone could see what was going on, what was deployed to production and when, down to the service delivery teams that would get the first call when something went wrong.

One thing that I learned to pay attention to is that although DevOps encourages autonomy and ownership, teams can sometimes defend behind the process: e.g. the user story is not groomed enough, the responsibility is with someone else before it gets to my lane. The recommendation here is to not let the process be the excuse for things not moving forward or a shield for contributing to forward movement. While the user story needs to be clarified of course, ping ponging emails and user stories from one lane to another defeats the collaboration and ownership that we aim to get. Make that statement and expectation clear within your teams, deal with outliers, but expect that everyone works towards getting things done.