As an engineering student I was impressed by the intricate harmony of development processes. It was probably the same part of me that loved the uniformity and audacity that is xml.
These days I use JSON 100 times more often than xml. I’ve similarly fallen out of love with process. But there’s still a grudging respect there. Like an old lover that done you wrong, but is a sort of evil superhero so you’ve got to respect.
Now I’m a manager, and many developers dismiss my thoughts on process. But there’s good process and there’s bad process. The trick is telling them apart.
I’ve been trying to foster one good process: The 5 Whys.
The Holy Grail: Time and Effort Free Improvement
I’ve dealt with many production incidents — goofs where we went down and failed to serve our users. We have a process for reviewing those incidents. I feel like they often only address a shallow understanding and patching of the problem.
In my view, if the investigation of a production incident doesn’t eventually lead to management being wrong, then you’re just not digging deeply enough. That is, I should have to change.
I’ve tried to install a 5-whys process to make these reviews more effective. (Read about them anywhere on the web, or in Eric Ries’ The Lean Startup.) It’s hard to get support.
Everyone’s busy. Add to that a bias against “process” and it really is an uphill battle.
I figured that I need to make it palatable by keeping it lightweight and low burden to the team.
Incident Review SHOULD Slow You Down
Then I remembered that Eric Ries introduced the 5-Whys with the sentence:
“Adaptive processes force you to slow down and invest in preventing the kinds of problems that are currently wasting time.”
— The Lean Startup, pp. 229. Eric Ries
The 5 Whys is supposed to slow you down right when you need to. You fix the causes and speed up again. When others arise you slow down again.
Not a Holy Grail: A Contradiction
Trying to improve without taking any time away from “real work” isn’t possible. I can’t make brakes that work and don’t slow you down:
The most innovative and exciting feature of Cheforlet’s new concept car is that the brakes bring you to a stop without slowing down. Picture arriving at your destination just as if you sped straight there without a stop – even in heavy city traffic. The imagineers have done it again!
— Said Noone Ever
Improvement Costs Time – Requires Partnering
I can’t improve the team’s track record without costing time. That means that developers need to take time to understand why production went down. Then we need to understand why the work was done that way. Then we need to understand how management is going to enable and support a new way of doing work.
All of that takes data. Sometimes it takes dark and inaccessible data that we need to write custom tools and reports to get at. We must force ourselves to get the data we need and not settle for the data that’s easy (availability bias).
While it is a dev ops concern it can’t be done without buy-in from the business. We need to carefully manage their expectations around incidents. They should expect that incidents will cause a slow-down beyond the flurry to restore service.
Custom Grow Your Good Processes
Mr. Ries bills The 5 Whys as a way to grow just the right amount of process. The kind of process that addresses problems so credible that they’ve happened to you before.
Of course, if you just address the surface issues and promise to “be more careful” then you avoid building more process. And you also pass up helpful guard rails that make your job safer and more fulfilling.
