If it hurts, do it often
“In software, when something is painful, the way to reduce the pain is to do it more frequently, not less.” - David Farley
Imagine for a moment that you're training for a marathon. The first day you run, your muscles ache, and you feel exhausted. If you stop because of that pain and never run again, you'll never be prepared for the marathon. But if you persist, gradually increasing your distance and frequency, your body adapts, becomes stronger, and the process gets smoother.
In the context of reliability, the "pain" could be the challenges faced when deploying a new large scale system, conducting disaster recovery exercises, or making changes to an existing infrastructure. If we shy away from these challenges because they are hard or can cause disruptions, we'll never be able to make meaningful progress.
The truth is that activities that require significant effort upfront tend to pay the greatest dividends for site reliability in the long run:
Continuous deployment can be a headache at first. But frequent deployments in small batches reduce risk and provide rapid feedback. What seems extraordinarily painful if done monthly becomes straightforward and low stress when done daily.
Comprehensive disaster recovery testing simulates worst-case scenarios that reveal weaknesses in your systems or processes. Though disruptive and toily short-term, frequent testing gives you a better sleep at night and makes teams and systems ready when the disaster does strike.
Refactoring big legacy codebases or infrastructure parts feels risky. But incrementally improving them in small pieces reduces that risk while accelerating gains. Little refactoring wins compound over time.
1️⃣ Increased Frequency Reduces Fear
When you deploy changes or test new systems more frequently, it becomes a routine. The fear and uncertainty that once surrounded the process reduces. Over time, with repetition, the teams becomes better at managing and mitigating risks.
2️⃣ Small, Incremental Changes Enhance Reliability
Instead of making massive do small and incremental changes. The benefit is feedback. If something goes wrong, it's much easier to identify and fix the issue quickly. Rollbacks are simpler. Lessons are learned rapidly. Instead of massive pull requests, submit many small code changes that can be reviewed efficiently.
3️⃣ Confidence Comes with Practice
Reliability practices like testing, refactoring, and disaster recovery build confidence through repetition. At first, these activities feel uncomfortable and risky. But conducting them frequently makes them more familiar and routine. Things that once seemed difficult or error-prone become comfortable and trusted.
The idea of "If it hurts, do it often" is not about inviting unnecessary pain. It's about embracing the challenges that lead to growth, improved processes, and ultimately, a more reliable and resilient system.