As most of us know, making changes in a live environment is often something done with extreme care. In fact, best practices dictate that such changes should be made during a maintenance window wherever possible. Of course, there are often situations where changes are made outside of the aforementioned maintenance windows, though these are generally emergency situations in which services have failed or severe problems are being experienced.
Being human, though, there are also times when we make rash decisions, believing that our modifications are so insignificant that they can’t possibly cause problems. It’s times like this that the big red caution light should go off in our heads. Of course, we often ignore that light and move forward with our plan, certain that nothing will go wrong and that we’ll come out the other side as a hero. We’ll be the person who tweaked the service and made it run even smoother.
I was that person today. However, as I have a very good friend named Murphy, my insignificant change caused a wonderful chain reaction of events ending in a major service interruption. Go me.
In the end, we resolved the problem and life continues on. Thankfully I work with people who are willing to forgive the occasional mistake, provided it’s a learning experience and doesn’t happen again.
The point of all of this is to show that even the smallest changes, no matter how insignificant, can result in catastrophic failure. Every change should be evaluated, tested, and scheduled for deployment whenever possible. And when you make that inevitable mistake, learn from it. We’re all human and will, at one time or another, view ourselves as invincible. We’re not. We make mistakes, we screw things up.
So, learn from my mistake, go make your own, learn and move on. As they say, Live and Learn.