I woke up in a good mood this morning and then…
Sometimes one piece of technology makes another, seemingly unrelated, piece of technology misbehave. It happened this morning at 12:15am CDT when an automated program that I run in Amazon EC2 failed to do its thing. The EC2 instances (that’s geek speak for “virtual machines” which is geek speak for “computers which aren’t really there but act like they are”) started up but never got around to doing any useful work. Six hours later, all of the instances were still running; they should have finished their work and died off in about two hours. I killed all of the instances, grumbling because I had paid for six hours of time and gotten nothing for it and did not even know why.
The underlying problem, it turned out, was a new SSL certificate that we had installed on our e-commerce store yesterday. One of the first things that each EC2 instance does is to fetch the latest version of the software from a Subversion server, which, coincidentally, is on the same machine as our e-commerce store. With a new SSL certificate on the server, each instance was waiting for a human being to say that the new certificate was OK. Inconveniently, the human being was sound asleep.
Who would have thought that renewing the SSL certificate for our on-line store would break an unrelated Amazon EC2-based application? Hidden dependencies suck.
Now I am in a bad mood, grumbling because I did not get my relaxed waking-up time after my shower, sitting next to my wife, drinking coffee, cruising blogs. Instead I dove directly from the shower into debugging and it left me feeling edgy.
We have a mechanism at Hen’s Teeth Network which works pretty well to keep emotional baggage like this from blindsiding our coworkers: we check-in every morning. It is a chance for me to say, “I’m in a bad mood. Better watch out; I may bite.” Better forewarned than not.
I am finishing this post a couple of hours later, after checking in with my coworkers. The check-in worked beautifully, giving me a chance to blow off some of the steam. I am more relaxed and I got some support from sympathetic ears. We even laughed a bit about the situation.
We missed the hidden dependency between the e-commerce store’s SSL certificate and the EC2 application and were caught unawares. Fortunately, we did not miss the hidden dependency between my early morning upset and my interactions with my co-workers. Knowing about the dependency and having tools at hand and in daily use for handling the dependency, proved a good thing for all of us.
