Serenity – how ironic.

Today, Serenity opened across the country. This movie is the sequel to the commercially unpopular yet wildly successful series Firefly. I got hooked on Firefly a few months back and was absolutely elated to find out that a feature-length movie would be made. And shown in theaters, no less!

I was particularly feeling like celebrating because on Wednesday and Thursday, I had rolled out a massive code change to a vital system at work, and by the close of business, no critical errors had been reported. I had a list of five bugs which were annoyances more than anything else. I was elated.

So today after work I ran home, did a quick workout with Maya, grabbed my knitting and sped to the theater. I had bought tickets online for two of my co-workers whom I strong-armed into coming. Three or four other geeks from my workplace were going to the same showing. Yes! The geeks would show a united front of support for crazy neo-Western science fiction!

Ten minutes before the show started (showtime 7:40), my phone rang. It was work.

The massive code change, which I had worked on for five months and which was the single most thoroughly tested body of work I have ever done, was causing errors.

For a very high-profile customer. As in, websites were vanishing from the Internet.

I jumped in my car, screamed epithets at an irritated teenager in a huge pickup when he blocked my exit, and hauled ass back to work. In about twenty minutes I’d isolated the problem and come up with a fix.

But before you can roll out code, you test it. I believe this with the same degree of conviction that St. Augustine believed that Jesus was a great guy. I will not put code live without running it through an automated test first.

And my laptop, my usual development environment, was at my apartment. I could not run any of my automated tests without it.

So I hopped in my car and hauled even more ass back home, fired up the laptop, tested the fixes, and deployed them to our live system.

They didn’t work. When I looked at the code on the live systems, my changes weren’t there.

A few seconds later I realized that I had made a mistake with our version-control software. Putting it in English, I had corrected a mistake by editing a discarded manuscript. That doesn’t really fix anything in the real world. So I found the correct files, edited them, ran the tests, and pushed them live.

Hallelujah. The problem’s not completely fixed, but it has a really big band-aid on it that will hold for a few days. And I know what needs to be done to fix it for good.

I walked across the street to the local pub and bought myself a beer to celebrate being done with the fixes. On the television, they were playing ads for Serenity.

It was 9:50. The showing I had a ticket for had ended about ten minutes ago.


No Responses to “Serenity – how ironic.”

  1. nod Says:

    Ah, on my project, we roll on Wednesday! Enough time to fix before the weekend, and plenty of time if there is an emergency before the users get in on Monday morning.

    I saw Serenity last night, and loved it! I hope you get to see it this weekend.

  2. David Says:


    That’s the so very frustrating thing about this incident. I rolled the code in question out on Thursday. (I just updated the post to reflect that.)

    The bug just didn’t happen to get tripped until ten minutes before the show. I think I need to slaughter a fatted goat or something. Got a spare black rooster I can, uh, “borrow”?

  3. spoonix Says:

    And now you see why one of my rules is that you never, ever roll out new code on a Friday.

    I still haven’t seen it, either, so lemme know if you want to go catch it this afternoon or something.