To cut to the point, I was promoting some changes for a client last night. It was an update to part of our product that allows our customers to send automated and one off emails to lists of subscribers. I had done promotions similar to this in the past (in fact i’ve probably done the same thing maybe 100 times). But somehow, someway I managed to screw up. Not a small screw up either, I accidentally sent a test email message to the subscribers of one of the clients email lists.

That doesn’t sound so bad?
Sure it might not seem like a big deal, in fact I wasn’t even aware of it until the client pointed it out to me (there is nothing worse then this, NOTHING). You see our customers are publicly traded companies, and the software we develop helps them streamline and manage disclosure around their websites. I with an accidental mouse click had caused some level of damage in there shareholders eyes. I don’t even have to tell you the anxiety and frustration I felt when I was alerted to what I had done.

Ok, that’s bad
Yes it was bad, it was too late to undo anything so it was time to put into play this great saying one of my co-workers has:

First, get the cow out of the ditch, then figure out how the cow got in the ditch, and then make sure that cow never goes back in the ditch.

It has nothing and everything to do with software development all at the same time.

Obviously getting the cow out of the ditch was out of reach, but there were some measures that could be taken to ease the impact on our customer. I recommended that they send a one off email to the list that I had accidentally sent to, simply explaining that a test message was sent in error, they thought this was a great idea and did it.

Next, I had to examine all the various steps I had taken trying to reproduce what I had done. I had opened the wrong test email and sent it, an honest mistake but one that can’t ever be repeated again.

With that information in hand I put together some process recommendations for our company internally to implement when promoting these kinds of changes, as well as some product changes that could make this same scenario much more difficult to repeat in the future.

So in the end it worked out ok?
I guess if you look at it optimistically, yes everything turned out ok. For me, this was a situation that I was not comfortable with happening ever again. Fortunately the people I work with let me handle the problem start to finish, they didn’t breath down my neck and demand answers right then and there, they trusted that I would do the right thing, plus the client who was initially annoyed and frustrated appreciated the constant communication and recommendations that were made.

So what do you do when the shit hits the fan?
I’m sure this has happened to many a software developer. What do you do when this happens to you? I would be very interested in hearing about it, I think these hard learning lessons are ones that should be shared, never swept under the rug. If we can’t reflect and learn from these mistakes we as individuals and groups will continue to make them over and over again.