Going with your gut

Jan 11, 2010

This post is a response to Tim Ford’s Whose Blog Is It Anyway  challenge.  The opportunity to use the words: pony, nude and bong in a blog post about an actual experience was too much to pass up.

 

I’m a DBA and I’m logical – coldy logical, if you listen to K. Brian Kelley - but I’m here to tell you that sometimes you just have to go with your gut.  Most of us have an inner voice that clues us in on the things that you know but can’t rationalize or aren’t ready to deal with yet.  That’s the voice that lets you know that the URL in your son’s internet history with the word ‘PussyCat’, probably isn’t a site featuring live, nude cats.  Not everyone trusts that voice – just ask George “Let’s Have Padme Die Of A Broken Heart Instead Of Anakin Crushing Her To Death” Lucas – that would have been a far more awesome scene.

 

I’m here to talk about such a time, early in my career.   I had a great DBA to learn from, but he had moved on to another position.  I felt pretty firm in my knowledge and knew that, whatever came up I could fix or handle by simply using some magical tool, library or bong.

 

That’s when I ran into it – the problem that I couldn’t fix, but was going to cause me pain.  On a Friday evening I started seeing error messages in the SQL Server error logs that indicated that we were having disk issues.  Of course the error message didn’t read ‘Your disks are failing’, but everything that I was reading seemed to indicate that.  One thing that bears noting – this was the central order entry/documentation application for the hospital that I worked at – there was no acceptable downtime.  I contacted our system administrator who did some research and let me know that all of the disks looked fine.  Now, this was an SA whose knowledge I respected – he’d been in the field forever (not like ENIAC forever, but pretty close).   He alluded to the fact that I was still a newbie and probably didn’t diagnose the problem correctly.  At this point it was around noon on Saturday and even though I was getting tired of looking around, I figured that I’ve started, so I’ll finish.  I wasn’t able to find any information that was clearer than I had provided before, but I knew, in my gut, that we were about to have problems – big problems.  I called the SA again and tried to encourage him to look a little more closely.  I asked him to pretend that the RAID array was a horse stable.  From the outside, it might sound pretty happy.  On the inside it might look good initially, but as soon as he looks down he’s going to be very sad about the pony.  For some reason, that analogy didn’t work…

 

As the afternoon progressed, I kept with my gut feeling and bugged the heck out of that SA.  The disks actually gave up the ghost and 40 hours later (non-stop), with our application vendor and Microsoft on the line, we finally got the disks replaced and the application back online.  At the post-mortum, once everyone had gotten a few hours sleep, that same SA wanted to know how I knew what the problem was since none of the errors were absolutely clear about the cause.  I just told him that I knew in my gut that something bad was about to happen.  He said, “Well, if you’d told me that in the beginning, I would have done more research!”.

 

While the story above is true, some of the particulars have been changed to protect the innocent.

Posted by tledwards | Categories: Administration, DBAs, Uncategorized | Tagged: , , |

Share with others

No Responses so far | Have Your Say!

Leave a Feedback

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Spam Protection by WP-SpamFree Plugin