Going with your gut

11 January 2010

This post is a response to Tim Ford’s Whose Blog Is It Anyway  challenge.  The opportunity to use the words: pony, nude and bong in a blog post about an actual experience was too much to pass up.

 

I’m a DBA and I’m logical – coldy logical, if you listen to K. Brian Kelley - but I’m here to tell you that sometimes you just have to go with your gut.  Most of us have an inner voice that clues us in on the things that you know but can’t rationalize or aren’t ready to deal with yet.  That’s the voice that lets you know that the URL in your son’s internet history with the word ‘PussyCat’, probably isn’t a site featuring live, nude cats.  Not everyone trusts that voice – just ask George “Let’s Have Padme Die Of A Broken Heart Instead Of Anakin Crushing Her To Death” Lucas – that would have been a far more awesome scene.

 

I’m here to talk about such a time, early in my career.   I had a great DBA to learn from, but he had moved on to another position.  I felt pretty firm in my knowledge and knew that, whatever came up I could fix or handle by simply using some magical tool, library or bong.

 

That’s when I ran into it – the problem that I couldn’t fix, but was going to cause me pain.  On a Friday evening I started seeing error messages in the SQL Server error logs that indicated that we were having disk issues.  Of course the error message didn’t read ‘Your disks are failing’, but everything that I was reading seemed to indicate that.  One thing that bears noting – this was the central order entry/documentation application for the hospital that I worked at – there was no acceptable downtime.  I contacted our system administrator who did some research and let me know that all of the disks looked fine.  Now, this was an SA whose knowledge I respected – he’d been in the field forever (not like ENIAC forever, but pretty close).   He alluded to the fact that I was still a newbie and probably didn’t diagnose the problem correctly.  At this point it was around noon on Saturday and even though I was getting tired of looking around, I figured that I’ve started, so I’ll finish.  I wasn’t able to find any information that was clearer than I had provided before, but I knew, in my gut, that we were about to have problems – big problems.  I called the SA again and tried to encourage him to look a little more closely.  I asked him to pretend that the RAID array was a horse stable.  From the outside, it might sound pretty happy.  On the inside it might look good initially, but as soon as he looks down he’s going to be very sad about the pony.  For some reason, that analogy didn’t work…

 

As the afternoon progressed, I kept with my gut feeling and bugged the heck out of that SA.  The disks actually gave up the ghost and 40 hours later (non-stop), with our application vendor and Microsoft on the line, we finally got the disks replaced and the application back online.  At the post-mortum, once everyone had gotten a few hours sleep, that same SA wanted to know how I knew what the problem was since none of the errors were absolutely clear about the cause.  I just told him that I knew in my gut that something bad was about to happen.  He said, “Well, if you’d told me that in the beginning, I would have done more research!”.

 

While the story above is true, some of the particulars have been changed to protect the innocent.

 | Posted by tledwards | Categories: Administration, DBAs, Uncategorized | Tagged: , , |

I was tagged by Jorge Segarra (BlogTwitter) who had been tagged by Thomas LaRock (BlogTwitter) in his post about his goals and themeword for 2010.  I was going to try to remain blissfully ignorant about being tagged, but then Tim went and posted his goals.  So I guess I’m on the line now.  My theme word for this year is:

 

Recharge

While there are many things that I want to accomplish this year, I don’t know that (m)any of them will occur until I can figure out a way to recharge.  I’m typically a self motivated type of person, but it seems like, during the previous year, I’ve hit the wall.

 

I’m not entirely sure what has caused this, but I’m guessing that it is some combination of the cyclical nature of job satisfaction, having a boatload of things going on at home and the disconnect between the amount of things that I would like to learn and the amount of free time that I have.

 

recharger

Is there a human connector on that thing?

I realize that there is no magic button that will instantly recreate the hunger for knowledge that I had when I began learning to be a DBA.  What I can do, though, is set some goals, work hard to follow through on them and be patient.   My hope is that in the process of achieving these goals, I’ll rejuvenate my love of this career path.

 

Goals

Pick one or two topics to focus on

I have at least three SQL Server books sitting on my desk and more at home that I haven’t done much more than flip through.  Rather than setting a goal to read all 3000 pages (doable, but daunting), I’m going to pick a couple of subjects to focus on and learn them as thoroughly as possible.  This is ongoing – if it’s March and I know everything there is to know about database corruption (or whatever it is I end up focusing on), I’ll move on to the next subjects.

 

Blog more

My first love is teaching.  It invigorates me and gives me purpose.  Blogging provides me an arena to hopefully teach people that are learning to be DBAs and the chance to share what I’ve learned.

 

Become more involved with PASS

As I’ve mentioned in previous posts and as Tim mentioned in his goals for 2010, we’ve talked often about starting a PASS chapter in Tucson.  This ties into my love of teaching and will help us to connect with folks locally who have similar interests.  I would also like to take part in other committees within PASS as needed.    This will definitely require a balancing act with work and family, so I’ll be taking baby steps to ensure that I don’t shortchange other areas in my life.

 

What does this all mean?

None of these individual goals are earth-shattering and that’s intentional.  I have a tendency to swing for the bleachers, but end up hitting to the pitcher and it makes me grumpy.  My hope here is that I make some good, solid line drives and then I’ll be set up to hit it out of the park.

 

I’m tagging a couple of people that have unknowingly helped me to recharge (some thank you, eh?) 

TJay Belt (BlogTwitter)

Wendy Pastrick (BlogTwitter)

Kendal Van Dyke (BlogTwitter)

I went to install SQL Server 2008 on a Windows Server 2008 R2 box today for the first time and was greeted with the error “You must use the Role Management Tool to install or configure Microsoft .NET Framework 3.5.”  The text of this error was basically about as decipherable as the voice of the adults in the old Peanuts cartoons, so I fired up my old friend Google to find out what to really do.  It seems that Windows Server 2008 R2 ships with .NET Framework 3.5.1 and in order to proceed, you need to go to Server Manager (that window that pops up when you login and stays unless you close it) and enable (really, install) the .NET Framework 3.5.1 and any prerequisites.

ss2008_error_1

 

ss2008_error_2

PASS Summit Kool Aid

9 November 2009

Last week, I had the opportunity to attend the PASS Summit in Seattle.   As I had mentioned in a previous post, I was fairly anxious about attending, because I knew that there were going to be around 2,000 people there and I had met two in person.  Yep, 2 out of 2000.  Let’s just say that I wasn’t worrying about how to fit in time for catching up with those folks.

 

I knew that the PASS Summit would be a great learning opportunity.  I’ve attended Tech-Ed, SQL Server launches and other similar SQL Server events – the learning that occurred at those events was extremely valuable.  In looking at the sessions for the Summit, I knew that it was possible (probable) that my head would explode with newly gained knowledge.  There are plenty of folks that will be blogging about the sessions and all of the excellent speakers – I may be doing that in a future post, but that’s not my focus here.

 

koolaidman
Ohhhh yeeaaaahhhh!

My focus is on the PASS community.  While I already knew that there were helpful, friendly people that were already a part of PASS, I never thought that it would would pervade the entire conference.  I had the opportunity to meet an incredible number of people – those whom I was familiar with through Twitter, blogs or forums and those whose faces and names were new to me.  In every instance, they were accessible and welcoming.  In turn, these experiences encouraged me to go seek out and introduce myself to others.  This was truly a community in the best sense of the word.

 

Tim and I have been talking about getting more involved and have discussed starting a PASS chapter here in Tucson.  The experiences of last week have made me see that this is not only doable, but necessary.  I’ve supped of the PASS kool aid and it was not only yummy, it’s my new favorite drink.

 

I’m looking forward to keeping in touch with the people that I was fortunate enough to meet and becoming more involved in PASS, both locally and virtually.   My hope is to share this community with others and help it to grow.

 

On a more personal note, there were a few individuals that went above and beyond the call of  (professional) duty last week.  I hope that I’ve let you all know personally how much your thoughts and prayers meant.  Tim and I were pleasantly surprised and touched by your willingness to listen and help.  We truly thank you from the bottom of our hearts.

 

On a completely unprofessional note, I was overjoyed to be a part of the karaoke jollification (yeah, it’s a word) on Thursday night.  I was impressed with the singing (and dancing) talents of this crew.  I’m just hoping that there are no incriminating pictures…

I had been thinking of writing a blog post on the SQL Server community for the last couple of weeks.  Seeing Brent Ozar’s blog post “What Community Means to Me” helped me decide to go forward with it.

In my first draft of this post, I went into great detail about the beginning of my career, my quest for meaningful, reliable sources of information and my wish for a view of a larger community.  Unfortunately, I’m trying to get ready for a birthday party, Halloween, soccer games and, oh yes, the PASS Summit.  So that’s another story for another time.

 

When I first signed up to attend the PASS Summit, my hope was that my darling husband would be able to attend with me.  Regardless of what Tim might say, I’m not outgoing enough to walk up and talk with people I’ve never met.  Yet I know that those conversations will probably be the parts that I remember most and best from the Summit.  Unfortunately, it wasn’t in the cards for Tim and I’ll be attending solo.   I pictured three days of wandering around, trying to make conversations and going back to the hotel room to eat room service.

 

Enter the happy-happy-joy-joy land that is Twitter.  Tim and I both started using Twitter in April of this year.  It was interesting getting started – kind of like walking into a conference – you all have the same interests, but you don’t know anyone.  Slowly but surely we got involved.  Had some lively IM conversations at the spring SSWUG vConference in the Quest chat room, tried to write a rap song, got involved with PASS Virtual Chapters, started a blog, shared meals with a couple of great DBAs and got the kind of SQL Server advice and help that you can’t pay for.

 

Twitter is obviously not the only method for getting involved with the SQL Server community, but I’ve found it extremely helpful for becoming familiar with other people that do what we do.  By reading tweets and blog posts throughout the day, I’ve picked up tips and tricks as well as become exposed to features and functionality that I might not have been aware of.

 

Now, in addition to attending some excellent sessions, I’m also looking

It's not quite this, but close...

It's not quite this, but close...

forward to meeting a number of people that I’ve ‘met’ through Twitter.   I’ve felt more a part of the SQL Server community in the last six months than the previous 5 1/2 years of working as a DBA.  It’s a great community and I talk about the benefits of being involved any time I can.   I still wish Tim could have come along, but I also know that I won’t be feeling as alone.    Maybe when I’m there, I’ll meet someone who hasn’t yet had the chance to get involved in the community and be able to pass this along to them.

 | Posted by tledwards | Categories: DBAs, Miscellaneous, Personal | Tagged: , , , |

This weekend, one of my co-workers passed away.  He was 33 years old with a wife and toddler at home and a baby on the way.  I didn’t know him well, but we had worked together on a few projects and he was always very knowledgeable, thorough and helpful.  His passing was very unexpected and, as these things do, it caused me to think about my own life and my priorities.

 

Like many DBAs, the servers that Tim and I manage need to be available 24/7.  Tim is on call every third week, but being a lead, he often needs to step in on weeks that he isn’t on call.  I’m the sole DBA at my company.  We’re both proud of being dedicated professionals and we work hard to keep our current systems available as well as keeping our skills updated so that we can provide the best solutions possible.  I firmly believe that we’re setting a good example for our children in showing them the responsibility that we take in our positions.

 

The problem that we both face is in knowing when to step out of our work personas and focus on our family.  Tim is an excellent father and I work hard to be a good mom, but I know that there are many times that I’m talking about work, thinking about work, worrying about work when I should be more engaged as a wife and mother.  While I know that we both provide benefit to our businesses, I also recognize that, if we left, we would be replaced and work would continue as usual.  The time and effort that we put into our time together and our time as parents will shape all of us for the rest of our lives.

 

There isn’t an easy solution to this problem.  It’s not always that simple to walk out the door (especially for Tim, who works at home) and turn off the DBA part.  There will be times that I need to focus on an issue even after leaving work in order to sort it out, but I’m going to make the effort to do that only when it’s necessary.   I need to keep in mind that my first job is to take care of my family.  If I do that, the rest of life will work itself out.

A co-worker of mine had the following saying up on their wall:

 

                              Remember, the people that you work for are waiting for you at home.

 

I need to keep that in mind.

 | Posted by tledwards | Categories: DBAs, Miscellaneous, Personal | Tagged: , |

I have been managing DBAs for over ten years now and one question that always seems to come up, usually from management, is what do DBAs do anyway?  Hopefully, you have read my prior post, “Yes, Production DBAs Are Necessary!” and you know where I stand on this issue. 

 

I fully believe in the role of a production DBA and, as someone who has been in this role for well over a decade, would like to define, based on my experience, what I think that role should be.  The most effective way of doing this, I believe, is to define what a production DBA should and shouldn’t be expected to do.  Granted, this won’t work perfectly for every situation and this is not intended to be an exhaustive list, but the following should be a good guideline for defining what a production DBA does and why they are critical to ensuring that your data is safe and available.

 

User data

A production DBA should not have anything to do with data. There, I said it. This statement alone is going to generate tons of controversy, but let me explain and specify that I’m referring to user data in user databases. This confuses many people because, after all, DBA does stand for database administrator. Let me clear up that confusion right now by saying that database is one word and is a manageable entity just like a server. Data lives inside a database, much like an application lives on a server. Typically, there is a lot of data in a database, just like there can be many applications on a server, each of which may have their own application administrator. As in the case of the application analogy, we don’t expect the system/server administrator to necessarily manage all of the applications on the server, the production DBA should not be expected to manage the data in the database. The production DBA is responsible for the uptime of the database, the performance of that SQL Server instance and the safety (backups/DR) of the data in the database, not for the data itself. There should be other roles in the enterprise responsible for determining how the data gets into the database and how it is used from there, namely, data architects, database developers, etc.

 

Optimizing T-SQL

A production DBA should work with database developers to help them optimize T-SQL code. I emphasize work because production DBAs should not be writing code, outside of administration automation. This goes hand in hand with #1 above and, when you think about it in practical terms, makes sense. A production DBA may be responsible for managing hundreds or thousands of user databases all containing data for different uses. A production DBA can’t practically understand all of this data and be able to write code against it effectively. What the production DBA can do, though, is help the database developers optimize code by looking at available indexes and discussing with them how to best to arrange joins and where clauses to make the most effective use of indexes and/or create additional indexes. Additionally, if your organization falls under the Sarbanes-Oxley or PCI regulations, this segregation may be mandatory. Your developers should not have access to production databases. Their code should be promoted by the production DBA, who does have access to the production databases. This also means that you should have a development environment that reasonably approximates your production environment.

 

Managing Instances

The production DBA should manage the SQL Server instance(s). This is a big job and if your data is important and you have more than a handful of instances, it is a full time job. Let me breakdown what this actually includes to illustrate just how large a job it is:

  1. Install/patch SQL Server instances – The production DBA is responsible for installing any new instances of SQL Server and making sure that existing instances stay current on Microsoft patches. This isn’t just a matter of running through an install wizard. A number of things have to be considered by the production DBA when a new instance of SQL Server is installed. These include:
    • Collation settings. Questions have to be asked about how the application that uses this data expects the database to handle the case of words, accents on words, code pages that the application expects to be used (this gets into localization and language for those shops in other countries or that are responsible for servers that reside or are going to be accessed by users in other countries).
    • Drive/File Layout – To make the database instance run optimally, the production DBA has to consider what drive resources are going to be available and how the database files should be laid out across those resources. During this phase of the installation the production DBA has to consider whether only local storage is available or whether storage will be housed on a SAN. If it is going to be housed on a SAN, the production DBA needs to work with the SAN administrator to ensure that the LUNs are set up appropriately for optimal SQL Server access, which in and of itself requires a lot of knowledge and experience.
    • Scalability – The production DBA should be involved in developing the specifications for the hardware. Questions that the production DBA will be asking of the various parties include, how many concurrent users will be accessing the data, how many databases will there be, is the data static or changing and how does it change (i.e., batch load, transactional updates), etc. This will give the production DBA a better idea of what kind of resource utilization to expect and drive the hardware specification process. It will also help determine recovery model and backup strategies.
  2. Create, and periodically test, a backup and recovery scheme for your enterprise’s data. Things the DBA has to consider as part of this include:
    • Is the data development or production data? Yes, development data gets backed up and is considered in an effective backup/recovery scheme because, after all, you don’t want to lose any of your development effort; it just isn’t prioritized as highly as production data.
    • How often data is updated in databases and how is it updated (online transactions, batch, etc)? This determines the recovery model that should be used and leads the DBA to ask another question, what is the maximum acceptable data loss in terms of time? With the answers to these questions, the DBA can more effectively determine the type and the frequency of backups (full, differential, transaction log, etc).
  3. While somewhat tied with backup/recovery, the DBA is also tasked with helping to come up with a disaster recovery strategy for the SQL Server environment, within the constraints of the enterprise’s available resources.
  4. Uptime/performance – The DBA is tasked with managing/monitoring those things that would impact the availability of the databases. This includes CPU utilization, I/O performance, disk space, reviewing event and SQL Server logs,etc.
  5. Help design the security model for SQL Server instances. Within SQL Server, there are a number of built in security roles, both at the instance level and database level that users can made members of in addition to the ability to create custom roles. Additionally, SQL Server can operate in Windows Authentication mode, which integrates your domain security and uses the users’ Windows domain accounts or Mixed (or SQL Server and Windows Authentication Mode) Mode, which allows accounts to either be Windows domain or native SQL Server accounts. There are a number of factors to be considered here to make sure that your users can use your system while still protecting and safeguarding sensitive data and the SQL Server configuration.

 
So as you can see, the production DBA has plenty of things to contend with in order to effectively and efficiently maintain a SQL Server environment.  The bottom line comes down to how valuable is your data to you and how much are you willing to risk by not allowing your DBA to dedicate themselves to maintaining and safeguarding it.

I know that, for many, this is a controversial topic.  There are those that believe that there really is no such thing as a SQL Server production DBA and that DBAs should be jacks of all trades doing anything from database development to OLAP/BI development to .NET programming to being a webmaster or network/server/system administrator.  It seems that everywhere I turn anymore, job postings, sharing horror stories with colleagues, and even blogs from members of the SQL Server community, I see references to production DBAs being more than just a DBA.  It as if, somewhere, managers are thinking that they need to have someone manage their SQL Server databases, but they don’t know what that means, so the first thought is “let’s just make it part of someone’s duties that we already have on staff.”  The question is constantly asked, “Why can’t the database developer handle that, he/she already has to use SQL Server?” or the common mistake, “let’s just let the server administrator handle it.”  There has even been a term coined for this, “the reluctant DBA.”  I have actually heard SQL Server compared to Access since it has wizards and, because Access doesn’t require a production DBA, why the heck should SQL Server?  In my experience, this perception is especially prevalent in the small and medium-size (SMB) market, but there are large companies and managers that have grown up in medium-size companies that seem to reinforce this misconception.

 

Microsoft hasn’t really done anything to correct this misconception.  From a marketing perspective, I guess it is in their best interest for prospective SQL Server customers to think that it doesn’t take much to manage a production SQL Server environment.  When companies purchase Oracle or move their applications to Oracle databases, it is a foregone conclusion that dedicated production DBAs are necessary and so, these companies typically build that into their cost calculations.  I guess Microsoft feels that if customers don’t know a DBA is required, it makes their product look that much less expensive.

 

Now don’t get me wrong, I am not saying that database developers, server administrators, network administrators, etc. can’t become DBAs, they just can’t do it without training and experience, just like they didn’t fall into their current jobs without training and experience and the job of production DBA certainly can’t be done (at least not effectively) in someone’s “spare time.”  As SQL Server has matured, it has become extremely flexible and full featured, but these features come with a cost: complexity.  SQL Server can be used and configured in a myriad of different ways and for many different scenarios, but who is going to manage that and recommend how it should be configured and used for a particular purpose if you don’t have a production DBA?

 

In my next post, I will discuss what a production DBA does and what they shouldn’t do.