CommVault: I/O Frozen on Database
Over the last couple of years, I have met a number of you, who as I, are required to struggle daily to make CommVault work as the enterprise backup solution for your SQL Server databases. Given that, I thought I would share with you one of the issues that we have run into and possibly any third party product could run into that uses Microsoft’s Volume Shadow Copy Services (VSS). To be fair, I have to give the credit for the research and most of the write up of this issue and solution to one of the DBA’s that works for me (whom I am sure many of you know J), Samson Loo (twitter: @ayesamson).
Problem:
I/O operations are frozen on one or more databases due to CommVault issuing a “BACKUP DATABASE WITH SNAPSHOT” command and remain frozen until the operation completes successfully or is cancelled. (This is appears to be known behavior of VSS. If you wish to dig further into how VSS works, I would suggest reading these articles: http://msdn.microsoft.com/en-us/library/windows/desktop/aa384615(v=vs.85).aspx and http://msdn.microsoft.com/en-us/library/aa384589(v=VS.85).aspx).
This particular message gets logged in the SQL Server error log whenever any backup service makes use of the SQL Server Virtual Device interface to backup the database with snapshot. Microsoft Backup (ntbackup.exe), Volume Shadow Copy Services, Data Protection Manager, Symantec Backup Exec and other third party tools, in addition to CommVault can cause this message to be logged.
If ntbackup.exe is configured to take a backup of a drive that happens to house SQL Server data files, then the command “BACKUP DATABASE WITH SNAPSHOT” is issued to ensure the backup is consistent since the data files are in use. During this time, the I/O for the database that is currently being backed up is frozen until the backup operation is complete.
The message that you will typically see logged is:
Error:
I/O is frozen on database master. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.
Note: In the example error message above, the database master was referenced, but this could be any database on your instance that is being backed up.
Solution:
Disable VSS in the subclient following the steps below:
- Open the CommVault client (often called the “Simpana CommCell Console”)
- Navigate to the target subclient
-
Right click anywhere in the white area
- Select Properties
-
Uncheck “Use VSS” and click OK
Again, extreme thanks go out to Samson Loo (twitter: @ayesamson) for providing most of this content!
SeanB
May 3rd, 2012 at 1:45 pm #
What (if any) are the implications of unchecking “Use VSS”? Are there any other downsides to using CommVault agent to backup your SQL Server databases (other than having to install an agent, not being able to use your tried and true T-SQL scripts, and no ability to simply grab a .bak file and shipt it to a vendor)?
tledwards
May 3rd, 2012 at 3:12 pm #
The basic implication of unchecking the “Use VSS” box is that any changes to the database that occur while the backup is in process won’t be captured in the backup. If you are doing regular transaction log backups as you should, this really shouldn’t be an issue as they will get picked up in the next transaction log backup. As far as using CommVault for a SQL Server database backup solution, the downsides include those which you mentioned and the fact that you have to pay for an agent for both your production and development servers if you want to be able to refresh your development server with the latest production copy. Additionally, since this is a one size fits all solution (it is great for file system backups), it doesn’t give you some of the flexibility that you get with many of the SQL Server-centric backup solutions such as the ability to do object level restores, reading the sys.databases view to determine when a database has been dropped so that it is automatically skipped instead of continuously trying to backup the missing database for an extended period of time before issuing an error and moving on to the next database. I am not a big fan of “one size fits all” suites of products that are supposed to work on multiple products against multiple platforms. It seems that you overpay for what you get and end up having to do much more management of the tool than if you purchased the best of breed product for that specific purpose. Those are just my thoughts based on the experience that we have had over the last four years using CommVault for our enterprise backup solution.
where do i buy raspberry ketones
May 30th, 2013 at 7:59 pm #
I undoubtedly did not realizе that. Learnt somеthing nеw
аt the recent tіme! Thanks for that.
demo contests
June 2nd, 2013 at 2:12 am #
Neat blog! Is your theme custom made or did you download it
from somewhere? A theme like yours with a few simple tweeks
would really make my blog jump out. Please let me
know where you got your design. Thank you
Raphael
June 2nd, 2013 at 4:43 am #
In human terms, becoming “individuated” means moving from being unconsciously united with the
Creator or Ground of Being; to choosing to become divided
from the Creator and developing separate individual
consciousness; to finally returning to the Creator as conscious,
aware individuals. Waxing, body piercing, tattooing, permanent eye lineretc.
For interested readers we have some very handy and related information on our website about Taylor Refrigerator Thermometer and
Digital Refrigerator Freezer Thermometer.
Rolland
June 2nd, 2013 at 10:44 am #
It also cuts down the possibility possessing steady sickness, which can be should be attractive avoiding (or perhaps, slows down and also decreases the
link between) neuronal as well as , behavioural growing older based on issues
with regard to Alheimer’s. If you are juice extractors that you can purchase in every pricing. Considering the fact that we all want a good deal stand enough space rolling around in its residence, building a powerhouse counter top item of equipment doesn’t require
the actual whole worktop is really a .