none
DPM 2012 SP1 and Hyper-V 2012 Large System Volume information Folder RRS feed

  • Question

  • Over the past week we have seen a problem where the System Volume Information folder has been growing at the rate of 20GB - 30Gb a days and there are now 670GB of files in the folder. Thjis is not related to the DPM Filter Bitmap files of which there are a lot but the volume is small compared to the remaining 630 files sized between 700MB and 11GB in size.

    our setup is as follows:

    2 x Node Active Active Server 2012 cluster with a single 5TB CSV volume mounted for Hyper-V images. We have 75 images running accross the two nodes and these are backed up using DPM 2012 SP1. Within DPM the VM's are split into four protection groups with differnet protection and retention setups.

    I have checked the Shadop copy for the volume and this is disabled, checking on DiskShadow CLI using List Shadows All it returns 'No Shadow Copies found in Syatem'

    DPM has been performing well with not a single backup failure over the last three weeks but all of a sudden on the 7th August (date of the oldes file in System Volume) we are retaining huge amounts of data.

    Checking the event log I can see the following errors:

    we get the following 7-8 times per day

    Volume Shadow Copy Service warning: Writer received a Freeze event more than two minutes ago. The writer is still waiting for either an Abort or a Thaw event.

    Operation:
       Gathering Writer Data

    Context:
       Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}

       Writer Name: Microsoft Hyper-V VSS Writer
       Writer Instance ID: {7e0fa2fb-7392-485c-bb90-966d0198bd10}

    We get these errors about 7 - 10 times per day:

    VDS Basic Provider

    Unexpected failure. Error code: 48F@01000003

    I am really stumped by this so any help greatly appreaciated.

    Thanks,

    John

    Thursday, August 15, 2013 2:19 PM

Answers

  • Hi,

    Are the offending file names in the form of {GUID}-{GUID} ?

    There is a known code defect in Windows 2012 where VSS snapshots marked for deletion are not deleted on CSV volumes.  This is going to be addressed in a upcoming Windows 2012 cluster update. 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.



    Thursday, August 15, 2013 3:35 PM
    Moderator

All replies

  • Hi,

    Are the offending file names in the form of {GUID}-{GUID} ?

    There is a known code defect in Windows 2012 where VSS snapshots marked for deletion are not deleted on CSV volumes.  This is going to be addressed in a upcoming Windows 2012 cluster update. 


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.



    Thursday, August 15, 2013 3:35 PM
    Moderator
  • Hi,

    that worked great, bit difficult assigning the correct level of security to enable the {guid}{guid} files to be deleted.

    Thanks for the quick response

    Cheers,

    John

    Thursday, August 15, 2013 4:40 PM
  • Hi,

    Are there any further details around this fix? Or a related KB describing the issue in detail? We are experiencing exactly this issue and is has become quite problematic; we have a CSV totally packed with old VSS snapshots that we are unable to get rid of.

    Cheers

    Thursday, November 7, 2013 3:39 AM
  • Hi,

    The windows group has not released a fix for this issue yet. Please open a windows support case so they can help you delete them.

    UPDATE 1-2-14 - a fix is now available - see announcement here: http://social.technet.microsoft.com/Forums/en-US/home?forum=dpmhypervbackup&announcementId=6a85772e-cc77-4c46-ba3b-bab89ca95db1


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Thursday, November 7, 2013 5:12 AM
    Moderator
  • This is not a fix, its a statement, so to fix this issue can we remove the older dpmfilterbitmap files?

    William Lee

    Tuesday, November 26, 2013 5:20 PM
  • Hi William See

    <snip>
    This is not a fix, its a statement, so to fix this issue can we remove the older dpmfilterbitmap files?
    >snip<

    The Windows VSS snapshots are not the same as DPM bitmap files.  Windows has not yet released a fix for snapshots that cannot be removed after deletion.

    However - DPM 2012 Sp1 has a fix for large DPM bitmap files if you see that as being a problem. See http://www.microsoft.com/en-us/download/details.aspx?id=40318


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, November 26, 2013 6:20 PM
    Moderator
  • Hi,

    is there a release Date or KB number available? The Microsoft Support could not help us on the phone.

    Best regards

    Stefan

    Wednesday, November 27, 2013 1:16 PM
  • Is there already a fix?

    If so: What is the KB number?

    Thursday, January 2, 2014 12:04 PM
  • HI,

    Yes, the fix was released by the Windows group that will address the snapshot cleanup issues.

    2878635 - Update is available that improves the resiliency of the cloud service provider in Windows Server 2012: December 2013
    http://support.microsoft.com/kb/2878635

    Announcement updated here http://social.technet.microsoft.com/Forums/en-US/home?forum=dpmhypervbackup&announcementId=6a85772e-cc77-4c46-ba3b-bab89ca95db1


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Thursday, January 2, 2014 7:58 PM
    Moderator
  • Hello Mike.

    Can I delete files with names in the form of {GUID}-{GUID} too ? In the way like http://blogs.msdn.com/b/george_bethanis/archive/2014/09/09/dpm-2012-sp1-ru2-huge-system-volume-information-folder.aspx

    Monday, September 29, 2014 9:57 AM
  • Hi,

    I am facing the similar issue with a Win server 2012 R2 Cluster. I am using Software VSS provider on all the nodes. I am protecting this cluster using DPM 2012 R2 with update rollup 5 installed.

    Any suggestions would be appreciated.


    Thursday, March 12, 2015 8:12 AM