none
DPM backups using hardware VSS provider creating leftover snapshots as iSCSI targets RRS feed

  • Question

  • Hello all,

    This question was originally posted in the Windows Server File Services and Storage forum here; the personnel monitoring that forum have asked me to pose the question/situation here, so here we go...

    I have a Windows Server 2012 Failover cluster running a number of virtual machines. Storage for the cluster is provided by a Windows Storage Server 2012 instance and mapped to the nodes via iSCSI. In addition, I use DPM 2012 SP1 to protect the clustered VMs. The snapshot provider configuration was set up in accordance with the instructions and guidance at http://blogs.technet.com/b/filecab/archive/2012/10/08/iscsi-target-storage-vds-vss-provider.aspx . I had the same setup with Server 2008 R2, but recently upgraded to take advantage of the hardware snapshot provider and performance improvements claimed by Microsoft.

    Since configuring the protection group for the clustered virtual machines, the group shows that all of the members are in an ideal state and there are no errors in the cluster event log. However, there are some issues in the event logs for the cluster nodes around the time the backups are being taken. This isn't the reason for this post, however, though it may be related.

    While having a look at the iSCSI configuration on the storage server yesterday, I noticed the following...

    Get-iSCSIVirtualDisk
    
    ClusterGroupName   :
    ComputerName       : myserver
    Description        : Cluster Shared Volume
    DiskType           : Fixed
    HostVolumeId       : {67F71B49-BD09-457E-8E33-34607970BE1C}
    LocalMountDeviceId :
    OriginalPath       :
    ParentPath         :
    Path               : e:\csv.vhd
    SerialNumber       : C4A45A4D-E7E0-4AEE-B800-403DD32E8A61
    Size               : 8796093022208
    SnapshotIds        : {{19C35EF1-C02D-49E4-9C38-B3AE500E2524}, {82FA960F-529E-4A41-A0B7-4F4981846051},
                         {E59C4532-5280-4465-B2EB-9FD31175F665}, {ED2F4661-2882-4C11-9B27-C946D6695219}...}
    Status             : Connected
    VirtualDiskIndex   : 1777681073
    
    ClusterGroupName   :
    ComputerName       : myserver
    Description        : Snapshot of virtual disk 1777681073 taken at 12/8/2013 12:04:38 AM
    DiskType           : ReadOnlySnapshot
    HostVolumeId       : {67F71B49-BD09-457E-8E33-34607970BE1C}
    LocalMountDeviceId :
    OriginalPath       : e:\csv.vhd
    ParentPath         :
    Path               : \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy{2F56E973-560E-11E3-93F6-00259067F05B}\csv.vhd
    SerialNumber       : C4A45A4D-E7E0-4AEE-B800-403DD32E8A61
    Size               : 8796093022208
    SnapshotIds        :
    Status             : NotConnected
    VirtualDiskIndex   : 1977840284
    
    MANY others listed here...
    
    ClusterGroupName   :
    ComputerName       : myserver
    Description        : Snapshot of virtual disk 1777681073 taken at 12/18/2013 1:03:52 AM
    DiskType           : ReadOnlySnapshot
    HostVolumeId       : {67F71B49-BD09-457E-8E33-34607970BE1C}
    LocalMountDeviceId :
    OriginalPath       : e:\csv.vhd
    ParentPath         :
    Path               : \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy{F1BD4A63-60EC-11E3-93F6-00259067F05B}\csv.vhd
    SerialNumber       : C4A45A4D-E7E0-4AEE-B800-403DD32E8A61
    Size               : 8796093022208
    SnapshotIds        :
    Status             : NotConnected
    VirtualDiskIndex   : 2007241302

    My searches for an explanation for this has, so far, turned up nothing.

    When I do a "vssadmin list shadows" a whole bunch of crap is spit out as well:

    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2012 Microsoft Corp.
    
    Contents of shadow copy set ID: {2020c103-c81f-4c9f-a3e9-43ab496ba5b4}
       Contained 1 shadow copies at creation time: 11/29/2013 5:14:45 PM
          Shadow Copy ID: {09b6cd4f-28d2-4498-a132-3f9d6c716580}
             Original Volume: (E:)\\?\Volume{67f71b49-bd09-457e-8e33-34607970be1c}\
             Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1
             Originating Machine: myserver
             Service Machine: myserver
             Provider: 'Microsoft Software Shadow Copy provider 1.0'
             Type: DataVolumeRollback
             Attributes: Persistent, No auto release, No writers, Differential, Auto recovered
    
    ....
    
    Contents of shadow copy set ID: {b3650eb5-ae1c-4ff1-92f0-90a33d0f82b1}
       Contained 1 shadow copies at creation time: 12/18/2013 12:34:08 AM
          Shadow Copy ID: {757b8c99-265a-4206-8a16-31b28597cf2f}
             Original Volume: (E:)\\?\Volume{67f71b49-bd09-457e-8e33-34607970be1c}\
             Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy40
             Originating Machine: myserver
             Service Machine: myserver
             Provider: 'Microsoft Software Shadow Copy provider 1.0'
             Type: DataVolumeRollback
             Attributes: Persistent, No auto release, No writers, Differential, Auto recovered
    
    Contents of shadow copy set ID: {4f58d81d-ebe1-48fe-a88b-17ff0b20757f}
       Contained 1 shadow copies at creation time: 12/18/2013 1:03:52 AM
          Shadow Copy ID: {318a9fb3-1476-4a41-98eb-00a60718cb3f}
             Original Volume: (E:)\\?\Volume{67f71b49-bd09-457e-8e33-34607970be1c}\
             Shadow Copy Volume: \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy41
             Originating Machine: myserver
             Service Machine: myserver
             Provider: 'Microsoft Software Shadow Copy provider 1.0'
             Type: DataVolumeRollback
             Attributes: Persistent, No auto release, No writers, Differential, Auto recovered

    I can't find anything related in the system event log, but the timing of the majority of the creation times coincides somewhat with my backup schedule. There are 3 virtual machines in the protection group, not sure if this is coincidence or not, but every morning there are 3 more items in the iSCSI target list.

    I was wondering if anyone could possibly enlighten me as to what exactly is going on here? Why are these snapshots appearing as iSCSI virtual disks? This looks like a potential cause for a major disaster...

    Thanks for reading.

    Greg

    Monday, December 30, 2013 8:54 PM

All replies

  • Hope everyone's had a great holiday season!

    Here's a BUMP and a hope that someone in the Microsoft community can respond to this issue, it would be nice to not have to create a premier support case about this.

    Cheers,
    Greg

    Friday, January 3, 2014 10:17 PM
  • Hi,

    Install this Windows fix and if that does not help, please open a support case with the Windows group.

    2878635 - Update is available that improves the resiliency of the cloud service provider in Windows Server 2012: December 2013
    http://support.microsoft.com/kb/2878635


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.



    Friday, January 3, 2014 10:27 PM
    Moderator
  • Thanks for the response Mike. I've installed the update, which sadly prompted for a reboot.

    Since this server provides the storage for all of the virtual machines which our company relies on for basic operations, this isn't a trivial thing, so I've scheduled a reboot for this weekend after hours (Friday January 10).

    A few questions:

    -Do you think it is safe to remove the iSCSI target artifacts before the reboot? Or should I wait until after the reboot to do this?
    -Are there any recommended steps or best practices I should follow in order to safely remove the shadow copies being left behind (which now total 101)?

    Essentially, I'm concerned that all of these artifacts and shadow copies could compromise the integrity of my cluster shared volume, which would be absolutely catastrophic to our environment and my sanity. What will happen to these artifacts when the system is rebooted?...

    Thanks for any guidance or suggestions.

    Greg


    • Edited by gr0x0rd Tuesday, January 7, 2014 7:39 PM
    Tuesday, January 7, 2014 7:23 PM
  • Hi,

    I'm not on the Windows team, so don't know about the inner workings of the fix and what to expect.  However, you can try using diskshadow.exe to delete all the snapshots for the given volume and see if that clears them up.

    DISKSHADOW> delete shadows /?

    DELETE SHADOWS { ALL | VOLUME <volume> | OLDEST <volume> | SET <setID> | ID <shadowID> | EXPOSED <drive letter, mountPoint or share> }

            Delete shadow copies, both persistent and non-persistent.

            ALL                     All shadow copies.
            VOLUME <volume>         Delete all shadow copies of the given volume.
            OLDEST <volume>         Delete the oldest shadow copy of the given volume.
            SET <setID>             Delete the shadow copies in the shadow copy set specified by the setId parameter.
            ID <shadowID>           Delete the shadow copy specified by the shadowId parameter.
            EXPOSED <exposeName>    Delete the shadow copy that is exposed at the specified drive letter, mount point or share.

            Examples: DELETE SHADOWS ALL
                      DELETE SHADOWS EXPOSED p:
                      DELETE SHADOWS EXPOSED ShareName

    So something like:  DELETE SHADOWS VOLUME E:   or  DELETE SHADOWS ALL


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, January 7, 2014 8:00 PM
    Moderator
  • Thanks Mike.

    A few moments ago I deleted the shadows and the corresponding iSCSI target artifacts. My heart was in my throat for part of the process, but at the moment, everything appears to be stable.

    I will post again after I bring the cluster down and reboot the storage server on Friday afternoon.

    Again, thanks for the help.

    Greg

    Wednesday, January 8, 2014 9:21 PM
  • Just an update- I rebooted the server Friday, as scheduled, after applying the fix. Sadly, it did not fix the issue. I'm still getting an iSCSI target and a shadow for each VM I back up using my DPM.

    As I mentioned earlier in this thread, I used the instructions at

    http://blogs.technet.com/b/filecab/archive/2012/10/08/iscsi-target-storage-vds-vss-provider.aspx

    to configure the provider. I've been reviewing these steps, and recollecting my going through them originally. In particular, I'm curious about the following commands:

    PS C:\> $PrvdSubsystemPath = New-Object System.Management.ManagementPath("root\wmi:WT_iSCSIStorageSubsystem")
    
    PS C:\> $PrvdSubsystemClass = New-Object System.Management.ManagementClass($PrvdSubsystemPath)
    
    PS C:\> $PrvdSubsystemClass.AddStorageSubsystem("<remote-machine>")

    I recall being hazy on this step, and why it was necessary. I'd like to revisit what I did here, but as yet I haven't been able to view the results of these commands via any GUI console or PowerShell commands. Right now my gut instinct is that some type of misconfiguration of the above commands may be causing this issue.

    I'll keep researching this to see if I can find more details on this; in the meantime, any feedback or suggestions are most welcome.

    Greg

    Tuesday, January 14, 2014 12:46 AM
  • In further researching these commands I've found a number of resources, but none of them have been particularly helpful. The following link has a description of the Add and Remove StorageSubsystem powershell commands... but nothing to view what is currently configured or installed on the system.

    http://msdn.microsoft.com/en-us/library/dn441533%28v=vs.85%29.aspx

    There's also an overview of VDS and Windows Storage Management API:

    http://msdn.microsoft.com/en-us/library/windows/desktop/aa381442%28v=vs.85%29.aspx

    http://msdn.microsoft.com/en-us/library/windows/desktop/hh830613%28v=vs.85%29.aspx

    However this information is fairly high-level and doesn't provide any detail about determining what a system's current configuration looks like or how to change it. I've tried using the RemoveStorageSubsystem command with what I would have entered when adding them- notably my three cluster nodes- but it returns a "You cannot call a method on a null-valued expression".

    My background is linux so I am really missing my man pages right about now. Any help or suggestions are definitely appreciated.

    Greg

    Tuesday, January 14, 2014 5:38 PM