none
DPM 2010 + Dell MD3200 VSS Hardware Provider Issues RRS feed

  • Question

  • I'm trying to backup virtual machines hosted on a 4 nodes Windows Server 2008 R2 Hyper-V cluster. The cluster has two cluster shared volumes. One is on Netapp iSCSI storage, the other one on the Dell MD3200. If I try to backup machines on the MD3200 CSV they are not snapped using the hardware provider.

    Looking further into this issue with vssadmin,  diskshadow and a simple NTFS volume on the MD3200:

    C:\Program Files (x86)\Dell\MD Storage Software>vssadmin list providers
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.
    
    Provider name: 'SmVssProvider'
     Provider type: Hardware
     Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
     Version: 10.70.08
    
    Provider name: 'Microsoft Software Shadow Copy provider 1.0'
     Provider type: System
     Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
     Version: 1.0.0.7
    
    C:\Program Files (x86)\Dell\MD Storage Software>diskshadow
    Microsoft DiskShadow version 1.0
    Copyright (C) 2007 Microsoft Corporation
    On computer: FCHV-01A, 16.11.2010 16:19:09
    
    
    DISKSHADOW> set context persistent
    
    DISKSHADOW> add volume e: provider {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
    The registered provider does not support shadow copy of this volume.

    Not even snapping this simple volume is possible!

    Dell claims it's the backup software and DPM 2010 is not supported. But at least the above test with diskshadow should work in my opinion....

    I have configured the HW provider according to Dell's document "Data Protection in a Hyper-V Environment" but to no avail. Now I'm stuck, spent money on the array and the snapshot license for it and it doesn't even do the basic stuff.

     

    • Moved by MarcReynolds Tuesday, November 16, 2010 3:43 PM (From:Data Protection Manager)
    Tuesday, November 16, 2010 3:23 PM

Answers

  • Marcus I've now personally discussed this with some folks at Dell and they've confirmed that the hrdware provider does not support CSV snapshots on the MD3200 series of products.
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, November 24, 2010 9:55 PM
    Moderator

All replies

  • The commands which you're executing with Diskshadow should absolutely work and also eliminates DPM from the picture. Other than invoking a hardware snapshot directly on the MD3200 has Dell provided you with another way to test?
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    Thursday, November 18, 2010 10:08 PM
    Moderator
  • Hi,

    let's try this (bit more formal)
    Assuming E:\ is a simple volume on the MD3200

    BEGIN BACKUP
    SET CONTEXT PERSISTENT NOWRITERS  # we don't want any writer involvement now
    SET OPTION TRANSPORTABLE # we don't want local exposure/import now
    #you may also want to test SET OPTION PLEX as the MD might require this

    ADD VOLUME E:  # let VSS sort out what provider to use, TRANSPORTABLE needs hw provider anyway 
    CREATE
    END BACKUP

    This should work, if not DELL has to kick-in and help resolve.


    \R2 This posting is provided "AS IS" with no warranties, and confers no rights
    Sunday, November 21, 2010 1:31 PM
  • Thanks for the input!

    I have tried the command Ruud gave me:

     

    DISKSHADOW> begin backup
    
    DISKSHADOW> SET CONTEXT PERSISTENT NOWRITERS
    
    DISKSHADOW> SET OPTION TRANSPORTABLE
    
    DISKSHADOW> ADD VOLUME E:
    The provider does not support volume shadow copies for this volume in this context.
    You must either change the context or add a different volume.
    Note that changing the context might cause existing volumes in the shadow copy set to become unsupported.
    

    Failure :-(

    Dell told me that their hardware provider doesn't support CSV. Which seems rather odd because the have two documents on implementing Hyper-V on R2 with the MD32xx: "Implementing Hyper-V
    A Dell PowerVault™ MD3200 and MD3200i Technical White Paper" and "Data Protection in a Hyper-V Environment".

    The commands also didn't work with "set option plex". I'm at a total loss here :-(

     

     

    Tuesday, November 23, 2010 11:32 AM
  • No tests from Dell I can try so far.
    Tuesday, November 23, 2010 11:34 AM
  • Marcus I've now personally discussed this with some folks at Dell and they've confirmed that the hrdware provider does not support CSV snapshots on the MD3200 series of products.
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.
    Wednesday, November 24, 2010 9:55 PM
    Moderator
  • Tyler, thanks for the info. This is what they have been telling me lately as well. But I'm still wondering why Dell has whitepapers for Hyper-V on R2 which also mentions CSV and shows how to do hardware VSS snapshots... I haven't found anything where they clearly state that they don't or do support CSV. This is very annoying...
    Friday, November 26, 2010 9:28 AM
  • Maybe we are after the same problem

    Our hosts have this update: http://support.microsoft.com/kb/975354
    Does the second note mean that the host will not use VSS?

    ...

    Issue 2

    Consider the following scenario:

    • Cluster shared volumes are enabled on a failover cluster for Hyper-V.
    • Some virtual machines are saved on the same volume. But they are running on different nodes.
    • These virtual machines are backed up in parallel.

    In this scenario, the virtual machine backup operation fails.
    ...

    Notes

    • This update contains fixes for both VSS and Hyper-V. The Hyper-V update is only applicable to the x64 versions of Windows Server 2008 R2.
    • After this rollup is installed, Hyper-V will only use the VSS system provider inside the virtual machine when performing a host-level backup.

    ...

     

    Friday, December 3, 2010 4:02 PM
  • Is there any work around? Or you have to get a different DAS if you want to use DPM?


    Darwaish
    Thursday, December 23, 2010 5:22 PM
  • You can use the default system provider but then have to serialize VM backup.

    I you want/must have parallel backup you need storage with a working VSS hardware provider.


    \R2 This posting is provided "AS IS" with no warranties, and confers no rights
    Thursday, December 23, 2010 9:09 PM
  • We have used DPM 2010 with serialized backup and Software VSS and this is of cource not an option when doing large backups. We have just invested in a HP 4300G2 SAN from HP (Former Lefthand) to be able to use Hardware Snapshots. And what happens.... We are getting alot of different errors. BSOD, Rebooting VM:s, Cluster issues, you name it.

    We have installed ALL available hotfixes, All the latest firmware in SAN, Servers (HP Blade), Switches (Cisco), All the latest drivers in every system.
    There is alot of strange thing going on at Microsoft at the moment. For example: http://support.microsoft.com/kb/2406705" witch hade a fix to the mpio.sys file. Then this fix came along: http://support.microsoft.com/kb/979743 witch also updated the mpio.sys and had a newer version than KB:2406705.
    I of cource installed the latest version that came with KB: 979743. Then some days later I returned to KB 2406705 because 979743 did not help Microsoft had updated the mpio.sys file in 2406705 and the file version now was newer than 979743. There are already 4 hotfixes for Windows Server 2008 R2 SP1 that is related to HA stuff like, BSOD with CSV, BSOD when doing Snapshots, etc...
    http://support.microsoft.com/kb/2494016
    http://support.microsoft.com/kb/2496089
    http://support.microsoft.com/kb/2460912
    http://support.microsoft.com/kb/2494162

    Come On.... If there is someting that need to be stable it is HA related services.
    We have now halted the implementation of our new Hyper-V cluster and are thinking about going with vmware instead.
    Can someone at Microsoft please make a list of all hotfixes that is needed to get a Windows R2 Hyper-V CSV Cluster to work with hardware VSS Provider and Microsoft DPM 2010 Server.

     

    Monday, February 14, 2011 7:30 PM
  • Hi,

    Since the only changes made was going from a software provider to a hardware provider, then their must be something wrong with the provider. You will need to investigate the system crashes by opening a support ticket, it could be a driver issue, but we cannot help you in a forum setting. 

    Below is a list of tested hardware providers used for Hyper-V backups, you may want to get in touch with HP for their guidance.

    http://blogs.technet.com/b/dpm/archive/2010/02/05/tested-hardware-vss-provider-table.aspx

    As far as other recommended hotfixes for the record.

    OTHER RECOMMENDED Updates for Windows 2008 R2
    ========================================

    Note: The hotfixes below contain the latest version of the file specified.


    VOLSNAP.SYS
    ===========
    2460912 "0x0000007E" Stop error when you create snapshots of a volume in Windows Server 2008 R2 or in Windows 7
    http://support.microsoft.com/default.aspx?scid=kb;en-US;2460912


    Clussvc.exe
    ===========
    The Cluster service stops unexpectedly on a Windows Server 2008 R2 failover cluster node when you perform multiple backup operations in parallel on a cluster shared volume
    http://support.microsoft.com/kb/2494162


    CSVFILTER.SYS
    =============
    2494016 Stop error 0x0000007a occurs on a virtual machine that is running on a Windows Server 2008 R2-based failover cluster with a cluster shared volume, and the state of the CSV is switched to redirected access
    http://support.microsoft.com/default.aspx?scid=kb;en-US;2494016


    MPIO.SYS
    ========
    2277904 You cannot access an MPIO-controlled storage device in Windows Server 2008 R2 after you send the "IOCTL_MPIO_PASS_THROUGH_PATH_DIRECT" control code that has  an invalid MPIO path ID
    http://support.microsoft.com/default.aspx?scid=kb;en-US;2277904

     


    Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.
    Monday, February 14, 2011 10:59 PM
    Moderator
  • Marcus I've now personally discussed this with some folks at Dell and they've confirmed that the hrdware provider does not support CSV snapshots on the MD3200 series of products.
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.

    Just wondering if you ever heard any more on this issue?  It is strange since Dell's whitepapers mention this feature.  I have an MD3000i and MD3220i that have this issue.
    Thursday, April 28, 2011 4:37 PM
  • You can use the default system provider but then have to serialize VM backup.

    I you want/must have parallel backup you need storage with a working VSS hardware provider.


    \R2 This posting is provided "AS IS" with no warranties, and confers no rights

    I was reading http://technet.microsoft.com/en-us/library/ff634192.aspx about serializing and I wasn't sure if both the host and LUN settings needed to be done or if it was an either choice?  The main downside to serialized backup is performance correct?  If I do it at off peak times there shouldn't be an issue?
    Thursday, April 28, 2011 4:39 PM
  • No solution. I believe Dell is only interested in selling other, way more expensive storage for virtualization like Equallogic. I'm pretty annoyed that Dell nowhere clearly states, that the HW provider does not support CSV.
    Friday, April 29, 2011 8:44 AM
  • Serialization only needs to be done for Hyper-V on CSV volumes. And you need to do it per cluster per CSV. So if you have one CSV and four nodes, you need to do it once. If you have four nodes and two CSVs then I believe you need to have two XML files. The downside of serialization is that it takes a lot longer for the Hyper-V backup of all your machines. But since DPM doesn't really do true express full backups (it reads source and destination and only transfers changed blocks which causes very high IO load on both the DPM server and the Hyper-V host) running parallel backups should only be done during off peak hours. This is my experience. So serial backups probably can be done during work hours. As I understand it this limitation of DPM is because of the Hyper-V VSS writer. But I might be wrong with that.
    Friday, April 29, 2011 8:51 AM
  • Marcus I've now personally discussed this with some folks at Dell and they've confirmed that the hrdware provider does not support CSV snapshots on the MD3200 series of products.
    Cheers, Tyler F [MSFT] - This posting is provided "AS IS" with no warranties, and confers no rights.

    There is a new release of the MD3XX0i VSS providers available at http://support.dell.com/support/downloads/download.aspx?releaseid=R302057&fileid=454613.  I was curious if anyone could check with Dell to see if anything has been fixed yet?  It is hard from a customer standpoint to troubleshoot since you have to start at zero with finger point at DPM.  Thanks. 
    Wednesday, August 17, 2011 12:06 AM
  • Does anyone have an update on this?
    Tuesday, August 30, 2011 1:14 PM
  • I'd also be interested to know if the new VSS providers work with DPM. Anybody?
    Friday, September 23, 2011 6:10 PM
  • Any update on this, at all??? I'm in the same boat...HyperV 2008 R2 CSV on Powervault 3200i, DPM 2010...no dice!
    Keep the Funk alive!
    Saturday, November 19, 2011 5:25 PM
  • I spoke to a Dell technician this afternoon about my 3220i as I'd raised the same problem with the Hardware VSS not working. He told me that it still isn't supported but would be in a "future" firmware release, but couldn't give a timeframe.

    I too read the deployment guides without seeing any warnings about the 3220i not supporting Hardware VSS for CSV's.

    Thursday, November 24, 2011 5:05 PM
  • I've had some promising results from talking to Dell, below is a method that does get the hardware VSS working, it was provided as a "best effort" attempt and is not supported officially.

    The important thing to remember is the snapshot is presented to the node which currently owns the virtual machine not to the DPM server. The DPM server then backups up over the LAN network. Hence DPM backups are typically slow by design.

     

    The registry on each node must be edited using the SmRegTool so the VDS provider correctly recognises the MD array.

     

    Note: It is not necessary to run this tool on the DPM server, only on each node, since the DPM server makes the request of the Hyper-V node to perform the backup.

     

    This is the standard location of the tool:

    C:\Program Files\Dell\MD Storage Software\VDS_VSS\provider\SmRegTool.exe

     

    (however in my case I found it here)

     

    C:\Program Files (x86)\Dell\SMprovider\provider\SmRegTool.exe

     

    These are the settings used (make sure the array name matches exactly what yours is called):

     

     

    I left my % of base volume at 20% as I have the disk space spare and a very large fileserver VHD.

     

     

    These IP addresses are of the Management NIC's on the Hosts and the SAN, don't use any of your iSCSI connections. 

     

    I ran a test backup and it puts the CSV into redirected mode while the snapshot is taken and then straight back to online while the backup copies from the snapshot.

     

    I've not yet tested concurrent backups, but this is a huge step forward for me.

     

    Matt

    Friday, December 2, 2011 12:03 AM
  • Microsoft DPM 2010 + Dell PowerVault MD3200 = Who to blame?

     

    "VSS Hardware Provider" seemed to fail in Cluster Shared Volume environment.

    There is no problem if we backup many VMs of the same Hyper-V host.

    But, if we try to backup many VMs of different hosts, many recovery-points fail randomly.

     

    So we tried to follow "Enabling Per Node Serialization" of this page: http://technet.microsoft.com/en-us/library/ff634192.aspx

    After setting "MaxAllowedParallelBackups" to "1" and rebooting DPM Server, we tested again

    And right now, DPM2010 is backing up one virtual machine per each Hyper-V node.

    Does this mean that "VSS Hardware Provider" is working properly in CSV environment?

     

    Summary:

    With "MaxAllowedParallelBackups"="3", I can backup three VMs of the ONE node in parallel.

    With "MaxAllowedParallelBackups"="1", I can backup one VM of MANY nodes in parallel.



    • Edited by HamletMun Tuesday, December 27, 2011 10:09 PM
    Tuesday, December 27, 2011 10:08 PM
  • I found this hotfix, but no idea if it is related or not.
     
    http://support.microsoft.com/kb/2494162/en-us
     
    Article ID: 2494162 - Last Review: February 10, 2011 - Revision: 1.0
     
    The Cluster service stops unexpectedly on a Windows Server 2008 R2 failover cluster node when you perform multiple backup operations in parallel on a cluster shared volume
    Wednesday, December 28, 2011 6:24 PM