none
DPM 2010, Hyper-V Cluster (CSV), Storage Server 2008, Hardware VSS RRS feed

  • Question

  • I have been trying with little luck to get DPM 2010 on it's feet backing up VMs on our cluster nodes. 

    We have a cluster of (5) servers, running 2008 R2.  The cluster is running only Hyper-V VMs, distributed across the nodes.  The SAN is running Storage Server 2008 (R1), which is providing CSVs for the VMs to reside on. 

    I have been trying to get the hardware VSS providers working in order to perform "best practice" backups of the VMs.   I do not want to take the performance hit of serialized backups. 

    Here is what I have done so far, as far as modifying our environment to support this:

    - I have added our storage server to the domain.  It was not previously.

    - I created an account called 'sys_iscsi' in AD.   I added this account to the local administrators group of all cluster nodes, and the storage server.

    - I installed iscsitargetclient.msi on all cluster nodes, using the above username and it's password as part of the setup process. 

    I am not sure what I am missing, as from what I've studied from the relatively scant documentation I've found, that's about all there is supposed to be to it.  

     

    On the DPM server, when I create a protection group, I target the cluster's object, and a hyper-v host contained underneath it.  However, the initial backup fails, with "Replica is inconsistent."  When I attempt to perform a consistency check, the job fails after 30-60 seconds, generating this error:

    Failure occured while adding one or more of the volumes involved in the backup operation to snapshot set.  Please check the event log on [hyper-v VM that was targeted] to troubleshoot the issue.  (ID 30290 Details:  Internal Error Code:  0x80990A00)

    What is strange is that the error is telling me to look on the event log of the VM that was targeted, even though the backup takes place at the cluster level, so the VM should not even be aware that it's being backed up.  Regardless, I checked the event logs on the target VM, and there was nothing there. 

    All of the information I have found relating to this error is pertaining to third party SAN appliances, and not Windows Storage Server 2008, so any assistance I can get would be appreciated.

    Monday, May 16, 2011 5:13 PM

All replies

  • The Microsoft iSCSI Software does not act as a hardware provider capable of backing up multiple VMs simultaneously using a CSV.  It doesn't support hardware snapshots for CSV's. 

    The "How to protect Hyper-v with DPM 2010" white paper available at http://www.microsoft.com/downloads/en/details.aspx?displaylang=en&FamilyID=c9d141cf-c839-4728-af52-928f61bebdca includes the statement "If the Hyper-V host will use Cluster Shared Volumes (CSV), ensure that the appropriate Volume Shadow Copy Services (VSS) hardware provider is installed. The VSS hardware provider allows you to back up multiple VMs simultaneously. If the VSS hardware provider is not available, the VSS software provider is used and backup jobs are serialized."

    The main item related to DPM to be aware of when using a hardware provider that supports hardware snapshots for CSV's is the registry key MaxAllowedParallelBackups listed in "Considerations for Backing Up Virtual Machines on CSV with Hardware VSS Providers" at http://technet.microsoft.com/en-us/library/ff634220.aspx.

    There is a list of tested hardware VSS providers for DPM 2007 at http://blogs.technet.com/b/dpm/archive/2010/07/08/tested-hardware-vss-provider-table.aspx.  We do not have a list of tested hardware VSS providers for DPM 2010.

    Friday, May 20, 2011 6:50 PM
    Moderator
  • I have to admit that at this point, I am pretty confused in light of what you are saying. 

    Your comments and accompanying whitepaper regarding DPM's serial backup functionality are understood.  But you are also saying that the Microsoft iSCSI target software does not have a hardware VSS provider, or at least that's what I'm reading into it... ??

    However, this technet article directly contradicts what you are saying if I am understanding all of this correctly. http://technet.microsoft.com/en-us/library/dd573327(WS.10).aspx  Additionally, this article http://technet.microsoft.com/en-us/library/ff634189.aspx discusses the performance impact of doing non-hardware-VSS based DPM backups, an impact which I wish to avoid.  In essence, the latter article is saying that if I do not use hardware VSS in my backup scheme, that when backing up a VM on a cluster node, all other VMs on the same cluster node will sit in redirected I/O for the duration of the backup.  Since I have quite a few VMs on each cluster node, that would carry some serious performance implications for us. 

    So there is either completely conflicting information, or I am misunderstanding either what you are saying or what the technet articles are saying. 

    The articles seem pretty clear over the process of using the iscsitargetclient.msi to enable CSV/DPM backups using hardware-VSS, which avoids serialization, but my issue is that it just doesn't work.  I've followed the instructions, but I receive the errors in my original post. 

    So really what this comes down to is either I've misinterpreted the articles and am trying to do something that completely will not work, or what I am trying to do is supposed to work, it's just not. 

     

     



    Monday, May 23, 2011 2:36 PM
  • The Microsoft iSCSI Software is a hardware VSS provider. However, it doesn't support hardware snapshots for CSV's.  You can use the Microsoft iSCSI Software with the CSV but you will need to serialize the backups when using it as the hardware provider.  You must purchase a hardware provider solution that supports hardware snapshots for CSV's in order to have the ability to to perform parallel backups.

     

    Monday, May 23, 2011 7:15 PM
    Moderator
  • Okay. 

    So do you believe that I have done everything correctly up to this point?  Should changing that registry key (and bouncing the server) allow it to work properly in your estimation? 

    If so I'll give it a shot and post the results.

    Monday, May 23, 2011 7:20 PM
  • Once you apply the settings from "Considerations for Backing Up Virtual Machines on CSV with the System VSS Provider" at http://technet.microsoft.com/en-us/library/ff634192.aspx the consistency checks should work.  Look for VSS errors on the Hyper-v if the backups continue to fail.
    Monday, May 23, 2011 8:27 PM
    Moderator
  • I had previously been through the document you linked to above, but I went through it again just to make sure. 

    - Used the powershell script to generate the XML file.

    - Place the XML file in the config folder on DPM

    - Verified registry key

    - Restarted DPM server just out of an abundance of caution. 

    - Created a new protectiong group, targeting a VM I knew to be on node 0 of the cluster. 

     

    DPM generated the same error:

    Failure occurred while adding one or more of the volumes involved in backup operation to snapshot set. Please check the event log on xxxxxx.xxxxx.xxxxx.xxx to troubleshoot the issue. (ID 30290 Details: Internal error code: 0x80990A00). 

    On node 0 of the cluster, I received this message: 

    Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {2f900f90-00e9-440e-873a-96ca5eb079e5}. Routine details BeginPrepareSnapshot({2f900f90-00e9-440e-873a-96ca5eb079e5},\\?\Volume{cfa9d014-5119-4465-b757-9d4d9ee6f70a}\) [hr = 0x800706ba, The RPC server is unavailable.

    ].

    Operation:

    Processing BeginPrepareSnapshot

    Add a Volume to a Shadow Copy Set

    Context:

    Provider Name: Microsoft iSCSI Target VSS Hardware Provider

    Provider Version: 3.2.3688

    Provider ID: {2f900f90-00e9-440e-873a-96ca5eb079e5}

    Volume Name: \\?\Volume{cfa9d014-5119-4465-b757-9d4d9ee6f70a}\

    Snapshot ID: {69fbcf9f-ad7b-4b7e-bbe7-8ab56da21d79}

    Snapshot Context: 4194304

    Execution Context: Coordinator

     

     

    The iSCSI Software Target Volume Shadow Copy Service Hardware Provider service could not prepare the snapshot for LUN 0x00000000 on (null). 0x800706ba.



    as well as this error:
    Tuesday, May 24, 2011 5:07 PM
  • The steps you outlined are correct and should allow DPM to back up the Hyper-V guest. At this point the problem appears to be caused by VSS errors on node 0 of the cluster. You can run "vssadmin list writers" on the host to see if the Microsoft Hyper-V VSS Writer is in a failed state or has any errors. Restarting the Hyper-V Machine Management Service will get you out of an error state. You should also look for VSS errors in the event viewer application log. If you find VSS errors you can try the VSS forums or open a support incident if you want additional help troubleshooting the problem.

    Thursday, May 26, 2011 9:11 PM
    Moderator
  • Here is the output of writer list on the cluster node in question.  I'm not sure which one relates to the iSCSI hardware provider... is it missing from this list?  Perhaps it is the Hyper-V VSS writer?

     

    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2005 Microsoft Corp.

    Waiting for responses.
    These may be delayed if a shadow copy is being prepared.

    Writer name: 'Task Scheduler Writer'
       Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       State: [1] Stable
       Last error: No error

    Writer name: 'VSS Metadata Store Writer'
       Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       State: [1] Stable
       Last error: No error

    Writer name: 'Performance Counters Writer'
       Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       State: [1] Stable
       Last error: No error

    Writer name: 'Microsoft Hyper-V VSS Writer'
       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
       Writer Instance Id: {e29e3a6d-3633-472f-b75a-e196aa47d7ea}
       State: [1] Stable
       Last error: No error

    Writer name: 'System Writer'
       Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
       Writer Instance Id: {8312dcfb-c391-46d6-aa78-0c86aa37611b}
       State: [1] Stable
       Last error: No error

    Writer name: 'ASR Writer'
       Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Writer Instance Id: {74bc2892-af96-4c39-9368-e22378cf3a52}
       State: [1] Stable
       Last error: No error

    Writer name: 'Registry Writer'
       Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Writer Instance Id: {2a35e7a3-403b-44c9-b3fd-14dc1e8a9d00}
       State: [1] Stable
       Last error: No error

    Writer name: 'Shadow Copy Optimization Writer'
       Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Writer Instance Id: {cac758a8-bec6-42fd-a238-77e2805e6e1f}
       State: [1] Stable
       Last error: No error

    Writer name: 'BITS Writer'
       Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
       Writer Instance Id: {53b76f6c-2a66-4e0c-8a1c-3ae0112e6bff}
       State: [1] Stable
       Last error: No error

    Writer name: 'WMI Writer'
       Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Writer Instance Id: {b7960452-974f-4795-bf10-b6cf82572166}
       State: [1] Stable
       Last error: No error

    Writer name: 'Cluster Database'
       Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
       Writer Instance Id: {5b97b3d0-07e5-45ff-ab47-9a74eacad718}
       State: [1] Stable
       Last error: No error

    Writer name: 'COM+ REGDB Writer'
       Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Writer Instance Id: {9d5e6b1d-6d5e-4c51-84cc-869a6a8b861d}
       State: [1] Stable
       Last error: No error

     


    Tuesday, May 31, 2011 7:16 PM
  • To verifiy the hardware or software VSS provider being used run vssadmin list providers.

     

    My apologies that this has fallen off our radar.  I have found no errors referencing the VSS errors with error codes indicating the "RPC server is unavailable."  Are you still having trouble with DPM backups of the Hyper-V hosted VM's?

     

    Please review the following blog for some additional recommendations.  "Troubleshooting Hyper-V protection failures due to networking problems" at  http://blogs.technet.com/b/dpm/archive/2011/12/06/troubleshooting-hyper-v-protection-failures-due-to-networking-problems.aspx

     

     

    Thursday, January 19, 2012 6:26 PM
    Moderator
  • To be honest, we pretty much shelved this initiative when we couldn't get any further on it.  Please allow me some time to find some downtime where I can get back on this box and get myself back to where I was. 
    Thursday, January 19, 2012 6:49 PM