none
DPM 2016, Hyper-V 2016 CSV backup error (0x80042313) RRS feed

  • Question

  • Hi,

    we are dealing with the persisting error during backup jobs on our cluster. Originally, we had two node Hyper-V 2012 R2 cluster, SC VMM 2012 R2 and DPM 2012 R2 - this was a fully functional setup and we were enjoying our periodically successful backups. A month ago, we´ve decided to upgrade to the latest. So firstly, we´ve upgraded DPM to 2016 (but still sitting on 2012 R2 OS), then VMM to 2016 (and also underlying OS to 2016), a then we´ve performed rolling cluster update.

    Now, all backup tasks against clustered VMs fails (even the first synchronization). Backup job fails and in DPM console shows:

    DPM encountered a retryable VSS error. (ID 30112 Details: VssError:The shadow copy provider timed out while flushing data to the volume being shadow copied. This is probably due to excessive activity on the volume. Try again later when the volume is not being used so heavily. (0x80042313)).

    Both nodes show error 12297 repeatedly. Details of each error are:

    <Event
    xmlns
    ="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="VSS" />
      <EventID
    Qualifiers
    ="0">12297</EventID>
      <Level>2</Level>
      <Task>0</Task>
      <Keywords>0x80000000000000</Keywords>
      <TimeCreated SystemTime="2016-11-28T16:22:06.289900800Z" />
      <EventRecordID>5492</EventRecordID>
      <Channel>Application</Channel>
      <Computer>Clymene.spstr.cz</Computer>
      <Security />
      </System>
    - <EventData>
      <Data>\\?\Volume{fcca4af5-9739-4152-ae90-051ec16043e2}\</Data>
      <Data>0</Data>
      <Data>0x00000000, The operation completed successfully.</Data>
      <Data>0x80042313, The shadow copy provider timed out while flushing data to
    the volume being shadow copied. This is probably due to excessive activity on
    the volume. Try again later when the volume is not being used so
    heavily.
    </Data>
      <Data>0x00000000, The operation completed successfully.</Data>
      <Data>0x00000000, The operation completed successfully.</Data>
      <Data>Operation: Executing Asynchronous Operation Context: Current State:
    DoSnapshotSet
    </Data>
      <Binary>2D20436F64653A20434F524C4F564C4330303030303938322D2043616C6C3A20434F524C4F564C4330303030303738302D205049443A202030303030383439322D205449443A202030303030353931322D20434D443A2020433A5C57696E646F77735C73797374656D33325C76737376632E6578652020202D20557365723A204E616D653A204E5420415554484F524954595C53595354454D2C205349443A532D312D352D313820</Binary>
      </EventData>

     </Event>

    But on the mentioned volume (CSV), there is almost no activity during backup job.

    From HW perspective, the cluster is made of two Dell R720 servers and one Dell MD3220i SAN connected over iSCSI. Both servers are running WS 2016 Datacenter Core with enabled Hyper-V role and installed VMM agent, DPM agent and Dell MD Array Configuration Utility. Nothing more.

    I think the problem must be somewhere in the cluster, because when both system center components were updated, but before cluster was updated, everything was ok. So DPM 2016 was able to backup old 2012 R2 cluster. Old cluster was also WS 2012 R2 Datacenter Core with enabled Hyper-V role and installed VMM agent, DPM agent and Dell MD Array Configuration Utility.

    Only backup jobs containing clustered VMs fail, bare metal recovery and system state of both nodes are behaving normally.

    Does anyone experienced the same or similar issue?

    George

    Monday, November 28, 2016 6:31 PM

All replies

  • Hi,

    Windows server 2016 in combination with DPM 2016 totally do away with VSS snapshots when performing host level hyper-V backups.  However, in order for that happen you need to perform a few steps.  You can try this for one failing VM and see if it helps.

    1) Update the VM's to Windows 2016 version by running a Windows powershell command.
             Update-VMVersion -Name VMNAME

    2) Stop protection of the VM you just updated and retain the replica so previous backups are maintained.

    3) Re-protect that VM.

    DPM will then utilize Windows 2016 native resilient change tracking (RCT) feature to protect that VM.

    If that works, repeat for the remaining VMs.   Also make sure there are no VSS errors inside the VM's as vss snapshots are still performed inside the VM's to ensure file integrity when a backup is performed.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, November 28, 2016 9:26 PM
    Moderator
  • Hi,

    thank you for the reply. It works for one VM, so I've added another three. The problem is, that DPM started to backup all three VMs at once - and at this point everything screwed up. It shows same errors like I've posted yesterday. Is there a way how to tell DPM to backup VMs one by one (or are we missing any configuration to be able to perform multiple backups at one time)?

    And better, do not perform another backup jobs in the VMs, when Hyper-V backup is performed? We are backing up all VMs over night, and for example one VM running Exchange and when Hyper-V backup this VM (it takes approx. one hour) and DPM starts to backing up mailboxes, both jobs are failing over time and only restart of VM helps. So it would be nice, if I can tell the Exchange backup job (which performs every 15 minutes) to wait, until whole VM backup will be finished. The same is with SQL or any other service. After restarting the VM, it mostly survives the first double backup, but he fourth, fifth or sixth brokes it down.

    Regards

    George

    Tuesday, November 29, 2016 10:07 PM
  • Hi,

    For VM's that you are perfroming both guest level application backup and host level VM backup, you can uncheck the Intergration Services option: backup (volume checkpoint).  That will disable VSS snapshots when host level backup is done.   After doing a full VM recovery, you would then want to perform a restore of the application data so it's consistent.

    To control the maximum number of parallel Hyper-V backups, you can add this registry value on the DPM Server.  The default if not present is 3.


    [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft Data Protection Manager\2.0\Configuration\MaxAllowedParallelBackups]
    "Microsoft Hyper-V"=dword:00000001


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Tuesday, November 29, 2016 11:25 PM
    Moderator