none
DPM 2019 - Unexpected error occured - Re-occuring backup job failure RRS feed

  • Question

  • Hello, 

    I've been trying to chase down this issue at my work and have not been able to find a solution or even the same exact error message anywhere online. 

    Continually keep getting the following for one of our Hyper-V VM(RCT/SCCM) backups: 

    Replica is inconsistent

    "An unexpected error occurred while the job was running. (ID 104 Details: Unknown error (0x80041069) (0x80041069))"

    The recommended actions states to retry the operation, run a synchronization job with consistency check which does not resolve the issue, the consistency check runs for a few minutes and then it fails again. 

    The VM is on a host running Windows 2012 R2. I understand this would be a limitation of earlier versions of DPM, but we're running DPM 2019 on Windows Server 2016

    SCCM backup(BMR and System State) is failing as well with the following: 

    Replica is inconsistent

    "DPM is out of disk space for the replica. (ID 58 Details: Internal error code: 0x80990ED2)"

    The recommended action is to modify disk allocation and run synchronization job with consistency check which does not resolve the issue and it immediately fails. 

    The disk has plenty of free space, 8.6TB to be exact. 

    Any suggestions or advice would be much appreciated. 

    Thanks

    Tuesday, September 10, 2019 2:05 PM

All replies

  • Hello,

    What is the Hyper-V configuration version for your SCCM virtual machine running on the Windows Server 2012 R2 Hyper-V host?

    As for the Bare Metal Recovery (BMR) backup, have you tried modifying the default BMR replica size? 
    Create the registry key below to change the default replica size for all future BMR protection:

    Registry Path:
    HKLM\Software\Microsoft\Microsoft Data Protection Manager\Configuration 

    Registry Key (DWORD): 
    ReplicaSizeInGBForSystemProtectionWithBMR


    You can refer to the following links for more information:

    How to increase DPM 2016 replica when using Modern Backup Storage (MBS)

    https://docs.microsoft.com/en-us/system-center/dpm/back-up-system-state-and-bare-metal?view=sc-dpm-2019#prerequisites-and-limitations

    ----------------------------------------------------------------------------------------------------------------------

    Please also check the DPM logs for any more clues to why your backup jobs are failing, you'll find the logs here:

    DPM Protected Server:

    • C:\Program Files\Microsoft Data Protection Manager\DPM\Temp

    DPM Server:

    • C:\Program Files\Microsoft System Center\DPM\DPM\Temp

    You can then upload the log files to OneDrive and share the link here, I can always take a look at them to check if I see anything obvious. If you decide to upload the logs, please make sure the logs are clean first, otherwise there will be a lot of events to go through.

    You can simply move the logs out of the Temp folder, then reproduce the errors you get and then upload the logs.

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, September 10, 2019 2:27 PM
  • Hello Leon, 

    Thank you for the quick response. 

    We are running configuration version 8.0. 

    As for the BMR backup, I have not tried modifying the default replica size. I will give that a try and report my findings. 

    I'll take a look at the logs and will upload them if need be. 

    Regards,

    Vitaliy

    Tuesday, September 10, 2019 6:10 PM
  • Okay, since the Hyper-V host is a Windows Server 2012 R2, DPM will not be able to make use of the Resilient Change Tracking (RCT) feature, you could also try unchecking the "Backup (volume shadow copy)" integration service on the virtual machine.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, September 10, 2019 6:30 PM
  • Here are the logs: 

    https://1drv.ms/t/s!AiBVTHFKTV0AnTk07CgEnEAsS3XO

    https://1drv.ms/t/s!AiBVTHFKTV0AnTpOeL-fZGwIqEJa

    I created the registry key to change the default BMR replica size and that did not seem to work. 

    Thank you

    Wednesday, September 11, 2019 5:51 PM
  • Did you restart the DPM server after applying the registry change?

    You might also have to remove the server from the protection group and re-add it.

    Please also run the following command on the Hyper-V host and post the result here:

    vssadmin list writers

    Also what is the configuration of the failing virtual machine?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, September 11, 2019 6:26 PM
  • I restarted the DPM server after the registry change and removed the sccm server from the protection group and re-added it, and it's working now, 

    The VM that's failing to backup is on a 2019 host not 2012 R2, sorry for the confusion. And the configuration version is also 8.0. 

    Both of the backups described in this thread are for SCCM. The VM-level is the primary and the OS-level is the fallback. 

    Thursday, September 12, 2019 8:19 PM
  • Okay, so it's all working now and all good?

    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, September 12, 2019 8:40 PM
  • The OS level SCCM backup is all good now, thank you for that. 

    The VM level/RCT back-up is still failing with Replica is inconsistent "An unexpected error occurred while the job was running. (ID 104 Details: Unknown error (0x80041069) (0x80041069))"

    Monday, September 16, 2019 1:14 PM
  • Could you post the results of the following two commands?

    vssadmin list providers

    vssadmin list writers

    Please run them on the Hyper-V host.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, September 16, 2019 1:50 PM
  • vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2013 Microsoft Corp.
    Provider name: 'Microsoft CSV Shadow Copy Helper Provider'
       Provider type: Software
       Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d}
       Version: 1.0.0.1
    Provider name: 'Microsoft CSV Shadow Copy Provider'
       Provider type: Software
       Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
       Version: 1.0.0.1
    Provider name: 'Nimble Storage VSS Provider'
       Provider type: Hardware
       Provider Id: {5253d47a-55bb-4b3a-928d-5f4baec6fbf4}
       Version: 3.4.0.2
    Provider name: 'VSS Null Provider'
       Provider type: Software
       Provider Id: {8202aeda-45bd-48c4-b38b-ea1b7017aec3}
       Version: 10.19.58.0
    Provider name: 'Microsoft File Share Shadow Copy provider'
       Provider type: Fileshare
       Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
       Version: 1.0.0.1
    Provider name: 'Microsoft Software Shadow Copy provider 1.0'
       Provider type: System
       Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
       Version: 1.0.0.7

    Provider name: 'VSS Null Provider'
       Provider type: Fileshare
       Provider Id: {f4a69dd4-f712-40e3-a6b3-faeff03cb2b8}
       Version: 10.19.58.0

    __________________________________________________________________________________

    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2013 Microsoft Corp.
    Writer name: 'Task Scheduler Writer'
       Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       State: [1] Stable
       Last error: No error
    Writer name: 'VSS Metadata Store Writer'
       Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       State: [1] Stable
       Last error: No error
    Writer name: 'Performance Counters Writer'
       Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       State: [1] Stable
       Last error: No error
    Writer name: 'Microsoft Hyper-V VSS Writer'
       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
       Writer Instance Id: {f9b54af8-4131-4679-8424-21f6ef55d777}
       State: [1] Stable
       Last error: No error
    Writer name: 'System Writer'
       Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
       Writer Instance Id: {183ad992-badb-4bb3-99e7-b926543f2a3a}
       State: [1] Stable
       Last error: No error
    Writer name: 'Cluster Shared Volume VSS Writer'
       Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
       Writer Instance Id: {8bfee02b-4391-4934-a7f8-6626b29bcaee}
       State: [1] Stable
       Last error: No error
    Writer name: 'ASR Writer'
       Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Writer Instance Id: {409f35a3-2f66-44fd-9967-0f28086158b3}
       State: [1] Stable
       Last error: No error
    Writer name: 'Shadow Copy Optimization Writer'
       Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Writer Instance Id: {8be08b40-d7f1-402f-aa63-337d9c629831}
       State: [1] Stable
       Last error: No error
    Writer name: 'Registry Writer'
       Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Writer Instance Id: {eb6f0a64-dc08-40ca-adbb-91b2041f0066}
       State: [1] Stable
       Last error: No error
    Writer name: 'WMI Writer'
       Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Writer Instance Id: {a2bd5d6a-a2db-4786-a930-99af1a9d8a91}
       State: [1] Stable
       Last error: No error
    Writer name: 'COM+ REGDB Writer'
       Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Writer Instance Id: {3e308a48-b454-4adb-90f0-6f633d3e7486}
       State: [1] Stable
       Last error: No error
    Writer name: 'Cluster Database'
       Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
       Writer Instance Id: {092392da-7902-4e3e-97f0-c57e33958f97}
       State: [1] Stable
       Last error: No error

    Monday, September 16, 2019 2:46 PM
  • The VSS looks healthy, and all VSS providers are there.

    I believe the 0x80041069 in your error: "An unexpected error occurred while the job was running. (ID 104 Details: Unknown error (0x80041069) (0x80041069))" means "Operation Timed out"

    If I understood you right, you also have the DPM agent installed on the SCCM VM?

    Did you try unchecking the Backup (volume shadow copy) integration service of the virtual machine? Might want to try rebooting the VM as well after that.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, September 16, 2019 4:19 PM
  • We did check the integration service on the VM, and "backup (volume shadow copy)" is checked off. Our solution architect advised to not uncheck that option. Since the host is on 2019, would DPM be able to make use of the RCT feature?
    Monday, September 16, 2019 6:20 PM
  • RCT is enabled by default since Windows Server 2016, so DPM can only uses the RCT feature on Windows Server 2016/2019 Hyper-V hosts, but note well that the Hyper-V configuration version has to be version 8.0 or higher.

    Has this virtual machine in question been migrated from a previous Hyper-V version?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, September 17, 2019 8:04 AM
  • Sorry for the delayed response. 

    Yes, the VM was previously on Hyper-V version 5.0 and was upgraded to 8.0

    Tuesday, September 24, 2019 4:39 PM
  • I’ve previously encountered some strange issues with older guest virtual machines that have been upgraded.

    Do you have the possibility to create a new virtual machine and attach the existing VHDX files to it?

    This would then require you to:

    • Remove the guest virtual machine from the protection group.
    • Modify the protection group and refresh the Hyper-V host.
    • Re-add the guest virtual machine back to the protection group.

    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, September 24, 2019 5:12 PM
  • I will consult with our solution architect about this option. 

    will update you if we end up rebuilding the VM. 

    Thanks

    Wednesday, September 25, 2019 3:26 PM