none
Unable to backup VM's stored on a CSV RRS feed

  • Question

  • I have a 2 node hyper-v cluster with a csv shared between both nodes. I just set up dpm 2016 and installed the agent on both hosts and added the vm's to a protection group, but when I try to take the initial backup it looks like it is about to but after a few seconds it just goes back to "replica is inconsistent". I looked in the event viewer, but the only logs i can see that pertain to this is event ID 33223, which just says it failed and nothing else. 

    Kind of lost, still pretty new to hyper-v clusters as well as DPM so i could use some guidance, thanks!

    Wednesday, February 13, 2019 3:17 PM

All replies

  • Hello!

    What operating system version are your Hyper-V hosts running?

    Also what integration services version are your Hyper-V virtual machines running?

    Could you go into the Monitoring tab in the DPM console and provide us with the exact error(s)?


    Could you also run the following command on your Hyper-V nodes and show us the result:

    vssadmin list writers

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, February 13, 2019 3:56 PM
  • Both nodes are running server 2016 datacenter, and vm's are running 8.0. I backed up these same vm's when i just had a single node with dpm without issue.

    and here's there error code dpm gives me:

    The replica of Microsoft Hyper-V [VM] on [VM].[hvcluster].domain is inconsistent with the protected data source. All protection activities for data source will fail until the replica is synchronized with consistency check. You can recover data from existing recovery points, but new recovery points cannot be created until the replica is consistent. 
    
    For SharePoint farm, recovery points will continue getting created with the databases that are consistent. To backup inconsistent databases, run a consistency check on the farm. (ID 3106)
    
    An unexpected error occurred while the job was running. (ID 104 Details: Internal error code: 0x80990E54)

    All vss writers are showing as stable with no errors

    Writer name: 'Task Scheduler Writer'
       Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'VSS Metadata Store Writer'
       Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Performance Counters Writer'
       Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'System Writer'
       Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
       Writer Instance Id: {9e9cf53b-46a6-4087-b2e7-66bf8e1708d0}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Cluster Shared Volume VSS Writer'
       Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
       Writer Instance Id: {a0dafb12-6263-4d26-96ec-dae936ff1882}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Microsoft Hyper-V VSS Writer'
       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
       Writer Instance Id: {ee3545eb-bbd2-4424-8cd2-333d540d5eeb}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'ASR Writer'
       Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Writer Instance Id: {4ba6aa8f-355e-41e3-a03b-4bad6224c497}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Shadow Copy Optimization Writer'
       Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Writer Instance Id: {7f95ed69-10df-4a8e-b803-59687da2c44e}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Registry Writer'
       Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Writer Instance Id: {c4df0069-9215-4217-93d9-72b565163778}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Storage Replica Writer'
       Writer Id: {7e82ba6f-d7ac-4c8e-b66a-e319a0d3e6dc}
       Writer Instance Id: {50f908d0-b414-4708-8c03-6215369f11ca}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'COM+ REGDB Writer'
       Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Writer Instance Id: {b8515729-9f21-465a-9d0a-919b1b7871de}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'WMI Writer'
       Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Writer Instance Id: {c31d4684-2f23-4f7a-9ed2-8290591036ea}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Cluster Database'
       Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
       Writer Instance Id: {881bff89-ef5b-46d3-bf47-3ce41a41be9c}
       State: [1] Stable
       Last error: No error


    • Edited by brad_sbktx Wednesday, February 13, 2019 7:57 PM
    Wednesday, February 13, 2019 7:55 PM
  • Thanks for the information, could you still provide the vssadmin list writers result ?

    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, February 13, 2019 7:59 PM
  • Yep! i edited my reply with the output
    C:\Windows\system32>vssadmin list writers
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2013 Microsoft Corp.
    
    Writer name: 'Task Scheduler Writer'
       Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
       Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'VSS Metadata Store Writer'
       Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
       Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Performance Counters Writer'
       Writer Id: {0bada1de-01a9-4625-8278-69e735f39dd2}
       Writer Instance Id: {f0086dda-9efc-47c5-8eb6-a944c3d09381}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'System Writer'
       Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
       Writer Instance Id: {9e9cf53b-46a6-4087-b2e7-66bf8e1708d0}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Cluster Shared Volume VSS Writer'
       Writer Id: {1072ae1c-e5a7-4ea1-9e4a-6f7964656570}
       Writer Instance Id: {a0dafb12-6263-4d26-96ec-dae936ff1882}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Microsoft Hyper-V VSS Writer'
       Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
       Writer Instance Id: {ee3545eb-bbd2-4424-8cd2-333d540d5eeb}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'ASR Writer'
       Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
       Writer Instance Id: {4ba6aa8f-355e-41e3-a03b-4bad6224c497}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Shadow Copy Optimization Writer'
       Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
       Writer Instance Id: {7f95ed69-10df-4a8e-b803-59687da2c44e}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Registry Writer'
       Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
       Writer Instance Id: {c4df0069-9215-4217-93d9-72b565163778}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Storage Replica Writer'
       Writer Id: {7e82ba6f-d7ac-4c8e-b66a-e319a0d3e6dc}
       Writer Instance Id: {50f908d0-b414-4708-8c03-6215369f11ca}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'COM+ REGDB Writer'
       Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
       Writer Instance Id: {b8515729-9f21-465a-9d0a-919b1b7871de}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'WMI Writer'
       Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
       Writer Instance Id: {c31d4684-2f23-4f7a-9ed2-8290591036ea}
       State: [1] Stable
       Last error: No error
    
    Writer name: 'Cluster Database'
       Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
       Writer Instance Id: {881bff89-ef5b-46d3-bf47-3ce41a41be9c}
       State: [1] Stable
       Last error: No error

    Wednesday, February 13, 2019 8:15 PM
  • Thanks, indeed it looks good on the vss writers, could you provide the following output as well?

    vssadmin list providers


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, February 13, 2019 9:20 PM
  • Here you go

    C:\Windows\system32>vssadmin list providers
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2013 Microsoft Corp.
    
    Provider name: 'Microsoft CSV Shadow Copy Helper Provider'
       Provider type: Software
       Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d}
       Version: 1.0.0.1
    
    Provider name: 'Microsoft CSV Shadow Copy Provider'
       Provider type: Software
       Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
       Version: 1.0.0.1
    
    Provider name: 'VSS Null Provider'
       Provider type: Software
       Provider Id: {8202aeda-45bd-48c4-b38b-ea1b7017aec3}
       Version: 5.0.158.0
    
    Provider name: 'Microsoft File Share Shadow Copy provider'
       Provider type: Fileshare
       Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
       Version: 1.0.0.1
    
    Provider name: 'Microsoft Software Shadow Copy provider 1.0'
       Provider type: System
       Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
       Version: 1.0.0.7
    
    Provider name: 'VSS Null Provider'
       Provider type: Fileshare
       Provider Id: {f4a69dd4-f712-40e3-a6b3-faeff03cb2b8}
       Version: 5.0.158.0

    Wednesday, February 13, 2019 9:31 PM
  • Thanks! That looks as it should as well.

    Could you try the following:

    1. Uncheck the Backup (volume checkpoint) setting of a virtual machine in the Hyper-V cluster.

    2. Perform a backup of this Hyper-V virtual machine.

    3. Check how it goes.

    4. If all goes well, check the Backup (volume checkpoint) setting once again.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, February 13, 2019 9:52 PM
  • I tried that and it fails doing that as well. When i kick off a sync in DPM and watch it in hyper-v manager I can see that DPM starts to make a checkpoint but after a few seconds it goes away and the job does in dpm as well. I was able to backup one of my VM's however. From what i can tell it has similar configuration to the others that fail. 
    Wednesday, February 13, 2019 10:08 PM
  • Ok that's an very odd behavior.

    You could start investigating the DPMRACurr.err log on the Hyper-V cluster nodes
    (Default location: C:\Program Files\Microsoft Data Protection Manager 2016\DPM\Temp).


    Also the MSDPM.errlog on the DPM server might give some more clues.
    (Default location: C:\Program Files\Microsoft System Center 2016\DPM\DPM\Temp)


    Blog: https://thesystemcenterblog.com LinkedIn:

    • Proposed as answer by Leon Laude Thursday, February 14, 2019 1:42 AM
    Wednesday, February 13, 2019 10:18 PM
  • Thanks for the tip, I think I figured it out!! I saw in the logs this line-

    WARNING	Failed: Hr: = [0x80990e54] GetParentVhdPath failed. ssDiffVhdPath: D:\installation media\Windows OS's\en_windows_server_2016_updated_feb_2018_x64_dvd_11636692.iso

    So I removed the windows installation iso from the virtual machine is started backing up!! That was my old drive path, which doesn't exist anymore, so that seemed to have messed DPM up when creating the snapshots. So it seems that's all it was, thanks for all the help!! Sometimes it's just the simplest of things


    • Edited by brad_sbktx Thursday, February 14, 2019 1:20 AM
    • Proposed as answer by Leon Laude Thursday, February 14, 2019 1:42 AM
    Thursday, February 14, 2019 1:19 AM
  • Great to hear! Yeah I never come to think of the mounted ISOs... Glad it worked itself out!

    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, February 14, 2019 1:42 AM