none
DPM 2019 - RU1 - Unable to Backup Replica VM's RRS feed

  • Question

  • Hi,

    I have a DPM server which is running on DPM 2019 RU1 is also applied.

    I have a Hyperv Host in which i have 6 VM's . 3 VM's are running (Primary VM's hosted on the server) , Other 3 VM's are Replica VM's (These are getting replicated from a different Hyperv hosts and currently in Off state only). Hyperv configuration version is 9.0.

    So we have configured Hyper-v based backups on the 3 Replica VM's in DPM Servers. However, 2 VM's are getting backed up without any issues and there recovery points are up to date. However there is 1 VM out of 3 replica VM's which is Having issues.

    Error Says "An unexpected error occurred while the job was running . ID 104 details: The filename, directory name , or volume label syntax is correct 0x8007007B" 

    I have searched a blog and found this already.

    http://answers.flyppdevportal.com/MVC/Post/Thread/0e7eb3a3-57f6-4f1c-bfb8-d27ccc230c54?category=windowsazureonlinebackup

    Here i have seen most of the points covered in solution are already in place , Except CSV part since the VM's are not part of Cluster. 

    I have ensured VSS writers are running fine on the server. Checked if the Server has proper guest services enabled. DPM is up to date with all patches required etc. Still dont see what missing.

    Can someone please help me on this?


    Sai

    Friday, June 19, 2020 9:14 AM

Answers

  • Okay - Now since we have isolated the VM backup part, we need to get DPM verbose log to understand the failure from DPM side.

    There are few questions before we jump into the verbose logs.

    1. Have you tried to take the backup of the Primary VM using existing DPM server in the past? if yes, did it work and if it is still under active/inactive protection? if not, try to take the Primary VM backup using same DPM or any other DPM and see if it works? I suspect this is specific to the VHD issue.

    2. Is this your only Primary DPM server or if it has a relation with any other DPM server as a Primary/Secondary?

    3. In the previous DPMRA log, I saw the path as below:

    D:\VM\%ServerName%\Hyper-V Replica\Virtual hard disks\%VirtualMachineID%\

    However in the above WSB logs, I am see a bit different path:

    D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual Machines\

    I believe you may have changed the name manually for obvious reasons, but please compare the path from both the logs. It should be identical from where it is picking the VM objects and files.

    If the path looks fine, please enable verbose logs for the DPMRA on Protected Server and on the DPM server (MSDPM and DPMRA) and share it with me. Along with the logs, share the System time zone, start job time and failed job time so that I can precisely look into the logs. Verbose logs are going to be huge so I would need as much as data possible.

    4. I still think you should try doing a storage migration on the same volume. You can create a new folder in D:\ as NewFolder and migrate the VM data into it. Once done, delete the datasource from DPM, refresh the host while enumerating and protect it again. See if this helps.

    5. If above doesn't work, Please provide me the diskshadow output from the Protected server and the requested verbose log. Run below command on the PS to get the same.

    DiskShadow /L c:\temp\diskshadow.txt

    list writer detailed



    • Edited by Aayoosh Moitro Wednesday, June 24, 2020 9:19 AM
    • Marked as answer by Sai_phani Monday, June 29, 2020 7:18 AM
    Wednesday, June 24, 2020 8:09 AM
  • HI Aayoosh

    Thanks a lot for the ideas .

    I have finally sorted the issue using below.

    Below are the things done: 

    1. Took the primary VM backup on a different DPM server by disabling the replication for sometime and it also failed with the same error "An unexpected error occurred while the job was running . ID 104 details: The filename, directory name , or volume label syntax is correct 0x8007007B"

    Then thought of using the idea mentioned above .

    I have created a new VM on the server where the VM is replicated (Replica server) and then attached the VHDX files of the server that is having issues and then initiated the backup on a new DPM server. To my surprise it worked without any issues.
    Then i later gave a thought of trying it by enabling the replication for the same VM and try taking the backup just to ensure the replication related writers are not blocking the backup by any chance and indeed that didnt stop anything and backup worked successfully.

    Now i have enabled the backup for that test replicated VM on the replica server and i am sure that will be completed successfully too.

    Thanks a lot for helping me out in generating new ideas and sorting out this issue in a better way.


    Sai

    • Marked as answer by Sai_phani Monday, June 29, 2020 7:18 AM
    Monday, June 29, 2020 7:18 AM

All replies

  • Hi,

    Does this one virtual machine have different disks (VHD/VHDX)? Are there any mounted ISOs?

    I would start by checking the DPM logs to see if we can determine what file or folder is causing this error.

    DPM server log location:

    • %ProgramFiles%\Microsoft System Center\DPM\DPM\Temp\MSDPMCurr.errlog

    DPM agent log location:

    • C:\Program Files\Microsoft Data Protection Manager\DPM\Temp\DPMRACurr.errlog

    To make things easier, first move all the existing logs away from the Temp folder, this allows DPM to create new logs, then reproduce the error so that DPM writes it to the DPM log, this is to avoid excess logs.

    Then upload the DPM logs to a Microsoft OneDrive / Google Drive and share the link here (Please do not post the logs here).

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Friday, June 19, 2020 9:29 AM
  • Hello Sai,

    Since 2 of the VMs are getting backed up and 1vm is failing, it definitely means there is something wrong with that particular VM itself.

    Error:0x8007007B converts to below error message:

    C:\Temp\Err>Err.exe 0x8007007B
    # for hex 0x8007007b / decimal -2147024773 :
      STIERR_INVALID_DEVICE_NAME                                    stierr.h
    # 1 matches found for "0x8007007B"

    To me it seems like there is some issue with the VM configuration or device connected to it. Try to inspect the VHDs connected to it from Hyper-V manager console.


    You can also take VM backup using WSB (windows server backup) and see if it works. You can install it on Hyper-V host and it doesn't require a reboot. Please note that WSB can not take backup for Vms which are placed on CSV or in other words "highly available."

    DPMRA logs on the Hyper-V host would be more helpful.

    Friday, June 19, 2020 12:27 PM
  • let me check both the updates @Leon and @aayoosh

    Will get back in sometime


    Sai

    Friday, June 19, 2020 1:19 PM
  • @Leon

    Attaching the logs here.

    I believe there is some issue with the data that is replicated to the VM. Hence could be the issue?

    Please find the link here for the logs.

    https://drive.google.com/file/d/1jyhh_c4p9-c-DHS5VZpVFQXCWCRLnknJ/view?usp=sharing


    Sai

    Monday, June 22, 2020 10:29 AM
  • The log indicates an error code 998 which simply is "The operation failed because of a protection agent failure.".

    What is the Hyper-V host operating system version?

    Does the affected Hyper-V virtual machine have any special characters in the name that may be invalid?

    What generation is the affected virtual machine running, generation 1 or 2?

    Have you tried simply removing the affected VM from the protection group (try with retain data first), then re-add it. If this doesn't work, try removing the affected VM from the protection group and remove the data, then re-add it to the protection group.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, June 22, 2020 2:43 PM
  • Hey Leon

    Hyperv Host is running on Windows Server 2019

    VM is running on Gen 2 

    It does have a Special character, But as said , the other 2 replica VM's which are getting backed up are having the backups happening fine and they do consist of same special characters in their names as well.

    I have tried by removing and readding it. That didnt work.

    I am trying a different method by removing the replication and re adding the replication to the VM and then try configuring the backup again. it will take me around 24 hours atleast for the replication to happen and get back with an update. I will get back to you by tomorrow.

    Thank you so much for responding so quick


    Sai

    Monday, June 22, 2020 3:15 PM
  • If the removal of replication doesn't help, try exporting the VM from the source server first, then remove it from Hyper-V, let DPM refresh the inventory so that it doesn't see the VM.

    Then import the VM, enable replication and try adding the replicated VM to the protection group and see how it goes.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, June 22, 2020 3:18 PM
  • Hi Sai,

    Where have you placed the VHD of the replica VM? Could you please share the full path.

    Also, before you start from the scratch, try doing a storage migration (including VM config file and snapshots etc) for all the connected VHDs to a new Volume (if available) and then stop the protection from DPM and re-add it back. Don't forget to refresh the Hyper-V server while enumerating and then update.

    Also it would be helping if you can upload the DPMRA logs from the Hyper-V server. I guess you have uploaded the MSDPM/DPMRA from DPM server, at least that's what I see.


    Monday, June 22, 2020 8:08 PM
  • Hi Leon and Aayoosh

    I have tried all the ways mentioned by both of you and unfortunately it didnt work.

    Below are the things i have done.

    1. Removing the replication completely and re adding the replication back to the server. post the initial replication is done, i have refreshed the Agent in DPM console and also restarted the DPMRA Service on the client end as well. Then reconfigured the Protection group by adding the RCT\VM back . It failed with the same error after transferring around 7 GB of data. 
    Note : We have the replication interval set as 15 mins, which means replication will happen / synchronize to the replica server (where we have DPM agent installed) for every 15 mins with the replicated data that has gained since the VM is running on that server (This server doesn't have DPM configured)

    2. @Aayoosh, i cannot do the storage migration, because i dont have any other external storage nor a Volume available to get this done. For the path you have asked, here are the details.

    D:\VM\%ServerName%\Hyper-V Replica\Virtual hard disks\%Virtual Machine ID% . Please note i have the same structure for the other 2 replica VM's as well Where the Backup for them is happening absolutely fine without any issues.

    Below are the logs that you were looking for.

    https://drive.google.com/file/d/1aIKfNV3xdNMG4OvtfAgqoMARVQCKYbrh/view?usp=sharing

    Please Suggest if there are any other options that i could try.


    Sai

    Tuesday, June 23, 2020 10:01 AM
  • Hi Sai,

    I looked at the DPMRA logs and I am seeing the below error:

    497C 3D64 06/23 09:09:40.390 18 dsmsendersubtaskbase.cpp(157) [00000285F2FE85E0] 93ABF9B7-E633-4E05-AA97-EAA1984A4228 WARNING Failed: Hr: = [0x80072746] : Encountered Failure: : lVal : OnSessionClosed(dwNumberOfBytes, pAgentOvl, dwError)
    497C 3D64 06/23 09:09:40.390 18 dsmsendersubtaskbase.cpp(278) [00000285F2FE85E0] 93ABF9B7-E633-4E05-AA97-EAA1984A4228 WARNING Failed: Hr: = [0x80072746] : Encountered Failure: : lVal : ProcessWaitCompletion(dwNumberOfBytes, pAgentOvl, dwError)

    Error seems related to some network disconnect.


    C:\Temp\Err>Err.exe 0x80072746
    # The data value for one or more columns overflowed the type
    # used by the provider.
      WSAECONNRESET                                                 winerror.h
    # An existing connection was forcibly closed by the remote
    # host.
      WSAECONNRESET                                                 winsock2.h
    # 3 matches found for "0x80072746"


    This error could be misleading because your other VM backup is working fine which are running on the same host. Hence it could be a possibility of some corruption on the VM configuration file or Av is blocking the file access at the filter level.

    Lets isolate the issue first, install WSB on the Host, Take the same VM backup using WSB and save it on the DPM server by creating a share on the DPM server in advance and stream the backup to it and see if it completes.

    If above step works, then there is some thing wrong the way DPM is pulling/reading the VM's data and for that we would need verbose logs for DPMRA from Hyper-V host and MSDPM/DPMRA from DPM server.

    Use this link to enable verbose logs on the PS and on the DPM server. Make sure you disable the verbose logging after collecting the logs followed by a service restart.



    Tuesday, June 23, 2020 11:10 AM
  • Sure Aayoosh, I am taking the WSB backup for that VM alone. Its in progress. Will get back to you once its done or if any errors. Thank you!!

    Sai

    Tuesday, June 23, 2020 5:07 PM
  • Please make sure you take the backup on the share created on the DPM server only. So that we can check the network connectivity as well. 
    Tuesday, June 23, 2020 5:19 PM
  • hi Aayoosh

    Yes, i have done the same thing . I have mapped the DPM Server Drive as Network drive for the Host and then did a WSB backup and it is successful. Havent seen any issues as well.

    Below are the logs from WSB console.

    ```

    Backed up D:\
    Backed up D:\VM\
    Backed up D:\VM\%VirtualMachine%\
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Snapshots\
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Snapshots\%SnapshotID%.vmcx
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Snapshots\%SnapshotID%.VMRS
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Snapshots\%SnapshotID%.vmgs
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Snapshots\%SnapshotID%\
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual hard disks\
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual hard disks\%VirtualMachineID%\
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual hard disks\%VirtualMachineID%\Data.vhdx
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual hard disks\%VirtualMachineID%\Data_%SnapshotID%.avhdx
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual hard disks\%VirtualMachineID%\OS.vhdx
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual hard disks\%VirtualMachineID%\OS_%SnapshotID%.avhdx
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual Machines\
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual Machines\%VirtualMachineID%.VMRS
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual Machines\%VirtualMachineID%.vmcx
    Backed up D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual Machines\%VirtualMachineID%.vmgs
    Application backup
    Writer Id: {66841CD4-6DED-4F4B-8F17-FD23F8DDC3DE}
       Component: %VirtualMachineID%
       Caption     : Offline\%VirtualMachine%

    ```

    Any further suggestions on this?


    Sai

    Wednesday, June 24, 2020 7:16 AM
  • Okay - Now since we have isolated the VM backup part, we need to get DPM verbose log to understand the failure from DPM side.

    There are few questions before we jump into the verbose logs.

    1. Have you tried to take the backup of the Primary VM using existing DPM server in the past? if yes, did it work and if it is still under active/inactive protection? if not, try to take the Primary VM backup using same DPM or any other DPM and see if it works? I suspect this is specific to the VHD issue.

    2. Is this your only Primary DPM server or if it has a relation with any other DPM server as a Primary/Secondary?

    3. In the previous DPMRA log, I saw the path as below:

    D:\VM\%ServerName%\Hyper-V Replica\Virtual hard disks\%VirtualMachineID%\

    However in the above WSB logs, I am see a bit different path:

    D:\VM\%VirtualMachine%\Hyper-V Replica\Virtual Machines\

    I believe you may have changed the name manually for obvious reasons, but please compare the path from both the logs. It should be identical from where it is picking the VM objects and files.

    If the path looks fine, please enable verbose logs for the DPMRA on Protected Server and on the DPM server (MSDPM and DPMRA) and share it with me. Along with the logs, share the System time zone, start job time and failed job time so that I can precisely look into the logs. Verbose logs are going to be huge so I would need as much as data possible.

    4. I still think you should try doing a storage migration on the same volume. You can create a new folder in D:\ as NewFolder and migrate the VM data into it. Once done, delete the datasource from DPM, refresh the host while enumerating and protect it again. See if this helps.

    5. If above doesn't work, Please provide me the diskshadow output from the Protected server and the requested verbose log. Run below command on the PS to get the same.

    DiskShadow /L c:\temp\diskshadow.txt

    list writer detailed



    • Edited by Aayoosh Moitro Wednesday, June 24, 2020 9:19 AM
    • Marked as answer by Sai_phani Monday, June 29, 2020 7:18 AM
    Wednesday, June 24, 2020 8:09 AM
  • HI Aayoosh

    Thanks a lot for the ideas .

    I have finally sorted the issue using below.

    Below are the things done: 

    1. Took the primary VM backup on a different DPM server by disabling the replication for sometime and it also failed with the same error "An unexpected error occurred while the job was running . ID 104 details: The filename, directory name , or volume label syntax is correct 0x8007007B"

    Then thought of using the idea mentioned above .

    I have created a new VM on the server where the VM is replicated (Replica server) and then attached the VHDX files of the server that is having issues and then initiated the backup on a new DPM server. To my surprise it worked without any issues.
    Then i later gave a thought of trying it by enabling the replication for the same VM and try taking the backup just to ensure the replication related writers are not blocking the backup by any chance and indeed that didnt stop anything and backup worked successfully.

    Now i have enabled the backup for that test replicated VM on the replica server and i am sure that will be completed successfully too.

    Thanks a lot for helping me out in generating new ideas and sorting out this issue in a better way.


    Sai

    • Marked as answer by Sai_phani Monday, June 29, 2020 7:18 AM
    Monday, June 29, 2020 7:18 AM
  • You're welcome Sai.

    Monday, June 29, 2020 7:39 AM