none
DPM Backup fails backing up Server 2016 NTFS RRS feed

  • Question

  • Hi,

    I am at a new place and have been asked to look into why DPM (new to me) is failing on two (at least) 2016 Server backup.

    The environment is a mixture of 2012r2 and 2016 Servers, however only the upgraded 2016 Server backups fails.

    ++++++++++++++++++++++++++++++++++++++++++++++++

    On DPM we get the following message

    The following alert became inactive at 02/10/2018 06:38:27.

     

    Status: Inactive

    Data source type:

    Data source: Non VSS Datasource Writer

    Computer: srv25.hgss.local

    Description: Last 1 recovery points not created.

    DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:  0x68F5B0C0).

     

    For more information, open DPM Administrator Console and review the alert details in the Monitoring task area.

    ++++++++++++++++++++++++++++++++++++++++

    On the 2016 Server application event logs there seem to 3 related events

    ERROR:

    The backup operation that started at '‎2018‎-‎10‎-‎02T04:30:36.207277600Z' has failed with following error code '0x8078015B' (Windows Backup encountered an error when accessing the remote shared folder. Please retry the operation after making sure that the remote shared folder is available and accessible.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

    ++++++++++++++++++++++

    Information:

    The VSS service is shutting down due to idle timeout.

    ++++++++++++++++++++++

    Information:

    The Block Level Backup Engine service has stopped.

    ++++++++++++++++++++++

    Information:

    The Block Level Backup Engine service has started.

    Any thoughts would be appreciated

    Rob


    • Edited by Berttie Wednesday, October 3, 2018 10:00 AM changed title
    Tuesday, October 2, 2018 2:09 PM

All replies

  • Hi Rob!

    Sorry if I misunderstood something, how does your DPM environment look like, DPM version and build?

    So you're trying to backup a Windows Server 2016 on ReFS?


    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, October 2, 2018 3:09 PM
  • Hi Leon,

    Yep … trying to backup Windows Server 2016 on ReFS to DPM Version: 5.1.378.0 build 1807

    Thanks again

    Tuesday, October 2, 2018 3:43 PM
  • I'm afraid DPM doesn't support non-NTFS volumes, protecting Hyper-V virtual machines stored on ReFS-based SoFS clusters is supported from DPM 2016.


    For more information, see:

    https://docs.microsoft.com/en-us/system-center/dpm/dpm-support-issues?view=sc-dpm-1807#some-data-types-arent-supported


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, October 2, 2018 4:11 PM
  • Hi Leon,

    Thanks for your reply … 

    Asking around and digging a bit more, I have a better understanding of the 

    Looking at last nights failures ...I think we did fall into any of the unsupported data types.

    Server 7 DPM server o/s NTFS - Storage Pool ReFS.

    Server 7 tried to backup the following: 

    Server 23 Vsphere VM  - NTFS - Error: 

    The backup operation that started at '‎2018‎-‎10‎-‎03T04:30:25.329245200Z' has failed with following error code '0x8078015B' (Windows Backup encountered an error when accessing the remote shared folder. Please retry the operation after making sure that the remote shared folder is available and accessible.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.


    Wednesday, October 3, 2018 9:58 AM
  • So a bit more of my thoughts.

    I am thinking of reinstalling the DPM server and making the storage pool NTFS ….but before I do this I was thinking of making the splitting the protection group effect into two .. maybe more and schedule different backup times.

    Rob

    Wednesday, October 3, 2018 10:04 AM
  • If you have your DPM 1807 on a Windows Server 2016 it will automatically use the Modern Backup Storage (MBS), this allows you to store the backups on ReFS volumes.

    More information here:
    https://docs.microsoft.com/en-us/system-center/dpm/add-storage?view=sc-dpm-1807

    What vSphere version are you currently running on the VM that failed?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, October 3, 2018 10:07 AM
  • Hi Leon,

    we are version 6.7 of vSphere... 

    Wednesday, October 3, 2018 11:54 AM
  • Ok, the issue may lie there as vSphere 6.5 is the highest version that is officially supported for DPM 1807.

    Last week I spoke with the System Center team and they said that they are working on supporting vSphere/ESXi/ESX 6.7 for the next DPM version 1901, which is ti be released in Q1 of 2019.



    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, October 3, 2018 11:59 AM
  • Hi again,

    Thanks for taking the time.

    So .. we are not using DPM to backup the VM via ESXi … if that makes sense … I guess we would not need 1901 in this case ? 

    Rob

    Wednesday, October 3, 2018 1:06 PM
  • Looking at last nights failures ...I think we did fall into any of the unsupported data types.

    Server 7 DPM server o/s NTFS - Storage Pool ReFS.

    Server 7 tried to backup the following: 

    Server 23 Vsphere VM  - NTFS - Error: 

    The backup operation that started at '‎2018‎-‎10‎-‎03T04:30:25.329245200Z' has failed with following error code '0x8078015B' (Windows Backup encountered an error when accessing the remote shared folder. Please retry the operation after making sure that the remote shared folder is available and accessible.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

    Okay so you mean that your vSphere VM that is a Hyper-V virtual machine was the one that failed?

    Sorry if I misunderstood anything earlier, just want to make thing clear:

    • What is your Hyper-V host version?
    • What type of disks are your Hyper-V VMs stored on, NTFS or ReFS?


    As I understood, you have a Windows Server 2016 with DPM 1807, the storage pool of the DPM server is on ReFS right?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, October 3, 2018 1:17 PM
  • hi Leon,

    Sorry for not being clear.

    There is no Hyper-V in this case.

    We have a physical server run DPM which backs up around 27 servers. Amongst these backups are server 23 and 24 which are VMs on ESXI host… these two servers are running Windows 2016 NTFS. These are failing ...and we have to run consistency check every day more or less.

    Hope this helps,

    Rob 

    Wednesday, October 3, 2018 3:29 PM
  • Ah ok!

    Is there anything special or different with server 23 and server 24 comparing to the rest of the VMs?

    Also what type of backup are you performing on these VMs?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Wednesday, October 3, 2018 3:35 PM
  • no that I can tell at the moment .. will diver deeper ..

    What I can say though .. is last night 23 and 24 went though Ok.. however 25 and 19 failed.

    The only servers that fail to be backed up are all 2016 Windows server standard. All previous version of Windows server (mainly 2012r2) are  fine.

    Rob 

    Thursday, October 4, 2018 9:15 AM
  • Okay, backups do fail occasionally from time to time and there can be many reasons to why it fails.

    If it does fail often though, then there's a problem somewhere and then we need to start to investigate why it's failing and if there's a pattern.

    For how long have these Windows Server 2016 backups been failing?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, October 4, 2018 9:26 AM
  • Its seems the backups have been failing ever since the 2016 servers were implemented.

    Looking at DPM … its seems for server 25, the folders backups went through OK but the systemstate and BMR recovery point creation failed. I will keep on eye on this to see if that is always the case.

    At this time there are no backups running but NET SHARE shows

    Share name   Resource                        Remark
    -------------------------------------------------------------------------------
    IPC$                                         Remote IPC
    E$           E:\                             Default share
    C$           C:\                             Default share
    ADMIN$       C:\Windows                      Remote Admin
    MTATempStore$
                 C:\Program Files\Microsoft System Center\DPM\DPM\Temp\MTA\.

    011e191b98b54394b18621b7a1e445ce
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\8b78696a-fccb-4609-a1e9-a524ef47fe5d\011e191b-98b5-4394-b186-21b7a1e445ce\Full\E-Vol
                                                 011e191b98b54394b18621b7a1e445ce
    01813e3b32a3478295bebf0ed11fb30b
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\2d6dc92d-192d-4c0c-a0f0-d74df9b826eb\01813e3b-32a3-4782-95be-bf0ed11fb30b\Full\E-Vol
                                                 01813e3b32a3478295bebf0ed11fb30b
    04d5ec0110f54f6689be4ce0db3eca8f
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\abe7a9bc-e246-46ba-b3d4-3e17a59806f9\04d5ec01-10f5-4f66-89be-4ce0db3eca8f\Full\C-Vol
                                                 04d5ec0110f54f6689be4ce0db3eca8f
    1747f38170ff4c50840e752c10bd9297
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\19b53641-58b3-4ee1-8fab-0bacb1d38e04\1747f381-70ff-4c50-840e-752c10bd9297\Full\C-Vol
                                                 1747f38170ff4c50840e752c10bd9297
    2b077ebf5b554170b7fc4f3979bad634
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\ac5d4e06-a564-4eaf-871b-84dce56ea05d\2b077ebf-5b55-4170-b7fc-4f3979bad634\Full\E-Vol
                                                 2b077ebf5b554170b7fc4f3979bad634
    2ce7197ad751418fa623643f2ee6ab7d
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\c1d53abc-8565-4638-b798-074390da17c7\2ce7197a-d751-418f-a623-643f2ee6ab7d\Full\C-Vol
                                                 2ce7197ad751418fa623643f2ee6ab7d
    3917724d9d68451f8cef272fb33c7d96
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\0a08b56c-1560-47db-bc3e-f61a4235f269\3917724d-9d68-451f-8cef-272fb33c7d96\Full\H-Vol
                                                 3917724d9d68451f8cef272fb33c7d96
    3a27d909e82c4664959c36d3dcca5195
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\ec2d50fa-3a41-42c2-ae0e-e330333f5d0d\3a27d909-e82c-4664-959c-36d3dcca5195\Full\C-Vol
                                                 3a27d909e82c4664959c36d3dcca5195
    5850d75615f046b3a6000cb71270bd58
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\ce3d79ed-0b5e-46f1-ae0e-6e533c13a1af\5850d756-15f0-46b3-a600-0cb71270bd58\Full\C-Vol
                                                 5850d75615f046b3a6000cb71270bd58
    5db35fa5f7f74e67b2360f5f50fdac91
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\3b76d536-0346-4351-b0ab-5536a990c62b\5db35fa5-f7f7-4e67-b236-0f5f50fdac91\Full\C-Vol
                                                 5db35fa5f7f74e67b2360f5f50fdac91
    5f3a172bb528410183cf140583537249
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\7115e4bf-d212-4591-8dde-db36feee0d4b\5f3a172b-b528-4101-83cf-140583537249\Full\C-Vol
                                                 5f3a172bb528410183cf140583537249
    6201a7d2cc9a463eb0886460818bb41e
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\b17ef457-42d9-4309-9912-74dd1bf1b6c1\6201a7d2-cc9a-463e-b088-6460818bb41e\Full\C-Vol
                                                 6201a7d2cc9a463eb0886460818bb41e
    9630de5ad9c24a8d85169d8af43f85a5
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\d1c909a6-a6c1-4318-8938-2ba5fb289a1d\9630de5a-d9c2-4a8d-8516-9d8af43f85a5\Full\C-Vol
                                                 9630de5ad9c24a8d85169d8af43f85a5
    9c9068bb43564a22b9a59008eb4c16b6
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\3adf7826-47c5-4fe4-82d5-3b551853f31f\9c9068bb-4356-4a22-b9a5-9008eb4c16b6\Full\E-Vol
                                                 9c9068bb43564a22b9a59008eb4c16b6
    b2352783b294468db45c5fb424a51eba
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\7d63564e-3ea7-40e2-9dde-39a77354af47\b2352783-b294-468d-b45c-5fb424a51eba\Full\C-Vol
                                                 b2352783b294468db45c5fb424a51eba
    c49b17511cf9440ca4d836e64c10a489
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\66b31bb9-5c85-4bbb-8c43-905f335e97a7\c49b1751-1cf9-440c-a4d8-36e64c10a489\Full\E-Vol
                                                 c49b17511cf9440ca4d836e64c10a489
    d36bb5be82554b31a5c51699cb84f034
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\065a4e68-25c4-4a13-90e1-91cc80d5ec99\d36bb5be-8255-4b31-a5c5-1699cb84f034\Full\C-Vol
                                                 d36bb5be82554b31a5c51699cb84f034
    d69ebb33be67438b9073f448891a2c43
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\319932de-2e4b-4cbc-a6f9-44bb48e65c02\d69ebb33-be67-438b-9073-f448891a2c43\Full\E-Vol
                                                 d69ebb33be67438b9073f448891a2c43
    d7b8fdd974174334897569d04e686a6f
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\1f47387b-c65f-4e89-ba39-57e3e700a748\d7b8fdd9-7417-4334-8975-69d04e686a6f\Full\C-Vol
                                                 d7b8fdd974174334897569d04e686a6f
    f057461b4378473b92841facf7b43d43
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\93055efa-33a1-48a5-94a1-6b3ef126cdca\f057461b-4378-473b-9284-1facf7b43d43\Full\C-Vol
                                                 f057461b4378473b92841facf7b43d43
    f2c8562476594c8fa33a6621beb58dcf
                 \\?\C:\Program Files\Microsoft System Center\DPM\DPM\Volumes\Replica\ee9de113-e0de-4107-9f07-66acbfd67164\f2c85624-7659-4c8f-a33a-6621beb58dcf\Full\C-Vol
                                                 f2c8562476594c8fa33a6621beb58dcf

    Thursday, October 4, 2018 9:54 AM
  • Hi

    So I understand the error from WSB is this: Windows Backup encountered an error when accessing the remote shared folder. Please retry the operation after making sure that the remote shared folder is available and accessible.
     (The System State / BMR will be done over SMB to a network share on the DPM.)

     In this case please add this registry key on the protected server and reboot:
    Location: HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters
    Value type: Dword
    Value name: SessTimeout
    Value: decimal 1200

    Side Note: Make sure you also have enough space on the replica to acomodate the System State / BMR.
     (By default we allocate only 30GB). 
    • Proposed as answer by Tome Lopes Sunday, October 21, 2018 11:29 AM
    Sunday, October 21, 2018 11:29 AM
  • hi Tome,

    thanks for taking the time to reply.

    All three of you suggestions have already been checked and or implemented .. but without success.

    Regards

    Rob

    Monday, October 22, 2018 8:13 AM
  • Hi Rob. Thanks for the feedback. So the Reg Key I shared didn't worked…

    Can you do this on the protected server and then reboot:

    Delete those names from registry on the Protected Server:
    Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
    Name: Lmannounce
    Name: Size

    Set to zero those names on the Protected Server:
    Name: Enablesecuritysignature
    Name: Requiresecuritysignature

    Thanks.

    Friday, October 26, 2018 8:28 AM