none
DPM 2016 & BMR/Systemstate protection

    Question

  • Hi,

    first: sorry for my bad english.

    After a new Installation and connect all agents to a DPM 2016 Server, some server cant backup the systemstate

    DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:  0x6BAE1C0). (ID 30229 Details: Internal error code: 0x80990ED0)

    DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:  0x3F93650). (ID 30229 Details: Internal error code: 0x80990ED0)

    DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:  0x6BAE1C0). (ID 30229 Details: Internal error code: 0x80990ED0)

    DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:  0x3F93E20). (ID 30229 Details: Internal error code: 0x80990ED0)

    ... and so one. I found this solutions, and can excluded it:

    • WSB is enabled as role
    • reinstall WSB isn#s a solution
    • systemstate backup to a volume with local WSB is possible
    • Crypotographic service is running
    • vssadmin list writers found no error

      here is part from the DPM Logfile in the relevant peroiod:

    0DE0 43C0 12/07 09:52:07.062 22 ctimermgr.cpp(482) ACTIVITY CTimerMgr::TimerThread
    0DE0 3AE4 12/07 09:52:07.062 03 timer.cpp(490) ACTIVITY CTimerMgr::TimerThread
    0DE0 43C0 12/07 09:52:07.062 22 ctimermgr.cpp(495) [0000024F06628CB0] ACTIVITY CTimerMgr::TimerThreadInternal
    0DE0 3AE4 12/07 09:52:07.062 03 timer.cpp(503) [0000024F06628C40] ACTIVITY CTimerMgr::TimerThreadInternal
    0DE0 268C 12/07 09:52:07.062 03 runtime.cpp(781) [0000024F066A0FF0] ACTIVITY InitializeSecurity found 3 authorized entities
    0DE0 268C 12/07 09:52:07.062 03 runtime.cpp(840) [0000024F066A0FF0] ACTIVITY InitializeSecurity added COM perm for DPMRADCOMTrustedMachines
    0DE0 268C 12/07 09:52:07.078 03 runtime.cpp(840) [0000024F066A0FF0] ACTIVITY InitializeSecurity added COM perm for DPMRATrustedDPMRAs
    0DE0 268C 12/07 09:52:07.078 03 runtime.cpp(847) [0000024F066A0FF0] ACTIVITY InitializeSecurity added DM perm for DPMRADmTrustedMachines
    0DE0 268C 12/07 09:52:07.078 03 runtime.cpp(95) NORMAL Found 2 writer account(s) under VSS access control.
    0DE0 268C 12/07 09:52:07.078 03 runtime.cpp(163) NORMAL Added writer account [0] 'NT Authority\NetworkService' to the DACL.
    0DE0 268C 12/07 09:52:07.078 03 runtime.cpp(163) NORMAL Added writer account [1] 'NT SERVICE\SQLWriter' to the DACL.
    0DE0 268C 12/07 09:52:07.078 22 genericthreadpool.cpp(824) [0000024F066A0FF0] NORMAL Hr: = [0x80070002] CGenericThreadPool::m_dwMaximumNumberOfThreads[20]
    0DE0 268C 12/07 09:52:07.093 20 commengine.cpp(345) [0000024F066A26D0] ACTIVITY Hr: = [0x80070002] ConnectionNoActivityTimeout: 300
    0DE0 268C 12/07 09:52:07.093 20 commengine.cpp(359) [0000024F066A26D0] ACTIVITY Hr: = [0x80070002] m_dwConnectionNoActivityTimeoutForSlowReceiver: 3600
    0DE0 34FC 12/07 09:52:07.093 03 timer.cpp(490) ACTIVITY CTimerMgr::TimerThread
    0DE0 34FC 12/07 09:52:07.093 03 timer.cpp(503) [0000024F07B7AFE8] ACTIVITY CTimerMgr::TimerThreadInternal
    0DE0 268C 12/07 09:52:07.093 29 dpmra.cpp(154) [0000024F065F4B80] NORMAL CDPMRA::Initialize [0000024F065F4B80]
    0DE0 268C 12/07 09:52:07.093 29 dpmra.cpp(610) NORMAL CDPMRA: Initialize globals
    0DE0 268C 12/07 09:52:07.125 29 dpmra.cpp(597) ACTIVITY CDPMRA::AddSchemasToSchemaCacheForMTA => Initialize loaded schemas
    0DE0 268C 12/07 09:52:07.140 29 dpmra.cpp(172) [0000024F065F4B80] NORMAL CDPMRA::Initialize loaded schemas 
    0DE0 268C 12/07 09:52:07.140 22 genericthreadpool.cpp(75) [0000024F07B80EE0] NORMAL CGenericThreadPool: constructor
    0DE0 268C 12/07 09:52:07.140 22 genericthreadpool.cpp(824) [0000024F07B80EE0] NORMAL Hr: = [0x80070002] CGenericThreadPool::m_dwMaximumNumberOfThreads[256]
    0DE0 268C 12/07 09:52:07.140 29 dpmra.cpp(254) [0000024F065F4B80] NORMAL CreateDefaultSubTask [0000024F065F4B80]
    0DE0 268C 12/07 09:52:07.140 03 cbackupsubtasklimits.cpp(157) [0000024F06698580] NORMAL Resource Limit Config File: C:\Program Files\Microsoft System Center 2016\DPM\DPM\bin\DsResourceLimits.xml
    0DE0 268C 12/07 09:52:07.156 03 cbackupsubtasklimits.cpp(163) [0000024F06698580] NORMAL Loaded DS Resource Limits
    0DE0 268C 12/07 09:52:07.156 29 mtamethrottlehelper.cpp(112) [0000024F07BC7400] NORMAL Hr: = [0x80070002] Number of encryption threads = [2]
    0DE0 268C 12/07 09:52:07.156 29 dpmra.cpp(182) [0000024F065F4B80] NORMAL Temporary storage path reg key found
    0DE0 268C 12/07 09:52:07.156 29 dpmra.cpp(198) [0000024F065F4B80] NORMAL MTA Temp location Path = [C:\Program Files\Microsoft System Center 2016\DPM\DPM\temp\MTA\]
    0DE0 268C 12/07 09:52:07.156 09 nativeserviceproxy.cpp(200) NORMAL CoreServicesProxy::CreateCoreServicesObject(localhost)
    0DE0 268C 12/07 09:52:07.156 03 service.cpp(298) [00000092411EF910] ACTIVITY CService::AnnounceServiceStatus
    0DE0 70D0 12/07 09:52:07.156 03 cworkitem.cpp(429) [0000024F06E526A8] 90A82237-18F8-4347-AF9F-68245AE1B3F7 ACTIVITY Idle Timer created with timeout = 390000
    0DE0 7A68 12/07 09:52:45.519 03 cworkitem.cpp(23) [0000024F06E52630] 90A82237-18F8-4347-AF9F-68245AE1B3F7 ACTIVITY WorkItem stopping
    0DE0 7BD4 12/07 09:52:47.738 03 service.cpp(147) [00000092411EF910] ACTIVITY CService::ServiceHandlerExInternal

    Does anyone else hase an idea?

    Wednesday, December 7, 2016 9:57 AM

Answers

  • This has been fixed with DPM 2016 UR2.

    In order to make it work i had to remove the System State backup for the affected servers and re-protect the server, this time including the BMR protection. Just adding the BMR didn't work, i got an error that the storage has to be mounted. The replica creation took a while, but eventually finished.

    Wednesday, February 8, 2017 2:21 PM

All replies

  • I don't have a direct solution but you should be aware of this.

    • If you do a System State only the DPM Client will search for a disk which has the most disk space available. It will then create a local System State backup to that disk. After it is created it is copied to the DPM Server. The local System State backup is left behind. This is done to reservate the disk space. But you can delete it any time you want.
    • If you do a BMR (with System State) the DPM Client will backup directly to the DPM Server.

    Now, of course as you know you need to have WSB (Windows Server Backup) installed. But I have seen cases where 1 needed to remove the backup (and data) from the Protection Group when I first started with a System State only. And 2 you need to remove the cached copy of the backup on the local DPM Client.

    Probably this doesn't solve your issue, but just that you know and check to make sure.


    Boudewijn Plomp | Conclusion FIT

    Please remember, if you see a post that helped you please click "Vote as Helpful", and if it answered your question, please click "Mark as Answer". This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, December 7, 2016 1:22 PM
  • Hi Boudewijn,

    thank you! how do you mean "And 2 you need to remove the cached copy of the backup on the local DPM Client." ? i have sometimes remove and recreate the MBR in the protection group, without success.

    Wednesday, December 7, 2016 3:41 PM
  • Hi Boudewijn,

    thank you! how do you mean "And 2 you need to remove the cached copy of the backup on the local DPM Client." ? i have sometimes remove and recreate the MBR in the protection group, without success.

    Somewhere on a root of a disk you see a (hidden) directory named like "C:\WindowsImageBackup". You can safely delete it. It is only created when you create a System State backup only. When the directory does not exist it is re-created.

    If you remove the BMR backup from the System State, did you remove the data as well? If you didn't then when you re-add the BMR backup to the Protection Group it uses the old backup data again.

    But I have the feeling you might have a different problem. Please check the following link:

    Troubleshooting Data Protection Manager System State and Bare Metal Backup
    https://blogs.technet.microsoft.com/dpm/2011/10/31/troubleshooting-data-protection-manager-system-state-and-bare-metal-backup


    Boudewijn Plomp | Conclusion FIT

    Please remember, if you see a post that helped you please click "Vote as Helpful", and if it answered your question, please click "Mark as Answer". This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, December 7, 2016 7:04 PM
  • Hi,

    yes, i also remove the data. can't find a directory like "c:\WIndowsImageBackup".

    And, only all my 3  2008R2 Server are affected,  all 2012 and 2012R2 Systems backups normaly.


    Thursday, December 8, 2016 8:58 AM
  • Just wondering, did you manage to get this fixed? I've been searching the dark corners of the internet for a solution for this but so far, no luck..

    Every 2008 BMR backup fails (I have not tried 2012 machines).

    WSB gives the error "One of the backup files could not be created" and DPM gives the errors that you've already provided.

    I did notice a difference between 2012 and 2016 that if I browse to the shares of our DPM 2012 server I get a load of shares with GUIDs that I can actually access. If I try the same on our DPM 2016 server I get the same shares but I cannot access them.

    This might just be a permission issue..


    • Edited by M. Possamai Tuesday, January 24, 2017 11:34 AM
    Tuesday, January 24, 2017 11:33 AM
  • There is a thread about that somewhere here in one of the DPM forums. I can't find it atm. This is a known issue and should hopefully be fixed in the next rollup.
    Tuesday, January 31, 2017 3:35 PM
  • Known issue with DPM 2016 and protected machines older than Win2012 not supporting non-512 byte per sector size:

    https://social.technet.microsoft.com/Forums/en-US/ada0a0d1-0aa3-43e3-958d-cf4110d8986c/replica-is-inconsistent-errors?forum=dpmssandbmrbackup

    Thursday, February 2, 2017 9:13 PM
  • This has been fixed with DPM 2016 UR2.

    In order to make it work i had to remove the System State backup for the affected servers and re-protect the server, this time including the BMR protection. Just adding the BMR didn't work, i got an error that the storage has to be mounted. The replica creation took a while, but eventually finished.

    Wednesday, February 8, 2017 2:21 PM
  • I've got news for you, this still hasn't been fixed in UR5.  I have been complaining about this error (event 517) since DPM 2016 was released.  It is NOT restricted to old operating systems!  I have this problem protecting Windows Server 2016 as well.  I do not see why this cannot be fixed.  After a year of a premier support case covering this issue, Microsoft never even admitted a problem existed.

    See this too:  https://social.technet.microsoft.com/Forums/en-US/6d06ed6f-6241-41a5-bf2b-a6697db894a8/about-to-give-up-on-dpm-2016-ur2-anyone-else-in-the-same-situation?forum=dataprotectionmanager

    Tuesday, May 15, 2018 11:26 PM
  • Hello,

    We are using DPM 2016 UR5. and have system state backup failure for last one month.

    Error -

    DPM cannot create a backup because Windows Server Backup (WSB) on the protected computer encountered an error (WSB Event ID: 517, WSB Error Code:  0xE3E23E50). (ID 30229 Details: Internal error code: 0x80990ED0)

    when login on protected server and opened windows backup server console getting error "the backup catalog has been corrupted. Restore it from recent backup." Last successful backup of system state was on 27 April 2018 while catalog last modified time stamp is 21-08-2016.

    Is anyone know how to restore catalogue from DPM server. I have gone through the articleand it says about network drive but not DPM.

    https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/wbadmin-restore-catalog

    Thanks in advance

    Arun


    Arun Kumar

    Sunday, May 27, 2018 8:47 AM
  • hi all,

    Just wanted to bump this post … we are still having the same problems. 

    We are about to rebuild our BPM server without ReFS to see if that helps with BMR backups. 

    Can anyone confirm of they have any success with this ?

    Regards

    Rob 

    Friday, October 12, 2018 9:30 AM
  • I think it is required if you are backing up 2016 servers.  You may want to confirm.

    Friday, October 12, 2018 8:14 PM
  • Hello,

    we have the same Problem as described. Any News on that?

    Thursday, December 6, 2018 1:41 PM
  • Hi,

    I suggest you create a new thread about your issue, as this thread has been marked as answered.


    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Thursday, December 6, 2018 1:47 PM
  • "This has been fixed with DPM 2016 UR2."  -No it has not.  I have been using DPM 2016 since it was first released and this has NEVER been fixed!  Every update that comes out I hear it has been fixed, when the problem still exists.

    I am a Microsoft Premier customer and worked with their support team on and off for at least a year on this (and many other early DPM 2016 problems).  I finally gave up on this ever working again where I can trust it for this purpose.

     
    I used to really like DPM until version 2016 came out.  Then it became slow and unreliable.
    • Edited by simdoc Thursday, December 6, 2018 2:19 PM
    Thursday, December 6, 2018 2:13 PM
  • "This has been fixed with DPM 2016 UR2."  -No it has not.  I have been using DPM 2016 since it was first released and this has NEVER been fixed!  Every update that comes out I hear it has been fixed, when the problem still exists.

    I am a Microsoft Premier customer and worked with their support team on and off for at least a year on this (and many other early DPM 2016 problems).  I finally gave up on this ever working again where I can trust it for this purpose.

     
    We have the problem as well, since we used dpm 2016. Also the update UR6 did not bring any improvment.
    Thursday, December 6, 2018 2:19 PM
  • On and off problems with both 1807 and 2016 UR6 with this.

    We are currently transitioning away from DPM; this is one of many reasons why it doesn't work unfortunately.

    Monday, December 10, 2018 7:11 AM
  • What a pity, it can't be that there are such severe errors in a Backup Software.
    Wednesday, January 9, 2019 7:03 AM
  • Any News on that?
    Wednesday, May 15, 2019 9:25 AM
  • It's a shame to hear that.
    Tuesday, June 11, 2019 1:42 PM
  • mvds on here has suggested running DPM2016 or 2019 on Server 2012R2, which "disables" MBS.

    It works like a charm, even if 2019 isn't officially supported on 2012R2:

    https://social.technet.microsoft.com/Forums/en-US/7e4e4da4-1168-46cd-900f-9ca2bc364d5a/dpm-2016-mbs-performance-downward-spiral

    Maybe this will solve the OP's issue, too.

    Do note that DPM become veyr unhappy if you remove the contents of C:\Windows\Temp, as it does't bother recreating its contents there and it may result in very strange errors.

    Tuesday, June 11, 2019 2:50 PM