none
DC System State Backup Keeps Failing RRS feed

  • Question

  • Hello,

    One of our DC's fails to do a bare metal and system state backup every day. DPM Version is 2010. The DC is a Hyper-V Guest running 2012 R2. We are in the process of a procurement exercise for new hardware and software across the estate.

    I have reviewing the errors and browsed the web can’t seem to unravel what is going wrong and was hoping someone could help.

    The description of the fault on DPM is:

    The replica of System Protection Computer\System Protection is not consistent with the protected data source. (ID 91)

    I have tried running a consistency check which returns with the same error once complete.

    On the Application Event Log of the DC, at the time of the backup I have the 0x8078015b error:

    The backup operation that started at '‎2019‎-‎03‎-‎08T11:23:35.043747200Z' has failed with following error code '0x8078015B' (Windows Backup encountered an error when accessing the remote shared folder. Please retry the operation after making sure that the remote shared folder is available and accessible.). Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.

    I'm seeing this in the DPM Log:

    0A98      115C      03/08     11:33:07.416       31           vastatemachineutils.cpp(843)    [00000000012BE220]                       WARNING           Failed: Hr: = [0x809909fb] : F: lVal : m_pScriptExecutor->GetFailureHR()

    0A98      115C      03/08     11:33:07.416       05           fsmstate.cpp(167)           [00000000012BCCA0]                      WARNING                Failed: Hr: = [0x809909fb] : F: lVal : pTransition->Execute(pEvent)

    0A98      115C      03/08     11:33:07.416       05           genericfsm.cpp(225)      [00000000009DE410]                      WARNING                Failed: Hr: = [0x809909fb] : F: lVal : m_pCurrentState->SendEvent(pEvent, pNextState)

    0A98      115C      03/08     11:33:07.416       05           fsmtransition.cpp(111)  [00000000012BB9F0]                       WARNING                Failed: Hr: = [0x809909fb] HasEventErrorCode: completion: 0x27, signature: 0xaabbcc00

    0A98      115C      03/08     11:33:07.744       22           eventsutil.cpp(88)           [000000000332F650]                       WARNING                Failed: Hr: = [0x80070103] : F: lVal : m_pfnEvtNext(m_hQueryResult, 1, &m_hEvent, -1, 0, &dwEventsReturned)

    0A98      115C      03/08     11:33:07.744       31           aasubtask.cpp(1373)      [00000000012BDEC0]                      WARNING                Updating task status with hr: 809909fb8x, dls error code:30072

    0A98      115C      03/08     11:33:07.744       31           aasubtask.cpp(913)         [00000000012BDEC0]                      WARNING                <?xml version="1.0"?>

    0A98      115C      03/08     11:33:07.744       31           aasubtask.cpp(913)         [00000000012BDEC0]                      WARNING                <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2137454085" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="6fd713d6-8a4c-4f19-bbe6-2684156d6186" GuidWorkItem="70351c0a-dce7-42a2-bf37-4c285528674c" TETaskInstanceID="c2cc414d-5be5-4f43-ba25-a653ca84947c"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2137454085" DetailedSource="2"/><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>131965183877440000</BackupTime><DSStatus><ComponentName>System Protection</ComponentName><LogicalPath>Computer</LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>

    0A98      115C      03/08     11:33:07.744       03           runtime.cpp(1599)          [0000000001253190]                       FATAL                Subtask failure, sending status response XML=[<?xml version="1.0"?>

    0A98      115C      03/08     11:33:07.744       03           runtime.cpp(1599)          [0000000001253190]                       FATAL                <Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2137454085" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="6fd713d6-8a4c-4f19-bbe6-2684156d6186" GuidWorkItem="70351c0a-dce7-42a2-bf37-4c285528674c" TETaskInstanceID="c2cc414d-5be5-4f43-ba25-a653ca84947c"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="30072" DetailedCode="-2137454085" DetailedSource="2"><Parameter Name="WindowsBackupErrorCode" Value="0x8078015B"/><Parameter Name="WindowsBackupEventID" Value="517"/><Parameter Name="AgentTargetServer" Value="SERVERNAME"/></ErrorInfo><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>131965183877440000</BackupTime><DSStatus><ComponentName>System Protection</ComponentName><LogicalPath>Computer</LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>

    0A98      115C      03/08     11:33:07.744       03           runtime.cpp(1599)          [0000000001253190]                       FATAL   ]

    0A98      0D34      03/08     11:33:08.390       31           freesnapshotsubtask.cpp(46)     [00000000012C40F0]       C2CC414D-5BE5-4F43-BA25-A653CA84947C                NORMAL             CFreeSnapshotSubTask: constructor [00000000012C40F0]

    0A98      0D34      03/08     11:33:08.390       31           freesnapshotsubtask.cpp(123)   [00000000012C40F0]       C2CC414D-5BE5-4F43-BA25-A653CA84947C                NORMAL             CFreeSnapshotSubTask::TriggerDone [00000000012C40F0]

    0A98      0D34      03/08     11:33:08.405       31           freesnapshotsubtask.cpp(559)   [00000000012C40F0]       C2CC414D-5BE5-4F43-BA25-A653CA84947C                NORMAL             CFreeSnapshotSubTask: StartBackupComplete [00000000012C40F0]

    0A98      0D34      03/08     11:33:08.405       31           freesnapshotsubtask.cpp(671)   [00000000012C40F0]       C2CC414D-5BE5-4F43-BA25-A653CA84947C                NORMAL             CFreeSnapshotSubTask: ReleaseSnapshot [00000000012C40F0]

    0A98      115C      03/08     11:33:08.405       31           freesnapshotsubtask.cpp(181)   [00000000012C40F0]       C2CC414D-5BE5-4F43-BA25-A653CA84947C                NORMAL             CFreeSnapshotSubTask::GetFinalStatus [00000000012C40F0]

    0A98      0D34      03/08     11:33:08.421       31           freesnapshotsubtask.cpp(63)     [00000000012C40F0]       C2CC414D-5BE5-4F43-BA25-A653CA84947C                NORMAL             CFreeSnapshotSubTask: destructor [00000000012C40F0]

    0A98      115C      03/08     11:33:10.893       03           workitem.cpp(272)         [00000000009DE500]      C2CC414D-5BE5-4F43-BA25-A653CA84947C           ACTIVITY              WorkItem stopping

    0A98      13C8      03/08     11:38:09.634       22           genericthreadpool.cpp(804)        [0000000001253190]                       NORMAL             Hr: = [0x80070002] CGenericThreadPool::m_dwMaximumNumberOfThreads[20]

    0A98      06A4      03/08     11:38:09.634       22           genericthreadpool.cpp(804)        [00000000009DAFB0]                      NORMAL             Hr: = [0x80070002] CGenericThreadPool::m_dwMaximumNumberOfThreads[256]

    0A98      0370       03/08     11:38:10.914       03           runtime.cpp(1652)          [0000000001253190]                       NORMAL                CDLSRuntime::ProcessIdleTimeout

    0A98      0370       03/08     11:38:10.914       03           runtime.cpp(646)             [0000000001253190]                       NORMAL                CDLSRuntime::Uninitialize, bForce: 0

    0A98      0370       03/08     11:38:10.914       05           genericagent.cpp(264)  [00000000009D6A90]                      NORMAL                Agent Can Shutdown if there is only default wokitem active[1]

    0A98      0370       03/08     11:38:10.914       29           dpmra.cpp(360)                [00000000009D6A90]                      NORMAL                CDPMRA::Shutting down dpmra, force-shutdown :yes

    0A98      0370       03/08     11:38:10.914       03           workitem.cpp(391)                                         NORMAL             Timing out WI [0000000001298760], WI GUID = {B71B4544-7067-4A30-B5FB-BA320B10D82A}, ..last DM activity happened 66149093msec back, WI Idle Timeout = 390000msec

    0A98      0370       03/08     11:38:10.914       22           genericthreadpool.cpp(664)        [00000000009DAFB0]                      NORMAL             CGenericThreadPool: Waiting for threads to exit

    0A98      0134       03/08     11:38:12.927       20           cc_base.cpp(1226)          [00000000012A1FE0]                       NORMAL                DM: Aborting  http state machine for reason 0

    0A98      1124       03/08     11:38:12.927       20           cc_base.cpp(1226)          [00000000012A42D0]                      NORMAL                DM: Aborting  http state machine for reason 0

    0A98      0134       03/08     11:38:12.927       20           cc_base.cpp(1226)          [000000000129D130]                      NORMAL                DM: Aborting  http state machine for reason 0

    0A98      0370       03/08     11:38:12.927       20           cc_base.cpp(388)             [000000000129AE00]                      NORMAL                DM: Going to timeout connection

    0A98      0370       03/08     11:38:12.927       20           cc_base.cpp(388)             [000000000129F4A0]                       NORMAL                DM: Going to timeout connection

    0A98      0370       03/08     11:38:12.927       20           cc_base.cpp(388)             [00000000012A65C0]                      NORMAL                DM: Going to timeout connection

    0A98      0134       03/08     11:38:12.927       20           cc_base.cpp(1226)          [000000000129AE00]                      NORMAL                DM: Aborting  http state machine for reason 0

    0A98      0370       03/08     11:38:12.927       20           cc_base.cpp(388)             [00000000012A88B0]                      NORMAL                DM: Going to timeout connection

    0A98      0370       03/08     11:38:12.927       20           cc_base.cpp(388)             [00000000012AABA0]                     NORMAL                DM: Going to timeout connection

    0A98      1124       03/08     11:38:12.927       20           cc_base.cpp(1226)          [00000000012AF180]                       NORMAL                DM: Aborting  http state machine for reason 0

    0A98      0370       03/08     11:38:12.927       20           cc_base.cpp(388)             [00000000012ACE90]                      NORMAL                DM: Going to timeout connection

    0A98      0370       03/08     11:38:12.927       20           cc_base.cpp(388)             [00000000012AF180]                       NORMAL                DM: Going to timeout connection

    0A98      09B0      03/08     11:38:12.927       20           cc_base.cpp(1226)          [000000000129F4A0]                       NORMAL                DM: Aborting  http state machine for reason 0

    0A98      1124       03/08     11:38:12.927       20           cc_base.cpp(1226)          [00000000012AABA0]                     NORMAL                DM: Aborting  http state machine for reason 0

    0A98      1124       03/08     11:38:12.927       20           cc_base.cpp(1226)          [00000000012A88B0]                      NORMAL                DM: Aborting  http state machine for reason 0

    0A98      0584       03/08     11:38:12.927       20           cc_base.cpp(1226)          [00000000012ACE90]                      NORMAL                DM: Aborting  http state machine for reason 0

    0A98      09B0      03/08     11:38:12.927       20           cc_base.cpp(1226)          [00000000012A65C0]                      NORMAL                DM: Aborting  http state machine for reason 0

    0A98      0370       03/08     11:38:12.927       22           genericthreadpool.cpp(664)        [0000000001253190]                       NORMAL             CGenericThreadPool: Waiting for threads to exit

    0A98      0A54      03/08     11:38:14.971       03           timer.cpp(513)  [00000000012B1F98]                       ACTIVITY                Shutting down timer thread.

    0A98      0370       03/08     11:38:14.971       03           service.cpp(81)                                 ACTIVITY                CService::StopThisService

    0A98      0370       03/08     11:38:14.971       03           service.cpp(281)               [000000000065F9F0]                       ACTIVITY                CService::StopService()

    0A98      133C      03/08     11:38:14.971       03           service.cpp(298)               [000000000065F9F0]                       ACTIVITY                CService::AnnounceServiceStatus

    Many thanks in advance for any help that can be offered.

    Gary


    Friday, March 8, 2019 12:52 PM

Answers

All replies

  • Hello Gary,

    Could you give us some more information regarding your Hyper-V environment?

    • What version of Hyper-V are you running (operating system on Hyper-V host)?
    • Are you backing up the Hyper-V virtual machine or do you have a DPM agent on the DC (guest)?
    • Have you already successfully backed up your DC or has this problem started to occur later?
    • What are the changes that have been done recently?

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Friday, March 8, 2019 2:08 PM
  • Hi Leon,

    Thanks for replying.

    The Hyper-V Guest OS is Server 2012 R2.

    DPM is installed on the DC, not the Host. The DPM logs on the original post are from this.

    It hasn't worked since 18th Feb which coincides with Windows update - KB4483469

    https://support.microsoft.com/en-gb/help/4483469/description-of-the-security-only-update-for-net-framework-4-6-to-4-7-2

    It is completely up to date, no updates required. System boot time shows 07/03/2019

    I checked the Application Event Log on the DC. There are no errors on the 18th Feb but I did see this event:

    lsass (540) A database location change was detected from 'C:\Windows\NTDS\ntds.dit' to '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy51\Windows\NTDS\ntds.dit'.

    It successfully completed recovery but I wonder if this could be linked?

    Friday, March 8, 2019 3:52 PM
  • The first thing that comes to my mind is that this is not a supported configuration.

    As per the supported operating systems for DPM 2010 on the link below, Windows 2012 R2 is not supported.

    Protected Computer Software Requirements
    https://docs.microsoft.com/en-us/previous-versions/system-center/data-protection-manager-2010/ff399140(v=technet.10)

    Now I know many have successfully managed to protect a Windows Server 2012/2012 R2 with DPM 2010, but this might lead to problems that could basically be anything.

    What I would advise you is to really think about upgrading your DPM 2010 to a newer version, for example DPM 2019 is going to be generally available in march 2019.


    Blog: https://thesystemcenterblog.com LinkedIn:

    • Marked as answer by Gary OConnor Saturday, March 9, 2019 10:47 AM
    Friday, March 8, 2019 4:26 PM
  • Thanks for the link that confirms this Leon. We are getting pricing for 3 backup solutions so I will make sure we get pricing for DPM 2019.

    Also, sounds like we need to make this a priority if 2012 R2 isn't supported.

    Many thanks for your help as always.

    Saturday, March 9, 2019 10:52 AM
  • You're very welcome Gary, if you have any other concerns feel free to ask!

    Blog: https://thesystemcenterblog.com LinkedIn:

    Saturday, March 9, 2019 10:55 AM
  • Just FYI, I removed the server from the protection group, deleted all data, re-created the backup and it backed up ok.

    At least this will give me a bit of breathing space until we can upgrade.

    Monday, March 11, 2019 9:16 AM
  • Okay, that's good news Gary, let's hope it buys you enough time!

    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, March 11, 2019 11:45 PM