none
DPM 2012 R2 (BMR) Replica allways inconsistent RRS feed

  • Question

  • Hello,

    I have two Server 2012 servers running DPM 2012 R2. I am trying to do a Bare Metal Recovery backup on one of our protected servers. The protected server is running Server 2008 R2. But since i added the server the replica is allways inconsistent.

    Error on the DPM server:

    An unexpected error occurred while the job was running. (ID 104 Details: The process cannot access the file because it is being used by another process (0x80070020))

    All jobs in Windows Server Backup on the protected server shows as: Succesfull.
    No errors in the log files on: C:\Windows\Logs\WindowsServerBackup

    DPMRACurr.errlog on the 'protected' server:

    createsnapshotsubtask.cpp(971)	[000000000228D680]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	Executing PreSnap step ""c:\Program Files\Microsoft Data Protection Manager\DPM\bin\BMRBackup.cmd"  \\dpmserver.domain.com\7bbdce9d91934eb9bedbcd71f7665a0d"
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    service.cpp(147)	[000000000018F7A0]		ACTIVITY	CService::ServiceHandlerExInternal
    createsnapshotsubtask.cpp(1261)	[000000000228D680]		NORMAL	CreateSnapshotSubTask: this:[000000000228D680], Component: System Protection
    vssbaserequestor.cpp(51)	[000000000228E640]		NORMAL	CVssBaseRequestor: constructor [000000000228E640]
    vsssnapshotrequestor.cpp(91)	[000000000228E640]		NORMAL	CVssSnapshotRequestor::CVssSnapshotRequestor [000000000228E640]
    createsnapshotsubtask.cpp(1302)	[000000000228D680]		NORMAL	CreateSnapshotSubTask: this:[000000000228D680], m_pVssRequestor = [36234816] 
    vssbaserequestor.cpp(88)	[000000000228E640]		NORMAL	CVssBaseRequestor::Initialize [000000000228E640]
    vsssnapshotrequestor.cpp(185)	[000000000228E640]		NORMAL	CVssSnapshotRequestor::InitializeSnapshotCreation [000000000228E640]
    miscellaneousutils.cpp(677)			WARNING	Failed: Hr: = [0x80070002] Could not open HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft Data Protection Manager\Agent\2.0
    iteratorutils.cpp(648)			NORMAL	CaseSensitivity NOT enabled on the machine
    systemstatewriterhelperplugin.cpp(1206)			WARNING	Failed: Hr: = [0x80070003] : Encountered Failure: : lVal : hr
    vdshelper.cpp(293)	[0000000002290D40]		NORMAL	VdsHelper: VDS Service started
    fsmtransition.cpp(111)	[00000000008ED4A0]		WARNING	Failed: Hr: = [0x80070020] HasEventErrorCode: completion: 0xa10c, signature: 0xaabbcc00
    vsssnapshotrequestor.cpp(2056)	[000000000228E640]		NORMAL	CVssSnapshotRequestor::DoFailureCleanup [000000000228E640]
    vsssnapshotrequestor.cpp(1608)	[000000000228E640]		NORMAL	CVssSnapshotRequestor::ReleaseVolumesForSnapshot [000000000228E640]
    aasubtask.cpp(906)	[000000000228D680]		WARNING	<?xml version="1.0"?>
    aasubtask.cpp(906)	[000000000228D680]		WARNING	<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147024864" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="7f686c79-a38a-4ed9-bf89-f78b72665634" GuidWorkItem="64d01a3c-e977-4f66-a7b8-860d7ce8c792" TETaskInstanceID="ebbbdb3b-f585-4753-8e72-af1d5f44285b"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147024864" DetailedSource="2"/><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>130444369564690000</BackupTime><DSStatus><ComponentName>System Protection</ComponentName><LogicalPath>Computer</LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    runtime.cpp(1376)	[000000000087B170]		FATAL	Subtask failure, sending status response XML=[<?xml version="1.0"?>
    runtime.cpp(1376)	[000000000087B170]		FATAL	<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147024864" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="7f686c79-a38a-4ed9-bf89-f78b72665634" GuidWorkItem="64d01a3c-e977-4f66-a7b8-860d7ce8c792" TETaskInstanceID="ebbbdb3b-f585-4753-8e72-af1d5f44285b"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147024864" DetailedSource="2"/><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>130444369564690000</BackupTime><DSStatus><ComponentName>System Protection</ComponentName><LogicalPath>Computer</LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    runtime.cpp(1376)	[000000000087B170]		FATAL	]
    systemstatewriterhelperplugin.cpp(373)	[00000000022933D0]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	WARNING	Failed: Hr: = [0x80070020] : Encountered Failure: : lVal : sfDisk.Open((0x80000000L), 0x00000001)
    systemstatewriterhelperplugin.cpp(202)	[00000000022933D0]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	WARNING	Failed: Hr: = [0x80070020] : Encountered Failure: : lVal : GetCriticalDisks(ssWindowsBackupSetDir, fCaseSensitive, arrCriticalDisks, arrCriticalDiskSizes)
    systemstatewriterhelperplugin.cpp(176)	[00000000022933D0]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	WARNING	Failed: Hr: = [0x80070020] : Encountered Failure: : lVal : GetBMRRecoveryInformation(ssWindowsBackupSetDir, fCaseSensitive, ssOSVersion, isServerOS, ssOSArchitecture, dwBootDiskNumber, arrCriticalDisks, arrCriticalDiskSizes)
    systemstatewriterhelperplugin.cpp(557)	[00000000022933D0]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	WARNING	Failed: Hr: = [0x80070020] : Encountered Failure: : lVal : DumpBMRRecoveryInformation()
    createsnapshotsubtask.cpp(1508)	[000000000228D680]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	WARNING	Failed: Hr: = [0x80070020] : Encountered Failure: : lVal : pBackupHelper->DoPreSnapshotProcessing()
    fsmstate.cpp(167)	[000000000228B7A0]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	WARNING	Failed: Hr: = [0x80070020] : Encountered Failure: : lVal : pTransition->Execute(pEvent)
    genericfsm.cpp(225)	[0000000000877300]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	WARNING	Failed: Hr: = [0x80070020] : Encountered Failure: : lVal : m_pCurrentState->SendEvent(pEvent, pNextState)
    freesnapshotsubtask.cpp(46)	[0000000002295B70]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CFreeSnapshotSubTask: constructor [0000000002295B70]
    freesnapshotsubtask.cpp(123)	[0000000002295B70]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CFreeSnapshotSubTask::TriggerDone [0000000002295B70]
    freesnapshotsubtask.cpp(562)	[0000000002295B70]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CFreeSnapshotSubTask: StartBackupComplete [0000000002295B70]
    vssbaserequestor.cpp(1068)	[000000000228E640]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CVssBaseRequestor::StartGatherWriterStatus [000000000228E640]
    vssbaserequestor.cpp(944)	[000000000228E640]		NORMAL	QueryStatus returned 0x4230a, Releasing VssAsync [000000000067A4C0]
    freesnapshotsubtask.cpp(675)	[0000000002295B70]		NORMAL	CFreeSnapshotSubTask: ReleaseSnapshot [0000000002295B70]
    vssbaserequestor.cpp(1101)	[000000000228E640]		NORMAL	CVssBaseRequestor::CheckWriterStatus [000000000228E640]
    vsssnapshotrequestor.cpp(1346)	[000000000228E640]		NORMAL	CVssSnapshotRequestor::ReleaseSnaphot [000000000228E640]
    vsssnapshotrequestor.cpp(2022)	[000000000228E640]		NORMAL	CVssSnapshotRequestor::CleanUp [000000000228E640]
    vsssnapshotrequestor.cpp(1608)	[000000000228E640]		NORMAL	CVssSnapshotRequestor::ReleaseVolumesForSnapshot [000000000228E640]
    freesnapshotsubtask.cpp(181)	[0000000002295B70]		NORMAL	CFreeSnapshotSubTask::GetFinalStatus [0000000002295B70]
    freesnapshotsubtask.cpp(63)	[0000000002295B70]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CFreeSnapshotSubTask: destructor [0000000002295B70]
    vsssnapshotrequestor.cpp(109)	[000000000228E640]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CVssSnapshotRequestor::~CVssSnapshotRequestor [000000000228E640]
    vsssnapshotrequestor.cpp(2022)	[000000000228E640]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CVssSnapshotRequestor::CleanUp [000000000228E640]
    vsssnapshotrequestor.cpp(1608)	[000000000228E640]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CVssSnapshotRequestor::ReleaseVolumesForSnapshot [000000000228E640]
    vssbaserequestor.cpp(69)	[000000000228E640]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	NORMAL	CVssBaseRequestor: destructor [000000000228E640]
    workitem.cpp(272)	[0000000002288BE0]	EBBBDB3B-F585-4753-8E72-AF1D5F44285B	ACTIVITY	WorkItem stopping

    When i run procmon on the 'protected' server during the bmr job:

    DPMRA.exe	CreateFile							\Device\Harddisk0\DR0															SHARING VIOLATION
    DPMRA.exe	QueryStandardInformationFile		C:\Program Files\Microsoft Data Protection Manager\DPM\Temp\DPMRACurr.errlog	SUCCESS
    DPMRA.exe	WriteFile							C:\Program Files\Microsoft Data Protection Manager\DPM\Temp\DPMRACurr.errlog	SUCCESS
    

    Things i tried:

    • wbadmin.exe Start Backup –backuptarget:\\justaserver\tempshare –AllCritical
      100% complete, no errors.
    • Removed the protection group (delete data).
    • Re-installed the DPM agent.
    • Moved the protected server from dpmserver1 to dpmserver2.


    i appreciate any help you can provide.

    Kind regards,

    BoRo90

    Tuesday, May 13, 2014 8:33 AM

All replies

  • Hi,

    See if you have HP Insight Storage Agent running on the server(s), if so stop and disable it.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.


    Tuesday, May 13, 2014 8:57 PM
    Moderator
  • Hi Mike,

    Thanks for your reply.
    No we don't have HP Insight Storage Agent on the servers.

    On the DPM server we have an HP service running:

    Service name: sysdown
    Displayname: HP ProLiant System Shutdown Service
    C:\Windows\System32\sysdown.exe

    I have disabled that service, but still no luck.

    Kind regards

    Wednesday, May 14, 2014 8:02 AM
  • Hi,

    please check Windows Server Backup for Errors.

    look for Wbadmin.1.elt on the protected server an open with Notepad and check for errors


    Seidl Michael | http://www.techguy.at | twitter.com/techguyat | facebook.com/techguyat

    Wednesday, May 14, 2014 1:21 PM
  • Hi Seidl Michael,

    Thanks for your reply.
    Wbadmin.1.elt is only 10KB. Wbadmin.0.etl is 1570kb

    I opened Wbadmin.0.etl but its almost unreadable. I copied it in Excel (Maybe there are better tools to read etl files?).
    There may very well be an error but i cant seem to spot it.
    The last view rows shows that the backup has finished successfully.

     
    «È[£$iޠọ¯¸ôE(P¹3¼°System Reserved (100.00 MB) «gŒª”˜6Ì
    M«/;ô‰‰ê·(P¹3¼°The backup of volume System Reserved (100.00 MB) successfully completed.
    ÿÿÿÿÿÿV
    «È[£$iޠọ¯¸ôÑ(P¹3¼°Local Disk(C:)ÿÿ¨ «gŒª”˜6Ì
    M«/;ô‰‰ê,)P¹3¼°The backup of volume Local Disk(C:) successfully completed.
    
    Any other ideas?

    kind regards

    Wednesday, May 14, 2014 2:34 PM
  • Hi

    when you start a Synchronisation in DPM, waht error you are getting in Windows Backup on the protected server?


    Seidl Michael | http://www.techguy.at | twitter.com/techguyat | facebook.com/techguyat

    Wednesday, May 14, 2014 2:37 PM
  • Hi Seidl Michael,

    No errors at all.. All jobs seems to complete successfully.
    I think the error occurs after the wbengine process.

    Kind regards

    Wednesday, May 14, 2014 2:44 PM
  • Is there a Antivirus Software on one of this Servers?

    Try to stop the Realtime Protection and see whats happening


    Seidl Michael | http://www.techguy.at | twitter.com/techguyat | facebook.com/techguyat

    Wednesday, May 14, 2014 2:52 PM
  • Hi,

    We had one customer report that this did the trick.  Try adding this on the DPM Server.

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
    Value name: SMB2
    Value: 0


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, May 14, 2014 3:16 PM
    Moderator
  • Hi Seidl Michael and Mike Jacquet,

    @Seidl Michael: I stopped System center endpoint protection, but with no luck.

    @Mike: i will try your suggestion tommorow. (it's 1:00 AM now).

    Kind regards

    Wednesday, May 14, 2014 10:55 PM
  • Hi Mike,

    I disabled SMB2 on the DPM server with powershell:

    Set-SmbServerConfiguration -EnableSMB2Protocol $false

    (using this cmdlet we dont have to reboot the server).

    Still no luck.... same error:

    An unexpected error occurred while the job was running. (ID 104 Details: The process cannot access the file because it is being used by another process (0x80070020))

    Any other ideas?

    Kind regards

    Thursday, May 15, 2014 7:53 AM
  • Hi

    please try this on the protected Server

    If you go to that server and from a command prompt type:  "vssadmin list writers"  Do the writers show up in a healthy state or a bad state?
    If you go to that server and from a command prompt type: "vssadmin list shadows" Do you see any snapshots at all listed?

    maybe try once more and post the DPMRACURR.log from protected server and DPM Server


    Seidl Michael | http://www.techguy.at | twitter.com/techguyat | facebook.com/techguyat

    Thursday, May 15, 2014 7:57 AM
  • Hi Seidl Michael and Mike Jacquet ,

    vssadmin list writers:
    All stable, No errors

    vssadmin list shadows:
    No items found that satisfy the query.

    I can send you the logging by e-mail.

    Could you drop an e-mail to 9 0 b o r o [a t] g m a i l [d o t] c o m 
    I will reply with the log files attached.

    Kind regards.

    Thursday, May 15, 2014 9:01 AM
  • Thursday, May 15, 2014 10:36 AM
  • have send you an email

    Seidl Michael | http://www.techguy.at | twitter.com/techguyat | facebook.com/techguyat

    I have sent you the log files. Thanks for your effort!

    Thursday, May 15, 2014 11:31 AM
  • Hi,

    Can you run FLTMC and provide the list of filter drivers.  Also, if you unselect BMR and just keep SystemState - does that job complete OK ?    Is there more than protected server effected ?


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, May 15, 2014 2:25 PM
    Moderator
  • Hi Mike,

    FLTMC on DPM server:

    Filter Name                     Num Instances    Altitude    Frame
    ------------------------------  -------------  ------------  -----
    DpmFilter                               0       385300         0
    PROCMON23                               0       385200         0
    Dedup                                 179       180450         0
    SIS                                   179       180400         0
    luafv                                   1       135000         0
    npsvctrig                               1        46000        

    FLTMC on Protected server:

    Filter Name                     Num Instances    Altitude    Frame
    ------------------------------  -------------  ------------  -----
    DpmFilter                               0       385300         0
    PROCMON23                               0       385200         0
    FsDepends                               4       370060         0
    MpFilter                                5       328000         0
    luafv                                   1       135000         0

    Running only a systemstate job works fine! I have already tried that.

    Kind regards

    Thursday, May 15, 2014 2:35 PM
  • Hi Guys,

    Great news, there were no client connections to the protected server at this moment so i just disabled all non-windows services (there were running 6 non-windows services) on the protected machine.

    And the BMR backup has completed successfully for the first time.

    within 2 weeks I am going to check which service is causing DPM to fail.
    I will keep you updated.

    Thanks guys for your effort.

    • Proposed as answer by Chris - [MSFT] Thursday, September 25, 2014 3:23 PM
    Thursday, May 15, 2014 3:27 PM
  • Hi,

    GR8 - yes please share what U find for root cause.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, May 15, 2014 5:07 PM
    Moderator
  • please give us an Update

    Seidl Michael | http://www.techguy.at | twitter.com/techguyat | facebook.com/techguyat

    Thursday, October 2, 2014 9:42 AM