none
DPM 2010 HyperV snapshot issues RRS feed

  • Question

  • Welcome to an interesting issue.. 

    We just migrated all of our HyperV servers from our old Lefthand FC SAN to our new NetApp 2240 ISCSI SAN. We went from 6 server 2008R2 hosts to 2 server 2012 hosts. All the DPM backup functions are working properly except for the HyperV Resource Snapshots. We're getting an unknown error. Below are the error logs. The things I've tried:

    Deleting and recreating the protection groups in question.
    Deleting the protection group, deleting the data, recreating the protection group.
    Deleting the protection group, deleting the data, uninstalling and reinstalling the agent, recreating the protection group. 
    Deleting the protection group, deleting the data, uninstalling the agent, restarting both DPM and the VM, reinstalling the agent, and recreating the protection group. 
    Cry a little. Google a little. Cry some more. Now I'm here..

    I end up with the same error each time when trying to run a consistency check for the snapshot. ANY help would be greatly appreciated!! Let me know if you need any more information.

    Current setup
    DPM2010 on a server 2008R2 stand alone box. 
    HyperV cluster with 2 server 2013 hosts
    All guests (including those in question) are server 2008R2

    6 VMs in question that are HyperV snapshots only. 

    Log from most recent consistency check below. **Server names have been changed to protect the innocent - Naming convention is still the same though so it should still work**

    0CF4	1294	03/14	16:45:31.786	02	EventManager.cs(88)			NORMAL	Publishing event from ActiveJob.cs(1043): JobStateChange, [JobID=2983420f-deba-4d37-a155-19d0be84a04d]
    0CF4	1294	03/14	16:45:31.793	16	JobManager.cs(602)			NORMAL	 Removed job entry from active list:2983420f-deba-4d37-a155-19d0be84a04d
    0CF4	1294	03/14	16:45:31.793	16	ActiveJob.cs(1240)			NORMAL	Completed execution of job:'2983420f-deba-4d37-a155-19d0be84a04d', final job state:'Succeeded'
    0CF4	17E8	03/14	16:46:28.476	09	coreservices.cpp(5453)		09315528-3E19-4972-A76E-A7E994C77992	NORMAL	CCoreServices::CheckForPendingReboot() entered
    0CF4	17E8	03/14	16:46:28.476	16	engineservices.cpp(185)		09315528-3E19-4972-A76E-A7E994C77992	NORMAL	CEngineServices::CheckForPendingReboot entered
    0CF4	17E8	03/14	16:46:28.476	16	engineservices.cpp(197)		09315528-3E19-4972-A76E-A7E994C77992	WARNING	Failed: Hr: = [0x80070002] : F: lVal : (UINT)RegOpenKeyExW((( HKEY ) (ULONG_PTR)((LONG)0x80000002) ), (LPCWSTR)rebootregkey, 0, ((((0x00020000L)) | (0x0001) | (0x0008) | (0x0010)) & (~(0x00100000L))), &key)
    0CF4	17E8	03/14	16:46:36.122	17	intentservices.cpp(633)		09315528-3E19-4972-A76E-A7E994C77992	NORMAL	CIntentServices::TriggerValidation(). dataSourceId={B498D14F-17A6-4058-B0B7-61E300F06A02}, jobDefinitionId={75E850C3-B13C-4A70-A9A4-82FAFA6D2363}, retriggeredForJobDefId={00000000-0000-0000-0000-000000000000}
    0CF4	17E8	03/14	16:46:36.135	16	JobDefinitionManager.cs(122)			NORMAL	 AddJobDefinition:75e850c3-b13c-4a70-a9a4-82fafa6d2363
    0CF4	17E8	03/14	16:46:36.157	16	JobManager.cs(354)			NORMAL	 Trigger job:75e850c3-b13c-4a70-a9a4-82fafa6d2363 with id:01405ab8-2111-4a7a-bb82-3be93e82dc65
    0CF4	17E8	03/14	16:46:36.161	16	Task.cs(235)			NORMAL	 Changing task state from 'Invalid' -> 'Start' (12b78186-0126-4f12-a8b3-498da9002f09)
    0CF4	17E8	03/14	16:46:36.161	16	Task.cs(235)			NORMAL	 Changing task state from 'Invalid' -> 'Start' (28d9a7ba-0164-4398-9793-5e99387850cb)
    0CF4	17E8	03/14	16:46:36.161	16	Task.cs(235)			NORMAL	 Changing task state from 'Invalid' -> 'Start' (479b3618-5b33-42ef-ac6b-2ee9d7c6db4a)
    0CF4	17E8	03/14	16:46:36.184	02	EventManager.cs(88)			NORMAL	Publishing event from ActiveJob.cs(272): JobStateChange, [JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65]
    0CF4	1294	03/14	16:46:36.189	16	ActiveJob.cs(290)			NORMAL	 Start execution of job:'01405ab8-2111-4a7a-bb82-3be93e82dc65' in state:'Start'
    0CF4	1294	03/14	16:46:36.189	08	ActivityWindow.cs(75)			NORMAL	 Add new wait : 01405ab8-2111-4a7a-bb82-3be93e82dc65(wait until: Wednesday, April 03, 2013) @ Thursday, March 14, 2013UTC
    0CF4	1294	03/14	16:46:36.189	08	ActivityWindow.cs(198)			NORMAL	 Setting timer for '1727999.9726552's...
    0CF4	1294	03/14	16:46:36.189	16	ActiveJob.cs(1024)			NORMAL	 Changing state of job: '01405ab8-2111-4a7a-bb82-3be93e82dc65' from 'Start' -> 'GenerateWorkplan'
    0CF4	1294	03/14	16:46:36.191	02	EventManager.cs(88)			NORMAL	Publishing event from ActiveJob.cs(1043): JobStateChange, [JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65]
    0CF4	1294	03/14	16:46:36.196	16	ActiveJob.cs(524)			NORMAL	 Begin GenerateWorkplan:JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65, ItemID=12b78186-0126-4f12-a8b3-498da9002f09
    0CF4	1294	03/14	16:46:36.196	01	TaskExecutor.cs(81)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	GenerateWorkplan(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Entering
    0CF4	1294	03/14	16:46:36.235	02	EventManager.cs(88)			NORMAL	Publishing event from TaskExecutor.cs(107): TaskCreated, [TaskID=12b78186-0126-4f12-a8b3-498da9002f09]
    0CF4	1294	03/14	16:46:36.240	01	TaskExecutor.cs(127)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	GenerateWorkplan(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Finished (error=False), executing GenerateWorkplanCallback
    0CF4	1294	03/14	16:46:36.240	16	Task.cs(235)			NORMAL	 Changing task state from 'Start' -> 'GenerateWorkplan' (12b78186-0126-4f12-a8b3-498da9002f09)
    0CF4	1294	03/14	16:46:36.241	16	ActiveJob.cs(524)			NORMAL	 Begin GenerateWorkplan:JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65, ItemID=28d9a7ba-0164-4398-9793-5e99387850cb
    0CF4	1294	03/14	16:46:36.241	01	TaskExecutor.cs(81)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	GenerateWorkplan(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Entering
    0CF4	1294	03/14	16:46:36.262	02	EventManager.cs(88)			NORMAL	Publishing event from TaskExecutor.cs(107): TaskCreated, [TaskID=28d9a7ba-0164-4398-9793-5e99387850cb]
    0CF4	1294	03/14	16:46:36.284	01	TaskExecutor.cs(127)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	GenerateWorkplan(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Finished (error=False), executing GenerateWorkplanCallback
    0CF4	1294	03/14	16:46:36.284	16	Task.cs(235)			NORMAL	 Changing task state from 'Start' -> 'GenerateWorkplan' (28d9a7ba-0164-4398-9793-5e99387850cb)
    0CF4	1294	03/14	16:46:36.285	16	ActiveJob.cs(524)			NORMAL	 Begin GenerateWorkplan:JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65, ItemID=479b3618-5b33-42ef-ac6b-2ee9d7c6db4a
    0CF4	1294	03/14	16:46:36.285	01	TaskExecutor.cs(81)		479B3618-5B33-42EF-AC6B-2EE9D7C6DB4A	NORMAL	GenerateWorkplan(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Entering
    0CF4	1294	03/14	16:46:36.302	02	EventManager.cs(88)			NORMAL	Publishing event from TaskExecutor.cs(107): TaskCreated, [TaskID=479b3618-5b33-42ef-ac6b-2ee9d7c6db4a]
    0CF4	1294	03/14	16:46:36.321	01	TaskExecutor.cs(127)		479B3618-5B33-42EF-AC6B-2EE9D7C6DB4A	NORMAL	GenerateWorkplan(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Finished (error=False), executing GenerateWorkplanCallback
    0CF4	1294	03/14	16:46:36.321	16	Task.cs(235)			NORMAL	 Changing task state from 'Start' -> 'GenerateWorkplan' (479b3618-5b33-42ef-ac6b-2ee9d7c6db4a)
    0CF4	1294	03/14	16:46:36.322	16	ActiveJob.cs(1024)			NORMAL	 Changing state of job: '01405ab8-2111-4a7a-bb82-3be93e82dc65' from 'GenerateWorkplan' -> 'Execute'
    0CF4	1294	03/14	16:46:36.323	02	EventManager.cs(88)			NORMAL	Publishing event from ActiveJob.cs(1043): JobStateChange, [JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65]
    0CF4	1294	03/14	16:46:36.328	16	ActiveJob.cs(774)			NORMAL	 Begin new execution batch at task:(True)-12b78186-0126-4f12-a8b3-498da9002f09
    0CF4	1294	03/14	16:46:36.329	16	ActiveJob.cs(794)			NORMAL	 Dispatching task for execution:True-12b78186-0126-4f12-a8b3-498da9002f09
    0CF4	1294	03/14	16:46:36.329	16	ActiveJob.cs(524)			NORMAL	 Begin ExecuteTask:JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65, ItemID=12b78186-0126-4f12-a8b3-498da9002f09
    0CF4	1294	03/14	16:46:36.329	01	TaskExecutor.cs(692)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Setting up Fsm: verb[85d11aaa-bbbc-427d-9932-eeb3fd4759f5]
    0CF4	1294	03/14	16:46:36.333	01	TaskExecutor.cs(489)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	ExecuteTask(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Starting task instance
    0CF4	15D0	03/14	16:46:36.333	03	timer.cpp(490)			ACTIVITY	CTimerMgr::TimerThread
    0CF4	15D0	03/14	16:46:36.333	03	timer.cpp(503)	[0000000019B87108]		ACTIVITY	CTimerMgr::TimerThreadInternal
    0CF4	1294	03/14	16:46:36.334	02	EventManager.cs(88)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Publishing event from TaskInstance.cs(407): TaskStart, [TaskID=12b78186-0126-4f12-a8b3-498da9002f09]
    0CF4	1294	03/14	16:46:36.351	01	TaskInstance.cs(444)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Task: Initial transition enqueued, startState='ConfigureProtection.StartState'
    0CF4	1294	03/14	16:46:39.865	27	FindActiveNodeBlock.cs(884)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Found Principal server for DatasourceId = b498d14f-17a6-4058-b0b7-61e300f06a02 Server= Hyper-v-HOST-1
    0CF4	1294	03/14	16:46:44.772	27	CommonErrorHandler.cs(165)		12B78186-0126-4F12-A8B3-498DA9002F09	WARNING	AgentStatus[RAForRead] - (CommandID=RAConfigureDeltaTracking, StatusReason=Error) failed with HRESULT 0x80004005, error -2147467259.
    0CF4	1294	03/14	16:46:44.773	27	CommonErrorHandler.cs(265)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Unmapped agent error code = GenericAgentFailure
    0CF4	15C8	03/14	16:46:45.030	04	cmdproc.cpp(2425)	[0000000019B86F50]	12B78186-0126-4F12-A8B3-498DA9002F09	WARNING	Failed: Hr: = [0x80000001] : F: lVal : hr
    0CF4	15C8	03/14	16:46:45.030	04	cmdproc.cpp(2482)	[0000000019B86F50]	12B78186-0126-4F12-A8B3-498DA9002F09	WARNING	CCommandProcessor::SendOutboundCommand this:[0000000019B86F50], ServerName: Hyper-v-HOST
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	Task: Received tiemout message from CmdProc '<?xml version="1.0" encoding="utf-16"?>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	<TECommand xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.microsoft.com/2003/dls/Commands.xsd">
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	  <Command xmlns="http://schemas.microsoft.com/2003/dls/GenericCommand.xsd">
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	    <CommandInstanceID>60cfba9f-1974-4838-ad5d-db80d535720f</CommandInstanceID>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	    <AgentTypeID>da6aa17a-d61c-4e9c-8cea-db25dea52a95</AgentTypeID>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	    <WorkItemID>12b78186-0126-4f12-a8b3-498da9002f09</WorkItemID>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	    <TETaskInstanceID>12b78186-0126-4f12-a8b3-498da9002f09</TETaskInstanceID>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	    <ServerIdFilter>c870b70b-6386-4a9a-8fdc-fa446250de8b</ServerIdFilter>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	    <VerbIndexFilter>13</VerbIndexFilter>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	    <DatasourceIndexFilter>522</DatasourceIndexFilter>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	  </Command>
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	  <RASetVolumeControlBlockInfo />
    0CF4	0D90	03/14	16:46:45.031	01	TaskInstance.cs(253)			NORMAL	</TECommand>' Error CmdProcCommunicationError
    0CF4	1294	03/14	16:46:45.032	27	FsmBlock.cs(178)			WARNING	ConfigureProtection.PropagateConfigInfo : RASetVolumeControlBlockInfo, StatusReason = Timeout (StatusCode = -2147483647, ErrorCode = CmdProcCommunicationError, workitem = 12b78186-0126-4f12-a8b3-498da9002f09)
    0CF4	1294	03/14	16:46:45.032	27	FsmBlock.cs(178)			WARNING	Response: <?xml version="1.0" encoding="utf-16"?>
    0CF4	1294	03/14	16:46:45.032	27	FsmBlock.cs(178)			WARNING	<Status xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" StatusCode="-2147483647" Reason="Timeout" CommandInstanceID="60cfba9f-1974-4838-ad5d-db80d535720f" CommandID="RASetVolumeControlBlockInfo" GuidWorkItem="12b78186-0126-4f12-a8b3-498da9002f09" TETaskInstanceID="12b78186-0126-4f12-a8b3-498da9002f09" xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd">
    0CF4	1294	03/14	16:46:45.032	27	FsmBlock.cs(178)			WARNING	  <ErrorInfo ErrorCode="259" DetailedCode="-2147483647" DetailedSource="2" ExceptionDetails="" xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd">
    0CF4	1294	03/14	16:46:45.032	27	FsmBlock.cs(178)			WARNING	    <Parameter Name="servername" Value="Hyper-v-HOST" />
    0CF4	1294	03/14	16:46:45.032	27	FsmBlock.cs(178)			WARNING	  </ErrorInfo>
    0CF4	1294	03/14	16:46:45.032	27	FsmBlock.cs(178)			WARNING	</Status>
    0CF4	1294	03/14	16:46:45.033	27	CommonErrorHandler.cs(104)		12B78186-0126-4F12-A8B3-498DA9002F09	WARNING	AgentStatus[RAForWrite] - (CommandID=RASetVolumeControlBlockInfo, StatusReason=Timeout) failed with HRESULT 0x80000001, error -2147483647.
    0CF4	1294	03/14	16:46:45.034	27	RARequestor.cs(188)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Sending Command RACancelAllSubTasks to RAForWrite on server Hyper-v-HOST for workitem 12b78186-0126-4f12-a8b3-498da9002f09
    0CF4	15D0	03/14	16:46:45.959	03	timer.cpp(513)	[0000000019B87108]		ACTIVITY	Shutting down timer thread.
    0CF4	1448	03/14	16:46:45.959	02	EventManager.cs(88)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Publishing event from TaskInstance.cs(825): TaskStop, [TaskID=12b78186-0126-4f12-a8b3-498da9002f09]
    0CF4	1448	03/14	16:46:45.963	01	TaskExecutor.cs(849)		12B78186-0126-4F12-A8B3-498DA9002F09	NORMAL	Task stopped (state=Completed, error=RmDCOMAgentCommunicationError; -2147483647; WindowsHResult)
    0CF4	1448	03/14	16:46:45.963	16	Task.cs(235)			NORMAL	 Changing task state from 'GenerateWorkplan' -> 'Execute' (12b78186-0126-4f12-a8b3-498da9002f09)
    0CF4	1448	03/14	16:46:45.964	16	ActiveJob.cs(774)			NORMAL	 Begin new execution batch at task:(True)-28d9a7ba-0164-4398-9793-5e99387850cb
    0CF4	1448	03/14	16:46:45.966	16	ActiveJob.cs(794)			NORMAL	 Dispatching task for execution:True-28d9a7ba-0164-4398-9793-5e99387850cb
    0CF4	1448	03/14	16:46:45.966	16	ActiveJob.cs(524)			NORMAL	 Begin ExecuteTask:JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65, ItemID=28d9a7ba-0164-4398-9793-5e99387850cb
    0CF4	1448	03/14	16:46:45.970	01	DPMTaskController.cs(70)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	(job=01405ab8-2111-4a7a-bb82-3be93e82dc65, taskId=28d9a7ba-0164-4398-9793-5e99387850cb) DPMTaskControllerHelper : ExecuteTask - CheckTaskStatus returned ReadyToExecute
    0CF4	1448	03/14	16:46:45.971	01	TaskExecutor.cs(692)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Setting up Fsm: verb[c4cae2f7-f068-4a37-914e-9f02991868da]
    0CF4	1448	03/14	16:46:45.974	27	CommonLoop.cs(343)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	DatamovePollingCmdTimeoutInMin = 30
    0CF4	1448	03/14	16:46:45.977	01	TaskExecutor.cs(489)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	ExecuteTask(job=01405ab8-2111-4a7a-bb82-3be93e82dc65) - Starting task instance
    0CF4	061C	03/14	16:46:45.978	03	timer.cpp(490)			ACTIVITY	CTimerMgr::TimerThread
    0CF4	061C	03/14	16:46:45.978	03	timer.cpp(503)	[0000000019B87108]		ACTIVITY	CTimerMgr::TimerThreadInternal
    0CF4	1448	03/14	16:46:45.979	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from TaskInstance.cs(407): TaskStart, [TaskID=28d9a7ba-0164-4398-9793-5e99387850cb]
    0CF4	1448	03/14	16:46:46.030	01	TaskInstance.cs(444)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Task: Initial transition enqueued, startState='Backup.StartState'
    0CF4	1448	03/14	16:46:46.030	27	BackupMachine.cs(1897)		28D9A7BA-0164-4398-9793-5E99387850CB	ACTIVITY	BackupMachine : Starting machine execution
    0CF4	1448	03/14	16:46:46.032	27	DecideConsistencyCheckNeededBlock.cs(292)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Consistency Check (Needed) for datasource=b498d14f-17a6-4058-b0b7-61e300f06a02
    0CF4	1448	03/14	16:46:48.352	27	FindActiveNodeBlock.cs(884)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Found Principal server for DatasourceId = b498d14f-17a6-4058-b0b7-61e300f06a02 Server= Hyper-v-HOST-1
    0CF4	1448	03/14	16:46:48.356	27	OperationTypeLock.cs(766)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	In LockServer() of ReplicationOperationTypeLock, returning lockStatus = AcquiredLock
    0CF4	1448	03/14	16:46:48.357	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from Replica.cs(1948): ReplicaStatusChange, [DataSourceID=a21965d3-71a2-4c95-9a56-d614a36da516]
    0CF4	1448	03/14	16:46:48.362	27	OperationTypeLock.cs(671)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	In Lock() of ReplicationOperationTypeLock, returning lockStatus = AcquiredLock
    0CF4	1448	03/14	16:46:48.367	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from ServerAlert.cs(650): AlertStateChange, [ObjectId=b498d14f-17a6-4058-b0b7-61e300f06a02]
    0CF4	1448	03/14	16:46:48.372	23	AlertVisibility.cs(263)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	StopAutoHealActionTimer: Timer for alertId=[dabadca3-e14a-43ec-909e-80195c87902f] not found
    0CF4	1448	03/14	16:46:48.386	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from ServerAlert.cs(556): AlertStateChange, [ObjectId=b498d14f-17a6-4058-b0b7-61e300f06a02]
    0CF4	1448	03/14	16:46:48.431	27	BackupMachine.cs(2247)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	RaiseReplicaVerificationInProgressAlert = True for ReplicaId = a21965d3-71a2-4c95-9a56-d614a36da516
    0CF4	10E0	03/14	16:46:48.599	09	coreservices.cpp(7011)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	CCoreServices::UpdateAlertVisibilityIfInvisible() entered
    0CF4	10E0	03/14	16:46:48.599	16	engineservices.cpp(356)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	CEngineServices::UpdateAlertVisibilityIfInvisible entered. alertId={7ECE9C4B-DBD5-40E5-8ADE-07147A646F8A}, newVisibility=0, invisibilityTimeout=0
    0CF4	10E0	03/14	16:46:48.603	02	EventManager.cs(88)			NORMAL	Publishing event from ServerAlert.cs(722): AlertStateChange, [ObjectId=b498d14f-17a6-4058-b0b7-61e300f06a02]
    0CF4	1448	03/14	16:46:49.152	27	DeleteBitmapsBlock.cs(202)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	No bitmap deleted physically, deleting bitmaps from Database for datasource=b498d14f-17a6-4058-b0b7-61e300f06a02
    0CF4	0A48	03/14	16:46:49.156	18	fsutils.cpp(3710)			WARNING	Failed: Hr: = [0x80070003] : CreateFile failed for \\?\Volume{e02293f2-89ca-11e2-b64b-002481b2d5fd}\b498d14f-17a6-4058-b0b7-61e300f06a02\DatasourceVolumeMapping.xml
    0CF4	1558	03/14	16:46:49.157	18	fsutils.cpp(3710)			WARNING	Failed: Hr: = [0x80070003] : CreateFile failed for \\?\Volume{e02293f2-89ca-11e2-b64b-002481b2d5fd}\b498d14f-17a6-4058-b0b7-61e300f06a02\Incremental\
    0CF4	1448	03/14	16:46:49.177	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from RecoverySourceFactory.cs(230): DatasetStateChange, [DatasetID=e84d4a30-6807-4a74-8153-bc911a781781]
    0CF4	1448	03/14	16:46:51.503	27	FsmBlock.cs(178)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	Backup.ReplicaPreBackupBlock : RAPreBackup, StatusReason = Error (StatusCode = -2147467259, ErrorCode = GenericAgentFailure, workitem = 7dbb6217-5141-44cd-bbea-a404fae1a2d7)
    0CF4	1448	03/14	16:46:51.503	27	FsmBlock.cs(178)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	Response: <?xml version="1.0"?>
    0CF4	1448	03/14	16:46:51.503	27	FsmBlock.cs(178)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	<Status xmlns="http://schemas.microsoft.com/2003/dls/StatusMessages.xsd" StatusCode="-2147467259" Reason="Error" CommandID="RAPreBackup" CommandInstanceID="50c29dcd-c883-42a4-bf0b-9bb829506a39" GuidWorkItem="7dbb6217-5141-44cd-bbea-a404fae1a2d7" TETaskInstanceID="28d9a7ba-0164-4398-9793-5e99387850cb"><ErrorInfo xmlns="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" ErrorCode="998" DetailedCode="-2147467259" DetailedSource="2"/><RAStatus><RAPreBackup xmlns="http://schemas.microsoft.com/2003/dls/ArchiveAgent/StatusMessages.xsd"><BackupTime>130077532113110000</BackupTime><DSStatus><ComponentName>AFE745ED-22B9-41DF-AB29-7411AFA5760A</ComponentName><LogicalPath></LogicalPath><BackupStamp></BackupStamp><Metadata></Metadata></DSStatus></RAPreBackup></RAStatus></Status>
    0CF4	1448	03/14	16:46:51.505	27	CommonErrorHandler.cs(165)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	AgentStatus[RAForRead] - (CommandID=RAPreBackup, StatusReason=Error) failed with HRESULT 0x80004005, error -2147467259.
    0CF4	1448	03/14	16:46:51.505	27	CommonErrorHandler.cs(265)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Unmapped agent error code = GenericAgentFailure
    0CF4	1448	03/14	16:46:51.508	27	FsmBlock.cs(130)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	Backup.ReplicaPreBackupBlock : <-- Exited FSM block with FAILURE (errorCode = RmGenericError)
    0CF4	1448	03/14	16:46:51.508	27	RARequestor.cs(188)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Sending Command RACancelAllSubTasks to RAForRead on server Hyper-v-HOST-1 for workitem 7dbb6217-5141-44cd-bbea-a404fae1a2d7
    0CF4	0EE8	03/14	16:46:52.854	27	FsmBlock.cs(118)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Backup.OnFailure(1).RAPostBackupBlock : <-- Exited FSM block with SUCCESS
    0CF4	0EE8	03/14	16:46:52.855	27	DeleteBitmapsBlock.cs(202)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	No bitmap deleted physically, deleting bitmaps from Database for datasource=b498d14f-17a6-4058-b0b7-61e300f06a02
    0CF4	0EE8	03/14	16:46:52.867	27	RARequestor.cs(188)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Sending Command RACancelAllSubTasks to RAForWrite on server backup.company.net for workitem ad86c0db-21be-4f94-8cf6-e589144feabf
    0CF4	0C74	03/14	16:46:53.837	27	RARequestor.cs(188)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Sending Command RACancelAllSubTasks to RAForRead on server Hyper-v-HOST-1 for workitem 7dbb6217-5141-44cd-bbea-a404fae1a2d7
    0CF4	1658	03/14	16:46:54.979	27	BackupMachine.cs(2737)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	BackupMachine : DBCleanup on backup failure with errorCode=RmGenericError
    0CF4	1658	03/14	16:46:54.979	27	BackupMachine.cs(2745)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	BackupMachine: Cleaning up ROlist for DatasetId [e84d4a30-6807-4a74-8153-bc911a781781] on backup failure, will also mark Dataset as Invalid
    0CF4	0ECC	03/14	16:46:55.015	18	fsutils.cpp(3710)			WARNING	Failed: Hr: = [0x80070002] : CreateFile failed for \\?\C:\Program Files\Microsoft DPM\DPM\Temp\28d9a7ba-0164-4398-9793-5e99387850cb\
    0CF4	1658	03/14	16:46:55.018	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from ServerAlert.cs(650): AlertStateChange, [ObjectId=b498d14f-17a6-4058-b0b7-61e300f06a02]
    0CF4	1658	03/14	16:46:55.023	23	AlertVisibility.cs(263)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	StopAutoHealActionTimer: Timer for alertId=[7ece9c4b-dbd5-40e5-8ade-07147a646f8a] not found
    0CF4	1658	03/14	16:46:55.032	27	BackupMachine.cs(2523)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	BackupComplete : ResolveReplicaVerificationInProgressAlert = True for ReplicaId = a21965d3-71a2-4c95-9a56-d614a36da516
    0CF4	1658	03/14	16:46:55.087	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from ServerAlert.cs(556): AlertStateChange, [ObjectId=b498d14f-17a6-4058-b0b7-61e300f06a02]
    0CF4	1658	03/14	16:46:55.094	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from Replica.cs(1948): ReplicaStatusChange, [DataSourceID=a21965d3-71a2-4c95-9a56-d614a36da516]
    0CF4	1658	03/14	16:46:55.104	27	OperationTypeLock.cs(812)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	UnlockServer() of ReplicationOperationTypeLock, returning
    0CF4	1658	03/14	16:46:55.104	27	OperationTypeLock.cs(702)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	In Unlock() of ReplicationOperationTypeLock, returning
    0CF4	1658	03/14	16:46:55.105	27	BackupMachine.cs(2596)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	BackupMachine : FAILURE - BACKUP, errorCode=RmGenericError
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	Task Diagnostic Information - <?xml version="1.0" encoding="utf-16"?>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	<TaskExecutionContext>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmWriterId>66841cd4-6ded-4f4b-8f17-fd23f8ddc3de</PrmWriterId>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmDatasourceId>b498d14f-17a6-4058-b0b7-61e300f06a02</PrmDatasourceId>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmPhysicalReplicaId>e5df17c0-5416-43a1-85b4-7aa6d7868954</PrmPhysicalReplicaId>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmReplicaValidity>Invalid</PrmReplicaValidity>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmReplicaStatus>Idle</PrmReplicaStatus>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmOwnerLockId>00000000-0000-0000-0000-000000000000</PrmOwnerLockId>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmActiveNodeName>SCVMM guestvm.company.netResources.Cluster.company.net</PrmActiveNodeName>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmLogicalReplicaId>a21965d3-71a2-4c95-9a56-d614a36da516</PrmLogicalReplicaId>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <PrmDatasetId>e84d4a30-6807-4a74-8153-bc911a781781</PrmDatasetId>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <TEVerb>ValidateFixupReplica</TEVerb>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <TEErrorState>Backup.RAPreBackupPending</TEErrorState>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	  <TEErrorDetails>&lt;?xml version="1.0" encoding="utf-16"?&gt;
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	&lt;q1:ErrorInfo ErrorCode="104" DetailedCode="-2147467259" DetailedSource="2" ExceptionDetails="" xmlns:q1="http://schemas.microsoft.com/2003/dls/GenericAgentStatus.xsd" /&gt;</TEErrorDetails>
    0CF4	1658	03/14	16:46:55.106	01	TaskInstance.cs(800)		28D9A7BA-0164-4398-9793-5E99387850CB	WARNING	</TaskExecutionContext>
    0CF4	061C	03/14	16:46:55.107	03	timer.cpp(513)	[0000000019B87108]		ACTIVITY	Shutting down timer thread.
    0CF4	1658	03/14	16:46:55.107	02	EventManager.cs(88)		28D9A7BA-0164-4398-9793-5E99387850CB	NORMAL	Publishing event from TaskInstance.cs(825): TaskStop, [TaskID=28d9a7ba-0164-4398-9793-5e99387850cb]
    0CF4	1658	03/14	16:46:55.112	01	TaskExecutor.cs(843)		28D9A7BA-0164-4398-9793-5E99387850CB	FATAL	Task stopped (state=Failed, error=RmGenericError; -2147467259; WindowsHResult), search "Task Diagnostic Information" for details.
    0CF4	1658	03/14	16:46:55.116	16	ActiveJob.cs(688)			WARNING	 Fail: Task '28d9a7ba-0164-4398-9793-5e99387850cb' failed with error during execution.
    0CF4	1658	03/14	16:46:55.116	16	Task.cs(235)			NORMAL	 Changing task state from 'GenerateWorkplan' -> 'Failed' (28d9a7ba-0164-4398-9793-5e99387850cb)
    0CF4	1658	03/14	16:46:55.117	16	ActiveJob.cs(774)			NORMAL	 Begin new execution batch at task:(True)-479b3618-5b33-42ef-ac6b-2ee9d7c6db4a
    0CF4	1658	03/14	16:46:55.118	16	ActiveJob.cs(794)			NORMAL	 Dispatching task for execution:True-479b3618-5b33-42ef-ac6b-2ee9d7c6db4a
    0CF4	1658	03/14	16:46:55.384	16	ActiveJob.cs(524)			NORMAL	 Begin ExecuteTask:JobID=01405ab8-2111-4a7a-bb82-3be93e82dc65, ItemID=479b3618-5b33-42ef-ac6b-2ee9d7c6db4a
    0CF4	1658	03/14	16:46:55.385	01	TaskExecutor.cs(692)		479B3618-5B33-42EF-AC6B-2EE9D7C6DB4A	NORMAL	Setting up Fsm: verb[84021b5e-b4dc-9b27-2b7e-3b99bb1225ff]
    0	


    • Edited by TheDocFox Thursday, March 14, 2013 6:07 PM Edited to add guests OS
    Thursday, March 14, 2013 6:04 PM

Answers

  • Hi,

    DPM 2010 does not support protecting any Windows 2012 workloads including Hyper-V guests.  You must perform 2 upgrades before protection of Windows 2012 Hyper-V clusters will work.

    UPGRADE-1 = Upgrade from DPM 2010 to DPM 2012, then install UR3.  [KB2756127-Description of Update Rollup 3 for System Center 2012]
    UPGRADE-2 = Upgrade from DPM 2012 to DPM 2012 SP1

    Then install this Windows 2012 fix on the Windows Hyper-V servers.

    Virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster
    http://support.microsoft.com/kb/2799728

    NOTE: After you install the hotfix, CSV volumes do not enter paused states as frequently. Additionally, a cluster's ability to recover from expected paused states that occur when a CSV failover does not occur is improved. 

    To avoid CSV failovers, you may have to make additional changes to the computer after you install the hotfix. For example, you may be experiencing the issue described in this article because of the lack of hardware support for Offloaded Data Transfer (ODX). This causes delays when the operating system queries for the hardware support during I/O requests.

    In this situation, disable ODX by changing the FilterSupportedFeaturesMode value for the storage device that does not support ODX to 1. For more information about how to disable ODX, go to the following Microsoft website:

    General information about how to deploy ODX
    http://technet.microsoft.com/en-us/library/jj200627


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, March 14, 2013 11:27 PM
    Moderator

All replies

  • Hi,

    DPM 2010 does not support protecting any Windows 2012 workloads including Hyper-V guests.  You must perform 2 upgrades before protection of Windows 2012 Hyper-V clusters will work.

    UPGRADE-1 = Upgrade from DPM 2010 to DPM 2012, then install UR3.  [KB2756127-Description of Update Rollup 3 for System Center 2012]
    UPGRADE-2 = Upgrade from DPM 2012 to DPM 2012 SP1

    Then install this Windows 2012 fix on the Windows Hyper-V servers.

    Virtual machine enters a paused state or a CSV volume goes offline when you try to create a backup of the virtual machine on a Windows Server 2012-based failover cluster
    http://support.microsoft.com/kb/2799728

    NOTE: After you install the hotfix, CSV volumes do not enter paused states as frequently. Additionally, a cluster's ability to recover from expected paused states that occur when a CSV failover does not occur is improved. 

    To avoid CSV failovers, you may have to make additional changes to the computer after you install the hotfix. For example, you may be experiencing the issue described in this article because of the lack of hardware support for Offloaded Data Transfer (ODX). This causes delays when the operating system queries for the hardware support during I/O requests.

    In this situation, disable ODX by changing the FilterSupportedFeaturesMode value for the storage device that does not support ODX to 1. For more information about how to disable ODX, go to the following Microsoft website:

    General information about how to deploy ODX
    http://technet.microsoft.com/en-us/library/jj200627


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Thursday, March 14, 2013 11:27 PM
    Moderator
  • Mike,

    I appreciate the very thorough response! It sucks to hear that it's a 2012/2013 issue but at least I can stop freaking out about the issue. I've marked your response as the answer because it is going to take some planning and more than a few days to get things moved over and if someone stumbles across the same issue I don't want them to have to wait. If for some reason it doesn't work I'll come back. 

    Do you by chance have any good resources for a migration path from dpm 2010 to 2012? I haven't even begun the search yet but you seemed to really have a handle on it and I was hoping you might have some resources. 

    Thank you again for your help and information!!

    Thanks
    Dustin

    Friday, March 15, 2013 1:06 PM
  • Hi

    This TechNet atriclee should help you.

    Upgrading to System Center 2012 - Data Protection Manager
    http://technet.microsoft.com/en-us/library/hh848301.aspx


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT] This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, March 15, 2013 2:25 PM
    Moderator
  • Thank you very much!! It will be a hairy upgrade since we are short on hardware, and no passthrough disks to allow us to run it in a VM, but we'll get it done. Thank you for all of your help!!

    Thanks
    Dustin

    Wednesday, March 20, 2013 9:05 PM