none
DPM Service Crashes During Consistency Check RRS feed

  • Question

  • The DPM Service crashes each time at the same point performing a consistency check of a volume. We are currently running 1807

    This is the error I see in the logs

    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING Caught unhandled exception : System.OverflowException: Value was either too large or too small for an Int64.
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at System.Number.ParseInt64(String value, NumberStyles options, NumberFormatInfo numfmt)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at System.String.System.IConvertible.ToInt64(IFormatProvider provider)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.ConfigurationManager.InquiryAttributeFactory.Create(NEAttributeName name, String value)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.ConfigurationManager.RunInquiry.InquiryNE..ctor(ACInquiryResultType acResult)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.ConfigurationManager.RunInquiry.InquiryBlock.ACSendInquiryResponseSucceeded(Message msg)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Transition.Execute(Message msg)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Engine.ChangeState(Message msg)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskInstance.Process(Object dummy)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.FsmThreadFunction.Function(Object taskThreadContextObj)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
    07BC 230C 07/26 23:00:54.924 68 DpmThreadPool.cs(121) B665C198-1503-4288-B894-3CD44A266A5D WARNING    at System.Threading.ThreadPoolWorkQueue.Dispatch()
    07BC 230C 07/26 23:00:54.931 68 DpmThreadPool.cs(149) B665C198-1503-4288-B894-3CD44A266A5D WARNING Attempting to write error to the event log Value was either too large or too small for an Int64.
    07BC 230C 07/26 23:00:54.932 02 EventManager.cs(143) B665C198-1503-4288-B894-3CD44A266A5D NORMAL Logging event to NT log from DpmThreadPool.cs(163): FatalServiceError
    07BC 230C 07/26 23:00:54.944 68 DpmThreadPool.cs(170) B665C198-1503-4288-B894-3CD44A266A5D WARNING Attempting to invoke Watson
    07BC 230C 07/26 23:00:54.954 09 everettexception.cpp(762) B665C198-1503-4288-B894-3CD44A266A5D CRITICAL Exception Message = Value was either too large or too small for an Int64. of type System.OverflowException, process will terminate after generating dump

    Has anyone else had this issue? and possibly know the solution?

    Regards

    Saturday, July 27, 2019 12:37 PM

All replies

  • We are running into the same error and same error stack trying to refresh the data sources for a DPM chaining setup:

    07D8	10B8	08/19	13:47:14.323	68	RunInquiry.cs(467)		AD0490D4-D4A5-465F-B948-AD92AFA2B33F	NORMAL	Agent Inquiry started on server dpm03.somedomain.local
    07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING Caught unhandled exception : System.OverflowException: Value was either too large or too small for an Int64. 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at System.Number.ParseInt64(String value, NumberStyles options, NumberFormatInfo numfmt) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at System.String.System.IConvertible.ToInt64(IFormatProvider provider) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at Microsoft.Internal.EnterpriseStorage.Dls.ConfigurationManager.InquiryAttributeFactory.Create(NEAttributeName name, String value) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at Microsoft.Internal.EnterpriseStorage.Dls.ConfigurationManager.RunInquiry.InquiryNE..ctor(ACInquiryResultType acResult) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at Microsoft.Internal.EnterpriseStorage.Dls.ConfigurationManager.RunInquiry.InquiryBlock.ACSendInquiryResponseSucceeded(Message msg) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Transition.Execute(Message msg) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.Fsm.Engine.ChangeState(Message msg) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.TaskInstance.Process(Object dummy) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at Microsoft.Internal.EnterpriseStorage.Dls.TaskExecutor.FsmThreadFunction.Function(Object taskThreadContextObj) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(121) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING at System.Threading.ThreadPoolWorkQueue.Dispatch() 07D8 1F58 08/19 13:48:09.408 68 DpmThreadPool.cs(149) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING Attempting to write error to the event log Value was either too large or too small for an Int64. 07D8 1F58 08/19 13:48:09.408 02 EventManager.cs(143) AD0490D4-D4A5-465F-B948-AD92AFA2B33F NORMAL Logging event to NT log from DpmThreadPool.cs(163): FatalServiceError 07D8 1F58 08/19 13:48:09.423 68 DpmThreadPool.cs(170) AD0490D4-D4A5-465F-B948-AD92AFA2B33F WARNING Attempting to invoke Watson 07D8 1F58 08/19 13:48:09.423 09 everettexception.cpp(762) AD0490D4-D4A5-465F-B948-AD92AFA2B33F CRITICAL Exception Message = Value was either too large or too small for an Int64. of type System.OverflowException, process will terminate after generating dump
    We are running DPM 2016; version 5.0.409.0 on both the source (DPM we're trying to pull backups from) and destination DPM server.

    Monday, August 19, 2019 2:12 PM
  • I got this fixed by opening a case with MS support

    It was an issue with auto protection of an SQL source and Prunning for me. 

    Monday, August 19, 2019 8:14 PM
  • Fixed our issue.

    <Attribute AttributeName="IncrementalSize" AttributeValue="18446744072482050048"/>

    This value reported by the source DPM server is larger than an int64.  I'm not sure how it got that big, but after removing/fixing the associated protection group on the source DPM server, the issue went away.

    For reference, I was able to find this value by enabling the verbose logging and looking at the data returned from the source DPM server when attempting to refresh the server in the protection wizard (a giant XML line gets logged in the log) and searching for an int that would be too big for an int64.

    Wednesday, August 21, 2019 4:36 PM
  • Yeah I just looked at the email we got back once the had looked at it

    • Symptom: DPM crashing

     

    • Cause: Failure of backup of a particular database replica size issue.

     

    Resolution: Increases replica size for data source 

    They literally just changed the replica size and did a manual prune and that fixed it.

    Thursday, August 22, 2019 6:33 AM