none
NullReferenceException in Custom Code Activity causes Workflow Manager to crash repeatedly, eventually consumes all remaining disk space. RRS feed

  • Question

  • Hi.

    I have a workflow that waits for an incoming notification before running a custom code activity. Once the notification is received and the custom code is called, if the custom code activity faults with a NullReferenceException then Workflow Manager proceeds to report a number of error and critical events to the EventLog (see below) before crashing (with crash dump) and restarting. On restart, Workflow Manager attempts to resume the workflow by calling the faulting code. It will keep doing this until the crash dumps (each approx 360MB) consume all remaining disk space; at a rate of one crash dump every 45 seconds. The only way to break the cycle before disk space is exhausted is to delete the published workflow.

    To rule-out any additional interaction within the code, I pared-down my custom activity to a simple "throw new NullReferenceException" in the Execute() body and got the same result.

    Incidentally, the crash dumps are written to "C:\Windows\System32\%LOCALAPPDATA%\CrashDumps" which is not a location where I would expect to find crash dumps for a service.


    Regards,

    Phil

    Event #1 (many instances):

    Dispatcher encountered an unexpected exception: System.NullReferenceException: Object reference not set to an instance of an object.

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)

    Event #2 (Critical):

    Dispatcher failfast is causing the Workflow Manager backend to terminate. Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.NullReferenceException: Object reference not set to an instance of an object.

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)

       --- End of inner exception stack trace ---

    ---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<---

    Event #3:

    Application terminating due to FailFast. Message : An unrecoverable error occurred. For diagnostic purposes, this English message is associated with the failure: 'The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures.'., Exception: System.AggregateException: The dispatcher failure rate has reached 11%. InnerException is the last failure; see traces for other failures. ---> System.NullReferenceException: Object reference not set to an instance of an object.

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)

       --- End of inner exception stack trace ---

    ---> (Inner Exception #0) System.NullReferenceException: Object reference not set to an instance of an object.

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSessionOnFailedConfiguration(String suspensionReason)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.SystemSuspendSession(String suspensionReason, SystemSuspendType suspendType)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.Isolate(AsyncCompletion callback, IAsyncResult result)

       at Microsoft.Activities.Dispatcher.DispatchLoopInstanceAsyncResult.ExceptionHandlingFrame(IAsyncResult result)<---

    .

    Monday, April 22, 2013 9:21 PM

All replies

  • Hi, did you find a solution about this? The only one I found is to completely remove the scope and then add it again with all its wf...but of course I prefer something else... 

    Thanks

    Monday, October 6, 2014 3:24 PM
  • Hi, I have the same issue, did you then find another solution? 

    Thanks, 

    Ale

    Friday, June 26, 2015 1:58 PM
  • same for me:

    • i deleted all site collection to stop all running workflows
    • removed farm from service bus and workflow manager
    • uninstalled both service bus and its CUS 
    • uninstalled both workflow manager CU and workflow manager

    then did fresh installation of workflow manger and service bus with all latest update

    and able to join workflow and service to farm then getting same errors as mentioned above

    Thursday, August 20, 2015 10:54 AM