none
RPC Service crashing when user deletes delegate Deleted Items

    Question

  • We have an administrative assistant that accesses the CEO's mailbox through Outlook 2010 and removes deleted items throughout the day.  However, when they are doing this, they occasionally run into a message indicating that the email they deleted has already been deleted or has been removed. This is in turn causes the Exchange RPC service to crash and users being temporary disconnected from Exchange.  (The Microsoft Exchange RPC Client Access service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 5000 milliseconds: Restart the service.)

    4 Application errors (4999 x2, 1026 and 1000) get logged and posted below at the bottom of this post.  We've discussed with MS support and exported/imported the users mailbox as well as migrated the CEO's mailbox to a different DB.  Nothing has really fixed the issue.  We are running Exchange 2013 CU17, so we are fully patched there.  I'm at a loss and users are aggravated with connectivity issues throughout the day.  Anyone have any ideas or alternate solutions for the user to do her daily audit of the delegates' deleted items?

    Faulting application name: Microsoft.Exchange.RpcClientAccess.Service.exe, version: 15.0.1320.0, time stamp: 0x58fe8b86
    Faulting module name: KERNELBASE.dll, version: 6.3.9600.18666, time stamp: 0x58f33794
    Exception code: 0xe0434352
    Fault offset: 0x00000000000095fc
    Faulting process id: 0x10ecc
    Faulting application start time: 0x01d311570978c5f2
    Faulting application path: C:\Program Files\Microsoft\Exchange Server\V15\bin\Microsoft.Exchange.RpcClientAccess.Service.exe
    Faulting module path: C:\Windows\system32\KERNELBASE.dll
    Report Id: 513d8f78-7d4a-11e7-811d-8231a8713008
    Faulting package full name:
    Faulting package-relative application ID:

    Application: Microsoft.Exchange.RpcClientAccess.Service.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.NullReferenceException
       at Microsoft.Exchange.Data.Storage.COWAudit.InternalGroupOperation(Microsoft.Exchange.Data.Storage.COWSettings, Microsoft.Exchange.Data.Storage.IDumpsterItemOperations, Microsoft.Exchange.Data.Storage.COWTriggerAction, Microsoft.Exchange.Data.Storage.FolderChangeOperationFlags, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.StoreObjectId, Microsoft.Exchange.Data.Storage.StoreObjectId[], Microsoft.Exchange.Data.Storage.GroupOperationResult, Boolean, Microsoft.Exchange.Data.Storage.CallbackContext)
       at Microsoft.Exchange.Data.Storage.COWAudit+<>c__DisplayClassa.<GroupOperation>b__9()
       at Microsoft.Exchange.Data.Storage.COWAudit.HandleExceptionsAtEntries(System.Action, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.CallbackContext, Boolean, System.String)
       at Microsoft.Exchange.Data.Storage.COWSession.InternalAfterFolderChange(Microsoft.Exchange.Data.Storage.COWTriggerAction, Microsoft.Exchange.Data.Storage.FolderChangeOperationFlags, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.StoreObjectId, Microsoft.Exchange.Data.Storage.StoreObjectId, System.Collections.Generic.ICollection`1<Microsoft.Exchange.Data.Storage.StoreObjectId>, Microsoft.Exchange.Data.Storage.GroupOperationResult, Microsoft.Exchange.Data.Storage.CallbackContext)
       at Microsoft.Exchange.Data.Storage.COWSession.OnAfterFolderChange(Microsoft.Exchange.Data.Storage.FolderChangeOperation, Microsoft.Exchange.Data.Storage.FolderChangeOperationFlags, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.StoreObjectId, Microsoft.Exchange.Data.Storage.StoreObjectId, System.Collections.Generic.ICollection`1<Microsoft.Exchange.Data.Storage.StoreObjectId>, Microsoft.Exchange.Data.Storage.GroupOperationResult, Microsoft.Exchange.Data.Storage.CallbackContext)
       at Microsoft.Exchange.Data.Storage.MailboxSession.OnAfterFolderChange(Microsoft.Exchange.Data.Storage.FolderChangeOperation, Microsoft.Exchange.Data.Storage.FolderChangeOperationFlags, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.StoreSession, Microsoft.Exchange.Data.Storage.StoreObjectId, Microsoft.Exchange.Data.Storage.StoreObjectId, System.Collections.Generic.ICollection`1<Microsoft.Exchange.Data.Storage.StoreObjectId>, Microsoft.Exchange.Data.Storage.GroupOperationResult, Microsoft.Exchange.Data.Storage.CallbackContext)
       at Microsoft.Exchange.Data.Storage.CoreFolder.DeleteItems(Microsoft.Exchange.Data.Storage.DeleteItemFlags, Microsoft.Exchange.Data.Storage.StoreObjectId[])
       at Microsoft.Exchange.Data.Storage.Folder.ExecuteGroupOperationAndAggregateResults(System.Collections.Generic.List`1<Microsoft.Exchange.Data.Storage.GroupOperationResult>, Microsoft.Exchange.Data.Storage.StoreObjectId[], GroupOperationDelegate)
       at Microsoft.Exchange.Data.Storage.Folder.ExecuteOperationOnObjects(ActOnFolderDelegate, ActOnItemsDelegate, Microsoft.Exchange.Data.Storage.StoreId[])
       at Microsoft.Exchange.Data.Storage.Folder.DeleteObjects(Microsoft.Exchange.Data.Storage.DeleteItemFlags, Boolean, Microsoft.Exchange.Data.Storage.StoreId[])
       at Microsoft.Exchange.Data.Storage.StoreSession.ExecuteOperationOnObjects(System.Collections.Generic.Dictionary`2<Microsoft.Exchange.Data.Storage.StoreObjectId,System.Collections.Generic.List`1<Microsoft.Exchange.Data.Storage.StoreId>>, System.Collections.Generic.List`1<Microsoft.Exchange.Data.Storage.GroupOperationResult>, ActOnObjectsDelegate)
       at Microsoft.Exchange.Data.Storage.StoreSession.Delete(Microsoft.Exchange.Data.Storage.DeleteItemFlags, Boolean, Microsoft.Exchange.Data.Storage.StoreId[])
       at Microsoft.Exchange.RpcClientAccess.Handler.DeleteMessagesSegmentedOperation.InternalDoNextBatchOperation()
       at Microsoft.Exchange.RpcClientAccess.Handler.ExceptionTranslator.TryExecuteCatchAndTranslateExceptions[[Microsoft.Exchange.RpcClientAccess.Handler.SegmentOperationResult, Microsoft.Exchange.RpcClientAccess.Handler, Version=15.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]](System.Func`1<Microsoft.Exchange.RpcClientAccess.Handler.SegmentOperationResult>, System.Func`2<Microsoft.Exchange.RpcClientAccess.Handler.SegmentOperationResult,Microsoft.Exchange.RpcClientAccess.ErrorCode>, Boolean, Microsoft.Exchange.RpcClientAccess.Handler.SegmentOperationResult ByRef, System.Exception ByRef, Microsoft.Exchange.RpcClientAccess.ErrorCode ByRef)
       at Microsoft.Exchange.RpcClientAccess.Handler.SegmentedRopOperation.DoNextBatchOperation()
       at Microsoft.Exchange.RpcClientAccess.Handler.AsyncOperationExecutor+<>c__DisplayClass9.<SegmentedOperation>b__3()
       at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Microsoft.Exchange.Common.IL.TryDelegate, Microsoft.Exchange.Common.IL.FilterDelegate, Microsoft.Exchange.Common.IL.CatchDelegate)
       at Microsoft.Exchange.RpcClientAccess.Handler.AsyncOperationExecutor.SegmentedOperation(System.Object)
       at Microsoft.Exchange.RpcClientAccess.Handler.Folder.DeleteMessages(Microsoft.Exchange.RpcClientAccess.StoreId[], Microsoft.Exchange.Data.Storage.DeleteItemFlags, Boolean, Boolean, System.Object)
       at Microsoft.Exchange.RpcClientAccess.Handler.RopHandler+<>c__DisplayClass2b.<DeleteMessages>b__2a()
       at Microsoft.Exchange.RpcClientAccess.Handler.ExceptionTranslator.TryExecuteCatchAndTranslateExceptions[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Func`1<System.__Canon>, System.Func`2<System.__Canon,Microsoft.Exchange.RpcClientAccess.ErrorCode>, Boolean, System.__Canon ByRef, System.Exception ByRef, Microsoft.Exchange.RpcClientAccess.ErrorCode ByRef)
       at Microsoft.Exchange.RpcClientAccess.Handler.RopHandlerHelper.CallHandler[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Func`1<System.__Canon>, System.Func`2<System.__Canon,Microsoft.Exchange.RpcClientAccess.ErrorCode>, Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory, System.Func`4<Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory,Microsoft.Exchange.RpcClientAccess.ErrorCode,System.Exception,System.__Canon>, System.Func`2<System.__Canon,Microsoft.Exchange.RpcClientAccess.Parser.RopId>)
       at Microsoft.Exchange.RpcClientAccess.Handler.RopHandlerHelper.CallHandler(Microsoft.Exchange.RpcClientAccess.Parser.IRopHandler, System.Func`1<Microsoft.Exchange.RpcClientAccess.Parser.RopResult>, Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory, System.Func`4<Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory,Microsoft.Exchange.RpcClientAccess.ErrorCode,System.Exception,Microsoft.Exchange.RpcClientAccess.Parser.RopResult>)
       at Microsoft.Exchange.RpcClientAccess.Handler.RopHandler.GenericRopHandlerMethod(Microsoft.Exchange.RpcClientAccess.Parser.IServerObject, System.Func`1<Microsoft.Exchange.RpcClientAccess.Parser.RopResult>, Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory, System.Func`4<Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory,Microsoft.Exchange.RpcClientAccess.ErrorCode,System.Exception,Microsoft.Exchange.RpcClientAccess.Parser.RopResult>)
       at Microsoft.Exchange.RpcClientAccess.Handler.RopHandler.StandardRopHandlerMethod(Microsoft.Exchange.RpcClientAccess.Parser.IServerObject, System.Func`1<Microsoft.Exchange.RpcClientAccess.Parser.RopResult>, Microsoft.Exchange.RpcClientAccess.Parser.IResultFactory)
       at Microsoft.Exchange.RpcClientAccess.Monitoring.ExMonHandler.DeleteMessages(Microsoft.Exchange.RpcClientAccess.Parser.IServerObject, Boolean, Boolean, Microsoft.Exchange.RpcClientAccess.StoreId[], Microsoft.Exchange.RpcClientAccess.Parser.DeleteMessagesResultFactory)
       at Microsoft.Exchange.RpcClientAccess.Parser.RopDeleteMessages.InternalExecute(Microsoft.Exchange.RpcClientAccess.Parser.IServerObject, Microsoft.Exchange.RpcClientAccess.Parser.IRopHandler, System.ArraySegment`1<Byte>)
       at Microsoft.Exchange.RpcClientAccess.Parser.InputRop.Execute(Microsoft.Exchange.RpcClientAccess.Parser.IConnectionInformation, Microsoft.Exchange.RpcClientAccess.Parser.IRopDriver, Microsoft.Exchange.RpcClientAccess.Parser.ServerObjectHandleTable, System.ArraySegment`1<Byte>)
       at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.ExecuteRops(System.Collections.Generic.List`1<System.ArraySegment`1<Byte>>, Microsoft.Exchange.RpcClientAccess.Parser.ServerObjectHandleTable, System.ArraySegment`1<Byte>, Int32, Int32, Boolean, Int32 ByRef, Microsoft.Exchange.RpcClientAccess.Parser.AuxiliaryData, Boolean, Byte[] ByRef)
       at Microsoft.Exchange.RpcClientAccess.Parser.RopDriver.Execute(System.Collections.Generic.IList`1<System.ArraySegment`1<Byte>>, System.ArraySegment`1<Byte>, Int32 ByRef, Microsoft.Exchange.RpcClientAccess.Parser.AuxiliaryData, Boolean, Byte[] ByRef)
       at Microsoft.Exchange.RpcClientAccess.Server.RpcDispatch+<>c__DisplayClass20.<Execute>b__1c()
       at Microsoft.Exchange.RpcClientAccess.Server.RpcDispatch.ExecuteWrapper(System.Func`1<ExecuteParameters>, System.Func`1<Microsoft.Exchange.RpcClientAccess.RpcErrorCode>, System.Action`1<System.Exception>)
       at Microsoft.Exchange.RpcClientAccess.Server.RpcDispatch.Execute(Microsoft.Exchange.Rpc.ProtocolRequestInfo, IntPtr ByRef, System.Collections.Generic.IList`1<System.ArraySegment`1<Byte>>, System.ArraySegment`1<Byte>, Int32 ByRef, System.ArraySegment`1<Byte>, System.ArraySegment`1<Byte>, Int32 ByRef, Boolean, Byte[] ByRef)
       at Microsoft.Exchange.RpcClientAccess.Server.WatsonOnUnhandledExceptionDispatch+<>c__DisplayClass10.<Microsoft.Exchange.RpcClientAccess.Server.IRpcDispatch.Execute>b__f()
       at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Microsoft.Exchange.Common.IL.TryDelegate, Microsoft.Exchange.Common.IL.FilterDelegate, Microsoft.Exchange.Common.IL.CatchDelegate)

    Exception Info: Microsoft.Exchange.Diagnostics.ExWatson+CrashNowException
       at Microsoft.Exchange.Diagnostics.ExWatson.CrashNow()
       at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
       at System.Threading.ThreadHelper.ThreadStart()

    Watson report about to be sent for process id: 69324, with parameters: E12IIS, c-RTL-AMD64, 15.00.1320.004, M.E.RpcClientAccess.Service, M.E.Data.Storage, M.E.D.S.COWAudit.InternalGroupOperation, System.NullReferenceException, 64c5, 15.00.1320.000.
    ErrorReportingEnabled: True

    Watson report about to be sent for process id: 69324, with parameters: E12IIS, c-RTL-AMD64, 15.00.1320.004, M.E.RpcClientAccess.Service, M.E.Data.Storage, M.E.D.S.COWAudit.InternalGroupOperation, System.NullReferenceException, 64c5, 15.00.1320.000.
    ErrorReportingEnabled: True

    Wednesday, August 9, 2017 10:11 PM

All replies

  • What cumulative update of Exchange 2013?

    Few of us have memorized all event ID numbers and few of us are inclined to look up the numbers you've posted, so please post the entire event log entry (without the XML).  You are welcome to use the handy-dandy "copy to clipboard" button in the event log detail window to assist you with this.


    Ed Crowley MVP "There are seldom good technological solutions to behavioral problems."
    Celebrating 20 years of providing Exchange peer support!

    Thursday, August 10, 2017 1:43 AM
    Moderator
  • Hi,

    How about deleting the items via OWA?

    Based on your description, I suggest you to check as follows:
    1. Grant the full access permissions to anther user and test again.
    2. Check the sever component states: Run Get-ServerComponentState -Identity <server name>.
    3. Repair the CEO’s mailbox: Run New-MailboxRepairRequest.
    4. Check the event log before this error.


    Best Regards,

    Manu Meng
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.
    Thursday, August 10, 2017 12:58 PM
    Moderator
  • Do we have similar behavior on different client machines ?

    Could you ask your user to use latest olk client  or at least olk 2013 with latest patch and see if it makes any difference. Also try and run offcat tool to identify client side issues.


    Thanks, AJ

    Thursday, August 10, 2017 2:53 PM
  • We are using Citrix XenApp published desktops so we're limited on upgrading Outlook clients.  We're in the middle of planning a migration to the cloud as well, so this will be moot in a few months, but it's still an issue for now.

    I've thought about having the delegate use OWA, but it may be a bit much for them to do so.  I did try to add a second Outlook profile to their profile and discovered that their windows profile was very slow to load.  So my last hope is that their windows profile was causing an issue and triggering this problem.  I reset their profile and am monitoring for a few days.

    Ed, we are running CU17, so the latest update.  I posted the corresponding errors in the app log at the bottom of the OP.

    Thursday, August 10, 2017 4:59 PM
  • In that case i believe your OLK profiles are in online mode ?? if yes then, afaik ... it can generate more load on exchange server as compare to OLK in cache mode.


    Thanks, AJ

    Thursday, August 10, 2017 5:14 PM
  • Yes, we are running in Online mode.  We did switch the user to Cached mode, but I believe it only caches the default profile, not any delegated mailboxes so it didn't help with the problem. 

    We did reference this KB as well, but unfortunately it didn't help:

    https://support.microsoft.com/en-us/help/3061079/rpc-client-access-service-crashes-and-event-4999-is-logged-in-exchange

    The RPC Client Access logs always show the same error at the time of crash:

    2017-08-09T21:33:00.146Z,314,21,/o=LTMAIL/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=b39dc217674d480689dfc6da9093d3ce-DELEGATENAME,,OUTLOOK.EXE,14.0.7180.5000,Classic,,,,ncacn_http,Client=MSExchangeRPC,,,,fault,00:00:01.1250000,,"Watson: [NullReferenceException] Object reference not set to an instance of an object.   at M.E.D.S.COWAudit.InternalGroupOperation(COWSettings settings, IDumpsterItemOperations dumpster, COWTriggerAction operation, FolderChangeOperationFlags flags, StoreSession sourceSession, StoreSession destinationSession, StoreObjectId destinationFolderId, StoreObjectId[] itemIds, GroupOperationResult result, Boolean onBeforeNotification, CallbackContext callbackContext)     at M.E.D.S.COWAudit.<>c__DisplayClassa.<GroupOperation>b__9()     at M.E.D.S.COWAudit.HandleExceptionsAtEntries(Action tryCode, StoreSession session, CallbackContext callbackContext, Boolean onBeforeNotification, String methodName)     at M.E.D.S.COWSession.InternalAfterFolderChange(COWTriggerAction operation, FolderChangeOperationFlags flags, StoreSession destinationSession, StoreObjectId sourceFolderId, StoreObjectId destinationFolderId, ICollection`1 itemIds, GroupOperationResult result, CallbackContext callbackContext)     at M.E.D.S.COWSession.OnAfterFolderChange(FolderChangeOperation operation, FolderChangeOperationFlags flags, StoreSession sourceSession, StoreSession destinationSession, StoreObjectId sourceFolderId, StoreObjectId destinationFolderId, ICollection`1 itemIds, GroupOperationResult result, CallbackContext callbackContext)     at M.E.D.S.MailboxSession.OnAfterFolderChange(FolderChangeOperation operation, FolderChangeOperationFlags flags, StoreSession sourceSession, StoreSession destinationSession, S",,,DELEGATEEMAILADDRESS,

    • Edited by azacura22 Thursday, August 10, 2017 5:37 PM
    Thursday, August 10, 2017 5:33 PM
  • We are using Citrix XenApp published desktops so we're limited on upgrading Outlook clients.  We're in the middle of planning a migration to the cloud as well, so this will be moot in a few months, but it's still an issue for now.

    I've thought about having the delegate use OWA, but it may be a bit much for them to do so.  I did try to add a second Outlook profile to their profile and discovered that their windows profile was very slow to load.  So my last hope is that their windows profile was causing an issue and triggering this problem.  I reset their profile and am monitoring for a few days.

    Ed, we are running CU17, so the latest update.  I posted the corresponding errors in the app log at the bottom of the OP.

    To narrow down the issue, I need to confirm whether it would be windows profile related or not, so please give me a response if you have a result.

    Best Regards,

    Manu Meng
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.
    Friday, August 11, 2017 6:27 PM
    Moderator
  • Hi,

    Was this ever solved?

    We appear to be having the same symptoms although it is with the MSExchangeMapiMailboxAppPool. We don't know what exactly is triggering it yet. We are on Exchange 2013 CU17 as well. Out of all the multi role servers we have, this is only happening on one of the servers.

    Thanks!

    Tuesday, May 29, 2018 4:34 PM
  • CU17 is dated - you would be asked to update if you called that issue into support.  

    I'm wondering if you have an OS corruption issue - have you run the DISM commands to rule that out ?


    Cheers,

    Rhoderick

    Microsoft Senior Exchange PFE

    Blog: http://blogs.technet.com/rmilne  Twitter:   LinkedIn:   Facebook:   XING:

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Wednesday, May 30, 2018 4:21 AM