locked
Mailbox Assistants crashing when processing certain mailboxes RRS feed

  • Question

  • We recently completed a migration from Exchange 2010 to 2016 on premise with O365 personal archiving.  We setup a retention policy and custom retention tags so our users could have messages moved to their archive as they desired.  It worked well in 2010, but we recently discovered that several mailboxes were no longer working since being moved to 2016.  When the mailbox assistants attempts to process to mailbox, it throws 4 errors in the event log, none of which are very helpful to me.  I'm able to trigger the error by running the start-managedfolderassistant -identity XYZ on the affected mailboxes (probably less than 5% of all of our mailboxes.  The problem is not isolated to any particular server or database, so it seems to be something specific with the mailbox.  All servers in DAG or Server 2012 R2/Exchange 2016 CU4.

    Here are the 4 Event Messages (in order):

    First Event:

    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="MSExchange Common" /> 
      <EventID Qualifiers="16388">4999</EventID> 
      <Level>2</Level> 
      <Task>1</Task> 
      <Keywords>0x80000000000000</Keywords> 
      <TimeCreated SystemTime="2017-01-09T17:53:34.000000000Z" /> 
      <EventRecordID>3825229</EventRecordID> 
      <Channel>Application</Channel> 
      <Computer>MBX1603</Computer> 
      <Security /> 
      </System>
    - <EventData>
      <Data>7812</Data> 
      <Data>E12IIS</Data> 
      <Data>c-RTL-AMD64</Data> 
      <Data>15.01.0669.032</Data> 
      <Data>MSExchangeMailboxAssistants</Data> 
      <Data>M.E.MailboxAssistants.Assistants</Data> 
      <Data>M.E.M.A.E.ELCAssistant.InvokeInternalAssistant</Data> 
      <Data>System.NullReferenceException</Data> 
      <Data>30b9</Data> 
      <Data>15.01.0669.032</Data> 
      <Data>True</Data> 
      <Data>False</Data> 
      <Data>MSExchangeMailboxAssistants</Data> 
      <Data /> 
      </EventData>
      </Event>

    Second Event:

    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Windows Error Reporting" /> 
      <EventID Qualifiers="0">1001</EventID> 
      <Level>4</Level> 
      <Task>0</Task> 
      <Keywords>0x80000000000000</Keywords> 
      <TimeCreated SystemTime="2017-01-09T17:53:37.000000000Z" /> 
      <EventRecordID>3825230</EventRecordID> 
      <Channel>Application</Channel> 
      <Computer>MBX1603</Computer> 
      <Security /> 
      </System>
    - <EventData>
      <Data>127732707094</Data> 
      <Data>5</Data> 
      <Data>E12IIS</Data> 
      <Data>Not available</Data> 
      <Data>0</Data> 
      <Data>c-RTL-AMD64</Data> 
      <Data>15.01.0669.032</Data> 
      <Data>MSExchangeMailboxAssistants</Data> 
      <Data>M.E.MailboxAssistants.Assistants</Data> 
      <Data>M.E.M.A.E.ELCAssistant.InvokeInternalAssistant</Data> 
      <Data>System.NullReferenceException</Data> 
      <Data>30b9</Data> 
      <Data>15.01.0669.032</Data> 
      <Data /> 
      <Data /> 
      <Data>C:\Windows\Temp\30d62b49-8290-4bbd-87c4-ff5b636a3821\report.txt C:\Windows\Temp\30d62b49-8290-4bbd-87c4-ff5b636a3821\report.xml</Data> 
      <Data>C:\ProgramData\Microsoft\Windows\WER\ReportArchive\NonCritical_c-RTL-AMD64_2dc258679c5a7f1af2fba0341f80616b8f898dce_00000000_264b3fb5</Data> 
      <Data /> 
      <Data>0</Data> 
      <Data>89db791a-d694-11e6-80c7-1402ec33af6b</Data> 
      <Data>0</Data> 
      <Data>57e5a87fe376b111a8eded2a846a4e14</Data> 
      </EventData>
      </Event>

    Third Event:

    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="Windows Error Reporting" /> 
      <EventID Qualifiers="0">1001</EventID> 
      <Level>4</Level> 
      <Task>0</Task> 
      <Keywords>0x80000000000000</Keywords> 
      <TimeCreated SystemTime="2017-01-09T17:53:37.000000000Z" /> 
      <EventRecordID>3825231</EventRecordID> 
      <Channel>Application</Channel> 
      <Computer>MBX1603</Computer> 
      <Security /> 
      </System>
    - <EventData>
      <Data /> 
      <Data>0</Data> 
      <Data>E12IIS</Data> 
      <Data>Not available</Data> 
      <Data>0</Data> 
      <Data>c-RTL-AMD64</Data> 
      <Data>15.01.0669.032</Data> 
      <Data>MSExchangeMailboxAssistants</Data> 
      <Data>M.E.MailboxAssistants.Assistants</Data> 
      <Data>M.E.M.A.E.ELCAssistant.InvokeInternalAssistant</Data> 
      <Data>System.NullReferenceException</Data> 
      <Data>30b9</Data> 
      <Data>15.01.0669.032</Data> 
      <Data /> 
      <Data /> 
      <Data>C:\Windows\Temp\30d62b49-8290-4bbd-87c4-ff5b636a3821\report.txt C:\Windows\Temp\30d62b49-8290-4bbd-87c4-ff5b636a3821\report.xml</Data> 
      <Data /> 
      <Data /> 
      <Data>0</Data> 
      <Data>89db791a-d694-11e6-80c7-1402ec33af6b</Data> 
      <Data>262144</Data> 
      <Data /> 
      </EventData>
      </Event>

    Fourth Event:

    - <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    - <System>
      <Provider Name="MSExchange Assistants" /> 
      <EventID Qualifiers="32772">9041</EventID> 
      <Level>3</Level> 
      <Task>1</Task> 
      <Keywords>0x80000000000000</Keywords> 
      <TimeCreated SystemTime="2017-01-09T17:53:37.000000000Z" /> 
      <EventRecordID>3825232</EventRecordID> 
      <Channel>Application</Channel> 
      <Computer>MBX1603</Computer> 
      <Security /> 
      </System>
    - <EventData>
      <Data>MSExchangeMailboxAssistants</Data> 
      <Data>Microsoft.Exchange.Assistants.AIGrayException ---> Microsoft.Exchange.Common.GrayException ---> System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Exchange.MailboxAssistants.Assistants.ELC.ELCAssistant.InvokeInternalAssistant(MailboxSession mailboxSession, InvokeArgs invokeArgs, List`1 customDataToLog, Int32 totalAttempts) at Microsoft.Exchange.MailboxAssistants.Assistants.ELC.ELCAssistant.DoWork(AssistantTaskContext context) at Microsoft.Exchange.Assistants.TimeBasedDatabaseJob.ProcessStoreMailbox(MailboxQueueItem item, AssistantTaskContext context) at Microsoft.Exchange.Assistants.TimeBasedDatabaseJob.ProcessStoreMailboxUnderPoisonControl(MailboxQueueItem item, AssistantTaskContext context) at Microsoft.Exchange.Assistants.TimeBasedDatabaseJob.<>c__DisplayClassc.<ProcessMailboxUnderPoisonControl>b__a() at Microsoft.Exchange.Assistants.Util.<>c__DisplayClass4.<CoreCatchMeIfYouCan>b__1() at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate) at Microsoft.Exchange.Diagnostics.ExceptionUtil.TryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate, Action`1 finallyDelegate) at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate) --- End of inner exception stack trace --- at Microsoft.Exchange.Common.GrayException.ExceptionCatcher(Object exception) at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate) at Microsoft.Exchange.Assistants.Util.CoreCatchMeIfYouCan(Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions) at Microsoft.Exchange.Assistants.Util.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions) --- End of inner exception stack trace --- at Microsoft.Exchange.Assistants.Util.TraceAndThrow(Action function, AIException aiException, String nonLocalizedAssistantName) at Microsoft.Exchange.Assistants.Util.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions) at Microsoft.Exchange.Assistants.Base.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions)</Data> 
      </EventData>
      </Event>

    Please let me know if you have any suggestions on how to troubleshoot this further or potential solutions.  Everything I have found so far doesn't seem to apply to 2016.  Thanks in advance.


    • Edited by Jeff Foydl Monday, January 9, 2017 6:15 PM
    Monday, January 9, 2017 6:11 PM

Answers

  • The final solution turned out to be removing the retention policy from the user, then waiting 48+ hours before reapplying the retention policy.  It not works properly and is processing the mailbox again.  I tried this with the few other mailboxes and it worked with them as well.  My guess is that some how the associated tags were corrupted during migration and Exchange didn't know what to do when it hit those tagged messages or folders.
    • Marked as answer by Jeff Foydl Monday, January 23, 2017 3:51 PM
    Monday, January 23, 2017 3:51 PM

All replies

  • Hi,

    What's the description in event 1001 and 9041? That will be very helpful.

    If this problem is not isolated to any particular server or database, and it seems to be something specific with the mailbox. You can try the following several things for troubleshooting.

    1. Move these mailboxes to another new mailbox database on another mailbox server.
    2. Schedule your time to run The Microsoft Exchange Mailbox Assistants service on all Exchange 2016 Servers.
    3. Run the following command to detect and repair the mailbox assistant's health on all Exchange 2016 servers, please post the result.

    Test-AssistantHealth -ServerName ServerName -IncludeCrashDump -ResolveProblems | FL


    Best Regards,

    Lynn-Li
    TechNet Community Support


    Please remember to mark the replies as answers.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    • Proposed as answer by Lynn-Li Monday, January 16, 2017 3:23 AM
    • Unproposed as answer by Jeff Foydl Tuesday, January 17, 2017 7:20 PM
    Tuesday, January 10, 2017 7:37 AM
  • I moved the mailbox to a different server and and database, but I know even less as to what is going on now.  The ELCLastSuccessTimestamp is getting updated and there are no longer crash messages in the event viewer, but nothing is being processed in the mailbox according to the retention policy.  I removed and re-added the retention policy last week, so I'm going to wait and see if the folders that are set to 1 week process tonight automatically before I decide if it is working or not.
    Monday, January 16, 2017 10:22 PM
  • After restarting the servers, the 9041 error comes back now on the server hosting this mailbox.

    The description is:

    Event 9041, MSExchange Assistants

    Service MSExchangeMailboxAssistants.  An exception has been thrown: Microsoft.Exchange.Assistants.AIGrayException ---> Microsoft.Exchange.Common.GrayException ---> System.NullReferenceException: Object reference not set to an instance of an object.
       at Microsoft.Exchange.MailboxAssistants.Assistants.ELC.ELCAssistant.InvokeInternalAssistant(MailboxSession mailboxSession, InvokeArgs invokeArgs, List`1 customDataToLog, Int32 totalAttempts)
       at Microsoft.Exchange.MailboxAssistants.Assistants.ELC.ELCAssistant.DoWork(AssistantTaskContext context)
       at Microsoft.Exchange.Assistants.TimeBasedDatabaseJob.ProcessStoreMailbox(MailboxQueueItem item, AssistantTaskContext context)
       at Microsoft.Exchange.Assistants.TimeBasedDatabaseJob.ProcessStoreMailboxUnderPoisonControl(MailboxQueueItem item, AssistantTaskContext context)
       at Microsoft.Exchange.Assistants.TimeBasedDatabaseJob.<>c__DisplayClassc.<ProcessMailboxUnderPoisonControl>b__a()
       at Microsoft.Exchange.Assistants.Util.<>c__DisplayClass4.<CoreCatchMeIfYouCan>b__1()
       at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
       at Microsoft.Exchange.Diagnostics.ExceptionUtil.TryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate, Action`1 finallyDelegate)
       at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
       --- End of inner exception stack trace ---
       at Microsoft.Exchange.Common.GrayException.ExceptionCatcher(Object exception)
       at Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
       at Microsoft.Exchange.Assistants.Util.CoreCatchMeIfYouCan(Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions)
       at Microsoft.Exchange.Assistants.Util.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions)
       --- End of inner exception stack trace ---
       at Microsoft.Exchange.Assistants.Util.TraceAndThrow(Action function, AIException aiException, String nonLocalizedAssistantName)
       at Microsoft.Exchange.Assistants.Util.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions)
       at Microsoft.Exchange.Assistants.Base.CatchMeIfYouCan(Boolean translateToPermanentException, Action function, String nonLocalizedAssistantName, ISet`1 permanentExceptions)

    Tuesday, January 17, 2017 7:23 PM
  • I moved the mailbox to a different server and and database, but I know even less as to what is going on now.  The ELCLastSuccessTimestamp is getting updated and there are no longer crash messages in the event viewer, but nothing is being processed in the mailbox according to the retention policy.  I removed and re-added the retention policy last week, so I'm going to wait and see if the folders that are set to 1 week process tonight automatically before I decide if it is working or not.

    The Microsoft Exchange Mailbox Assistants service is broken now, and retention policy is relied on it, so no emails are being processed according to retention policy. We need to fix that first, then check retention policy.

    Have you tried my suggested command Test-AssistantHealth?

    And do you have mail flow problems on these mailboxes or on exchange 2016 mailbox server?

    Similar thread about event 9041 for reference, make sure DNS is configured properly on NIC.

    https://social.technet.microsoft.com/Forums/office/en-US/30e36211-5f7f-4547-a29a-2455a14b63f5/internal-mail-flow-not-working-and-event-id-9041?forum=Exch2016MFSM


    Best Regards,

    Lynn-Li
    TechNet Community Support


    Please remember to mark the replies as answers.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, January 18, 2017 2:56 AM

  • The Microsoft Exchange Mailbox Assistants service is broken now, and retention policy is relied on it, so no emails are being processed according to retention policy. We need to fix that first, then check retention policy.

    Have you tried my suggested command Test-AssistantHealth?

    And do you have mail flow problems on these mailboxes or on exchange 2016 mailbox server?

    Similar thread about event 9041 for reference, make sure DNS is configured properly on NIC.

    https://social.technet.microsoft.com/Forums/office/en-US/30e36211-5f7f-4547-a29a-2455a14b63f5/internal-mail-flow-not-working-and-event-id-9041?forum=Exch2016MFSM


    Best Regards,

    Lynn-Li
    TechNet Community Support


    Please remember to mark the replies as answers.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    The mailbox assistants work fine on nearly all of our mailboxes, there are just specific mailboxes that cause the process to crash.  The test-assistanthealth cmdlet does not detect any problems with the Assistant server on any of my servers.

    No, we don't have any mailflow or DNS problems.  I don't think I would be worried about retention policies and mailbox assistants if my users couldn't send email.

    Wednesday, January 18, 2017 2:28 PM
  • Ok,

    Since the mailbox assistants works fine, then check if ExchangeLegacyDN attribute of the server object matches the ExchangeLegacyDN attribute for the effected user.

    Get-ExchangeServer 16Servername |FL name, ExchangeLegacyDN

    Get-Mailbox UserName | FL Displayname, ServerLegacyDN


    Best Regards,

    Lynn-Li
    TechNet Community Support


    Please remember to mark the replies as answers.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, January 20, 2017 10:06 AM
  • The final solution turned out to be removing the retention policy from the user, then waiting 48+ hours before reapplying the retention policy.  It not works properly and is processing the mailbox again.  I tried this with the few other mailboxes and it worked with them as well.  My guess is that some how the associated tags were corrupted during migration and Exchange didn't know what to do when it hit those tagged messages or folders.
    • Marked as answer by Jeff Foydl Monday, January 23, 2017 3:51 PM
    Monday, January 23, 2017 3:51 PM