none
RPC Requests failed (%) on Passive DAG Members RRS feed

  • Question

  • We're on CU2 on Server 2019 Core. We have a 4 server DAG, 2 servers in each of our sites. There are 3 HA database copies and a lagged replay on the 4th server. None of these systems are in production yet except 1 which is just doing SMTP relay. All of the systems seem to be fine performance wise.

    Our monitoring system though, SolarWinds Server and Application Monitor, is showing that all 3 of the passive nodes have a high percentage of failed RPC Requests. I've looked at \Logging\RPC Client Access\ for info. And I'm seeing a repeated connections from Microsoft.Exchange.RpcClientAccess.Monitoring. It appears to be connecting to a health monitoring mailbox?

    Here is a sample of the Failure field from the log.

      failures
      RpcEndPoint: [ServerTooBusyException] Client is being backed off -> [ClientBackoffException] Mailbox was moved to a different mailbox server. A client needs to retry. -> [IllegalCrossServerConnectionException] Cannot open mailbox /o=MY Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=47c455b6918749109f123569a82d1bfd-Healt. -> [MapiExceptionIllegalCrossServerConnection] Monitoring mailbox [] with application ID [Client=MSExchangeRPC] is not allowed to make cross-server calls from [AD-EXCH19-MS01.domain.com] to [PD-EXCH19-MS01.domain.com] [diag::AAAOAAAA/wAAAAAAAAAAAgAAAAA=]


    • Edited by infinit_e Thursday, September 12, 2019 5:49 PM Updated info
    Thursday, September 12, 2019 4:57 PM

All replies

  • Hi

    Have you tried to remove the health mailbox and restart the exchange health service?


    Hope this helps. Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Thursday, September 12, 2019 6:29 PM
    Moderator
  • I had not tried that. Would it be best to remove just this one, or clear all of them? It appears I have just shy of 80 HealthMailboxes, and about a half dozen appear to have validation errors.
    Thursday, September 12, 2019 7:15 PM
  • Start with the one that is reporting the error and see after that if the others report an error.

    Hope this helps. Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Thursday, September 12, 2019 7:23 PM
    Moderator
  • I did as you suggested. Removed the problem mailbox and restarted the health service. The problem came back, but now a different mailbox is referenced.

    Inspecting the mailbox shows the DisplayName as 'HealthMailbox-AD-EXCH19-MS01-DB-02', but the server is 'PD-EXCH19-MS01'. The latter is the server where DB-02 is mounted and active.

    Friday, September 13, 2019 3:31 PM
  • I went ahead and removed all the health mailboxes then restarted services to create new ones. Still seeing the same error in the RPC log but with a new mailbox.
    Friday, September 13, 2019 6:49 PM
  • maybe open a case with Microsoft, it might be something in CU2, not saying it is but if the re-creation does not fix it let them look.

    Hope this helps. Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Friday, September 13, 2019 6:56 PM
    Moderator
  • Hi,

     

    I am writing here to confirm with you how the thing going now?

     

    If you need further help, please provide more detailed information, so that we can give more appropriate suggestions.

     

    Regards,

    Kelvin Deng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com


    Wednesday, September 18, 2019 1:22 AM