locked
ActiveSync Clients lost their connection RRS feed

  • Question

  • Hi,

    I have a strange behavor with Exchange 2013, and I need some extra help to identify the culprit.

    For the technical context, we have a pair of Exchange 2013 servers, and a Netscaler V12 appliance which acts a load balancer.
    Each of is server (Let's say SRV01 and SRV02) is hosting a mailbox database as master (Let's say DB01 and DB02), and the other one as passive.
    Since a few weeks, we noticed that application pool "MSExchangeSyncAppPool" fails on the server hosting DB02 (SRV02).
    As a matter of fact, service is degraded, and some of our mobile devices are not syncing anymore.

    A further analysis showed that, on the server hosting DB02 active database (SRV02), IIS Active connections were growing over the time, but were never ended. The only way to solve the problem was to recycle the application pool, and IIS connections were released.

    We used this article as a basis for investigation : https://support.microsoft.com/en-us/help/4456227/activesync-clients-cannot-connect-or-synchronizing-delayed-in-exchange
    We noticed that : 

    - Indeed ActiveSync Application pool memory usage went from 200Mo to 5Go, on SRV02, in a few days.
    - W3SVC_W3WP/Active Requests counter for the MSExchangeSyncAppPool is growing, on SRV02, until recycling it.

    We first act on "\Exchange Server\V15\FrontEnd\HttpProxy\sync\web.config", in order to increase value "HttpProxy.ConcurrencyGuards.TargetBackendLimit". We upgraded it from 150 to 5000. 
    At first, it seemed to bring a better situation. But the problem is now happening between day 2 and 4 after the recycle.

    We dig a bit further with powershell scripts, in order to properly investigate ActiveSync connections from our mobiles.
    Here is the one we used : https://gallery.technet.microsoft.com/scriptcenter/ActiveSyncReport-script-a2417a84#content
    With this, we generate a report for a 3 days timelapse.

    We noticed that the first user in the list has 18000 hits recorded for the timelapse, alongside 16000 pings.
    If we examine this user's activity during the day, we see : 

     - During 23 of the 24 hours daytime, we have a stable amount of hits/pings (25 to 75)
     - For the remaining hour, we reach a peak (From 2249 hits / 2220 pings  to 7402 hits / 7377 pings)

    These users are connecting to Exchange with Microsoft Outlook app from Google PlayStore. Nothing unofficial.
    So you understand us when we deduced that this was quite unusual to see high numbers in short period of time.

    According to our research, it may seem that ActiveSync ping requestes are stranded. 
    We can always recycle the application pool manually or with a scheduled task. But we'd rather like to understand why endpoints are sudenly acting like this.

    Any help ?

    Thanks a lot.
    Monday, February 17, 2020 1:29 PM

All replies

  • Hi

    What CU of Exchange 2013 are you on?

    If you run both DB's on 1 server do you see the same problem?

    Are your outlook clients also affected or just activesync? Does the problem only happen with the netscalar in place? Have you tried to bypass the netscalar and see what happens?

    Can you check the IIS logs for a corrupt activesync partnership and does it only affect android or iOS as well?


    Hope this helps. Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    Monday, February 17, 2020 5:15 PM
  • Hi,

    First, thanks for the answer.

    What CU of Exchange 2013 are you on?

    I'm running Exchange 2013 CU19 on both servers.

    If you run both DB's on 1 server do you see the same problem?

    I didn't took the risk. Right now only a part of my users (The ones trying to connect to the failing app pool through netscaler) are impacted when the problem occurs.

    Are your outlook clients also affected or just ActiveSync? Does the problem only happen with the NetScaler in place? Have you tried to bypass the Netscaler and see what happens?

    Immediate symptom is the fact that our mobiles can't sync their mailboxes anymore.
    Netscalers are in place since 4 years, and the problem started to appear at mid-january this year.
    This date may be related to a new database that we filled with mailboxes coming from another Exchange, which was in 2010.
    Finally, It's gonna be difficult to avoid using Netscalers. They're acting as HTTPS proxies, which means that avoiding them will result in putting Exchange in DMZ. This is a no-go, as you can imagine ;-)

    Can you check the IIS logs for a corrupt activesync partnership and does it only affect android or iOS as well?

    Can you tell me how to find them ?
    What I can say is that all the high ranking hits/pings mobile devices are using the Microsoft Outlook app from Google Play Store.

    Again, thanks,


    • Edited by Raph12345 Tuesday, February 18, 2020 7:23 AM
    Tuesday, February 18, 2020 7:06 AM
  • Hi Raph12345,

    You can check the IIS log under the path C:\inetpub\logs\LogFiles\W3SVC1. You can take the steps mentioned in this article How to use Excel to analyse IIS Logs to check if there is a corrupt activesync partnership.

    Regards,

    Beverly Gao


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Tuesday, February 18, 2020 8:56 AM
  • Hi,

    I've looked for our IIS Logs, and nothing fancy so far.

    Another thing that we've spotted : As I said, we've got a Netscaler appliance which acts as a load balancer.

    To monitor the load on each Exchange/IIS server, we use Centreon/NSClient++ software to grab this performance counter : "\Web Service(*)\Current Connections"

    As you can see, It seems that SRV02 never releases IIS connections. This, until I recycle the "MSExchangeSyncAppPool".

    Is it related to ping commands issued by mobile devices ?

    Wednesday, February 19, 2020 8:02 AM
  • Hi Raph12345,

    Did you get any errors in the event viewer? This is a known issue that occurs when ActiveSync Ping commands become stranded in the application pool. For Exchange Server 2013, to work around this issue, periodically cycle the ActiveSync application pool for all the servers to release the stranded commands. For details, you can refer to this official KB.

    Regards,

    Beverly Gao


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Monday, February 24, 2020 10:24 AM
  • Hi, I'm here to confirm with you if your issue has been resolved. If the problem is successfully solved, you can share your solution and mark them or the helpful reply as answer, this will make answer searching in the forum easier and be beneficial to other community members as well.

    Regards,

    Beverly Gao


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Thursday, February 27, 2020 5:47 AM