locked
Exchange 2016 constant performance issues RRS feed

  • Question

  • Hi

    We have Exchange 2016 setup as follows:

    2 x Client Access Servers using NLB to load balance

    2x Mail servers hosting 5 databases in a DAG which are also setup in a Microsoft cluster to provide failover.

    The CAS servers are used as the witness server

    Versions:

    Name                : BQEXM01
    Edition             : Standard
    AdminDisplayVersion : Version 15.1 (Build 544.27)

    Name                : BQEXC02
    Edition             : Standard
    AdminDisplayVersion : Version 15.1 (Build 544.27)

    Name                : BQEXC01
    Edition             : Standard
    AdminDisplayVersion : Version 15.1 (Build 544.27)

    Name                : BQEXM02
    Edition             : StandardEvaluation
    AdminDisplayVersion : Version 15.1 (Build 845.34)

    Note: Server BQEXM02 has recently been rebuilt due to constant crashing hence the version is newer.  We intend to bring the other servers inline asap.

    So the problem we are having is that every Monday morning we are experiencing the servers running slowly.  By this I mean that email delivery externally (inbound & outbound) is delayed and inernal also suffers with delays of up to 20-30 minutes.

    To give you a background we have 531 mailboxes, 58 Shared Mailboxes and 12 REsource mailboxes.

    The 531 mailboxes, apart from 184, are all accessed using Outlook 2016 primarily with up to 70-100 accessed via mobile devices as well.  The other 184 mailboxes are for our retail outlets and are accessed only using OWA on between 1-3 PCs simultateously, so anywhere from 184 to 552 OWA connections to these 184 mailboxes.

    The servers are all sitting on a Microsoft Cluster and the OS is serverd from SSD and the data is on 7.2K SAS drives.  We use Starwind to create the Virtual SAN that all this data sits on and the physical servers have 12GB SAS cards with multiple 10GB network cars to provide the syncronisation of the data in the virtual SAN.

    I must point out, I was not involved in building this setup so do not know the in and outs of it in detail.

    I belive that the problem is one of two things:

    1. Slow disk access, but this is hard to gauge as Starwinds gets in the way and also doesn't explain why we dont see the problem more during the week.
    2. The influx of hundreds of users accessing the server all at once between 8:15am - 9:15am after the weeken which could either relate back to point 1 above, or there is another underlying issue that is causing Exchnage to grid to a halt on a Monday morning.

    Sorry I haven't been able to give you any real specifics, but just ask and I will do my best to post any info required.

    Anyone with any ideas, similar experiences or thought on areas to monitor/check etc would be very much appreciated.

    Thanks for any help you can provide.
    Monday, July 17, 2017 1:24 PM

All replies

  • Rob,

    Are all Outlook clients configured for cached mode? Are you seeing any back pressure events? Exchange 2016 doesn't offer a CAS role, all roles are collocated so how do you have 2 CAS?

    Very Respectfully,
    Dana Garcia - MCP (Designing and Deploying Microsoft Exchange Server 2016)

    Monday, July 17, 2017 1:54 PM
  • Monday, July 17, 2017 2:01 PM
  • Rob,

    Are all Outlook clients configured for cached mode? Are you seeing any back pressure events? Exchange 2016 doesn't offer a CAS role, all roles are collocated so how do you have 2 CAS?

    Very Respectfully,
    Dana Garcia - MCP (Designing and Deploying Microsoft Exchange Server 2016)

    Yes all clients are configured for Cached mode.

    We have also Disabled Download Shared Folders via a GPO.

    I agree there is no CAS role, but the person who set this up insists that best practice it to have two server that are not part of the DAG to act as a point of entry for all clients connecting.  For me this seems ridiculous as it is requiring 2 Exchange licenses and all these servers are doing is proxying or redirecting the client connections to the server holding the DB/Mailbox.

    Could you elaborate on what you mean by a back pressure event?  I did some research and came up with this info if this is what you mean?

    Resource            : PrivateBytes
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=75 HighToMedium=73 LowToMedium=72 MediumToLow=71]
    Pressure            : 2

    Resource            : QueueLength[SubmissionQueue]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=15000 HighToMedium=10000 LowToMedium=9999 MediumToLow=2000]
    Pressure            : 0

    Resource            : SystemMemory
    CurrentResourceUse  : High
    PreviousResourceUse : High
    PressureTransitions : [PressureTransitions: MediumToHigh=94 HighToMedium=89 LowToMedium=88 MediumToLow=84]
    Pressure            : 96

    Resource            : UsedVersionBuckets[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=1500 HighToMedium=1000 LowToMedium=999 MediumToLow=800]
    Pressure            : 1

    Resource            : DatabaseUsedSpace[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=99 HighToMedium=97 LowToMedium=96 MediumToLow=94]
    Pressure            : 71

    Resource            : UsedDiskSpace[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=99 HighToMedium=90 LowToMedium=89 MediumToLow=80]
    Pressure            : 72

    Resource            : UsedDiskSpace[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=99 HighToMedium=90 LowToMedium=89 MediumToLow=80]
    Pressure            : 72


    Monday, July 17, 2017 3:35 PM
  • Rob,

    Are all Outlook clients configured for cached mode? Are you seeing any back pressure events? Exchange 2016 doesn't offer a CAS role, all roles are collocated so how do you have 2 CAS?

    Very Respectfully,
    Dana Garcia - MCP (Designing and Deploying Microsoft Exchange Server 2016)

    Yes all clients are configured for Cached mode.

    We have also Disabled Download Shared Folders via a GPO.

    I agree there is no CAS role, but the person who set this up insists that best practice it to have two server that are not part of the DAG to act as a point of entry for all clients connecting.  For me this seems ridiculous as it is requiring 2 Exchange licenses and all these servers are doing is proxying or redirecting the client connections to the server holding the DB/Mailbox.

    Could you elaborate on what you mean by a back pressure event?  I did some research and came up with this info if this is what you mean?

    Resource            : PrivateBytes
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=75 HighToMedium=73 LowToMedium=72 MediumToLow=71]
    Pressure            : 2

    Resource            : QueueLength[SubmissionQueue]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=15000 HighToMedium=10000 LowToMedium=9999 MediumToLow=2000]
    Pressure            : 0

    Resource            : SystemMemory
    CurrentResourceUse  : High
    PreviousResourceUse : High
    PressureTransitions : [PressureTransitions: MediumToHigh=94 HighToMedium=89 LowToMedium=88 MediumToLow=84]
    Pressure            : 96

    Resource            : UsedVersionBuckets[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=1500 HighToMedium=1000 LowToMedium=999 MediumToLow=800]
    Pressure            : 1

    Resource            : DatabaseUsedSpace[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=99 HighToMedium=97 LowToMedium=96 MediumToLow=94]
    Pressure            : 71

    Resource            : UsedDiskSpace[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=99 HighToMedium=90 LowToMedium=89 MediumToLow=80]
    Pressure            : 72

    Resource            : UsedDiskSpace[E:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data]
    CurrentResourceUse  : Low
    PreviousResourceUse : Low
    PressureTransitions : [PressureTransitions: MediumToHigh=99 HighToMedium=90 LowToMedium=89 MediumToLow=80]
    Pressure            : 72


    Rob,

    In the event logs you will see an event that states that Exchange is taking an action due to back pressure. Here is a example:

    Log Name:      Application
    Source:        MSExchangeTransport
    Date:          01/08/2014 08:07:39
    Event ID:      15004
    Task Category: ResourceManager
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      myserver.mydomain.local
    Description:
    The resource pressure increased from Medium to High.
    
    The following resources are under pressure:
    Version buckets = 358 [High] [Normal=80 Medium=120 High=200]
    
    The following components are disabled due to back pressure:
    Inbound mail submission from Hub Transport servers
    Inbound mail submission from the Internet
    Mail submission from Pickup directory
    Mail submission from Replay directory
    Mail submission from Mailbox server
    Mail delivery to remote domains
    Content aggregation
    Mail resubmission from the Message Resubmission component.
    Mail resubmission from the Shadow Redundancy Component
    
    The following resources are in normal state:
    Queue database and disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que") = 46% [Normal] [Normal=95% Medium=97% High=99%]
    Queue database logging disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\") = 48% [Normal] [Normal=94% Medium=96% High=98%]
    Private bytes = 3% [Normal] [Normal=71% Medium=73% High=75%]
    Physical memory load = 62% [limit is 94% to start dehydrating messages.]
    Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]
    Temporary Storage disk space ("C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp") = 48% [Normal] [Normal=95% Medium=97% High=99%]

    Very Respectfully,
    Dana Garcia - MCP (Designing and Deploying Microsoft Exchange Server 2016)

    Monday, July 17, 2017 6:29 PM
  • Thank you for the update.

    I have searched the Application Log and cannot see any events with the ID 15004 or any 1500X.

    The events that I am seeing from the time period yesterday where we were having problems are in a text file linked below.

    https://dreamsltd-my.sharepoint.com/personal/robwooders_dreams_co_uk/_layouts/15/guestaccess.aspx?docid=1d1e4786ef5fc47a3ab009ef2c829725f&authkey=AT0MTstzdSsi0YpO8O0uUk4

    I have also noticed that the W3WP.EXE Process is consuming approximately 30%+ of the server memory.  Running appcmd list wp I can see that the process is for the MSExchangePowerShellAppPool.  I have set the Recycling so that if the Private Memory for this App Pool is higher than 5GB.  Does this sound sensible?

    Thanks


    • Edited by rob_dreams Tuesday, July 18, 2017 8:17 AM
    Tuesday, July 18, 2017 8:14 AM
  • Hi,

    Is there any periodical task or process (for example: back up) running on Monday morning?

    Please open Computer Management, locate to Task Scheduler ---> Task Scheduler Library.
    Figure as below:

    As far as I know, Exchange 2013 and later uses a work cycle model to define the Exchange process assistant, for example Managed folder assistant, OAB generation schedule. Thus, I suppose that the Exchange process might not cause high resource usage.

    More details about it, for your reference:
    https://blogs.technet.microsoft.com/anya/2014/11/19/understanding-of-managed-folder-assistant-with-retention-policies/

    Regards,

    Allen Wang


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    • Proposed as answer by Allen_WangJF Monday, July 24, 2017 2:48 PM
    Tuesday, July 18, 2017 9:26 AM