none
Sporadic Internal Email Delay RRS feed

  • Question

  • We are having sporadic issues with internal email delayed delivery.

    This is the topology:

    Single Exchange 2013 (Version 15.0 ‎Build 995.29) running on Windows 2012R2

    ‎Here is a message tracking log example of the problem:

    **This is the STOREDRIVER RECEIVE**

    RunspaceId              : ef020c77-b0f5-470d-85c4-1de5df3a78ed
    Timestamp               : 1/4/2016 8:21:26 AM
    ClientIp                : fe80::d974:571a:8e74:dca6
    ClientHostname          : server.domain.com
    ServerIp                : fe80::d974:571a:8e74:dca6%12
    ServerHostname          : server
    SourceContext           : 08D2F4D6F76C2B95
    ConnectorId             : 
    Source                  : STOREDRIVER
    EventId                 : RECEIVE
    InternalMessageId       : 0
    MessageId               : <e02a50e8518649eb8a36316731d1be38@RXMAIL.rxbenefits.local>
    Recipients              : {user@domain.com, user@domain.com}
    RecipientStatus         : {To, To}
    TotalBytes              : 49512
    RecipientCount          : 2
    RelatedRecipientAddress : 
    Reference               : 
    MessageSubject          : *********************
    Sender                  : user@domain.com
    ReturnPath              : user@domain.com
    Directionality          : Originating
    TenantId                : 
    OriginalClientIp        : ************
    MessageInfo             : 04I: 
    MessageLatency          : 
    MessageLatencyType      : None
    EventData               : {[MailboxDatabaseGuid, a20269a8-f110-4887-8434-00acd1e6d988], [ItemEntryId, 00-00-00-00-2A-A8
                              -DB-69-13-CB-F1-43-B6-E2-D8-10-29-07-FE-F0-07-00-53-06-B1-2C-1B-01-E1-4A-A5-EF-E2-41-F7-39-5B
                              -58-00-00-00-80-0F-88-00-00-F2-46-95-A2-41-7B-FC-4D-A4-74-3B-03-B3-87-BF-A2-00-01-00-DD-3F-A7
                              -00-00], [DeliveryPriority, Normal], [PurportedSender, user@domain.com]}

    **This is when it finally got submitted 35 minutes later**

    RunspaceId              : ef020c77-b0f5-470d-85c4-1de5df3a78ed
    Timestamp               : 1/4/2016 8:56:34 AM
    ClientIp                : fe80::d974:571a:8e74:dca6%12
    ClientHostname          : server
    ServerIp                : 
    ServerHostname          : server.domain.com
    SourceContext           : MDB:a20269a8-f110-4887-8434-00acd1e6d988, Mailbox:cc57c1d6-6755-442a-921d-82cccd31498a, 
                              Event:197459615, MessageClass:IPM.Note, CreationTime:2016-01-04T14:21:25.534Z, 
                              ClientType:MOMT
    ConnectorId             : 
    Source                  : STOREDRIVER
    EventId                 : SUBMIT
    InternalMessageId       : 
    MessageId               : <e02a50e8518649eb8a36316731d1be38@RXMAIL.rxbenefits.local>
    Recipients              : {*******.com, *******.com}
    RecipientStatus         : {}
    TotalBytes              : 
    RecipientCount          : 2
    RelatedRecipientAddress : 
    Reference               : 
    MessageSubject          : *******
    Sender                  : *******
    ReturnPath              : 
    Directionality          : Originating
    TenantId                : 
    OriginalClientIp        : ***********
    MessageInfo             : 2016-01-04T14:21:25.534Z;LSRV=********:TOTAL=2109.081|SA=2088.066|MTSSDC=0.047
                              |MTSSDMO=20.311|MTSSDPL=0.015|MTSSDSS=0.312|MTSSDS=0.656|MTSS=21.024
    MessageLatency          : 00:35:09.0810000
    MessageLatencyType      : LocalServer
    EventData               : {[ItemEntryId, 00-00-00-00-2A-A8-DB-69-13-CB-F1-43-B6-E2-D8-10-29-07-FE-F0-07-00-53-06-B1-2C-
                              1B-01-E1-4A-A5-EF-E2-41-F7-39-5B-58-00-00-00-80-0F-88-00-00-F2-46-95-A2-41-7B-FC-4D-A4-74-3B-
                              03-B3-87-BF-A2-00-01-00-DD-3F-A7-00-00], [PurportedSender, **********]}

    Note the MessageInfo SA value is 2088 seconds. I cannot find a reference to what this value refers to.

    If I export the logs to HTML the logs indicate the following:

    8:21AM  STOREDRIVER RECEIVE

    8:21AM-8:56AM this repeats over and over: STOREDRIVER RECEIVE then STOREDRIVER SUBMITDEFER

    8:56AM finally STOREDRIVER SUBMIT







    Wednesday, January 6, 2016 10:37 PM

Answers

  • Check event logs for indication of Back Pressure - Event ID 15004, 15005, 15006, 15007

    More details here on Exchange Back Pressure https://technet.microsoft.com/en-us/library/bb201658(v=exchg.141).aspx


    __________________________________________

    Please mark as Answer if this answers your question

    Regards,

    Shane Jackson

    Blog: https://shanejacksonitpro.wordpress.com/

    Twitter: https://twitter.com/shane00jackson

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    • Marked as answer by Joel Sargent Friday, January 8, 2016 1:57 PM
    Thursday, January 7, 2016 4:12 PM

All replies

  • Try disabling antivirus on mailbox server and see if it helps. Also it could be because of disk IOPS.
    Wednesday, January 6, 2016 11:31 PM
  • Hi,

    When this problem occurs? Is there any change to Exchange Server before this problem occurs? Would it be delay when receive external email?

    To make further troubleshoot with this problem, firstly, please use OWA to check if it works normally?

    Then, use queue viewer to check if it identify any problem. You can also use get-queue command instead. For more details using get-queue cmdlet, access to the link below:

    https://technet.microsoft.com/en-us/library/aa998047(v=exchg.150).aspx

    Please also check Event Viewer if it identify this problem.


    Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Thursday, January 7, 2016 3:47 PM
  • Check event logs for indication of Back Pressure - Event ID 15004, 15005, 15006, 15007

    More details here on Exchange Back Pressure https://technet.microsoft.com/en-us/library/bb201658(v=exchg.141).aspx


    __________________________________________

    Please mark as Answer if this answers your question

    Regards,

    Shane Jackson

    Blog: https://shanejacksonitpro.wordpress.com/

    Twitter: https://twitter.com/shane00jackson

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    • Marked as answer by Joel Sargent Friday, January 8, 2016 1:57 PM
    Thursday, January 7, 2016 4:12 PM
  • This issue has occurred with and without AV installed.

    Below are some performance monitor counters I pulled from the Exchange Diagnostics Daily Performance Log. I don't think this is a resource bottleneck based on these numbers:

    Memory Available Mbytes: average 16GB out of 32GB (no spikes)

    CPU % Processor Time Total: average 69% with some (7 in an hour) spikes to 90% but these last less than 1 minute each

    Disk Queue Length: average .133 few spikes to 1 but only sustained for 1-2 minutes

    MSExchange Database ==> Instances(*)\I/O Database Reads (Attached) Average Latency: average 11ms no spikes higher than 34ms

    MSExchange Database ==> Instances(*)\I/O Database Writes (Attached) Average Latency: average 5ms no spikes higher than 67ms

    MSExchangeIS Store(*)\RPC Average Latency: average 1ms spikes to 10ms

    %Failures Due to Back Pressure: average 0 / maximum 0

    Reviewed all MSExchange Transport SMTP Availability failure counters and they are all: average 0 / maximum 0

    MSExchange Transport Queues Submission Queue Length: average 0 / maximum 0




    Thursday, January 7, 2016 7:32 PM
  • The problem has been reported mostly with internal email.

    OWA has the same problem so I don't think it is cleint side.

    I'll try to review the queue stats next time it occurs. The problem is very sporadic and seems to only occur for 20-30 minute every couple weeks.

    Thursday, January 7, 2016 7:35 PM
  • Found two entries at exactly the time the last reported issue occurred and ended:

    Start of the event:

    Log Name:      Application
    Source:        MSExchangeTransport
    Date:          1/4/2016 8:20:53 AM
    Event ID:      15004
    Task Category: ResourceManager
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      ********************
    Description:
    The resource pressure increased from Normal to High.

    The following resources are under pressure:
    Version buckets = 275 [High] [Normal=80 Medium=120 High=200]

    The following components are disabled due to back pressure:
    Inbound mail submission from Hub Transport servers
    Inbound mail submission from the Internet
    Mail submission from Pickup directory
    Mail submission from Replay directory
    Mail submission from Mailbox server
    Mail delivery to remote domains
    Content aggregation
    Mail resubmission from the Message Resubmission component.
    Mail resubmission from the Shadow Redundancy Component

    The following resources are in normal state:
    Queue database and disk space ("D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que") = 78% [Normal] [Normal=95% Medium=97% High=99%]
    Queue database logging disk space ("D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\") = 78% [Normal] [Normal=95% Medium=97% High=99%]
    Private bytes = 3% [Normal] [Normal=71% Medium=73% High=75%]
    Physical memory load = 49% [limit is 94% to start dehydrating messages.]
    Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]
    Temporary Storage disk space ("D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp") = 78% [Normal] [Normal=95% Medium=97% High=99%]

    End of the event:

    Log Name:      Application
    Source:        MSExchangeTransport
    Date:          1/4/2016 8:54:49 AM
    Event ID:      15005
    Task Category: ResourceManager
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      ******************
    Description:
    The resource pressure decreased from High to Normal.

    No components disabled due to back pressure.
    The following resources are in normal state:
    Queue database and disk space ("D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\mail.que") = 78% [Normal] [Normal=95% Medium=97% High=99%]
    Queue database logging disk space ("D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Queue\") = 78% [Normal] [Normal=95% Medium=97% High=99%]
    Version buckets = 0 [Normal] [Normal=80 Medium=120 High=200]
    Private bytes = 3% [Normal] [Normal=71% Medium=73% High=75%]
    Physical memory load = 48% [limit is 94% to start dehydrating messages.]
    Submission Queue = 0 [Normal] [Normal=2000 Medium=4000 High=10000]
    Temporary Storage disk space ("D:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\data\Temp") = 78% [Normal] [Normal=95% Medium=97% High=99%]




    Thursday, January 7, 2016 7:43 PM
  • Also found using PowerShell that a few emails with large attachments were sent during this time:

    Sender                        MessageSubject                Recipients                                       TotalBytes
    ------                        --------------                ----------                                       ---------- 

    sender@domain.com         subject {recip@domain.com}                         60234659
    sender@domain.com         subject {recip@domain.com}                         60234659
    sender@domain.com         subject {recip@domain.com}                         60234659
    sender@domain.com         subject {recip@domain.com}                         60234961
    sender@domain.com         subject {recip@domain.com}                         60234961
    sender@domain.com         subject {recip@domain.com}                         60235546
    sender@domain.com         subject {recip@domain.com}                         60235612
    sender@domain.com         subject {recip@domain.com}                         60235495
    sender@domain.com         subject {recip@domain.com}                         60235495
    sender@domain.com         subject {recip@domain.com}                         60235936
    sender@domain.com         subject {recip@domain.com}                         82410969
    sender@domain.com         subject {recip@domain.com}                         82410955
    sender@domain.com         subject {recip@domain.com}                         60235612
    Thursday, January 7, 2016 8:47 PM
  • I've updated the version buckets settings in EdgeTransport.exe.config

    Original:

     <add key="VersionBucketsHighThreshold" value="200" />
        <add key="VersionBucketsMediumThreshold" value="160" />
        <add key="VersionBucketsNormalThreshold" value="80" />

    New:

     <add key="VersionBucketsHighThreshold" value="400" />
        <add key="VersionBucketsMediumThreshold" value="240" />
        <add key="VersionBucketsNormalThreshold" value="160" />

    I'll update the post if this clears the problem.


    Thursday, January 7, 2016 9:22 PM
  • Hi,

    It seems like the big attachment of email was the reason of this problem. Looking forward to your feedback with this problem.


    Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Friday, January 8, 2016 6:48 AM
  • You could also implement message size limits to minimize the risk of emails with large attachments causing a resource constraint.  This link has more details https://technet.microsoft.com/en-us/library/bb124345%28v=exchg.160%29.aspx?f=255&MSPPError=-2147217396

    There are alternative ways to share 60MB & 80MB files other than sending via email.  For example - One Drive

    https://support.office.com/en-us/article/Share-files-and-folders-and-change-permissions-9fcc2f7d-de0c-4cec-93b0-a82024800c07

     

    __________________________________________

    Please mark as Answer if this answers your question

    Regards,

    Shane Jackson

    Blog: https://shanejacksonitpro.wordpress.com/

    Twitter: https://twitter.com/shane00jackson

    Note: Posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.

    Friday, January 8, 2016 9:44 AM
  • Hi,

    I am just writing to check the status of this thread. Was the information provided in previous reply helpful to you? Do you have any further questions or concerns? Please feel free to let us know.

    Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact tnmff@microsoft.com.

    Sunday, January 10, 2016 4:02 PM