locked
On-Premise Exchange 2016 and Office 365 Mail delivery problems (571) RRS feed

  • Question

  • Hello,

    I have a client with an On-Premise Exchange Server 2016 Standard, single server deployment. They are having problems receiving emails from some of their vendors that use Office 365 Cloud servers for email. Some come through, but for others they get a "571 delivery not authorized" bounce message from the On-Premise Server. I've already checked blacklists and they aren't listed on any. Some other information:

    1. The On-Premise Server is using the AntiSpam Agents.
    2. It's NOT affecting ALL emails from Office 365. Just certain server on MS cloud pool 40.107.0.0.
    3. I have ruled out perimeter protections accordingly.
    4. Nothing in Message Tracking.

    The following is an excerpt from the SMTP Receive protocol log of a failed message:

    2019-09-24T14:16:26.288Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,0,192.168.2.19:25,40.107.80.91:1216,+,,
    2019-09-24T14:16:26.288Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,1,192.168.2.19:25,40.107.80.91:1216,>,220 <client domain>,
    2019-09-24T14:16:26.315Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,2,192.168.2.19:25,40.107.80.91:1216,<,EHLO NAM03-DM3-obe.outbound.protection.outlook.com,
    2019-09-24T14:16:26.316Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,3,192.168.2.19:25,40.107.80.91:1216,>,250  <internal server> Hello [40.107.80.91] SIZE 52428800 PIPELINING DSN ENHANCEDSTATUSCODES STARTTLS X-ANONYMOUSTLS AUTH NTLM X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XRDST,
    2019-09-24T14:16:26.343Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,4,192.168.2.19:25,40.107.80.91:1216,<,STARTTLS,
    2019-09-24T14:16:26.343Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,5,192.168.2.19:25,40.107.80.91:1216,>,220 2.0.0 SMTP server ready,
    2019-09-24T14:16:26.343Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,6,192.168.2.19:25,40.107.80.91:1216,*, CN=CS4v CN=CS4v 1C51B34FBC82009947E3AB058252ED3D 1B32CAC7B6337D73B719B5F4215F8AC6FBA35EA6 2019-05-11T15:33:30.000Z 2024-05-11T15:33:30.000Z CS4v;<internal server>,Sending certificate Subject Issuer name Serial number Thumbprint Not before Not after Subject alternate names
    2019-09-24T14:16:26.344Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,7,192.168.2.19:25,40.107.80.91:1216,*,,"TLS protocol SP_PROT_TLS1_2_SERVER negotiation succeeded using bulk encryption algorithm CALG_AES_256 with strength 256 bits, MAC hash algorithm CALG_SHA_384 with strength 0 bits and key exchange algorithm CALG_RSA_KEYX with strength 2048 bits"
    2019-09-24T14:16:26.431Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,8,192.168.2.19:25,40.107.80.91:1216,<,EHLO NAM03-DM3-obe.outbound.protection.outlook.com,
    2019-09-24T14:16:26.431Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,9,192.168.2.19:25,40.107.80.91:1216,*,,Client certificate chain validation status: 'EmptyCertificate'
    2019-09-24T14:16:26.431Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,10,192.168.2.19:25,40.107.80.91:1216,*,,TlsDomainCapabilities='None'; Status='NoRemoteCertificate'
    2019-09-24T14:16:26.431Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,11,192.168.2.19:25,40.107.80.91:1216,>,250  <internal server> Hello [40.107.80.91] SIZE 52428800 PIPELINING DSN ENHANCEDSTATUSCODES AUTH NTLM LOGIN X-EXPS GSSAPI NTLM 8BITMIME BINARYMIME CHUNKING XRDST,
    2019-09-24T14:16:26.494Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,12,192.168.2.19:25,40.107.80.91:1216,<,MAIL FROM:<KMcNerney@<office365 hosted domain> SIZE=17135,
    2019-09-24T14:16:26.494Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,13,192.168.2.19:25,40.107.80.91:1216,*,08D74096A58A9C49;2019-09-24T14:16:26.288Z;1,receiving message
    2019-09-24T14:16:26.494Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,14,192.168.2.19:25,40.107.80.91:1216,>,250 2.1.0 Sender OK,
    2019-09-24T14:16:26.557Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,15,192.168.2.19:25,40.107.80.91:1216,<,RCPT TO:<RHall@<client domain>>,
    2019-09-24T14:16:26.557Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,16,192.168.2.19:25,40.107.80.91:1216,>,250 2.1.5 Recipient OK,
    2019-09-24T14:16:26.620Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,17,192.168.2.19:25,40.107.80.91:1216,<,DATA,
    2019-09-24T14:16:26.620Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,18,192.168.2.19:25,40.107.80.91:1216,>,354 Start mail input; end with <CRLF>.<CRLF>,
    2019-09-24T14:16:26.873Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,19,192.168.2.19:25,40.107.80.91:1216,-,,Local

    Your help is appreciated.


    Rob Holmes

    Tuesday, September 24, 2019 6:55 PM

All replies

  • Hi Rob,

    I only see the following exceptions in your protocol log:

    2019-09-24T14:16:26.431Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,9,192.168.2.19:25,40.107.80.91:1216,*,,Client certificate chain validation status: 'EmptyCertificate'
    2019-09-24T14:16:26.431Z,<internal server>\Default Frontend <internal server>,08D74096A58A9C49,10,192.168.2.19:25,40.107.80.91:1216,*,,TlsDomainCapabilities='None'; Status='NoRemoteCertificate'


    It seems that issue lies in TLS encryption in the mail flow process, we could verify it by creating a new receive connector on Frontend Transport service, only accept message from the certain IP ranges of the O365 server, untick the Transport Layer Security(TLS), and tick Anonymous users in Permission groups.

    Regards,

    Manu Meng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    • Edited by Manu Meng Wednesday, September 25, 2019 8:39 AM
    • Proposed as answer by Manu Meng Friday, September 27, 2019 10:56 AM
    Wednesday, September 25, 2019 8:38 AM
  • Manu,

    Seems like the same problem exists when the Office 365 vendor sends to Google as well, they are opening a ticket with Office 365 support. I'm betting it has to do with that server cluster. I still want to check this though so I will create this Connector and report my findings.

    Thanks,

    Rob


    Rob Holmes

    Thursday, September 26, 2019 2:45 AM
  • Manu,

    Seems like the same problem exists when the Office 365 vendor sends to Google as well, they are opening a ticket with Office 365 support. I'm betting it has to do with that server cluster. I still want to check this though so I will create this Connector and report my findings.

    Thanks,

    Rob


    Rob Holmes

    OK, feel free to post back!

    Regards,

    Manu Meng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Thursday, September 26, 2019 8:49 AM
  • Manu,

    I created the connector and had the vendor send an email to it but they got the same bounce, logs reveal the same issue. Just be clear though, do I need to set a priority on that connector of some sort to make sure the Default Frontend (which gets any remote mail) isn't processed? I assume Exchange 2016 does the specific scoped connectors first, but want to make sure.

    Thank you,

    Rob


    Rob Holmes

    Monday, September 30, 2019 10:01 PM
  • Manu,

    I created the connector and had the vendor send an email to it but they got the same bounce, logs reveal the same issue. Just be clear though, do I need to set a priority on that connector of some sort to make sure the Default Frontend (which gets any remote mail) isn't processed? I assume Exchange 2016 does the specific scoped connectors first, but want to make sure.

    Thank you,

    Rob


    Rob Holmes

    We could not specify the priority of the receive connector, as you said, Exchange 2016 does the specific scoped connectors first. We could check which connector is being used in protocol log.

    If creating a new receive connector didn't help, I would consider that issue lies in the Office 365 sender side, did you get some clues from O365 support?

    Regards,

    Manu Meng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Wednesday, October 2, 2019 8:37 AM
  • Manu,

    I was able to finally get a logging of them using the new connector.

    2019-10-03T18:13:30.754Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,0,<internal server>:25,40.107.82.79:13184,+,,
    2019-10-03T18:13:30.755Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,1,<internal server>:25,40.107.82.79:13184,>,"220 <internal server> Microsoft ESMTP MAIL Service ready at Thu, 3 Oct 2019 14:13:30 -0400",
    2019-10-03T18:13:30.810Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,2,<internal server>:25,40.107.82.79:13184,<,EHLO NAM01-SN1-obe.outbound.protection.outlook.com,
    2019-10-03T18:13:30.810Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,3,<internal server>:25,40.107.82.79:13184,>,250  <internal server> Hello [40.107.82.79] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES 8BITMIME BINARYMIME CHUNKING,
    2019-10-03T18:13:30.901Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,4,<internal server>:25,40.107.82.79:13184,<,MAIL FROM:<SHenley@<office365 hosted domain>> SIZE=3967062,
    2019-10-03T18:13:30.901Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,5,<internal server>:25,40.107.82.79:13184,*,08D745D3504EB6D0;2019-10-03T18:13:30.754Z;1,receiving message
    2019-10-03T18:13:30.901Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,6,<internal server>:25,40.107.82.79:13184,>,250 2.1.0 Sender OK,
    2019-10-03T18:13:30.989Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,7,<internal server>:25,40.107.82.79:13184,<,RCPT TO:<AMatthews@<client domain>>,
    2019-10-03T18:13:31.047Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,8,<internal server>:25,40.107.82.79:13184,>,250 2.1.5 Recipient OK,
    2019-10-03T18:13:31.135Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,9,<internal server>:25,40.107.82.79:13184,<,RCPT TO:<JThomas@<client domain>>,
    2019-10-03T18:13:31.138Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,10,<internal server>:25,40.107.82.79:13184,>,250 2.1.5 Recipient OK,
    2019-10-03T18:13:31.225Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,11,<internal server>:25,40.107.82.79:13184,<,RCPT TO:<KGarris@<client domain>>,
    2019-10-03T18:13:31.228Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,12,<internal server>:25,40.107.82.79:13184,>,250 2.1.5 Recipient OK,
    2019-10-03T18:13:32.411Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,13,<internal server>:25,40.107.82.79:13184,<,DATA,
    2019-10-03T18:13:32.411Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,14,<internal server>:25,40.107.82.79:13184,>,354 Start mail input; end with <CRLF>.<CRLF>,
    2019-10-03T18:13:32.896Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,15,<internal server>:25,40.107.82.79:13184,-,,Remote(SocketError)
    2019-10-03T18:13:35.754Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D1,0,<internal server>:25,40.107.82.45:2177,+,,

    I notice a Remote(SocketError) present now. It looks like it's on the Office 365 server end though, correct?


    Rob Holmes

    Thursday, October 3, 2019 6:35 PM
  • Manu,

    I was able to finally get a logging of them using the new connector.

    2019-10-03T18:13:30.754Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,0,<internal server>:25,40.107.82.79:13184,+,,
    2019-10-03T18:13:30.755Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,1,<internal server>:25,40.107.82.79:13184,>,"220 <internal server> Microsoft ESMTP MAIL Service ready at Thu, 3 Oct 2019 14:13:30 -0400",
    2019-10-03T18:13:30.810Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,2,<internal server>:25,40.107.82.79:13184,<,EHLO NAM01-SN1-obe.outbound.protection.outlook.com,
    2019-10-03T18:13:30.810Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,3,<internal server>:25,40.107.82.79:13184,>,250  <internal server> Hello [40.107.82.79] SIZE 37748736 PIPELINING DSN ENHANCEDSTATUSCODES 8BITMIME BINARYMIME CHUNKING,
    2019-10-03T18:13:30.901Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,4,<internal server>:25,40.107.82.79:13184,<,MAIL FROM:<SHenley@<office365 hosted domain>> SIZE=3967062,
    2019-10-03T18:13:30.901Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,5,<internal server>:25,40.107.82.79:13184,*,08D745D3504EB6D0;2019-10-03T18:13:30.754Z;1,receiving message
    2019-10-03T18:13:30.901Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,6,<internal server>:25,40.107.82.79:13184,>,250 2.1.0 Sender OK,
    2019-10-03T18:13:30.989Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,7,<internal server>:25,40.107.82.79:13184,<,RCPT TO:<AMatthews@<client domain>>,
    2019-10-03T18:13:31.047Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,8,<internal server>:25,40.107.82.79:13184,>,250 2.1.5 Recipient OK,
    2019-10-03T18:13:31.135Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,9,<internal server>:25,40.107.82.79:13184,<,RCPT TO:<JThomas@<client domain>>,
    2019-10-03T18:13:31.138Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,10,<internal server>:25,40.107.82.79:13184,>,250 2.1.5 Recipient OK,
    2019-10-03T18:13:31.225Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,11,<internal server>:25,40.107.82.79:13184,<,RCPT TO:<KGarris@<client domain>>,
    2019-10-03T18:13:31.228Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,12,<internal server>:25,40.107.82.79:13184,>,250 2.1.5 Recipient OK,
    2019-10-03T18:13:32.411Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,13,<internal server>:25,40.107.82.79:13184,<,DATA,
    2019-10-03T18:13:32.411Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,14,<internal server>:25,40.107.82.79:13184,>,354 Start mail input; end with <CRLF>.<CRLF>,
    2019-10-03T18:13:32.896Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D0,15,<internal server>:25,40.107.82.79:13184,-,,Remote(SocketError)
    2019-10-03T18:13:35.754Z,CS4V\FrontEnd 40.107.0.0 (Office365 Cluster Test),08D745D3504EB6D1,0,<internal server>:25,40.107.82.45:2177,+,,

    I notice a Remote(SocketError) present now. It looks like it's on the Office 365 server end though, correct?


    Rob Holmes

    I'm afraid yes.

    Regards,

    Manu Meng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.

    Monday, October 7, 2019 8:22 AM