none
Plugins in 2013/2019 environment fail when trying to retrieve headers

    Question

  • I have a mixed 2013/2019 environment. EWS/OWA/etc URLS are set to the same shared value (generic: web.mydomain.com), and currently it points to the 2013 servers. Trying to run a plugin (such as Message Header Analyzer) results in this on the 2013 server HTTP Proxy EWS logs. 

    For 2013 users, it all works fine. When a 2019 (proxied from the 2013 server) user tries to use the same plugin, it fails.

    2019-05-06T14:07:59.277Z,d6196d8e-76c9-4ece-96b7-4a5b64f64ae0,15,0,1395,0,E829FDDF34D04BF89349C84769DFB400_155715167926351,Ews,mail.myserver.com,/ews/exchange.asmx,,Bearer,false,,,,EWSProxy/MailApp/db13d144-e2b7-48ee-9fad-f7da1ac66c25,192.168.0.5,server-redacted,401,,,POST,,,,,,,,,862,,,,,,,,,,,,,,,1,,,,,,,,,,,,,,1,,1,1,,,,BeginRequest=2019-05-06T14:07:59.277Z;CorrelationID=<empty>;EndRequest=2019-05-06T14:07:59.277Z;S:ActivityStandardMetadata.Component=Owa;S:ActivityStandardMetadata.Action=ExecuteEwsProxy;S:ServiceCommonMetadata.OAuthError=Microsoft.Exchange.Security.OAuth.InvalidOAuthTokenException: An unexpected error occurred.\r\n   at Microsoft.Exchange.Security.OAuth.OAuthTokenHandler.CreateTokenHandler(JwtSecurityToken token  Uri targetUri)\r\n   at Microsoft.Exchange.Security.OAuth.OAuthTokenHandler.CreateTokenHandler(String rawToken  Uri targetUri  String& loggableToken)\r\n   at Microsoft.Exchange.Security.OAuth.OAuthHttpModule.InternalOnAuthenticate(HttpContext context);S:ServiceCommonMetadata.OAuthErrorCategory=InternalError;S:ServiceCommonMetadata.OAuthToken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Ink4bjRYdG1LN19XT1ZZRWUwanZBTXRpZ2VCOCJ9LnsibmFtZWlkIjoiZGIxM2QxNDQtZTJiNy00OGVlLTlmYWQtZjdkYTFhYzY2YzI1QERSV0hvbGRpbmdzLmNvbSIsInZlciI6IkV4Y2hhbmdlLkNhbGxiYWNrLlYxIiwiYXBwY3R4c2VuZGVyIjoiaHR0cHM6Ly90cmFpbmluZy5rbm93YmU0LmNvbS9waGlzaGFsZXJ0b25saW5lLzVDODlEODU5MjVCNTcyOEZCNkQxN0ZDNDU4NUNGMEJGL0FwcFJlYWQvSG9tZS9Ib21lLmh0bWwiLCJhcHBjdHgiOiJ7XCJzbXRwXCI6XCJtb2xzemV3c2tpQERSV0hvbGRpbmdzLmNvbVwiLFwibXNleGNocHJvdFwiOlwiZXdzXCJ9IiwiaXNzIjoiMDAwMDAwMDItMDAwMC0wZmYxLWNlMDAtMDAwMDAwMDAwMDAwQERSV0hvbGRpbmdzLmNvbSIsImF1ZCI6IjAwMDAwMDAyLTAwMDAtMGZmMS1jZTAwLTAwMDAwMDAwMDAwMC93ZWJtYWlsLmRyd2hvbGRpbmdzLmNvbUBEUldIb2xkaW5ncy5jb20iLCJleHAiOjE1NTcxNzg4NzcsIm5iZiI6MTU1NzE1MDA3N30=,

    It also works fine if I point the web.mydomain.com name to the 2019 server so there is no proxying taking place. Something is broken with this process if a 2019 user is hitting the 2013 server first. Confirmed across multiple plugins.
    • Edited by mark49808 Monday, May 6, 2019 5:29 PM
    Monday, May 6, 2019 4:02 PM

All replies

  • Hi Mark,

     

    If client access moves to Exchange 2019, does this issue also happen on Exchange 2013 users?

    When you run the Message Header Analyzer through web,  will the issue reproduce?

     

    Here are the brief steps about Exchange 2013 up-version proxy to Exchange 2019, please check if your settings are right.  

     

    1. Prep your environment (server versions, DFL/FFL, schema, AD, domains, etc…)
    2. Install 2019 (use a deployment AD site.)
    3. Configure Exchange 2019 server URLs as you would have Exchange 2013.
    4. Import the certificate(s) to 2019 server(s)
    5. Setup your DAG(optional)
    6. Start moving mailboxes
    7. Repeat for all Internet facing sites and then repeat for non-Internet facing
    8. Move incoming mail flow to deliver to 2019 first once it makes sense (>50% moved) 

    Supported: Cutover to all 2019 servers at once

    Regards,

    Kelvin Deng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Tuesday, May 7, 2019 6:29 AM
  • I think there is a misunderstanding running that web site wont solve anything as the issue is specific to plugins. Another example that reproduces is a Symantec Email Submission plugin.

    Yes, my settings are correct, Schema/etc are all updated. Everything else works (50 users are on 2019) except these plugins when proxying up from 2013 to 2019 for a 2019 mailbox user.

    Pointing the OWA name at 2019 makes it work for 2019 users. Also makes it work for 2013 users. So it seems there is some reproducible issue that up-version proxying between 2013 and 2019 breaks plugins. Is this a known issue? i can reproduce it 100% of the time with the same log message in prod and a very clean test environment. 

    Understood the obvious solution is to point the OWA URL to 2019 and have proxying only work in that direction, but this is not always easy for all organizations during coexistance. 

    Tuesday, May 7, 2019 1:16 PM
  • Hi Mark,

     

    How did you install these plugins? On server side or client side?

     

    With my research, there's no related known issue released. If the plugins are installed on server side, please try to uninstall them and reinstall on another version server, then check the result.

    Regards,

    Kelvin Deng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Wednesday, May 8, 2019 3:09 AM
  • Installed server side using xml via via ECP. Originally on 2013

    I uninstalled and re-installed multiple plugins on 2019. Same issue. EWS proxy returns a 401 Unauthorized for users on 2019 being proxied via 2013. 

    Wednesday, May 8, 2019 12:35 PM
  • Installed server side using xml via via ECP. Originally on 2013

    I uninstalled and re-installed multiple plugins on 2019. Same issue. EWS proxy returns a 401 Unauthorized for users on 2019 being proxied via 2013. 

    Hi mark,

    If the issue still exists when the plugins are only installed on Outlook clients?

    Regards,

    kelvin Deng



    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Thursday, May 9, 2019 1:44 AM
  • Unfortunately thats not how these plugins i'm interested in work. They are delivered as xml files designed to be installed server side. 
    Thursday, May 9, 2019 12:17 PM
  • Hi Mark,

     

    Is there any related event warning/error when the issue occurs?

     

    According the EWS logs in your initial post, I noticed the information

    "Microsoft.Exchange.Security.OAuth.InvalidOAuthTokenException".

    Please run the following command in the Exchange Management Shell to check whether the certificate is missing:

     

    Get-ExchangeCertificate (Get-AuthConfig).CurrentCertificateThumbprint

     

    If the OAuth certificate is present on both of Exchange servers, please check the certifcate SAN and expiry date.

     

    It is also suggested that using the Get-AuthConfig cmdlet to get the authorization configuration for partner applications.

    Regards,

    Kelvin Deng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Friday, May 10, 2019 12:55 PM
  • Auth cert is valid. And as mentioned it works when proxying from 2019->2013, or exclusively on 2013. Just not proxying 2013->2019. I would expect that if something was expired or incorrect, it would never work.
    Friday, May 10, 2019 12:59 PM
  • Hi Mark,

     

    To meet Modern Authentication prerequisites, please make sure your Exchange server 2013 is CU19 or higher, and Outlook version is 2013 or 2016. If not, please update them and check the result.

     

    Exchange Server specific

    https://docs.microsoft.com/en-us/office365/enterprise/hybrid-modern-auth-overview#do-you-meet-modern-authentication-prerequisites

    Regards,

    Kelvin Deng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Friday, May 10, 2019 1:26 PM
  • Yes I am fully up to date. I don't understand the reference to modern authentication. Seems unrelated to my issue.
    Tuesday, May 14, 2019 11:43 AM
  • Hi Mark,

    To avoid the plugins influence, will Exchange 2019 users, 2013 up-version proxy to 2019 successfully when all plugins are removed from the server side?

    Regards,

    Kelvin Deng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Sunday, May 19, 2019 5:55 AM
  • The plugins are the issue, though. Everything else works fine. Yes, users up-proxy just fine. But plugins are the only thing that dont work, so i'm not sure what removing the plugins answers. Its just masking the problem by removing the feature i want to use.
    Sunday, May 19, 2019 5:56 PM
  • Hi Mark,

     

    Now we could confirm the issue is that the Exchange 2019 users log in OWA through 2013 up-version proxy to 2019 successfully, but when they use the plugins, it fails.

     

    According to the HTTP Proxy EWS logs you referred in your initial post, we could see the "invalidOAuthToken" exception. After researching and confirming, the Exchange environment meets Modern Authentication prerequisites and the OAuth certificate configuration is correct. In my point of view, you could contact the 3rd party plugins vendors to know about the OAuth Token for the plugins and the setting in the proxy.

     

    Thank you for your understanding and patience! 

    Regards,

    Kelvin Deng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Thursday, May 23, 2019 1:51 AM