none
Hybrid agent setup failing on "Validating Hybrid Agent for Exchange usage" RRS feed

  • Question

  • I really can't find much in the way of troubleshooting this... I'm setting up a hybrid between a 2013 on-prem Exchange and O365, doing the hybrid agent method. It gets through Download Hybrid Agent, Install agent, Register Agent, then after a few minutes fails on Validating Hybrid Agent for Exchange usage. Looking through the logs I'm seeing this:

    Microsoft.Exchange.Management.Migration.MigrationService.Endpoint.TestMigrationServerAvailability.InternalProcessEndpoint(Boolean fromAutoDiscover)' IsValid=1 Message='The connection to the server '<GUID>.resource.mailboxmigration.his.msappproxy.net' could not be completed.' ObjectState='New' Result='Failed' SupportsCutover=0} 2019.03.28 15:26:19.498 10341 [Client=UX, Page=HybridConnectorInstall, Step=TestOrgRoute, Thread=26] FINISH Time=345.4s Results=Failed The connection to the server '<GUID>.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' timed out. Error details: The request channel timed out while waiting for a reply after 00:00:00.0024533. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. --> The HTTP request to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' has exceeded the allotted timeout of 00:00:00.0020000. The time allotted to this operation may have been a portion of a longer timeout. --> The operation has timed out, The request channel timed out while waiting for a reply after 00:00:00.0024533. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout., The HTTP request to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' has exceeded the allotted timeout of 00:00:00.0020000. The time allotted to this operation may have been a portion of a longer timeout., The operation has timed out

    I've been googling and can't find ANYTHING on this error.

    Thursday, March 28, 2019 3:50 PM

All replies

  • Do you have a proxy server?

    Ed Crowley MVP "There are seldom good technological solutions to behavioral problems."
    Celebrating 20 years of providing Exchange peer support!

    Thursday, March 28, 2019 11:26 PM
    Moderator
  • Hi GrdLock,

    You can refer the following method to confirm if all the port requirements have been met prior to installing:

    • On the server where you will be running the Hybrid Configuration Wizard (Hybrid Agent install and subsequent hybrid configuration steps), download the following sample script and save it to any directory: http://aka.ms/hybridconnectivity.
    • Open PowerShell and change directory to the location of the script.
    • Import the cmdlets: Import-Module .\HybridManagement.psm1
    • Next run Test-HybridConnectivity with the testO365Endpoints option to verify the machine you are installing on can reach out to all required endpoints for the Hybrid Agent installation and Hybrid Configuration Wizard setup.

    Details see: The Microsoft Hybrid Agent Public Preview -- Verifying connectivity


    Best Regards,
    Niko Cheng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.


    Friday, March 29, 2019 7:38 AM
    Moderator
  • Hi GrdLock,

    I'm just writing to check how's everything going? If you have any questions or needed further help on this issue, please feel free to post back. If the issue has been resolved, please mark the helpful replies as answers, this will make answer searching in the forum easier and be beneficial to other community members as well.

    Thanks for your understanding.


    Best Regards,
    Niko Cheng


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.


    Click here to learn more. Visit the dedicated forum to share, explore and talk to experts about Microsoft Teams.

    Wednesday, April 3, 2019 9:44 AM
    Moderator
  • Dear

    We have the same error, the same issue.
    Using the most recent version of the Hybrid Agent (16.0.3054.9) running on an Exchange 2013 CU21.
    The Hybrid server has direct internet access, bypassing the Proxy server and has NO restrictions on the outbound firewall, the client told me.

    When we run the Test-HybridConnectivity we get the following output. 

    Testing connection to mscrl.microsoft.com on port 80
    Testing connection to crl.microsoft.com on port 80
    Testing connection to ocsp.msocsp.com on port 80
    Testing connection to www.microsoft.com on port 80
    Testing connection to login.windows.net on port 443
    WARNING: Ping to login.windows.net failed -- Status: TimedOut
    Testing connection to login.microsoftonline.com on port 443
    WARNING: Ping to login.microsoftonline.com failed -- Status: TimedOut
    Testing connection to aadap-portcheck-seaus.connectorporttest.msappproxy.net on port 8080
    WARNING: Ping to aadap-portcheck-seaus.connectorporttest.msappproxy.net failed -- Status: TimedOut
    Performing GET on https://aadap-portcheck-seaus.connectorporttest.msappproxy.net:8080
    Testing connection to watchdog.servicebus.windows.net on port 443
    WARNING: Ping to watchdog.servicebus.windows.net failed -- Status: TimedOut
    Performing GET on https://watchdog.servicebus.windows.net:443
    Browser is configured for proxy:
    Ensure connector is configured for proxy or whitelisted to bypass proxy.

    It seems that some locations can’t be pinged, although all URLs are accessible from the hybrid server.
    Is this a requirement, does the ping needs to be successful?

    This is the error from the hybrid setup log files:

    TestMigrationServerAvailabilityOutcome {ErrorDetail='Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server '6f82a11b-99a4-4ef0-a0a1-b1009bc0a7c7.resource.mailboxmigration.his.msappproxy.net' could not be completed. ---> Microsoft.Exchange.MailboxReplicationService.RemoteTransientException: The call to 'https://6f82a11b-xxxx-xxxx-xxxx-b1009bc0a7c7.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' timed out. Error details: The request channel timed out while waiting for a reply after 00:00:09.9990046. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. --> The remote server returned an error: (504) Gateway Timeout. --> The remote server returned an error: (504) Gateway Timeout. ---> Microsoft.Exchange. MailboxReplicationService.RemotePermanentException: The request channel timed out while waiting for a reply after 00:00:09.9990046. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. --->

    Anyone an idea why we can’t Validating the Hybrid agent?

    Thanks for the feedback.
    Regards
    Peter





    Peter Van Keymeulen, IT Infrastructure Solution Architect, www.edeconsulting.be

    Thursday, May 16, 2019 3:01 PM
  • Hi guys,

    i'm currrenlty facing exactly same issue like @Peter regards. What else can i check? I tried adding all Domain names the Test-HybridConnectivity -testO365Endpoints Script tells doesnt help...

    Greetings,

    Markus

    Wednesday, May 22, 2019 12:13 PM
  • I never use that.  I use http://exrca.com.

    Ed Crowley MVP "There are seldom good technological solutions to behavioral problems."
    Celebrating 20 years of providing Exchange peer support!

    Wednesday, May 22, 2019 11:56 PM
    Moderator
  • Hi Niko,

    After the user provided his results did you have any luck fixing the issue?

    I'm having the exact same issue with my customer.

    I've spent two hours with the firewall company ensuring it isn't an issue coming in, and we have TCP dumps from the firewall showing that it's not receiving any ping response.

    If I do a Test-Netconnection login.windows.net -Port 443 -InformationLevel Detailed

    WARNING: Ping to login.windows.net failed -- Status: TimedOut
    ComputerName             : login.windows.net
    RemoteAddress            : 40.126.12.34
    RemotePort               : 443
    AllNameResolutionResults : 20.190.140.100
                               20.190.140.98
                               20.190.140.99
                               40.126.12.101
                               40.126.12.32
                               40.126.12.33
                               40.126.12.34
                               40.126.12.99
    MatchingIPsecRules       :
    NetworkIsolationContext  : Internet
    InterfaceAlias           : Ethernet
    SourceAddress            : 192.168.1.41
    NetRoute (NextHop)       : 192.168.1.1
    PingSucceeded            : False
    PingReplyDetails (RTT)   : 0 ms
    TcpTestSucceeded         : True

    Ping fails but TCPTest works.

    And I believe this could be related too my InTune setup not working as expected also.

    Can we get some feedback on how to resolve this issue please.

    Monday, June 10, 2019 10:24 PM
  • Hi Peter,

    Did you ever manage to resolve this issue?

    There doesn't seem to be too much information about the error anywhere.

    Let me know if you ended up being successful.

    Thanks.

    Tuesday, June 11, 2019 4:51 AM
  • I had the same issue and found the solution on this page

    https://www.reddit.com/r/exchangeserver/comments/b6kbxq/validating_hybrid_agent_for_exchange_usage_is/

    from user "txlessor"

    "I ran into the same issue on one client of dozens we've done.

    My theory is that it has to do with choosing the wrong MRSProxy URL in the HCW / not configuring the MSAppProxy. Fortunately, it doesn't matter.

    Make sure you followed the steps to enable BasicAuth and MRSProxy on EWS. As long as you got the 'congratulations' notice, the link was created.

    Set-WebServicesVirtualDirectory –identity EXCHANGESERVER\"EWS (Default Web Site)"  -MRSProxyEnabled $false
    Set-WebServicesVirtualDirectory –identity EXCHANGESERVER\"EWS (Default Web Site)"  -MRSProxyEnabled $true
    Set-WebServicesVirtualDirectory –identity EXCHANGESERVER\"EWS (Default Web Site)" -BasicAuthentication $TRUE
    

    I was able to hop on to ECP (Always IE!) and move the mailboxes without issue. You'll notice that in ECP as to migration to O365 it has the proper FQDN of your on premise exchange server. If it doesn't, your issue lies in your Internal/External URIs.

    Use these PS scripts to make sure those are set right: https://github.com/cunninghamp/ConfigureExchangeURLs.ps1

    Good luck!"

    Solved it for me!


    Saturday, June 15, 2019 3:32 AM
  • I'm having a similar issue when I test connectivity prior to installing the Hybrid Agent. It seems I can't resolve the host name

    watchdog.servicebus.windows.net

    I've tried resolving it from work, home and all have the same DNS issue. Has anyone run into this before?

    [PS] C:\Hybrid Agent>Test-HybridConnectivity testO365Endpoints

    Testing connection to mscrl.microsoft.com on port 80

    Testing connection to crl.microsoft.com on port 80

    Testing connection to ocsp.msocsp.com on port 80

    Testing connection to www.microsoft.com on port 80

    Testing connection to login.windows.net on port 443

    Testing connection to login.microsoftonline.com on port 443

    Testing connection to aadap-portcheck-seaus.connectorporttest.msappproxy.net on port 8080

    Performing GET on https://aadap-portcheck-seaus.connectorporttest.msappproxy.net:8080

    Testing connection to watchdog.servicebus.windows.net on port 443

    WARNING: Name resolution of watchdog.servicebus.windows.net failed -- Status: No

    such host is known

    Performing GET on https://watchdog.servicebus.windows.net:443

    Invoke-WebRequest : The remote name could not be resolved:

    'watchdog.servicebus.windows.net'

    At C:\Hybrid Agent\HybridManagement.psm1:196 char:19

    +         $result = Invoke-WebRequest -Method Get -Uri $uri

    +                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

        + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebReque

       st) [Invoke-WebRequest], WebException

        + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Comman

       ds.InvokeWebRequestCommand

     

    [PS] C:\Hybrid Agent>


    Orange County District Attorney

    Thursday, July 18, 2019 3:02 PM
  • Did you manage to resolve this issue?

    As we are experiencing the exact same issue.

    Tuesday, July 30, 2019 4:00 PM
  • Sadly these steps did not work for me. It's very frustrating now.
    Wednesday, September 4, 2019 9:25 AM
  • same here

    this is a nonsense

    Thursday, September 5, 2019 3:00 PM
  • Same here. Anybody has a solution for this?

    Monday, October 21, 2019 6:55 PM
  • Today i faced the same problem.

    For me the solution was enabeling basic authentication on my on-premise Exchange 2013 EWS virtual directory.

    Monday, November 4, 2019 9:51 AM
  • His mistake was in the connection test. In the hybrid log which message occurred? in my trim the error below;

    mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc 'failed Error details: The remote server returned an unexpected response: (400) Bad Request.

    I count on everyone's help.

    Thank you very much
    Tuesday, November 5, 2019 7:15 PM
  • Same here. So also the Bad Request (400).

    Actually a case is open. Lets see where we are going to :)
    I will post the solution if there is one then.

    Thursday, November 14, 2019 12:44 PM
  • Same here

    I have a case open at Microsoft for weeks now. Still getting Bad Request.

    Peter

    Wednesday, November 20, 2019 8:10 PM
  • news about the error?

    Contacting Microsoft was prompted to install in the classic topology, the error occurs at the end and does not create the endpoint.

    Created the endpoint manually, but there is an error in free / busy by Exchange Online.

    When debugging the error by Fiddler the same 400 bad request message occurs.
    Thursday, November 21, 2019 2:39 PM
  • BUMP. I'm running into this as well at a client's site. It's failing right at: 
    Monday, November 25, 2019 2:29 PM
  • It's a backend problem at the Microsoft side, someone at Microsoft is keeping me up-to-date while the engineers are looking at it.

    Don't try anything which could damage your environment, just wait for an update on the issue, changing things on the on-premise side won't resolve the problem.

    What you could do is open a ticket at portal.office.com with the logs included from trying to install the hybrid modern 

    The 400 bad request is important to mention.



    Tuesday, November 26, 2019 8:18 AM
  • Hi Jurre,

    Even i was getting same error, Please keep us updated once issue has been resolved 

    Tuesday, November 26, 2019 11:37 AM
  • Hello,

    i have a similar issue but a slightly different error message:

    Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: There was no endpoint listening at https://cab6a948-xxxx-41c8-a479-a0016cffba17.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The remote name could not be resolved: 'cab6a948-xxxx-41c8-a479-a0016cffba17.resource.mailboxmigration.his.msappproxy.net'

    If i try to lookup the endpoint in public DNS it cannot be found which could be the issue here.

    I compared this to another Hybrid agent install where i can successfully lookup the xxxx.his.msappproxy.net endpoint in public DNS.

    Opened a ticket with Microsoft now.

    Do you guys can successfully resolve your xxxxx.his.msappproxy.net endpoints or do you have similar results?

    Thursday, November 28, 2019 3:04 PM
  • Hello,

    i have a similar issue but a slightly different error message:

    Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: There was no endpoint listening at https://cab6a948-xxxx-41c8-a479-a0016cffba17.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The remote name could not be resolved: 'cab6a948-xxxx-41c8-a479-a0016cffba17.resource.mailboxmigration.his.msappproxy.net'

    If i try to lookup the endpoint in public DNS it cannot be found which could be the issue here.

    I compared this to another Hybrid agent install where i can successfully lookup the xxxx.his.msappproxy.net endpoint in public DNS.

    Opened a ticket with Microsoft now.

    Do you guys can successfully resolve your xxxxx.his.msappproxy.net endpoints or do you have similar results?

    Did you try the HybridManagement script as advised above by Niko Cheng? Delete the   watchdog.servicebus.windows.net part as this is old crap.

    Because I am able to resolve to the msappproxy.net URL.

    Also don't forget to force TLS 1.2

      

    Friday, November 29, 2019 1:19 PM
  • Same for me right now.. don't know how to solve this... :(

    Thursday, December 5, 2019 9:41 AM
  • Hi Guys,

    I gave up on Microsoft support, they even created a similar situation on their side and acknowledged the problem.

    But to no avail, it's taking way too long in my opinion. I decided to use the Classic Toplogy, security wise my least favorite option, but some Firewall tweaking on the on-premise side and tweaking the receive/send connectors and the difference is almost dismissible

    The biggest problem for me is the engineer you're communicating with has little to no knowledge on the subject and is literally the middleman who needs to translate my findings to the senior engineer and the other way around. This only makes it more complex to find a solution.

    Good luck guys!

    Tuesday, December 10, 2019 6:59 AM
  • Today i faced the same problem.

    For me the solution was enabeling basic authentication on my on-premise Exchange 2013 EWS virtual directory.

    this worked for me! log in to exchange server, click servers link. click ews server, edit, enable basic authentication. worked like a dream.
    Wednesday, January 22, 2020 4:42 PM
  • Have a look at the internal an external URL's for the virtual directories, de EWS virtual directory specifically (although usually you would configure all URL's to use the same FQDN). Make sure the internal and external URL's are the same.

    For me the sollution was to make sure that the Exchange server itself could actually resolve the FQDN used in these URL's and to make sure that it resolved to the INTERNAL IP-address of the Exchange server.

    If your URL's use https://mail.domain.com/...  , either implement split-brain DNS for domain.com by adding a DNS zone domain.com and add an A-record for 'mail' pointing to the internal IP-address of the Exchange server, or add a zone mail.domain.com and add an A-record with a blank name (same as parent folder) and again, have this point to the internal IP-address of the Exchange server.


    Thursday, March 5, 2020 10:38 AM
  • Has anyone solved this issue in the meantime?

    Our exchange servers have direct internet access and still getting the message

    The call to 'https://GUID.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' timed out

    The strange thing is, that these URL is not hitting the firewall we could see through the agent - its just hitting if we try it directly in the browser

    could onyone help please?

    this is really annoying

    Friday, May 1, 2020 8:57 AM
  • We just tried to install it. Exchange 2016 / W2k12R2

    Also having the timeout problem. But the connector was registered in our tenant. So internet connection must be in place. We excluded the server in any policy that filters its traffic.

    We can open up the app proxy URL from the server. What are we doing wrong?


    Regards Stephan

    Friday, June 26, 2020 1:57 PM