locked
Exchange 2007 - Outlook anywhere stopped working RRS feed

  • Question

  • Hi there,

    Our environment is Windows 2003 x64 on Exchange 2007 SP3 on a CCR cluster with a 2 hub/cas servers.

    Over a month ago, outlook anywhere stopped working.
    I've followed the Exchange blog and made the changes suggested in:

    http://blogs.technet.com/b/exchange/archive/2008/06/20/3405633.aspx

    Ive disabled Outlook anywhere, uninstalled RPC/HTTP, rebooted and re-enabled Outlook anywhere again but still not working.

    Below is the error message i get on the www.testexchangeconnectivity.com

    Attempting to ping RPC endpoint 6001 (Exchange Information Store) on server mbxserver.internal.local
    The attempt to ping the endpoint failed.
    Additional Details
    An RPC error was thrown by the RPC Runtime process. Error 1818 1818

     I've even added an entry into the host file on our mailbox server and our cas server so the external fqdn points to the internal ip address of our cas server.

    When tracing the IIS logs i see these entries in the W3SVC1 folder


    2011-10-06 04:49:49 W3SVC1 10.254.72.56 RPC_IN_DATA /rpc/rpcproxy.dll - 443 - 10.254.91.212 MSRPC 401 2 2148074254
    2011-10-06 04:49:49 W3SVC1 10.254.72.56 RPC_IN_DATA /rpc/rpcproxy.dll - 443 - 10.254.91.212 MSRPC 401 2 2148074254
    2011-10-06 04:49:49 W3SVC1 10.254.72.56 RPC_IN_DATA /Rpc/RpcProxy.dll - 443 - 10.254.91.212 MSRPC 401 2 2148074254
    2011-10-06 04:49:50 W3SVC1 10.254.72.56 RPC_IN_DATA /Rpc/RpcProxy.dll - 443 - 10.254.91.212 MSRPC 401 1 0

    and these entries in the HTTPERR folder

    2011-10-07 02:38:03 10.254.91.212 60214 10.254.72.56 443 HTTP/1.1 RPC_IN_DATA /rpc/rpcproxy.dll?mbxserver.internal.local:6001 400 1 BadRequest DefaultAppPool
    2011-10-07 02:38:57 10.254.91.212 60214 10.254.72.56 443 HTTP/1.1 RPC_IN_DATA /rpc/rpcproxy.dll?mbxserver.internal.local:6001 400 1 Connection_Dropped DefaultAppPool

    Externally when i go to externalfqdn/rpc/rpcproxy.dll, i dont get prompted for a logon, all i get is Error:Access is Denied. However i do see the correct SSL certificate.

    When i try an rpcping from an external client, it is successfull when using the -E option.

    but when i trying the rpcping to ports 6001,6002,6004 i get prompted for a password and then get the error message:

    Exception 5 (0x000000005)

    So it looks like rpc call to the front end server works but then when trying to get to the internal mailbox server it fails.

    I have edited the validports registry key and tested from the internal cas server it can telnet to the mailbox server on ports 6001,6002,6004.

    Whats different about this environment is that our internal AD domain is named something different to their external domain. Internally we cant get to externalurl.com.au.
    Also the reverse proxy is done through a blue coat device and the external certificate is installed on this device but not installed on the internal CAS server. The internal CAS server has a certificate from our local CA authority and in the SAN it does include both the internal and external name. Not sure if this has anything to do with it, but OWA and ActiveSync work im thinking this may not be the issue.

    I've been stuck on this problem for a week and looking for some expert advice.

    Please help......

     

     

     

     

    Sunday, October 9, 2011 11:44 PM

Answers

  • Finally, this issue has been resolved.

    We performed another test by creating a new OWA and Outlook anywhere URL and published it externally but this time the site was going through the old reverse proxy device.
    After installing the internal cert on the client machine, it worked straight away.
    So I threw the ball into the network guys court and after some investigation he found a new feature on the device called network threat protection, seems like this feature was scanning the RPC packets to the mailbox server and stopping OA from working.

    So after 2 weeks of pain, it wasnt my problem in the first place, lesson learnt is push the network guys to check their kit first.

    Thanks for your help Fiona

    Winga

    • Marked as answer by wingadean Wednesday, October 19, 2011 3:58 AM
    Wednesday, October 19, 2011 3:58 AM

All replies

  •  

    Hi Wingadean,

     

    The IIS log means the logon failed due to server configuration. And I would suggest you verify the following configuration:

     

    1.    Logon to CAS server which is acting your RPC proxy server, launch IE and try the URL: https://localhost/rpc/rpcproxy.dll. The expected result is an error code 600. This test can tell us if the Outlook Anywhere component is working.

    2.    Verify the authentication in EMC/Server Configuration/<your CAS server>/Properties/Outlook Anywhere.

    3.    Verify the certificate configuration in IIS, make sure it is setup correctly. Refer to: http://blogs.technet.com/b/exchange/archive/2008/02/01/3404755.aspx

    4.    Run a test with an internal Outlook. To do this, change the local host file and point the external URL to the internal IP address of your CAS server; and then change the Outlook profile to enable option “In fast network, connect HTTP first”. When Outlook is launched, verify the Connection Status and make sure it is showing as HTTP.

     

    Hope it is helpful.

     

     


    Fiona
    Tuesday, October 11, 2011 4:07 AM
  • Hi Fiona,

    Thanks for your reply.

    1. When navigating to the website i get a blank page which shows that is it working. However externally when i navigate to https://externalfdqdn.com.au/rpc, i dont get a logon prompt and all i get is : Error:Access is denied.  Does this have anything to do with it?

    2. Authentication is set to NTLM , ive tried with basic as well and same error.

    3. The IIS configuration is setup correctly.

    4. Internally Outlook anywhere works , i can see the connection status is via https.

    The problem is externally, The strange thing about our envrionment is that our Verisign cert is installed on the Blue Coat reverse proxy device but not installed on the CAS server, we have an internal certificate from our internal CA on our CAS that does include the internal hostname as well as the external fdqn, ive never worked with an environment like this before but my manager assures me that this used to work.

    Also yesterday we changed the reverse proxy to point to our second CAS server to see if that helps but same issue.

    I've run netmon on the CAS server and see messages below when i was attempting to connect to outlook anywhere:

    w3wp.exe caserver.local mbxserver.local MSRPC MSRPC:c/o Request: unknown   Call=0x6808D  Opnum=0x0  Context=0x2  Hint=0x5C  {MSRPC:7, TCP:6, IPv4:5}

     

    any more ideas??? any help appreciated as im clutching at straws.

     

     

     

     

    Tuesday, October 11, 2011 5:18 AM
  • Hi,

     

    Thanks for your reply. Outlook Anywhere won't work if the certificate is not trusted by the client computer. Your enterprise certificates must be manually copied to the trusted root certificate store on the client computer or mobile device. Refer to:

     

    Installing a Self-Signed Certificate as a Trusted Root CA in Windows Vista
    http://blogs.technet.com/b/sbs/archive/2007/04/10/installing-a-self-signed-certificate-as-a-trusted-root-ca-in-windows-vista.aspx

     

    Please take a try and let me know if this works. Thanks.


    Fiona
    • Proposed as answer by Fiona_Liao Tuesday, October 11, 2011 5:54 AM
    Tuesday, October 11, 2011 5:54 AM
  • Hi Fiona,

    The Verisign certificate is installed on the Blue Coat reverse proxy device and this device does the SSL offloading.
    OWA and ActiveSync work perfectly externally so not sure what the difference with Outlook anywhere is.

    The CAS server doesnt have the Verisgin cert, it doesnt have a self signed certificate too, it has an internal certificate from our internal CA. This internal cert has the following SAN in this order:

    DNS Name = internalfqdn
    DNS Name = internal host name
    DNS Name = externalfdqdn
    DNS Name = autodiscover.internal
    DNS Name = autodiscvoer.external

    I read somewhere that for OA to work properly, the order of the DNS names in the SAN matter, is this true?

    I have also tried what you said and installed the cert onto the local machines store but still the same issue.

    Do you think that the Verisign cert needs to be installed onto the CAS's Trusted Root Certificate store?

    appreciate your help with this

    Kin

     

    Tuesday, October 11, 2011 10:09 PM
  • Hi Fiona,

    The Verisign certificate is installed on the Blue Coat reverse proxy device and this device does the SSL offloading.
    OWA and ActiveSync work perfectly externally so not sure what the difference with Outlook anywhere is.

    The CAS server doesnt have the Verisgin cert, it doesnt have a self signed certificate too, it has an internal certificate from our internal CA. This internal cert has the following SAN in this order:

    DNS Name = internalfqdn
    DNS Name = internal host name
    DNS Name = externalfdqdn
    DNS Name = autodiscover.internal
    DNS Name = autodiscvoer.external

    I read somewhere that for OA to work properly, the order of the DNS names in the SAN matter, is this true?

    I have also tried what you said and installed the cert onto the local machines store but still the same issue.

    Do you think that the Verisign cert needs to be installed onto the CAS's Trusted Root Certificate store?

    appreciate your help with this

    Kin

     

    Hi Kin,

    Not the Verisgin cert, but the certificate installed on your Exchange server. You can get the detail via "Get-ExchangeCertificate |FL". The certificate install on CAS with IIS service binding must be trusted by your client computer. Actually, it is recommended to be a thrid party certificate.

    I read somewhere that for OA to work properly, the order of the DNS names in the SAN matter, is this true?---Not ture. It is the commond name which is listed in "Issue To" in your certificate must match your external URL which is used to connect from external network.

    I have also tried what you said and installed the cert onto the local machines store but still the same issue.---I guess you have install the  Verisign cert  not the internal certificate install on your exchange server. so try again to install the "internal certificate".

    Do you think that the Verisign cert needs to be installed onto the CAS's Trusted Root Certificate store?--Actually it is the recommended by Microsoft to install a third party certificate, but not in this case unless this cert contain your internal name. otherwise it will cause internal autodiscover related issue.

     

     

     


    Fiona
    Wednesday, October 12, 2011 10:40 AM
  • Thanks for replying ,

    still not getting anywhere.
    ive installed the internal cert in the Trusted Root Certification Authorities on the client PC's and still same issue, ive even installed the internal cert on the Exchange mailbox server.

    I cant replace the internal cert on the CAS with the Verisign cert because the verisign cert doesnt have the autodiscover and internal dns names in the SAN.
    The internal cert has many DNS names such as autodiscover and local hostnames in the SAN and its needed for Lync and UM to work properly, so if i replaced the internal cert with the Verisgin cert on the CAS then Lynch and UM would break. Not sure why they have set it up this way but its the setup i've been given.

    I can tell the connection is hitting our network and our domain controllers because when prompted for a username and password, i hit the wrong password on purpose 3 times in a row and the account does get locked.
    So its the point where its trying to go to port 6001 to the mailbox server that fails.

    any idea what this means when i do an rpcping to 6001?

    Exception 1727 (0x000006BF)


     

     

     

    Wednesday, October 12, 2011 10:21 PM
  • Hi,

    Port 6001 is used to connect mailbox server. I don't think it is a problem on Exchange server since you are able to connect in local LAN.

    I am suspecting it is a problem on your firewall. Did you check the log? As I know in ISA/TMG, there is a rule that control Outlook Anywhere access. So please take a check to see if the firewall has the similar one.

    Besides, I reviewed the case again and performed a deep research, the possible resolution below might be helpful:

    1.  Backup IIS configuration file C:\Windows\System32\inetsrv\config\applicationHost.config
    2. Verify if there is the follow:

      <requestFiltering>
      <requestLimits maxAllowedContentLength="2147483648" />
      </requestFiltering>
    3. If not, add the above and then restart IIS.

    Thanks for your time and patience.

     

     


    Fiona
    Thursday, October 13, 2011 8:57 AM
  • Hi,

    rpcping fails with 6001, 6002 and 6004. it just hangs there and then i get this error

    Exception 1727 (0x000006BF)

    but works when using the -E switch.

    Ill check with our network architect tomorrow regarding firewall logs and a rule for OA.

    I did see that fix you suggested in a forum somewhere but the problem is that i dont have that applicationhost.config file on the server, i've performed a search and still cant find it.
    Another thing i noticed is that in the asp.net tab on the rpc folder, it seems to point to a web.config file that doesnt exist.

    could these be related to my issue?

     

    Thursday, October 13, 2011 10:22 AM
  • The the file path is c:\windows\system32\inetsrv\config\applicatiohost.config.

    If the issue continues, you might need to reinstall RPC over HTTP from scretch.

     


    Fiona
    Friday, October 14, 2011 9:08 AM
  • Finally, this issue has been resolved.

    We performed another test by creating a new OWA and Outlook anywhere URL and published it externally but this time the site was going through the old reverse proxy device.
    After installing the internal cert on the client machine, it worked straight away.
    So I threw the ball into the network guys court and after some investigation he found a new feature on the device called network threat protection, seems like this feature was scanning the RPC packets to the mailbox server and stopping OA from working.

    So after 2 weeks of pain, it wasnt my problem in the first place, lesson learnt is push the network guys to check their kit first.

    Thanks for your help Fiona

    Winga

    • Marked as answer by wingadean Wednesday, October 19, 2011 3:58 AM
    Wednesday, October 19, 2011 3:58 AM