none
Issue in Global Services Monitor GSM

    Question

  • Dear all,

    I'm facing a problem in Global Services Monitor.

    The Resources pool contains 4 Management Servers; 2 old and 2 recently installed.
    The GSM was installed and was working normally on the old Management Servers.
    But, after increasing the number of management servers to be 4 instead of two, the problem appeared.
    The GSM is firing alerts on the new MS's and their state are critical (old 2 servers are healthy).
    The alert description is as below:

    Global Service Monitor Modules: Failed to discover Global Service Monitor locations.
    Failure step: 'Couldn't get the ACS endpoint from discovery service. SubscriptionId: 'a6846da0-e5d7-4bea-ab13-836d89364b60', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/''
    Message: 'Could not establish trust relationship for the SSL/TLS secure channel with authority 'gsm-prod.systemcenter.microsoft.com'.'
    Details: 'System.ServiceModel.Security.SecurityNegotiationException: Could not establish trust relationship for the SSL/TLS secure channel with authority 'gsm-prod.systemcenter.microsoft.com'. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
    at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
    at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
    at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
    at System.Net.ConnectStream.WriteHeaders(Boolean async)
    --- End of inner exception stack trace ---
    at System.Net.HttpWebRequest.GetResponse()
    at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
    --- End of inner exception stack trace ---

    Server stack trace: 
    at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
    at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
    at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
    at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]: 
    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.IDiscovery.GetEndpoints(String subscriptionId)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.<>c__DisplayClass1.<DiscoverAcsEndpoint>b__0(IDiscovery service)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.RestCallHelper.ExecuteRestCall[TContract](Uri endpointUri, WebProxy webProxy, String accessToken, RestMethod`1 method)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.DiscoverAcsEndpoint(String subscriptionId, Uri outsideInServiceBaseUri, WebProxy proxy)
    at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.DiscoveryWriteActionModule.Execute()'

    Any clue?

    Regards,
    Khaled A. Hamad

    Friday, September 20, 2013 9:21 PM

Answers

  • Dear all,

    The issue was due to one of the Microsoft Trusted Root CA certificates (Baltimore Cyber Trust Root) was missing from the 2 Management Servers due to the fact that the KB931125 was not installed - which is responsible to install the needed Microsoft Trusted Root CA certificates which are used in general by Microsoft web-sites and applications.

    Issue resolved by exporting the missing certificate (Baltimore Cyber Trust Root) from one of the other Management Servers which were working and imported it onto the 2 new Management Servers.

    A better (more complete) solution would be install KB931125 directly: http://www.microsoft.com/en-us/download/details.aspx?id=6149

    Thanks.
    Khaled A. Hamad


    Thursday, October 17, 2013 10:25 AM

All replies

  • is the Windows Identity Foundation installed on both new MSs (if yes, did you restart the Health service)? do both have Internet access?

    is there a proxy server to connect to the internet, check this link http://technet.microsoft.com/en-us/library/jj860373.aspx

    Saturday, September 21, 2013 3:41 AM
  • Khaled,

    Looks to me like the 2 new servers don't have internet access, either directly or via a proxy server. Make sure they are set up exactly the same as the 2 which are working.

    Cheers

    Luke

    Sunday, September 22, 2013 1:56 AM
  • is the Windows Identity Foundation installed on both new MSs (if yes, did you restart the Health service)? do both have Internet access?

    is there a proxy server to connect to the internet, check this link http://technet.microsoft.com/en-us/library/jj860373.aspx

    Windows Identity Foundation is installed on both servers.

    Also, both servers have internet access and there is no proxy connection required.

    Monday, September 23, 2013 9:52 AM
  • Khaled,

    Looks to me like the 2 new servers don't have internet access, either directly or via a proxy server. Make sure they are set up exactly the same as the 2 which are working.

    Cheers

    Luke

    Windows Identity Foundation is installed on both servers.

    Also, both servers have internet access and there is no proxy connection required.


    Monday, September 23, 2013 9:55 AM
  • Dear all,

    The issue was due to one of the Microsoft Trusted Root CA certificates (Baltimore Cyber Trust Root) was missing from the 2 Management Servers due to the fact that the KB931125 was not installed - which is responsible to install the needed Microsoft Trusted Root CA certificates which are used in general by Microsoft web-sites and applications.

    Issue resolved by exporting the missing certificate (Baltimore Cyber Trust Root) from one of the other Management Servers which were working and imported it onto the 2 new Management Servers.

    A better (more complete) solution would be install KB931125 directly: http://www.microsoft.com/en-us/download/details.aspx?id=6149

    Thanks.
    Khaled A. Hamad


    Thursday, October 17, 2013 10:25 AM