none
Issue in Global Services Monitor GSM

    Question

  • Dear all,

    I'm facing a problem in Global Services Monitor.

    The Resources pool contains 4 Management Servers; 2 old and 2 recently installed.
    The GSM was installed and was working normally on the old Management Servers.
    But, after increasing the number of management servers to be 4 instead of two, the problem appeared.
    The GSM is firing alerts on the new MS's and their state are critical (old 2 servers are healthy).
    The alert description is as below:

    Global Service Monitor Modules: Failed to discover Global Service Monitor locations.
    Failure step: 'Couldn't get the ACS endpoint from discovery service. SubscriptionId: 'a6846da0-e5d7-4bea-ab13-836d89364b60', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/''
    Message: 'Could not establish trust relationship for the SSL/TLS secure channel with authority 'gsm-prod.systemcenter.microsoft.com'.'
    Details: 'System.ServiceModel.Security.SecurityNegotiationException: Could not establish trust relationship for the SSL/TLS secure channel with authority 'gsm-prod.systemcenter.microsoft.com'. ---> System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
    at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
    at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
    at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
    at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
    at System.Net.ConnectStream.WriteHeaders(Boolean async)
    --- End of inner exception stack trace ---
    at System.Net.HttpWebRequest.GetResponse()
    at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
    --- End of inner exception stack trace ---

    Server stack trace: 
    at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
    at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
    at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
    at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
    at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]: 
    at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.IDiscovery.GetEndpoints(String subscriptionId)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.<>c__DisplayClass1.<DiscoverAcsEndpoint>b__0(IDiscovery service)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.RestCallHelper.ExecuteRestCall[TContract](Uri endpointUri, WebProxy webProxy, String accessToken, RestMethod`1 method)
    at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.DiscoverAcsEndpoint(String subscriptionId, Uri outsideInServiceBaseUri, WebProxy proxy)
    at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.DiscoveryWriteActionModule.Execute()'

    Any clue?

    Regards,
    Khaled A. Hamad

    Friday, September 20, 2013 9:21 PM

Answers

  • Dear all,

    The issue was due to one of the Microsoft Trusted Root CA certificates (Baltimore Cyber Trust Root) was missing from the 2 Management Servers due to the fact that the KB931125 was not installed - which is responsible to install the needed Microsoft Trusted Root CA certificates which are used in general by Microsoft web-sites and applications.

    Issue resolved by exporting the missing certificate (Baltimore Cyber Trust Root) from one of the other Management Servers which were working and imported it onto the 2 new Management Servers.

    A better (more complete) solution would be install KB931125 directly: http://www.microsoft.com/en-us/download/details.aspx?id=6149

    Thanks.
    Khaled A. Hamad


    Thursday, October 17, 2013 10:25 AM

All replies

  • is the Windows Identity Foundation installed on both new MSs (if yes, did you restart the Health service)? do both have Internet access?

    is there a proxy server to connect to the internet, check this link http://technet.microsoft.com/en-us/library/jj860373.aspx

    Saturday, September 21, 2013 3:41 AM
  • Khaled,

    Looks to me like the 2 new servers don't have internet access, either directly or via a proxy server. Make sure they are set up exactly the same as the 2 which are working.

    Cheers

    Luke

    Sunday, September 22, 2013 1:56 AM
  • is the Windows Identity Foundation installed on both new MSs (if yes, did you restart the Health service)? do both have Internet access?

    is there a proxy server to connect to the internet, check this link http://technet.microsoft.com/en-us/library/jj860373.aspx

    Windows Identity Foundation is installed on both servers.

    Also, both servers have internet access and there is no proxy connection required.

    Monday, September 23, 2013 9:52 AM
  • Khaled,

    Looks to me like the 2 new servers don't have internet access, either directly or via a proxy server. Make sure they are set up exactly the same as the 2 which are working.

    Cheers

    Luke

    Windows Identity Foundation is installed on both servers.

    Also, both servers have internet access and there is no proxy connection required.


    Monday, September 23, 2013 9:55 AM
  • Dear all,

    The issue was due to one of the Microsoft Trusted Root CA certificates (Baltimore Cyber Trust Root) was missing from the 2 Management Servers due to the fact that the KB931125 was not installed - which is responsible to install the needed Microsoft Trusted Root CA certificates which are used in general by Microsoft web-sites and applications.

    Issue resolved by exporting the missing certificate (Baltimore Cyber Trust Root) from one of the other Management Servers which were working and imported it onto the 2 new Management Servers.

    A better (more complete) solution would be install KB931125 directly: http://www.microsoft.com/en-us/download/details.aspx?id=6149

    Thanks.
    Khaled A. Hamad


    Thursday, October 17, 2013 10:25 AM
  • Hi

    Is there any requirement to install the Microsoft Root Certificate on the server where SCOM console is working? Shall I need to purchase Windows Azure Subscription also for GSM? Please let me know.

    The scenario is - I have one SCOM server (Including all the server roles on the single server) and other server where VMM server and SCOM console is installed. I have installed GSM Management Packs on the SCOM server and configured one Web Availability Monitor to be monitored from external servers (e.g. Chicago).

    I am getting the below error:-

     

    Log Name:      Operations Manager
    Source:        Health Service Modules Ex
    Date:          9/11/2014 7:14:26 PM
    Event ID:      10001
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      SCOMCLOUD.abc.in
    Description:
    Global Service Monitor Modules:  Failed step: 'Couldn't get the ACS endpoint from discovery service. SubscriptionId: '1f156904-532e-416f-b570-1141438392a3', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/''. Diagnostic context: RequestId = '0fe72d85-989c-4c1b-89c1-1f4b641c1578', New ConfigHash = '65afc4b6-c18d-5e68-56d3-482e2db1851a', '1' tests, Last ConfigHash = '00000000-0000-0000-0000-000000000000'. Exception: 'There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.'


    One or more workflows were affected by this. 

    Workflow name: Microsoft.SystemCenter.Omonline.OutsideIn.Discovery.ConfigUploaderRule 
    Instance name: Global Service Monitor 
    Instance ID: {298CB0DA-4453-EFD2-A7AC-C2E8F2F7100D} 
    Management group: SCOMGROUP
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Health Service Modules Ex" />
        <EventID Qualifiers="0">10001</EventID>
        <Level>3</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2014-09-11T13:44:26.000000000Z" />
        <EventRecordID>149790</EventRecordID>
        <Channel>Operations Manager</Channel>
        <Computer>SCOMCLOUD.abc.in</Computer>
        <Security />
      </System>
      <EventData>
        <Data>SCOMGROUP</Data>
        <Data>Microsoft.SystemCenter.Omonline.OutsideIn.Discovery.ConfigUploaderRule</Data>
        <Data>Global Service Monitor</Data>
        <Data>{298CB0DA-4453-EFD2-A7AC-C2E8F2F7100D}</Data>
        <Data>Failed step: 'Couldn't get the ACS endpoint from discovery service. SubscriptionId: '1f156904-532e-416f-b570-1141438392a3', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/''. Diagnostic context: RequestId = '0fe72d85-989c-4c1b-89c1-1f4b641c1578', New ConfigHash = '65afc4b6-c18d-5e68-56d3-482e2db1851a', '1' tests, Last ConfigHash = '00000000-0000-0000-0000-000000000000'. Exception: 'There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.'</Data>
        <Data>Global Service Monitor Modules</Data>
        <Data>Couldn't get the ACS endpoint from discovery service. SubscriptionId: '1f156904-532e-416f-b570-1141438392a3', OutsideInServiceBaseUri: 'https://gsm-prod.systemcenter.microsoft.com/'</Data>
        <Data>RequestId = '0fe72d85-989c-4c1b-89c1-1f4b641c1578', New ConfigHash = '65afc4b6-c18d-5e68-56d3-482e2db1851a', '1' tests, Last ConfigHash = '00000000-0000-0000-0000-000000000000'</Data>
        <Data>There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.</Data>
        <Data>System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at https://gsm-prod.systemcenter.microsoft.com/DiscoveryService/1f156904-532e-416f-b570-1141438392a3/Endpoints that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---&gt; System.Net.WebException: The remote name could not be resolved: 'gsm-prod.systemcenter.microsoft.com'
       at System.Net.HttpWebRequest.GetResponse()
       at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
       --- End of inner exception stack trace ---

    Server stack trace: 
       at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason)
       at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)
       at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
       at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.IDiscovery.GetEndpoints(String subscriptionId)
       at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.&lt;&gt;c__DisplayClass1.&lt;DiscoverAcsEndpoint&gt;b__0(IDiscovery service)
       at Microsoft.SystemCenter.Cloud.SharedLibrary.RestCallHelper.ExecuteRestCall[TContract](Uri endpointUri, WebProxy webProxy, String accessToken, RestMethod`1 method)
       at Microsoft.SystemCenter.Cloud.SharedLibrary.Discovery.DiscoveryHelper.DiscoverAcsEndpoint(String subscriptionId, Uri outsideInServiceBaseUri, WebProxy proxy)
       at Microsoft.SystemCenter.Cloud.OutsideInUnitModule.ConfigUploaderWriteActionModule.Execute()</Data>
      </EventData>
    </Event>

    Any HELP would be really Appreciated.

    Thanks in advance.


    Abhinav | MCTS-Server Virtualization


    • Edited by abhiagg Sunday, September 14, 2014 6:25 AM
    Sunday, September 14, 2014 5:49 AM