locked
SCOM 1807 Management Server - Not Monitored RRS feed

  • Question

  • Hi -

    I would like to ask what should I check and perform why our newly installed SCOM 1807 Management Server is still in "Not Monitored" state for couple days now.

    I have performed that following troubleshooting:

    - Telnet RMS and SQL port (OK)

    - Stop HealthService / Rename "Health Service State" folder / Start HealthService

    - Restart RMS HealthService

    - Check Operations Manager Event Logs

    Thanks!

    Monday, March 23, 2020 8:52 AM

All replies

  • Hi,

    What does the Operations Manager event log say? There should be some indications there, either errors or warnings.


    Also note that SCOM 1807 is out of support, it would be advisable to upgrade to either SCOM 2016 or SCOM 2019 as soon as possible.

    Best regards,
    Leon


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, March 23, 2020 9:34 AM
  • Hi -

    There's no significant event log found aside from this one.

    Log Name:      Operations Manager
    Source:        OpsMgr Management Configuration
    Date:          3/23/2020 11:33:02 PM
    Event ID:      29120
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      XXXXXXXXXXXXXX
    Description:
    OpsMgr Management Configuration Service failed to process configuration request (Xml configuration file or management pack request) due to the following exception

    Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessException: Data access operation failed
       at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.CmdbDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingConfigurationDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.ProcessManagementPackRequest(ManagementPackRequestDescriptor requestDescriptor)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.Execute()
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ThreadManager.ResponseThreadStart(Object state)
    -----------------------------------
    System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
       at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
       at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
       at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
       at System.Data.SqlClient.TdsParserStateObject.TryReadUInt32(UInt32& value)
       at System.Data.SqlClient.TdsParserStateObject.TryReadPlpLength(Boolean returnPlpNullIfNull, UInt64& lengthLeft)
       at System.Data.SqlClient.TdsParserStateObject.TryReadPlpBytes(Byte[]& buff, Int32 offst, Int32 len, Int32& totalBytesRead)
       at System.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj, SqlCommandColumnEncryptionSetting columnEncryptionOverride, String columnName)
       at System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32 i, Boolean readHeaderOnly)
       at System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn)
       at System.Data.SqlClient.SqlDataReader.GetSqlBinary(Int32 i)
       at System.Data.SqlClient.SqlDataReader.TryGetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length, Int64& remaining)
       at System.Data.SqlClient.SqlDataReader.GetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
       at System.Data.SqlClient.SqlDataReader.GetBytes(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadResourceList(SqlDataReader reader)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadData(SqlDataReader reader)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ReaderSqlCommandOperation.SqlCommandCompleted(IAsyncResult asyncResult)
    ClientConnectionId:02025c4f-c9df-4614-894c-c1ccdf257927
    Error Number:-2,State:0,Class:11
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="OpsMgr Management Configuration" />
        <EventID Qualifiers="32768">29120</EventID>
        <Level>3</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2020-03-23T22:33:02.201567600Z" />
        <EventRecordID>167606</EventRecordID>
        <Channel>Operations Manager</Channel>
        <Computer>XXXXXXXXXXXXXXX</Computer>
        <Security />
      </System>
      <EventData>
        <Data>OpsMgr Management Configuration Service</Data>
        <Data>Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessException: Data access operation failed
       at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.DataAccessOperation.ExecuteSynchronously(Int32 timeoutSeconds, WaitHandle stopWaitHandle)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.CmdbDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.TracingConfigurationDataProvider.GetManagementPack(Guid managementPackVersionDependentId)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.ProcessManagementPackRequest(ManagementPackRequestDescriptor requestDescriptor)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.AgentRequestProcessor.Execute()
       at Microsoft.EnterpriseManagement.ManagementConfiguration.Engine.ThreadManager.ResponseThreadStart(Object state)
    -----------------------------------
    System.Data.SqlClient.SqlException (0x80131904): Execution Timeout Expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---&gt; System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out
       at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniError(TdsParserStateObject stateObj, UInt32 error)
       at System.Data.SqlClient.TdsParserStateObject.ReadSniSyncOverAsync()
       at System.Data.SqlClient.TdsParserStateObject.TryReadNetworkPacket()
       at System.Data.SqlClient.TdsParserStateObject.TryPrepareBuffer()
       at System.Data.SqlClient.TdsParserStateObject.TryReadUInt32(UInt32&amp; value)
       at System.Data.SqlClient.TdsParserStateObject.TryReadPlpLength(Boolean returnPlpNullIfNull, UInt64&amp; lengthLeft)
       at System.Data.SqlClient.TdsParserStateObject.TryReadPlpBytes(Byte[]&amp; buff, Int32 offst, Int32 len, Int32&amp; totalBytesRead)
       at System.Data.SqlClient.TdsParser.TryReadSqlValue(SqlBuffer value, SqlMetaDataPriv md, Int32 length, TdsParserStateObject stateObj, SqlCommandColumnEncryptionSetting columnEncryptionOverride, String columnName)
       at System.Data.SqlClient.SqlDataReader.TryReadColumnInternal(Int32 i, Boolean readHeaderOnly)
       at System.Data.SqlClient.SqlDataReader.TryReadColumn(Int32 i, Boolean setTimeout, Boolean allowPartiallyReadColumn)
       at System.Data.SqlClient.SqlDataReader.GetSqlBinary(Int32 i)
       at System.Data.SqlClient.SqlDataReader.TryGetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length, Int64&amp; remaining)
       at System.Data.SqlClient.SqlDataReader.GetBytesInternal(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
       at System.Data.SqlClient.SqlDataReader.GetBytes(Int32 i, Int64 dataIndex, Byte[] buffer, Int32 bufferIndex, Int32 length)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadResourceList(SqlDataReader reader)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.CmdbOperations.ManagementPackGetOperation.ReadData(SqlDataReader reader)
       at Microsoft.EnterpriseManagement.ManagementConfiguration.DataAccessLayer.ReaderSqlCommandOperation.SqlCommandCompleted(IAsyncResult asyncResult)
    ClientConnectionId:02025c4f-c9df-4614-894c-c1ccdf257927
    Error Number:-2,State:0,Class:11</Data>
      </EventData>
    </Event>

    Also, thanks for the heads up. We will discuss the upgrade to either SCOM 2016 and SCOM 2019. We just decided to upgrade from 2012 R2 last Dec 2018.

    Monday, March 23, 2020 11:16 PM
  • The following is the interesting part:
    "System.Data.SqlClient.SqlException (0x80131904): Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. ---> System.ComponentModel.Win32Exception (0x80004005): The wait operation timed out" 

    I would say there are issues with inserting data into the database, could you verify the database connection is working properly? Also are the SCOM databases healthy?

    Any 2115 events (Warnings) in the Operations Manager event log?


    Blog: https://thesystemcenterblog.com LinkedIn:

    Monday, March 23, 2020 11:28 PM
  • Yes. The SCOM databases are healthy. There's also no Event 2115 on the RMS and the affected MS.

    The only different setup here is that the new MS is located on another IP segment (from different location) but within the same domain.

    Thanks.

    Tuesday, March 24, 2020 12:36 AM
  • Did you check that the firewall is not blocking the communication?

    Configuring a Firewall for Operations Manager
    https://docs.microsoft.com/en-us/system-center/scom/plan-security-config-firewall?view=sc-om-1807


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, March 24, 2020 12:38 AM
  • Already check that all required TCP and UDP ports are not block in Firewall.

    TCP 5723/5724

    TCP 1433 / UDP 1434

    TCP 135 / UDP 137

    TCP 445

    Tuesday, March 24, 2020 12:53 AM
  • Please verify these with the Test-NetConnection PowerShell cmdlet.

    For example from the SCOM management server:

    Test-NetConnection -Port 1433 -ComputerName "SQL Server Name"

    The TcpTestSucceeded should return "Success".

    Make sure you check all firewall appliances (both software and hardware appliances).

    It is recommended to have all SCOM servers in the same subnet as the database servers to minimize latency, SCOM requires a good latency between the database server, you could monitor your network to check if there are any latency issues.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, March 24, 2020 1:01 AM
  • Thanks.

    Verified that port is reachable. The network latency is the one we are currently looking now. Is there network latency requirements in SCOM communication from MS to DB?

    Tuesday, March 24, 2020 1:38 AM
  • Yes there is an older documentation of the minimum requirements:

    https://docs.microsoft.com/en-us/previous-versions/system-center/system-center-2012-R2/dn249696%28v%3dsc.12%29#minimum-network-connectivity-speeds

    This may also depend on the size of your SCOM environment, the amount of management packs and so on.


    Blog: https://thesystemcenterblog.com LinkedIn:

    Tuesday, March 24, 2020 1:47 AM
  • Thank you Leon in providing suggestions and possible solutions.

    I have already reached out MS support for further investigation what's the root cause.

    Will keep this post updated for any developments.

    Tuesday, March 24, 2020 1:50 AM