locked
High utilization/ServerManager.Log updated every 80 seconds RRS feed

  • Question

  • Hi.  I have a Windows Server 2008 system that is having problems with slow response times.  One app usually takes 20-30 times as long to return a query than it does immediately after rebooting the server.

    In procexp, I see that svchost.exe -k LocalServiceNetworkRestricted jumps up to about 25% of the processor every 80 seconds or so and remains there for about 20 seconds.  I don't know if it's related, but I also see that some sort of cycle is logged in the ServerManager.Log file about every 80 seconds also.  There is also a set of messages in the CBS log at the same times that go like this:

    Session: 29977493:1658306072 initialized.                  
    Session: 29977493:1658306072 finalized.  Reboot required: no
    Session: 29977493:2443618572 initialized.                  

    The SeverManger.Log entries are over 500 lines each, starting with a "[CBS] LastModified CBS Time (UTC)" message and including several [DISCOVERY] and [ApplicationServer] entries each interval.

    Can anyone tell me why these entries are being generated so often and/or how I can stop them or at least lengthen the interval they occur?

    Thanks!
    Wednesday, December 31, 2008 11:59 PM

Answers

  • Hi,

     

    According to the research, by default the Server Manager will periodically refresh the information that is shown on Server Manager. When the server manager refresh occurs, it will query all the CBS components of the installed server roles and feature. Meanwhile, the server manager records the "[CBS] LastModified CBS Time (UTC)" logs in the Servermanager.log.

     

    In this way, you may try configuring the refresh interval in server manager console to lengthen the interval they occur, or you may disable the refresh to improve the system performance as you need.

     

    Steps:

     

    1. Launch the Server Manager.

    2. Click on "Configure refresh" under the end of the Server Manager.

    3. Configure the refresh interval as you need. You may select "Minutes", “Hours", or “Days"

    4. You may also disable the refresh by selecting "Disable refresh".

    5. Click ok to confirm.

    6. Reboot the server.

     

    Afterwards, you may monitor if these entries will re-occur within the servermanager.log. If the issue still continues, please send the servermanager.log to tfwst@micorosoft.com for further research.

     

    Hope it helps.


    David Shen - MSFT
    • Edited by David Shen Friday, January 2, 2009 10:36 AM modify
    • Marked as answer by David Shen Tuesday, January 6, 2009 1:49 AM
    Friday, January 2, 2009 10:35 AM
  • Hi Wendell,

     

    Thanks for the reply.

     

    I am sorry that I have little knowledge on Wireshark. Meanwhile, I am not sure whether it will notice all the network protocols when capturing the network traffic.

     

    There is a Network Monitor 3.2 which is available via the following link. I would like to suggest that you use it to capture all the network traffic between the two computers during the copy operation.

     

    Download: Microsoft Network Monitor 3.2

    http://www.microsoft.com/downloads/details.aspx?FamilyID=f4db40af-1e08-4a21-a26b-ec2f4dc4190d&DisplayLang=en

     

    General steps:

     

    1. Enable the Capture Filter "IPv4.Address == <ip of the client>" and start capture.

    2. Restart one of clients to reproduce the issue.

    3. Stop capture and save to *.cap file.

     

    For more information, please refer to:

     

    How to use Network Monitor to capture network traffic

    http://support.microsoft.com/kb/812953

     

    Please use Windows Live SkyDrive (http://www.skydrive.live.com/) to upload the capture file, and then give me the download address so that we can perform further analysis on it.

     

    Hope you like it!


    David Shen - MSFT
    • Marked as answer by David Shen Sunday, January 11, 2009 11:41 AM
    Friday, January 9, 2009 2:48 AM

All replies

  • Hi,

     

    According to the research, by default the Server Manager will periodically refresh the information that is shown on Server Manager. When the server manager refresh occurs, it will query all the CBS components of the installed server roles and feature. Meanwhile, the server manager records the "[CBS] LastModified CBS Time (UTC)" logs in the Servermanager.log.

     

    In this way, you may try configuring the refresh interval in server manager console to lengthen the interval they occur, or you may disable the refresh to improve the system performance as you need.

     

    Steps:

     

    1. Launch the Server Manager.

    2. Click on "Configure refresh" under the end of the Server Manager.

    3. Configure the refresh interval as you need. You may select "Minutes", “Hours", or “Days"

    4. You may also disable the refresh by selecting "Disable refresh".

    5. Click ok to confirm.

    6. Reboot the server.

     

    Afterwards, you may monitor if these entries will re-occur within the servermanager.log. If the issue still continues, please send the servermanager.log to tfwst@micorosoft.com for further research.

     

    Hope it helps.


    David Shen - MSFT
    • Edited by David Shen Friday, January 2, 2009 10:36 AM modify
    • Marked as answer by David Shen Tuesday, January 6, 2009 1:49 AM
    Friday, January 2, 2009 10:35 AM
  • Hi David.  Thanks for the info on how to disable Server Manager refreshes.  I did that and it immediately stopped running up the CPU utilization like it was.

    Unfortunately, that didn't turn out to help the original problem of the server slowing down some time after rebooting.  However, I may have gotten closer last night.  After rebooting, I noticed that response was very good until some time after I had accessed the server from a Windows 2000 client, and after I had started a RoboCopy job that syncs a shared volume with an external USB2 drive I use for backups.  This sounds really strange, but after I rebooted the server I ran Wireshark and it didn't show ANY network traffic between my PC and the server until I restarted RoboCopy.  After that there were thousands of SMB2 protocol entries, showing two LOCK requests and responses before each READ request.  The application I use to check response changed from a couple of seconds to over a minute!

    Can you or anyone else think of a reason the Windows 2008 Server would change how it communicates with an application based on something RoboCopy could be doing?  They really sound unrelated to me.

    Thanks,
    Wendell
    Tuesday, January 6, 2009 4:44 PM
  • Hi Wendell,

     

    Thanks for the reply.

     

    As you encountered the performance issue when you try Robocopy some files from a Windows 2000 client to the Windows Server 2008 and you found some of SMB2 protocol entries in the network traffic. Personally, I suspect that the issue may be related to SMB2 protocol on Windows Server 2008.

     

    Would you please refer to the following steps to test and check if it will be helpful to resolve the performance issue?

     

    1. Please disable SMB2 on Windows Server 2008

     

    Steps:

     

    a. Run "regedit" on Windows Server 2008 based computer.

     

    b. Expand and locate the sub tree as follow.

     

    HKLM\System\CurrentControlSet\Services\LanmanServer\Parameters

     

    c. Add a new REG_DWORD key with the name of "Smb2" (without quotation mark)

     

    Value name: Smb2

    Value type: REG_DWORD

     

    d. Set the value to 0 to disable SMB 2.0

     

    2. Please disable Raw SMB on the Windows Server 2008

     

    a. Run "regedit" on Windows Server 2008 based computer.

     

    b. Expand and locate the sub tree as follow

     

    HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters

     

    c. From the Edit menu choose Add Value.

     

    d. Add the following:

     

    Value Name: EnableRaw

    Value Type: REG_DWORD

    Data: 0 

     

    3. Please change the registry key value of LMcompatibilitylevel to 2 on the Windows Server 2008

     

    HKLM\System\CurrentControlSet\Control\LSA

     

    Lmcompatibilitylevel [REG_DWORD] = 2 to only NTLM Response

     

    4. Please configure the Local Policy on Windows Server 2008

     

    a. Open Local Group Policy Editor with gpedit.msc

     

    b. Expand and local to the following node

     

    Computer Configuration\Windows Settings\Security Settings\Local Policies\Security options

     

    c. Adjust the corresponding security settings as followed.

     

    Microsoft network client: Digitally sign communications (if server agrees) to "enabled"

    Microsoft network client: Digitally sign communications (always) to "disabled"

    Microsoft network server: Digitally sign communications (if server agrees) to "enabled"

    Microsoft network server: Digitally sign communications (always) to "disabled"

     

    Please note: after making the modification of these registry keys and policies settings, you may need to reboot the server to make it take into effect.

     

    Hope it helps.


    David Shen - MSFT
    • Edited by David Shen Wednesday, January 7, 2009 6:53 AM modify
    Wednesday, January 7, 2009 6:53 AM
  • Wow David, thanks for the awesome help!

    I disabled the scheduled task that runs the Robocopy backup and the response was good for several hours.  But it did eventually degrade again.  Not sure who started what that affected it so much.

    I've made the registry changes, but in gpedit.msc I am not able to change the 'Microsoft network server: Digitally sign communications (always) to "disabled"' setting--the Enable & Disable buttons are both grayed out.  Could this be because I've already disabled the SMB support in the registry?

    Thanks again,
    Wendell
    Wednesday, January 7, 2009 5:20 PM
  • Hi Wendell,

     

    The local group policy setting "Microsoft network server: Digitally sign communications (always)" maybe grey out if there is GPO enabled as the same setting in Default Domain Controller Policy or other up level OU group policy which overwrite the registry value on the problematic server.

     

    You may also disable this setting by set the value of EnableSecurtiySignature of the following registry key to 0

     

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\lanmanserver\parameters\EnableSecuritySignature

     

    Hope it helps.


    David Shen - MSFT
    Thursday, January 8, 2009 6:09 AM
  • Thanks again David.

    I opened gpedit.msc again and the setting showed "Disabled" already.  I guess I'd opened the original gpedit.msc before finishing the registry changes.  Anyway, it's off.

    The changes did give us an improvement.  Wireshark now shows far fewer read requests/responses--I think it went down from requests to one.  And I'm not seeing a bunch of entries in the Windows/Security event log showing "A Kerberos service ticket was renewed."  I do still get a lot of entries saying "Special privileges assigned to new logon."--up to 7 logon/logoff entries in the same second, 'tho not every second.

    Anyway, after the change the SMB traffic appears to be doing better without the digital certificates, but I still can't explain why right after the server nothing gets noticed in Wireshark to scan a (Clarion--not SQL Server) database, but after some time and/or something runs (Robocopy seemed to be one of the later), it goes back to SMB protocol.  There seem to be about the same number of requests as before, but the query completes in about 1/3 of the time it was.

    Do you think there could be another protocol (not noticed by Wireshark?) that is used up to something changing on the Win2008 server that causes it to switch to SMB?

    Thanks,
    Wendell

    Thursday, January 8, 2009 7:05 PM
  • Hi Wendell,

     

    Thanks for the reply.

     

    I am sorry that I have little knowledge on Wireshark. Meanwhile, I am not sure whether it will notice all the network protocols when capturing the network traffic.

     

    There is a Network Monitor 3.2 which is available via the following link. I would like to suggest that you use it to capture all the network traffic between the two computers during the copy operation.

     

    Download: Microsoft Network Monitor 3.2

    http://www.microsoft.com/downloads/details.aspx?FamilyID=f4db40af-1e08-4a21-a26b-ec2f4dc4190d&DisplayLang=en

     

    General steps:

     

    1. Enable the Capture Filter "IPv4.Address == <ip of the client>" and start capture.

    2. Restart one of clients to reproduce the issue.

    3. Stop capture and save to *.cap file.

     

    For more information, please refer to:

     

    How to use Network Monitor to capture network traffic

    http://support.microsoft.com/kb/812953

     

    Please use Windows Live SkyDrive (http://www.skydrive.live.com/) to upload the capture file, and then give me the download address so that we can perform further analysis on it.

     

    Hope you like it!


    David Shen - MSFT
    • Marked as answer by David Shen Sunday, January 11, 2009 11:41 AM
    Friday, January 9, 2009 2:48 AM
  •  Hi David.

    Sorry for the delay.  If I've done this right, the .cap files can be downloaded from ZIP file containing two .cap files.

    There are two traces.  The MSCRM_Immediately_After_Reboot.cap file contains the trace immediately after the Win2008 serer is rebooted.  The MSCRM_After_Server_Runs_Awhile.cap file contains a trace of the same program doing the same thing sometime after the reboot.  The "somettime" is a mystery to me--I haven't identified what happens to change how it works.  I don't really know SMB protocols, but it appears to me that it's using port 139 and the trace tool is logging the I/O as "TCP" calls right after the server is booted, but after whateven happens happens, it changes to SMB protocols and is 15-20 times slower.  I did make the registry changes you mentioned above to, as best I understand, disable SMB.  So I really don't know why it seems to revert.  Could it have something to do with the SP level of the Windows XP or still a couple of Win2000 PCs?

    I really appreciate your help. 

    Wendell
    Tuesday, January 13, 2009 2:53 AM
  • Hi Wendell,

     

    Thanks for the reply.

     

    SMB 2.0 was introduced in Windows Vista and Windows Server 2008.  SMB 1.0 was designed for early Windows network operating systems such as Microsoft LAN Manager and Windows for Workgroups.  SMB 2.0 is designed for the needs of the next generation of file servers.  Both Windows Server 2008 and Windows Vista support SMB 1.0 and SMB 2.0. SMB uses TCP port 445 for file transmission in the network. You may find the SMB protocol in MSCRM_Immediately_After_Reboot.cap use 445 port.

     

    Since Windows supports both file and printer sharing traffic by using the Server Message Block (SMB) protocol directly hosted on TCP. This differs from earlier operating systems, in which SMB traffic requires the NetBIOS over TCP (NBT) protocol to work on a TCP/IP transport. If both the direct hosted and NBT interfaces are enabled, both methods are tried at the same time and the first to respond is used. NetBIOS over TCP (NBT) protocol use TCP port 139 for transmission.

     

    Computers running Windows Server 2008 or Windows Vista support both SMB 1.0 and SMB 2.0. The version of SMB that is used for file sharing operations is determined during the SMB session negotiation. The following table shows which version of SMB that is used for various combinations of client and server computers.

     

    Client

    Server

    Version of SMB used

    Windows Server 2008 or Windows Vista

    Windows Server 2008 or Windows Vista

    SMB 2.0

    Windows Server 2008 or Windows Vista

    Windows XP, Windows Server 2003, or Windows 2000

    SMB 1.0

    Windows XP, Windows Server 2003, or Windows 2000

    Windows Server 2008 or Windows Vista

    SMB 1.0

    Windows XP, Windows Server 2003, or Windows 2000

    Windows XP, Windows Server 2003, or Windows 2000

    SMB 1.0

     

    In this case, if the clients in your environment are Windows XP or Windows 2000, meanwhile the server is Windows Server 2008, the system should negotiate and use SMB 1.0 for file transmission.

     

    Based on the research of the MSCRM_After_Server_Runs_Awhile.cap, I found that the file transmission in your network uses NetBIOS session which use TCP 139 port, this is because of the reason that the system on the server side responds first with NetBIOS over TCP, which leads to it use TCP 139 (NBT) for file transmission.

     

    For more information, please refer to

     

    Direct hosting of SMB over TCP/IP

    http://support.microsoft.com/default.aspx/kb/204279

     

    New Networking Features in Windows Server 2008 and Windows Vista

    http://technet.microsoft.com/en-us/library/bb726965.aspx

     

    Overview of SMB 2.0

    http://blogs.technet.com/askperf/archive/2008/05/30/two-minute-drill-overview-of-smb-2-0.aspx

     

    Hope the information will be helpful.


    David Shen - MSFT
    Wednesday, January 14, 2009 3:25 AM
  • Hi David.

    Thanks again for being so helpful.

    I looked at the URLs you provided, and found that our real problem is not which version of SMB we are using--the XP users could not have been using anything other than SMB 1.0, and when I disabled Netbios over TCPIP on my Vista machine wasn't noticably faster than when it negotiated SMB 1.0 in the "After_Server_Runs_Awhile" trace.

    So, there is something else that is really slowing down response on our Win2008 server.  Can you please make any suggestions as to how I would start diagnosing the problem? 

    Thanks,
    Wendell
    Wednesday, January 14, 2009 6:21 PM