Bug: halt when deleting unneeded update files

Unanswered Bug: halt when deleting unneeded update files

  • Monday, December 10, 2012 2:28 PM
     
     

    If WSUS installed on a heavy IO loaded harddrive (LUN) of a VM, when deleting unneeded update files by Server Cleanup Wizard, MMC node may halt.

    Server OS: Windows Server 2012 Standard

    Errors can be located in Event Log:
    An unhandled exception occurred and the process was terminated.
    Application ID: /LM/W3SVC/1881423753/ROOT/ServerSyncWebService
    Process ID: 3288
    Exception: System.NullReferenceException
    Message: Object reference not set to an instance of an object.
    StackTrace:    at Microsoft.UpdateServices.Internal.DataAccess.GetRevisionIdListForCache(Int64 deploymentChangeNumber, Int64& highestChangeNumber, CategoryIdCacheEntry[]& newCategoryDeployments, CategoryIdCacheEntry[]& deletedCategoryDeployments, RevisionIdCacheEntry[]& newRevisionDeployments, RevisionIdCacheEntry[]& deletedRevisionDeployments)
    at Microsoft.UpdateServices.Internal.RevisionIdCache.UpdateCategoryAndRevisionIdCache(Int64 deploymentChangeNumberAtRefresh)
    at Microsoft.UpdateServices.Internal.ServerImplementation.UpdateCache()
    at Microsoft.UpdateServices.Internal.ServerImplementation.DatabaseChangeHandler(Object byTimer)
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.TimerQueueTimer.CallCallback()
    at System.Threading.TimerQueueTimer.Fire()
    at System.Threading.TimerQueue.FireNextTimers()

    Application: w3wp.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.NullReferenceException
    Stack:
    at Microsoft.UpdateServices.Internal.DataAccess.GetRevisionIdListForCache(Int64, Int64 ByRef, Microsoft.UpdateServices.Internal.CategoryIdCacheEntry[] ByRef, Microsoft.UpdateServices.Internal.CategoryIdCacheEntry[] ByRef, Microsoft.UpdateServices.Internal.RevisionIdCacheEntry[] ByRef, Microsoft.UpdateServices.Internal.RevisionIdCacheEntry[] ByRef)
    at Microsoft.UpdateServices.Internal.RevisionIdCache.UpdateCategoryAndRevisionIdCache(Int64)
    at Microsoft.UpdateServices.Internal.ServerImplementation.UpdateCache()
    at Microsoft.UpdateServices.Internal.ServerImplementation.DatabaseChangeHandler(System.Object)
    at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
    at System.Threading.TimerQueueTimer.CallCallback()
    at System.Threading.TimerQueueTimer.Fire()
    at System.Threading.TimerQueue.FireNextTimers()

    Faulting application name: w3wp.exe, version: 8.0.9200.16384, time stamp: 0x50108835
    Faulting module name: KERNELBASE.dll, version: 6.2.9200.16384, time stamp: 0x5010ab2d
    Exception code: 0xe0434352
    Fault offset: 0x00000000000189cc
    Faulting process id: 0xcd8
    Faulting application start time: 0x01cdcff0fa98da3e
    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
    Faulting module path: C:\Windows\system32\KERNELBASE.dll
    Report Id: 332ed25c-3c39-11e2-93f1-005056b26389
    Faulting package full name: 
    Faulting package-relative application ID:

    Step to repo:

    1 Open Update Services MMC.
    2 Click Options from the left panel.
    3 Select Server Cleanup Wizard from the right.
    4 Press Next.
    MMC node MAY(only when lots of files need to be deleted) be prompted to be reset when deleting unneeded files if it's running longer than usual. Re-open Update Services MMC will fix it. But it will be appeared again another day when many unneeded files exist.


    Eternal Snow

All Replies

  • Tuesday, December 11, 2012 5:43 AM
    Moderator
     
     

    Hi,

    How many updates file did you try to delete at one time?How heavy is your IO loaded harddrive?When doing the cleanup wizard,the performance will be affected during the cleanup.I don't suggest do any other tasks during the cleanup.Your operations on the WSUS console will make the connection to the SQL DB as well.So it may break the connection to WSUS console since the DB is in a heavy workload.Also,in your description, it is not in the production environment,is it?If it is in a large WSUS deployment with several downstream servers,it is recommended installing on a physical machine with a full version of back-end SQL server.So it will not affected by overload in performance.

    In addition,a well-maintained WSUS should run the cleanup wizard at least once a month to always keep the updates file and SQL DB fit.So each cleanup wizard will not saturate too many system resources when doing other tasks to the DB.

    Regards,

    Clarence

    TechNet Subscriber Support

    If you are TechNet Subscription user and have any feedback on our support quality, please send your feedback here.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.


  • Tuesday, December 11, 2012 8:24 AM
     
     

    Thanks for your reply.

    How can I move WSUS database from Internal Database to a SqlServer?

    I've checked by google but it seems not suitable for WinSvr 2012.

    I'm not able to access Internal database by SqlServer 2005 Management Studio, nor SqlServer 2012 Management Studio either.

    TITLE: Connect to Server

    ------------------------------

    Cannot connect to \\.\pipe\mssql$microsoft##ssee\sql\query.

    ------------------------------
    ADDITIONAL INFORMATION:

    A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 2)

    For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&EvtSrc=MSSQLServer&EvtID=2&LinkId=20476

    ------------------------------
    BUTTONS:

    OK
    ------------------------------


    Eternal Snow

  • Tuesday, December 11, 2012 10:14 AM
     
     

    I've tried to move database from WID (\\.\pipe\MICROSOFT##WID\tsql\query) to local SqlServer Express 2012 and change the registry value SqlServerName to MSSQL$SQLEXPRESS.

    but this kill WSUS.

    WSUS service started, but management node cannot work.

    I've use server backup to rollback. wait for more detailed operation guide or document.


    Eternal Snow

  • Wednesday, December 12, 2012 2:59 AM
    Moderator
     
     

    Hi,

    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.
    Thank you for your understanding and support.

    Regards,

    Clarence

    TechNet Subscriber Support

    If you are TechNet Subscription user and have any feedback on our support quality, please send your feedbackhere.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

  • Wednesday, December 12, 2012 3:07 AM
     
     

    That's OK.

    the default named pipe of sqlserver contains \sql\query just like WSUS 3.0 and previous. but the new version of WSUS/WID uses a named pipe contains \tsql\query.

    Can I simply change the pipe name of SqlServer from sql to tsql?

    I'm backing up server to test.


    Eternal Snow

  • Wednesday, December 12, 2012 3:37 AM
    Moderator
     
     

    the default named pipe of sqlserver contains \sql\query just like WSUS 3.0 and previous. but the new version of WSUS/WID uses a named pipe contains \tsql\query.

    Can I simply change the pipe name of SqlServer from sql to tsql?

    Well, actually, it contains a couple of differences:

    WID on Win2008: \\.\pipe\MICROSOFT##SSEE\sql\query

    WID on Win2012: \\.\pipe\MICROSOFT##WID\tsql\query

    But yes, using the correct connection string is all that is necessary.


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Distribution (2005-2012)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.


  • Wednesday, December 12, 2012 3:45 AM
     
     
    Yup, but in registry, there is only one key of SqlServerName which is set to MICROSOFT##WID by default. There is no related setting about the whole connection string.

    Eternal Snow

  • Wednesday, December 12, 2012 6:13 AM
     
     

    OK. I changed my database from WID to SqlServer. I wrote a migration steps document about this: http://wp.secretnest.info/?p=804

    I'll go on check if this bug still exist.


    Eternal Snow

  • Wednesday, December 12, 2012 9:44 AM
     
     

    Hi,

    Have you tried to delete the update files in safe mode and clean boot?

    How to start in Safe mode
    1) Print these instructions before you go to step 2). They will not be available after you shut down the computer in step 2.
    2) Restart your computer.
    3) Use the F8 key. On a computer that is configured to start to multiple operating systems, you can press F8 when you see the Startup menu.
    4) Use the arrow keys to select a Safe mode option, and then press ENTER.
    Note: NUM LOCK functionality must be turned off for the arrow keys on the numeric keypad to work.
    5) If you have a dual-boot or multiple-boot system, use the arrow/2keys to select the installation that you want to access, and then press ENTER. 

    How to perform a clean boot
    1) Log on to the computer by using an account that has administrator rights.
    2) Run “msconfig.exe” in the Start Search box, and then press ENTER to start the System Configuration Utility.
    3) On the General tab, click Selective Startup, and then click to clear the Load startup items check box. (The Use Original Boot.ini check box is unavailable.)
    4) On the Services tab, click to select the Hide all Microsoft services check box, and then click Disable all.
    5) Click OK, and then click Restart.


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

  • Thursday, December 13, 2012 1:40 AM
     
     

    After changing database to sqlserver, the problem is still existed.

    When deleting unused updates, a popup message box from visual studio (installed by SqlServer 2012) says w3wp has unexpected exception and has been closed.

    3 Errors in Application event view as above.

    1 new Warning in Application event also:

    The WSUS administration console was unable to connect to the WSUS Server via the remote API. 

    Verify that the Update Services service, IIS and SQL are running on the server. If the problem persists, try restarting IIS, SQL, and the Update Services Service.

    System.Net.Sockets.SocketException -- An existing connection was forcibly closed by the remote host

    Source
    System.Windows.Forms

    Stack Trace:
       at System.Windows.Forms.Control.MarshaledInvoke(Control caller, Delegate method, Object[] args, Boolean synchronous)
       at System.Windows.Forms.Control.Invoke(Delegate method, Object[] args)
       at Microsoft.UpdateServices.UI.SnapIn.Wizards.ServerCleanup.ServerCleanupWizard.OnCleanupComplete(Object sender, PerformCleanupCompletedEventArgs e)

    And an Information:

    Fault bucket , type 0
    Event Name: CLR20r3
    Response: Not available
    Cab Id: 0

    Problem signature:
    P1: w3wp.exe
    P2: 8.0.9200.16384
    P3: 50108835
    P4: Microsoft.UpdateServices.CoreCommon
    P5: 6.2.9200.16384
    P6: 501070f9
    P7: 113
    P8: 5b9
    P9: System.NullReferenceException
    P10: 

    Attached files:

    These files may be available here:


    Analysis symbol: 
    Rechecking for solution: 0
    Report Id: 6b5541fb-44c5-11e2-93fb-005056b26389
    Report Status: 0
    Hashed bucket: 


    Eternal Snow

  • Thursday, December 13, 2012 9:36 AM
     
     

    Hi,

    Have you tried to delete the update files in safe mode and clean boot to see if the issue is related to third party drivers and services?


    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

  • Thursday, December 13, 2012 1:36 PM
     
     

    I'll try it next time.

    BTW, there is only WSUS installed on this computer.


    Eternal Snow

  • Saturday, December 15, 2012 12:04 AM
    Moderator
     
     
    Yup, but in registry, there is only one key of SqlServerName which is set to MICROSOFT##WID by default. There is no related setting about the whole connection string.

    True. SQLServerName and ConnectionString are two completely different entities. Your statement was about the ConnectionString, not the instance name. The ConnectionString is dependent upon the connectivity methodology being used. Since the WID only supports Named Pipes for connectivity, that string must be used.

    For SQL2008R2 and earlier (including WID on Win2008R2 and earlier), the Named Pipes ConnectionString is \\.\pipe\InstanceName\sql\query

    For SQL2012 (including WID on Win2012), the Named Pipes ConnectionString is \\.\pipe\InstanceName\tsql\query


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Distribution (2005-2012)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

  • Saturday, December 15, 2012 12:05 AM
    Moderator
     
     

    I'll go on check if this bug still exist.

    What "bug" are you talking about?

    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Distribution (2005-2012)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

  • Saturday, December 15, 2012 4:33 AM
     
     

    I'll go on check if this bug still exist.

    What "bug" are you talking about?

    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Distribution (2005-2012)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

    Listed above. Unhandled exception occurred when deleting unused updates.

    Eternal Snow

  • Tuesday, December 18, 2012 11:14 PM
    Moderator
     
     

    What "bug" are you talking about?

    Listed above. Unhandled exception occurred when deleting unused updates.

    Hmmm... given the conditions under which this was encountered, I'd be skeptical of calling it a 'bug'...

    If WSUS installed on a heavy IO loaded harddrive (LUN) of a VM, when deleting unneeded update files by Server Cleanup Wizard, MMC node may halt.

    Can you reproduce the behavior on a VM that doesn't have a "heavy IO loaded harddrive (LUN)"?

    Is there some reason you would expect the most database intensive process of WSUS (the Server Cleanup Wizard) wouldn't choke trying to do multi-join table scans when reading the database from "a heavy IO loaded LUN"?

    Incidentally... is that a single-spindle LUN, or is the VHD hosting the database actually sitting on a multi-spindle array?


    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Distribution (2005-2012)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.


  • Wednesday, December 19, 2012 1:09 AM
     
     

    WSUS cannot start in Safe mode:

    Service Dependence Tree: WSUS - BITS - COM+ (cannot start in safe mode)

    ---

    I've migrated this server to another LUN which is not used by other VM. The problem still exists.


    Eternal Snow


    • Edited by EternalSnow Wednesday, December 19, 2012 1:09 AM
    •  
  • Wednesday, December 19, 2012 1:11 AM
     
     
    This server is running in VM. Sqlserver Express is installed on the same server for this server use only. Database files and others are not shared with other servers.

    Eternal Snow

  • Wednesday, December 19, 2012 1:53 PM
    Moderator
     
     
    Sqlserver Express is installed on the same server for this server use only.
    Are you using SQL Server Express Edition for WSUS? If so, that's probably a contributing factor also. Migrate the WSUS database back to the Windows Internal Database, or a full edition of SQL Server, and then we'll have a functional implementation that we can actually perform diagnostics on.

    Lawrence Garvin, M.S., MCITP:EA, MCDBA, MCSA
    SolarWinds Head Geek
    Microsoft MVP - Software Distribution (2005-2012)
    My MVP Profile: http://mvp.support.microsoft.com/profile/Lawrence.Garvin
    The views expressed on this post are mine and do not necessarily reflect the views of SolarWinds.

  • Wednesday, December 19, 2012 1:57 PM
     
     

    At the beginning, some guy said I have to migrate database OUT OF WID. Now, I have to migrate it back?

    SqlServer license is not purchased for this server. I have no way to use a full edition legally.

    Please check above. when this database running in WID, the problem is much likely.


    Eternal Snow

  • Wednesday, April 24, 2013 1:39 PM
     
     
    With a high speed harddrive, no more halt but only an unhandled exception occurred every time while running server cleanup.

    Eternal Snow