locked
w3wp.exe event 1000 crash due to onetutil.dll on SharePoint 2007 RRS feed

  • Question

  • Hello,
    We have been experiencing w3wp.exe event 1000 due to onetutil.dll crash randomly on our SharePoint 2007 site (MOSS 2007 SP2). Our issue seems to be similar to the one described in: 
    http://social.technet.microsoft.com/Forums/en-SG/sharepointadminlegacy/thread/add16327-f0c3-4ba2-93de-0eb776d59e9e
    http://social.technet.microsoft.com/Forums/en-US/sharepointgenerallegacy/thread/4881f133-fd57-4399-9b19-de0cd37831d4

    I have googled this issue and try to find the root cause of this issue without success so far. 
    We have increased paging file size and restarted the server multiple times and none of these seem to help.
    Crash occurs on w3wp.exe from onetutil.dll on 12/6/12 7:26:49 AM again.
    The event viewer shows Event 1000, Application Error
    Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
    Faulting module name: onetutil.dll, version: 12.0.6421.1000, time stamp: 0x499414d5
    Exception code: 0xc0000005
    Fault offset: 0x00000000000abd8b
    Faulting process id: 0x1b4c
    Faulting application start time: 0x01cdd376ef8b85d6
    Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
    Faulting module path: C:\PROGRA~1\COMMON~1\MICROS~1\WEBSER~1\12\BIN\onetutil.dll
    Report Id: 34872082-3fa0-11e2-b9c8-0050569e0030

    Other times when event 1000 has occured on our site are:
    6/7/2012 6:37:35 AM
    6/15/2012 6:23:14 AM
    6/20/2012 8:01:26 AM
    7/6/2012 7:04:34 AM
    7/10/2012 6:25:26 AM
    7/12/2012 5:44:02 AM
    7/25/2012 5:50:27 AM
    10/18/2012 5:38:21 AM
    10/25/2012 7:50:28 AM
    12/6/2012 7:26:49 AM

    This crash usually occurs in the morning ranging from 5:30 - 8 AM (when the first user logs in) and I could not find a pattern (very random). When this error occurs, users see the login page and can enter their credentials but the system is not responsive, i.e the login page keeps spinning, until I perform  iisreset. 
    This site is very vanilla, no customization. Only logo is inserted in the login page. I could not find any indication of the error in 12 hive logs around this time. I try to look for any apparent erros in the IIS log around this time and also find nothing suspicious.

    In the event viewer, there are always a lot of database connectivity errors (27745, 5554, 7888, 2424, 10036 the night before around 9, 10 PM or early morning 1 or 2 AM  as usual) that happen daily. But the event 1000 crash on w3wp.exe does not always occur. Physical Memory utilization was at 57% in the Task Manager when the crash was found. There are not many users accessing this application daily (between 10 - 15 users / day).

    Our farm consists of 1 application server and 1 database server. The application server hosts Central Admin, My Site (not enabled or used in the application), SSP, SharePoint 80 and FBA web applications. We are using search though.
    Application server is on Windows Server 2008 R2 Standard SP1 and RAM is 4 GB. C drive has 45 GB with 14.4 GB free, C is where MOSS 2007 SP2 is installed. D drive has 80 GB with 48.3 free and D is where the index and log files are located. Total paging file for all drives is 16384 MB. C initial and max paging file size is 4096 MB. D has no paging file set. S (swap drive) initial and max paging file size is 12288 MB. Application server is a virtual box and is using VmWare. 
    Databases are on a physical box running Windows 2003 SP2 and SQL Server 2008 SP3. The site collection size is 4096 MB.
    The application pools has recycling set at different times (1 AM, 1.44 AM, 2:45 AM, 2:51 AM), none of them conflict.

    I would really appreciate any help to diagnose this issue. 
    Thank you.
    Regards,
    Judith
    Friday, December 7, 2012 6:11 PM

All replies

  • Hi,

    For this issue, I’m trying to involve someone familiar with this topic to further look at it.

    Thanks,
    Qiao
    Forum Support
    Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact
    tnmff@microsoft.com.


    Qiao Wei
    TechNet Community Support

    Monday, December 10, 2012 2:41 AM
    Moderator
  • Hi Judith,

    The exception code  0xc0000005 means Access Violation.  You can use Debugdiag 1.2 and setup a rule to monitor for it.  http://www.microsoft.com/en-us/download/details.aspx?id=26798

    This tool also has an analyzer that can give a quick overview of the dump it creates.  If you need an in-depth review of the memory dump please open a support ticket.  We would need the memory dump and the ULS logs from that server that created dump file for a proper review.

    You can also check the ULS logs and follow it until it crashed.  Filter by PID value in the process column; try to find a repeating pattern to see what the last action was.  Then you can focus on this area to trouble shoot further.

    Also make sure your .NET 2.0 versions is fully patched.  There is an issue with access violations if your build is lower than the build listed here: http://support.microsoft.com/kb/974168


    Regards, Savoeurn Va Microsoft Online Community Support

    Monday, December 10, 2012 7:29 PM
  • Hi Qiao and Savoeurn,

    Thank you for your responses. I will read the article on Debugdiag to monitor the crash and check the .NET 2.0 version the server has, per your suggestion.

    My current subscription is Visual Studio Premium with MSDN. Am I elligible to create a support ticket?

    I will report back with my findings.

    Thank you.

    Regards,

    Judith

    Monday, December 10, 2012 9:08 PM
  • Hi Savoeurn,

    I am getting ready to run Debug Diag tool in Production and would like to confirm the following.

    In DebugDiag, which rule should I create to monitor: crash, performance or native (non .NET) memory and handle leak. I assume I would have to create a crash rule but would like to confirm.

    The ULS log in CA > Operations > Diagnostic Logging > Event Throttling is set to Error for least critical event to report to the Event log and Monitorable for least critical event to report to the trace log. Would this be sufficient?

    To check whether the .NET 2.0 is fully patched, I check the Mscorlib.dll, Mscordacwks.dll, Mscorwks.dll and Sos.dll in the server C:\Windows\Microsoft.NET\Framework\v2.0.50727. They all have 2.0.50727.5466 version (which is higher than 2.0.50727.5000) with date modified of 12/1/2012 7:11 AM.  But I see that there is another set of the files above in C:\Windows\Microsoft.NET\Framework\v4.0.30319 with version of 4.0.30319.296 with date modified as 9/6/2012 9:50 AM. Is this okay or could this create a conflict?

    Thank you.

    Regards,

    Judith

    Tuesday, December 11, 2012 11:52 PM
  • Yes it would be a crash rule.

    Logging in SharePoint must be configured as "Verbose"

    SharePoint actually just uses .NET 2.0 so 4.0 should not cause a conflict.


    Regards, Savoeurn Va Microsoft Online Community Support

    Wednesday, December 12, 2012 12:20 AM
  • Hi Savoeurn,

    I was able to reproduce the event 1000 on w3wp.exe on the Test server today. I was just navigating to a list after logging in and it kept spinning. I check the server and the event 1000 on w3wp.exe happens on 12/13/2012 4:21:24 PM. I check DebugDiag tool and it didn't create a dump file. It just logs the activity of the pid 6380 w3wp.exe that errors out in a text file (w3wp__USAccess - 80__PID__2268__Date__12_12_2012__Time_01_41_19AM__439__Log.txt). Do you know why it didn’t automatically create a dump file?

    Therefore, I manually create a full user dump on pid 6380. I tried to analyze the dump on the server but each time I click Start Analysis, the report runs with error although I already enable "Allow active content to run in files on My Computer*" under Tools > Internet Options > Advanced > Security:

    CrashHangAnalysis.asp Failed 0x800a01ad Microsoft VBScript runtime error ActiveX component can't create object Line 717, Column 12

    So I analyze the dump on my other computer and this time, it creates the mht and the summary is as follows:

    Type

    Description

    Recommendation

    Detected possible blocking or leaked critical section at 0x081231a8 owned by thread 21 in w3wp.exe__USAccess - 80__PID__6380__Date__12_13_2012__Time_04_31_24PM__922__Manual Dump.dmp
    Impact of this lock
    13.16% of threads blocked
    (Threads
    27 28 29 30 34)
    The following functions are trying to enter this critical section
    ONETUTIL!COWSThreadWithHeap::Uninitialize+9b5
    The following module(s) are involved with this critical section
    C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN\ONETUTIL.DLL from Microsoft Corporation

    One or more of the blocking functions is a heap function. In most cases this implies leaked critical section caused by heap corruption. Please follow the steps outlined in the following Knowledge Base article:
    300966 Howto debug heap corruption issues in Internet Information Services (IIS)

    3 client connection(s) in w3wp.exe__USAccess - 80__PID__6380__Date__12_13_2012__Time_04_31_24PM__922__Manual Dump.dmp have been executing a request for more than 90 seconds.

    Please see the Client Connections section of this report for more detailed information about the connection(s).


    The process w3wp.exe in w3wp.exe__USAccess - 80__PID__6380__Date__12_13_2012__Time_04_31_24PM__922__Manual Dump.dmp is a 64 bit application. Some of the native (non-.NET) analysis, such as COM call information, may be missing. Further manual inspection of the dump may be required to extract such information.

     

    The only thing odd that I find in the ULS log around the time of the event 1000 error (12/13/2012 4:21:24 PM) is the following:

    12/13/2012 16:21:31.33                w3wp.exe (0x18EC)                       0x10A4  Windows SharePoint Services               General                               8nca      Verbose               Application error when access /teammso/_themes/Lacquer/topshape_lacquer.jpg, Error=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.   at System.Web.UnsafeNativeMethods.EcbExecuteUrlUnicode(IntPtr pECB, String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, IntPtr pEntity, ISAPIAsyncCompletionCallback asyncCompletionCallback)     at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.BeginExecuteUrl(String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, Byte[] entity, AsyncCallback cb, Object state)     at System.Web.HttpResponse.BeginExecuteUrlForEntireResponse(String pathOverride, ...      

    12/13/2012 16:21:31.33*              w3wp.exe (0x18EC)                       0x10A4  Windows SharePoint Services               General                               8nca      Verbose               ...NameValueCollection requestHeaders, AsyncCallback cb, Object state)     at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state)     at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

    But I could not make sense of it to understand what may be the root cause. I have the ULS log (verbose) around the time the error occurred and the DebugDiag Logs folder that contain the manual dump I generated on the pid and the text log file on the pid. Do you mind helping me analyze what the root cause may be?

    Thank you.
    Regards,

    Judith

    Thursday, December 13, 2012 11:34 PM
  • Hi,

     

    It looks like the ULS contained more useful information than the dump.  This piece is key:

     

    • Application      error when access /teammso/_themes/Lacquer/topshape_lacquer.jpg, Error=Attempted to read or write protected memory. This is often an      indication that other memory is corrupt. at      System.Web.UnsafeNativeMethods.EcbExecuteUrlUnicode(IntPtr pECB, String      url, String method, String childHeaders, Boolean sendHeaders, Boolean      addUserIndo, IntPtr token, String name, String authType, IntPtr pEntity,      ISAPIAsyncCompletionCallback asyncCompletionCallback) at      System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.BeginExecuteUrl(String      url, String method, String childHeaders, Boolean sendHeaders, Boolean      addUserIndo, IntPtr token, String name, String authType, Byte[] entity,      AsyncCallback cb, Object state) at      System.Web.HttpResponse.BeginExecuteUrlForEntireResponse(String      pathOverride

     

    What is the full  out put of that?  Basically some thing made a call and that was last piece of the stack.  But we need to know the full stack to see what actually made the call.

     

    Also what OS and IIS is running your SharePoint server?

     

     


    Regards, Savoeurn Va Microsoft Online Community Support

    Monday, December 17, 2012 10:43 PM
  • Hi Savoeurn,
    I use the SharePoint LogViewer to parse through the ULS log, filtering by the process that generates the error above.
    The full output of the highlighted line above is below. The area is Windows SharePoint Services, category is General, EventID is 8nca, level is Verbose:
    Application error when access /teammso/_themes/Lacquer/topshape_lacquer.jpg, Error=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.   at System.Web.UnsafeNativeMethods.EcbExecuteUrlUnicode(IntPtr pECB, String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, IntPtr pEntity, ISAPIAsyncCompletionCallback asyncCompletionCallback)     at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.BeginExecuteUrl(String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, Byte[] entity, AsyncCallback cb, Object state)     at System.Web.HttpResponse.BeginExecuteUrlForEntireResponse(String pathOverride, NameValueCollection requestHeaders, AsyncCallback cb, Object state)     at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state)     at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

    Regarding which OS and IIS are running on the SharePoint Test server:
    Our farm consists of 1 application server and 1 database server. The application server hosts Central Admin, My Site (not enabled or used in the application), SSP, SharePoint 80 and FBA web applications. We are using search though.
    Application server is on Windows Server 2008 R2 Standard SP1 (Windows NT 6.1 SP1) 64 bit and RAM is 4 GB. C drive has 45 GB with 20.5 GB free, C is where MOSS 2007 SP2 is installed. D drive has 100 GB with 8.11 GB free and D is where the index and log files are located. Total paging file for all drives is 12688 MB. C initial and max paging file size is 400 MB. D has no paging file set. S (swap drive) initial and max paging file size is 12288 MB. Application server is a virtual box and is using VmWare.
    IIS version in application server is 7.5.7600.16385.
    Databases are on a physical box running Windows 2003 SP2 and SQL Server 2008 SP3.

    Is there anywhere I can upload the files for you to look at as well? I tried to summarize what I see in the ~25 MB ULS log file below by using SharePoint LogViewer filtered by the w3wp process causing the error. It logs errors such as "There is no Web named "/teammso/Deployment/Forms/AllItems.aspx"." But these pages/views actually exist. Here is the summary of events leading up to the application error in the ULS log file:
    DatabaseConnectionString acquired
    Reverting to process identity
    SqlCommand: 'dbo.proc_MIP_GetObjectUpdates'
    SearchService.GetService: Getting the search service from the farm.
    Topology cache updated
    Portal application domain refreshed
    Repeated above
    Entering... then Exiting MRU trim routine.
    Repeated above
    Diagnostics settings: 32768
    Initializing managed SQM
    BlobCache initialized with values location:'C:\blobCache\233779004\', path:'\.(gif|jpg|png|css|js)$', InstanceID:'233779004', maxSize:'10737418240', enabled:'False', changeCheckInterval:'5', policyCheckInterval:'60'
    Waiting for, Acquired semaphore for ID CMS...
    Successfully incremented handle count of an existing item in Perfmon Memory
    Registered hosting environment for ID CMS...
    PublishingHttpModule.Init() calling AppDomainUnloadListener.Register()
    AppDomainUnloadListener.RegisterSelf() entered lock(this=34945278)...about to call HostingEnv
    Partition resolver initialized
    Initializing the configuration database connection..found connection string...
    SqlCommand: 'dbo.proc_getObjectsByClass' called twice
    SqlCommand: 'dbo.proc_getObjectsByBaseClass' called 4 times
    Successfully incremented handle count of an existing item in Perfmon Memory
    Looking up context site https://...
    Site lookup is replacing https://...with alternate access url https://...
    Looking up typical site
    SqlCommand: 'dbo.proc_getSiteMap'
    Found typical site ...it calls some SQL Select stmts
    Releasing SPRequest with allocation Id ...
    spHttpHandler:GetHash started...SPSecurity::GetHash started then both finished
    PublishingHttpModule.Init() calling AppDomainUnloadListener.Register()
    AppDomainUnloadListener.RegisterSelf() entered lock(this=34945278)
    Partition resolver initialized
    Repeated above Publishing...
    Releaseing SPRequest with allocation Id
    Looking up context site https://...
    Site lookup is replacing https://...with alternate access url https://...
    Looking up typical site
    Found typical site...
    Videntityinfo::isFreshToken reported failure.
    SqlCommand: 'dbo.proc_getObjectsByClass'
    PermissionMask check failed. asking for 0x00000005, have 0x00011001 (repeated 3 times)
    Usage: Rights and Roles: GetWebPartPage: Shared
    SqlCommand: 'dbo.proc_MIP_GetObjectUpdates'
    Failed to look up string with key "XomlUrl", keyfile core. (repeated 2 times)
    Localized resource for token 'XomlUrl' could not be found for file with path: "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\Template\Features\Fields\fieldswss.xml". (repeated 2 times)
    SqlCommand: 'dbo.proc_getObject'
    SqlCommand: 'dbo.proc_getObjectsByBaseClass'
    SearchService.GetService: Getting the search service from the farm.
    Topology cache updated
    Portal application domain refreshed
    SqlCommand: 'dbo.proc_getObjectsByClass'
    DatabaseConnectionString acquired.
    Starting shared object cache refresh timer with an interval of 0 seconds.Shared object cache refresh timer started.Cached shared object store for connection...
    SqlCommand: 'dbo.proc_MIP_GetObjectVersion'
    Initialized new shared object cache at version 74. Cache miss for shared object ... of type 'Microsoft.Office.Server.Administration.UserProfileApplication'.
    SqlCommand: 'dbo.proc_MIP_GetObject'  (repeated 2 times)
    SharedServiceAccessControlList: Constructing WSS Acl from persisted text =...
    SharedServiceAccessControlList: Constructing WSS ACE ...
    SqlCommand: 'dbo.proc_getObjectsByClass'...SqlCommand: 'dbo.proc_getDependentObjectsByBaseClass'
    SqlCommand: 'dbo.proc_getDependentObjectsByBaseClass'
    Constructing PortalSiteMapProvider, Initializing PortalSiteMapProvider...
    SqlCommand: 'dbo.proc_getSiteMapById'
    Releasing SPRequest with allocation Id ... (multiple times for different Ids)
    Successfully assigned an item in Perfmon Memory 
    SqlCommand: 'proc_GetCurrent'
    SqlCommand: 'proc_GetWebExtendedMetaData'
    PersonalActions::CreateChildControls() - Was called.
    SqlCommand: 'dbo.proc_getObjectsByClass'  (repeated 2 times)
    SqlCommand: 'dbo.proc_getObject'
    Failed to determine definition for Feature with ID '73049794-7edf-44bd-ab0f-6392893d02de'.  Skipping this feature for element querying consideration. (repeated 2 times)
    Menu::OnLoad() - Was called. ID="Menu"
    Constructing PortalSiteMapProvider
    Looking up context site https://...
    Site lookup is replacing https://...with alternate access url https://...
    Looking up typical site
    Found typical site...
    SqlCommand: 'proc_GetWebExtendedMetaData'
    Releasing SPRequest with allocation Id ...
    Cache miss for shared object ... of type  'Microsoft.Office.Server.Search.Administration.SearchSharedApplication'.
    SqlCommand: 'dbo.proc_MIP_GetObject'
    '... SSP', scope cache: creating
    SQL: dbo.proc_MSS_ScopeCacheGetChangedConsumers 0
    '... SSP', scope cache: removing unused consumer caches...creating...refreshing
    '... SSP', consumer scope cache '00000000-0000-0000-0000-000000000000': adding scope 'People' (ready)...'All Sites' (ready)... 'Global Query Exclusion' (not ready)...'Rank Demoted Sites' (not ready)
    Menu::OnLoad() - Was called. ID="SiteActionsMenu"
    SELECT stmts...
    SqlCommand: 'dbo.proc_MSS_GetManagedPropertiesForOM'
    PersonalActions::OnPreRender() - Was called.
    SqlCommand: 'dbo.proc_MSS_GetMappingsForMangedProperty'
    SqlCommand: 'dbo.profile_GetProfilePropertyInfo'
    SqlCommand: 'dbo.profile_GetProfilePropertyLoc'
    SqlCommand: 'dbo.proc_MSS_GetMappingsForMangedProperty' (repeated 126 times with different input values)
    SqlCommand: 'dbo.privacy_getFeaturePolicy' 
    In CachedArea.GetChildAreaIds, about to fetch list of child web IDs for: /.
    Retrieved a list of 5 child webs. Successfully cached 5 child web IDs.
    There is no Web named "/agency/default.aspx".
    Possible mismatch between the reported error with code = 0x81070504 and message: "There is no Web named "/agency/default.aspx"." and the returned error with code 0x80070002.
    Releasing SPRequest with allocation Id ... (repeated 3 times with different Ids)
    Unknown SPRequest error occurred. More information: 0x80070001
    There is no Web named "/teammso/Program Deliverables/Forms/Standard View.aspx".
    Possible mismatch between the reported error with code = 0x81070504 and message: "There is no Web named "/teammso/Program Deliverables/Forms/Standard View.aspx"." and the returned error with code 0x80070002.
    There is no Web named "/teammso/Communication Tools/Forms/AllItems.aspx".
    Possible mismatch between the reported error with code = 0x81070504 and message: "There is no Web named "/teammso/Communication Tools/Forms/AllItems.aspx"." and the returned error with code 0x80070002.
    There is no Web named "/teammso/Deployment/Forms/AllItems.aspx".
    Possible mismatch between the reported error with code = 0x81070504 and message: "There is no Web named "/teammso/Operations/Forms/AllItems.aspx"." and the returned error with code 0x80070002.
    There is no Web named "/teammso/Hot Issues/Forms/AllItems.aspx".
    Possible mismatch between the reported error with code = 0x81070504 and message: "There is no Web named "/teammso/Hot Issues/Forms/AllItems.aspx"." and the returned error with code 0x80070002.
    Menu::Render() - Was called. ID="Menu"
    PostCacheSubstitutionText::Render() - Was called.
    SqlCommand: 'dbo.proc_getObject'
    Failed to determine definition for Feature with ID '73049794-7edf-44bd-ab0f-6392893d02de'.  Skipping this feature for element querying consideration.
    Releasing SPRequest with allocation Id ... (repeated 8 times for different ids)
    PublishingHttpModule.Init() calling AppDomainUnloadListener.Register()
    AppDomainUnloadListener.RegisterSelf() entered lock(this=34945278)
    Partition resolver initialized

    0x0841ec00 metabase: inserting: . (repeated hundreds of times at 12/13/2012 16:21:21.36)

    Looking up context ... for various gifs
    Site lookup is replacing ...gifs with the alternate access url https:...
    Found typical site xxx in web application SPWebApplication Name=SharePoint - 80 Parent=SPWebService.
    Application error when access /teammso/_themes/Lacquer/topshape_lacquer.jpg, Error=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.   at System.Web.UnsafeNativeMethods.EcbExecuteUrlUnicode(IntPtr pECB, String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, IntPtr pEntity, ISAPIAsyncCompletionCallback asyncCompletionCallback)     at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.BeginExecuteUrl(String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, Byte[] entity, AsyncCallback cb, Object state)     at System.Web.HttpResponse.BeginExecuteUrlForEntireResponse(String pathOverride, NameValueCollection requestHeaders, AsyncCallback cb, Object state)     at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state)     at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) occurs at 12/13/2012 16:21:31.33

    Thank you for your help.
    Regards,
    Judith

    Thursday, December 20, 2012 11:18 PM
  • Hi Savoeurn,

    I am sorry to bug you again. I was just wondering if you have any feedback regarding the full output of the crash stack in the ULS logging I provided on 12/20/12.

    Another observation I would like to report, which may not be related to the w3wp.exe crash, is that the owstimer.exe increases to a huge amount of memory from the time it started, that reduces the performance / responsiveness of the application. For example, when starting the windows sharepoint timer service, it starts consuming ~65 MB of memory. It gradually increases to ~ 666 MB of memory after about a month or so, causing the memory consumption on the server to be up in the 90% range.  When I restart the timer service, the memory consumption on the server reduces down to about 63% and it slowly comes back up again. So I have to watch for the owstimer.exe memory and restart this service if it becomes too high. Is this normal? When the w3wp.exe crash occurs on the Production server, the memory consumption in task manager was only at 57% though, so it may not be related. A similar trend is observed on w3wp.exe that's running the application too, meaning it starts with ~ 150 MB then gradually increases to 420 MB of memory consumption.

    Thank you.

    Regards,

    Judith


    Judith Teguh

    Friday, January 4, 2013 11:28 PM
  • Hi Judith,

    I’ve looked at the post on the 20<sup>th</sup> and there should be some more regarding this piece:

    General, EventID is 8nca, level is Verbose:
    Application error when access /teammso/_themes/Lacquer/topshape_lacquer.jpg, Error=Attempted to read or write protected memory. This is often an indication that other memory is corrupt. at System.Web.UnsafeNativeMethods.EcbExecuteUrlUnicode(IntPtr pECB, String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, IntPtr pEntity, ISAPIAsyncCompletionCallback asyncCompletionCallback) at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.BeginExecuteUrl(String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, Byte[] entity, AsyncCallback cb, Object state) at System.Web.HttpResponse.BeginExecuteUrlForEntireResponse(String pathOverride, NameValueCollection requestHeaders, AsyncCallback cb, Object state) at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

    Something is calling topshape_lacquer.jpg but it is erroring out for some reason.  Based off the stack, I can’t see if it was a user call or a piece of code that called it.   As for the rest of the output, it looks like a regular load of a page. 

    Now for the Ostimer issue you’re describing.  Does the Oswtimer.exe get smaller at all?  It should go grow and shrink depending upon how much work it has to do.  I know we fixed a handle leak on Owstimer in the SP2 release.  Do a double check on the build number for that .exe.  It is located in C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\BIN

    Right click the .exe and select properties then select Details tab.   The build number should be at least 12.0.6421.1000.  If the number is smaller than that, then I would double check and ensure all the patches are installed.  Since this is MOSS, you would need to install the WSS and MOSS patches at the same time for a proper patch. http://technet.microsoft.com/en-us/sharepoint/bb735839.aspx

    If you want us to look at this deeper, you would need to open a support ticket to send this information for review.  We would need the ULS in verbose + Memory dump from the same time to understand what is happening on your farm.


    Regards, Savoeurn Va Microsoft Online Community Support

    Sunday, January 6, 2013 6:31 PM
  • Hi Savoeurn,

    Thank  you for your feedback. I have submitted a support ticket along with the files. Please let me know if you need to know the SR number. 

    I check the owstimer.exe version, per your suggestion and it is 12.0.6415.1000 modified on 11/21/08 (lower than 12.0.6421.1000). So does this mean that I have to install both the December CU (released 12/11/12) for WSS 3 (KB 2760571) and Office SharePoint Server 2007 (KB 2760570) although we are running MOSS 2007 SP2? Or should I just install the KB 2760570 in the link you provided http://technet.microsoft.com/en-us/sharepoint/bb735839.aspx?

    Thank you.

    Regards,

    Judith


    Judith Teguh

    Monday, January 7, 2013 11:33 PM
  • Hi Savoeurn,
    I submitted SR with Microsoft. The recommendation was to clear configuration cache, setup portal reader and portal super user accounts, clear all stuck timer jobs, run psconfig installfeatures, stop and restart IIS, SharePoint services and the server. The problem still reoccurs after some random period (could be days, 1 week, 1 month) after performing all these.

    I also now get what you were asking regarding the missing piece of the ULS log. I look at the ULS log again and find the following. Do you have any insight as to what maybe causing the error?

    Thank you.
    Regards,
    Judith

    12/13/2012 16:21:31.33  w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        8nca Verbose  Application error when access /xxxx/_themes/Lacquer/topshape_lacquer.jpg, Error=Attempted to read or write protected memory. This is often an indication that other memory is corrupt.   at System.Web.UnsafeNativeMethods.EcbExecuteUrlUnicode(IntPtr pECB, String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, IntPtr pEntity, ISAPIAsyncCompletionCallback asyncCompletionCallback)     at System.Web.Hosting.ISAPIWorkerRequestInProcForIIS6.BeginExecuteUrl(String url, String method, String childHeaders, Boolean sendHeaders, Boolean addUserIndo, IntPtr token, String name, String authType, Byte[] entity, AsyncCallback cb, Object state)     at System.Web.HttpResponse.BeginExecuteUrlForEntireResponse(String pathOverride, ... 
    12/13/2012 16:21:31.33* w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        8nca Verbose  ...NameValueCollection requestHeaders, AsyncCallback cb, Object state)     at System.Web.DefaultHttpHandler.BeginProcessRequest(HttpContext context, AsyncCallback callback, Object state)     at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()     at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) 
    12/13/2012 16:21:31.35  mssearch.exe (0x0C48)                    0x1730 Search Server Common           GatherSvc                      0 Medium   CGatherApplication::CGatherApplication::GetRole - What is the role of this server? - File:d:\office\source\search\search\gather\gthrsvc\gthrapp.cxx Line:745 
    12/13/2012 16:21:31.35  mssearch.exe (0x0C48)                    0x1730 Search Server Common           GatherSvc                      0 Medium   CGatherApplication::CGatherApplication::GetRole - The role of this server is 3 - File:d:\office\source\search\search\gather\gthrsvc\gthrapp.cxx Line:767 
    12/13/2012 16:21:31.42  OWSTIMER.EXE (0x08FC)                    0x1004 Windows SharePoint Services    Timer                          8e46 Verbose  End invoke timer job Indexing Schedule Manager on xxxx, id {2AC4C637-5F4E-4F7F-8A72-67E94EC5529E}, DB n/a 
    12/13/2012 16:21:31.58  w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        6t8b Verbose  Looking up context  site https://xxxx/_themes/Lacquer/topshape_lacquer.jpg in the farm Test_xxxx_SharePoint_Config 
    12/13/2012 16:21:31.58  w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        6t8d Verbose  Looking up the additional information about the typical site https://xxxx/_themes/Lacquer/topshape_lacquer.jpg
    12/13/2012 16:21:31.58  w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        6t8f Verbose  Site lookup is replacing https://xxxx/_themes/Lacquer/topshape_lacquer.jpg with the alternate access url https://xxxx
    12/13/2012 16:21:31.58  w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        6t8g Verbose  Looking up typical site https://xxxx/_themes/Lacquer/topshape_lacquer.jpg in web application SPWebApplication Name=SharePoint - 80 Parent=SPWebService. 
    12/13/2012 16:21:31.58  w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        6t8h Verbose  Found typical site / (a0765a74-38c5-40b0-a525-9121d118d5b7) in web application SPWebApplication Name=SharePoint - 80 Parent=SPWebService. 
    12/13/2012 16:21:31.61  w3wp.exe (0x18EC)                        0x10A4 Windows SharePoint Services    General                        0 Verbose  Releasing SPRequest with allocation Id {5334B5ED-ECBB-43EF-B96E-AB76772C4B1B} 


    Judith Teguh

    Friday, June 28, 2013 6:33 PM