none
SCOM reports "A significant portion of the database buffer cache has been written out to the system paging file. This may result in severe performance degradation"

    Question

  • This was discussed here, with no resolution

    http://social.technet.microsoft.com/Forums/en-US/exchange2010/thread/bb073c59-b88f-471b-a209-d7b5d9e5aa28?prof=required

    I have the same issue.  This is a single-purpose physical mailbox server with 320 users and 72GB of RAM.  That should be plenty.  I've checked and there are no manual settings for the database cache.  There are no other problems with the server, nothing reported in the logs, except for the aforementioned error (see below).

    The server is sluggish.  A reboot will clear up the problem temporarily.  The only processes using any significant amount of memory are store.exe (using 53GB), regsvc (using 5) and W3 and Monitoringhost.exe using 1 GB each.  Does anyone have any ideas on this?

    Warning ESE Event ID 906. 

    Information Store (1497076) A significant portion of the database buffer cache has been written out to the system paging file.  This may result in severe performance degradation. See help link for complete details of possible causes. Resident cache has fallen by 213107 buffers (or 11%) in the last 207168 seconds. Current Total Percent Resident: 79% (1574197 of 1969409 buffers)

    Thursday, March 01, 2012 2:01 AM

All replies

  • Hi

       store.exe no longer has a database cache limit, it will consume all RAM available minus 2GB approximately. It
    monitors the system itself and gives memory back to the system as it's needed.
    It's dynamic. IE: It's normal in Exchange 2007/10 to see store.exe eating up most RAM, it'll give it back if it needs to.

       Can you find any error from event viewer?

    Terence Yu

    TechNet Community Support

    Friday, March 02, 2012 2:53 AM
  • Nothing aside from that Warning w/ Event ID 906 saying essentially the same thing as SCOM.  (Emphasis below mine)

    Warning ESE Event ID 906. 

    Information Store (1497076) A significant portion of the database buffer cache has been written out to the system paging file.  This may result in severe performance degradation. See help link for complete details of possible causes. Resident cache has fallen by 213107 buffers (or 11%) in the last 207168 seconds. Current Total Percent Resident: 79% (1574197 of 1969409 buffers)

    Friday, March 02, 2012 2:56 AM
  • Nothing aside from that Warning w/ Event ID 906 saying essentially the same thing as SCOM.  (Emphasis below mine)

    Warning ESE Event ID 906. 

    Information Store (1497076) A significant portion of the database buffer cache has been written out to the system paging file.  This may result in severe performance degradation. See help link for complete details of possible causes. Resident cache has fallen by 213107 buffers (or 11%) in the last 207168 seconds. Current Total Percent Resident: 79% (1574197 of 1969409 buffers)

    Were your virus definitions updated within a few minutes when this event was triggered, or was there a recent A/V application update? I've seen antivirus updates force a re-scan of processes after definition updates and it can force store.exe to dump its entire working set. It can sometimes take a full 24 hours to build up the store.exe cache depending on the server and then the definition updates again and you start all over. Essentially you never get to full performance.

    If the definition update did happen 5-10 minutes before this 906 gets triggered, then make sure you have all of the required the file and processes level exclusions in place. This does not mean just exclude the .exe file itself from scanning, but exclude it from scanning while it is running in memory. If it appears all of the exclusions are in place and this is still happening when the definitions are updated then please open a ticket with your A/V vendor to determine why this is happening as it should not be.

    http://technet.microsoft.com/en-us/library/bb332342.aspx


    Microsoft Premier Field Engineer, Exchange
    MCSA 2000/2003
    MCTS: Win Server 2008 AD, Configuration MCTS: Win Server 2008 Network Infrastructure, Configuration
    MCITP: Enterprise Messaging Administrator 2010
    Former Microsoft MVP, Exchange Server

    NOTICE: My posts are provided “AS IS” without warranty of any kind, either expressed or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.


    Friday, March 02, 2012 3:40 AM
  • Thanks for responding Brian.

    My last a/v update prior to this was 3.5 hours earlier.  We're running Microsoft Forefront Endpoint Protection v. 2.1.1116.0 with the Microsoft recommended exclusions in both files, locations, file types and Excluded processes for that product.  I don't think this is the culprit.

    Friday, March 02, 2012 6:57 PM
  • Brian,

    We had this event log entry as well which SCOM picked up on, and 10 seconds before it the Forefront Protection 2010 for Exchange updated all of its engines.

    We are running Exchange 2010 SP2 RU3 with no file system antivirus (the boxes are restricted and have UAC turned on as mitigations). We are running the servers primarily as Hub Transport servers with 16GB of RAM, but they do have the mailbox role installed for the sole purpose of serving as our public folder servers.

    So we theroized the STORE process was just grabbing a ton of RAM, and occasionally it was told to dump the memory so the other processes could grab some - thus generating the alert. Up until last night we thought nothing of it, but ~25 seconds after the cache flush to paging file, we got the following alert:

    Log Name:      Application
    Source:        MSExchangeTransport
    Date:          8/2/2012 2:08:14 AM
    Event ID:      17012
    Task Category: Storage
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      HTS1.company.com
    Description:
    Transport Mail Database: The database could not allocate memory. Please close some applications to make sure you have enough memory for Exchange Server. The exception is Microsoft.Exchange.Isam.IsamOutOfMemoryException: Out of Memory (-1011)
       at Microsoft.Exchange.Isam.JetInterop.CallW(Int32 errFn)
       at Microsoft.Exchange.Isam.JetInterop.MJetOpenDatabase(MJET_SESID sesid, String file, String connect, MJET_GRBIT grbit, MJET_WRN& wrn)
       at Microsoft.Exchange.Isam.JetInterop.MJetOpenDatabase(MJET_SESID sesid, String file, MJET_GRBIT grbit)
       at Microsoft.Exchange.Isam.JetInterop.MJetOpenDatabase(MJET_SESID sesid, String file)
       at Microsoft.Exchange.Isam.Interop.MJetOpenDatabase(MJET_SESID sesid, String file)
       at Microsoft.Exchange.Transport.Storage.DataConnection..ctor(MJET_INSTANCE instance, DataSource source).

    Followed by:

    Log Name:      Application
    Source:        MSExchangeTransport
    Date:          8/2/2012 2:08:15 AM
    Event ID:      17106
    Task Category: Storage
    Level:         Information
    Keywords:      Classic
    User:          N/A
    Computer:      HTS1.company.com
    Description:
    Transport Mail Database: MSExchangeTransport has detected a critical storage error, updated the registry key (SOFTWARE\Microsoft\ExchangeServer\v14\Transport\QueueDatabase) and as a result, will attempt self-healing after process restart.


    Log Name:      Application
    Source:        MSExchangeTransport
    Date:          8/2/2012 2:13:50 AM
    Event ID:      17102
    Task Category: Storage
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      HTS1.company.com
    Description:
    Transport Mail Database: MSExchangeTransport has detected a critical storage error and has taken an automated recovery action.  This recovery action will not be repeated until the target folders are renamed or deleted. Directory path:E:\EXCHSRVR\TransportRoles\Data\Queue is moved to directory path:E:\EXCHSRVR\TransportRoles\Data\Queue\Queue.old.

    So it seems as if the Forefront Protection 2010 for Exchange inadvertently trigger the cache flush which didn't appear to happen quick or thuroughly enough for the transport service to do what it needed to do, so it freaked out and performed the subsequent actions.

    Do you have any ideas on how to prevent this 906 warning, which cascaded into a transport service outage?

    Thanks!

    Thursday, August 02, 2012 1:31 PM
  • Interesting. I've seen this on all of my mailbox servers at one point of another now, but as of about 2 RUs ago, it has largely cleared up. Except on my public folder servers.

    Currently running Exchange 2010 SP2 RU3

    Forefront for Exchange 2010 V 11.0.727.0

    Forefront Endpoint Protection  

    Forefront Endpoint Protection Version: 2.1.1116.0

    Antimalware Client Version: 3.0.8402.0

    Engine Version: 1.1.8601.0

    Antivirus definition: 1.131.1245.0

    Antispyware definition: 1.131.1245.0

    Friday, August 03, 2012 12:02 AM
  • It appears that the last time I got this was during a Forefront update.
    Friday, August 03, 2012 12:05 AM