none
Exchange services crash due to low memory during online defragmentation RRS feed

  • Question

  • Hi everyone

    We are experiencing problems on our Exchange 2007 server (we will migrate to 2013 at the end of this year) and I would greatly appreciate if someone could help. Our problems seem to be related to the maintenance task which is scheduled to run on weekends. Every weekend, memory consumption on our server is so high that certain services (e.g. the information store) crash.

    For testing purposes, I changed the maintenance schedule for one particular storage group yesterday and let it run between 6 PM and 12 AM. After it started at 6 PM, memory consumption was still normal (8 GB of 30 GB). I noticed event ID 703, saying that it “resumed a pass” (probably the one that started on the weekend but never finished). This was followed by event ID 700. At 7 PM, memory consumption was already alarmingly high.

    The following events stood out to me:

    7:00:13 PM: Exchange VSS Writer (instance c52ac2b4-e291-4d06-adda-5b79eb355fcc:6) has processed the post-snapshot event successfully. (DPM synchronizes every hour).

    7:00:17 PM: COM+ has determined that your machine is running very low on available memory.  In order to ensure proper system behavior, the activation of the component has been refused.  If this problem continues, either install more memory or increase the size of your paging file.  Memory statistics are: dwMemoryLoad = 84

    ullTotalPhys = 0x4e1e97000

    ullAvailPhys = 0x0c7830000

    ullTotalPageFile = 0x53aa57000

    ullAvailPageFile = 0x07c675000

    ullTotalVirtual = 0x7fffffe0000

    ullAvailVirtual = 0x7fffc95a000

    Process Name: svchost.exe

    Comsvcs.dll file version:  not loaded

    7:00:28 PM: Volume Shadow Copy Service information: The COM Server with CLSID {79079d5c-ef20-4952-b9fc-cfd443b38641} and name SW_PROV cannot be started. Most likely the server is in the process of shutting down or the system is out of memory. [0x8007000e]

    Almost one hour later I noticed event IDs 701 and 1221:

    07:52:14 PM: MSExchangeIS (3852): Online defragmentation has completed a full pass on database…, freeing 766 pages. This pass started on 19/08/2014 and ran for a total of 303 seconds, requiring 1 invocations over 1 days. Since the database was created, it has been fully defragmented 1695 times over 2759 days.

    07:52:14 PM: The database … has 50 megabytes of free space after online defragmentation has terminated.

    To my surprise, at exactly 8 PM, another defragmentation started on the same database. Why?

    08:00:00 PM: MSExchangeIS (3852) SG01: Online defragmentation is beginning a full pass on database ….

    Shortly afterwards, the Exchange services crashed and I had to reboot the server.

    Any advice how to narrow down this problem would be greatly appreciated!

    Thank you

    Stefan

    Wednesday, August 20, 2014 8:46 AM

Answers

  • There should be 3 events 701 700 1221
    Event 700 indicates that online defragmentation of a specific database has begun full defragmentation. This is followed by ESE event 701, which indicates that online defragmentation has finished full defragmentation. Event 701 is followed by MSExchangeIS event 1221, which indicates how much space has been freed by the online defragmentation process. These messages indicate that online maintenance has successfully completed.

    Please check this as well
    http://technet.microsoft.com/en-us/library/bb218224(v=exchg.80).aspx


    Thanks, MAS Please mark as helpful if you find my comment helpful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Wednesday, August 20, 2014 10:16 AM
  • Did you try increasing your paging file size? If not please try that


    Thanks,
    MAS
    Please mark as helpful if you find my comment helpful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Wednesday, August 20, 2014 10:31 AM
  • If you schedule online maintenance to run for six hours it will run for six hours!

    If the maintenance task only runs for a few minutes it will run every 15 minutes until the end of the scheduled time.

    I'd shorten that maintenance interval -- and I'd schedule the backup to NOT run during that time period.


    --- Rich Matheisen MCSE&I, Exchange MVP

    Wednesday, August 20, 2014 4:26 PM
  • The memory use increases because you're reading every page in the database. That uses buffers, and those are kept in memory.

    Brien Posey? Well . . . consider the source.

    "Exchange extends the maintenance cycle by up to an hour to ensure that the defragmentation has time to complete."

    Suppose it's a really big database? An hour is no guaranty that it'll complete.

    The point is that the tasks will eventually complete. If not today, than tomorrow. For example, it doesn't matter a whole lot if unused indexes hang around for a while, it just matters that they'll be removed in a reasonable period of time. Removing expired mailboxes doesn't take all that long, nor does removing expired messages.

    Other statements in that article are incorrect, too. For example:

    "If you have to perform any type of repair on the database, you'll need enough free space on the volume for a full copy of the database, plus another 10 to 20 percent for overhead."

    In the context of running ISINTEG (which was the subject of that quote) you certainly don't need to have enough room for a full copy of the database.


    --- Rich Matheisen MCSE&I, Exchange MVP

    Friday, August 29, 2014 3:21 AM

All replies

  • What about the database status now? 
    Is it mounted or dismounted.


    Thanks, MAS Please mark as helpful if you find my comment helpful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Wednesday, August 20, 2014 9:06 AM
  • Yes, they are all mounted. The server works like a charm but as soon as the maintenance tasks start the problems begin.
    Wednesday, August 20, 2014 9:13 AM
  • There should be 3 events 701 700 1221
    Event 700 indicates that online defragmentation of a specific database has begun full defragmentation. This is followed by ESE event 701, which indicates that online defragmentation has finished full defragmentation. Event 701 is followed by MSExchangeIS event 1221, which indicates how much space has been freed by the online defragmentation process. These messages indicate that online maintenance has successfully completed.

    Please check this as well
    http://technet.microsoft.com/en-us/library/bb218224(v=exchg.80).aspx


    Thanks, MAS Please mark as helpful if you find my comment helpful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Wednesday, August 20, 2014 10:16 AM
  • Yep, all three events are logged, so the online defragmentation certainly completes (see events at 07:52:14 PM in my first post). Nevertheless it seems it started again at 8 PM.
    Wednesday, August 20, 2014 10:25 AM
  • Did you try increasing your paging file size? If not please try that


    Thanks,
    MAS
    Please mark as helpful if you find my comment helpful or as an answer if it does answer your question. That will encourage me - and others - to take time out to help you.

    Wednesday, August 20, 2014 10:31 AM
  • If you schedule online maintenance to run for six hours it will run for six hours!

    If the maintenance task only runs for a few minutes it will run every 15 minutes until the end of the scheduled time.

    I'd shorten that maintenance interval -- and I'd schedule the backup to NOT run during that time period.


    --- Rich Matheisen MCSE&I, Exchange MVP

    Wednesday, August 20, 2014 4:26 PM
  • Thanks a lot for your comment Rich, I was not aware of this. I assumed this is just the time period it could run in case it needs more time to finish all the tasks. I will adjust the maintenance tasks accordingly and will see if it makes a difference.

    Thursday, August 21, 2014 7:47 AM
  • I have changed the maintenance schedule and this seems to work much better. I am still surprised at how much memory is used during that process but since I only tackle one storage group per day the server does not crash anymore.

    What I am still wondering is how to verify that every one of the 11 maintenance tasks has completed? I have verified that event ID 700 is followed by 701 and 1221 but that's not conclusive according to this article since Exchange will always make sure that the defragmentation task completes, even though it might not be able to finish the other 10 tasks in time.

    Thursday, August 28, 2014 12:56 PM
  • The memory use increases because you're reading every page in the database. That uses buffers, and those are kept in memory.

    Brien Posey? Well . . . consider the source.

    "Exchange extends the maintenance cycle by up to an hour to ensure that the defragmentation has time to complete."

    Suppose it's a really big database? An hour is no guaranty that it'll complete.

    The point is that the tasks will eventually complete. If not today, than tomorrow. For example, it doesn't matter a whole lot if unused indexes hang around for a while, it just matters that they'll be removed in a reasonable period of time. Removing expired mailboxes doesn't take all that long, nor does removing expired messages.

    Other statements in that article are incorrect, too. For example:

    "If you have to perform any type of repair on the database, you'll need enough free space on the volume for a full copy of the database, plus another 10 to 20 percent for overhead."

    In the context of running ISINTEG (which was the subject of that quote) you certainly don't need to have enough room for a full copy of the database.


    --- Rich Matheisen MCSE&I, Exchange MVP

    Friday, August 29, 2014 3:21 AM