locked
Exchange 2010 large database RRS feed

  • Question

  • Ok - I tried to do a little research on my own before asking this but I could use some guidance.

    We have about 100 users that I have on 4 databases.

    Each database is about 30GB

    Powershell to get all mailbox sizes came back and only showed about 32GB

    So my total for all mailboxes is about 25% of the total database size.

    - about 1.5GB total whitespace so it is not that

    - No disconnected mailboxes (cleaned that up as well)

    I have a feeling it is the dumspter - recoverable items.  I looked through the folders using the code below

    Get-Mailbox -Identity -server servername | Get-MailboxFolderStatistics -FolderScope RecoverableItems | Select Identity, ItemsInFolder, FolderSize, FolderType | List Identity, ItemsInFolder, FolderSize, FolderType

    There are alot of users that have over 100k items in recoverable items folders.  This doesnt seem right.  I have a 14 retention policy set up for the deleted items folder with an action of permanently delete.  Also the default for the dumpster is set at the default of 14 days as well so I am not sure why alot of users are showing over 100k items in their dumspter.

    Any guidance is appreciated.

     

    Thanks
    John

    Wednesday, October 5, 2011 5:37 PM

Answers

  • http://www.symantec.com/business/support/index?page=content&id=TECH164949

     

    Getting closer - I think you are on the right track.  I didnt mean to doubt you I just like to find the root cause.  It may seem that right now there isnt a fix other than a temp fix as listed above.

    Symantec wants me to do an offline defrag to fix the issue - I have heard you should never do this without MS support because it can invalidate the databases??

    Personally - an offline defrag is no different than creating a new DB, moving the users??

    Anyways - Ill keep posting when I find more as perhaps it will help others

     

     


    It doesnt invalidate the database, it invalidates any previous backups. By that , you cant restore an older backup and then replay logs from the defragged database against it. The solution of course for that is to get a full backup of the store after you do an offline defrag.

    Creating a new database and moving mailboxes is preferred because there is no downtime. And mailbox moves in 2010 are online and with SP1 RU2 and above, endusers wont get that "Your administrator made a change so you have restart Outlook" prompt thingy.

     

    • Marked as answer by Evan Liu Wednesday, October 19, 2011 6:20 AM
    Wednesday, October 5, 2011 9:54 PM
  • This problem has to be around the way the data is removed from the database to leave the stub behind. It confuses the database.
    I also think it is interesting that Microsoft doesn't support another of the Symantec Exchange database technologies, their Granular Restore process (the replacement for brick level backups). There appears to be a demand (whether it is right or wrong) for working at an item level, which Microsoft do not want to support, but Symantec (and others) have decided to provide.

    An offline defrag creates a new database anyway, so you either create a new database and move mailboxes with no downtime, or get a new database (just with the same name) and an extended period of downtime. I have never done an offline defrag on Exchange 2007 or 2010. Waste of time.

    Simon.


    Simon Butler, Exchange MVP
    Blog | Exchange Resources | In the UK? Hire Me.
    • Marked as answer by Evan Liu Wednesday, October 19, 2011 6:20 AM
    Wednesday, October 5, 2011 10:16 PM
  • eseutil /d is straight forward. Back in the old days with Exchange 5.0 and 5.5, it was sort of common practice. Even with Exchange 2000 and 2003, I happened to do it, perhaps more out of old habit.

    But never with Exchange 2007 or Exchange 2010, other than with my home servers. There should be no need to call MS support. I have never experienced or heard of an invalid database after an offline defrag, given that your Exchange environment is healthy.

    Anyhow, the recommendation these days is to move your mailboxes to a new database and delete the old one. It does incur considerable less downtime. Users are only locked out of their accounts for a brief time at the end of the process of an online move, when the final synchronization occurs.

    An offline defragmentation will handle about 10 GB an hour, depending on your storage subsystem.

    Moving Mailboxes in Exchange 2010 (Part 1 through 4)
    http://www.msexchange.org/articles_tutorials/exchange-server-2010/management-administration/moving-mailboxes-exchange-2010-part1.html


    MCTS: Messaging | MCSE: S+M
    • Marked as answer by Evan Liu Wednesday, October 19, 2011 6:20 AM
    Wednesday, October 5, 2011 10:18 PM

All replies

  • PS - dont get tunnel vision on the dumspter.  It is just my initial thoughts - it could be anything so any ideas are appreciated.

     

    Thanks

    John

    Wednesday, October 5, 2011 5:38 PM
  • Do you have the option enabled to not remove items from DIR until a backup has completed? Are your backups completing successfully? Are you using any third party archiving products against the database? That can often confuse things.

    At 30gb per database, it might just be easier to merge all of the users in to two databases, drop the others, create new ones and move them back.

    Simon.


    Simon Butler, Exchange MVP
    Blog | Exchange Resources | In the UK? Hire Me.
    Wednesday, October 5, 2011 5:41 PM
  • I am not sure how to check the option you are referring to.  We are using eVault for archiving after 30 days.

    I am sure I could move and fix, but if I dont resolve the root issue it will just come back I would assume.

     

    John

    Wednesday, October 5, 2011 5:45 PM
  • PS - backups are running and sucessful - nightly
    Wednesday, October 5, 2011 5:46 PM
  • Do you use Symantec Enterprise Vault or another 3rd-party archiving solution that uses stubbing, that is removing data from Exchange and replacing it with a message stubs, i.e. shortcuts? That can cause your mailbox databases to continue to grow and grow.

    See: Exchange 2010 Database overhead
    http://social.technet.microsoft.com/Forums/en/exchange2010/thread/55776c0b-6760-4c2d-bd3b-f8b10f90ce81


    MCTS: Messaging | MCSE: S+M
    Wednesday, October 5, 2011 5:53 PM
  • I think there is a lot of unexplained bloat in 2010 databases. I see similar posts to yours it seems at least once a week.

    Once thing you may find interesting. I have seen a store go from 150GB to 50GB simply by moving mailboxes. And remember, when you move mailboxes in 2010, the dumpster moves with it, so clearly there was a lot of "junk" in that trunk.

     

     

     

    Wednesday, October 5, 2011 5:59 PM
  • We do use eVault as I mentioned above and are stubbing after 30 days.  I would assume though that the total for a mailbox size would include the stubs though.

    The total for all mailboxes is 32GB / this should include the stubs correct?  I would assume so.

    The total for the databases is over 120GB - 75% of the total seems like an alful lot of junk.

    It would be nice to be able to narrow down the junk or cause of the junk to fix the issue.

    Thoughts?

    John

    Wednesday, October 5, 2011 6:03 PM
  • Any thoughts on the high counts of recoverable items?

    Here is a great example...


    Identity      : domain/Username\Recoverable Items
    ItemsInFolder : 40561
    FolderSize    : 142.1 MB (149,019,893 bytes)
    FolderType    : RecoverableItemsRoot

     

    This user left the company 30 days ago - this should have exceeded the 14 day deleted item retention and dumspter policy and should have 0 counts - correct?

    Wednesday, October 5, 2011 6:11 PM
  • Any thoughts on the high counts of recoverable items?

    Here is a great example...


    Identity      : domain/Username\Recoverable Items
    ItemsInFolder : 40561
    FolderSize    : 142.1 MB (149,019,893 bytes)
    FolderType    : RecoverableItemsRoot

     

    This user left the company 30 days ago - this should have exceeded the 14 day deleted item retention and dumspter policy and should have 0 counts - correct?


    You would think. You can clean that up of course:

    http://technet.microsoft.com/en-us/library/ff678798.aspx

    But I dont have a great answer for you on why its not being cleaned up automatically. Not sure if things get better with the latest RU or if things will improve with 2010 SP2.

     

     

    Wednesday, October 5, 2011 6:17 PM
  • We do use eVault as I mentioned above and are stubbing after 30 days.  I would assume though that the total for a mailbox size would include the stubs though.

    The total for all mailboxes is 32GB / this should include the stubs correct?  I would assume so.

    The total for the databases is over 120GB - 75% of the total seems like an alful lot of junk.

    It would be nice to be able to narrow down the junk or cause of the junk to fix the issue.

    Thoughts?

    John


    Stubbing - that appears to be causing most of the problems with database size issues.
    Microsoft don't recommend stubbing with Exchange 2010 and I have dealt with at least two questions and one client who is stubbing and seeing some odd things happening with the database. Something about the way that stubbing removes the item confuses Exchange. Moving the mailbox to a new database is being reported to usually sort it out.

    Simon.


    Simon Butler, Exchange MVP
    Blog | Exchange Resources | In the UK? Hire Me.
    Wednesday, October 5, 2011 6:54 PM
  • Here are two interesting links, which might support A_ D_'s view.

    AvailableNewMailboxSpace on Exchange 2010 database not growing according to archiving
    http://www.symantec.com/business/support/index?page=content&id=TECH164949

    The Effects Of Archival Stubs On Database Space Management
    http://blogs.technet.com/b/bill_long/archive/2011/09/02/the-effects-of-archival-stubs-on-database-space-management.aspx

    And the link below is removed. Here James Carroll explained the effects on stubbing, and that the only solution was "to move users to a new blank database until the size becomes too large once again and you have to repeat the moves."

    Exchange 2010 Database Page Fragmentation Caused By Archive Solutions (1 Aug 2011)
    http://blogs.technet.com/b/jamec/archive/2011/08/01/exchange-2010-database-page-fragmentation-caused-by-archive-solutions.aspx


    MCTS: Messaging | MCSE: S+M
    Wednesday, October 5, 2011 7:06 PM
  • It doesnt suprise me that Microsoft doesnt recommend any product that is not there own.

    I am not saying it is not evault, but I am not sure what stubbing would have to do with the high dumspter counts.

    For example - my mailbox

    Get-MailboxFolderStatistics -Identity "jdole" -FolderScope RecoverableItems | fl foldertype, it
    ems*, folderpath, foldersize


    FolderType                 : RecoverableItemsRoot
    ItemsInFolder              : 5723
    ItemsInFolderAndSubfolders : 6029
    FolderPath                 : /Recoverable Items
    FolderSize                 : 23.03 MB (24,152,674 bytes)

    FolderType                 : RecoverableItemsDeletions
    ItemsInFolder              : 2
    ItemsInFolderAndSubfolders : 2
    FolderPath                 : /Deletions
    FolderSize                 : 276 B (276 bytes)

    FolderType                 : RecoverableItemsPurges
    ItemsInFolder              : 304
    ItemsInFolderAndSubfolders : 304
    FolderPath                 : /Purges
    FolderSize                 : 10.28 MB (10,783,326 bytes)

    FolderType                 : RecoverableItemsVersions
    ItemsInFolder              : 0
    ItemsInFolderAndSubfolders : 0
    FolderPath                 : /Versions
    FolderSize                 : 0 B (0 bytes)

     


    I then ran the purge as suggested above

    Search-Mailbox -Identity "jdole" -SearchDumpsterOnly -DeleteContent


    After that I ran the command to look at the recoverable counts and they were exactly the same.  Nothing is getting purged from the dumpster????

    What am I missing here...

    JD

    Wednesday, October 5, 2011 7:09 PM
  • Your last link there was very interesting...  I found it cached on google.

     

    JD


    Wonder why it was pulled.
    • Edited by dolejh Wednesday, October 5, 2011 7:12 PM
    Wednesday, October 5, 2011 7:12 PM
  • I would believe, because it either directly or indirectly puts the blame on Symantec Enterprise Vault, EMC OneSource, Mimosa Systems, and favors Microsoft's own archiving solution. In addition it is not so evident that stubbing is the sole cause.

    I was reproducing what highly knowledgeable Exchange experts have said in this forum about stubbing and read a lot of papers about it lately. What I still don't understand is:

    * Exchange 2010 uses 32 K pages.
    * Stubs or short cuts use a lot less.

    Now, why can't Exchange consolidate these stubs and fill up the 32 K pages?


    MCTS: Messaging | MCSE: S+M
    Wednesday, October 5, 2011 7:32 PM
  • http://www.symantec.com/business/support/index?page=content&id=TECH164949

     

    Getting closer - I think you are on the right track.  I didnt mean to doubt you I just like to find the root cause.  It may seem that right now there isnt a fix other than a temp fix as listed above.

    Symantec wants me to do an offline defrag to fix the issue - I have heard you should never do this without MS support because it can invalidate the databases??

    Personally - an offline defrag is no different than creating a new DB, moving the users??

    Anyways - Ill keep posting when I find more as perhaps it will help others

     

     

    Wednesday, October 5, 2011 9:47 PM
  • http://www.symantec.com/business/support/index?page=content&id=TECH164949

     

    Getting closer - I think you are on the right track.  I didnt mean to doubt you I just like to find the root cause.  It may seem that right now there isnt a fix other than a temp fix as listed above.

    Symantec wants me to do an offline defrag to fix the issue - I have heard you should never do this without MS support because it can invalidate the databases??

    Personally - an offline defrag is no different than creating a new DB, moving the users??

    Anyways - Ill keep posting when I find more as perhaps it will help others

     

     


    It doesnt invalidate the database, it invalidates any previous backups. By that , you cant restore an older backup and then replay logs from the defragged database against it. The solution of course for that is to get a full backup of the store after you do an offline defrag.

    Creating a new database and moving mailboxes is preferred because there is no downtime. And mailbox moves in 2010 are online and with SP1 RU2 and above, endusers wont get that "Your administrator made a change so you have restart Outlook" prompt thingy.

     

    • Marked as answer by Evan Liu Wednesday, October 19, 2011 6:20 AM
    Wednesday, October 5, 2011 9:54 PM
  • This problem has to be around the way the data is removed from the database to leave the stub behind. It confuses the database.
    I also think it is interesting that Microsoft doesn't support another of the Symantec Exchange database technologies, their Granular Restore process (the replacement for brick level backups). There appears to be a demand (whether it is right or wrong) for working at an item level, which Microsoft do not want to support, but Symantec (and others) have decided to provide.

    An offline defrag creates a new database anyway, so you either create a new database and move mailboxes with no downtime, or get a new database (just with the same name) and an extended period of downtime. I have never done an offline defrag on Exchange 2007 or 2010. Waste of time.

    Simon.


    Simon Butler, Exchange MVP
    Blog | Exchange Resources | In the UK? Hire Me.
    • Marked as answer by Evan Liu Wednesday, October 19, 2011 6:20 AM
    Wednesday, October 5, 2011 10:16 PM
  • eseutil /d is straight forward. Back in the old days with Exchange 5.0 and 5.5, it was sort of common practice. Even with Exchange 2000 and 2003, I happened to do it, perhaps more out of old habit.

    But never with Exchange 2007 or Exchange 2010, other than with my home servers. There should be no need to call MS support. I have never experienced or heard of an invalid database after an offline defrag, given that your Exchange environment is healthy.

    Anyhow, the recommendation these days is to move your mailboxes to a new database and delete the old one. It does incur considerable less downtime. Users are only locked out of their accounts for a brief time at the end of the process of an online move, when the final synchronization occurs.

    An offline defragmentation will handle about 10 GB an hour, depending on your storage subsystem.

    Moving Mailboxes in Exchange 2010 (Part 1 through 4)
    http://www.msexchange.org/articles_tutorials/exchange-server-2010/management-administration/moving-mailboxes-exchange-2010-part1.html


    MCTS: Messaging | MCSE: S+M
    • Marked as answer by Evan Liu Wednesday, October 19, 2011 6:20 AM
    Wednesday, October 5, 2011 10:18 PM
  • I ended up just creating new databases and moving the users over.  Alot faster and easier on the users.

    JD

    Wednesday, December 28, 2011 4:31 PM