locked
Help needed with Exchange 2010 DB corruption RRS feed

  • Question

  • Problem

    Getting an error moving mailboxes to a new database, Exchange 2010 SP3:

    DisplayName           : <User's Name>
    SyncStage             : FinalIncrementalSync
    FailureCode           : -2147467259
    FailureType           : MapiExceptionJetErrorIndexNotFound
    FailureSide           : Source
    FailureTimestamp      : 6/19/2017 5:34:10 PM
    

    Google didn't help, I'm hoping Reddit can.

    Background

    After the reboot Saturday (due to patch Tuesday), when I got in Monday morning, a bunch of users were reporting they can't access public folders. I noticed all the complaints had one thing in common: The users were all on the same database. Users in a different database on the same server had no problems. Interestingly, the problem users could access via OWA, but not Outlook.

    Some googling seemed to indicate database corruption was the most likely cause. So I took the DB offline, made a backup copy, then ran eseutil /mh <dbname>. That gave me a clean shutdown, so I didn't have to play with the logs. Next I ran eseutil /p <dbname> and that found and "repaired" some errors (2 IIRC). Then I ran eseutil /d <dbname>. That finished and shrunk the DB by about 6 Gigs.

    Since I'm at SP3, and the DB was still 93 Gigs, I skipped the isintegsince that supposedly takes about 1 hour per gig and the DB is offline the whole time. Plus, I've read that it is mostly obsolete since Exchange 2010 SP1.

    Instead I ran

    New-MailboxRepairRequest -Mailbox “Doe, John” -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView -detectonly
    

    on two of the mailboxes I tried to move earlier. Both came back with no errors. At this point, I think I'm good to move off the old DB. When I try to move those two accounts, it failed. The command

    get-moverequest -movestatus Failed|get-moverequeststatistics|select DisplayName,SyncStage,Failure*,Message,PercentComplete,largeitemsencountered,baditemsencountered|clip
    

    returned:

    DisplayName           : <username>
    SyncStage             : FinalIncrementalSync
    FailureCode           : -2147467259
    FailureType           : MapiExceptionJetErrorIndexNotFound
    FailureSide           : Source
    FailureTimestamp      : 6/19/2017 5:34:10 PM
    FailureContext        : --------
                        Operation: ISourceMailbox.CopyTo
                        OperationSide: Source
                        Primary (e06b6ac3-ba8f-47b1-94e1-d97953ea3976)
                        PropTags: [ContainerHierarchy; ContainerContents]
    Message               : Error: MapiExceptionJetErrorIndexNotFound: Unable to copy to target. (hr=0x80004005, ec=-1404)
                        Diagnostic context:
                            Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=82]
                            Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=170][latency=0]
                            Lid: 23226   --- ROP Parse Start ---
                            Lid: 27962   ROP: ropGetPropsSpecific [7]
                            Lid: 21921   StoreEc: 0x40380   
                            Lid: 31418   --- ROP Parse Done ---
                            Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=45]
                            Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=216][latency=0]
                            Lid: 23226   --- ROP Parse Start ---
                            Lid: 27962   ROP: ropGetReceiveFolderTable [104]
                            Lid: 31418   --- ROP Parse Done ---
                            Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=69]
                            Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=176][latency=0]
                            Lid: 23226   --- ROP Parse Start ---
                            Lid: 27962   ROP: ropGetAllPerUserLtids [125]
                            Lid: 17082   ROP Error: 0xFFFFFA84
                            Lid: 29793  
                            Lid: 21921   StoreEc: 0xFFFFFA84
                            Lid: 27962   ROP: ropExtendedError [250]
                            Lid: 1494    ---- Remote Context Beg ----
                            Lid: 26426   ROP: ropGetAllPerUserLtids [125]
                            Lid: 15348   StoreEc: 0xFFFFFA84
                            Lid: 47040  
                            Lid: 2087    StoreEc: 0xFFFFFA84
                            Lid: 1750    ---- Remote Context End ----
                            Lid: 26849  
                            Lid: 21817   ROP Failure: 0xFFFFFA84
                            Lid: 25738  
                            Lid: 18570   StoreEc: 0xFFFFFA84
                            Lid: 23370   StoreEc: 0xFFFFFA84
                            Lid: 24302  
                            Lid: 32494   StoreEc: 0xFFFFFA84
    PercentComplete       : 95
    largeitemsencountered : 
    BadItemsEncountered   : 0
    

    Both had 0 bad items. Googling the error returned nothing matching other than a post about a mailbox that did have bad items, which isn't the case here.

    For kicks, on the same server, I have two more DBs. I did a move from one of those to the other and the move worked, so it's almost certainly a DB problem, not a server problem.

    I also, while the DB was offline, ran a chkdsk on the hard drive. No errors. It's a virtual drive, so physical problems aren't likely as the SAN checks out.

    Any ideas at this point?

    Tuesday, June 20, 2017 3:08 PM

Answers

  • Andy is correct that /P should be the last action since its a destructive process and best way forward is to

    1. Move all the mailboxes within the questionable DB into a new DB

    2. After moving all that you can then dismount the database and do a dial tone action as outlined here http://support.lucid8.com/support/solutions/articles/6000167684-dial-tone-recovery and note that even if they all move out of that DB I would still do the folder rename action outlined in the article vs just deleting the EDB and log files within the file system.  I say this just in case something is missed then you will still have the EDB to open via DigiScope


    Search, Recover, Export Mailboxes, Folders, Email, Contacts, Calendars, Tasks, etc. from Offline Exchange Databases (EDBs), On-Premise Exchange Servers and Office 365. Migrate/Recover direct from any offline EDB into any On-Premises Exchange Server, even cross version i.e. 2003 → 2007 → 2010 →2013 → 2016 → Office 365 with Lucid8's DigiScope

    • Marked as answer by Rob S2 Thursday, June 22, 2017 8:24 PM
    Tuesday, June 20, 2017 5:29 PM

All replies

  • Problem

    Getting an error moving mailboxes to a new database, Exchange 2010 SP3:

    DisplayName           : <User's Name>
    SyncStage             : FinalIncrementalSync
    FailureCode           : -2147467259
    FailureType           : MapiExceptionJetErrorIndexNotFound
    FailureSide           : Source
    FailureTimestamp      : 6/19/2017 5:34:10 PM
    

    Google didn't help, I'm hoping Reddit can.

    Background

    After the reboot Saturday (due to patch Tuesday), when I got in Monday morning, a bunch of users were reporting they can't access public folders. I noticed all the complaints had one thing in common: The users were all on the same database. Users in a different database on the same server had no problems. Interestingly, the problem users could access via OWA, but not Outlook.

    Some googling seemed to indicate database corruption was the most likely cause. So I took the DB offline, made a backup copy, then ran eseutil /mh <dbname>. That gave me a clean shutdown, so I didn't have to play with the logs. Next I ran eseutil /p <dbname> and that found and "repaired" some errors (2 IIRC). Then I ran eseutil /d <dbname>. That finished and shrunk the DB by about 6 Gigs.

    Since I'm at SP3, and the DB was still 93 Gigs, I skipped the isintegsince that supposedly takes about 1 hour per gig and the DB is offline the whole time. Plus, I've read that it is mostly obsolete since Exchange 2010 SP1.

    Instead I ran

    New-MailboxRepairRequest -Mailbox “Doe, John” -CorruptionType SearchFolder,AggregateCounts,ProvisionedFolder,FolderView -detectonly
    

    on two of the mailboxes I tried to move earlier. Both came back with no errors. At this point, I think I'm good to move off the old DB. When I try to move those two accounts, it failed. The command

    get-moverequest -movestatus Failed|get-moverequeststatistics|select DisplayName,SyncStage,Failure*,Message,PercentComplete,largeitemsencountered,baditemsencountered|clip
    

    returned:

    DisplayName           : <username>
    SyncStage             : FinalIncrementalSync
    FailureCode           : -2147467259
    FailureType           : MapiExceptionJetErrorIndexNotFound
    FailureSide           : Source
    FailureTimestamp      : 6/19/2017 5:34:10 PM
    FailureContext        : --------
                        Operation: ISourceMailbox.CopyTo
                        OperationSide: Source
                        Primary (e06b6ac3-ba8f-47b1-94e1-d97953ea3976)
                        PropTags: [ContainerHierarchy; ContainerContents]
    Message               : Error: MapiExceptionJetErrorIndexNotFound: Unable to copy to target. (hr=0x80004005, ec=-1404)
                        Diagnostic context:
                            Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=82]
                            Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=170][latency=0]
                            Lid: 23226   --- ROP Parse Start ---
                            Lid: 27962   ROP: ropGetPropsSpecific [7]
                            Lid: 21921   StoreEc: 0x40380   
                            Lid: 31418   --- ROP Parse Done ---
                            Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=45]
                            Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=216][latency=0]
                            Lid: 23226   --- ROP Parse Start ---
                            Lid: 27962   ROP: ropGetReceiveFolderTable [104]
                            Lid: 31418   --- ROP Parse Done ---
                            Lid: 55847   EMSMDBPOOL.EcPoolSessionDoRpc called [length=69]
                            Lid: 43559   EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=176][latency=0]
                            Lid: 23226   --- ROP Parse Start ---
                            Lid: 27962   ROP: ropGetAllPerUserLtids [125]
                            Lid: 17082   ROP Error: 0xFFFFFA84
                            Lid: 29793  
                            Lid: 21921   StoreEc: 0xFFFFFA84
                            Lid: 27962   ROP: ropExtendedError [250]
                            Lid: 1494    ---- Remote Context Beg ----
                            Lid: 26426   ROP: ropGetAllPerUserLtids [125]
                            Lid: 15348   StoreEc: 0xFFFFFA84
                            Lid: 47040  
                            Lid: 2087    StoreEc: 0xFFFFFA84
                            Lid: 1750    ---- Remote Context End ----
                            Lid: 26849  
                            Lid: 21817   ROP Failure: 0xFFFFFA84
                            Lid: 25738  
                            Lid: 18570   StoreEc: 0xFFFFFA84
                            Lid: 23370   StoreEc: 0xFFFFFA84
                            Lid: 24302  
                            Lid: 32494   StoreEc: 0xFFFFFA84
    PercentComplete       : 95
    largeitemsencountered : 
    BadItemsEncountered   : 0
    

    Both had 0 bad items. Googling the error returned nothing matching other than a post about a mailbox that did have bad items, which isn't the case here.

    For kicks, on the same server, I have two more DBs. I did a move from one of those to the other and the move worked, so it's almost certainly a DB problem, not a server problem.

    I also, while the DB was offline, ran a chkdsk on the hard drive. No errors. It's a virtual drive, so physical problems aren't likely as the SAN checks out.

    Any ideas at this point?

    Running eseutil / p was a bad idea. That should never be run except as a last resort and its now unsupported to run that DB in production. Moving mailboxes should be the first option - before running eseutil/p.

    Can you replace the repaired DB with the saved one that didn't have eseutil/p ran against it? Then try to move mailboxes. Do any mailboxes move at all?

    If that doesn't work, restore from backup is possible, use a 3rd party solution that accesses the edb or open a case with Microsoft support.


    Tuesday, June 20, 2017 3:20 PM
  • I will keep what you said in mind for the next corruption. In the meantime...

    Yes, I still have the backup copy from before the eseutil and other copies from even earlier. But using that presents a new problem - what about all the emails in some 40+ boxes, that have come in since, which even for a particular mailbox are likely scattered over multiple folders? Any way to reconcile the two?

    Right now, the problems are the same as before - Users can access their email just fine, but the ones on the troubled DB can't access public folders through Outlook, but 2 that I know have tried, can do it through OWA. So I suspect the eseutil really did nothing.

    Oh, if you're thinking it's a problem with Outlook, as I did, my account is on another database. I logged into an affected user's computer and using their outlook, can access public folders just fine. I then had him try on my workstation. I can, he can't.


    • Edited by Rob S2 Tuesday, June 20, 2017 4:04 PM clarification
    Tuesday, June 20, 2017 3:59 PM
  • Andy is correct that /P should be the last action since its a destructive process and best way forward is to

    1. Move all the mailboxes within the questionable DB into a new DB

    2. After moving all that you can then dismount the database and do a dial tone action as outlined here http://support.lucid8.com/support/solutions/articles/6000167684-dial-tone-recovery and note that even if they all move out of that DB I would still do the folder rename action outlined in the article vs just deleting the EDB and log files within the file system.  I say this just in case something is missed then you will still have the EDB to open via DigiScope


    Search, Recover, Export Mailboxes, Folders, Email, Contacts, Calendars, Tasks, etc. from Offline Exchange Databases (EDBs), On-Premise Exchange Servers and Office 365. Migrate/Recover direct from any offline EDB into any On-Premises Exchange Server, even cross version i.e. 2003 → 2007 → 2010 →2013 → 2016 → Office 365 with Lucid8's DigiScope

    • Marked as answer by Rob S2 Thursday, June 22, 2017 8:24 PM
    Tuesday, June 20, 2017 5:29 PM
  • Your answer is incomplete, but did get me pointed in the correct direction. I don't have DigiScope, so I am currently working through doing something similar with powershell. It's probably more time consuming, but it working.

    If anyone is hitting this via google in the future, the Cliff Note's version is I use exchange powershell to export the mailbox to a pst file, record the settings for the mailbox, disable (which deletes it effectively), then recreate it in a new database. Then I use more powershell to re-import the pst file. Finally, I have to kill the user's Outlook profile and recreate it. Messy, but it does work.
    • Edited by Rob S2 Thursday, June 22, 2017 9:03 PM clarification
    • Proposed as answer by Allen_WangJFModerator Friday, June 23, 2017 2:12 AM
    Thursday, June 22, 2017 8:24 PM
  • Glad it all worked out and thanks for the update on your workaround method.  

    Search, Recover, Export Mailboxes, Folders, Email, Contacts, Calendars, Tasks, etc. from Offline Exchange Databases (EDBs), On-Premise Exchange Servers and Office 365. Migrate/Recover direct from any offline EDB into any On-Premises Exchange Server, even cross version i.e. 2003 → 2007 → 2010 →2013 → 2016 → Office 365 with Lucid8's DigiScope

    Thursday, June 22, 2017 8:49 PM