none
Exchange Server 2007 Disaster Recovery

    Question

  • Our lone mail server went down.  When I couldn't bring the physical server back online, I used Backup Exec to restore to VM. None of the databases would mount without running eseutil /p due to "dirty shutdown".  Because time was of the essence and we had been down for about 20 hours, I just mounted the DBs so that mail would start flowing.  My server is now up and running.  The problem I'm facing now is that Backup Exec won't successfully complete a GRT backup so I can't restore individual items anymore.  This is because the old transaction logs are still there for each database and not being committed.

    1. My question is, in those logs, is that data in people's mailboxes now? or can it be deleted obviously with the understanding that that much data will be gone for ever.
    2. If I were to attempt to replay those old logs on a non-crucial database, is there a chance of at least getting some data in the mailboxes or because the new logs exist the old ones are worthless? 

    3. Is it safe to assume that the total amount of data the old logs take up for each DB are the approximate amount of data that would be lost for each?

    Sorry for the complex description but it's a complex problem.  Trying to see if I should just cut my losses or attempt to replay them.  I wouldn't want to mess up any new data, just add old.  If that's even possible.  Any feedback would be greatly appreciated.  I do understand that Exchange 2007 is EOL and am planning an upgrade this year.


    • Edited by T3kky Tuesday, June 5, 2018 5:48 PM
    Tuesday, June 5, 2018 5:02 PM

All replies

  • So a few things

    1. When a database is backed up via an Exchange aware agent its ALWAYS in a DIRTY state

    2. The backups contain the EDB as well as the associated logs which you should ensure are restored as well

    3. When you roll logs into a DB its a one and done process, i.e. Lets assume you have a sequential set of logs from the 1st to the 15th in the backup and the logs from the 16th -31st still on the server. So you would want to put all the logs into the same location and roll the entire log set into the EDB because once a log set is committed the DB goes into a clean/consistent state. So with that in mind if you rolled the logs from the 1-15th and the DB completes the rollup and becomes consistent/clean you cannot go back and rollup the logs from the 16th to the 31st, make sense?

    4. Logs contain items sent, received, modified, deleted etc so in terms of data loss its not just about what was received sent.

    5. As far as the OLD logs being present for GRT that should be ok as long as you have the LOGS from the backup comingled with the ones from the backup and you remove the checkpoint file so that you can roll up the entire set.  Now as stated above the log files must be sequential or else it will not complete, i.e. if you have logs 1-30 and log 10 is missing or corrupted its not going to be able to complete

    6. You can ONLY rolls logs into a database from which they were created, i.e. there are internal stamps that tell the logs and DB if they can be rolled up

    7. Depending on how your backups were created you should be able to restore the EDB and log sets as an RDB, i.e. spoof alternate location and just ensure you tell the backup to NOT finalize, complete, or mount the DB since you may want to be able to merge in other log files.

    8.  You should only do a /P when there is no other option since that is a destructive process

    9. When you say that you mounted the DB's so that email would begin flowing are you saying you did a dial-tone of the DB's or that you did a /P against the EDB's and then upon completion you were able to mount them?  If the latter what was the date of the backup you restored and the date of the failure.  Do you still have the logs available from the failure date?


    Search, Recover, Export Mailboxes, Folders, Email, Contacts, Calendars, Tasks, etc. from Offline Exchange Databases (EDBs), On-Premise Exchange Servers and Office 365. Migrate/Recover direct from any offline EDB into any On-Premises Exchange Server, even cross version i.e. 2003 → 2007 → 2010 →2013 → 2016 → Office 365 with Lucid8's DigiScope

    Tuesday, June 5, 2018 6:59 PM
  • Thank you for all the information it helps me understand the logs a little better.  I wasn't having any luck getting the stores mounted so I did a eseutil /p against the EDBs to get them to be able to mount and didn't touch them after that.  If you're saying that once they're in a clean state they can't be updated with the old logs then I'm guessing I have my answer.  Unfortunately I do not have the luxury of another maintenance window that could result in mail being down.  Since then I have a problem with my PF DB where it has old emails in some of the folders that can't be deleted.  I was thinking about taking it offline and trying to repair and then run an integrity check, but not sure I want to do that anymore.  Everything is now working there is just a few days worth of transactions logs that haven't been applied.  So that data then represents the loss if I were to calculate each DB's size and log size?  So these transaction logs have the changes in them and users don't see that unless I go through and commit them, correct?
    Tuesday, June 5, 2018 8:44 PM
  • Here is an example of what I have for logs...  Some current and then a bunch from a few days before the outage.  Not sure that I should even bother with them...  If nobody will notice any change by deleting them that might be the easiest route.  If I can calculate possible loss by dividing logs size and DB size to give me an estimated MB loss then at least I have that.  For this DB there is 326MB of old logs...
    Tuesday, June 5, 2018 9:04 PM
  • 1. Yes once a DB is in a clean/consistent state you can no longer apply historical logs.  Also even when you have a DB that is dirty because it was backed up you can only roll logs from the point in time the DB was backed up to the latest log file, i.e. you cant take a backup of a DB on Wednesday and apply logs from the previous Monday/Tuesday

    2. Unknown whats in the logs, it could be email that was inbound, state changes i.e. read/unread deletions etc

    3. You shouldn't have to take the production server offline to restore an EDB from backup as long as you are using the RSG process https://technet.microsoft.com/en-us/library/bb124039%28v=exchg.80%29.aspx?f=255&MSPPError=-2147217396

    4. You only have 350 Logs so at most you lost 350MB of transactions being committed to the DB assuming the 350 logs your showing were from the point in recovery, i.e. if the logs were from May 19th to June 5th and lets say you restored the EDB from May 10th you would have more loss, i.e. all logs that may have been backed up yet not committed to that backup and of course all the logs from that point in time forward

    5. Alas you cannot use RSG for PF's so your a bit stuck on that one, however that said

    • What caused the server to go down? 
    • I would look at the SYSTEM event log to ensure there are no disk related error or critical events and if so correct them immediately else the DB's are sure to crash again
    • If the SYSTEM events are clean look within the APPLICATION event logs for error and critical events regarding those databases.  if there are critical and error application events within a DB you may need to do some additional work, i.e. in Exchange 2007 and earlier best practice post /P was to always run Isinteg against the DB until it reported no additional errors were found. What I mean is at the end of the Inisteg process it would tell you how many errors and warnings were fixed.  Warnings are not so much an issue, however if we see any errors being fixed you need to run isinteg against that DB until you see that 0 errors were fixed/corrected.
    • Once isinteg is completed you should then run a eseutil /d to defragment the DB

     


    Search, Recover, Export Mailboxes, Folders, Email, Contacts, Calendars, Tasks, etc. from Offline Exchange Databases (EDBs), On-Premise Exchange Servers and Office 365. Migrate/Recover direct from any offline EDB into any On-Premises Exchange Server, even cross version i.e. 2003 → 2007 → 2010 →2013 → 2016 → Office 365 with Lucid8's DigiScope

    Tuesday, June 5, 2018 11:12 PM
  • Great info!

    Ok, so the log files contain the changes and they are not applied to the mailboxes so no one will know anything is missing until they do.  I have already informed my executive team that we had some minor loss due to the nature of the mounting process that had to be done.

    As far as the server going down, MS updates were applied and then the system was restarted. It just never came back up.  Hours were spent trying and even MS Server Support was called and they worked on it with no success.

    The new server is a VM and I have been monitoring the event logs since.  Only a couple databases showed error and I have already fixed them by creating new ones and moving the affected mailboxes over.  Overall the system seems pretty stable at this point as far as event logs are concerned.  Only warninigs coming up now are related to my mobile device users.  A few needed to be cleaned up, but the weird thing is even the ones mentioned now in the events say their mobile email is working.  I think I'm going to reset them just to clear those as well.

    My only problem to fix now is the PF DB and I've looked at tons of articles that talk about what you mentioned integrity check until 0 erros and then defrag.  To do that I'd have to dismount the DB which I'm hesitating to do at this point, but need to so it'll happen this Friday night.  I have 2 problems left I hope to resolve with the PFs after I complete this process.

    1. I get the MSExchangeFBPulish Event 8207 constantly.

    Log Name:      Application
    Source:        MSExchangeFBPublish
    Date:          6/7/2018 10:03:08 AM
    Event ID:      8207
    Task Category: General
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      MAIL.domain.local
    Description:
    Error updating public folder with free/busy information on virtual machine MAIL. The error number is 0x80004005.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="MSExchangeFBPublish" />
        <EventID Qualifiers="49153">8207</EventID>
        <Level>2</Level>
        <Task>1</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2018-06-07T15:03:08.000Z" />
        <EventRecordID>4251450</EventRecordID>
        <Channel>Application</Channel>
        <Computer>MAIL.domain.local</Computer>
        <Security />
      </System>
      <EventData>
        <Data>MAIL</Data>
        <Data>0x80004005</Data>
      </EventData>
    </Event>

    Goofy thing is when I check to see replica setup everything looks to be in order, unless I'm not reading that right.  I got rid of the Exchange 2003 server like 8 years ago:

    Name     : SCHEDULE+ FREE BUSY
    Replicas : {}

    Name     : EX:/O=Company Name/OU=CMPNM
    Replicas : {MAIL\PublicFolders\PFDB}

    Name     : EX:/o=Company Name/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)
    Replicas : {MAIL\PublicFolders\PFDB}

    2. Can't delete certain emails, in PFs, from before the outage even as Domain Admin.





    • Edited by T3kky Friday, June 8, 2018 1:43 PM
    Thursday, June 7, 2018 3:31 PM