none
Exchange 2010 DAG transaction logs not truncating after VSS Full backup

    Question

  • I have four Exchange 2010 servers running under VMWare 4.  All servers run on Server 2008 R2.  Two are running CAS/HT with WNLB and two running mailbox roles in a DAG.  The databases are stored externally on an HP fiber channel EVA.

    Both servers in DAG have two nics; one for MAPI traffic and one for replication (on two separate subnets).  Get-DatabaseAvailabilityGroupNetwork shows MapiAccessEnabled : True and ReplicationEnabled : False on DAGNetwork01 and the opposite on DAGNetwork02.  Under Database Availability Groups in EMC, all the interfaces show as UP and all databases show Mounted and Healthy as expected.  Under Get-MailboxDatabaseCopyStatus, the ContentIndexState shows Healthy for all databases.  All transaction logs replicate just fine to all databases and email works perfectly.

    I have tested backups with both Windows Server Backup and HP Data Protector.  I've tested backups on both active and passive copies of the databases.  Since Rollup3, I can use Windows Server Backup to backup both active and passive copies.  With both software packages, the backups run and complete fine.  This is what I get logged on the MB1 with the active copies:

    Information Store (2672) Database1: No log files can be truncated.

    Exchange VSS Writer (instance ceecab1d-e306-4f44-9269-4f29306f91cf:6) has successfully completed the full or incremental backup of replicated database 'Database1'. The log files will be truncated after they have been replayed.

    Then on MB2:

    The replication instance for database Database1 has copied and replayed multiple logs.

    However, no logs get truncated.  Actually, with every backup I do, it adds a log or two!  Going in the wrong direction here.  I did add the second NIC to the mailbox servers and split MAPI and replication traffic after the Exchange installation.  So maybe I missed some setting in splitting traffic and I did not add any static routes to the replication NICs because the mailbox servers replicate transaction logs just fine.  The only problem I have is backup log truncation.

    Thanks for any help!

    Brian

     

    Tuesday, April 27, 2010 9:35 PM

Answers

  • After about six hours on the phone with MS Support, the problem is related to the Microsoft Exchange Search Indexer Service.  I don't know if it is a bug in Exchange or something in my design, but I had to dismount databases, stop the Exchange Information Store Service then delete catalogs from all database copies (active and passive).  After restarting Exchange Information Store Service and remounting the databases the backs now truncate logs.  The rep said the Microsoft Exchange Search Indexer Service wasn't allowing the Replication Service to truncate logs.

    In addition to this issue I am seeing problems with the ContentIndexState, upon replica creation, being set to Crawling and never coming out.  If I restart the Microsoft Exchange Search Indexer Service, the state moves to Healthy.

    It appears all my issues point to the Microsoft Exchange Search Indexer Service and it's manifesting itself in different ways.  My case is in the process of being escalated so I'll post back with an update.  Hopefully someone else won't have to spend weeks fighting this too.

    Monday, May 03, 2010 7:20 PM

All replies

  • Are all of the copies for the database either healthy or mounted?
    Tuesday, April 27, 2010 11:37 PM
  • Yes, all copies are healthy or mounted.
    Wednesday, April 28, 2010 3:25 PM
  • The factors that determine truncation (when you're using backups insted of circular logging) are a successful backup, the checkpoint depth on the active, and the required range on all of the copies. If you've had a successful backup, look at the checkpoint depth perf counter of the active database copy instance.

    What's the log generation rate on the database?

    Wednesday, April 28, 2010 3:35 PM
  • After about six hours on the phone with MS Support, the problem is related to the Microsoft Exchange Search Indexer Service.  I don't know if it is a bug in Exchange or something in my design, but I had to dismount databases, stop the Exchange Information Store Service then delete catalogs from all database copies (active and passive).  After restarting Exchange Information Store Service and remounting the databases the backs now truncate logs.  The rep said the Microsoft Exchange Search Indexer Service wasn't allowing the Replication Service to truncate logs.

    In addition to this issue I am seeing problems with the ContentIndexState, upon replica creation, being set to Crawling and never coming out.  If I restart the Microsoft Exchange Search Indexer Service, the state moves to Healthy.

    It appears all my issues point to the Microsoft Exchange Search Indexer Service and it's manifesting itself in different ways.  My case is in the process of being escalated so I'll post back with an update.  Hopefully someone else won't have to spend weeks fighting this too.

    Monday, May 03, 2010 7:20 PM
  • Any updates on this from the escalation team?
    We're having the exact same issue.
    Sunday, May 16, 2010 6:06 PM
  • Here's one other solution we discovered ...

    On closer inspection of the logs it appears that only one database was not truncating files after backup. This was a newly created database with several moved mailboxes. Because of bandwidth costs we ended up canceling several of the move requests before they completed. It would appear that the files not truncating were those created during these move requests. After moving everyone out of the new database, removing it and all related files, creating a brand new database and moving everyone back in, the backups finished immediately and the newly created log files were truncated.

    Most likely as a result of the canceled move requests the associated log files were discarded as valid from the indexing service and never truncated.

    Hope that helps!

    Sunday, May 16, 2010 11:06 PM
  • Hi Kris.  I've been working with MS Support now for weeks without a resolution.  My scenario is similar in that these are brand new databases, but I didn't move any accounts.  I'm glad you got yours working but my complaint to MS is that we shouldn't have to do things like delete databases, catalogs etc to make truncation work.  So, unfortunately, I'm still having issues.  We've decided to go live with 2,000 users without the DAG passive node.  Hopefully we can find a solution quickly.

     

    Thanks for your help!

    Monday, May 17, 2010 8:44 PM
  • Hi,

    I stand corrected on the resolution.
    It would appear that after a couple nights of backup the files are again not truncating.
    We have a few items in the replication cue that need to finish up, i'm starting to think this may have something to do with it?
    Then again after hearing your story I'm not going to hold my breath.

    Please let me know when Microsoft finds that resolution!
    I agree with your position completely

    Monday, May 17, 2010 11:25 PM
  • Looks like i was correct on the replication aspect
    Once the replication finished, the logs were truncated.
    That actually makes sense as the replication cycles are based on the log files themselves if I understand it correctly.
    Had the files truncated before the replication cycle completed, well .. that would be bad news :)

    Hopefully Microsoft figures your issues out, keep me posted

    Tuesday, May 18, 2010 7:53 PM
  • This problem is still ongoing.  I've decided to back out the second mailbox server and reinstall it from scratch.  I'll post back.
    Saturday, May 29, 2010 9:46 PM
  • I had very similar problems.  In my case, when I scheduled the backup, I selected the Logs and the Database disks, along with the system state (old school logic I guess).  I was advised to remove the system state from the selection and re-run my schedule.  My next backup ran successfully and generated a successful backup event (ID9827) Source: MSExchangeIS "Exchange VSS Writer has successfully completed the full or incremetal backup of replicated database 'Mailbox Database 1'.  The log files will be truncated after they have been replayed."

    Hope this helps someone.

    Thursday, July 15, 2010 5:23 PM
  • any more detail here? I'm seeing similar behavior in my DAG; seems related to logs from mailbox moves but i'm not sure. Any final word from MS?
    http://exchangedeepthoughts.blogspot.com
    Tuesday, July 27, 2010 3:51 PM
  • I don't have a DAG but am getting the same behaviour with my backups. I have SP1 and am using a simple Windows Backup as are 3rd party solution wasn't working and since it's not working with windows Backup it's clearly not their issue? Any word on this one?
    Tuesday, November 16, 2010 5:55 PM
  • Just to be sure. You Guys do know that MS-Backup only supports the active mailboxes not the copies?!

    (ignoring this fact does give these errors when trying to backup the stores).

    A workAround has can be seen here: http://www.mikepfeiffer.net/2010/10/windows-server-backup-with-volumes-that-include-active-and-passive-exchange-databases/

     

    Regards

    Monday, January 31, 2011 10:43 AM
  • Hi vukovin, We are also having the same kind of setup and we are facing the same issues. Did you get any resolution from MS.
    Tuesday, April 19, 2011 6:00 AM
  • All, Does anyone have any resolution for this issues. I have dismounted the store stoppen MS IS service and deleted the catlog file in both the dag server, but no luck, the backup completed successful and requested for log deletion, but logs are not cleared after full backup in DAG.
    Thursday, April 28, 2011 6:41 AM
  • Hi Jorgen

    i tried the link you provided,

    but the server got restarted when the backup starts, do you have any update on that.

    i have a db server with DAG member and passive db.

    Kindly advice 

    Monday, July 25, 2011 12:02 PM
  • you are right MS only supports only active dbs. Which is why we use Netbackup to backup passive databases from our DR site.

     

    anyway, i do want to mention that admins need to check their Exchange VSS writer status when logs are not truncating.


    got a question? guarantee answers at: http://www.infotechguyz.com/forum/
    Thursday, July 28, 2011 12:53 AM
  • Dear Vukovin,

     

    I have the same problem logs are not purging (truncates)

    as we are aware WSB does not support passive backups but should not aslo forgot that Enabling the VSS writers can perform a backup in my case i got error "application not available" and logs are not truncating.

    shall we dismount all databases and move all the logs in to USB drive then mountback the database?

    is any confirmation from MSFT?

    I can also agree with your point Microsoft Exchange search services behaviour

    Please let me know if you fixed the problem

    Regards

     


    RaSa
    • Edited by Syed_R Monday, August 08, 2011 3:54 AM More to input
    Friday, July 29, 2011 5:06 AM
  • Dear All,

    My diskspace was filling i have made room for logs

    i have dismounted and move the logs and re-mounted the databases the logs files were cleared and i got hte sufficient space after that i run a full backup but still the logs are not truncates.

    intrestingly Application for exchange a radio button is disabled i though there must be some procudere to re-register the WSBEXCHANGE.EXE while backup is permorming services msft extension for exchange is stopped

    MSFT should immidiatley look in to the issue. or i think we may to swithc to symentec backup exec solution

     

    regards

     


    RaSa
    • Proposed as answer by Syed_R Tuesday, August 23, 2011 8:55 AM
    • Unproposed as answer by Syed_R Tuesday, August 23, 2011 8:55 AM
    Monday, August 08, 2011 3:59 AM
  • Dear All,

    WSB services were automatic set back to "Manual"

    Enable the VSS writes registry HKLM\software\microsoft\exchangeserver\v14\replay\parameters =0

    run the BETEST and download the hotfix http://support.microsoft.com/kb/977096

    i have resolve my issue

     

     


    RaSa
    • Proposed as answer by Syed_R Tuesday, August 23, 2011 8:55 AM
    Tuesday, August 23, 2011 8:55 AM
  • Boyz and Girlz: Fixit.  Switch the DB to Circular Logging, do a backup, switch it back to normal logging.  This will allow the logs to all truncate, and the Exchange Server knows what to do during the switch.  Problem is usually caused by missing logs.  Rick.
    Monday, October 10, 2011 2:55 AM
  • "Microsoft Exchange Server 2010 databases should not be configured to use circular logging if you also plan to use the Volume Shadow Copy Service to enable third-party (or custom) backup and recovery operations." See the article below.

    http://msdn.microsoft.com/en-us/library/aa579094(v=exchg.140).aspx

    • Edited by Quinnpumps Sunday, October 23, 2011 3:10 AM
    Sunday, October 23, 2011 3:09 AM
  • Did anyone get the logs to truncate on a persistent basis? This is very annoying to say the least. Our system is a mixed environment of Exchange 2003 and 2010. I have tried several methods and all work for just a day, then the following day I am back to where I began, no truncation of logs.

    I followed this forum and dismounted the stores, the Information store, then deleted the catalogs on both active and passive (only two servers in our DAG). Started the Information Store, mounted the databases, waited for all to replicate and the Index state to complete crawling, ran the backup (Windows backup - no passive databases on node) and  that worked - logs got truncated, only to stop the following day.

    Sometimes I restart the active, let it fail over to the passive, then wait to ensure everything is healthy, restart the (now active) database) to let it fail back to the original.  The backups will truncate then the following day same problem.

    Another think I have done is to dismount the active database, clear the "transaction logs" (not the database) on the PASSIVE only, then re-mount the database . This immediately reseeds the logs and the backup will go through and truncate. The problem is that all users are affected whenever the databases are dismounted. Surely there has to be a better way.

    Any help is appreciated.

    Vaughn

     

    Thursday, December 01, 2011 3:06 AM
  • any update frnds .. now m stuck with filling of log files .. and DAG.. please advise...

    Thanks
    Happiness Always
    Jatin


    • Edited by 'Jatin' Tuesday, February 07, 2012 4:39 PM
    Tuesday, February 07, 2012 4:39 PM
  • Any news on this one? We have the same issue with no truncation of log files.
    Wednesday, May 02, 2012 8:23 AM
  • That is strange, we all are facing same issue but no MSFT is replying?

    Hello MSFTs this is crucial issue. Please support.

    Sunday, May 27, 2012 11:36 AM
  • Were stuck with logs not being truncated as well.

    Any help will be appreciated

    Tuesday, May 29, 2012 10:11 AM
  • Dear

    Hope you are using WSB if correct then follow the

    WSB services were automatic set back to "Manual"

    Enable the VSS writes registry HKLM\software\microsoft\exchangeserver\v14\replay\parameters =0

    run the BETEST and download the hotfix http://support.microsoft.com/kb/977096

    i have resolve my issue thats all

    BETEST is a must in this senerio pls.


    RaSa

    • Proposed as answer by Syed_R Wednesday, May 30, 2012 9:05 AM
    Wednesday, May 30, 2012 9:05 AM
  • Please check to see if the database shows as being backed up in Exchange (regardless of whether your backup job shows it completed successfully). If the archive bit is not being reset (does not show as being backed up successfully), then you may need to create a new mailbox database and move the users over to it. This was Microsoft's suggestion when I spoke to them about it. Also, if you use a BES server, don't forget to set the permissions on the new mailbox to allow the BES administrator read access to the mailbox.
    Wednesday, June 06, 2012 9:06 PM
  • thnks for sharing this

    Thanks Rajat Swain "Make iT Simple"

    Sunday, August 19, 2012 1:07 PM
  • I thought I would share the command to check on the time of the last full backup for each database....

    it's this one:

    Get-MailboxDatabase -status  -id mb1 | fl

    (where mb1 is the name of my database)

    In my environment, this command is the only one showing the correct 'LastFullBackup' time in CET (which is the same timezone as where my exchange server is located).

    Nevertheless, the logs only truncate sometimes, not for every backup.

    Perhaps this is due to too few changes in my test environment.

    Regardless, it's shameful that Microsoft does not bother to document this properly.

    Perhaps Microsoft takes advantage of this because their customers would log a case with the Backup application Vendor, thinking that it's the VSS requestor which should truncate the logs.

    For those who read this, please note that it is EXCHANGE VSS Writer responsibility to truncate logs and the only VSS requestor's responsibility, in this context, is to make a successful backup.

    Their sole and incomplete explanation I could find was this:

    http://technet.microsoft.com/en-us/library/dd335158.aspx

    There is a much more interesting blog instead, here:

    http://blogs.technet.com/b/timmcmic/archive/2012/03/12/exchange-2010-log-truncation-and-checkpoint-at-log-creation-in-a-database-availability-group.aspx

    and this:

    http://blogs.technet.com/b/timmcmic/archive/2012/05/14/verifying-log-file-truncation.aspx

    (but those links do not cover all scenarios where logs are truncated or where they aren't).

    I report the most significant part (in case that link gets dead):

    "Why do we see logs remaining on disk for longer?  Exchange 2010 SP1 and newer introduces a change in the behavior of log truncation.  The changes were taken to ensure that replicated copies of databases within a database availability group always had the appropriate log files on the source server to complete an incremental resynchronization."

    "The change to log truncation is the tracking of Checkpoint At Log Creation.  Remember that in a database availability group we can expect the checkpoint to be approximately 100 logs (or slightly more) off the current log file – this is known as checkpoint depth"

    enjoy!

    Domenico.




    Wednesday, November 21, 2012 1:39 PM
  • Just wondering how you guys got on with this one..? 

    I had the same issue over the last couple of days and it came down to the oul' chestnut of UAC... I've disabled UAC now and all is well..!

    Hope this helps someone out..

    Reg..

    Friday, December 07, 2012 10:48 AM
  • So I am facing this exact problem on June 1st 2013. I ran the Get-MailboxDatabase -status  -id mb1 | fl command  and the results show that the database reports the correct backup time.

    No back up errors and the logs did not truncate, I verified this by running eseutil /ml E02 in the logs directory and piping it out to a text file before and after the backup. This confirmed that the same log file was listed as the first file in each result.

    I logged on to all of the DAG Copy servers and found one with the Catalog status of failed, Not visible from EMC and when checking replication from the primary server everything reported healthy.

    On the DAG Copy server with the failed catalog I was able to identify that this server could not communicate with the primary server, the request would just time out. From the primary server back to this copy server no issues with communication.

    So my problem was the copy server could not communicate with to the primary server.  It ended up being a corrupt arp entry, so I deleted the bad entry arp -d (IPADDRESS).

    In seconds 40GB worth of logs flushed away.

    I hope this helps someone else.

    Saturday, June 01, 2013 3:51 PM
  • How is that related to the Log file truncation?

    At the moment my Exchange Server 2007 CCR mailbox is also not truncating log successfully even after the successfully Pbackup from the passive node.

    The strange thing is that the LatestFullBackupTime column shows older date (more than 4 months ago) when the backup was finished last night at 11:11 PM yesterday.

    As now I speak, the transaction log are still growing.


    /* Server Support Specialist */

    Tuesday, December 03, 2013 10:27 PM