CCR - Newly copied logfile corrupt,re-seeding required
Hello,
I'm transitioning from Exchange 2003 to 2007, failover cluster to CCR cluster.
While moving a mailbox from the old to the new server i get the following errors on the standby server:
Newly copied log file E:\Exchange Server\Mailbox\First Storage Group\inspector\E0000000790.log for CCRServer\First Storage Group is corrupt. Re-seeding the passive node is now required. Use the Update-StorageGroupCopy cmdlet in the Exchange Management Shell to perform a re-seed operation.
Microsoft.Exchange.Cluster.ReplayService (1720) Log Verifier 8464211: The log file E:\Exchange Server\Mailbox\First Storage Group\inspector\E0000000790.log is damaged, invalid, or inaccessible (error -501) and cannot be used. If this log file is required for recovery, a good copy of the log file will be needed for recovery to complete successfully.
The active node is running fine, but a failover is not possible anymore.
After that i reseed the storage group it seems to be ok again.
This happened on every three mailboxes i tested to move. Any AV scanner or other non MS Service is switched off.
Can anyone explain me where that behavior could come from?
Is that usual or only coming up in the process of moving a mailbox?
Thank you in advance for your answers.
All Replies
That is very unusual. When you say it happened 3 times, do you mean that the same log file failed three different timesDid you do the full reseed in between? Was it just reported three times? Or was it with three different log files?
During development sometimes I'd forget about the log Inspection phase since it failed so infrequently. It is not directly related to doing a mailbox move. (It could be related in that doing a mailbox move puts an IO load on the systems.)
-501 means that the log file is corrupted. The big question is where did the corruption happen?
Log on to the source server, and run the following:
e:\Program Files\Microsoft\Exchange Server\Mailbox\First Storage Group > eseutil -k E0000000790.log
Choose your own adventure: if the original was:
A. OK. Then something got messed up in the copying. It might be a networking problem, or storage subsystem problem on the target machine.
B. Corrupted. Then something is probably wrong with the storage subsystem on the source machine (could be drivers, overheating, configuration, etc.). Or Exchange has a bug in creating log files, which of course is unlikely.
Look around in the event logs on both machines for disk or networking errors.
-martin
Hello Martin
Thank you for your time!
It happened 3 different times with full reseed in between on three completely different log files.
I did not find any logfiles concerning network or disk errors. But will check for driver updates, as this is always a good approach.
I have now already reseeded, but as soon as i move the next few mailboxes, i will post the results of eseutil -k.
Andy.
Now - after 15 successful mailbox-moves again: re-seeding required.
Here is the output of eseutil -k of source server:
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 08.00
Copyright (C) Microsoft Corporation. All Rights Reserved.Initiating CHECKSUM mode...
Log File: E:\Exchange Server\Mailbox\First Storage Group\inspector\E0000002177.logERROR: Log damaged (unusable). Last Lgpos: (0x2177,8,16). Error -501.
Operation terminated with error -501 (JET_errLogFileCorrupt, Log file is corrupt) after 0.141 seconds.The last application eventlog entry before the ESE error is the following: (30 seconds before)
Logon Failure on database "First Storage Group\Mailbox Database" - Windows account NT AUTHORITY\SYSTEM; mailbox /o=organisation/ou=First Administrative Group/cn=Recipients/cn=username.
Error: 1144
Client Machine: <target-Server>
Client Process: MSExchangeMailboxAssistants.exe
Client ProcessId: 0
Client ApplicationId: Client=MSExchangeMailboxAssistants;Action=EventDispatcher
Drivers are all up to date. System event log is clean. No errors there.
Well... if this happens only when moving mailboxes, i'm happy :-)
Is there anything else i could try to solve that?Since it's on the source, it sounds like the corruption isn't getting introduced during the copying.
Does this happen regardless of which node is the source machine during your move mailbox?
I'd be suspicious of antivirus software, but you clearly said you turned them off.
So maybe it's still an IO subsystem problem. Have you downloaded and run jetstress?
Although you're only seeing the errors while moving mailboxes, it is a cause for future concern -- you might not be able to switch over to the redundant server when you need to.
-martin
