Exchange 2003-Exchange 2010 SP1 Local Move Request Completes Successful but with transient erors
-
Monday, May 14, 2012 7:15 PM
Performing local move requests from Exchange 2003 to Exchange 2010 SP1 environment. Target has 2 dedicated CAS/HT servers with F5 load balancers in front of them. I have performed about 112 test moves and pretty much any mailbox > 200MB gets the following transient errors while moving the mailboxes. The transient errors occur repeatedly for xmins-xhrs and then the mailboxes ALL complete successfully, eventually, but some have up to 8 hours of delays due the transient errors. I am also seeing about a 3:1 ratio of data moved across the network in terms of mailbox size and data moved. I found something similar to this on this forum, but not exactly the same related to this error I am seeing but wanted to see if someone could address my issue directly. I removed some information (attributes about the mailbox on the succesful migration part as it is irrelevant to the error).
Appreciate the help!
5/5/2012 3:58:50 PM [XSJ-PVEXCAS01] 'companyxx.com/People/XSJ/XSJ/service-user' created move request.
5/5/2012 3:58:52 PM [XSJ-PVEXCAS01] The Microsoft Exchange Mailbox Replication service 'XSJ-PVEXCAS01.companyxx.com' (14.1.355.1 caps:07) is examining the request.
5/5/2012 3:58:52 PM [XSJ-PVEXCAS01] Connected to target mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)', database 'XSJ-PSEXMBX02-DB007', Mailbox server 'XSJ-PSEXMBX02.companyxx.com' Version 14.1 (Build 218.0).
5/5/2012 3:58:52 PM [XSJ-PVEXCAS01] Connected to source mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)', database 'XSJ-EXCHVS1\XSJ-SG1-VS1\XSJ-SG1-PRIV3-VS1', Mailbox server 'XSJ-EXCHVS1.companyxx.com' Version 6.0 (Build 7655.0).
5/5/2012 3:59:51 PM [XSJ-PVEXCAS01] Request processing started.
5/5/2012 3:59:51 PM [XSJ-PVEXCAS01] Mailbox signature will not be preserved for mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)'. Outlook clients will need to restart to access the moved mailbox.
5/5/2012 3:59:52 PM [XSJ-PVEXCAS01] Source Mailbox information before the move:
Regular Items: 3490, 346.3 MB (363,172,454 bytes)
Regular Deleted Items: 1, 1.466 KB (1,501 bytes)
FAI Items: 10, 0 B (0 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
5/5/2012 3:59:52 PM [XSJ-PVEXCAS01] Initializing folder hierarchy in mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)': 25 folders total.
5/5/2012 3:59:52 PM [XSJ-PVEXCAS01] Folder hierarchy initialized for mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)': 25 folders total.
5/5/2012 3:59:52 PM [XSJ-PVEXCAS01] Stage: CreatingInitialSyncCheckpoint. Percent complete: 15.
5/5/2012 3:59:52 PM [XSJ-PVEXCAS01] Stage: LoadingMessages. Percent complete: 20.
5/5/2012 3:59:56 PM [XSJ-PVEXCAS01] Stage: CopyingMessages. Percent complete: 25.
5/5/2012 3:59:56 PM [XSJ-PVEXCAS01] Copy progress: 0/3499 messages, 0 B (0 bytes)/346.3 MB (363,172,320 bytes).
5/5/2012 3:59:56 PM [XSJ-PVEXCAS01] Messages have been enumerated successfully. 3499 items loaded. Total size: 346.3 MB (363,172,320 bytes).
5/5/2012 4:04:34 PM [XSJ-PVEXCAS01] Transient error MapiExceptionNetworkError has occurred. The system will retry (1/60).
Error details: MapiExceptionNetworkError: IExchangeFastTransferEx.TransferBuffer failed (hr=0x80040115, ec=0)
Diagnostic context:
Lid: 45025
Lid: 45345 StoreEc: 0x80040115
Lid: 22630
at Microsoft.Exchange.MailboxReplicationService.MapiUtils.ExportMessageBatch(ISourceMailbox mailbox, List`1 messages, GetProxyPoolDelegate getProxyPool, ExportMessagesFlags flags, PropTag[] propsToCopyExplicitly, PropTag[] excludeProps, TestIntegration testIntegration, LocalizedException& failure)
at Microsoft.Exchange.MailboxReplicationService.MapiUtils.ExportMessagesWithBadItemDetection(ISourceMailbox mailbox, List`1 messages, GetProxyPoolDelegate getProxyPool, ExportMessagesFlags flags, PropTag[] propsToCopyExplicitly, PropTag[] excludeProps, TestIntegration testIntegration, List`1& badMessages)
at Microsoft.Exchange.MailboxReplicationService.MailboxCopierBase.CopyMessageBatch(List`1 batch, MailboxChanges mailboxChanges, Int32& numberOfUpdates)
at Microsoft.Exchange.MailboxReplicationService.MailboxCopierBase.CopyMessages(List`1 batch)
at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.WriteMessages(Object[] wiParams)
at Microsoft.Exchange.MailboxReplicationService.CommonUtils.CatchKnownExceptions(GenericCallDelegate del, FailureDelegate failureDelegate)
Error context: --------
Operation: IMapiFxProxy.ProcessRequest
OpCode: TransferBuffer
DataLength: 31680
--------
Operation: IMapiFxProxy.ProcessRequest
OperationSide: Target
Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)
OpCode: TransferBuffer
DataLength: 31680
5/5/2012 4:04:34 PM [XSJ-PVEXCAS01] Removing target mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' due to an offline move failure.
5/5/2012 4:49:33 PM [XSJ-PVEXCAS01] The Microsoft Exchange Mailbox Replication service 'XSJ-PVEXCAS01.companyxx.com' (14.1.355.1 caps:07) is examining the request.
5/5/2012 4:49:34 PM [XSJ-PVEXCAS01] Connected to target mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)', database 'XSJ-PSEXMBX02-DB007', Mailbox server 'XSJ-PSEXMBX02.companyxx.com' Version 14.1 (Build 218.0).
5/5/2012 4:49:34 PM [XSJ-PVEXCAS01] Connected to source mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)', database 'XSJ-EXCHVS1\XSJ-SG1-VS1\XSJ-SG1-PRIV3-VS1', Mailbox server 'XSJ-EXCHVS1.companyxx.com' Version 6.0 (Build 7655.0).
5/5/2012 4:50:06 PM [XSJ-PVEXCAS01] Request processing started.
5/5/2012 4:50:06 PM [XSJ-PVEXCAS01] Mailbox signature will not be preserved for mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)'. Outlook clients will need to restart to access the moved mailbox.
5/5/2012 4:50:07 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (1/60).
5/5/2012 4:50:17 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (2/60).
5/5/2012 4:50:28 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (3/60).
5/5/2012 4:50:39 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (4/60).
5/5/2012 4:50:50 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (5/60).
5/5/2012 4:51:01 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (6/60).
5/5/2012 4:51:12 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (7/60).
5/5/2012 4:51:23 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (8/60).
5/5/2012 4:51:34 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (9/60).
5/5/2012 4:51:45 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (10/60).
5/5/2012 4:51:56 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (11/60).
5/5/2012 4:52:07 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (12/60).
5/5/2012 4:52:18 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (13/60).
5/5/2012 4:52:29 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (14/60).
5/5/2012 4:52:40 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (15/60).
5/5/2012 4:52:51 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (16/60).
5/5/2012 4:53:02 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (17/60).
5/5/2012 4:53:13 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (18/60).
5/5/2012 4:53:24 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (19/60).
5/5/2012 4:53:35 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (20/60).
5/5/2012 4:53:46 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (21/60).
5/5/2012 4:53:57 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (22/60).
5/5/2012 4:54:08 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (23/60).
5/5/2012 4:54:19 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (24/60).
5/5/2012 4:54:31 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (25/60).
5/5/2012 4:54:43 PM [XSJ-PVEXCAS01] Mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' tombstone hasn't been cleaned up from the destination database. The operation will try again in 10 seconds (26/60).
5/5/2012 4:55:51 PM [XSJ-PVEXCAS01] The Microsoft Exchange Mailbox Replication service 'XSJ-PVEXCAS01.companyxx.com' (14.1.355.1 caps:07) is examining the request.
5/5/2012 4:55:52 PM [XSJ-PVEXCAS01] Connected to target mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)', database 'XSJ-PSEXMBX02-DB007', Mailbox server 'XSJ-PSEXMBX02.companyxx.com' Version 14.1 (Build 218.0).
5/5/2012 4:55:52 PM [XSJ-PVEXCAS01] Connected to source mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)', database 'XSJ-EXCHVS1\XSJ-SG1-VS1\XSJ-SG1-PRIV3-VS1', Mailbox server 'XSJ-EXCHVS1.companyxx.com' Version 6.0 (Build 7655.0).
5/5/2012 4:56:20 PM [XSJ-PVEXCAS01] Request processing started.
5/5/2012 4:56:20 PM [XSJ-PVEXCAS01] Mailbox signature will not be preserved for mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)'. Outlook clients will need to restart to access the moved mailbox.
5/5/2012 4:56:22 PM [XSJ-PVEXCAS01] Source Mailbox information before the move:
Regular Items: 3490, 346.3 MB (363,172,454 bytes)
Regular Deleted Items: 1, 1.466 KB (1,501 bytes)
FAI Items: 10, 0 B (0 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
5/5/2012 4:56:22 PM [XSJ-PVEXCAS01] Initializing folder hierarchy in mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)': 25 folders total.
5/5/2012 4:56:22 PM [XSJ-PVEXCAS01] Folder hierarchy initialized for mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)': 25 folders total.
5/5/2012 4:56:22 PM [XSJ-PVEXCAS01] Stage: CreatingInitialSyncCheckpoint. Percent complete: 15.
5/5/2012 4:56:23 PM [XSJ-PVEXCAS01] Stage: LoadingMessages. Percent complete: 20.
5/5/2012 4:56:28 PM [XSJ-PVEXCAS01] Stage: CopyingMessages. Percent complete: 25.
5/5/2012 4:56:28 PM [XSJ-PVEXCAS01] Copy progress: 0/3499 messages, 0 B (0 bytes)/346.3 MB (363,172,320 bytes).
5/5/2012 4:56:28 PM [XSJ-PVEXCAS01] Messages have been enumerated successfully. 3499 items loaded. Total size: 346.3 MB (363,172,320 bytes).
5/5/2012 5:01:30 PM [XSJ-PVEXCAS01] Stage: CopyingMessages. Percent complete: 79.
5/5/2012 5:01:30 PM [XSJ-PVEXCAS01] Copy progress: 2267/3499 messages, 272 MB (285,207,440 bytes)/346.3 MB (363,172,320 bytes).
5/5/2012 5:05:03 PM [XSJ-PVEXCAS01] Initial seeding completed, 3499 items copied, total size 346.3 MB (363,172,320 bytes).
5/5/2012 5:05:03 PM [XSJ-PVEXCAS01] Final sync has started.
5/5/2012 5:05:38 PM [XSJ-PVEXCAS01] Stage: FinalIncrementalSync. Percent complete: 95.
5/5/2012 5:05:39 PM [XSJ-PVEXCAS01] Waiting for mailbox changes to replicate.
5/5/2012 5:06:23 PM [XSJ-PVEXCAS01] Mailbox data before finalization:
5/5/2012 5:06:23 PM [XSJ-PVEXCAS01] Move has completed and final clean up has started.
5/5/2012 5:06:23 PM [XSJ-PVEXCAS01] Target mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' was successfully reset after the move.
5/5/2012 5:06:23 PM [XSJ-PVEXCAS01] MBI cache of the target mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' database was successfully seeded after the move. If the target mailbox was on an Exchange 2003 or Exchange 2007 Mailbox server, this operation isn't supported and wasn't performed.
5/5/2012 5:06:25 PM [XSJ-PVEXCAS01] Source mailbox 'Primary (eebfda2e-d6fe-4975-a52b-a39a481e74a2)' was successfully cleaned up after the move.
5/5/2012 5:06:26 PM [XSJ-PVEXCAS01] Target mailbox information after the move:
Regular Items: 3492, 344.1 MB (360,830,005 bytes)
Regular Deleted Items: 0, 0 B (0 bytes)
FAI Items: 10, 0 B (0 bytes)
FAI Deleted Items: 0, 0 B (0 bytes)
5/5/2012 5:06:26 PM [XSJ-PVEXCAS01] Request is complete.
All Replies
-
Monday, May 14, 2012 11:28 PMu r in which roll up?
-
Tuesday, May 15, 2012 6:15 PMThanks for the reply. All Exchange servers are running SP1 RU6
-
Wednesday, May 16, 2012 8:10 AMModerator
Hi,
It seems to something wrong with the delegates of the mailbox or corrupt items in mailbox .
Please test use “BadItemLimit ” parameter.
New-MoveRequest:
http://technet.microsoft.com/en-us/library/dd351123.aspx
Try moving to another mail store before migrating to 2010 SP1.
You can also run ISINTEG on the database to look for any logical corruption.
NOTE:Make a backup of your Exchange databases even if you think the files are damaged and lost before run ISINTEG.
Using the Exchange tools ISINTEG and ESEUTIL to Ensure the Health of your Information Store:
http://www.msexchange.org/tutorials/Exchange-ISINTEG-ESEUTIL.html
Please try to fix DACL use PFDAVadmin tool.
Please check if it has any error log in Event Viewer. Please post the detail information.
Wendy Liu
TechNet Community Support
- Edited by wendy_liuMicrosoft Contingent Staff, Moderator Wednesday, May 16, 2012 8:12 AM
-
Wednesday, May 16, 2012 4:44 PMI am using the badlimit parameter and its set for 50. On the mailboxes that have this issue there are NO corrupt items listed in the log. If you look at the error, it fails continually after it enumerates the message count. Is there a known issue with move requests and orphaned delegates, meaning a source 2003 mailbox with an orphaned delegate (user left the company etc). Can the move request not deal with that properly. The move actually completes successfully, EVENTUALLY. It just takes a lot of partial failures first. If you look at the error order and then the successful move at the bottom there is no difference in what it is trying to do. If it was something wrong enough such as baditemlimit it would fail the move. I have 5000 users to migrate. There has to be a better answer than manually fixing the delegates on 3000 of the 5000 mailboxes

