Answered by:
Secondary Site Repliation Error

Question
-
One of our secondary servers was in transit for a couple of weeks to a remote location. It’s a satellite link, so not very strong. Currently the status in spdiagdrs is as follows:
GlobalData Replication Status
Secondary to P01 – Error
P01 to Secondary – Failed
Site Data Replication Satus
Not Applicable
Rcmctrl.log on Primary:
- ERROR: Replication group "Secondary Site Data" has failed to initialize for subscribing Secondary, setting link state to Error.
Rcmctrl.log on Secondary
- ERROR: Replication group "Secondary Site Data" has failed to initialize for subscribing site Secondary, setting link state to Error.
The Replication link analyser recommends reinitializing the Secondary Site Data. Upon execution the sender on P01 sends about 15mb of data to the Secondary without errors. It takes 2 – 5 minutes for it to finish sending.
In the rcmctrl.log file on the Secondary you can see the reinitializing begin to take place with:
- Building request init for 87 tables in publication Secondary Site Data from site P01.
- ColumnXML output: <TABLE NAME="ActiveDirectoryForestPublishingStatus"><COLUMN COLUMN_ID="1" NAME="FORESTID" TYPENAME="int" TYPEID="56" MAX_LENGTH="4" IS_NULLABLE="0" HAS_DEFAULTVALUE="0" /><COLUMN
(snip)
And then the following errors appear:
(snip)
- Removing old data for global table ... [~~ WHILE EXISTS (SELECT * FROM [PkgPrograms_G])~~ BEGIN~~DELETE TOP(1000) [PkgPrograms_G] ~~ END~~ ] Rowcount from file [E:\ SCCM\inboxes\rcm.box\D57735B4-579B-4D65-911C-88BA7D6512F3\PkgPrograms_G.bcp.rowcount] is [294].
- CSqlBCP::BCPIN: bcp_exec failed.
- *** DRS_Init_BCPIN() failed
- *** BCP fails due to internal sql error. Check if this table has a trigger failed to execute. CBulkInsert::DRS_Init_BCPIN : Failed to BCP in
- BCP in result is 2147500037.
- ERROR: Failed to BCP in for table PkgPrograms_G with error code 2147500037.
- ERROR: Failed to apply BCP for all articles in publication Secondary Site Data.
- Info: Currently Site SECONDARY is initializing Site Control Data. Can't get Master SCF from database at this time. Will try later.
- STATMSG: ID=7806 SEV=E LEV=M SOURCE="SMS Server" COMP="" SYS=Secondary.server.domain SITE=SECONDARY PID=1652 TID=4588 GMTDATE=Sat Nov 03 17:18:29.507 2012 ISTR0="Secondary Site Data" ISTR1="P01" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0
- ERROR: Replication group "Secondary Site Data" has failed to initialize for subscribing site Secondary, setting link state to Error.
The folder (inboxes\rcm.box\ D57735B4-579B-4D65-911C-88BA7D6512F3) on the Secondary remains there until reinitialization, and then disappear once the new folder is created and initialized.
When running RLA again it fails with the following message:
Replication initialization is unable to apply replication package from site P01 to Secondary for the replication groups: Secondary Site Data (Tables: PkgPrograms_G). Replication link analyser is unable to remediate this issue.
Eventually RLA gets to a point where it will recommend to reinitialize and the cycle above starts again. BCP fails on table PkgPrograms_G every time.
Thanks for your time.
- Edited by Johan.R Saturday, November 3, 2012 7:13 PM
Saturday, November 3, 2012 7:09 PM
Answers
-
Update:
A couple of days on the phone with support and the issue have been resolved. Very well done MS Support! :)
SQL Profiler helped to identify the following conflict in the database:
User Error Message Violation of PRIMARY KEY constraint 'PkgPrograms_L_PK'. Cannot insert duplicate key in object 'dbo.PkgPrograms_L'. The duplicate key value is (CAS0013A, Install). BCP-SMS_REPLICATION_CONFIGURATION_MONITOR SYSTEM NT AUTHORITY\SYSTEM 5112 52 2012-11-06 15:18:31.650 5 CM_SEP 4701 NAEPSRV001 0X010100000000000512000000 NT AUTHORITY 0 naepsrv001 NT AUTHORITY\SYSTEM 703561 2627 14 1 223338299393
To remediate this, the support engineer scripted the table to new query and renamed the existing table and primary keys on the secondary site database. We then reinitialized replication to the secondary site, which cleared up the BPC errors we were seeing in the rcmctrl.log file.
So all's well that ends well!
- Edited by Johan.R Wednesday, November 7, 2012 4:57 PM
- Proposed as answer by TorstenMMVP Wednesday, November 7, 2012 6:33 PM
- Marked as answer by Johan.R Wednesday, November 7, 2012 7:13 PM
Wednesday, November 7, 2012 3:48 PM
All replies
-
Weeks?
There is definitely a limit to the amount of time between SQL synchronizations before ConfigMgr considers the data stale and requires a complete re-synch. I don't know what this value is for secondary sites but for primary to CAS the value is 3 days (I think) and so may be the same for secondaries.
Jason | http://blog.configmgrftw.com
Saturday, November 3, 2012 8:17 PM -
Thanks for your reply, Jason. How would I go about doing a complete re-synch?
---
Edit - I've found a useful post by Anoop on how to force replication (I can't post links yet as my account has not been verified).
I will dig around a bit further and provide feedback during the coming week.
SCCM 2012 SP1 will also feature added support for recovering secondary sites (a pity it's still in beta):
Technet - What's New in System Center 2012 SP1
"You can recover a secondary site by using the Recover Secondary Site action from the Sites node in the Configuration Manager console. During the recovery, the secondary site files are installed on the destination computer and then the secondary site data is reinitialized with data from the primary site. The secondary site that you recover must have the same FQDN, meet all secondary site prerequisites, and you must configure appropriate security rights for the secondary site. "
- Edited by Johan.R Sunday, November 4, 2012 10:00 AM
Saturday, November 3, 2012 9:00 PM -
Can you post the sql error? Certain triggers can block the application of BCPs particularly if they cause violate key relationships. The SQL error logs will provide additional insight into what's going on.
Chidozie Nwobilor
Monday, November 5, 2012 6:30 PM -
Hi Dagclo,
I've opened a support call with Microsoft and am working with them to resolve the issue.
Hoping to provide positive feedback soon.
- Edited by Johan.R Monday, November 5, 2012 8:01 PM
Monday, November 5, 2012 7:51 PM -
Update:
A couple of days on the phone with support and the issue have been resolved. Very well done MS Support! :)
SQL Profiler helped to identify the following conflict in the database:
User Error Message Violation of PRIMARY KEY constraint 'PkgPrograms_L_PK'. Cannot insert duplicate key in object 'dbo.PkgPrograms_L'. The duplicate key value is (CAS0013A, Install). BCP-SMS_REPLICATION_CONFIGURATION_MONITOR SYSTEM NT AUTHORITY\SYSTEM 5112 52 2012-11-06 15:18:31.650 5 CM_SEP 4701 NAEPSRV001 0X010100000000000512000000 NT AUTHORITY 0 naepsrv001 NT AUTHORITY\SYSTEM 703561 2627 14 1 223338299393
To remediate this, the support engineer scripted the table to new query and renamed the existing table and primary keys on the secondary site database. We then reinitialized replication to the secondary site, which cleared up the BPC errors we were seeing in the rcmctrl.log file.
So all's well that ends well!
- Edited by Johan.R Wednesday, November 7, 2012 4:57 PM
- Proposed as answer by TorstenMMVP Wednesday, November 7, 2012 6:33 PM
- Marked as answer by Johan.R Wednesday, November 7, 2012 7:13 PM
Wednesday, November 7, 2012 3:48 PM