locked
Secondary Site Repliation Error RRS feed

  • Question

  • One of our secondary servers was in transit for a couple of weeks to a remote location.  It’s a satellite link, so not very strong.  Currently the status in spdiagdrs is as follows:

    GlobalData Replication Status

    Secondary to P01 – Error

    P01 to Secondary – Failed

    Site Data Replication Satus

    Not Applicable

    Rcmctrl.log on Primary:

    • ERROR: Replication group "Secondary Site Data" has failed to initialize for subscribing Secondary, setting link state to Error.

    Rcmctrl.log on Secondary    

    • ERROR: Replication group "Secondary Site Data" has failed to initialize for subscribing site Secondary, setting link state to Error.

    The Replication link analyser recommends reinitializing the Secondary Site Data.  Upon execution the sender on P01 sends about 15mb of data to the Secondary without errors.  It takes 2 – 5 minutes for it to finish sending.

    In the rcmctrl.log file on the Secondary you can see the reinitializing begin to take place with:

    • Building request init for 87 tables in publication Secondary Site Data from site P01.
    • ColumnXML output: <TABLE NAME="ActiveDirectoryForestPublishingStatus"><COLUMN COLUMN_ID="1" NAME="FORESTID" TYPENAME="int" TYPEID="56" MAX_LENGTH="4" IS_NULLABLE="0" HAS_DEFAULTVALUE="0" /><COLUMN

    (snip)

    And then the following errors appear:

    (snip)

    • Removing old data for global table ... [~~                      WHILE EXISTS (SELECT * FROM [PkgPrograms_G])~~                      BEGIN~~DELETE TOP(1000) [PkgPrograms_G] ~~               END~~ ]     Rowcount from file [E:\ SCCM\inboxes\rcm.box\D57735B4-579B-4D65-911C-88BA7D6512F3\PkgPrograms_G.bcp.rowcount] is [294].                                    
    • CSqlBCP::BCPIN: bcp_exec failed.                                           
    • *** DRS_Init_BCPIN() failed                                      
    • *** BCP fails due to internal sql error. Check if this table has a trigger failed to execute. CBulkInsert::DRS_Init_BCPIN : Failed to BCP in                               
    • BCP in result is 2147500037.                                        
    • ERROR: Failed to BCP in for table PkgPrograms_G with error code 2147500037.                                  
    • ERROR: Failed to apply BCP for all articles in publication Secondary Site Data.                                      
    • Info: Currently Site SECONDARY is initializing Site Control Data. Can't get Master SCF from database at this time. Will try later.                                    
    • STATMSG: ID=7806 SEV=E LEV=M SOURCE="SMS Server" COMP="" SYS=Secondary.server.domain SITE=SECONDARY PID=1652 TID=4588 GMTDATE=Sat Nov 03 17:18:29.507 2012 ISTR0="Secondary Site Data" ISTR1="P01" ISTR2="" ISTR3="" ISTR4="" ISTR5="" ISTR6="" ISTR7="" ISTR8="" ISTR9="" NUMATTRS=0                                  
    • ERROR: Replication group "Secondary Site Data" has failed to initialize for subscribing site Secondary, setting link state to Error.

    The folder (inboxes\rcm.box\ D57735B4-579B-4D65-911C-88BA7D6512F3) on the Secondary remains there until reinitialization, and then disappear once the new folder is created and initialized.

    When running RLA again it fails with the following message:

    Replication initialization is unable to apply replication package from site P01 to Secondary for the replication groups:  Secondary Site Data (Tables:  PkgPrograms_G).  Replication link analyser is unable to remediate this issue.

    Eventually RLA gets to a point where it will recommend to reinitialize and the cycle above starts again.  BCP fails on table PkgPrograms_G every time.

    Thanks for your time.





    • Edited by Johan.R Saturday, November 3, 2012 7:13 PM
    Saturday, November 3, 2012 7:09 PM

Answers

  • Update:

    A couple of days on the phone with support and the issue have been resolved. Very well done MS Support! :)

    SQL Profiler helped to identify the following conflict in the database:

    User Error Message  Violation of PRIMARY KEY constraint 'PkgPrograms_L_PK'. Cannot insert duplicate key in object 'dbo.PkgPrograms_L'. The duplicate key value is (CAS0013A, Install).            BCP-SMS_REPLICATION_CONFIGURATION_MONITOR     SYSTEM          NT AUTHORITY\SYSTEM                                                  5112    52        2012-11-06 15:18:31.650                                5          CM_SEP         4701            NAEPSRV001                       0X010100000000000512000000        NT AUTHORITY       0            naepsrv001     NT AUTHORITY\SYSTEM  703561            2627                                                                14            1                      223338299393

    To remediate this, the support engineer scripted the table to new query and renamed the existing table and primary keys on the secondary site database.  We then reinitialized replication to the secondary site, which cleared up the BPC errors we were seeing in the rcmctrl.log file.

    So all's well that ends well!


    • Edited by Johan.R Wednesday, November 7, 2012 4:57 PM
    • Proposed as answer by TorstenMMVP Wednesday, November 7, 2012 6:33 PM
    • Marked as answer by Johan.R Wednesday, November 7, 2012 7:13 PM
    Wednesday, November 7, 2012 3:48 PM

All replies

  • Weeks?

    There is definitely a limit to the amount of time between SQL synchronizations before ConfigMgr considers the data stale and requires a complete re-synch. I don't know what this value is for secondary sites but for primary to CAS the value is 3 days (I think) and so may be the same for secondaries.


    Jason | http://blog.configmgrftw.com

    Saturday, November 3, 2012 8:17 PM
  • Thanks for your reply, Jason.  How would I go about doing a complete re-synch?

    ---

    Edit - I've found a useful post by Anoop on how to force replication (I can't post links yet as my account has not been verified).

    I will dig around a bit further and provide feedback during the coming week.

    SCCM 2012 SP1 will also feature added support for recovering secondary sites (a pity it's still in beta): 

    Technet - What's New in System Center 2012 SP1

    "You can recover a secondary site by using the Recover Secondary Site action from the Sites node in the Configuration Manager console. During the recovery, the secondary site files are installed on the destination computer and then the secondary site data is reinitialized with data from the primary site. The secondary site that you recover must have the same FQDN, meet all secondary site prerequisites, and you must configure appropriate security rights for the secondary site. "



    • Edited by Johan.R Sunday, November 4, 2012 10:00 AM
    Saturday, November 3, 2012 9:00 PM
  • Can you post the sql error?  Certain triggers can block the application of BCPs particularly if they cause violate key relationships.  The SQL error logs will provide additional insight into what's going on.

    Chidozie Nwobilor

    Monday, November 5, 2012 6:30 PM
  • Hi Dagclo,

    I've opened a support call with Microsoft and am working with them to resolve the issue.

    Hoping to provide positive feedback soon.




    • Edited by Johan.R Monday, November 5, 2012 8:01 PM
    Monday, November 5, 2012 7:51 PM
  • Update:

    A couple of days on the phone with support and the issue have been resolved. Very well done MS Support! :)

    SQL Profiler helped to identify the following conflict in the database:

    User Error Message  Violation of PRIMARY KEY constraint 'PkgPrograms_L_PK'. Cannot insert duplicate key in object 'dbo.PkgPrograms_L'. The duplicate key value is (CAS0013A, Install).            BCP-SMS_REPLICATION_CONFIGURATION_MONITOR     SYSTEM          NT AUTHORITY\SYSTEM                                                  5112    52        2012-11-06 15:18:31.650                                5          CM_SEP         4701            NAEPSRV001                       0X010100000000000512000000        NT AUTHORITY       0            naepsrv001     NT AUTHORITY\SYSTEM  703561            2627                                                                14            1                      223338299393

    To remediate this, the support engineer scripted the table to new query and renamed the existing table and primary keys on the secondary site database.  We then reinitialized replication to the secondary site, which cleared up the BPC errors we were seeing in the rcmctrl.log file.

    So all's well that ends well!


    • Edited by Johan.R Wednesday, November 7, 2012 4:57 PM
    • Proposed as answer by TorstenMMVP Wednesday, November 7, 2012 6:33 PM
    • Marked as answer by Johan.R Wednesday, November 7, 2012 7:13 PM
    Wednesday, November 7, 2012 3:48 PM