locked
Site replication does not appear to work RRS feed

  • Question

  • Hi guys, here's my current setup:

    • Primary site (PIT): SCCM 2012 SP1 CU2 on Server 2008 R2 SP1
    • Primary site DB: SQL Server 2008 R2 SP1 CU6 on Server 2008 R2 SP1
    • Secondary site (QTR): Server 2008 R2 SP1

    Both primary and secondary sites appear in my console as "active." The database replication link state has said "link is being configured" for about a day now, with description reading "initializing data between the parent and child site." Parent site state says "replication recovery in progress" while child state is "replication install complete."

    I looked inside the despoolr.box\receive directory on my secondary site and noticed a large (3.3gb) PCK file has been received (I'm thinking this is my OS image?) along with a .sni.ina file, but the despooler has done nothing with it. Despool.log shows the secondary is in maintenance mode:

    Site currently is in maintenance mode and C:\Program Files\Microsoft Configuration Manager\inboxes\despoolr.box\receive\2001cpit.sni is not DRS init or certificate exchange package, backup it until site becomes active.	SMS_DESPOOLER	8/12/2013 1:26:36 PM	3548 (0x0DDC)
    

    From what I've read, this means site replication is still occuring and I should check rcmctrl.log. I've checked this file on the primary site and it shows the same thing over and over again:

    There are 3 Drs Activations sprocs running.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:12 PM	628 (0x0274)
    No connector role installed	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:25 PM	2976 (0x0BA0)
    DRS sync started.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:28 PM	4900 (0x1324)
    InvokeRcmMonitor thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:28 PM	1312 (0x0520)
    InvokeRcmConfigure thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:28 PM	4392 (0x1128)
    Wait for inbox notification timed out.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	1852 (0x073C)
    Cleaning the RCM inbox if there are any *.RCM files for further change notifications....	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	1852 (0x073C)
    Initializing RCM.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	1852 (0x073C)
    Processing Replication Configure	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	4392 (0x1128)
    Running configuration EnsureServiceBrokerEnabled.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	4392 (0x1128)
    Processing Replication Monitor	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	1312 (0x0520)
    Running configuration EnsureServiceBrokerQueuesAreEnabled.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	4392 (0x1128)
    Summarizing all replication links for monitoring UI.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	1312 (0x0520)
    Processing replication pattern global_proxy.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	4392 (0x1128)
    Checking if we need to create an initialization package for replication group Secondary_Site_Replication_Configuration for site QTR.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	4392 (0x1128)
    Tracking guid string is empty, so skipping	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	4392 (0x1128)
    ReplicationConfiguration said it is safe to continue: True	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	4392 (0x1128)
    The current site status: RecoveryInProgress.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:29 PM	1312 (0x0520)
    Processing Replication success.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:30 PM	1312 (0x0520)
    Checking if we need to create an initialization package for replication group Secondary Site Data for site QTR.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:30 PM	4392 (0x1128)
    Tracking guid string is empty, so skipping	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:30 PM	4392 (0x1128)
    Retry for process replication is requested.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:30 PM	4392 (0x1128)
    Rcm control is waiting for file change notification or timeout after 60 seconds.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:30 PM	1852 (0x073C)
    Cleaning the RCM inbox if there are any *.RCM files for further change notifications....	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:30 PM	1852 (0x073C)
    Rcm control is waiting for file change notification or timeout after 60 seconds.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:45:30 PM	1852 (0x073C)
    DRS change application started.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:12 PM	628 (0x0274)
    Launching 3 sprocs on queue ConfigMgrDRSQueue and 0 sprocs on queue ConfigMgrDRSSiteQueue.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:12 PM	628 (0x0274)
    The asynchronous command finished with return message: [spDRSActivation finished at 8/12/2013 1:46:12 PM. End execute query finished at 8/12/2013 1:46:12 PM.].	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:12 PM	628 (0x0274)
    There are 3 Drs Activations sprocs running.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:12 PM	628 (0x0274)
    DRS sync started.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:28 PM	4900 (0x1324)
    InvokeRcmMonitor thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:30 PM	1312 (0x0520)
    InvokeRcmConfigure thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:30 PM	4392 (0x1128)
    Wait for inbox notification timed out.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:30 PM	1852 (0x073C)
    Cleaning the RCM inbox if there are any *.RCM files for further change notifications....	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:30 PM	1852 (0x073C)
    Initializing RCM.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:30 PM	1852 (0x073C)
    Processing Replication Configure	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:30 PM	4392 (0x1128)
    Processing Replication Monitor	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:30 PM	1312 (0x0520)
    Running configuration EnsureServiceBrokerEnabled.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    Running configuration EnsureServiceBrokerQueuesAreEnabled.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    Summarizing all replication links for monitoring UI.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	1312 (0x0520)
    Processing replication pattern global_proxy.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    Checking if we need to create an initialization package for replication group Secondary_Site_Replication_Configuration for site QTR.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    The current site status: RecoveryInProgress.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	1312 (0x0520)
    Processing Replication success.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	1312 (0x0520)
    Tracking guid string is empty, so skipping	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    ReplicationConfiguration said it is safe to continue: True	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    Checking if we need to create an initialization package for replication group Secondary Site Data for site QTR.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    Tracking guid string is empty, so skipping	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    Retry for process replication is requested.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:31 PM	4392 (0x1128)
    Rcm control is waiting for file change notification or timeout after 60 seconds.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:32 PM	1852 (0x073C)
    Cleaning the RCM inbox if there are any *.RCM files for further change notifications....	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:32 PM	1852 (0x073C)
    Rcm control is waiting for file change notification or timeout after 60 seconds.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:46:32 PM	1852 (0x073C)
    DRS change application started.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:47:12 PM	628 (0x0274)
    Launching 3 sprocs on queue ConfigMgrDRSQueue and 0 sprocs on queue ConfigMgrDRSSiteQueue.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:47:13 PM	628 (0x0274)
    The asynchronous command finished with return message: [spDRSActivation finished at 8/12/2013 1:47:12 PM. End execute query finished at 8/12/2013 1:47:12 PM.].	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 1:47:13 PM	628 (0x0274)
    

    This goes on even after running the Replication link analyzer.

    TcpView shows no incoming connections from my primary site server or my site database, so it looks like the secondary site isn't even receiving anything now. At this point I've reached a dead end in my troubleshooting. Can anybody give any advice?

    Thanks.

    Monday, August 12, 2013 5:58 PM

All replies

  • What do rcmctrl.log, despool.log and sender.log on the secondary and sender.log on the primary site tell?

    Torsten Meringer | http://www.mssccmfaq.de

    Monday, August 12, 2013 7:45 PM
  • Despool.log from the secondary site was included in my first post. The log indicates that the site is in "maintenance mode." Everything after that just says "waiting for ready instruction file."

    Sender.log on the primary site seems to indicate that it's waiting for data to send. This is from the primary site:

    Checking for sending capacity.  Used 0 out of 30.	SMS_LAN_SENDER	8/12/2013 2:32:02 PM	2800 (0x0AF0)
    Connecting to C:\Program Files\Microsoft Configuration Manager\inboxes\schedule.box\outboxes\LAN.	SMS_LAN_SENDER	8/12/2013 2:32:02 PM	2800 (0x0AF0)
    COutbox::TakeNextToSend(pszSiteCode)	SMS_LAN_SENDER	8/12/2013 2:32:02 PM	2800 (0x0AF0)
    No (more) send requests found to process.	SMS_LAN_SENDER	8/12/2013 2:32:02 PM	2800 (0x0AF0)
    Waiting for new/rescheduled send requests, Maximum Sleep Time = 60 minutes	SMS_LAN_SENDER	8/12/2013 2:32:02 PM	2800 (0x0AF0)
    We have 0 active connections	SMS_LAN_SENDER	8/12/2013 3:32:02 PM	2800 (0x0AF0)
    Checking for sending capacity.  Used 0 out of 30.	SMS_LAN_SENDER	8/12/2013 3:32:02 PM	2800 (0x0AF0)
    Connecting to C:\Program Files\Microsoft Configuration Manager\inboxes\schedule.box\outboxes\LAN.	SMS_LAN_SENDER	8/12/2013 3:32:02 PM	2800 (0x0AF0)
    COutbox::TakeNextToSend(pszSiteCode)	SMS_LAN_SENDER	8/12/2013 3:32:02 PM	2800 (0x0AF0)
    No (more) send requests found to process.	SMS_LAN_SENDER	8/12/2013 3:32:02 PM	2800 (0x0AF0)
    Waiting for new/rescheduled send requests, Maximum Sleep Time = 60 minutes	SMS_LAN_SENDER	8/12/2013 3:32:02 PM	2800 (0x0AF0)
    SMS_EXECUTIVE signalled SMS_LAN_SENDER to stop.	SMS_LAN_SENDER	8/12/2013 3:47:35 PM	6952 (0x1B28)
    We have 0 active connections	SMS_LAN_SENDER	8/12/2013 3:47:35 PM	2800 (0x0AF0)
    

    And this is sender.log on the secondary site. It's practically empty:

    SMS_EXECUTIVE started SMS_LAN_SENDER as thread ID 2984 (0xBA8).	SMS_LAN_SENDER	8/10/2013 10:16:21 PM	4660 (0x1234)
    Starting MS_LAN sender...	SMS_LAN_SENDER	8/10/2013 10:16:21 PM	2984 (0x0BA8)
    Registry indicates no outbox for the sender to scan.	SMS_LAN_SENDER	8/10/2013 10:16:21 PM	2984 (0x0BA8)
    Registry indicates no outbox for the sender to scan.	SMS_LAN_SENDER	8/10/2013 10:16:21 PM	2984 (0x0BA8)
    Waiting for new/rescheduled send requests, Maximum Sleep Time = 1 minutes	SMS_LAN_SENDER	8/10/2013 10:16:21 PM	2984 (0x0BA8)
    SMS_EXECUTIVE signalled SMS_LAN_SENDER to stop.	SMS_LAN_SENDER	8/10/2013 10:16:38 PM	4660 (0x1234)
    We have 0 active connections	SMS_LAN_SENDER	8/10/2013 10:16:38 PM	2984 (0x0BA8)
    

    rcmctrl.log on the secondary site has been saying the same thing every minute:

    DRS sync started.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:26 PM	616 (0x0268)
    InvokeRcmMonitor thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:54 PM	2820 (0x0B04)
    InvokeRcmConfigure thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:54 PM	2168 (0x0878)
    Wait for inbox notification timed out.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	3684 (0x0E64)
    Cleaning the RCM inbox if there are any *.RCM files for further change notifications....	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	3684 (0x0E64)
    Initializing RCM.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	3684 (0x0E64)
    Processing Replication Configure	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    Processing Replication Monitor	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2820 (0x0B04)
    Summarizing all replication links for monitoring UI.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2820 (0x0B04)
    Running configuration EnsureServiceBrokerEnabled.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    Running configuration EnsureServiceBrokerQueuesAreEnabled.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    Processing replication pattern global_proxy.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    The current site status: ReplicationMaintenance.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2820 (0x0B04)
    Processing Replication success.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2820 (0x0B04)
    Checking if initialization request is needed for replication group Secondary_Site_Replication_Configuration from site PIT.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    Checking if there are bcp file to apply for replication group Secondary_Site_Replication_Configuration from site PIT.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    No tracking data for bcp files	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    ReplicationConfiguration said it is safe to continue: False	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    Retry for process replication is requested.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:55 PM	2168 (0x0878)
    Rcm control is waiting for file change notification or timeout after 60 seconds.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:56 PM	3684 (0x0E64)
    Cleaning the RCM inbox if there are any *.RCM files for further change notifications....	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:56 PM	3684 (0x0E64)
    Rcm control is waiting for file change notification or timeout after 60 seconds.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 3:59:56 PM	3684 (0x0E64)
    DRS change application started.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 4:00:25 PM	3084 (0x0C0C)
    Site is NOT active, so not calling the DrsActivation procedures.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 4:00:25 PM	3084 (0x0C0C)
    There are 0 Drs Activations sprocs running.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/12/2013 4:00:25 PM	3084 (0x0C0C)
    

    The only thing that jumps out there is "Site is NOT active, so not calling the DrsActivation procedures."

    Monday, August 12, 2013 8:02 PM
  • Is "replication recovery in progress" normal? I'm considering reinstalling the primary if this doesn't go away.
    Monday, August 12, 2013 8:28 PM
  • Yesterday I reinstalled the primary site using the same database, and deleted/reinstalled the secondary site. Status looks the same, except rcmctrl.log on the secondary now says this:

    Initializing RCM.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:42:06 AM	3760 (0x0EB0)
    ERROR: Received unhandled exception, printing info and throwing it again.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:42:06 AM	3760 (0x0EB0)
    ERROR: Microsoft.ConfigurationManager.ReplicationConfigurationAndMonitoring.LocalServerDataNotFoundException: Unable to find SiteCode or SiteNumber~~   at Microsoft.ConfigurationManager.ReplicationConfigurationAndMonitoring.SiteData.Refresh()~~   at Microsoft.ConfigurationManager.ReplicationConfigurationAndMonitoring.ReplicationConfigureAndMonitor.DoReplicationChannelsExist()	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:42:06 AM	3760 (0x0EB0)
    Rcm control is waiting for file change notification or timeout after 60 seconds.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:42:07 AM	3760 (0x0EB0)
    InvokeRcmMonitor thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:42:25 AM	2900 (0x0B54)
    DRS sync started.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:42:44 AM	4700 (0x125C)
    InvokeRcmConfigure thread wait one more minute for incoming event...	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:43:01 AM	4012 (0x0FAC)
    Wait for inbox notification timed out.	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:43:07 AM	3760 (0x0EB0)
    Cleaning the RCM inbox if there are any *.RCM files for further change notifications....	SMS_REPLICATION_CONFIGURATION_MONITOR	8/13/2013 9:43:07 AM	3760 (0x0EB0)

    I ran SELECT dbo.fnGetSiteCode() on secondary database and get the correct site code.

    • Edited by _nickb Tuesday, August 13, 2013 1:45 PM
    Tuesday, August 13, 2013 1:44 PM
  • I discovered that the outbound connections on my primary site's database cluster nodes were being blocked by the network firewall. Once those were opened, I noticed the following SQL Broker error:

    Message

    Service Broker login attempt failed with error: 'A previously existing connection with the same peer was detected during connection handshake. This connection lost the arbitration and it will be closed. All traffic will be redirected to the previously existing connection. This is an informational message only. No user action is required. State 80.'. 

    Tuesday, August 13, 2013 11:00 PM
  • Did you receive a response yet on your last post?
    Wednesday, September 11, 2013 7:56 PM
  • Hi YoungOne,

    Haven't gotten the secondary site to work yet, and due to time constraints I have temporarily opted to use a remote distribution point for package delivery. In the future I plan to revisit the secondary site configuration.

    Wednesday, September 11, 2013 8:01 PM