locked
Lync 2013 Edge Not replicating after moving CMS store RRS feed

  • Question

  • Hi, we have 3 datacenters and each datacenter has it front-end farm, with a regional Edge server.

    We have moved the Central Management Store to the SkypeForBusiness Pool. After the pool we saw that two Edge Servers (out of three) are failing to report uptodate in the Get-csmanagementStoreReplicationStatus:

    

    When i check the eventviewer on the Edge servers, i can see that the Edge is reporting that is using the latest version:

    Microsoft Lync Server 2013, Replica Replicator Agent reported the latest replica status.

    Status report reason: ServerStartup

    The firewall ports are open, as there was no replication issue before we moved the CMS. Nevertheless have checked with telnet and am able to browse to https://edgeserverfqdn:4443/ReplicationWebService. The certificate checkes out and have added registry settings, discussed here https://jaapwesselius.com/2014/07/11/lync-2013-replication-not-working/.

    Have also recreated the Replica folder (RtcReplicaRoot) on the Edge server, but still no go.

    https://digitalbamboo.wordpress.com/2017/06/22/edge-replication-status-is-false-and-the-last-update-creation-time-stops-updating-for-command-get-csmanagementstorereplicationstatus/

    However still no go. It seems to be like a split brain, where the edge is uptodate, but the Front-end pool failes to retrieve the current status.

    Does anyone have any idea's?


    Answers provided are coming from personal experience, and come with no warranty of success. I as everybody else do make mistakes.

    Wednesday, December 11, 2019 3:53 PM

Answers

  • Hi jimmyy, first of all thanks for your help.

    PLease review my answers below your tips.

    Hi Killerbe!

    In my research, you could try to solve the issue in these ways:

    1)Check you SQL server FQDN in topology builder. If you specify the fqdn in topology builder but the computer name is just edgeservername without the domain name you will have an issue.

    We have used FQDN's in the topology builder.

    2)Run the command “get-CsManagementConnection” and “get-CsService -CentralManagement” to see if it shows destination pool.

    Yes it does.

    3)Run the command “invoke-CsManagementStoreReplication -ReplicaFQDN <servername>” then wait a few minutes to see if status changes. For more details, you can refer to:

    http://www.gecko-studio.co.uk/troubleshooting-lync-edge-server-replication/?LMCL=Jw95Uc

    If the above ways cannot solve this issue, we recommend you use Lync Server 2013 Debugging Tools to trace the logs. For more details about use this tool to troubleshoot the CMS replication status on Edge Server, you can refer to:

    http://pipe2text.com/?page_id=2777

    As this issue happened after moving CMS store, you can also check your process of moving CMS store. The following article for you referring:

    https://blogs.technet.microsoft.com/uctalks/2016/05/09/moving-the-cms-make-it-part-of-your-upgrade-action-plan-part-1/

    Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make sure that you completely understand the risk before retrieving any suggestions from the above link.

    We have used the OCSLogger, and no errors where found, we did find warnings:

    TL_WARN(TF_COMPONENT) [3]0F60.367C::12/12/2019-09:54:02.849.1534f8b5 (XDS_Replica_Replicator,FileDistributeTask.Execute:filedistributetask.cs(165))
    (0000000000ECE0F8)Could not distribute the file. Exception: [System.IO.IOException: The process cannot access the file because it is being used by another process.

    then i see 

    C:\RtcReplicaRoot\xds-replica\from-master access failed

    however three events later i see

    C:\RtcReplicaRoot\xds-replica\from-master is processed successfuly.

    So i expect that this is expected behaviour?

    Best Regards,
    Jimmy Yang


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.



    Answers provided are coming from personal experience, and come with no warranty of success. I as everybody else do make mistakes.


    • Edited by Killerbe Monday, December 16, 2019 1:33 PM Bold
    • Marked as answer by Killerbe Monday, December 23, 2019 10:29 AM
    Monday, December 16, 2019 1:32 PM

All replies

  • Hi Killerbe!

    In my research, you could try to solve the issue in these ways:

    1)Check you SQL server FQDN in topology builder. If you specify the fqdn in topology builder but the computer name is just edgeservername without the domain name you will have an issue.

    2)Run the command “get-CsManagementConnection” and “get-CsService -CentralManagement” to see if it shows destination pool.

    3)Run the command “invoke-CsManagementStoreReplication -ReplicaFQDN <servername>” then wait a few minutes to see if status changes. For more details, you can refer to:

    http://www.gecko-studio.co.uk/troubleshooting-lync-edge-server-replication/?LMCL=Jw95Uc

    If the above ways cannot solve this issue, we recommend you use Lync Server 2013 Debugging Tools to trace the logs. For more details about use this tool to troubleshoot the CMS replication status on Edge Server, you can refer to:

    http://pipe2text.com/?page_id=2777

    As this issue happened after moving CMS store, you can also check your process of moving CMS store. The following article for you referring:

    https://blogs.technet.microsoft.com/uctalks/2016/05/09/moving-the-cms-make-it-part-of-your-upgrade-action-plan-part-1/

    Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make sure that you completely understand the risk before retrieving any suggestions from the above link.

    Best Regards,
    Jimmy Yang

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.
    Thursday, December 12, 2019 7:39 AM
  • Hi jimmyy, first of all thanks for your help.

    PLease review my answers below your tips.

    Hi Killerbe!

    In my research, you could try to solve the issue in these ways:

    1)Check you SQL server FQDN in topology builder. If you specify the fqdn in topology builder but the computer name is just edgeservername without the domain name you will have an issue.

    We have used FQDN's in the topology builder.

    2)Run the command “get-CsManagementConnection” and “get-CsService -CentralManagement” to see if it shows destination pool.

    Yes it does.

    3)Run the command “invoke-CsManagementStoreReplication -ReplicaFQDN <servername>” then wait a few minutes to see if status changes. For more details, you can refer to:

    http://www.gecko-studio.co.uk/troubleshooting-lync-edge-server-replication/?LMCL=Jw95Uc

    If the above ways cannot solve this issue, we recommend you use Lync Server 2013 Debugging Tools to trace the logs. For more details about use this tool to troubleshoot the CMS replication status on Edge Server, you can refer to:

    http://pipe2text.com/?page_id=2777

    As this issue happened after moving CMS store, you can also check your process of moving CMS store. The following article for you referring:

    https://blogs.technet.microsoft.com/uctalks/2016/05/09/moving-the-cms-make-it-part-of-your-upgrade-action-plan-part-1/

    Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make sure that you completely understand the risk before retrieving any suggestions from the above link.

    We have used the OCSLogger, and no errors where found, we did find warnings:

    TL_WARN(TF_COMPONENT) [3]0F60.367C::12/12/2019-09:54:02.849.1534f8b5 (XDS_Replica_Replicator,FileDistributeTask.Execute:filedistributetask.cs(165))
    (0000000000ECE0F8)Could not distribute the file. Exception: [System.IO.IOException: The process cannot access the file because it is being used by another process.

    then i see 

    C:\RtcReplicaRoot\xds-replica\from-master access failed

    however three events later i see

    C:\RtcReplicaRoot\xds-replica\from-master is processed successfuly.

    So i expect that this is expected behaviour?

    Best Regards,
    Jimmy Yang


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.



    Answers provided are coming from personal experience, and come with no warranty of success. I as everybody else do make mistakes.


    • Edited by Killerbe Monday, December 16, 2019 1:33 PM Bold
    • Marked as answer by Killerbe Monday, December 23, 2019 10:29 AM
    Monday, December 16, 2019 1:32 PM
  • Hi Killerbe!

    According to your description, it seems that FTA service cannot complete the Kerberos authentication process that is required to securely transfer a copy of the CMS replica information to one or more the Lync Server role servers.

    We recommend you check if the xds-replica folder that is located on the remote Lync Server is missing the default Access Control Entries(ACE) from its Access Control List(ACL).

    You could try to solve the problem in these ways:

    1)In the Local Security Policy > Local Policies > User Rights Assignment > Access this computer from the network>local security Setting, see if there is only Administrator Group.

    2)add the normal groups back


    You can refer to the following article to know more details:

    https://blogs.technet.microsoft.com/dodeitte/2017/03/06/issue-with-cms-replication-in-a-hardened-environment/

    For more detail about CMS replication process, please refer to:

    https://blogs.technet.microsoft.com/jenstr/2010/10/13/what-is-central-management-store-cms/

    Best Regards,
    Jimmy Yang

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.
    Thursday, December 19, 2019 9:39 AM
  • Got it resolved. I was thinking that the edge was updating the CMS by contacting the local front-end pool and not the front-end pool which holds the CMS master. So the failing edges were able to comunicate with their associated Front-end pool on port 4443, but not with the front-end pool holding the CMS master. We updated the firewall rules in which all servers are able to communicate (TCP4443) with the Front-end pool holding the CMS master, and the issue was resolved.

    Answers provided are coming from personal experience, and come with no warranty of success. I as everybody else do make mistakes.

    Monday, December 23, 2019 10:29 AM
  • Here I will provide a brief and temporary summary of this post:

    <Issue Symptom>:

    Environment: 3 datacenters and each datacenter has it front-end farm, with a regional Edge Server.

    Operation before issue happen: Move the Central management store to the Skype for Business Pool

    Two Edge Servers(out of three) are failing to report update in the Get-CsManagementStoreReplicationStatus.

    <Possible cause>:

    1.Check the SQL Server FQDN in topology builder.

    2. Destination pool configuration is not right.

    3.FTA Service cannot complete the Kerberos authentication

    4.Firewall probelm

    <Solution>:

    1.Run the command "get-CsManagementConnection" and "get-CsService -CentralManagement" to see if it shows destination pool.

    2.Run the command "invoke-CsManagementStoreReplication -ReplicaFQDN <servername>" then wait for a few minutes to see if status changes

    3.Use Lync Server 2013 Debugging Tools to trace the logs.

    4.check if the xds-replica folder that is located on the remote Lync Server is missing the default ACE from its ACL.

    5.Update the firewall rules in which all servers are able to communicate(TCP4443) with the Front-End pool holding the CMS master.

    <Reference Links>:

    http://www.gecko-studio.co.uk/troubleshooting-lync-edge-server-replication/?LMCL=Jw95Uc

    http://pipe2text.com/?page_id=2777

    https://blogs.technet.microsoft.com/uctalks/2016/05/09/moving-the-cms-make-it-part-of-your-upgrade-action-plan-part-1/

    https://blogs.technet.microsoft.com/dodeitte/2017/03/06/issue-with-cms-replication-in-a-hardened-environment/

    https://blogs.technet.microsoft.com/jenstr/2010/10/13/what-is-central-management-store-cms/

    Note: Microsoft is providing this information as a convenience to you. The sites are not controlled by Microsoft. Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. Please make sure that you completely understand the risk before retrieving any suggestions from the above link.

    Best Regards,
    Jimmy Yang

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnsf@microsoft.com.
    Thursday, December 26, 2019 6:26 AM