locked
CCE HA unit > cmdlet Publish-CcAppliance error at Backup-RootCA function script RRS feed

  • Question

  • In my CCE HA deployment, I've successfully brought up the first CCE appliance. However, during the installation of 2nd appliance, when running the step "Publish-CcAppliance" on the first CCE appliance, I got the following error:

    PS C:\Users\Administrator> Publish-CcAppliance
    The PublishInstance log is in C:\CCE\appliance\Log\<sitename>_CCE1_PublishInstance+08_00_2017_09_11_13_04_28.log.
    Task:PublishInstance starts at 2017-09-11T13:04:28.2204182+08:00
    The current scripts version is 2.0.1.
    The version of current running instance is 2.0.1.
    Trying to enter manual maintenance mode.
    Successfully enter manual maintenance mode.
    Opening online connection...
    Online connection established.
    WARNING: Network configuration has not been set for this tenant.
    Remove online connection.
    The AD domain controller configuration is created in memory.
    Trying to exit current manual maintenance mode.
    Successfully exit manual maintenance mode.
    Wait-MtMachineOn : Can't connect to machine <local_CCE_DC_IP_address> after waiting 61 seconds.
    At C:\Program Files\WindowsPowerShell\Modules\CloudConnector\Internal\MtHostCommon.ps1:477 char:5
    +     Wait-MtMachineOn -MachineIP $adIPAddress -Credential $domainAdmin ...
    +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Wait-MtMachineOn

    Wait-MtMachineOn : Can't connect to machine <local_CCE_DC_IP_address> after waiting 61 seconds.
    At C:\Program Files\WindowsPowerShell\Modules\CloudConnector\Internal\MtHostCommon.ps1:477 char:5
    +     Wait-MtMachineOn -MachineIP $adIPAddress -Credential $domainAdmin ...
    +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
        + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Wait-MtMachineOn

    PS C:\Users\Administrator>

    I can confirm that:
    Site Directory = same shared folder 
    SIP Domains = only single SIP domain name and it is the same 
    Site Name = same site name 
    Edge Certificate = same certificate
    VM admin password = same password
    VM IP address = different on all VMs
    Edge External Pool FQDN = same on both CCE appliances
    External IP address Edge = different on both Edge servers
    PSTN GW settings = same on both CCE appliances
    DNS records = same External Edge FQDN with different public IP addresses
    SIP domain name & VM domain (cce.local) do not contain same word

    CCEservice, local domain admin password = same

    CCE version 2.0.1

    CCE Appliances = AudioCodes Mediant 800B CCE

    Could you please advise what went wrong related to the error above?

    Thank you.


    • Edited by Nurmawan Tuesday, September 12, 2017 4:49 AM fine tune problem title
    Monday, September 11, 2017 8:37 AM

All replies

  • Further troubleshooting the above, on CCE1 host, I can ping the local_CCE_DC_IP_address:

    PS C:\Users\Administrator> ping <local_CCE_DC_IP_address>

    Pinging <local_CCE_DC_IP_address> with 32 bytes of data:
    Reply from <local_CCE_DC_IP_address>: bytes=32 time<1ms TTL=128
    Reply from <local_CCE_DC_IP_address>: bytes=32 time<1ms TTL=128
    Reply from <local_CCE_DC_IP_address>: bytes=32 time<1ms TTL=128
    Reply from <local_CCE_DC_IP_address>: bytes=32 time<1ms TTL=128

    Ping statistics for <local_CCE_DC_IP_address>:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 0ms, Maximum = 0ms, Average = 0ms
    PS C:\Users\Administrator>

    Checking on MtHostCommon.ps1 line 477, revealed:

        # When upgrading from 1.4.1 or 1.4.2 to V2, CA need be backup from current running version AD server.
        # But, at this time, AD server may NOT ever be ping or reached. The first ping or invoke command call may failed for network connection problem.
        # Check and wait AD server is on and network is reday. Base on our test expirience, first ping and invoke will fail, the second try will success.
        # In V2, we already restart server and wait the machine on in switch-ccversion logic. So it is not a problem after V2.
        Wait-MtMachineOn -MachineIP $adIPAddress -Credential $domainAdminCredential -WaitTimeoutInSeconds 60

    At this point, I'm not very sure what went wrong. Could anyone enlighten me about the error when executing Publish-CcAppliance cmdlet, please?

    Thank you.

    Monday, September 11, 2017 10:40 AM
  • More info on the part from MtHostCommon.ps1. The error is part of Backup-RootCA function.

    function Backup-RootCA {
        param(
            [hashtable] $config
        )

        # Export root CA on AD and store it in SiteRoot\CA folder
        $addcConfig = Export-MtAdDomainControllerConfig $config -InMemory;
        $adIPAddress = $addcConfig.AdCorpnetIPAddress;
        $domainAdminCredential = Get-DomainAdminCredential $config;
        $caBackupFileCredential = Get-MtCredential -Role $CredentialRoles.CABackupFile -FilePath $config.CredentialFilePath;
        $vmSharedFolder = $addcConfig.VmSharedFolder;

        # When upgrading from 1.4.1 or 1.4.2 to V2, CA need be backup from current running version AD server.
        # But, at this time, AD server may NOT ever be ping or reached. The first ping or invoke command call may failed for network connection problem.
        # Check and wait AD server is on and network is reday. Base on our test expirience, first ping and invoke will fail, the second try will success.
        # In V2, we already restart server and wait the machine on in switch-ccversion logic. So it is not a problem after V2.
        Wait-MtMachineOn -MachineIP $adIPAddress -Credential $domainAdminCredential -WaitTimeoutInSeconds 60

        $shareName = "\\$adIPAddress\$vmSharedFolder";
        New-MtiVmFileShare $adIPAddress $domainAdminCredential $vmSharedFolder

        $backupCAString = Get-MtMessage -FormatKey "InternalCommon_Information_CopyBackupCA"
        Invoke-command -ComputerName $adIPAddress -Credential $domainAdminCredential -ScriptBlock {
            param (
                [string] $vmSharedFolder,
                [string] $password
            )
            try {
                Backup-CARoleService -Path "$($env:SystemDrive)\$($vmSharedFolder)\" -Password (ConvertTo-SecureString $password -AsPlainText -Force) -Force
            } catch {
                Write-Host $using:backupCAString
            }
        } -Argumentlist @($vmSharedFolder, (Get-MtPlainText $caBackupFileCredential.Password)) -ErrorAction Stop;

        $sharedCaDirectory = Get-InccSharedCaDirectory;
        $caBackupFileName = Get-InccSharedCaFileName;
        $internalCertInSharedFolder = Join-Path $sharedCaDirectory $caBackupFileName;

        $vmCopyDriveName = "BackupDrive";
        New-MtPSDrive -DriveName $vmCopyDriveName -ServerName $adIPAddress -ShareRoot $shareName -Credential $domainAdminCredential;

        $caBackupFileFullName = Join-Path ($vmCopyDriveName + ":") $caBackupFileName;
        Write-Host (Get-MtMessage -FormatKey "InternalCommon_Information_CopyingFileToFolder" -Values @($caBackupFileFullName, $sharedCaDirectory))
        Copy-MtItem -Source $caBackupFileFullName -Destination $internalCertInSharedFolder;

        Remove-MtPSDrive -DriveName $vmCopyDriveName -ServerName $adIPAddress;

        Remove-MtiVmFileShare $adIPAddress $domainAdminCredential $vmSharedFolder
    }

    Could anyone help, please?

    Tuesday, September 12, 2017 4:51 AM
  • Hi Nurmawan,

     

    1.Confirm no VLAN’s configured on the Host appliance.

    2.If using a Proxy server ,confirm WinHTTP  proxy settings bypass list.

    3.Check if the switches on the Cloud Connector VM have Automatic Private IP Addresses of 169.x. It they have 169.x addresses, the Convert-CcIsoToVhdx did not complete correctly. To resolve:

    Run:

    Convert-CcIsoToVhdx -IsoFilePath <Windows ISO File Path, including file name> -PauseBeforeUpdate

    When prompted, connect to the Base VM using credentials provided in the PowerShell output.

    Make any necessary adjustments to allow the computer to connect to Windows Update Service.

    Complete all Windows Updates and restart the VM.

    Return to PowerShell and answer prompt complete the process.

     

    4.Check NetBIOS name of the Hyper-V server hosting CCE and NetBIOS name of the CCE is different, there is similar case, you could have a reference

    https://social.technet.microsoft.com/Forums/lync/en-US/cefe9673-f434-4132-a677-c6ddce9ac43c/error-execute-installccappliance?forum=sfbfr


    Best Regards,

    Leon-Lu
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, September 12, 2017 7:04 AM
  • Hello Leon,

    Thanks for the pointers. Appreciate your efforts & time.

    1) The CCE appliance is an AudioCodes Mediant 800B. There is internal private VLAN 4021 configured on it. I did try to disable it and ran Publish-CcAppliance again but to no avail.

    Is there anything else I can do beside disable it?

    2) Confirmed, my setup does not use a Proxy server

    3) The switches on Cloud Connector VM confirmed do not have Automatic Private IP Address.

    4) SIP domain is <domain_name>.com.sg
    CCE domain is cce.local

    Hyper-V server hosting CCE doesn't join any domain.

    Tuesday, September 12, 2017 8:06 AM
  • Hi Nurmawan,

    Check the Mediation Server FQDN and IP addresses of the PSTN site,and new DNS a recods were added to the AD Server for Mediation server IP address.

    If you have done all troubleshooting steps,please check the steps of configure SFB CCE,you can refer to this link.

    https://technet.microsoft.com/en-us/library/mt605228.aspx


    Best Regards,

    Leon-Lu
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Monday, September 18, 2017 10:17 AM
  • Hello Leon,

    Yes, I have re-checked the Mediation Server FQDN and IP addresses as well as the DNS "A" records. I've also checked the steps of configuring SFB CCE on the given link.

    When the error stop at CCE script MtHostCommon.ps1, instead of checking 'outside' CCE, what is 'inside' CCE that we should check?

    When the error pointed to timeout reaching local CCE AD server, who/what component/service is trying to access the local CCE AD server at MtHostCommon.ps1 line 477?

    For the fact that network connectivity / reachability to CCE AD server is not an issue, what could possibly cause failure in accessing local CCE AD server? is it the credential? is it the service account? is it the local policy setting? is it the group policy setting?

    For your info, the password is different between CceService and CABackupFile service accounts, but CceService service account password is the same on CCE1 and CCE2. Likewise CABackupFile service account password is the same on CCE1 and CCE2. Are these password an area of concerns?

    What else could contribute to the error at MtHostCommon.ps1 line 477?

    I did further troubleshooting and below is the finding:

    MediationServerDnsRecords = returns no value 

    AdIPAddress = return the private address 192.168.213.x

    CorpnetDNSIPAddress = return customer internal DNS servers IP address

    ManagementIPPrefixLength = 24

    VmSharedFolder = CloudConnector  

    PoolDnsRecords = return 'edgepool' with corpnet IP address and 'mspool' with corpnet IP address

    AdServerName = ADA03407                                                          

    CredentialFilePath             C:\ProgramData\CloudConnector\credentials..Administrator.xml      

    AdCorpnetIPAddress = return local CCE AD corpnet IP address

    CertificateFolderName          Certificates                                                       

    EdgeMachineDnsRecords = return edge hostname and its corpnet IP address

    DomainName = cce.local

    GatewayDnsRecords = return GW1 and GW2 FQDN and their IP addresses

    ManagementIPPrefix = 192.168.213.0

    CaCommonName = SfB CCE Root                       

    UserName : cce.local\Administrator

    Password : System.Security.SecureString

    UserName : Administrator

    Password : System.Security.SecureString

    What causes the MediationServerDnsRecords returns empty value?

    Could you give more pointers on what to check based on the above questions & info?

    Many thanks.

    Thursday, September 21, 2017 8:24 AM
  • Hi Nurmawan,

    I noticed you have deployed two CCE appliance for HA deployment, please review these items for “Single-Site with HA” in the following link: (According your mentioned, you have checked most of these, please also check the left.)

    https://technet.microsoft.com/en-us/library/mt740650.aspx

     

    If there is no problem, please check the CCE status from Skype for Business Online Admin Portal, and check the DNS records for both the CCE appliances in Online portal.

    This message “MediationServerDnsRecords = returns no value” seems it cannot get the related DNS records. 

     

    And for the first error message, it is related to this command “Wait-MtMachineOn -MachineIP $adIPAddress -Credential $domainAdminCredential -WaitTimeoutInSeconds 60”, according your above information, please check this folder “..\CloudConnector\SiteRoot\CA” to see if the Root CA is exported under it.

     

    Thanks for your understanding and patience!


    Best Regards,

    Leon-Lu
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Tuesday, September 26, 2017 9:48 AM
  • Are there any update for this issue, if the reply is helpful to you, please try to mark it as an answer, it will help others who has similar issue.

    Best Regards,

    Leon-Lu
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Wednesday, September 27, 2017 9:05 AM
  • hi Leon,

    Yes, I did check https://technet.microsoft.com/en-us/library/mt740650.aspx?f=255&MSPPError=-2147217396

    I also checked the CCE status on Skype for Business Online Admin Portal and it said CCE1 was running (registeredanddeployed), CCE2 was None (registeredandfailed)

    I checked the DNS records, the "A" record was and is still valid pointing to both CCE1 and CCE2

    i checked the folder “..\CloudConnector\SiteRoot\CA” , and the Root CA was not exported because that was where the script stopped.

    In the end, I rebuilt the CCE1 and CCE2,  unregistered both, uninstalled both, and redo the installation from scratch. The difference in my 2nd rebuilt setup was I used the SFB admin account with onmicrosoft.com domain instead. Initially was using SFB admin with SIP domain.

    I don't think that was the root cause of my issue. Maybe there was other 'things' that failed my setup and was reset when i rebuilt it again.

    Many thanks for your pointers, guidance, and patience.

    Wednesday, September 27, 2017 2:31 PM
  • Hi Nurmawan,

    Thanks for your reply.

    According your description, you had re-built the environment. I noticed you only changed the admin account, does it work now?

    And yes, something may be reset during the re-built process.

    If anything other I can help you, please feel free to let me know.

    Thanks for your patience! 


    Best Regards,

    Leon-Lu
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, September 29, 2017 2:37 AM
  • Hi Leon,

    Yes, it's working now. Both CCE1 and CCE2 are working as HA pair.

    Thank you.

    Monday, October 2, 2017 9:46 AM
  • Hi Nurmawan,

    If the reply is helpful to you, please try to mark it as an answer, it will help others who has similar issue.


    Best Regards,

    Leon-Lu
    TechNet Community Support


    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, October 6, 2017 1:35 AM