none
Cloud SSA error while uploading to Azure

    Question

  • Hi all,

    We are implementing Hybrid Cloud Search. 

    The on-boarding script has been successfully executed.

    While crawling, I can see the following errors in the ULS Viewer:

    AzureServiceProxy::GetCerts caught AggregateException: Unable to connect to the remote server
    
    AzureServiceProxy::GetCerts: Failed to get encryption certificates from cert server * for realm *, documents will be send unencrypted (if unecrypted submit is allowed)
    
    AzureServiceProxy::GetAzureTenantInfo caught AggregateException: Unable to connect to the remote server, unable to get ServiceProperties, submit is blocked
    
    AzureServiceProxy caught Exception: *** Microsoft.Office.Server.Search.AzureSearchService.AzureException: AzurePlugin was not able to get Tenant Info from configuration server    
     at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.GetAzureTenantInfo(String portalURL, String realm, String& returnPropertyValue, String propertyName)    
     at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.SubmitDocuments(String azureServiceLocation, String authRealm, String SPOServiceTenantID, String SearchContentService_ContentFarmId, String portalURL, String testId, String encryptionCert, Boolean allowUnencryptedSubmit, sSubmitDocument[] documents, sDocumentResult[]& results, sAzureRequestInfo& RequestInfo) ***

    I am able to access the Internet from the server where the crawling is taking place, so it doesn't seem to be a proxy issue.

    Is there anything else I could look for?


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Thursday, October 22, 2015 1:16 PM

All replies

  • According to my understanding it seems that some part of onboarding script got failed.

    Can you make sure that below part of the script was done as expected,

    Write-Host "Connecting on-prem sharepoint farm to Office 365 tenant..." -foreground Yellow
    Connect-SPFarmToAAD -AADRealm $AADRealm -O365Credentials $credentials
    Write-Host "Done connecting!" -foreground Green

    This will make sure that your on-premise will connect to O365 tenant. Check Onboarding script log to make sure that this point not failed.

    Regards !

    Ravi


    Ravi De Alwis

    Thursday, October 22, 2015 8:59 PM
  • Once you followed the steps of re-running the onboarding script that Ravi mentioned above and assuming it completes successfully , can you also let us know that if you farm has multiple accounts for search implementation like separate crawl account and separate account running search service ?. Also do you have outbound proxy for users to go out to internet . If yes can you confirm by logging in with those accounts that the accounts has rights to access internet.

    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    • Marked as answer by Nico Martens Tuesday, November 24, 2015 1:30 PM
    • Unmarked as answer by Nico Martens Thursday, February 18, 2016 10:30 AM
    Friday, October 23, 2015 5:12 AM
  • Hi, the Connect-SPFarmToAAD has run and it showed Done connecting. Is there any way to verify this?

    For this purpose, I have only 1 account for the Cloud SSA Application Pool, which is also used as the default content access account.

    We have an outbound proxy for the SharePoint server. At this point, we configured the proxy in the web.config and using netsh winhttp set proxy. Also, for testing purposes, I added the same proxy to Internet Explorer.

    I can browse to any site to the internet after setting the proxy for Internet explorer.

    I'm thinking it is still a proxy issue, but I'm not sure where to configure the proxy.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Friday, October 23, 2015 5:52 AM
  • One more thing.

    The first time I ran the on-boarding script, it failed because there was no Internet access from the server. At this point, the web.config was already configured to use a proxy.

    However, in order to make the on-boarding script work, i had to set the proxy using Netsh winhttp set proxy cmd.

    After this, the on-boarding script ran successfully. Are there any services (or the server altogether) I have to restart in order to make it use the new proxy?


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Friday, October 23, 2015 7:19 AM
  • In the crawl log I can see the following Top-level error:

    An unexpected error occurred in the Azure plugin. This item will be retried in the next incremental crawl


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Friday, October 23, 2015 9:08 AM
  • Can you please re-run the onboarding script one more time , ensure that it completes successfully and see if you can still reproduce the issue.

    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    Friday, October 23, 2015 10:52 AM
  • Hi Manas,

    I have scheduled to run the on-boarding script in 1 hour, I will post the results here when it is done.

    Thanks for your feedback.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Friday, October 23, 2015 10:56 AM
  • Hi,

    The on-boarding script has run succesfully. See attached image.

    However, the errors still appear and no documents are being crawled.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Friday, October 23, 2015 1:42 PM
  • Just a little follow up. We were getting calls from people telling us that apps didn't work anymore. Also, the Workflow Manager seemed to have stopped working.

    The onboarding script changes the (SPSecurityTokenConfig).NameIdentifier to the AAD Realm. 

    This caused our SPSecurityTokenIssuers to stop working, as they were listening on the local SharePoint STS realm. We have been troubleshooting this issue for 8 hours before fixing it.

    However, is there no other way to configure Hybrid search then to change the NameIdentifier? And if not, how should we go about this regarding our apps or workflow services?

    Thanks in advance.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Tuesday, October 27, 2015 9:39 AM
  • Thank you for letting us know Nico. We are investigating this and will get back to you with an update.

    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    Tuesday, October 27, 2015 1:15 PM
  • Thanks very much!

    Nico Martens
    SharePoint/Office365/Azure Consultant

    Tuesday, October 27, 2015 1:18 PM
  • Hello Nico,

    I have an environment with high trust apps and did manage to configure Cloud Search Service application. We would want to understand more about your environment before we come up with recommendations. Can you let us know the below details

    1. Details on your farm topology where you have configured Cloud SSA . Do give us details on your apps , you have on this environment.  

    2. I assume you are reverting value of (SPSecurityTokenConfig).NameIdentifier , is that accurate . If you could provide any additional info would add value to our repro and we can come back to you with next actions. 


    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    Friday, October 30, 2015 1:15 PM
  • Hi Manas,

    Is there any private way I can provide information, as this is a client's environment. I am not allowed so send specific information on public forums.

    Yes I reverted the NameIdentifier to make sure the apps and workflow manager started working again.

    The SPTrustedSecuritytokenIssuers have the following RegisteredIssuerName: 

    <IssuerID>@<SPRealm>

    As the realm was changed by the on-boarding script, the SPTrustedSecurityTokenIssuer doesn't recoginize the <SPRealm> part anymore, as it was changed.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Friday, October 30, 2015 1:31 PM
  • Hello Nico,

    Thank you for sharing the details. We have made some changes to the onboarding script. Can you download the new version of the script from Connect and re-run onboarding again . Please let us know your observations.


    Regards,

    Manas Biswas MSFT

    SharePoint Online Escalation Services


    Tuesday, November 03, 2015 5:33 AM
  • Hi Manas,

    Thanks for coming back to me on this.

    I will review the script and let you know the outcome.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Tuesday, November 03, 2015 7:27 AM
  • Hi Manas,

    I have quickly checked what changed in the script. I noticed the line

    $LocalSTS.NameIdentifier = '{0}@{1}' -f $SP_APPPRINCIPALID,$AADRealm   

    is gone. That looks promising.

    However, I can see that the AAD Realm is still put in the Set-SPAuthenticationRealm. What exactly is this used for?

    When rolling back the last script, I had to set the AuthenticationRealm back to the old value as well.

    Will this break my S2S trust with apps / Workflow manager if it is changed to the AAD Realm?


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Tuesday, November 03, 2015 7:46 AM
  • Hello Nico,

    The Set-SPAuthenticationRealm –Realm <AAD realm> is required because of how we acquire bearer tokens based on SPServiceContext. Is it possible for you to execute the script we provided for onboarding on some test Sharepoint setup and let us know what you see. 


    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services



    Sunday, November 08, 2015 6:40 AM
  • Hello Nico,

    When you have some time , appreciate if you share your findings with the testing.


    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services


    Wednesday, November 11, 2015 9:51 AM
  • Hi Manas,

    We had a SharePoint issue with Microsoft services, sorry for the late response.

    I'll try the new script today or tomorrow on a test environment.

    I will let you know the outcome.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Wednesday, November 11, 2015 9:54 AM
  • Hi Manas,

    Yesterday we tried running the new version (The Hybrid Configuration Wizard version 0.9). 

    We tried running the Hybrid Configuration Wizard with the default STS certificate and after that didn't work, we tried using the wizard using a new HybridWizard self-signed certificate.

    Both give the same error when trying to crawl:

    AzureServiceProxy caught Exception: *** Microsoft.Office.Server.Search.AzureSearchService.AzureException: Neither cert server nor farm label configured, unable to retrieve encryption certificates, cannot submit documents to Azure (unecrypted submit is not allowed)     at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.SubmitDocuments(String azureServiceLocation, String authRealm, String SPOServiceTenantID, String SearchContentService_ContentFarmId, String portalURL, String testId, String encryptionCert, Boolean allowUnencryptedSubmit, sSubmitDocument[] documents, sDocumentResult[]& results, sAzureRequestInfo& RequestInfo) ***

    Are there any steps we can take to verify if all steps are performed correctly? 

    I have also added some attachments which show the Hybrid Configuration Wizard and some settings from SPO.

    If you require any more information, please let me know.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Tuesday, November 24, 2015 9:59 AM
  • Hello Nico,

    The wizard you are referring to from our blog post is for Hybrid Query federation and not for on boarding Cloud Search Service Application. The latest version of the on boarding script can be downloaded from Microsoft Connect Site . Can you download a copy of the same ,execute the same in your farm and let us know if you still see any issues.



    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    Tuesday, November 24, 2015 10:43 AM
  • Hi Manas,

    Sorry. Last friday I tried downloading the latest version for the onboarding script. However, it was not available. After searching I came across the other article on hybrid search so I figured it was replaced.

    We have run the newest version of the onboarding script (20-11-2015). The apps and workflow farm are still functioning as before, so that is a good thing.

    However, the original error is occurring again when we start a crawl:

    AzureServiceProxy caught Exception: *** Microsoft.Office.Server.Search.AzureSearchService.AzureException: AzurePlugin was not able to get Tenant Info from configuration server    
     at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.GetAzureTenantInfo(String portalURL, String realm, String& returnPropertyValue, String propertyName)    
     at Microsoft.Office.Server.Search.AzureSearchService.AzureServiceProxy.SubmitDocuments(String azureServiceLocation, String authRealm, String SPOServiceTenantID, String SearchContentService_ContentFarmId, String portalURL, String testId, String encryptionCert, Boolean allowUnencryptedSubmit, sSubmitDocument[] documents, sDocumentResult[]& results, sAzureRequestInfo& RequestInfo) ***

    To make sure this is not a SSL traffic proxy issue, I will be replacing the proxy being used with a gateway, so we have direct internet acccess.

    If there are any more things I can check, please let me know.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Tuesday, November 24, 2015 12:01 PM
  • Hi Manas,

    After disabling the proxy for internet access, I ran the on-boarding script again, and crawling went successfully.

    It seems our proxy server is interrupting the traffic flow to Azure. After setting the proxy on again, crawling won't work anymore. I guess SSL inspection does something that breaks it.

    Thanks for your continued support in this case, maybe others can use our case to find the problem faster :-)


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Tuesday, November 24, 2015 12:30 PM
  • Hello Nico,

    Thanks for the update . Good to know crawling went fine. If any questions or observations do let us know.

     

    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    Tuesday, November 24, 2015 12:50 PM
  • Hi Manas,

    One additional piece of configuration was missing, I posted a blog on it:

    http://sharepointrelated.com/2015/12/11/cloud-hybrid-search-proxy-settings/

    If you are using a proxy, you have to change the machine.config in the "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config"


    Nico Martens
    SharePoint/Office365/Azure Consultant


    Friday, December 11, 2015 10:19 AM
  • Hi Manas,

    Thanks for your continued support on this issue. 
    Unfortunately, after some more testing, we found that the current script (updated 14-12-2015) still breaks Server 2 Server trusts. I have created a new thread for this, to avoid confusion: https://social.technet.microsoft.com/Forums/en-US/f524732b-f683-4522-8940-9ac7f393e121/cloud-hybrid-search-breaking-other-s2s-trusts?forum=CloudSSA

    I would be very thankful if you could take a look at this issue.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Thursday, February 18, 2016 10:35 AM
  • Thank you for the heads-up Nico. We are working on the ask and would get back with an update.

    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    Wednesday, March 09, 2016 7:54 AM
  • Hi Manas,

    Thanks for your reply. Currently, we are in a mail conversation with Neil Hodgkinson on this issue.

    We have proposed a solution that involves changing the SPSecurityTokenIssuers before running the onboarding script. :-)

    If I can be of any help, please let me know.


    Nico Martens
    SharePoint/Office365/Azure Consultant

    Wednesday, March 09, 2016 8:03 AM
  • Thanks Nico , Neil and I are discussing on the same update and been doing some additional validation. We will keep you posted on the progress.

    Regards, Manas Biswas MSFT- SharePoint Online Escalation Services

    Wednesday, March 09, 2016 8:25 AM