none
Issue deploying SharePoint 2016 compiled solutions on 2019 Farm? RRS feed

  • Question

  • Hi,

    We have number of legacy WSP solutions built for SharePoint 2016 platform.

    We are trying to deploy them on 2019 without recompiling. We hope it should work without issues.

    The behavior however is very odd, solution gets deployed to only of the servers in 2 WFE servers. For some it shows "Not Deployed" on solutions page but after clicking on the solution, it shows "Deployed" but again only for one server.

    Does the solution need to be recompiled specifically for 2019.

    Please let me know if you have any suggestions.

    Thanks,

    Rahul Babar


    ASP.NET, C# 4.0, Sharepoint 2007/2010, Infopath 2007/2010 Developer http://sharepoint247.wordpress.com/

    Monday, March 4, 2019 8:49 PM

Answers

  • We got an update from Microsoft support:

    There is a defect in the product (SP2019) which causes update conflicts when multiple servers simultaniously writes to the config cache during farm solution deployment on farms with multiple servers.

    No ETA yet, expect months before there is a fix in CU.

    Recommended workaround is to use the -Local parameter on Install-SPSolution and install on each server separately in sequence.

    When deployed on all servers, "Globally Deployed" will be true.

    From ULS:

    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Config Cache be3x5 Medium Name lookup failure in derived cache for [Name:global-deployment],[Parent:c0644b57-1398-4fe2-bff0-f76761e7898d], [type:f24c0c10-bcf5-447d-8f71-94f6e631e087 [Microsoft.SharePoint.Administration.SPSolutionLanguagePack+DeploymentConfig]]. Stack Trace:   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(String name, Guid parentId, Type type)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.StoreObject(SPPersistedObject obj, Boolean storeClassIfNecessary, Boolean ensure)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.Microsoft.SharePoint.Administration.ISPPersistedStoreProvider.PutObject(SPPersistedObject persistedObject, Boolean ensure)     at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdateCore(Boolean legacyUpdate)     at Microsoft.SharePoint.Administration.SPPersistedObject.Update()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.UpdateDeploymentConfig(DeploymentConfig& config, Int32 numServers)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.AddGlobalDeploymentConfig()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployFilesInstallFeatures(SPWebApplication webApp, Boolean globalInstallWPPackDlls, Boolean installFeatures, Boolean force, Int32 tries, Int32 compatibilityLevel)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployLocalCore(Boolean globalInstallWPPackDlls, Collection`1 webApplications, Boolean useAdminService, Boolean force, Boolean checkIfDeployed, SPCompatibilityRange compatibilityRange)     at Microsoft.SharePoint.Administration.SPSolutionDeploymentJobDefinition.Execute(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPAdministrationServiceJobDefinition.ExecuteAdminJob(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)     at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result) 2d32db9e-5450-c0dd-289a-1fabee027e38

    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Timer 6759 Critical The Execute method of job definition "SPSolutionDeploymentJobDefinition" (id "fe7467e2-b6ed-4319-8a2d-17fb6103fc4f") threw an exception. Object reference not set to an instance of an object. 2d32db9e-5450-c0dd-289a-1fabee027e38
    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Timer 8udt Unexpected Solution Deployment : Exception stack trace:    at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.UpdateDeploymentConfig(DeploymentConfig& config, Int32 numServers)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.AddGlobalDeploymentConfig()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployFilesInstallFeatures(SPWebApplication webApp, Boolean globalInstallWPPackDlls, Boolean installFeatures, Boolean force, Int32 tries, Int32 compatibilityLevel)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployLocalCore(Boolean globalInstallWPPackDlls, Collection`1 webApplications, Boolean useAdminService, Boolean force, Boolean checkIfDeployed, SPCompatibilityRange compatibilityRange)     at Microsoft.SharePoint.Administration.SPSolutionDeploymentJobDefinition.Execute(Guid targetInstanceId) 2d32db9e-5450-c0dd-289a-1fabee027e38

    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Config Cache 8xqz Medium Updating SPPersistedObject SolutionOperationStatus Name=SolutionOperationStatus. Version: -1 Ensure: False, SkipObjectCallbackCheck: False, HashCode: 13530048, Id: 6f2b1977-2d76-4f7c-8320-3586a4a08f7c, Stack:    at Microsoft.SharePoint.Administration.SPPersistedObject.Update()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.SetOperationResult(SPSolutionOperationResult opResult, String msg, SPWebApplication webApp, Int32 updatesPerServer)     at Microsoft.SharePoint.Administration.SPSolutionDeploymentJobDefinition.Execute(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPAdministrationServiceJobDefinition.ExecuteAdminJob(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)     at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result) 2d32db9e-5450-c0dd-289a-1fabee027e38

    • Proposed as answer by Johan Lidberg Tuesday, May 14, 2019 9:21 AM
    • Marked as answer by Rahul Babar Wednesday, May 15, 2019 1:10 AM
    Tuesday, May 14, 2019 9:19 AM

All replies

  • Hi Rahul Babar,

    I suggest you could recompile the solution for SharePoint 2019.

    Best regards,

    Sara Fan


    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    SharePoint Server 2019 has been released, you can click here to download it.
    Click here to learn new features. Visit the dedicated forum to share, explore and talk to experts about SharePoint Server 2019.

    Tuesday, March 5, 2019 3:18 AM
  • Thanks for the response.

    We want to avoid recompiling as much as possible to avoid maintaining two different solutions for 2016 and 2019. So we want to confirm if it is supported to have 2016 solutions run on the 2019 farm.

    The solutions worked just fine on 1 server farms. Only issue is with multi server production farms.

    Thanks,

    Rahul Babar


    ASP.NET, C# 4.0, Sharepoint 2007/2010, Infopath 2007/2010 Developer http://sharepoint247.wordpress.com/

    Tuesday, March 5, 2019 1:15 PM
  • Are they need content Databases / site collections? Its possible that the features are already in the existing site collections.  Try re-deploying with the -force switch.

    Install-SPSolution –Identity “NAME.wsp” –GACDeployment -Force

    Mike Lee

    Tuesday, March 5, 2019 3:03 PM
  • Hi Rahul Babar,

    If the reply is helpful to you, you could mark the reply as answer. Thanks for your understanding.

    Best regards,

    Sara Fan

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    SharePoint Server 2019 has been released, you can click here to download it.
    Click here to learn new features. Visit the dedicated forum to share, explore and talk to experts about SharePoint Server 2019.

    Wednesday, March 6, 2019 9:44 AM
  • Hi Rahul Babar,

    There might be an issue with deploying farm solutions in SP2019 multi server farm.
    We have also encountered this same kind of problem you described.

    We have managed to replicate this issue in multiple farm (azure farm and on-premise farm (sp2019 + win2019)) and it seems that farm solution deployment is currently unstable somehow. We opened a support case and Microsoft is currently investigating the problem.

    Installing solution with powershell (install-spsolution) and using -force parameter works sometimes. Needless to say you have to deploy solution with force parameter a couple of times before succeeding. After force deployment CA status shows deployed but there still might be servers missing on operation details! Weird.

    It seems that solution artifacts (features, dlls) are actually deployed to all servers but after installing solution artifacts, CA server somehow cannot agree on successful deployment on all servers. Issue is random and deployment works sometimes ok.  Error occurs more often if you have more servers in your farm. Looks like some kind of update conflict(?) Like you said, in a single server farm this error does not seem to happen.

    Recompiling in SP2019 environment did not fix our problem. We even tried "empty solution package" with just one global assembly DLL.  Btw: Update-spsolution might also have same kind of problem.

    It could be also some sort of configuration issue, which we are not aware of.  Anyway, (legacy) farm solutions are still supported, and SP2019 is still quite new, so we opened a support case and Microsoft is currently investigating our problem.  I really suggest you open a support case also if needed.


    --Juha The Password is "Ken Sent Me"

    Tuesday, March 26, 2019 7:54 AM
  • Hi Rahul Babar,

    If the reply is helpful to you, you could mark the reply as answer. Thanks for your understanding.

    Best regards,

    Sara Fan

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    SharePoint Server 2019 has been released, you can click here to download it.
    Click here to learn new features. Visit the dedicated forum to share, explore and talk to experts about SharePoint Server 2019.

    Tuesday, March 26, 2019 10:05 AM
  • Hi Juha,

    We are experiencing the exact same problem with SharePoint 2019 and multi-server farm. Have you received any more information from your support case? When we check the logs for the server that not gets successfully deployed(can be a different one every try) we detect a

    "Solution Deployment : Try 1 : DuplicateObjectException adding server XXXXXXXX to deploymentConfig global-deployment XXX.sharepoint.foundation.wsp. Trying again."

    Followed by a

    "The Execute method of job definition "SPSolutionDeploymentJobDefinition" (id "fa5d6899-c2f9-4a3a-a1b1-b04231f4553f") threw an exception. Object reference not set to an instance of an object."

    Monday, April 15, 2019 12:55 PM
  • Hi Örjan,

    According to your description ("duplicateobjectexception...") this is the same issue we have encountered.  We have contacted Microsoft last week and they were still investigating the issue. So we are still waiting for fix.



    --Juha

    Tuesday, April 16, 2019 11:49 AM
  • Hi Juha,

    Thanks for replying, we will also open a support case for this issue. Please let me know when u recieve new information on the issue and i will do the same

    //Örjan

    Wednesday, April 17, 2019 9:14 AM
  • I have the same issue also with multi server farms, have tried several and CA shows the status as Not Deployed.

    Tried PowerShell and CA GUI.  Some solutions deploy ok and others we need to keep retrying till it works.

    Any update from Microsoft on this?


    mmm... coffee...

    Friday, May 3, 2019 3:20 AM
  • We are still facing same issue and currently working with MS Support to find out about the issue.

    Please let me know if there are any findings.

    Thanks,

    Rahul Babar


    ASP.NET, C# 4.0, Sharepoint 2007/2010, Infopath 2007/2010 Developer http://sharepoint247.wordpress.com/

    Tuesday, May 7, 2019 12:25 PM
  • We got an update from Microsoft support:

    There is a defect in the product (SP2019) which causes update conflicts when multiple servers simultaniously writes to the config cache during farm solution deployment on farms with multiple servers.

    No ETA yet, expect months before there is a fix in CU.

    Recommended workaround is to use the -Local parameter on Install-SPSolution and install on each server separately in sequence.

    When deployed on all servers, "Globally Deployed" will be true.

    From ULS:

    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Config Cache be3x5 Medium Name lookup failure in derived cache for [Name:global-deployment],[Parent:c0644b57-1398-4fe2-bff0-f76761e7898d], [type:f24c0c10-bcf5-447d-8f71-94f6e631e087 [Microsoft.SharePoint.Administration.SPSolutionLanguagePack+DeploymentConfig]]. Stack Trace:   at Microsoft.SharePoint.Administration.SPConfigurationDatabase.GetObject(String name, Guid parentId, Type type)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.StoreObject(SPPersistedObject obj, Boolean storeClassIfNecessary, Boolean ensure)     at Microsoft.SharePoint.Administration.SPConfigurationDatabase.Microsoft.SharePoint.Administration.ISPPersistedStoreProvider.PutObject(SPPersistedObject persistedObject, Boolean ensure)     at Microsoft.SharePoint.Administration.SPPersistedObject.BaseUpdateCore(Boolean legacyUpdate)     at Microsoft.SharePoint.Administration.SPPersistedObject.Update()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.UpdateDeploymentConfig(DeploymentConfig& config, Int32 numServers)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.AddGlobalDeploymentConfig()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployFilesInstallFeatures(SPWebApplication webApp, Boolean globalInstallWPPackDlls, Boolean installFeatures, Boolean force, Int32 tries, Int32 compatibilityLevel)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployLocalCore(Boolean globalInstallWPPackDlls, Collection`1 webApplications, Boolean useAdminService, Boolean force, Boolean checkIfDeployed, SPCompatibilityRange compatibilityRange)     at Microsoft.SharePoint.Administration.SPSolutionDeploymentJobDefinition.Execute(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPAdministrationServiceJobDefinition.ExecuteAdminJob(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)     at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result) 2d32db9e-5450-c0dd-289a-1fabee027e38

    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Timer 6759 Critical The Execute method of job definition "SPSolutionDeploymentJobDefinition" (id "fe7467e2-b6ed-4319-8a2d-17fb6103fc4f") threw an exception. Object reference not set to an instance of an object. 2d32db9e-5450-c0dd-289a-1fabee027e38
    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Timer 8udt Unexpected Solution Deployment : Exception stack trace:    at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.UpdateDeploymentConfig(DeploymentConfig& config, Int32 numServers)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.AddGlobalDeploymentConfig()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployFilesInstallFeatures(SPWebApplication webApp, Boolean globalInstallWPPackDlls, Boolean installFeatures, Boolean force, Int32 tries, Int32 compatibilityLevel)     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.DeployLocalCore(Boolean globalInstallWPPackDlls, Collection`1 webApplications, Boolean useAdminService, Boolean force, Boolean checkIfDeployed, SPCompatibilityRange compatibilityRange)     at Microsoft.SharePoint.Administration.SPSolutionDeploymentJobDefinition.Execute(Guid targetInstanceId) 2d32db9e-5450-c0dd-289a-1fabee027e38

    05-10-2019 10:57:17.57 OWSTIMER.EXE (S0005103:0x2068) 0x1B84 SharePoint Foundation Config Cache 8xqz Medium Updating SPPersistedObject SolutionOperationStatus Name=SolutionOperationStatus. Version: -1 Ensure: False, SkipObjectCallbackCheck: False, HashCode: 13530048, Id: 6f2b1977-2d76-4f7c-8320-3586a4a08f7c, Stack:    at Microsoft.SharePoint.Administration.SPPersistedObject.Update()     at Microsoft.SharePoint.Administration.SPSolutionLanguagePack.SetOperationResult(SPSolutionOperationResult opResult, String msg, SPWebApplication webApp, Int32 updatesPerServer)     at Microsoft.SharePoint.Administration.SPSolutionDeploymentJobDefinition.Execute(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPAdministrationServiceJobDefinition.ExecuteAdminJob(Guid targetInstanceId)     at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)     at Microsoft.SharePoint.Administration.SPTimerJobInvoke.Invoke(TimerJobExecuteData& data, Int32& result) 2d32db9e-5450-c0dd-289a-1fabee027e38

    • Proposed as answer by Johan Lidberg Tuesday, May 14, 2019 9:21 AM
    • Marked as answer by Rahul Babar Wednesday, May 15, 2019 1:10 AM
    Tuesday, May 14, 2019 9:19 AM
  • Thanks for posting the MS Response. We also got confirmation about 2019 defect.

    That is really helpful.

    Thanks,

    Rahul Babar


    ASP.NET, C# 4.0, Sharepoint 2007/2010, Infopath 2007/2010 Developer http://sharepoint247.wordpress.com/

    Tuesday, May 14, 2019 12:26 PM
  • Hi Rahul Babar,

    If the reply is helpful to you, you could mark the reply as answer. Thanks for your understanding.

    Best regards,

    Sara Fan

    Please remember to mark the replies as answers if they helped. If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    SharePoint Server 2019 has been released, you can click here to download it.
    Click here to learn new features. Visit the dedicated forum to share, explore and talk to experts about SharePoint Server 2019.

    Wednesday, May 15, 2019 12:51 AM
  • This needs to be a sticky on this thread. I've just run into this issue also. So happy I found this before I started to pull my hair out over it.

    If this is helpful please mark it so. Also if this solved your problem mark as answer.

    Friday, August 9, 2019 8:19 AM
  • Good news! Our product group identified and fixed the issue.
    The fix is going to be included in October or November 2019 CU.
    Please carefully check the KBs of upcoming CUs to check if it is included. 
    Friday, August 9, 2019 9:13 AM
  • We were told this fix will be part of the Sept 19 CU and the notes clearly stated that this fix was included. 
    FIX : "Installing a SharePoint Server 2019 solution on a multi-server farm is an unreliable process and will not necessartily succeed."

    https://support.microsoft.com/sq-al/help/4475596/security-update-for-sharepoint-server-2019-september-10

    But after installing the Sept CU we still see this issue with solution deployment. This is very unprofessional, how can you release a CU claiming a bug was fixed without actually fixing it. Our go-live plan is all messed up because of this issue. Do you have a confirmed release date?



    • Edited by Renjii Tuesday, September 17, 2019 11:32 PM
    Tuesday, September 17, 2019 11:30 PM
  • Hi Renjii, the fix was included in the September 2019 Public Update for SharePoint Server 2019.  If you're still seeing an issue with SharePoint solutions failing to deploy successful in a multi-server farm after installing that update, then your issue may have a different root cause than the issue that was fixed.

    I recommend opening a support case with Microsoft Support so they can investigate why this is failing in your environment and if additional fixes are needed.

    - Troy Starr [MSFT]

    Wednesday, September 18, 2019 6:55 AM
  • Hi

    did someone installed the Sept CU, did it fixed this issue?

    Thanks

    Octavian

    Thursday, September 19, 2019 12:37 PM
  • 2nd EDIT:

    I've been able to replicate successful deployment of all five solutions this way:

    • retract/remove all solutions
    • clean any remants of the solutions from GAC on both servers
    • reboot both servers
    • run my script to install the solutions

    I'm two for two with this so far. 

    ------first update-----

    EDIT: I've been removing and redeploying my solutions, and I'm seeing some continued flakiness with the deployments only hitting one server or another.

    I've found no rhyme nor reason - at first, it looked like -force would resolve it, but then a -forced deployment only hit a single server, after three previous successful deployments. 

    So it looks like this patch is not working as intended.

    ----original post------

    I just installed the Sep CU (remember, you need both the CU and the language pack). 


    I still had the issue after running the configuration wizard, but I then rebooted both servers and the solutions deployed to each server as expected.

    -Pete

    • Edited by PeteMIII Friday, September 20, 2019 9:15 PM
    Friday, September 20, 2019 7:19 PM
  • Hi

    I confirm that the patch didn't fixed the issue in my case neither.

    Octavian

    Friday, October 4, 2019 12:18 PM
  • October patch doesn't fix the issue. Hoping for November if done via PowerShell. Went through UI and selected deploy and it seems to have worked


    If this is helpful please mark it so. Also if this solved your problem mark as answer.


    • Edited by taylor.l Tuesday, October 15, 2019 1:48 PM
    Tuesday, October 15, 2019 1:47 PM
  • We had also installed the Oct CU and still see this issue. Some times it works and other times it fails while done from Powershell or UI. So it is not consistent. Try deploying multiple times and you might see the issue. We still have the ticket open with MS and so far no luck.


    • Edited by Renjii Tuesday, October 15, 2019 8:57 PM
    Tuesday, October 15, 2019 8:55 PM