none
Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data...

    Question

  • We have SharePoint Server 2016 Minrole deployed with Distributed cache successfully running on its own server role. However We suddenly start getting "Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data... (Correlation=xxxxxx) for both The Execute method of job definition Microsoft.Office.Server.UserProfiles.LMTRepopulationJob  and The Execute method of job definition Microsoft.Office.Server.UserProfiles.FeedCacheRepopulationJob

    We have done the following so far but still getting the error every 5 min:

    • Stopped the Timer Service, Cleared Config Cache, Restarted Timer Service
    • Restarted App server
    • Looked at the logs with following output:
    Timestamp              	Process                                 	TID   	Area                          	Category                      	EventID	Level     	Message 	Correlation
    10/27/2016 08:50:00.58 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Foundation         	Logging Correlation Data      	xmnv	Medium	Name=Timer Job User Profile Service Application Proxy_LMTRepopulationJob	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.58 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Portal Server      	User Profiles                 	arkm4	Medium	ChannelInvoke::IsRepopulationNeeded::1 -- Executing code block on endpoint [http://xxxx:32843/851910283e774515b1f4146ed3e91de8/FeedCacheService.svc].	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.58 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Foundation         	Topology                      	e5mc	Medium	WcfSendRequest: RemoteAddress: 'http://xxxx:32843/851910283e774515b1f4146ed3e91de8/FeedCacheService.svc' Channel: 'Microsoft.Office.Server.UserProfiles.IFeedCacheService' Action: 'http://Microsoft.Office.Server.UserProfiles/IsRepopulationNeeded' MessageId: 'urn:uuid:8c2454e5-adb8-48db-acf7-93e9bb858afd'	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Topology                      	e5mb	Medium	WcfReceiveRequest: LocalAddress: 'http://xxxx.xxxx.com:32843/851910283e774515b1f4146ed3e91de8/FeedCacheService.svc' Channel: 'System.ServiceModel.Channels.ServiceChannel' Action: 'http://Microsoft.Office.Server.UserProfiles/IsRepopulationNeeded' MessageId: 'urn:uuid:8c2454e5-adb8-48db-acf7-93e9bb858afd'	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Monitoring                    	nasq	Medium	Entering Monitored Scope (FeedCacheService.IsRepopulationNeeded). Parent=ExecuteWcfServerOperation (http://xxxx.xxxx.com:32843/851910283e774515b1f4146ed3e91de8/FeedCacheService.svc)	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Monitoring                    	nasq	Medium	Entering Monitored Scope (FeedCacheImplementation.IsRepopulationNeeded). Parent=FeedCacheService.IsRepopulationNeeded	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Monitoring                    	nasq	Medium	Entering Monitored Scope (FeedCacheDataCacheWrapper.GetObject). Parent=FeedCacheImplementation.IsRepopulationNeeded	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Server             	Distributed Cache             	aa6e5	Unexpected	Unexpected error in Decrypt - Exception 'System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.     at System.Security.Cryptography.CapiSymmetricAlgorithm.DepadBlock(Byte[] block, Int32 offset, Int32 count)     at System.Security.Cryptography.CapiSymmetricAlgorithm.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)     at System.Security.Cryptography.CryptoStream.FlushFinalBlock()     at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)     at System.IO.Stream.Close()     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Transform(Byte[] buffer, ICryptoTransform transformer)     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Transform(Byte[] secretKey, Byte[] buff...	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60*	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Server             	Distributed Cache             	aa6e5	Unexpected	...er, Boolean encrypt)     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Decrypt(Byte[] secretKey, Byte[] buffer)'.	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Server             	Distributed Cache             	a7l3j	High	Unable to decrypt using secret key. Exception 'Microsoft.Office.Server.DistributedCaching.SPDistributedCachedObjectDecryptionFailedException: Failed to Decrypt data.     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Decrypt(Byte[] secretKey, Byte[] buffer)     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.PostProcessData(Byte[] secretKey, Object cachedObject, Boolean bDecrptyData, SPDistributedCacheTelemetryData teleData)'.	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Server             	Distributed Cache             	a7l3m	Unexpected	Unable to PostProcessData - Exception 'Microsoft.Office.Server.DistributedCaching.SPDistributedCachedObjectDecryptionFailedException: Failed to Decrypt data.     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Decrypt(Byte[] secretKey, Byte[] buffer)     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.PostProcessData(Byte[] secretKey, Object cachedObject, Boolean bDecrptyData, SPDistributedCacheTelemetryData teleData)'.	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Server             	Distributed Cache             	agyfy	Unexpected	Unexpected error occurred in method 'GetObject' , usage 'FeedCache' - Exception 'Microsoft.Office.Server.DistributedCaching.SPDistributedCachedObjectDecryptionFailedException: Failed to Decrypt data.     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Decrypt(Byte[] secretKey, Byte[] buffer)     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.PostProcessData(Byte[] secretKey, Object cachedObject, Boolean bDecrptyData, SPDistributedCacheTelemetryData teleData)     at Microsoft.Office.Server.DistributedCaching.SPDistributedCache.GetObject(String key, String regionName)'.	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Monitoring                    	b4ly	Medium	Leaving Monitored Scope: (FeedCacheDataCacheWrapper.GetObject) Execution Time=2.9878; CPU Milliseconds=1; SQL Query Count=0; Parent=FeedCacheImplementation.IsRepopulationNeeded	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Portal Server      	Feed Cache                    	afr68	High	FeedCacheImplementation.IsRepopulationNeeded() - Unexpected exception while trying to check is repopulation is needed for a given UPA: Microsoft.Office.Server.DistributedCaching.SPDistributedCachedObjectDecryptionFailedException: Failed to Decrypt data.     at Microsoft.Office.Server.DistributedCaching.SPDistributedCache.GetObject(String key, String regionName)     at Microsoft.Office.Server.FeedCache.FeedCacheDataCacheWrapper.GetObject(String key, String regionName)     at Microsoft.Office.Server.FeedCache.FeedCacheImplementation.IsRepopulationNeeded(Guid callerID)	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Monitoring                    	b4ly	Medium	Leaving Monitored Scope: (FeedCacheImplementation.IsRepopulationNeeded) Execution Time=3.2846; CPU Milliseconds=1; SQL Query Count=0; Parent=FeedCacheService.IsRepopulationNeeded	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Portal Server      	Feed Cache                    	afr7r	Unexpected	Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data..:    at Microsoft.Office.Server.DistributedCaching.SPDistributedCache.GetObject(String key, String regionName)     at Microsoft.Office.Server.FeedCache.FeedCacheDataCacheWrapper.GetObject(String key, String regionName)     at Microsoft.Office.Server.FeedCache.FeedCacheImplementation.IsRepopulationNeeded(Guid callerID)     at Microsoft.Office.Server.UserProfiles.FeedCacheService.Microsoft.Office.Server.UserProfiles.IFeedCacheService.IsRepopulationNeeded(Guid callerID)	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Monitoring                    	b4ly	Medium	Leaving Monitored Scope: (FeedCacheService.IsRepopulationNeeded) Execution Time=3.8474; CPU Milliseconds=2; SQL Query Count=0; Parent=ExecuteWcfServerOperation (http://xxxx.xxxxx.com:32843/851910283e774515b1f4146ed3e91de8/FeedCacheService.svc)	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Micro Trace                   	uls4	Medium	Micro Trace Tags: 0 nasq,0 e5mb,0 nasq,0 nasq,0 nasq,2 aa6e5,0 a7l3j,0 a7l3m,0 agyfy,0 b4ly,0 afr68,0 b4ly,0 afr7r,0 b4ly	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	w3wp.exe (AppServer:0x0E44)                	0x0CC4	SharePoint Foundation         	Monitoring                    	b4ly	Medium	Leaving Monitored Scope: (ExecuteWcfServerOperation (http://xxxxx.xxxx.com:32843/851910283e774515b1f4146ed3e91de8/FeedCacheService.svc)) Execution Time=4.506; CPU Milliseconds=3; SQL Query Count=0; Parent=None	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Portal Server      	User Profiles                 	arkm7	High	ChannelInvoke::IsRepopulationNeeded::1 -- FaultException occurred: [FaultCode=Sender][FaultSubCode=UnspecifiedSubCode] System.ServiceModel.FaultException`1[Microsoft.Office.Server.UserProfiles.FeedCacheFault]: Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data.. (Fault Detail is equal to Microsoft.Office.Server.UserProfiles.FeedCacheFault).	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Portal Server      	Feed Cache                    	adbpm	Unexpected	Exception occured in ExecuteOnChannel: System.ServiceModel.FaultException`1[Microsoft.Office.Server.UserProfiles.FeedCacheFault]: Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data.. (Fault Detail is equal to Microsoft.Office.Server.UserProfiles.FeedCacheFault).	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Portal Server      	Feed Cache                    	7155	Warning	FeedCache last modified time repopulation failed, social experience will suffer if this happens many times in an hour.  Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data..	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Portal Server      	Feed Cache                    	ajs4n	High	LMTRepopulationJob.Execute - Unexpected exception: Microsoft.Office.Server.UserProfiles.UserProfileException: Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data.. ---> System.ServiceModel.FaultException`1[Microsoft.Office.Server.UserProfiles.FeedCacheFault]: Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data..    Server stack trace:      at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)     at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)     at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationR...	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60*	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Portal Server      	Feed Cache                    	ajs4n	High	...untime operation)     at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)    Exception rethrown at [0]:      at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)     at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)     at Microsoft.Office.Server.UserProfiles.IFeedCacheService.IsRepopulationNeeded(Guid callerID)     at Microsoft.Office.Server.UserProfiles.FeedCacheServiceClient.<>c__DisplayClass2c.<IsRepopulationNeeded>b__2b(IFeedCacheService channel)     at Microsoft.Office.Server.Infrastructure.ChannelInvoke`1.ExecuteOnChannel(String operationName, Action`1 codeBlock, ChannelInvokeSettings channelInvokeSettings)     at Microsoft.Office.Server.UserProfiles.MossClientBase`1.Exe...	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60*	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Portal Server      	Feed Cache                    	ajs4n	High	...cuteOnChannelWithRetries(String operationName, Action`1 codeBlock)     at Microsoft.Office.Server.UserProfiles.FeedCacheServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock)     --- End of inner exception stack trace ---     at Microsoft.Office.Server.UserProfiles.FeedCacheServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock)     at Microsoft.Office.Server.UserProfiles.FeedCacheServiceClient.IsRepopulationNeeded(Guid callerID)     at Microsoft.Office.Server.UserProfiles.LMTRepopulationJob.IsRepopulationNeeded(UserProfileApplicationProxy proxy, SPPersistedObject parent)     at Microsoft.Office.Server.UserProfiles.LMTRepopulationJob.IsRepopulationNeeded()     at Microsoft.Office.Server.UserProfiles.LMTRepopulationJob.Execute()	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Foundation         	Timer                         	6398	Critical	The Execute method of job definition Microsoft.Office.Server.UserProfiles.LMTRepopulationJob (ID 40577422-e844-4f52-b110-7139ab7564b0) threw an exception. More information is included below.  Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data... (Correlation=ad74b19d-9028-3097-17b0-c9c8ea30d7d9)	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Foundation         	Timer                         	72ae	Unexpected	Exception stack trace:    at Microsoft.Office.Server.UserProfiles.FeedCacheServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock)     at Microsoft.Office.Server.UserProfiles.FeedCacheServiceClient.IsRepopulationNeeded(Guid callerID)     at Microsoft.Office.Server.UserProfiles.LMTRepopulationJob.IsRepopulationNeeded(UserProfileApplicationProxy proxy, SPPersistedObject parent)     at Microsoft.Office.Server.UserProfiles.LMTRepopulationJob.IsRepopulationNeeded()     at Microsoft.Office.Server.UserProfiles.LMTRepopulationJob.Execute()     at Microsoft.SharePoint.Administration.SPTimerJobInvokeInternal.Invoke(SPJobDefinition jd, Guid targetInstanceId, Boolean isTimerService, Int32& result)	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.60 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Foundation         	General                       	a479h	Medium	SMART API does not exist or Job is null. Skip alerting.	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.61 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Foundation         	Timer                         	an9i8	High	Completed processing of timer job [User Profile Service Application Proxy_LMTRepopulationJob] with lock type [Job] against target instance [40577422-e844-4f52-b110-7139ab7564b0] [LMTRepopulationJob Name=User Profile Service Application Proxy_LMTRepopulationJob]. Start time: [10/27/2016 07:50:00]. Duration: [00:00:00.0156224]. Status result: [Failed]. Status ressage (if any): [Unexpected exception in FeedCacheService.IsRepopulationNeeded: Failed to Decrypt data... (Correlation=ad74b19d-9028-3097-17b0-c9c8ea30d7d9)].	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    10/27/2016 08:50:00.61 	OWSTIMER.EXE (AppServer:0x0864)            	0x1D64	SharePoint Foundation         	Monitoring                    	b4ly	Medium	Leaving Monitored Scope: (Timer Job User Profile Service Application Proxy_LMTRepopulationJob) Execution Time=24.5323; CPU Milliseconds=6; SQL Query Count=1; Parent=None	ad74b19d-9028-3097-17b0-c9c8ea30d7d9
    

    By the way the App Server is not part of the cluster and does not have the app fapric service running - we only have the Distribute cache server in the custer and that is running with no issues. Search and WFE have also no errors.

    We are stuck a this stage, please help.

    • Changed type Ali Yusuf Thursday, October 27, 2016 8:09 AM
    Thursday, October 27, 2016 8:01 AM

Answers

  • Can you remove the server from the farm and readd it? Looks like, for whatever reason, it is unable to decrypt the key associated with that farm. This key has two pairs, one stored in the registry (after you add a SharePoint server to the farm) while the other piece is stored in the Config db.

    Trevor Seward

    Office Servers and Services MVP



    Author, Deploying SharePoint 2016

    This post is my own opinion and does not necessarily reflect the opinion or view of Microsoft, its employees, or other MVPs.

    • Marked as answer by Ali Yusuf Tuesday, April 4, 2017 7:38 AM
    Monday, November 21, 2016 4:28 AM
    Moderator

All replies

  • Hi,

    Please try to add your account to the "WindowsFabricAllowedUsers" group.

    Or try to restart the app fabric service and wait a couple of second with a line of PowerShell:

    Restart-Service AppFabricCachingService

    Or try to remove and re-add the service:

    Remove-SPDistributedCacheServiceInstance
    Add-SPDistributedCacheServiceInstance

    Or restart your app server.

    Best Regards,

    Andy Wu


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, November 4, 2016 8:45 AM
  • Hi Andy,

    Many thanks for the suggestions. We did all the steps you mention and still getting the error.

    Thanks,

    Ali

    Tuesday, November 8, 2016 4:23 PM
  • I was seeing this on our application server as well in our 2016 farm. Distributed cache roll running on 2 other dedicated boxes(minrole). No errors anywhere other than the application server. I cleared the cache with this script (https://spcachecleaner.codeplex.com/) and the issue appears to be resolved.
    Sunday, November 13, 2016 11:55 PM
  • Hi _Clarky_

    Thanks for the helpful solution. I was able to run the script successfully on all servers. However the error has now shifted on to one of our search server in the minrole deployment. 

    Any tips to getting rid of this error from all servers would be really helpful for us, and anyone else facing the issue.

    Kind regards,

    Ali

    Friday, November 18, 2016 10:28 AM
  • Hi Ali,

    I had the same behavior but I believe that the script I used to clear the cache may not have run correctly on SP2016. The error started appearing on one (only one) of my search servers (Minrole) after I used that  script to clear the cache. I cleared the cache manually and I can't see it on any servers now. I think (hope) its gone for good!

    Hope this help 

    clarky


    Monday, November 21, 2016 4:15 AM
  • Can you remove the server from the farm and readd it? Looks like, for whatever reason, it is unable to decrypt the key associated with that farm. This key has two pairs, one stored in the registry (after you add a SharePoint server to the farm) while the other piece is stored in the Config db.

    Trevor Seward

    Office Servers and Services MVP



    Author, Deploying SharePoint 2016

    This post is my own opinion and does not necessarily reflect the opinion or view of Microsoft, its employees, or other MVPs.

    • Marked as answer by Ali Yusuf Tuesday, April 4, 2017 7:38 AM
    Monday, November 21, 2016 4:28 AM
    Moderator
  • Many thanks Trevor,

    That solved the search server, but the issue now moved to one of our App Servers. This is weird issue, any further ideas on what we can do?

    Ali

    Tuesday, April 4, 2017 7:38 AM
  • My App server is throwing the Feedcache error people are reporting here every 5 minutes after I applied March CU 2018.  I cleared the cache on all servers with SharePoint binaries as instructed a couple of different times, with and without an IISReset afterwards, and it did not resolve the issue. 

    My App server does not have Distributed Cache running but this is the server that is failing.  DC is configured to run on the WFE's.  In the logs on the App server, I see: "Unable to decrypt using secret key. "

    I believe I should take your advice and remove the App server from the farm and re-add it back in.  I consider myself a novice SharePoint Administrator and I could use some advice about this process, please. 

    The App server runs Central Admin (CA).  To remove and re-add the App server,  would I run PSConfig.exe?  Do I run the upgrade commands? then go back to the App server and run PSConfig.exe to re-assign the server as an App server?

    Should I then run a Full Crawl because the Search role is assigned to this server?

    Thank you for any advice.  It is greatly appreciated.

    Our Test and Production environments use:  Minrole topology.  1 App/Search (combined roles); 2 WFE/Dist Cache (combined roles); 1 Custom.  Platform:  SP2016; March 2018 CU; Windows Server 2016.

    Wednesday, June 6, 2018 8:14 PM
  • I just (starting 6/6/2018) had same Feed Cache Repopulation error with same cause "Failed to decrypt data". I did not look in ULS Log so I can't say if got same "secret key" info. So I may have had different cause. Still, fwiw, here's my experience:

    We have 3 Custom role servers we are getting ready for migrating from 2010. All were recently installed, April 2018 patch level. We want #1 and #2 to be WFE/App except Search, and #3 to be Search (and backup for #1 or #2 if either fails). I did not want the Distributed Cache service running on #3, so on 6/5 I stopped it via PowerShell, then when I got the "service is stopped, may cause cache issues" event, I removed the cache instance via PowerShell. The next day 6/6 I started the User Profile service on #1 and #2 and created the User Profile Service Application, then the MySite web app. Half an hour later - in that half hour time I also configured UPSA including AD connection and My Site Host - I started getting the Feed Cache error on #3, the server *not* running either the Distributed Cache service or the User Profile service. I did not get error on #1 or #2.

    Troubleshooting = no issues with the cache cluster, both AppFabric and SharePoint see same #1 and #2 servers in cluster but not #3, both #1 and #2 up. iisreset on #3, reboot on #3, still error. Thought maybe related to error I had trying to do first full sync, which failed, with ULS Log pointing to "failed to decrypt data" as cause. But editing AD connection and retyping UPS account password fixed that issue, sync been working fine since and no errors. Thought #3 had bad cache (cobwebs needed clearing out, to use analogy). Was going to clear the Config cache on #3 but then not sure it is related and MSFT article on doing that in 2013 says do on all servers, which sounded too drastic, #1 and #2 did not have errors. Then thought, UPS is the complainer. So, maybe UPS needed its cobwebs cleared out. So today (6/12) I rebooted #1 and #2 simultaneously. And that stopped the error on #3.

    So then I found out by studying the job history in CA (which only goes back 4-5 days, to 6/9) that the failing job was the Feed Cache Full Repopulation Job, not the Feed Cache Repopulation Job. And, filtering the history on each job definition one at  a time, I see that each job only runs on one server, not all three as I thought. And that either job can switch to another server. For example, the Feed Cache Repopulation Job was running on #2 until I rebooted #1 and #2, after which it ran on #1. (Maybe because I simultaneously rebooted the both servers running the distributed cache and UPS services, and #1 came up first ?? Wild guess.) And the Feed Cache Full Repopulation Job had, in the 4-5 days timeframe, run on #3 for two days then #2 for two days, then back to #3, and each switchover occurred at same time, 6:15 AM. (Have not yet looked in App Log to see if can determine cause for switchover.) Too, when the job ran on #3, it succeeded every 5 minutes and failed every other 5 minutes; e.g. 1:30 succeeded, 1:35 failed, 1:40 succeeded, 1:45 failed, and so on. When it ran on #2 it always succeeded.

    Finally, when I rebooted #1 and #2, the error stopped on #3. Job History shows the job is still on #3 and it is now succeeding each time it runs.

    As my job history in CA only goes back 4-5 days, I also looked at Application log and that's when I noticed the Feed Cache Full Repopulation Job error started about half hour after creating the MySite web app, and despite #3 not running the distributed cache service or User Profile service, the job started on #3 from the get-go.

    I'm happy that the job is now succeeding on #3 and fingers crossed. But maybe someone can explain or point to an explanation as to why a User Profile job would run on a server not running the User Profile service, and especially why a User Profile job related to Distributed Cache would run on a server not running either the User Profile or Distributed Cache service.


    Wednesday, June 13, 2018 2:24 AM
  • Did you ever get this resolved. Having the same issues and I've tried everything that is mentioned here and other threads with no success
    Wednesday, June 27, 2018 7:25 AM
  • I'm seeing the same problem for one of my customers. Did you find the cause?

    Dean MCTS-SQL 2005 Business Intelligence, MCITP SharePoint 2010, MCSA Office 365

    Friday, August 3, 2018 5:17 PM
  • Still experiencing periodically. This morning, on SP server #1 (see my 6/13 post above). The job had been running on SP #2 successfully, then (reason unknown) switched to SP server #1 and promptly failed, has continued to fail. A "SharePoint booted" event on #1 immediately preceded job switch. In ULS Viewer, filtering on Correlation ID, this log entry milliseconds before:

    Unexpected error in Decrypt - Exception 'System.Security.Cryptography.CryptographicException: Padding is invalid and cannot be removed.   
     at System.Security.Cryptography.CapiSymmetricAlgorithm.DepadBlock(Byte[] block, Int32 offset, Int32 count)   
     at System.Security.Cryptography.CapiSymmetricAlgorithm.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)   
     at System.Security.Cryptography.CryptoStream.FlushFinalBlock()   
     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Transform(Byte[] buffer, ICryptoTransform transformer)   
     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Transform(Byte[] secretKey, Byte[] buffer, Boolean encrypt)   
     at Microsoft.Office.Server.DistributedCaching.SPDistributedCacheUtils.Decrypt(Byte[] secretKey, Byte[] buffer)'.

    I have no idea what "Padding is invalid and cannot be removed" means.

    Stopping then starting distributed cache service in CA did not fix.

    Joan

    Monday, August 6, 2018 8:18 PM
  • Searching on "windows cryptographic service padding invalid" gets lots of results. Including lots from the TechNet and MSDN forums (non-SharePoint)  https://social.msdn.microsoft.com/Forums/vstudio/en-US/d1788582-bf8c-43ec-a686-49647c359136/unexplained-cryptographicexception-padding-is-invalid?forum=netfxbcl - note the marked answer. I've given up on trying to fix, because the jobs don't always fail and the fail/succeed thing is regardless of which server the job runs on. Maybe someone will come up with an answer that I can implement but at this point/until then, I think it's a bug. It's happening on my Azure test network with 2 (custom role) WFE/App servers as well as Production with 2 (custom role) WFE/App and 1 (custom role) Search. But it's never happened on my office test net with a single SP (custom role) WFE/App server. I configured the DC the same on all, following Trevor Seward's book and MSFT.

    Monday, August 20, 2018 5:36 PM
  • Ok, correction on my last post. The error is not happening on Azure test farm. (Got confused, sorry.)

    Anyway, fixed error; jobs steadily succeeding (for now at least; been over 24 hours but can't say it won't return). Did 2 things at once, so not sure which made jobs stop failing. First, I used Clear-SPDistributedCacheItem against each of the 16 container types (ran command against each, one at a time, on both servers running the DC service). Second, I again compared production farm to office and Azure test farms (now that realized error not occurring on Azure farm) and noticed one difference: I had auto provisioned the DC service on both test farms (meaning, on Services in Farm page I had clicked Enable Auto Provision), but not on the production farm. So I enabled auto provision on the production farm.

    So, either clearing the cache items in all containers on both servers running the DC service, or enabling auto provision for the DC service (even though all servers are custom role) made error go away. For now.

    Joan

    Wednesday, August 22, 2018 8:39 PM
  • I had the same "Failed to Decrypt data" issue with SharePoint 2016. After opening a support case with Microsoft we found that the server was unable to decrypt the DistributedCache key within the registry here:

    HKLM\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\16.0\Secure\Credentials

    The Solution was to rename the "Credentials" key then reboot the machine. After the reboot, the "Credentials" key will be automatically recreated. Confirm that the correct permissions are set for the "Credentials" key after the reboot:

    The permissions should be: WSS_WPG – Read, WSS_ADMIN_WPG  - full control

    If the permissions were not correct you, you will need to correct them and then reboot again.

    I can confirm that after this I no longer received any Event ID: 6398 Errors in the Application event log.

    Hopefully this will save someone from having to remove and readd the server to the farm.

    • Proposed as answer by Jamests Tuesday, January 22, 2019 10:34 PM
    Thursday, January 17, 2019 11:51 PM