none
SharePoint 2013 so many 6398 errors RRS feed

  • Question

  • Why SharePoint 2013 has so many 6398 errors?

    Here are some errors:

    ==============================================================================================================================================================

    DEV01 6398 Critical Microsoft-SharePoint Products-SharePoint Foundation Application 2012-11-13 10:35:06 AM

    The Execute method of job definition Microsoft.Office.Server.UserProfiles.LMTRepopulationJob (ID f6b8af32-0dbb-4459-8f0b-f9ed376e9324) threw an exception. More information is included below.

    Unexpected exception in FeedCacheService.IsRepopulationNeeded: Communications with the cache cluster has experienced a delay past the timeout value,please increase the RequestTimeout of the client..

    ==============================================================================================================================================================
    DEV01 6398 Critical Microsoft-SharePoint Products-SharePoint Foundation Application 2012-11-13 10:45:05 AM

    The Execute method of job definition Microsoft.Office.Server.UserProfiles.LMTRepopulationJob (ID f6b8af32-0dbb-4459-8f0b-f9ed376e9324) threw an exception. More information is included below.

    An update conflict has occurred, and you must re-try this action. The object LMTRepopulationJob Name=User Profile Service Application_LMTRepopulationJob was updated by WS12DC\SP_ServiceApps, in the w3wp (4952) process, on machine DEV01.  View the tracing log for more information about the conflict.

    ==============================================================================================================================================================
    DEV01 6398 Critical Microsoft-SharePoint Products-SharePoint Foundation Application 2012-11-13 11:04:53 AM

    The Execute method of job definition Microsoft.Office.Server.Search.Monitoring.HealthStatUpdateJobDefinition (ID 8bda72f6-0c4d-41c3-84c1-2d4e7adb790b) threw an exception. More information is included below.

    A connection was successfully established with the server, but then an error occurred during the pre-login handshake. (provider: SSL Provider, error: 0 - The handle specified is invalid)

    ==============================================================================================================================================================

    Thursday, November 15, 2012 3:02 AM

Answers

  • I think I probably figured out what is my issue,  I know that someone having this issue because of other reason, but my reason is, the FeedCacheService.BulkLMTUpdate: Region not found happened all the time when I restart my 2012 server, because I'm doing development, so I need to restart server everyday, every time when restart server, the AppFabric Caching Service restart, and the cache region will rebuild, and there is no LMT region, and then this exception happened.

    so if you look at the code in FeedCacheService.BulkLMTUpdate, it will

    call FeedCacheImplementation.BulkLMTUpdate, in FeedCacheImplementation.BulkLMTUpdate, it calls 

    this.PutInLMTRegion(CacheIdentifier, this.GetCacheLMTTTL(LastModifiedUtcTime), null);

    
    

    this cause the region not found exception, and it throw SPDistributedCacheRegionNotFoundException exception, and try to call another method to recreate the LMT region:

    this.EnsureAndRepopulateLMTRegion(false);

    after the LMT region recreated. and the region not found event will never happened, unless the AppFabric Caching Service restart.

    I just don't understand why MS developers assuming the LMT region always be there, how about the first running of the cache service after the fresh installation? just like my case. and also sometime I got another 6398 error, said the cache cluster is probably down, I believe this is also related with the AppFabric Caching Service, looks like AppFabric Caching Service is not stable.

    some links about Distributed Cache service in SharePoint Server 2013

    http://technet.microsoft.com/en-us/library/jj219700.aspx

    http://technet.microsoft.com/en-us/library/jj219560.aspx

    and here is the code BulkLMTUpdate:

    internal bool BulkLMTUpdate(Guid callerID, List<KeyValuePair<FeedCacheEntity, LMTEntry>> lastModifiedTimes)
    {
        bool flag;
        if (lastModifiedTimes == null)
        {
            throw new ArgumentNullException("lastModifiedTimes");
        }
        using (new SPMicroblogUsageMonitoredScope("FeedCacheImplementation.BulkLMTUpdate", SPMicroblogUsageField.FeedCacheImpl_Duration))
        {
            try
            {
                foreach (KeyValuePair<FeedCacheEntity, LMTEntry> pair in lastModifiedTimes)
                {
                    this.PutInLMTRegion(pair.Key.CacheIdentifier, pair.Value, this.GetCacheLMTTTL(pair.Value.LastModifiedUtcTime), null);
                    AddToLocalUrls(pair.Key);
                }
                this.LMTCacheWrapper.Put(callerID.ToString("N"), new LMTEntry(DateTime.UtcNow).Serialze(), FeedCacheManager.Constants.ConfigTTLTimeSpan, this.LastModifiedTimeRegionName);
            }
            catch (Exception exception)
            {
                if (exception is SPDistributedCacheRegionNotFoundException)
                {
                    ULS.SendTraceTag(0x10645f, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.Medium, "FeedCacheImplementation.BulkLMTUpdate() - LMT region missing");
                    this.EnsureAndRepopulateLMTRegion(false);
                    try
                    {
                        foreach (KeyValuePair<FeedCacheEntity, LMTEntry> pair2 in lastModifiedTimes)
                        {
                            this.PutInLMTRegion(pair2.Key.CacheIdentifier, pair2.Value, this.GetCacheLMTTTL(pair2.Value.LastModifiedUtcTime), null);
                        }
                        this.DataCacheWrapper.Put(callerID.ToString("N"), new LMTEntry(DateTime.UtcNow).Serialze(), FeedCacheManager.Constants.ConfigTTLTimeSpan, this.LastModifiedTimeRegionName);
                        goto Label_01C1;
                    }
                    catch (Exception exception2)
                    {
                        ULS.SendTraceTag(0x20178c, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.High, "FeedCacheImplementation.BulkLMTUpdate() - Unexpected exception while trying to update LMT in bulk: {0}", new object[] { exception2 });
                        throw;
                    }
                }
                ULS.SendTraceTag(0x151821, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.High, "FeedCacheImplementation.BulkLMTUpdate() - Unexpected exception while trying to update LMT in bulk: {0}", new object[] { exception });
                throw;
            }
        Label_01C1:
            flag = true;
        }
        return flag;
    }

     

     


    • Edited by 10031103 Wednesday, November 21, 2012 7:12 PM
    • Marked as answer by GuYuming Thursday, November 22, 2012 2:33 AM
    Wednesday, November 21, 2012 7:07 PM
  • you have answered your own question, according to the second link about distributed cache in sharepoint 2013 you posted, by default, the User Profile Service Application - Feed Cache Repopulation Job timer job is configured to run every 5 minutes .

    So, the LMT cache will be repopulated by the timer job within 5 minutes after restart.  Hence, MS developers can assume it is always there (although you will still get the exception if you need the cache data before first run of the timer job after restart). Please ensure that the timer job is configured properly in your system.

    • Marked as answer by GuYuming Monday, November 26, 2012 4:38 AM
    Thursday, November 22, 2012 2:32 AM

All replies

  • Another one, this one happened all the time:

    DEV01 6398 Critical Microsoft-SharePoint Products-SharePoint Foundation Application 2012-11-14 10:00:19 PM

    The Execute method of job definition Microsoft.Office.Server.UserProfiles.LMTRepopulationJob (ID d55dfdda-c513-415b-9809-77aba0cce788) threw an exception. More information is included below.

    Unexpected exception in FeedCacheService.BulkLMTUpdate: Region not found..

    Thursday, November 15, 2012 3:12 AM
  • Hi

    User profile synchonisation is not a part of foundation version.What are you trying to do ?

    Thanks


    Praveen

    Thursday, November 15, 2012 9:46 AM
  • Hi

    User profile synchonisation is not a part of foundation version.What are you trying to do ?

    Thanks


    Praveen

    Hi Praveen,

    I am not doing anything, I just had a fresh installation of Windows Server 2012, SQL Server 2012, and SharePoint Server 2013, I haven't do anything, after the fresh installation, every time when I login, I always see some 6398 Critical errors in the Event list, and the "Region not found" happened all the times.

    Thursday, November 15, 2012 2:19 PM
  • Verify the following steps

    1)Have you deployed all the prerequisities

    2)Have you congigured the Service Application(through wizard or powershell)

    3)Is the central admin site working

    4)Are you able to create web application and Site collection


    Praveen


    Friday, November 16, 2012 9:36 AM
  • Verify the following steps

    1)Have you deployed all the prerequisities

    2)Have you congigured the Service Application(through wizard or powershell)

    3)Is the central admin site working

    4)Are you able to create web application and Site collection


    Praveen


    Hi Praveen,

    The prerequisites are deployed, Service Application configured through wizard, central admin works fine, and I can create web application and site collection, all things are good.

    Just the 6398 error keep annoying me. is this related with the AppFabric Caching Service?

    Friday, November 16, 2012 4:45 PM
  • Another one

    DEV01 6398 Critical Microsoft-SharePoint Products-SharePoint Foundation Application 2012-11-19 9:45:06 AM

    The Execute method of job definition Microsoft.Office.Server.UserProfiles.LMTRepopulationJob (ID d55dfdda-c513-415b-9809-77aba0cce788) threw an exception. More information is included below.

    Unexpected exception in FeedCacheService.IsRepopulationNeeded: Unable to create a DataCache. SPDistributedCache is probably down..

    Monday, November 19, 2012 3:19 PM
  • Hi

    From the error .I guess your UserProfile Sync is not working because App fabric caching is not working .Verify if you can add users to the profile store and also you are able to sync

    These errors you are getting when navigationg to any page or these errors are from event or log viewer

    Thanks


    Praveen

    Tuesday, November 20, 2012 5:05 AM
  • Hi

    From the error .I guess your UserProfile Sync is not working because App fabric caching is not working .Verify if you can add users to the profile store and also you are able to sync

    These errors you are getting when navigationg to any page or these errors are from event or log viewer

    Thanks


    Praveen


    this mean, we must update our profile ?
    Tuesday, November 20, 2012 5:12 AM
  • I think I probably figured out what is my issue,  I know that someone having this issue because of other reason, but my reason is, the FeedCacheService.BulkLMTUpdate: Region not found happened all the time when I restart my 2012 server, because I'm doing development, so I need to restart server everyday, every time when restart server, the AppFabric Caching Service restart, and the cache region will rebuild, and there is no LMT region, and then this exception happened.

    so if you look at the code in FeedCacheService.BulkLMTUpdate, it will

    call FeedCacheImplementation.BulkLMTUpdate, in FeedCacheImplementation.BulkLMTUpdate, it calls 

    this.PutInLMTRegion(CacheIdentifier, this.GetCacheLMTTTL(LastModifiedUtcTime), null);

    
    

    this cause the region not found exception, and it throw SPDistributedCacheRegionNotFoundException exception, and try to call another method to recreate the LMT region:

    this.EnsureAndRepopulateLMTRegion(false);

    after the LMT region recreated. and the region not found event will never happened, unless the AppFabric Caching Service restart.

    I just don't understand why MS developers assuming the LMT region always be there, how about the first running of the cache service after the fresh installation? just like my case. and also sometime I got another 6398 error, said the cache cluster is probably down, I believe this is also related with the AppFabric Caching Service, looks like AppFabric Caching Service is not stable.

    some links about Distributed Cache service in SharePoint Server 2013

    http://technet.microsoft.com/en-us/library/jj219700.aspx

    http://technet.microsoft.com/en-us/library/jj219560.aspx

    and here is the code BulkLMTUpdate:

    internal bool BulkLMTUpdate(Guid callerID, List<KeyValuePair<FeedCacheEntity, LMTEntry>> lastModifiedTimes)
    {
        bool flag;
        if (lastModifiedTimes == null)
        {
            throw new ArgumentNullException("lastModifiedTimes");
        }
        using (new SPMicroblogUsageMonitoredScope("FeedCacheImplementation.BulkLMTUpdate", SPMicroblogUsageField.FeedCacheImpl_Duration))
        {
            try
            {
                foreach (KeyValuePair<FeedCacheEntity, LMTEntry> pair in lastModifiedTimes)
                {
                    this.PutInLMTRegion(pair.Key.CacheIdentifier, pair.Value, this.GetCacheLMTTTL(pair.Value.LastModifiedUtcTime), null);
                    AddToLocalUrls(pair.Key);
                }
                this.LMTCacheWrapper.Put(callerID.ToString("N"), new LMTEntry(DateTime.UtcNow).Serialze(), FeedCacheManager.Constants.ConfigTTLTimeSpan, this.LastModifiedTimeRegionName);
            }
            catch (Exception exception)
            {
                if (exception is SPDistributedCacheRegionNotFoundException)
                {
                    ULS.SendTraceTag(0x10645f, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.Medium, "FeedCacheImplementation.BulkLMTUpdate() - LMT region missing");
                    this.EnsureAndRepopulateLMTRegion(false);
                    try
                    {
                        foreach (KeyValuePair<FeedCacheEntity, LMTEntry> pair2 in lastModifiedTimes)
                        {
                            this.PutInLMTRegion(pair2.Key.CacheIdentifier, pair2.Value, this.GetCacheLMTTTL(pair2.Value.LastModifiedUtcTime), null);
                        }
                        this.DataCacheWrapper.Put(callerID.ToString("N"), new LMTEntry(DateTime.UtcNow).Serialze(), FeedCacheManager.Constants.ConfigTTLTimeSpan, this.LastModifiedTimeRegionName);
                        goto Label_01C1;
                    }
                    catch (Exception exception2)
                    {
                        ULS.SendTraceTag(0x20178c, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.High, "FeedCacheImplementation.BulkLMTUpdate() - Unexpected exception while trying to update LMT in bulk: {0}", new object[] { exception2 });
                        throw;
                    }
                }
                ULS.SendTraceTag(0x151821, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.High, "FeedCacheImplementation.BulkLMTUpdate() - Unexpected exception while trying to update LMT in bulk: {0}", new object[] { exception });
                throw;
            }
        Label_01C1:
            flag = true;
        }
        return flag;
    }

     

     


    • Edited by 10031103 Wednesday, November 21, 2012 7:12 PM
    • Marked as answer by GuYuming Thursday, November 22, 2012 2:33 AM
    Wednesday, November 21, 2012 7:07 PM
  • you have answered your own question, according to the second link about distributed cache in sharepoint 2013 you posted, by default, the User Profile Service Application - Feed Cache Repopulation Job timer job is configured to run every 5 minutes .

    So, the LMT cache will be repopulated by the timer job within 5 minutes after restart.  Hence, MS developers can assume it is always there (although you will still get the exception if you need the cache data before first run of the timer job after restart). Please ensure that the timer job is configured properly in your system.

    • Marked as answer by GuYuming Monday, November 26, 2012 4:38 AM
    Thursday, November 22, 2012 2:32 AM
  • Hi, Cool number.

    But I got some questions. Actually, I don't understand what the FeedCacheService.BulkLMTUpdate is. Is that kinda dll? or file? or library stuff? I don't get a meaning of your code. Do I need to recorrect recent BulkLMTUpdate?(file? dll? I don't know) If not, Do I need to create a BulkLMTUpdate? and if it could be, where can I find a file? If that exsits, pls let me know the following directory. 6398 makes me crazy. pls help me out.........

    Thank You.

    Friday, December 21, 2012 12:23 AM
  • Hi, GuYuming

    I don't understand your sentense that the configuration of proper timer job in a system. Is there any proper manual to configure timer job? If there is, pls tell me how to configure the timer job of the cache. I know the default value of repopulation job is 'run every 5 minutes.' Is that movable value depend on my system? If it is, how can I find out and ensure correct value? 6398 makes me crazy pls help me out...

    thank you

    Friday, December 21, 2012 12:35 AM
  • I was getting the "increase the RequestTimeout" message in Event ID 6398, which led me to this forum post. Even tried following saantil's blog post that GuYuming points out, but it didn't help. 

    In my case, it turned out to be that the three servers in my farm which had caching enabled did not have the same amount of physical memory in them and, thus, the Cache Host hadn't configured for the same size on all servers. Soon as I manually adjusted the size to match across the three servers, the error went away. Picked up on this from the Plan and use the Distributed Cache service in SharePoint Server 2013 model.

    Monday, August 5, 2013 8:19 PM
  • I think I probably figured out what is my issue,  I know that someone having this issue because of other reason, but my reason is, the FeedCacheService.BulkLMTUpdate: Region not found happened all the time when I restart my 2012 server, because I'm doing development, so I need to restart server everyday, every time when restart server, the AppFabric Caching Service restart, and the cache region will rebuild, and there is no LMT region, and then this exception happened.

    so if you look at the code in FeedCacheService.BulkLMTUpdate, it will

    call FeedCacheImplementation.BulkLMTUpdate, in FeedCacheImplementation.BulkLMTUpdate, it calls 

    this.PutInLMTRegion(CacheIdentifier, this.GetCacheLMTTTL(LastModifiedUtcTime), null);

    
    

    this cause the region not found exception, and it throw SPDistributedCacheRegionNotFoundException exception, and try to call another method to recreate the LMT region:

    this.EnsureAndRepopulateLMTRegion(false);

    after the LMT region recreated. and the region not found event will never happened, unless the AppFabric Caching Service restart.

    I just don't understand why MS developers assuming the LMT region always be there, how about the first running of the cache service after the fresh installation? just like my case. and also sometime I got another 6398 error, said the cache cluster is probably down, I believe this is also related with the AppFabric Caching Service, looks like AppFabric Caching Service is not stable.

    some links about Distributed Cache service in SharePoint Server 2013

    http://technet.microsoft.com/en-us/library/jj219700.aspx

    http://technet.microsoft.com/en-us/library/jj219560.aspx

    and here is the code BulkLMTUpdate:

    internal bool BulkLMTUpdate(Guid callerID, List<KeyValuePair<FeedCacheEntity, LMTEntry>> lastModifiedTimes)
    {
        bool flag;
        if (lastModifiedTimes == null)
        {
            throw new ArgumentNullException("lastModifiedTimes");
        }
        using (new SPMicroblogUsageMonitoredScope("FeedCacheImplementation.BulkLMTUpdate", SPMicroblogUsageField.FeedCacheImpl_Duration))
        {
            try
            {
                foreach (KeyValuePair<FeedCacheEntity, LMTEntry> pair in lastModifiedTimes)
                {
                    this.PutInLMTRegion(pair.Key.CacheIdentifier, pair.Value, this.GetCacheLMTTTL(pair.Value.LastModifiedUtcTime), null);
                    AddToLocalUrls(pair.Key);
                }
                this.LMTCacheWrapper.Put(callerID.ToString("N"), new LMTEntry(DateTime.UtcNow).Serialze(), FeedCacheManager.Constants.ConfigTTLTimeSpan, this.LastModifiedTimeRegionName);
            }
            catch (Exception exception)
            {
                if (exception is SPDistributedCacheRegionNotFoundException)
                {
                    ULS.SendTraceTag(0x10645f, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.Medium, "FeedCacheImplementation.BulkLMTUpdate() - LMT region missing");
                    this.EnsureAndRepopulateLMTRegion(false);
                    try
                    {
                        foreach (KeyValuePair<FeedCacheEntity, LMTEntry> pair2 in lastModifiedTimes)
                        {
                            this.PutInLMTRegion(pair2.Key.CacheIdentifier, pair2.Value, this.GetCacheLMTTTL(pair2.Value.LastModifiedUtcTime), null);
                        }
                        this.DataCacheWrapper.Put(callerID.ToString("N"), new LMTEntry(DateTime.UtcNow).Serialze(), FeedCacheManager.Constants.ConfigTTLTimeSpan, this.LastModifiedTimeRegionName);
                        goto Label_01C1;
                    }
                    catch (Exception exception2)
                    {
                        ULS.SendTraceTag(0x20178c, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.High, "FeedCacheImplementation.BulkLMTUpdate() - Unexpected exception while trying to update LMT in bulk: {0}", new object[] { exception2 });
                        throw;
                    }
                }
                ULS.SendTraceTag(0x151821, ULSCat.msoulscat_SPS_FeedCache, ULSTraceLevel.High, "FeedCacheImplementation.BulkLMTUpdate() - Unexpected exception while trying to update LMT in bulk: {0}", new object[] { exception });
                throw;
            }
        Label_01C1:
            flag = true;
        }
        return flag;
    }

     

     


    Great analysis there!

    This made me write a fix for the error.

    Code that runs Repopulation should be built to console app (for example RegionFeeder.exe):

    using Microsoft.SharePoint.Administration;
    using Microsoft.SharePoint.DistributedCaching;
    using System;
    using System.Collections;
    using System.Collections.Generic;
    using System.Linq;
    using System.Reflection;
    using System.Text;
    using System.Threading.Tasks;
    
    namespace RegionFeeder
    {
        class Program
        {
            static int attempts = 0;
            static void Main(string[] args)
            {
                attempts++;
    
                //TODO:Generalize so that it works with any version of SP
                Type type = Type.GetType("Microsoft.Office.Server.Administration.UserProfileServiceProxy, Microsoft.Office.Server.UserProfiles, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c");
                MethodInfo MI = typeof(SPServiceCollection).GetMethod("GetValue", new[] { typeof(string) });
                MethodInfo genericMethod = MI.MakeGenericMethod(new[] { type });
                object userProfileServiceProxy = genericMethod.Invoke(SPFarm.Local.Services, new[] { string.Empty });
                IEnumerable userProfileApplicationProxyCollection = (IEnumerable)type.GetProperty("ApplicationProxies").GetValue(userProfileServiceProxy);
                object targetUserProfileApplicationProxy = null;
                //TODO:Find UPA ID somehow
                foreach (object userProfileApplicationProxy in userProfileApplicationProxyCollection)
                {
                    targetUserProfileApplicationProxy = userProfileApplicationProxy;
                    break;
                }
                Type typeFeedImpl = Type.GetType("Microsoft.Office.Server.FeedCache.FeedCacheImplementation, Microsoft.Office.Server.UserProfiles, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c");
                object feedImpl = Activator.CreateInstance(
                    typeFeedImpl,
                    BindingFlags.NonPublic | BindingFlags.Instance,
                    null,
                    new object[] { targetUserProfileApplicationProxy },
                    null);
                MethodInfo ensureMethod = feedImpl.GetType().GetMethod(
                    "EnsureAndRepopulateLMTRegion",
                    BindingFlags.NonPublic | BindingFlags.Instance,
                    null,
                    new[] { typeof(bool) },
                    null);
    
                try
                {
                    ensureMethod.Invoke(feedImpl, new object[] { false });
                }
                catch (TargetInvocationException tie)
                {
                    if ((tie.InnerException.GetType().Name == typeof(SPDistributedCacheServerStillConfiguringException).Name) &&
                        (attempts < 20))
                    {
                        Main(args);
                    }
                    else
                    {
                        throw;
                    }
                }
            }
        }
    }
    

    Then create a task scheduler task which will trigger on App fabric service start. Add NuGet Microsoft.Win32.TaskScheduler. It will run the app and Repopulate LMT region. This should be run only once to create the task:

    using Microsoft.Win32.TaskScheduler;
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Threading.Tasks;
    
    namespace TaskCreator
    {
        class Program
        {
            static void Main(string[] args)
            {
                using (TaskService ts = new TaskService())
                {
                    TaskDefinition td = ts.NewTask();
                    td.RegistrationInfo.Description = "Repopulates LMT Region after restart of distibuted cache service";
                    EventTrigger trigger = new EventTrigger();
                    trigger.Subscription=
                        @"<QueryList> 
                           <Query Id='0'> 
                              <Select Path='System'> 
                                 *[EventData[Data[@Name='param1'] and (Data='AppFabric Caching Service')]] 
                                 and
                                 *[EventData[Data[@Name='param2'] and (Data='running')]] 
                               </Select> 
                           </Query> 
                        </QueryList>";
                    td.Triggers.Add(trigger);
                    td.Actions.Add(new ExecAction("c:\\RegionFeeder.exe"));
                    ts.RootFolder.RegisterTaskDefinition(@"Repopulator", td);
                }
            }
        }
    }
    

    Monday, September 2, 2019 10:35 AM