none
FIM MA and Stopped-server error RRS feed

  • Question

  • Hi All,

    I am getting a stopped-server error on export from the FIM MA.  This doesn't happen with every run, but at random times, for example everything will run fine for 3 - 4 days and the suddenly this error.  It almost seems like a buffer problem.  Interestingly, I can still do an import on the FIM MA as well as sync.  It is only the export that fails with this error. The only way to recover from this is to start all the servers in the solution.

    This is a production deployment running on three servers with Windows Server 2012 and Share Point 2013. The FIM version is 2010 R2.

    The following is logged in the application log on the synchronization server:

    The management agent controller encountered an unexpected error.
     
    "BAIL: MMS(19764): d:\bt\2172\private\source\miis\ma\managed\manhost\manhost.cpp(2089): 0x80230709 (unable to get error text)
    BAIL: MMS(19764): d:\bt\2172\private\source\miis\ma\managed\nathost\nathost.cpp(837): 0x80230020 (unable to get error text)
    BAIL: MMS(19764): d:\bt\2172\private\source\miis\cntrler\cntrler.cpp(3677): 0x80230020 (unable to get error text)
    BAIL: MMS(19764): d:\bt\2172\private\source\miis\ma\managed\manhost\manhost.cpp(2114): 0x80230709 (unable to get error text)
    BAIL: MMS(19764): d:\bt\2172\private\source\miis\ma\managed\nathost\nathost.cpp(869): 0x80230709 (unable to get error text)
    Forefront Identity Manager 4.1.3114.0"

    Have looked at various posts for this type of error but is not closer to a solution.  I am also not sure what is happening to cause this. 

    Has anyone else experienced this or knows why this happens?  Where can one get explanation of the MMS BAIL error codes?

    Thank you

    Johan Marais


    JkM6228

    Friday, July 26, 2013 8:58 AM

Answers

  • Hi,

    it seems that we have found and resolved the cause of our stopped-server-error.  In our case it seemed to be the OEM network card drivers.  I initially posted about this here: http://social.technet.microsoft.com/Forums/en-US/d2dc0116-3b5b-48aa-8d64-7a29da15fcd4/fim-portal-unstable?forum=ilm2.

    The OEM drivers were replaced with the generic MS NDIS driver.  It seems that the OEM driver fails under high transaction loads, specifically with an export on the FIM MA.  My solution has three servers running different components of our FIM solution on a HP G5 and two G6 servers with Windows Server 2012.   Initially I only replaced the network card driver on the G5 which was running the FIM service and FIM portal, but have now also replaced the driver on the two G6s.

    No events were logged to the any event log and the only way to recover from this was to restart all three servers.

    We have replaced the network card drivers about three weeks ago and have not had any problems to date.

    Regards

    Johan Marais


    JkM6228

    • Marked as answer by Johan Marais Friday, October 11, 2013 9:41 AM
    Friday, October 11, 2013 9:40 AM

All replies

  • http://social.technet.microsoft.com/wiki/contents/articles/11331.troubleshooting-fim-r2-stopped-server-error-on-the-fim-service-management-agent.aspx  Here is the post that normally helps resolve FIMMA stopped server in FIM R2.


    Tim Macaulay Security Identity Support Team Support Escalation Engineer


    Friday, July 26, 2013 4:26 PM
  • Tim,

    Thanks for your reply. Will try the steps in the link when it happens again.

    Regards

    Johan Marais


    JkM6228

    Monday, July 29, 2013 4:58 AM
  • Tim,

    Experienced the Stopped-Server error on export of the FIM MA again during the night.  I have tried the steps in the link you have posted, but it didn't work for me.  I still had to restart all three my servers after which it was resolved.

    It almost seem as if this happens on a three day cycle, almost like a buffer filling up. 

    Any other ideas I can try to locate the cause of this?

    Thanks

    Johan Marais


    JkM6228

    Tuesday, July 30, 2013 7:38 AM
  • What is the FIM full version? What hotfix are you up to?

    Dave Nesbitt | Architect | Oxford Computer Group

    Wednesday, July 31, 2013 1:59 PM
  • Dave,

    Apologies for only replying now, was out of office for a day.  To answer your question, I am running FIM 2010 R2  Ver 4.1.3114.0 on Windows Server 2012.  The solution consists of three servers:

    1. Synchronization server and database

    2. FIM Servive DB

    3. FIM Portal, Password registration & reset portal and FIM Service

    Thanks

    Johan Marais


    JkM6228

    Thursday, August 1, 2013 6:09 AM
  • Hi,

    It seems the frequency at which this happens has increased.  Server stopped error occurs now regularly.  The following 3 events are logged on the FIM Synchronization server in the application log:

    FIM Synchronization Error

    Application log

    Log Name:      Application

    Source:        FIMSynchronizationService

    Date:          01/08/2013 08:31:55

    Event ID:      6500

    Task Category: None

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      <FIM Sync server name>

    Description:

    The description for Event ID 6500 from source FIMSynchronizationService cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

     

    If the event originated on another computer, the display information had to be saved with the event.

     

    The following information was included with the event:

     

    An error occurred during export.

    Type: System.Reflection.TargetInvocationException

     

    Message: Exception has been thrown by the target of an invocation.

     

    Stack Trace:    at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck)

       at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)

       at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark)

       at System.Activator.CreateInstance[T]()

       at Microsoft.ResourceManagement.Utilities.SingletonObjectBase`1.get_Instance()

       at Microsoft.ResourceManagement.Utilities.DefaultSingletonObjectClassFactory`2.CreateInstance()

       at Microsoft.ResourceManagement.Utilities.ClassFactoryManager.CreateInstance[T]()

       at MIIS.ManagementAgent.RavenMA.DoOpenDataSourceForExport(String stepData, Guid exportSessionIdentifier)

     

    the message resource is present but the message is not found in the string/message table

     

    Event Xml:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

      <System>

        <Provider Name="FIMSynchronizationService" />

        <EventID Qualifiers="0">6500</EventID>

        <Level>2</Level>

        <Task>0</Task>

        <Keywords>0x80000000000000</Keywords>

        <TimeCreated SystemTime="2013-08-01T06:31:55.000000000Z" />

        <EventRecordID>291358</EventRecordID>

        <Channel>Application</Channel>

        <Computer><FQDN of FIM Sync server></Computer>

        <Security />

      </System>

      <EventData>

        <Data>An error occurred during export.

    Type: System.Reflection.TargetInvocationException

     

    Message: Exception has been thrown by the target of an invocation.

     

    Stack Trace:    at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean&amp; canBeCached, RuntimeMethodHandleInternal&amp; ctor, Boolean&amp; bNeedSecurityCheck)

       at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark&amp; stackMark)

       at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark&amp; stackMark)

       at System.Activator.CreateInstance[T]()

       at Microsoft.ResourceManagement.Utilities.SingletonObjectBase`1.get_Instance()

       at Microsoft.ResourceManagement.Utilities.DefaultSingletonObjectClassFactory`2.CreateInstance()

       at Microsoft.ResourceManagement.Utilities.ClassFactoryManager.CreateInstance[T]()

       at MIIS.ManagementAgent.RavenMA.DoOpenDataSourceForExport(String stepData, Guid exportSessionIdentifier)</Data>

      </EventData>

    </Event>

     

    Log Name:      Application

    Source:        FIMSynchronizationService

    Date:          01/08/2013 08:31:55

    Event ID:      6401

    Task Category: Server

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      <FIM Sync server name>

    Description:

    The management agent controller encountered an unexpected error.

     

     "BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\manhost\manhost.cpp(1379): 0x80230709 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\nathost\nathost.cpp(811): 0x80230020 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\cntrler\cntrler.cpp(3562): 0x80230020 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\manhost\manhost.cpp(2114): 0x80230709 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\nathost\nathost.cpp(869): 0x80230709 (unable to get error text)

    Forefront Identity Manager 4.1.3114.0"

    Event Xml:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

      <System>

        <Provider Name="FIMSynchronizationService" />

        <EventID Qualifiers="49152">6401</EventID>

        <Level>2</Level>

        <Task>3</Task>

        <Keywords>0x80000000000000</Keywords>

        <TimeCreated SystemTime="2013-08-01T06:31:55.000000000Z" />

        <EventRecordID>291359</EventRecordID>

        <Channel>Application</Channel>

        <Computer><FQDN of FIM Sync server></Computer>

        <Security />

      </System>

      <EventData>

        <Data>BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\manhost\manhost.cpp(1379): 0x80230709 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\nathost\nathost.cpp(811): 0x80230020 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\cntrler\cntrler.cpp(3562): 0x80230020 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\manhost\manhost.cpp(2114): 0x80230709 (unable to get error text)

    BAIL: MMS(6284): d:\bt\2172\private\source\miis\ma\managed\nathost\nathost.cpp(869): 0x80230709 (unable to get error text)

    Forefront Identity Manager 4.1.3114.0</Data>

      </EventData>

    </Event>

     

    Log Name:      Application

    Source:        FIMSynchronizationService

    Date:          01/08/2013 08:31:55

    Event ID:      6056

    Task Category: Management Agent Run Profile

    Level:         Error

    Keywords:      Classic

    User:          N/A

    Computer:      <FIM Sync Server name>

    Description:

    The management agent "Telkom FIMMA" failed on run profile "E" because the server encountered errors.

    Event Xml:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

      <System>

        <Provider Name="FIMSynchronizationService" />

        <EventID Qualifiers="49152">6056</EventID>

        <Level>2</Level>

        <Task>1</Task>

        <Keywords>0x80000000000000</Keywords>

        <TimeCreated SystemTime="2013-08-01T06:31:55.000000000Z" />

        <EventRecordID>291360</EventRecordID>

        <Channel>Application</Channel>

        <Computer><FQDN of FIM Sync server></Computer>

        <Security />

      </System>

      <EventData>

        <Data>Telkom FIMMA</Data>

        <Data>E</Data>

        <Data>No Additional Data</Data>

      </EventData>

    </Event>

    Any idea as to what is wrong, or how to resolve this?

    Thanks

    Johan Marais


    JkM6228

    Thursday, August 1, 2013 6:56 AM
  • You're a couple of hotfixes behind the current (which is 4.1.3451.0) plus build 4.1.3441.0 (http://support.microsoft.com/kb/2832389/en-gb) does have a fix for a FIM MA "stopped server" issue. It might not be your issue but I would recommend patching to the latest version anyway, just in case you need to escalate to PSS

    Good luck

    Dave


    Dave Nesbitt | Architect | Oxford Computer Group

    Thursday, August 1, 2013 8:58 AM
  • Dave,

    I have applied the latest patch yesterday and my FIM 2010 R2 version is now 4.1.3451.0.  Will monitor the situation and see if the error returns.

    Thanks for your help so far.

    Regards

    Johan Marais


    JkM6228

    Friday, August 2, 2013 4:39 AM
  • Dave,

    The stopped Server error seems to be resolved, but I will wait for another day or so just to be sure. But since applying the patch, I get an unexpected error when synchronizing certain groups to a SQL database.  The error description is just "unexpected-error" on the export flow and it lists the FunctionLibrary.dll

    Must point out, that it is not all groups which encounter this problem.  Also note that the groups with this problem have more than 500 members.

    The following error event is logged in  the Application log on the Synchronization server:

    Log Name:      Application
    Source:        FIMSynchronizationService
    Date:          05/08/2013 09:21:15
    Event ID:      6301
    Task Category: Server
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      <FQDN of server>
    Description:
    The server encountered an unexpected error in the synchronization engine:
     
     "BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15192): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15042): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15707): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15640): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(1400): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(1503): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(1482): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(3118): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(3327): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): D:\bt\5417\private\source\MIIS\server\shared\inc\ScriptManager.h(821): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\eafam.cpp(1708): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(3017): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(3276): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(1878): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(2101): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\eaf.cpp(1417): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\eaf.cpp(657): 0x80230405 (The operation failed because the object cannot be found)
    ERR_: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(5252): 0x80230405 - export-flow failed 0x80230405
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(5253): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(4862): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(10880): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(7121): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(4364): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(8685): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(3792): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(3573): 0x80230405 (The operation failed because the object cannot be found)
    Forefront Identity Manager 4.1.3451.0"
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="FIMSynchronizationService" />
        <EventID Qualifiers="49152">6301</EventID>
        <Level>2</Level>
        <Task>3</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-08-05T07:21:15.000000000Z" />
        <EventRecordID>293079</EventRecordID>
        <Channel>Application</Channel>
        <Computer><FQDN of Server></Computer>
        <Security />
      </System>
      <EventData>
        <Data>BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15192): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15042): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15707): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\sqlstore\csobj.cpp(15640): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(1400): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(1503): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(1482): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(3118): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(21744): d:\bt\5417\private\source\miis\server\rules\scriptmanagerimpl.cpp(3327): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): D:\bt\5417\private\source\MIIS\server\shared\inc\ScriptManager.h(821): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\eafam.cpp(1708): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(3017): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(3276): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(1878): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\amexec.cpp(2101): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\eaf.cpp(1417): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\rules\eaf.cpp(657): 0x80230405 (The operation failed because the object cannot be found)
    ERR_: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(5252): 0x80230405 - export-flow failed 0x80230405
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(5253): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(4862): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(10880): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(7121): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(4364): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(8685): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(3792): 0x80230405 (The operation failed because the object cannot be found)
    BAIL: MMS(5264): d:\bt\5417\private\source\miis\server\sync\synccoreimp.cpp(3573): 0x80230405 (The operation failed because the object cannot be found)
    Forefront Identity Manager 4.1.3451.0</Data>
      </EventData>
    </Event> 

    Thanks

    Johan Marais


    JkM6228

    Monday, August 5, 2013 7:59 AM
  • HI Johan,

    Did you patch all the services (SyncEngine, Portal, Service, etc, etc) with the same patches? If not, they should be patched to the same level. After patching, also ensure that you do a refresh schema on the FIM MA then a full import full sync on all the MA's


    Visit My Blog: http://theidentityguy.blogspot.com/

    Tuesday, August 6, 2013 11:49 AM
  • JSSting,

    Thanks for reply.  Yes I have applied the same patch to all components and verified that by checking the version number across them.  But I haven't done the schema refresh and FI and FS.

    is this standard procedure after a patch is applied?

    Will do that.

    Regards

    Johan marais


    JkM6228

    Tuesday, August 6, 2013 11:57 AM
  • Hi Johan,

    I typically follow this procedure with doing updates especially if you skipped a few patches in between the current patch and the version you have loaded.


    Visit My Blog: http://theidentityguy.blogspot.com/

    Tuesday, August 6, 2013 3:49 PM
  • All,

    Thanks for your help.  After all of this I am still getting the Stopped-server error :-).  I have logged a call with MS PSS.

    Will provide feedback about that.

    Regards

    Johan Marais


    JkM6228

    Wednesday, August 7, 2013 9:56 AM
  • Hi,

    We have similar issue on our FIM 2010 R2 environment (4 servers, FIM Sync, FIM Service/Portal, FIM SSPR and SQL 2008R2, FIM version 4.1.3451) but "stopper-server" error actually occurs when running either Full or Delta Sync on FIM MA. Full and Delta import runs ok and Export as well. All run profiles on AD MA runs ok as well so it´s only the Sync run profile that fails. Sync run just stops since FIM Synchronization Service stops.

    Some examples from the FIM Operational log at FIM Sync Server:

    *******************

    HRESULT: '0x80070002' Source: 'd:\bt\5417\private\source\miis\shared\utils\libutils.cpp(10808)'  Thread ID: '0x8d4' Additional Info: ''

    HRESULT: '0x80070057' Source: 'd:\bt\5417\private\source\miis\shared\xmlpe\xparse.cpp(436)'  Thread ID: '0x8d4' Additional Info: ''

    HRESULT: '0x80070057' Source: 'd:\bt\5417\private\source\miis\shared\xmlpe\xstack.cpp(405)'  Thread ID: '0x8d4' Additional Info: ''

    HRESULT: '0x80070057' Source: 'd:\bt\5417\private\source\miis\shared\maxml\cdext.cpp(416)'  Thread ID: '0x8d4' Additional Info: ''

    HRESULT: '0x0' Source: 'd:\bt\5417\private\source\miis\server\mgmt\perfmon\prfdata.cpp(654)'  Thread ID: '0x7e0' Additional Info: ''

    HRESULT: '0x80070002' Source: 'd:\bt\5417\private\source\miis\server\mgmt\perfmon\prfdata.cpp(956)'  Thread ID: '0x7e0' Additional Info: ''

    ***********************

    Application and System logs do not contain anything revealing.

    We have also tried some tips and tricks including the ones mentioned in this thread to fix this but without success:

    http://social.technet.microsoft.com/Forums/en-US/cd5d3974-ddf3-4296-95a6-b71d413fa703/management-agent-encountered-an-error-exporting-to-the-connected-directory

    http://social.technet.microsoft.com/wiki/contents/articles/18544.troubleshooting-failed-to-retrieve-schema-and-event-id-6331.aspx

    http://social.technet.microsoft.com/wiki/contents/articles/11331.troubleshooting-fim-r2-stopped-server-error-on-the-fim-service-management-agent.aspx

    Connector Spaces for both FIM MA and AD MA has been deleted but does not help to fix the issues. We have also deleted whole FIM MA and restored it from backup but this does not help either. Schema updates were done before running FIM MA Full Import and Full Sync.

    Therefore we have also opened a support case with MS PSS. When and if we get solution to fix this, I´ll post it here too.

    -Pappa75

    Friday, August 16, 2013 5:51 AM
  • I usually get this error when i have more than one step in my run profiles. For Exampe if i have a run profile with a step Import and another step sync and another step export.

    I usually then create a run profile for each step separately and then put a pause or something for about 10 seconds between the run profiles. This usually stops the issue for me atleast.

    Try doing that :)


    Regards Furqan Asghar

    Sunday, August 18, 2013 7:55 AM
  • Furqan Asghar,

    Thanks for your reply.  I only have single step run profiles. I have a case open with PSS.

    Will provide feedback once we know what is causing this and how to fix it.

    Regards

    Johan Marais


    JkM6228

    Monday, August 19, 2013 9:03 AM
  • Thanks a lot Johan.

    I will be waiting for your feedback, it will help a lot.


    Regards Furqan Asghar

    Monday, August 19, 2013 9:19 AM
  • I was able to locate the cause for the Sync run profile which fails with "stopped-server" error. I investigated some of the objects in the connector space of FIM MA and found out that two of the outbound synchronization rules that were using Outbound Scoping Filter was causing this. I run preview of Full Sync for these two synchronization rules and both failed with the following error code in the popup window:

    **********************

    Unable to get preview XML from server. The remote procedure call failed. (Exception from HRESULT: 0x800706BE)

    **********************

    At the same time, the FIM Sync Service stopped. Then I tested to just delete these two synchronization rules from the FIM Portal and run again FULL Import and then FULL Sync and this time Sync run profile worked with success.

    I also tested to recreate these two synchronization rules with Outbound Scoping Filter again and run again FULL Import (with success) and then FULL Sync which failed again with "stopped-server" error. As a final test, I once again deleted these two synchronization rules with Outbound Scoping Filter and run both run profiles without issues.

    Then I decided to create two more "classic" synchronization rules (without Outbound Scoping Filter) for Exchange mailbox provisioning in addition to our existing sync rules (for user/service account provisioning) and run again both FULL Import and FULL Sync without any issues; users/service account are again successfully provisioned to AD with or without mailbox based on certain attribute value. :) So the cause for our issue was the Outbound Scoping Filter if used in synchronization rule configuration but the root cause for such behaviour is not yet found by us or by MS Premium Support. It is suspected that the .NET might have something to do with this.... I´ll provide feedback when we know more.

    -Pappa75

    Tuesday, August 20, 2013 6:06 AM
  • Hi,

    Still busy with MS PSS on this. No solution yet. We have implemented a lot of things which improved the stability. So far running for a week without errors.

    Regards

    Johan Marais


    JkM6228

    Friday, September 27, 2013 10:36 AM
  • Hi,

    it seems that we have found and resolved the cause of our stopped-server-error.  In our case it seemed to be the OEM network card drivers.  I initially posted about this here: http://social.technet.microsoft.com/Forums/en-US/d2dc0116-3b5b-48aa-8d64-7a29da15fcd4/fim-portal-unstable?forum=ilm2.

    The OEM drivers were replaced with the generic MS NDIS driver.  It seems that the OEM driver fails under high transaction loads, specifically with an export on the FIM MA.  My solution has three servers running different components of our FIM solution on a HP G5 and two G6 servers with Windows Server 2012.   Initially I only replaced the network card driver on the G5 which was running the FIM service and FIM portal, but have now also replaced the driver on the two G6s.

    No events were logged to the any event log and the only way to recover from this was to restart all three servers.

    We have replaced the network card drivers about three weeks ago and have not had any problems to date.

    Regards

    Johan Marais


    JkM6228

    • Marked as answer by Johan Marais Friday, October 11, 2013 9:41 AM
    Friday, October 11, 2013 9:40 AM