none
FIM 2010 R2 CM - SC issuance works only from the server itself

    Question

  • Hello!

    I am implementing a new installation of FIM 2010 R2 CM in a test environment.

    The test environment is "half-production", as it was built to be as close as possible to the active production environment.

    I have 2 DC servers (2008r2 sp1), a CA server, an MSSQL 2008 R2 server, and an application server for the FIM installation.

    I have done everything that's written in the "Test Lab Guide" document (blogs.technet.com/b/tlgs/archive/2010/11/04/test-lab-guide-demonstrate-fim-cm-2010.aspx), and when I reached the part of "Perform FIM CM subscriber tasks",  

    When renewing sets of certificates, issuing a temporary or permanent smartcard, everything works perfectly, IF AND ONLY IF the operation is done via the fim portal on the fim application server itself!

    If I go to the portal from a regular station (Win7, Win XP, or even another 2008r2sp1 server), I get one of the following errors:

    • The RPC server is unavailable
    • The version of OLE on the client and server machines does not match

    I can't find any relevant log on the CA, end-station, FIM server or the DC for that matter, to help me figure out what's happening. 

    It seems as though the requests don't even "go out" from the station/server to the FIM or to the CA server.

    How can I debug this? What can I check?

    I already set all of the debugging levels to "4" in the web.config file, and edited the registry according to some post I found regarding FIM logging.

    Kind Regards,

    Marom. 

    Tuesday, May 14, 2013 10:16 AM

Answers

  • OK, The issue has been resolved.

    It sure was a weird one:

    Apparently, the applicationHost.config file was modified.

    One of our programmers must have modified it for debugging purposes, and failed to revert the file to the original version.

    Luckily I keep backups of important files such as this file and the web.config file, so I could replace the defected file with the previous one.

    The file is located in this path: %windir%/System32/inetsrv/config

    iisreset is required after the modification of the file, for changes to apply.

    Jacques - Thanks a lot for your time and effort! :)

    • Marked as answer by MaromG Monday, May 20, 2013 3:07 PM
    Monday, May 20, 2013 12:55 PM

All replies

  • Hi Marom,

    Here are a few things you need to check:

    • Have you installed the Fim CM client on the machine you are trying to do the enrolment from?
    • Make sure that you open the x64 version of IE if you installed a x64 version of the FIM CM Client? (do not install the x64 client under windows 8 as IE10 does not have a separate x64 version)
    • Have you set up SPN's for the SQL Server?
    • Have you delegated Kerberos access for the SQL server?

    Regards

    Jacques


    Visit My Blog: http://theidentityguy.blogspot.com/


    • Edited by Jssting Wednesday, May 15, 2013 7:19 AM
    Wednesday, May 15, 2013 7:19 AM
  • Hi Marom,

    Here are a few things you need to check:

    • Have you installed the Fim CM client on the machine you are trying to do the enrolment from?
    • Make sure that you open the x64 version of IE if you installed a x64 version of the FIM CM Client? (do not install the x64 client under windows 8 as IE10 does not have a separate x64 version)
    • Have you set up SPN's for the SQL Server?
    • Have you delegated Kerberos access for the SQL server?

    Regards

    Jacques


    Visit My Blog: http://theidentityguy.blogspot.com/


    Hello, Jacques!

    I have done everything according to this page (including the delegation part : http://technet.microsoft.com/en-us/library/hh230239(v=ws.10).aspx

    The FIM CM client is installed. I've reinstalled it, only to get the same errors.

    The client PC from which I am trying to open the FIM CM portal, is a 32 bit computer, but even when I try accessing the portal from another 2008 r2sp1 server, with the x64 IE browser, I get the same errors.

    I didn't set up SPNs for the SQL server, because I didn't see any reference for this procedure in the documentation. The SPNs are set for the FIM CM server (HTTP/FIM_SRV and HTTP/FIM_SRV.domain.local)

    Regarding the SQL Kerberos delegation, could you please elaborate on that? meaning what exactly I need to configure, and where?

    Thanks in advance,

    Marom.

    Sunday, May 19, 2013 8:34 AM
  • I just want to emphasize the fact that this USED TO work.

    Meaning: up until a few weeks ago, this environment was working properly, and SC issuance could occur via the FIM CM web portal, using any client connected to the domain.

    But suddenly, one day, it stopped working entirely.

    I reinstalled the FIM CM server application, and ran the wizard again, and by that point, we managed to issue certificated and smart cards via the FIM CM portal on the application server itself.

    Hope this helps in trying to figure out a solution to this odd and specific issue :)

    Thanks!

    Sunday, May 19, 2013 9:14 AM
  • Hi,

    I've seen this quite a few times where the solution works for a time and then stops working. In our case it worked until the SQL server was rebooted. From a SQL Delegation perspective you need to ensure that the SQL server is delegated Kerberos permissions for all services to the FIM CM server. Please refer to the screenshot below and http://social.technet.microsoft.com/Forums/en-US/ilm2/thread/7bfb4a34-e294-43e2-82d2-57a618a63e43/ for some handy info posted by Matthias Heil:


    Visit My Blog: http://theidentityguy.blogspot.com/

    Sunday, May 19, 2013 12:44 PM
  • Jacques,

    Thanks for your reply!

    There wasn't any delegation configuration in regards to the SQL server.

    I modified this, according to the screenshot you've provided, and the error is still not resolved.

    Should I be rebooting or restarting anything, for this to work? the FIM server? the CA?

    PS- 

    I also checked the following items, recommended by Matthias:

    Most time I' ve seen "RPC server unavailble" errors something with Authentication /Kerberos Delegation went wrong. Investigate security logs on your CA server for failed logins by anonymous users. In this case Kerberos delegation fails and FIM CM tries anonymously to perform the task.

    • Have you completed the Kerberos constrained delegation for the clmWebpool-Account and the FIM CM server object?
    • Did you try to configure the Kerberos delegation on both accounts not for the dedicated services HOST and RPCSS, but for all services.
    • Have you checked on the FIM CM client the IE configuration:  the IE option Integrated Windows authentication should be enabled. Furthermore the site security settings should allow the Automatic Logon with current user name and password option should be configured

    All of the above, seems to be configured correctly. 

    Thanks,

    Marom.

    Sunday, May 19, 2013 1:45 PM
  • you will need to restart the sql server

    Visit My Blog: http://theidentityguy.blogspot.com/

    Sunday, May 19, 2013 1:53 PM
  • OK.

    I will try that tomorrow. I will need the DBA's approval, and he has already left for the day.

    I am wondering how could it have been working prior to this situation?

    I mean, delegating the SQL server itself was never configured, and it WAS working.

    Intriguing... :)

    I will keep updating, when possible.

    Thanks!

    Sunday, May 19, 2013 2:23 PM
  • you will need to restart the sql server

    Visit My Blog: http://theidentityguy.blogspot.com/

    It seems as though rebooting the SQL server might be problematic. It's a major cluster, with many production instances.

    Is there any other way of checking this, instead of rebooting the server? such as resetting the instance (taking the cluster resource offline and then back online), or resetting something else? or running a command of some sort? 

    The DBAs REALLY don't want to reboot this server....

    Monday, May 20, 2013 8:17 AM
  • Hi Marom,

    Just a few more questions:

    • Have you set the delegation on all of the nodes of the cluster?
    • Have you checked against the security eventlogs. If the delegation is not working as expected you will see it in the logs.
    • Have you done the same delegation on the CA server?

    I also would suggest checking the logs for Kerberos failures and starting with a IISReset on the FIM CM server and well as a CA service restart on the CA Server (if the CA server has an HSM you will need to re-enter the HSM pin as well). As the delegation is done on machine level I suspect that a SQL service restart may not solve the problem. If the delegation still fails you may have to revert to a SQL server restart. As far as I know the new security token is not issued until the reboot is done, But I will verify this in my lab and advise accordingly on the actual config.


    Visit My Blog: http://theidentityguy.blogspot.com/

    Monday, May 20, 2013 8:51 AM
  • Hi,

    In your initial post you mentioned this is a development FIM CM instance. If so could you move the SQL Database to a Development server which you can better test against without having to reboot Production SQL server?

    just a quick thought


    Visit My Blog: http://theidentityguy.blogspot.com/

    Monday, May 20, 2013 8:54 AM
  • Hi,

    In your initial post you mentioned this is a development FIM CM instance. If so could you move the SQL Database to a Development server which you can better test against without having to reboot Production SQL server?

    just a quick thought


    Visit My Blog: http://theidentityguy.blogspot.com/

    Yes, that was my intention. 

    Except for the SQL connection string I have to reconfigure, are there any other places that should be reconfigured after moving the DB?

    Thanks again!

    Marom.

    Monday, May 20, 2013 9:04 AM
  • OK, The issue has been resolved.

    It sure was a weird one:

    Apparently, the applicationHost.config file was modified.

    One of our programmers must have modified it for debugging purposes, and failed to revert the file to the original version.

    Luckily I keep backups of important files such as this file and the web.config file, so I could replace the defected file with the previous one.

    The file is located in this path: %windir%/System32/inetsrv/config

    iisreset is required after the modification of the file, for changes to apply.

    Jacques - Thanks a lot for your time and effort! :)

    • Marked as answer by MaromG Monday, May 20, 2013 3:07 PM
    Monday, May 20, 2013 12:55 PM
  • Glad to hear you have gotten it resolved. If the applicationhost.config file was changed they probably changed the Kerberos Delegation configured for FIM CM in the file. you can check out the "Configure IIS for Kerberos Delegation" section under http://technet.microsoft.com/en-us/library/hh230239(v=ws.10).aspx for more info.

    Visit My Blog: http://theidentityguy.blogspot.com/

    Tuesday, May 21, 2013 12:11 AM
  • Glad to hear you have gotten it resolved. If the applicationhost.config file was changed they probably changed the Kerberos Delegation configured for FIM CM in the file. you can check out the "Configure IIS for Kerberos Delegation" section under http://technet.microsoft.com/en-us/library/hh230239(v=ws.10).aspx for more info.

    Visit My Blog: http://theidentityguy.blogspot.com/

    Thanks, man!

    You are the best! :)

    Tuesday, May 21, 2013 10:09 AM