none
High CPU usage in MSIExec due to enumeration of print GUIDs in HKU\.Default\Software

    Pregunta

  • Hi,

    Our setup:

    Windows Server 2008 x64 terminal servers with Citrix XenApp 5.5

    Windows Server 2008 R2 file/print servers

    x64 printer drivers

    Network printers are mapped with RES PowerFuse

     

    Our problem:

    When installing or updating products using msiexec the process uses hours to complete, with about 50% cpu usage on a dual cpu terminal server.

    We used process explorer to monitor the msiexec process to find out what it was doing, and found that it was enumerating a huge amount of GUIDs in the HKLM\Software\Microsoft\Windows NT\Current Version\Terminal Server\Install\RefHive.

    After some digging on the internet we found that the GUIDs came from HKU\.Default\Software\PrintVendor\CSR|Printserver\{GUID}. With another vendors driver they are created directly under the "printvendor" key.

    We found the same keys under HKU\S-1-5-18.

     

    We then set up a testenvironment to find out why that huge amounts of GUID were created under the HKU\.Default... key, and it seems it is created each time a user logs on and connects a network printer.

    On further investigation we found that RES PowerFuse disconnects the network printer at each log off, so there is a new network printer connection each time the user logs on and hence we get a new GUID each time. The disconnect feature is a good thing as we don't have to worry about removing mapped network printers when they are phased out.

    But even if we disable the disconnect feature we'll have the same problem, but it's limited to the number of users logging on the terminal servers or Citrix servers for the first time. So if there are 1000 users with a couple of print mappings each the number of GUIDS will eventually be the number of users multiplied with the number of print mappings pr user on each server.

    Since this is created under the HKU\.Default key, all users getting a new profile also get the GUIDS in their user registry that makes the ntuser.dat file grow.

     

    We are able to reproduce this in a lab environment without Citrix or Res Powerfuse by deleting the print mapping and then connect the printer again. The printer drivers have also been tested in several different versions, with and without native print processor and with and without CSR enabled.

     

    The workaround is to delete the offending keys in HKU\.Default, but this is a very cumbersome procedure and we don't know what effect it has on the users print setup.

    So far we have seen the issue with some Sharp drivers, HP Universal Printer v5.0 drivers and Konica Minolta drivers.

    On the Sharp driver an equal amount of GUIDs are created under HKEY_USERS\.DEFAULT\Software\Microsoft\Windows NT\CurrentVersion\ICM\ProfileAssociations\Print, but it's only one key with some sub values (key: CSR|printserver,{GUID}).

     

    It seems that this is vendor related, but I'm wondering if anyone else have seen this behavior and found some way to fix it or work around it?

    I think it's strange that all the keys that are created does not get cleaned up when a printer connection is deleted...

     

    Regards,

    Martin Eilertsen

    jueves, 21 de octubre de 2010 15:18

Respuestas

Todas las respuestas

  • Hi,

    I have the same exact problem.  We have W2K8 x32 SP2.  This only seems to happen on our citrix servers.  Did you get any more info on this?

    martes, 09 de noviembre de 2010 15:57
  • Hi,

    We have not been able to find a permanent solution, but our workaround is as follows (if you map your printers with RES PowerFuse):

    • Disable that networked printers are disconnected at each logoff
    • Delete the offending registry keys in HKU\.Default\... (as described in the first post)
    • Then the .Default part of the registry stops growing indefinetly for each user at each logoff/logon. It will still give you an entry pr user pr server, but it's fewer and the msi installer handles it.

     

    We actually had to give a few users new profiles as well after cleaning up the offending keys in HKU\.Default, becuause print was failing from a number of applications when using the offending drivers (some applications did not print and no error message was given, and some applications such as Microsoft Office Word gave an error message).

     

    I found a few links refering to the same problem, when searching for refhive:

    http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1247687

    http://www.rb303.net/2010/01/terminal-server-2003-msiexec-high-cpu.html

    http://forum.sysinternals.com/svchostexe-eating-100-cpu_topic8937_post113433.html

     

    None of these provided a complete solution, but they pointed us in the workaround direction.

     

    Regards,

    Martin Eilertsen

    miércoles, 10 de noviembre de 2010 9:18
  • thanks for this.
    it helped explain exactly the behavior i've seen.

    i've been installing windows patches for one and a half day one one of our servers and it still isn't finnished.
    your post pointed me in the right direction and also explained some of our longstanding issues.

    after a little registry cleanup the remaining patches seems to install much quicker than before and i also belive this will fix the high cpu usage at each logon due to one of the hp printer drivers trying to install a msi file when someone loggs on.

     

    viernes, 26 de noviembre de 2010 15:01
  • Hi Martin, thank you very much for your solution. Our server suffered from the same symptoms (msiexec hanging for long time before continuing). I also found quite a lot of GUIDs in HKU\.Default\Software\Hewlett-Packard. In our case the problem was not solved by removing those keys though.

    On further investigation (using Process Monitor and Process Explorer) I found that our server had more than 30000 keys in HKEY_USERS\.DEFAULT\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Digest\Hosts (for example ...\Digest\Hosts\digest01CA0000n0:digest01CA0000n0). Copying them to the RefHive shadow area kept msiexec pretty busy >:( ...

    As I could determine those keys were *not* created by a virus or other malware, but by UNS.exe, which is part of the "INTEL(R) management engine components" package, that I had installed. I could also see that UNS.exe kept on adding keys now and then. Uninstalling the "INTEL(R) management engine components" package put a stop to this. I also removed the HKEY_USERS\*\Software\Microsoft\Windows\CurrentVersion\Internet Settings\Digest key for all users.

    Thanks again for pointing me in the right direction. I wasn't able to pinpoint the problem on several tries before I found your post here. Now my installs and updates run like hell again, which is a great relief.

    Christian
    • Editado skyball miércoles, 29 de junio de 2011 21:20 repair paragraphs
    miércoles, 29 de junio de 2011 21:16
  • Hi All,

    Thanks for your clarifications, i encountered the same problem with very slow msiexec. After investigations (Process Monitor/Explorer) i found the same offending registry keys: HKU\.Default\Software\Konica Minolta\CSR|Printserver\{GUID

    Did you find a more permanent solution? I have deleted these keys on a test machine, and like you, i don't know what effect it has on the users print setup.

    Regards.

    martes, 25 de octubre de 2011 15:22
  • Hi,

    Thanks for the interest on this topic and that our findings helps you in the right direction.

    I'm sorry to say that we have not yet found any solution to this, but we have moved more of our systems over to Windows Server 2008 R2 platform with XenApp 6.0, and it seems that this plattform isn't affected in the same way. At least we have not noticed any msiexec cpu usage, so either this has been fixed in the install routines in Windows Server 2008 R2 or the registry keys are not present.

    I'll check up on the registry keys on a couple of Windows Server 2008 R2 servers in the next couple of days and return with my findings.

    As for deleting the keys. We have done this several times now, even during install and it does not seem to impact the users as they log in again. But I'm not able to offer support on this issue. So make a backup of the keys first, just to be safe. If it goes wrong, I don't know how to fix it :-).

    Regards,

    Martin

    martes, 25 de octubre de 2011 17:42
  • Hi (again)....,

    I came across an article while searching for a solution. Yes, we also see this behavior on Windows Server 2008 R2, but I can't say that we have seen slow MSIExec installs...

    Either way this articel explains how CSR (Client Side Rendering works):

    http://blogs.technet.com/b/askperf/archive/2008/02/10/ws2008-client-side-rendering.aspx

    Notice the bullet with GuidPrinter (CSR is default in Win 2008):

    •GuidPrinter: this is a random Globally Unique Identifier (GUID) assigned to the remote printer connection on the client.  Each connection to a shared network printer creates a GuidPrinter that is used to identify the printer as a remote printer.  Print jobs sent to a GuidPrinter are rendered as if they are being printed to a local printer but are then sent to the print server using the Client Side Port monitor (see below) instead of being written to a local port.

    As they describe this GUID should reside in the users registry under HKEY_CURRENT_USER\Printers\Connections\<connection>.

    I looked inside a users registry and noted the printer GUID for one of our network printers (Sharp MX-2700N) residing on one of our print servers.

    I then searched in HKU\.Default and found the GUID in HKEY_USERS\.DEFAULT\Printers\DevModes2 (contains thousands of GUIDs) and beneath HKEY_USERS\.DEFAULT\Software\SHARP\CSR|<printservername>.

    So it seems to me, if the print server is set up to publish the print queue as CSR (default) the GUIDs are genereated as unique each time a terminal server user logs on and the printers are mapped, and in time, this will generate a huge amount of GUID entries in .Default for no reason what so ever? (except generate a lot of waiting time for the poor souls installing msi packages).

    A possible solution to this, is to disable the CSR on the print queue and see if the GUID generation goes away. We have not yet tested this.

    I don't know what will happen with the existing print mapping, and I don't know what is safe to delete beneath HKU\.Default (devmodes2 and so on).

    And it's possible that RES Workspace Manager causes the GUID to be generated at each logon (we are still running the same configuration as stated in the earlier posts except a newer verison of RES, so the print mappings should not be removed at log off). I've not been able to test this again, to see if it's the same behavior without RES Workspace Manager.

    If you are running without RES or some kind of printmapping deletion/connection and are still seeing this, I would be interested to hear from you in this thread.

     

    Some advice from Microsoft on:

    • how to clean up
    • if this is by design
    • if it's a best practice to disable CSR print queues when intended use are on Terminal Servers (as stated in the article you can disable rendering on the terminal server "client" by using Group Policy, but that is just the type of rendering in effect, maybe not the generation of the GUIDs?)
    • and if the above setting on the queue; will it disable the generation of the GUID entry?

    would be appreciated.

     

    Regards,

    Martin Eilertsen

     

     

    miércoles, 26 de octubre de 2011 14:34
  • Hello

    We are running 4 TS servers with Windows Server 2008 x64 SP2 (non-R2)

    Print server is located on other server

     

    We do not run REW manager in our environment, but still experience the high CPU msi installers on HP Universal Driver 5.3

    I tried disabling CSR on the effected printers and restart the print spoolers. But with no effect. Still msi install and CPU @ max on one core.

    I now have to remove the HP Universal Driver to get a functional environment tomorrow, we have been running on 6P black & white driver on the HP printers, but we need color printing. We had the setup running fine on HP Universal 5.0.3 before, but then I wanted to upgrade to a "better" or at least newer HP driver because we had some new printers installed (HP CP1515n). Bad misstake, User profiles were corrupted, Logon duration incresed in some extreme cases it took users 30minutes-2hours just to log on.

    Alan are you there?

    All the best

    // J

    miércoles, 26 de octubre de 2011 21:48
  • I have uninstalled all HP Universal drivers from the printserver and terminal server.

    In the meantime we have found a temp solution for the problem. It's far from a permanent resolution of the problem but a good workaround.

    We already had another HP LaserJet Color driver for another printer (CM1410fwn) so I tried to use that driver for the HP CP1515n.
    The result so far is good, color print is working. I have requested feedback from the "live" users tomorrow to se if I can use this driver instead of the Universal one and still get color prints to work fine in all programs and with margins and stuff.

    The driver used is:

    CM1410Series_FNW_Basic_Solution for Windows Server 2008 x64

    Greetings

    J

    jueves, 27 de octubre de 2011 15:45
  • Thank you all very very much!! This complete thread has helped me sorting out a long lasting problem! My Citrix servers that were regulalry rebuild didn't has any issue, but the ones that lived longer had. I had no idea where this was coming from, but I think the msiexec should leave the print subsystem alone when installing a complete different application!

    Thanks again!

    martes, 15 de septiembre de 2015 13:06