locked
RDS - Insufficient system resources exist to complete the requested service for users ntuser.dat RRS feed

  • Question

  • Hi all

    We have the following RDS Environment:

    - 2 physical RDS Hosts with 32GB RAM and Desktop Composition and RemoteFX enabled

    - 1 RDS Session Broker and Session Broker Loadbalancer

    - Kaspersky AntiVirus 8.0 for Windows Servers Enterprise Edition (validated for RDS Enviroment)

    Now one server is damaged and i need to reinstall it. So the other server must handle all Users.

    But after just 20 Users logged in or disconnected the server tells me the following:

    ------------------------------

    EventID: 1508
    Source: User Profile Service

    Windows was unable to load the registry. This problem is often caused by insufficient memory or insufficient security rights.

     DETAIL - Insufficient system resources exist to complete the requested service.
     for C:\Users\%username%\ntuser.dat

    ------------------------------

    EventID: 1502
    Source: User Profile Service

    Windows cannot load the locally stored profile. Possible causes of this error include insufficient security rights or a corrupt local profile.

     DETAIL - Insufficient system resources exist to complete the requested service.

    ------------------------------

    When i now log off some disconnected users, everything is fine again until the user count reaches again 20 users.

    I have already changed the following Registrykeys:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagedPoolSize to 0xffffffff

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PoolUsageMaximum to 0x00000032

    i think the server can handle a little bit more users but not that much as it should.

    Is there somewhere a limitation how big the registry can be?

    Thanks for any tipps :)

    best regards

     

    JBAB


    • Edited by JBAB Tuesday, September 20, 2011 6:53 AM
    Tuesday, September 20, 2011 6:51 AM

Answers

  • Hi,

    Before working on this please make sure you have a backup of your server and that you are comfortable with what steps to take if your server will not boot.  This should be done during off hours with no users logged on (except you) and via the physical server console.

    There are many different ways to do this, below I will suggest one method:

    Exporting .DEFAULT hive

    1. Open regedit, in the left pane navigate to and right-click on .DEFAULT under HKEY_USERS, choose Export.
    2. Select Save as type: Registry Hive Files (*.*)
    3. Navigate to C:\Windows\System32\Config folder
    4. In the File name box, enter DEFAULT.new
    5. Click Save to export the hive.
    6. Using Windows Explorer, browse to C:\Windows\System32\Config folder and verify that DEFAULT.new file is very small in comparison to DEFAULT.
    7. If DEFAULT.new is small, insert the Windows Server 2008 R2 DVD into the server's drive, shut down your server, and then complete the next section.

    Replacing old .DEFAULT with compacted version

    1. Turn on your server, press key to boot to DVD (if needed, depends on your BIOS), when prompted with Press any key to boot from CD or DVD... or similar message press space to boot to the Windows DVD
    2. Click Next on the Install Windows screen
    3. Click Repair your computer
    4. Select Use Recovery Tools... option and click Next
    5. Click Command Prompt
    6. Change to the drive letter for your C: drive (likely it will not be C:, but something else like D:), for example, by typing D: and then pressing enter.  You can find the drive letter for your C: drive by trying different drives, doing a dir , and then seeing if Windows, Users, Program Files, etc. folders are present.
    7. Change directory to the Config folder by typing cd\Windows\System32\Config and then pressing enter
    8. Enter the following command to rename the DEFAULT file: ren DEFAULT DEFAULT.bak
    9. Enter the following command to rename the DEFAULT.new file: ren DEFAULT.new DEFAULT
    10. Exit the recovery environment by clicking Restart

    After completing all of the above please log on to your server, check the System, Application, and Application-specific logs for any (new, not seen in the past) errors.  Test out various things to make sure that the server appears to be operating fine.

    If for some reason the server does not boot or has major issues, then boot back into the recovery environment, rename DEFAULT to DEFAULT.new, rename DEFAULT.bak to DEFAULT, and then restart your server.

    Most likely the faulty sharp drivers are causing the resource problems you are seeing, however, it is always possible that another issue is present as well (like problem with antivirus software).  Also make sure you put the memory-related registry settings, registry size limitation, etc. back to factory default in order to avoid problems.

    Thanks.

    -TP

    • Marked as answer by JBAB Monday, September 26, 2011 10:14 AM
    Friday, September 23, 2011 8:01 AM
  • Hi,

    You most likely have faulty drivers/software running on your server.  Your local system user hive (DEFAULT) is almost 1.5GB, which is not typical.  For example, I usually see sizes like 4 MB or less, however, I would not be concerned if it were tens of megabytes, but 1.5 Gigabytes is excessive.

    One case where I have seen this occur is a faulty print driver that keeps saving data to the registry every time a users logs on and their printers are autocreated.  Do you have any print drivers installed on your server besides Remote Desktop Easy Print?  You can check this in Print Server Properties, Drivers tab.

    Another case where I have seen a similar type of thing occur is with antivirus/antimalware/security type of software on the server.

    Please review the following suggestions:

    1. Try to look under .DEFAULT in registry in order to determine exactly which key has a ton of data in it.  One technique you can use is to right click on each top-level key under .default and export it to a file.  Once you identify the exact subkey that is bloated please try to look at the data and determine what software is creating it.
    2. Consider removing/updating any third-party print drivers as needed.
    3. Consider removing all third-party anti-virus/anti-malware/security software.
    4. Please set the memory-related registry settings back to default (in other words, back to what they were before any changes were made) and then restart your server.  In the majority of cases you should have no need to change these on a 64-bit server.
    5. Use similar techniques as mentioned above to determine which key in the user's registry hive is taking up an excessive amount of space.

    After you determine the root cause you may need to remove the garbage from the DEFAULT hive and then rebuild it so that is is back to a normal size.  The rebuild is something that needs to be done while windows is not running.

    Thanks.

    -TP


    • Edited by TP []MVP Thursday, September 22, 2011 3:57 PM
    • Marked as answer by JBAB Monday, September 26, 2011 10:14 AM
    Thursday, September 22, 2011 3:53 PM

All replies

  • You can set the automatic log off disconnected users after x amount of time.

    Then you wont have all these hanging sessions that use system ressources.


    Also the user count doesnt sound completely out.

    1-2GB for System, leaves 30GB for users. With full client feature enabled, they will consume quite an amount of memory..(just look at your own machine how much it is consuming)

    Tuesday, September 20, 2011 7:39 AM
  • Hi Jesper

    Thank you for your input.

    13 users consume 30% of the physical ram. So from my point of view enough RAM for more Users. I mean in the "Remote Desktop Session Host Capacity Planning in Windows Server 2008 R2 and Microsoft RemoteFX in Windows Server 2008 R2 with Service Pack 1" Guide they talk about 80 Users for 24GB RAM. So i was expecting more then 20 Users for 32GB RAM :/

     

    best regards

    JBAB

    Tuesday, September 20, 2011 2:17 PM
  • Hiya,

    The document also talks about usage scenarios, which will influence your performane alot.

    For the test of it, let a user work for a few hours and then  take over this session and see what the users has running, how many ressources that user is utilizing etc.

    Or have a look at your Processes with all users enabled, have a look at what scores the top. Proberly your internet browser.

    Just for reference here is the doc:

    http://www.microsoft.com/download/en/details.aspx?id=17190 

    ----------------------------------------------------------------------

    Another thing that would be interesting is too see the size of your registry when you have around 20 users logged on.

     

    On NT-based versions of Windows, this key contains four subkeys, "SAM", "SECURITY", "SYSTEM", and "SOFTWARE", that are loaded at boot time within their respective files located in the %SystemRoot%\System32\config

    http://en.wikipedia.org/wiki/Windows_Registry#Hives

     

    Recently had an issue with some hives growing too big due to printers spamming the registry...

    Tuesday, September 20, 2011 2:37 PM
  • Hiya,

    The biggest application that is in use on the server is CLC Mainworkbench (a Life-Science Tool) that consume 400 - 500 MB each Session. Then CorelDraw, Illustrator and another Life-Science Tool.

    When i go in that folder i can see the following:

    DEFAULT --> 1'549'056 KB
    SOFTWARE --> 273'152 KB
    COMPONENTS --> 39'936
    SYSTEM --> 18'944 KB
    SAM --> 256 KB
    SECURITY --> 256 KB

    best regards

     

    JBAB

    Tuesday, September 20, 2011 2:56 PM
  • Then the user count will match - the application in reference are heavy utilizing.

    Which is also the scenario described in the referenced Capacity planning document.

     

    Try to calculate the total amount of memory consumed by 1 user. Also run a perfmon on your CPU and Average Disk Que Lenght on C:\ drive.

     

     

    Wednesday, September 21, 2011 8:09 AM
  • Hello,

    Just to make sure you are on x64 plateform. Your are running RDS which means windows 2008 R2 version of the OS on the TS where your events are logged ?

    Do you have other events(event log) or symptoms(application crashes, strange behaviors) that could tell us it's a memory problem ?

    Morever, could it be a security issue on the ntuser.dat file ? You can run process monitor (http://technet.microsoft.com/en-us/sysinternals/bb896645) to pinpoint a security cause search for access denied results in registry or files io.

     

    Wednesday, September 21, 2011 9:38 AM
  • Hi all

    It looks definitively like the server handles 17 Users. If User 18 log in the server gives me the logs from above.

    And i forgot to write that WSRM (Policy: Weighted Remote Sessions) is also active.

    @Jesper

    I tried to calculate our average memory consumption. It is around 250MB each User.

    Highest Value at the moment: 580MB
    Lowest Value at the moment: 22MB

    But with 250MB per User and 2GB for the System i would expect more than 100 users on it.

    The Average Disk Queue Length(Scale 100) is: 1.4 over 10 Minutes (depends on how the user using the server)
    The Average % Processor Time (Scale 1) is: 8 over 10 Minutes (depends on how the user using the server)

    @Angelo74

    Yes it is a x64 Plattform. Windows 2008 R2 with SP1. Sorry i could write it more clear.

    I checked with pocmon the ntuser.dat behavior. When a user can't log in procmon tells me every action with ntuser.dat is successful. Only QuerySecurityFile creates the first time a Buffer Overflow. But that happen also to a successful login.

     

    Wednesday, September 21, 2011 11:23 AM
  • OK for x64. Can you tell me is other strange behaviors are encoutered for users that successfully logged in like application crash when new windows are open, button clicked, like you open an server manager mmc ? in my experience when system memory (memory pools paged, non paged ,session pool) get out of space other symptoms do appear...

    Why did you set those registry keys below ?... On my side I did it in a special case with Microsoft support on a windows 2008 x32 and the reason was system memory fragmentation but I don't think this applies in the windows 2008 R2 x64 world where pools are very very large.... if you wannt to check if your are close to the limit your can use process explorer (load the symbols) http://technet.microsoft.com/en-us/sysinternals/bb896653 in a case where you are close to the limit to track down system memory consumtion you could use driver verifier(aval. out of the box)

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagedPoolSize to 0xffffffff

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PoolUsageMaximum to 0x00000032

    Have you tried to disable antivirus which sould be implemented as a system driver... could leak system memory... when you disable antivirus you have to reboot to make sure it is deactivated..

    .. I hope your users are not to angry...i know it's not a comfortable situation

    Let me know

     

    Wednesday, September 21, 2011 11:58 AM
  • On more thing can you tell me the size of your profiles ntuser.dat ? Also the size of the default profile ntuser.dat ?
    Wednesday, September 21, 2011 12:00 PM
  •  

    Hi Angelo74

    No no, our users are patiently :)

    I set the keys because during my searching i found that as a solution to fix the problems (http://support.microsoft.com/kb/312362/en-us), it is clear that was only for win2k3 but i found that people with the same problem and win2k8 R2 could fix it with that registry keys.

    That is the System Information:

    The server don't care about the user login problems. The Servermanager mmc is stable, the running applications running fine.

    C:\Users\Default\NTUSER.DAT is 256KB small.

    A fresh test user has 768KB.

    A normal user has files between 512KB up to 48'640KB. But the most users are under 10'000KB.

    During my tests i recognize that not the user count is the problem. It looks like the problem is how big the ntuser.dat file of the user is. So i can log multiple test users to our RDS Host and go over 17 or 20 Users. But with normal users can't. For me it looks like a limitation of how big the registry can be. Somehow :/

    Edit: it looks like the microsoft server has a problem showing the picture. i will upload tomorrow with a external hoster
    • Edited by JBAB Wednesday, September 21, 2011 3:40 PM
    Wednesday, September 21, 2011 2:26 PM
  • Hi Angelo,

    I had this problem in two TS Servers.

    I increase some existing registry keys:

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\

     

                                    RegistrySizeLimit (DWORD) – Decimal (234217728)

     

    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management

     

                                    SessionPoolSize (DWORD) – Decimal (64)

                                    SessionViewSize (DWORD) – Decimal (104)

     

    http://www.computing.net/answers/windows-2003/change-registry-size-windows-2003/1106.html
    http://www.eggheadcafe.com/software/aspnet/29453888/terminal-services-not-loading-user-profiles.aspx
    http://support.microsoft.com/kb/840342/pt-br
    http://technet.microsoft.com/en-us/library/cc776120%28WS.10%29.aspx

    Let we know the result.

     

    Tks.

     

     


    Paul Haro - Microsoft Certified Professional
    Wednesday, September 21, 2011 4:21 PM
  • Here is the System Information

    Thursday, September 22, 2011 6:05 AM
  • Hi Paul

    This registry settings has no effect. Same story. After 20 Users its finished :/

    I reset the registry settings that i wrote in my first post back to the default because after setting your registrysettings the server bluescreens during start-up.

    best regards

    JBAB

    Thursday, September 22, 2011 3:27 PM
  • Hi,

    You most likely have faulty drivers/software running on your server.  Your local system user hive (DEFAULT) is almost 1.5GB, which is not typical.  For example, I usually see sizes like 4 MB or less, however, I would not be concerned if it were tens of megabytes, but 1.5 Gigabytes is excessive.

    One case where I have seen this occur is a faulty print driver that keeps saving data to the registry every time a users logs on and their printers are autocreated.  Do you have any print drivers installed on your server besides Remote Desktop Easy Print?  You can check this in Print Server Properties, Drivers tab.

    Another case where I have seen a similar type of thing occur is with antivirus/antimalware/security type of software on the server.

    Please review the following suggestions:

    1. Try to look under .DEFAULT in registry in order to determine exactly which key has a ton of data in it.  One technique you can use is to right click on each top-level key under .default and export it to a file.  Once you identify the exact subkey that is bloated please try to look at the data and determine what software is creating it.
    2. Consider removing/updating any third-party print drivers as needed.
    3. Consider removing all third-party anti-virus/anti-malware/security software.
    4. Please set the memory-related registry settings back to default (in other words, back to what they were before any changes were made) and then restart your server.  In the majority of cases you should have no need to change these on a 64-bit server.
    5. Use similar techniques as mentioned above to determine which key in the user's registry hive is taking up an excessive amount of space.

    After you determine the root cause you may need to remove the garbage from the DEFAULT hive and then rebuild it so that is is back to a normal size.  The rebuild is something that needs to be done while windows is not running.

    Thanks.

    -TP


    • Edited by TP []MVP Thursday, September 22, 2011 3:57 PM
    • Marked as answer by JBAB Monday, September 26, 2011 10:14 AM
    Thursday, September 22, 2011 3:53 PM
  • Hi -TP

    Bingo. It was the Registry Key for our Sharp Copiers. That was 2.16GB big. I deleted now the big key.

    How can i compact now the registry? When i search i found just some tools that can do that. Is this save?

    Thanks and best regards

    JBAB

    Friday, September 23, 2011 6:57 AM
  • Hiya,

    I've previously covered the topic in few other threads, I had the same problem with Sharp and HP printers and Sharp support was unable to adress the issue. So I've created a log off script that cleans the hives, which seems to be working..

    http://social.technet.microsoft.com/Forums/en-US/winserverTS/thread/90cb4b9c-0afd-4a6e-b16e-d9ad3140038b

     

    Basically it removed the unwanted registry entries from Local Machine and Users hives.

     

     

    Friday, September 23, 2011 7:30 AM
  • Hi,

    Before working on this please make sure you have a backup of your server and that you are comfortable with what steps to take if your server will not boot.  This should be done during off hours with no users logged on (except you) and via the physical server console.

    There are many different ways to do this, below I will suggest one method:

    Exporting .DEFAULT hive

    1. Open regedit, in the left pane navigate to and right-click on .DEFAULT under HKEY_USERS, choose Export.
    2. Select Save as type: Registry Hive Files (*.*)
    3. Navigate to C:\Windows\System32\Config folder
    4. In the File name box, enter DEFAULT.new
    5. Click Save to export the hive.
    6. Using Windows Explorer, browse to C:\Windows\System32\Config folder and verify that DEFAULT.new file is very small in comparison to DEFAULT.
    7. If DEFAULT.new is small, insert the Windows Server 2008 R2 DVD into the server's drive, shut down your server, and then complete the next section.

    Replacing old .DEFAULT with compacted version

    1. Turn on your server, press key to boot to DVD (if needed, depends on your BIOS), when prompted with Press any key to boot from CD or DVD... or similar message press space to boot to the Windows DVD
    2. Click Next on the Install Windows screen
    3. Click Repair your computer
    4. Select Use Recovery Tools... option and click Next
    5. Click Command Prompt
    6. Change to the drive letter for your C: drive (likely it will not be C:, but something else like D:), for example, by typing D: and then pressing enter.  You can find the drive letter for your C: drive by trying different drives, doing a dir , and then seeing if Windows, Users, Program Files, etc. folders are present.
    7. Change directory to the Config folder by typing cd\Windows\System32\Config and then pressing enter
    8. Enter the following command to rename the DEFAULT file: ren DEFAULT DEFAULT.bak
    9. Enter the following command to rename the DEFAULT.new file: ren DEFAULT.new DEFAULT
    10. Exit the recovery environment by clicking Restart

    After completing all of the above please log on to your server, check the System, Application, and Application-specific logs for any (new, not seen in the past) errors.  Test out various things to make sure that the server appears to be operating fine.

    If for some reason the server does not boot or has major issues, then boot back into the recovery environment, rename DEFAULT to DEFAULT.new, rename DEFAULT.bak to DEFAULT, and then restart your server.

    Most likely the faulty sharp drivers are causing the resource problems you are seeing, however, it is always possible that another issue is present as well (like problem with antivirus software).  Also make sure you put the memory-related registry settings, registry size limitation, etc. back to factory default in order to avoid problems.

    Thanks.

    -TP

    • Marked as answer by JBAB Monday, September 26, 2011 10:14 AM
    Friday, September 23, 2011 8:01 AM
  • Ok thanks -TP.

    I will backup and replace the registry. That i can do on Monday morning.

    I will write back after that with the result :)

    JBAB

    Friday, September 23, 2011 8:15 AM
  • One thing you need to take notice to is the bottom line of TP's post.

    If its indeed printers that are clogging up, you will have the problem even after you replace the registry.

    Sunday, September 25, 2011 6:51 PM
  • Hi all

    Thanks for help. The server is now fine again. The replacing of the Default Hive was successful and now the kernel memory is back to more or less normal values.

    I must rethink how i connect the printers to the users and then i will do this cleanup again.

    Thanks to all and especial -TP for the important tipp.

    best regards

    JBAB

    Monday, September 26, 2011 10:14 AM
  • JBAB - I think I am having the same problem and I am almost positive it is related to printer. What did you do to resolve the printers clogging up the .default registry key? I installed all of the printer to the server using the global add command: rundll32 printui.dll,PrintUIEntry /ga /c\\%1 /n\\ServerName\PrinterShare and it constantly get these entries showing in the registry.

    DETAIL - 1 user registry handles leaked from \Registry\User\S-1-5-21-881422683-1183209535-925700815-2772:
    Process 552 (\Device\HarddiskVolume1\Windows\System32\svchost.exe) has opened key \REGISTRY\USER\S-1-5-21-881422683-1183209535-925700815-2772\Printers\DevModePerUser

    Any suggestions for uninstalling third party printer drivers would be a big help. The users do not have printers install on their local workstation since everything runs through the RemoteDesktop server.

    Tuesday, March 6, 2012 6:12 PM
  • Hi Wayne110

    I had the problem with a special key created by the sharp driver.

    What i did:

    - Install the printers as local printer on the RDS Servers
    - Remove the Creator Owner and set the user to read only on the specific registry key

    Then it stops.

    best regards

    JBAB

    Wednesday, March 7, 2012 8:18 AM
  • Hi everybody,

    On my side some buggy drivers added subkeys in the following to regitry keys. Based on a post on HP forum, the bug was linked to the printer redirection mechanism.

    [HKEY_USERS\.DEFAULT\Software\Hewlett-Packard] et [HKEY_CURRENT_USER\Software\Hewlett-Packard]

    I've impleted a little code in the logon script to clean-up thoses subkeys in HKEY_CURRENTE_USER. For the default wich is related to the default profil I clean it from time to time.

    As a guide line, we are working to simplify the number of drivers installed (Universal kind of like uniprint, ...) on our TS, avoid installing them when possible (RDS gives us TS Easy Print) and try to install only TS certified printer drivers. On the printing side of our RDS we try to implement the printer dirver isolation feature of windows 2008 R2 to avoid crashing of print spooler service with a buggy driver.

    Wednesday, March 7, 2012 9:25 AM
  • Here is what I did to "quick fix" the problem.  The [HKEY_USERS\.DEFAULT\Printers\DevModePerUser] and the [HKEY_USERS\.DEFAULT\Printers\DevModes2] Reg Keys were filled up with entries.  I deleted all of the entries in these reg keys (that took about 20-30 minutes each to complete) and then followed TP's post above exporting the cleaned .DEFAULT to the System32\Config folder and saving it as a Registry Hive Files and then booting into the Recovery Console command prompt and replacing DEFAULT with the newly created DEFAULT.new.  The DEFAULT file was 1.6GB and once cleaned it was 2.7MB.  Huge difference and this resolve my problem.

    Now on to the bigger problem.  Why are these reg keys filling up? [HKEY_USERS\.DEFAULT\Printers\DevModePerUser] and the [HKEY_USERS\.DEFAULT\Printers\DevModes2]  These are what the entries look like.  \\CSR|PRINTSERVERNAME\{4E833DF5-8ADE-4FA1-B657-3E4A13EE24E8}

    99% of my printer drivers installed are the HP Universal driver and the Ricoh universal printer driver, so I'm not sure that these are causing the problem.  In the .DEFAULT/Printers subkey I also have three RICOH driver folders below the DevModePerUser and DevModes2 and they only have a couple of entries in them and look normal unlike the DevMode keys that continually fill-up with entries. 

    JBAB - Makes and interesting suggestion that I might try and that is giving users READ ONLY access to the registry keys that are having problems.  I will try that and see if it causes any problems. 

    Wednesday, March 7, 2012 1:47 PM
  • I tried changing the permissions on these to registry keys [HKEY_USERS\.DEFAULT\Printers\DevModePerUser] and the [HKEY_USERS\.DEFAULT\Printers\DevModes2] to allow users "read only" and they are both still filling up with entries.  As a workaround I have created a batch file that I scheduled to run daily it deletes the entries in these registry keys.  I have not seen any problems on the user side when the entries in the reg keys are deleted. 

    Here is what the bat file looks like.

    reg delete "HKU\.DEFAULT\Printers\DevModePerUser" /f /va
    reg delete "HKU\.DEFAULT\Printers\DevModes2" /f /va

    Thursday, March 8, 2012 4:19 PM
  • Hmm but that will not work. The registry is a database and due to that it is needed to export and replace the default hive everytime when you delete some values to see a effect.

    As a test i would allow only Administrators write access to this reg keys. Set the system to read only. I think the system writes this keys.

    Best regards

    JBAB
    Friday, March 9, 2012 7:00 AM