none
Intermittent not enough space errors when doing LDAP queries against 2019 domain controller RRS feed

  • Question

  • Hi All,

    We have deployed some new 2019 domain controllers are are having issues where LDAP queries are coming back with:

    00000008: SysErr: DSID-0205199E, problem 12 (Not enough space), data 0

    Turning on diagnostic NTDS logging the domain controller logs (Event ID: 1535):

    Internal event: Active Directory Domain Services has encountered the following exception and associated parameters. 

    Exception:
    e0010003 
    Parameter:


    Additional Data 
    Error value:

    Internal ID:
    205199e

    Internal event: The LDAP server returned an error. 
     
    Additional Data 
    Error value:
    00000008: SysErr: DSID-0205199E, problem 12 (Not enough space), data 0

    The domain controller has plenty of space 80Gb. The issue seems to occur when LDAP clients use the filter syntax: memberOf:1.2.840.113556.1.4.1941:

    Has anyone seen this error before?.  Dcdiag and repladmin are clean.

    Friday, December 14, 2018 3:53 AM

All replies

  • It occurs both on Full GUI and server core.
    Friday, December 14, 2018 4:49 AM
  • We are having the similar issues when firing 'memberOf:1.2.840.113556.1.4.1941:' though the error does not seem to occur when the resulting search >~1000 or so. 

    As you we have recently deployed new 2019 domain controllers.


    Wednesday, December 19, 2018 8:21 AM
  • We are having exactly the same issue (with jenkins for example (sometimes the login works, sometimes it does not)).

    We also deployed Windows Server 2019 domain controllers recently.

    Have you found a solution/workaround?

    Thursday, January 3, 2019 7:59 AM
  • We rolled back to 2016 domain controllers. Clearly some bugs have appeared due to the memory changes they made in 2019 for Active Directory.
    Monday, January 7, 2019 6:30 AM
  • We have the same descriptive behavior in our environment.

    At any time the LDAP login fails, I can found a "not enough space" message at the same time in the directory log... :\

    I've created a MS Support case...

    Tuesday, January 8, 2019 10:02 AM
  • Please let others also know on what conclusion you became with the support.
    Tuesday, January 15, 2019 1:59 PM
  • Bad news...

    MS has closed the ticket as "won't fix" because I can't deliver a proper repro as case target.

    So: we rolled back to 2016 domain controllers too.

    Hopefully more users will report this behavior in future.

    Wednesday, January 16, 2019 3:36 PM
  • I just installed the new Jan update: https://support.microsoft.com/en-us/help/4476976/windows-10-update-kb4476976 and I can no longer reproduce this problem. 

    Are others able to confirm after installing this update.

    Thursday, January 24, 2019 10:14 PM
  • kb4476976 Didn't solve the problems for us unfortunately.
    Thursday, January 31, 2019 9:08 AM
  • still the same. Windows 2019 Server AD is unusable for us
    Friday, February 1, 2019 9:24 AM
  • This blog post describes the only known change in how AD handles memory, and seems relevant for this issue:

    https://blogs.technet.microsoft.com/askds/2018/10/02/deep-dive-active-directory-ese-version-store-changes-in-server-2019/

    In the comments, Daniel Appleby asked if this updated algorithm for the ESE version store caused the LDAP out of space problem. The blog author replied that he believes it does not. Still, to me this sounds like the cause, although admittedly after only a quick glance at the post.

    I would suggest anyone with Windows Server 2019 work through the troubleshooting section of the blog post, and perhaps make the registry changes mentioned, to see if it helps.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Friday, February 1, 2019 2:58 PM
  • In an environment of a customer we can reproduce the issue with the following PowerShell commando:

    Get-ADUSer -Server W2019DC -LDAPFilter "(&(objectClass=user)(objectCategory=person)(sAMAccountName=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(memberof:1.2.840.113556.1.4.1941:=CN=group name,OU=Groups,DC=company,DC=domain))"

    Result:
    Get-ADUSer : Server returned out of memory error

    When using OID: 1.2.840.113556.1.4.1941 for nested group extraction by AD the error occurs, without it doesn't. The eventviewer reports: SysErr: DSID-0205199E, problem 12 (Not enough space), data 0

    The error occurs most of the time, but once in a while it works. On Windows 2019 servers with more memory the command is working without problems more often than on servers with a lower amount of memory. The command process without problems on older OS versions.

    • Edited by Roy- Monday, February 4, 2019 12:04 PM
    Monday, February 4, 2019 12:03 PM
  • We are on KB4487044 it is still the same.
    Thursday, February 14, 2019 11:44 AM
  • I have been monitoring our bucket use with the perfmonitor and had a look at the event log, all seems clean.
    Thursday, February 14, 2019 12:33 PM
  • This is still occurring for us. Even after installing the update. My test had an  (&(cn=username)(memberOf:1.2.840.113556.1.4.1941=DN)) but just doing (memberOf:1.2.840.113556.1.4.1941=DN) returns the same error.
    Thursday, March 21, 2019 3:19 AM
  • Microsoft told me the root cause for this issue is found. A private hotfix will become available in 2/3 weeks.

    Tuesday, April 2, 2019 5:42 PM
  • Hi Roy, Could I ask you to notify, when you got a fix?
    Thursday, April 18, 2019 7:34 AM
  • Roy, 

    We are having similar issues using the memberOf:1.2.840.113556.1.4.1941  filter . 

    Would you be able to post here once you receive the hotfix and get it running? 

    Thanks... 

    Wednesday, April 24, 2019 2:03 AM
  • We have the same problem.
    Wednesday, April 24, 2019 1:21 PM
  • KB4495667 may have fixed this. Still testing - but looks promising so far.
    Monday, May 6, 2019 8:43 AM
  • No, worked only a few minutes after the reboot of the DCs.
    Monday, May 6, 2019 3:26 PM
  • Same issue on this end, reproducible in PowerShell.  I'm not seeing any events in the Directory Services log.

    EDIT: Also, the server itself has plenty of memory, and for the "version buckets allocated" limit (detailed in the Deep Dive article previously linked in this thread out of the 12,000 limit we have ... 2.

    • Edited by TMitera Thursday, May 9, 2019 6:15 PM
    Thursday, May 9, 2019 6:03 PM
  • For the purposes of anyone else either lurking on this thread currently, or for future posterity, I am working with Microsoft support on a support request of my own.  Current information is that the support rep believes that "[the issue] might be a known issue [and] will confirm it with [their] Product Team."  As I continue to get updated information I will similarly update this thread.
    Wednesday, May 15, 2019 5:56 PM
  • I got some more information from the Support Engineer, though it is far less definitive than I had hoped.  From the engineer:

    "I appreciate your time to report this problem to us and I am indeed sorry for any trouble this may have caused. I want to let you know Microsoft is aware of this problem. We have already reported this issue to our development team and they are currently working on this problem, it may take a long time to get a released fix."

    In short: Microsoft knows of the issue, is working on a fix, and has no ETA for said fix.

    What we are doing in the meantime is running a separate group of domain controllers (for example, running Server 2016) that sit behind a load balancer (ldap.domainname.local).  That way we can keep the Server 2019 DCs in place, but still have a place to point LDAP requests to that we know will work until the fix for 2019 is released and implemented.

    Monday, May 20, 2019 4:09 PM
  • Any news on this?
    Wednesday, May 22, 2019 7:39 AM
  • Disclaimer: I'm a Microsoft employee, and I am very familiar with this bug.

    Please don't blame my ESE version store change on this! ;)

    This isn't an ESE database issue; it's an LDAP memory arena issue. So that blog post is (fortunately for me) unrelated.

    We've root-caused the bug, but it will probably be a couple more months before the fix is public. (No firm ETA yet.)


    Wednesday, May 22, 2019 8:06 PM
  • Thanks for the update (and clarification)!

    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Wednesday, May 22, 2019 8:20 PM
  • *puts down somehow-ESE-shaped torch and pitchfork*

    I kid.  Actually Ryan I really appreciate this (obviously unofficial and not to be taken as gospel!) update.  It's good to hear that you / your team(s) are making progress on this bug.  What would the best way for us to keep ourselves up to date as to when the fix goes public?
    Thursday, May 23, 2019 6:35 PM
  • I'll update this thread since I basically "own" this bug. Current estimate is September.
    Wednesday, June 12, 2019 6:55 PM
  • A user recently reported a problem with the WinNT provider on Windows Server 2019. Only the first 100 users can be retrieved in a domain with over 2,000 users. The same WinNT query retrieves all users on Windows Server 2016. No errors were reported, and LDAP shouldn't be involved. But I wonder if it is possible that the same memory issue could cause this. Or is this an unrelated problem?

    The user replied to this old thread:

    https://social.technet.microsoft.com/Forums/en-US/c6539fc2-e0a4-41f9-b618-aa701a5219d4/vbscript-problem-with-winnt-object-only-returning-100-users-from-active-directory?forum=ITCG

    I have asked for the WinNT code that was used.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Thursday, June 13, 2019 6:13 PM
  • Its great its being worked on. 

    Is there a KB article describing it ? I couldn't find it in 2 weeks of searching. This thread is really the only place I can find mention of it. 

    Is there a workaround ? September is a long way off. Do I really need to re-promote a 2012 / 2016 server for LDAP queries in the meantime ?

    Sunday, June 16, 2019 11:52 PM
  • Near as I can tell, there is no workaround.  At the very least I haven't been able to find one, and in my support ticket with Microsoft the engineer was not aware of one that existed.
    Monday, June 17, 2019 1:48 PM
  • As I noted before, the problem reported with the WinNT provider on Windows Server 2019 may be different. But it is similar in that code that has worked for years on older versions of Windows Server, fails on Windows Server 2019. In this case, only 100 users are retrieved out of many thousands. The code used is similar to the following VBScript:

    Set objDomain = GetObject("WinNT://NetBIOSNameOfDC")
    objDomain.Filter = Array("user")
    k = 0
    For Each objUser In objDomain
        Wscript.Echo objUser.name
        k = k + 1
    Next
    Wscript.Echo "Number of users: " & CStr(k)
    


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Monday, June 17, 2019 3:56 PM
  • <rant>

    Over the last few years there have been a lot of complains about declining quality of windows OS and its cumulative updates. But this particular case is pretty telling. It been 6+ month since 2019 server GA and still "probably be a couple more months before the fix is public. (No firm ETA yet.)". And how come that such an obvious and easy to spot bug went into GA unnoticed? Did directory team forget to run their test suites?

    </rant>

    Tuesday, June 25, 2019 11:20 AM
  • We just deployed a couple of Windows Server 2019 domain controllers and experienced the exact same issue being described in this post in our environment. It's dependably reproducible using this Power Shell snippet posted by Roy a few month ago and is being logged at Windows' event log when setting the NTDS registry key "Internal processing" to DWORD:4. Since we got a lot of applications making use of LDAP queries containing (memberof:1.2.840.113556.1.4.1941:=...) we would be feasible if there was at least any workaround or Hotfix in the meantime waiting for an official bug fix.


    • Edited by 5EC2E7 Thursday, July 11, 2019 9:28 PM
    Thursday, July 11, 2019 9:09 PM
  • If your scripts retrieve direct and transitive members of a group, you can use the Get-ADGroupMember cmdlet and the -Recursive parameter. For example:

    Get-ADGroupMember -Identity "GroupName" -Recursive

    where "GroupName" should be either the DN, sAMAccountName, objectSID or objectGUID of a group.

    Edit: Otherwise, there is no workaround involving LDAP clauses, except to use more code to enumerate the direct members and recursively deal with members that have class group.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)


    Thursday, July 11, 2019 9:45 PM
  • For us this event has showed us that Server 2019 simply isn't ready for production usage.  It may operate well enough for the most part, but it worries me that an issue such as this is taking months to resolve with no workaround in the interim to mitigate the problem.

    Still eagerly awaiting news on this fix either way.

    Friday, July 12, 2019 12:59 PM
  • Get-ADGroupMember -Identity "GroupName" -Recursive.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Thx for your suggestion, Richard. While this doesn't directly help us with our problem it made me think of a possible temporary workaround to write a small PowerShell script which could resolve the nested group memberships and add those resulting user objects to another "flat" AD group which could be directly addressed by various LDAP queries instead of relying on the currently faulty OID 1.2.840.113556.1.4.1941 parameter. This script could be triggered by a time interval (i.e. hourly) and/or by a specific EventID.

    Nevertheless we'd appreciate if things just worked like they used to on previous versions of Windows Server ADDS (2012R2/2016). LDAP and nested groups are still a thing in 2019 despite all those various claims based alternatives which Windows offers as well ;-)

    Friday, July 12, 2019 5:10 PM
  • https://twitter.com/markmorow/status/1152226683866320897?s=11 

    Does anyone here on this thread have a premier contract and want to to chat with Ryan about testing the fix? https://twitter.com/JosephRyanRies/status/1152294734930407424?s=20

    If so can you reach out to him on twitter to get fixed up to test a patch?


    Friday, July 19, 2019 8:05 PM
  • Thank you everyone for coming forward with your willingness to help test the LDAP 0205199E hotfix. We have positive customer confirmation now. The fix should show up on Windows Update by August or September.

    Monday, July 22, 2019 5:17 PM
  • We have the issue.  Can we get a copy of the beta patch somewhere?

    Friday, July 26, 2019 1:12 PM
  • Hi Ryan, We have the issue.  Please can we get a copy of the beta patch somewhere?
    Saturday, July 27, 2019 9:36 AM
  • I believe to get the hotfix you need a premier contract.  Your best bet in the interim is likely to spin up 2016 DCs and use those until the hotfix for 2019 is officially released.
    Tuesday, July 30, 2019 4:46 PM
  • I'm the guy on twitter that moaned about this, can confirm the patch has fixed the bug in our environment.

    Just a little waiting game before it goes public.

    Friday, August 2, 2019 11:10 AM
  • That is great news.

    We do not have a premier support contact and this is really hurting us. Does Microsoft have a confirmed release date for this fix?

    Friday, August 2, 2019 2:38 PM
  • when will this patch be realeased… Same problem here, we not very amused !!!


    Tuesday, August 6, 2019 6:21 AM
  • @Ryan: is there any update when the patch will be released? We have major problem in our production environment about this issue.
    Tuesday, August 13, 2019 8:33 PM
  • Is there a KB article describing this bug? If not, why not?
    Friday, August 16, 2019 8:47 PM
  • For whatever reason, this bug doesn't seem to be as important as you'd think it should be considered.  I haven't found a KB article on it.  I don't know if the bug literally affects all implementations of Server 2019 or not, but it still results in an operating system that simply cannot be used as a production domain controller over 10 months after release.
    • Proposed as answer by vetter Tuesday, August 20, 2019 6:18 AM
    • Unproposed as answer by vetter Tuesday, August 20, 2019 6:18 AM
    Monday, August 19, 2019 6:53 PM
  • @Ryan: is there any update when the patch will be released? We have major problem in our production environment about this issue.
    Wednesday, August 21, 2019 12:11 PM
  • Hey imranh2, can you please share the patch with us?
    Wednesday, August 21, 2019 12:13 PM
  • Is there a KB article describing this bug? If not, why not?
    Wednesday, August 21, 2019 3:18 PM
  • Thank you everyone for coming forward with your willingness to help test the LDAP 0205199E hotfix. We have positive customer confirmation now. The fix should show up on Windows Update by August or September.

    Is there any way to get the hotfix now?
    Monday, August 26, 2019 9:32 PM
  • Thank you everyone for coming forward with your willingness to help test the LDAP 0205199E hotfix. We have positive customer confirmation now. The fix should show up on Windows Update by August or September.

    Any updates on this issue?
    Tuesday, August 27, 2019 8:48 PM
  • Any chance of getting this update early?  We too are having the same issue.
    Tuesday, September 3, 2019 2:06 PM
  • Any chance of getting this update early?  We too are having the same issue.
    Me too - this is a serious problem in our org....  So much discussion of this bug, but no fix released?
    Wednesday, September 4, 2019 2:49 PM
  • Any chance of getting this update early?  We too are having the same issue.

    Me too - this is a serious problem in our org....  So much discussion of this bug, but no fix released?

    At our org aswell. 

    This bug is entirely halting our already overdue migration to server 2019 

    Thursday, September 5, 2019 6:56 AM
  • we have this issue since putting in new 2019 dc's and Jenkins is affected, can anyone provide an update? 
    Thursday, September 5, 2019 11:02 AM
  • I wonder why Microsoft stopped updating this thread...
    Friday, September 6, 2019 6:09 PM
  • Disclaimer: I'm a Microsoft employee, and I am very familiar with this bug.

    Please don't blame my ESE version store change on this! ;)

    This isn't an ESE database issue; it's an LDAP memory arena issue. So that blog post is (fortunately for me) unrelated.

    We've root-caused the bug, but it will probably be a couple more months before the fix is public. (No firm ETA yet.)


    @Ryan - is there a KB article on this issue? Why isn't it listed on https://docs.microsoft.com/en-us/windows/release-information/status-windows-10-1809-and-windows-server-2019 ?
    Tuesday, September 10, 2019 7:20 PM
  • The fix is coming, third week of September. Keep an eye out for: 

    September 19, 2019—KB4516077 (OS Build 17763.769)

    • Proposed as answer by MatthewT79 Wednesday, September 11, 2019 3:58 PM
    Tuesday, September 10, 2019 11:38 PM
  • Thanks, Ryan.
    Wednesday, September 11, 2019 3:57 PM
  • The fix is coming, third week of September. Keep an eye out for: 

    September 19, 2019—KB4516077 (OS Build 17763.769)

    But the issue isn't listed here for some reason: https://docs.microsoft.com/en-us/windows/release-information/status-windows-10-1809-and-windows-server-2019 ?

    Will it ever be included there? If not, why not?

    Thursday, September 12, 2019 2:54 PM
  • Last-minute delay to September 24th.
    Thursday, September 19, 2019 3:49 PM
  • It's been a year of broken functionality, what's another week....
    Thursday, September 19, 2019 4:08 PM
  • What is the (high level) reason for the delay?  Is the stability of the patch in question?
    Friday, September 20, 2019 3:30 PM
  • Fix September 19, 2019—KB4516077 (OS Build 17763.769) is still not available. Any info on this?

    Thanks

    Christian

    Monday, September 23, 2019 8:34 AM
  • Still isn't showing up here:
    https://support.microsoft.com/help/4516077
    Tuesday, September 24, 2019 2:00 PM
  • Last-minute delay to September 24th.
    is it getting released today?
    Tuesday, September 24, 2019 2:00 PM
  • Our WSUS server has not pulled the KB4516077 update yet.  I did a Synchronization thinking it was missed but it's still not there.  Any word on when this will be available?  We too are having the same issue with Jenkins as others here.

    Thanks, Matt

    Tuesday, September 24, 2019 3:58 PM
    • Proposed as answer by 5EC2E7 Tuesday, September 24, 2019 8:04 PM
    Tuesday, September 24, 2019 5:21 PM
  • First tests passed without out of memory errors. Looks promising so far.
    Tuesday, September 24, 2019 8:06 PM
  • Thanks!
    Wednesday, September 25, 2019 6:47 AM
  • The Fix seems to work as expected.

    Many thanks to Ryan!

    Wednesday, September 25, 2019 9:37 AM