none
Intermittent not enough space errors when doing LDAP queries against 2019 domain controller RRS feed

  • Question

  • Hi All,

    We have deployed some new 2019 domain controllers are are having issues where LDAP queries are coming back with:

    00000008: SysErr: DSID-0205199E, problem 12 (Not enough space), data 0

    Turning on diagnostic NTDS logging the domain controller logs (Event ID: 1535):

    Internal event: Active Directory Domain Services has encountered the following exception and associated parameters. 

    Exception:
    e0010003 
    Parameter:


    Additional Data 
    Error value:

    Internal ID:
    205199e

    Internal event: The LDAP server returned an error. 
     
    Additional Data 
    Error value:
    00000008: SysErr: DSID-0205199E, problem 12 (Not enough space), data 0

    The domain controller has plenty of space 80Gb. The issue seems to occur when LDAP clients use the filter syntax: memberOf:1.2.840.113556.1.4.1941:

    Has anyone seen this error before?.  Dcdiag and repladmin are clean.

    Friday, December 14, 2018 3:53 AM

All replies

  • It occurs both on Full GUI and server core.
    Friday, December 14, 2018 4:49 AM
  • We are having the similar issues when firing 'memberOf:1.2.840.113556.1.4.1941:' though the error does not seem to occur when the resulting search >~1000 or so. 

    As you we have recently deployed new 2019 domain controllers.


    Wednesday, December 19, 2018 8:21 AM
  • We are having exactly the same issue (with jenkins for example (sometimes the login works, sometimes it does not)).

    We also deployed Windows Server 2019 domain controllers recently.

    Have you found a solution/workaround?

    Thursday, January 3, 2019 7:59 AM
  • We rolled back to 2016 domain controllers. Clearly some bugs have appeared due to the memory changes they made in 2019 for Active Directory.
    Monday, January 7, 2019 6:30 AM
  • We have the same descriptive behavior in our environment.

    At any time the LDAP login fails, I can found a "not enough space" message at the same time in the directory log... :\

    I've created a MS Support case...

    Tuesday, January 8, 2019 10:02 AM
  • Please let others also know on what conclusion you became with the support.
    Tuesday, January 15, 2019 1:59 PM
  • Bad news...

    MS has closed the ticket as "won't fix" because I can't deliver a proper repro as case target.

    So: we rolled back to 2016 domain controllers too.

    Hopefully more users will report this behavior in future.

    Wednesday, January 16, 2019 3:36 PM
  • I just installed the new Jan update: https://support.microsoft.com/en-us/help/4476976/windows-10-update-kb4476976 and I can no longer reproduce this problem. 

    Are others able to confirm after installing this update.

    Thursday, January 24, 2019 10:14 PM
  • kb4476976 Didn't solve the problems for us unfortunately.
    Thursday, January 31, 2019 9:08 AM
  • still the same. Windows 2019 Server AD is unusable for us
    Friday, February 1, 2019 9:24 AM
  • This blog post describes the only known change in how AD handles memory, and seems relevant for this issue:

    https://blogs.technet.microsoft.com/askds/2018/10/02/deep-dive-active-directory-ese-version-store-changes-in-server-2019/

    In the comments, Daniel Appleby asked if this updated algorithm for the ESE version store caused the LDAP out of space problem. The blog author replied that he believes it does not. Still, to me this sounds like the cause, although admittedly after only a quick glance at the post.

    I would suggest anyone with Windows Server 2019 work through the troubleshooting section of the blog post, and perhaps make the registry changes mentioned, to see if it helps.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Friday, February 1, 2019 2:58 PM
  • In an environment of a customer we can reproduce the issue with the following PowerShell commando:

    Get-ADUSer -Server W2019DC -LDAPFilter "(&(objectClass=user)(objectCategory=person)(sAMAccountName=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2))(memberof:1.2.840.113556.1.4.1941:=CN=group name,OU=Groups,DC=company,DC=domain))"

    Result:
    Get-ADUSer : Server returned out of memory error

    When using OID: 1.2.840.113556.1.4.1941 for nested group extraction by AD the error occurs, without it doesn't. The eventviewer reports: SysErr: DSID-0205199E, problem 12 (Not enough space), data 0

    The error occurs most of the time, but once in a while it works. On Windows 2019 servers with more memory the command is working without problems more often than on servers with a lower amount of memory. The command process without problems on older OS versions.

    • Edited by Roy- Monday, February 4, 2019 12:04 PM
    Monday, February 4, 2019 12:03 PM
  • We are on KB4487044 it is still the same.
    Thursday, February 14, 2019 11:44 AM
  • I have been monitoring our bucket use with the perfmonitor and had a look at the event log, all seems clean.
    Thursday, February 14, 2019 12:33 PM
  • This is still occurring for us. Even after installing the update. My test had an  (&(cn=username)(memberOf:1.2.840.113556.1.4.1941=DN)) but just doing (memberOf:1.2.840.113556.1.4.1941=DN) returns the same error.
    Thursday, March 21, 2019 3:19 AM
  • Microsoft told me the root cause for this issue is found. A private hotfix will become available in 2/3 weeks.

    Tuesday, April 2, 2019 5:42 PM
  • Hi Roy, Could I ask you to notify, when you got a fix?
    Thursday, April 18, 2019 7:34 AM
  • Roy, 

    We are having similar issues using the memberOf:1.2.840.113556.1.4.1941  filter . 

    Would you be able to post here once you receive the hotfix and get it running? 

    Thanks... 

    Wednesday, April 24, 2019 2:03 AM
  • We have the same problem.
    Wednesday, April 24, 2019 1:21 PM
  • KB4495667 may have fixed this. Still testing - but looks promising so far.
    Monday, May 6, 2019 8:43 AM
  • No, worked only a few minutes after the reboot of the DCs.
    Monday, May 6, 2019 3:26 PM
  • Same issue on this end, reproducible in PowerShell.  I'm not seeing any events in the Directory Services log.

    EDIT: Also, the server itself has plenty of memory, and for the "version buckets allocated" limit (detailed in the Deep Dive article previously linked in this thread out of the 12,000 limit we have ... 2.

    • Edited by TMitera Thursday, May 9, 2019 6:15 PM
    Thursday, May 9, 2019 6:03 PM
  • For the purposes of anyone else either lurking on this thread currently, or for future posterity, I am working with Microsoft support on a support request of my own.  Current information is that the support rep believes that "[the issue] might be a known issue [and] will confirm it with [their] Product Team."  As I continue to get updated information I will similarly update this thread.
    Wednesday, May 15, 2019 5:56 PM
  • I got some more information from the Support Engineer, though it is far less definitive than I had hoped.  From the engineer:

    "I appreciate your time to report this problem to us and I am indeed sorry for any trouble this may have caused. I want to let you know Microsoft is aware of this problem. We have already reported this issue to our development team and they are currently working on this problem, it may take a long time to get a released fix."

    In short: Microsoft knows of the issue, is working on a fix, and has no ETA for said fix.

    What we are doing in the meantime is running a separate group of domain controllers (for example, running Server 2016) that sit behind a load balancer (ldap.domainname.local).  That way we can keep the Server 2019 DCs in place, but still have a place to point LDAP requests to that we know will work until the fix for 2019 is released and implemented.

    Monday, May 20, 2019 4:09 PM
  • Any news on this?
    Wednesday, May 22, 2019 7:39 AM
  • Disclaimer: I'm a Microsoft employee, and I am very familiar with this bug.

    Please don't blame my ESE version store change on this! ;)

    This isn't an ESE database issue; it's an LDAP memory arena issue. So that blog post is (fortunately for me) unrelated.

    We've root-caused the bug, but it will probably be a couple more months before the fix is public. (No firm ETA yet.)


    Wednesday, May 22, 2019 8:06 PM
  • Thanks for the update (and clarification)!

    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Wednesday, May 22, 2019 8:20 PM
  • *puts down somehow-ESE-shaped torch and pitchfork*

    I kid.  Actually Ryan I really appreciate this (obviously unofficial and not to be taken as gospel!) update.  It's good to hear that you / your team(s) are making progress on this bug.  What would the best way for us to keep ourselves up to date as to when the fix goes public?
    Thursday, May 23, 2019 6:35 PM
  • I'll update this thread since I basically "own" this bug. Current estimate is September.
    Wednesday, June 12, 2019 6:55 PM
  • A user recently reported a problem with the WinNT provider on Windows Server 2019. Only the first 100 users can be retrieved in a domain with over 2,000 users. The same WinNT query retrieves all users on Windows Server 2016. No errors were reported, and LDAP shouldn't be involved. But I wonder if it is possible that the same memory issue could cause this. Or is this an unrelated problem?

    The user replied to this old thread:

    https://social.technet.microsoft.com/Forums/en-US/c6539fc2-e0a4-41f9-b618-aa701a5219d4/vbscript-problem-with-winnt-object-only-returning-100-users-from-active-directory?forum=ITCG

    I have asked for the WinNT code that was used.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Thursday, June 13, 2019 6:13 PM
  • Its great its being worked on. 

    Is there a KB article describing it ? I couldn't find it in 2 weeks of searching. This thread is really the only place I can find mention of it. 

    Is there a workaround ? September is a long way off. Do I really need to re-promote a 2012 / 2016 server for LDAP queries in the meantime ?

    Sunday, June 16, 2019 11:52 PM
  • Near as I can tell, there is no workaround.  At the very least I haven't been able to find one, and in my support ticket with Microsoft the engineer was not aware of one that existed.
    Monday, June 17, 2019 1:48 PM
  • As I noted before, the problem reported with the WinNT provider on Windows Server 2019 may be different. But it is similar in that code that has worked for years on older versions of Windows Server, fails on Windows Server 2019. In this case, only 100 users are retrieved out of many thousands. The code used is similar to the following VBScript:

    Set objDomain = GetObject("WinNT://NetBIOSNameOfDC")
    objDomain.Filter = Array("user")
    k = 0
    For Each objUser In objDomain
        Wscript.Echo objUser.name
        k = k + 1
    Next
    Wscript.Echo "Number of users: " & CStr(k)
    


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Monday, June 17, 2019 3:56 PM
  • <rant>

    Over the last few years there have been a lot of complains about declining quality of windows OS and its cumulative updates. But this particular case is pretty telling. It been 6+ month since 2019 server GA and still "probably be a couple more months before the fix is public. (No firm ETA yet.)". And how come that such an obvious and easy to spot bug went into GA unnoticed? Did directory team forget to run their test suites?

    </rant>

    Tuesday, June 25, 2019 11:20 AM
  • We just deployed a couple of Windows Server 2019 domain controllers and experienced the exact same issue being described in this post in our environment. It's dependably reproducible using this Power Shell snippet posted by Roy a few month ago and is being logged at Windows' event log when setting the NTDS registry key "Internal processing" to DWORD:4. Since we got a lot of applications making use of LDAP queries containing (memberof:1.2.840.113556.1.4.1941:=...) we would be feasible if there was at least any workaround or Hotfix in the meantime waiting for an official bug fix.


    Thursday, July 11, 2019 9:09 PM
  • If your scripts retrieve direct and transitive members of a group, you can use the Get-ADGroupMember cmdlet and the -Recursive parameter. For example:

    Get-ADGroupMember -Identity "GroupName" -Recursive

    where "GroupName" should be either the DN, sAMAccountName, objectSID or objectGUID of a group.

    Edit: Otherwise, there is no workaround involving LDAP clauses, except to use more code to enumerate the direct members and recursively deal with members that have class group.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)


    Thursday, July 11, 2019 9:45 PM
  • For us this event has showed us that Server 2019 simply isn't ready for production usage.  It may operate well enough for the most part, but it worries me that an issue such as this is taking months to resolve with no workaround in the interim to mitigate the problem.

    Still eagerly awaiting news on this fix either way.

    Friday, July 12, 2019 12:59 PM
  • Get-ADGroupMember -Identity "GroupName" -Recursive.


    Richard Mueller - MVP Enterprise Mobility (Identity and Access)

    Thx for your suggestion, Richard. While this doesn't directly help us with our problem it made me think of a possible temporary workaround to write a small PowerShell script which could resolve the nested group memberships and add those resulting user objects to another "flat" AD group which could be directly addressed by various LDAP queries instead of relying on the currently faulty OID 1.2.840.113556.1.4.1941 parameter. This script could be triggered by a time interval (i.e. hourly) and/or by a specific EventID.

    Nevertheless we'd appreciate if things just worked like they used to on previous versions of Windows Server ADDS (2012R2/2016). LDAP and nested groups are still a thing in 2019 despite all those various claims based alternatives which Windows offers as well ;-)

    Friday, July 12, 2019 5:10 PM