Server 2008 R2: DNS service stops with Event ID 7023 "A thread could not be created for the service" with 64 or more processors / cores
-
jeudi 27 janvier 2011 15:48
HP DL980 G7, 8 sockets of X7560 (8 cores each), 1TB RAM
8x8 = 64 logical processors, Hyper-Threading switched off so I can run Hyper-V
After a completely clean boot with no non-Microsoft startup applications or services, DNS server stops running after a few seconds, with "A thread could not be created for the service" in the System log. Event ID 7023. It consumes 3.9GB RAM before terminating.
When the same machine is restarted with only 4 sockets enabled (4x8 = 32 logical processors), DNS.EXE runs fine.
The response after several rounds of debugging and hotfixes from Microsoft support is:
This is a by-design problem in DNS.exe in this scenario with a very large amount of CPU resources available.
Inside the DNS Server service, there is a thread count limitation (maximal 120 threads). However, it does not limit processors (or cores).
On 64 cores server, DNS Dynamic update routines will create 64 Update threads, and UDP I/O and dispatch routine will create 64 UDP Listen threads. Besides that, the following threads will also be created for DNS operation,
"Recursion Timeout" thread
"Secondary Control" thread
"DsLoad_Thread" thread
"Timeout_Thread" thread
Therefore, the thread limitation will be reached: 64 *2 + 4 = 132 > 120, and ERROR_SERVICE_NO_THREAD error would occur.
On 32 cores server, the DNS service would be running fine as it only creates 32 * 2 + 4 = 68 threads.
This is a by-design behavior and it will not be fixed until Windows 8. Currently, the only solution is to use less than 32 cores.
To this situation, my suggestion is:
1. Install the DNS on another server.
2. Decrease your CPU cores less than 32 cores.
My perspective is:
Server 2008 R2 is supported on machines with 256 logical processors. There is no documentation anywhere that says that the DNS service will fail on machines with more than 58 logical processors. This is clearly a bug, it is clearly not “by design”.
DNS is a critical core service, required for Active Directory and basic name resolution across a network. This is the only server in a standalone DMZ data center, and it needs to run AD and DNS, in case a network link fails and VMs are unable to reach another DNS server. I have spent a great deal of money for a supported, 64 core machine to run Microsoft SQL Server with high performance. So running with fewer cores in unacceptable either.
I don’t need DNS to run with 64 update and 64 listen threads. It would be just fine with DNS to use 32 threads (or fewer) for each. This DNS server may have to serve just a few clients per second. It is unacceptable that DNS should completely fail on a supported configuration. The service should just run with fewer threads.
It should be a trivial code change to make this happen, to restrict the update and listen thread count to 58 (or fewer) threads each.
Microsoft support says that a hotfix is unlikely to be produced to fix this problem in advance of Windows 8 / Server 2012. So:
1. I wanted to document this issue so that others are aware. I have not found it documented anywhere else.
2. I wanted to find out if there are other ways to limit the number of processors visible to any particular process
3. Is there a way to increase the pressure on Microsoft to provide earlier resolution to this issue?
Thoughts anyone?
Toutes les réponses
-
mardi 2 août 2011 12:09
HI,
Try using Windows System Resource Manager feature and/or Core Parking.
Hope it helps...
MCTS... -
lundi 16 avril 2012 21:19
Marius ..
I see you can park CPU's via power management but DNS still crashes with the same error. Any more feedback since your last post?
-Tom
-
lundi 16 avril 2012 21:28
Running anything else on a hyper-V server is not recommended neither... You may create VM that will host DNS and AD services.
Since your post is old, i guess you found out another solution for your problem.
Mathieu Chateau http://www.lotp.fr
-
mercredi 9 mai 2012 01:48Yes, we are having to run a VM on this server just for DNS to get around this. Seems like an overly heavyweight solution to what should be a simple problem to fix.
-
lundi 20 août 2012 03:27
David,
Did you get anywhere with this? I am running into the issue at a client and I was informed that this is now deamed a bug but it is "DoNotFix". It is addressed in Win12. Not sure if that means a back port for us on Win08R2.
Also to add more wierdness to this issue. I have DNS working flawlessly on an server with 80 cores running datacenter. Both servers are HP DL580 G7 boxes with four procs. The unhappy server has 4 x 8-core CPUs and running enterprise. The happy one has 4 x 10-core CPUs and running datacenter.
Ron
-
lundi 20 août 2012 17:45
No, we just gave up and run DNS in a VM. We are migrating to Server 2012 asap, so I'm glad it has been fixed in the new version.
That's strange about it running fine on the machine with 80 logical cores (I assume you are using hyperthreading), given Microsoft's explanation of the root cause, in my original post above.
Perhaps Datacenter 2008 R2 has some corrections to the code to make it work, as it expects to be running on a machine with a large # cores, but that they didn't also make these fixes to Enterprise?
- Modifié David Trounce lundi 20 août 2012 17:48
-
lundi 27 août 2012 14:11
Funny part I checked the registry and the actual versions of the obvious files side-by-side for both servers they were the same.
I found a new list of oddball files to check. We ended up blowing the server away and adding a dedicated DC & DNS server.
Pretty unenlightened 'solution'. Also pretty lame that MSFT has chosen to keep the issue under the rug.
Hopefully since it is 'fixed' in 2012 they may back port a patch but I am only dreaming here. [I frequently find myself at times like this dreaming of rainbows and unicorns.]
-
vendredi 2 novembre 2012 06:07I can confirm that this DNS bug is fixed in Windows Server 2012. We just put that on our DL980 G7.
.png)
