none
KB3145126 Causing DNS.exe Crashes? RRS feed

  • Question

  • We applied KB3145126 hotfix (along the rest of April 2016's patches) on our two of our 2008 R2 DNS servers this past weekend. Unfortunately the DNS service failed to start after reboots and we suspect it's this optional hotfix.

    It was late and we weren't able to properly isolate if this hotfix was the culprit but after we uninstalled all of April 2016's patches, our DNS service started without a hitch. We won't be able to confirm our hypothesis until next month's maintenance window but decided to post this here just in case someone else runs into DNS.exe crashes.

    Here is an Event Viewer entry of DNS.exe failing to start:

    Log Name:      Application
    Source:        Application Error
    Date:          4/24/2016 1:22:30 AM
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      Server4
    Description:
    Faulting application name: dns.exe, version: 6.1.7601.23375, time stamp: 0x56e06454
    Faulting module name: dns.exe, version: 6.1.7601.23375, time stamp: 0x56e06454
    Exception code: 0xc0000005
    Fault offset: 0x000000000006138f
    Faulting process id: 0xecc
    Faulting application start time: 0x01d19de94af3fe67
    Faulting application path: C:\Windows\system32\dns.exe
    Faulting module path: C:\Windows\system32\dns.exe
    Report Id: 8a41937a-09dc-11e6-9922-e4115be71ecc

    

    

    Monday, April 25, 2016 1:00 PM

All replies

  • Hi Learned_Individual,

    >> We won't be able to confirm our hypothesis until next month's maintenance window but decided to post this here just in case someone else runs into DNS.exe crashes.

    Thanks for your sharing and sorry for the inconvenience caused to you.

    https://support.microsoft.com/en-us/kb/3145126

    Please try to upgrade the latest updates on the next month's maintenance to see if this issue is still persist.

    Hoping to get more feedback from you.

    Best regards,


    Andy_Pan

    Tuesday, April 26, 2016 5:43 AM
  • We saw the same behaviorist for all our Windows SErver 2008 SP1 DC's where the update was applied. 
    Tuesday, April 26, 2016 8:53 AM
  • Hi Martin,

    >>We saw the same behaviorist for all our Windows SErver 2008 SP1 DC's where the update was applied. 

    Please removing this hotfix and waiting for the next updates.

    And I will keep tracing of this issue, if I find a new hotfix or updates for this issue I will updating soon.

    Best regards,


    Andy_Pan



    • Edited by Hello_2018 Wednesday, April 27, 2016 6:13 AM
    Wednesday, April 27, 2016 3:23 AM
  • Hi,

    I'd just like to add that I've seen exactly the same problem on one of my 2008 R2 DCs - fortunately, I didn't patch all of them at the same time! I can confirm that removing 3145126 cured the problem.

    Thanks Learned_Individual, I was running out of ideas for what was causing it.

    Cheers,


    Dick

    Wednesday, April 27, 2016 9:32 AM
  • We suffered the same phenomenon on our 2008 R2 DC.  I can confirm that uninstalling KB3145126 allowed us to start the DNS Server service.

    Tony Sperbeck


    Wednesday, April 27, 2016 4:06 PM
  • Same problem in our network. Three 2008 r2 domain controllers, one for each location. All three of them went offline after this horror update. We had a long night of searching and evaluating before we found the faulty update and this post.

    In our case the problem occoured on standard and enterprise servers, all of them in german localization.

    Norbert

    Wednesday, April 27, 2016 9:40 PM
  • Hi all,

    Thanks for all of your feedback.

    Besides, please try to covert the ADI zone to Primary.

    In addition, please also follow the following link to see if it helps:

    https://support.microsoft.com/en-us/kb/3100465

    Best regards,


    Andy_Pan

    Thursday, April 28, 2016 1:50 AM
  • Same happened to our infrastructure on Tuesday. Panic stations until we figured out what it was. Only the 2008r2 DNS servers on the blade were effected, but due to some bad scoping our 2012 standalone didn't resolve the IPs. Worse still, we failed to disable the update in WSUS properly and it reapplied again overnight. Not our finest hour. 
    Thursday, April 28, 2016 7:56 AM
  • Same problem here with Win 2008 R2. Uninstall fixed the problem.
    Thursday, April 28, 2016 10:59 AM
  • Can confirm this is occurring for us as well. Server 2008 R2.
    Monday, May 2, 2016 2:26 AM
  • Thanks for posting this - it allowed me to resolve the issue immediately!  Microsoft is brutal - not sure how this passed QA
    Friday, May 6, 2016 1:22 PM
  • In my environment, DNS zones failed to load back in December right after we applied KB3100465. We then removed that patch, rebooted, and waited for Microsoft to come up with a fix for it.
    When you applied KB3145126, had you already implemented the
    "dnscmd /Config /DSMinimumBackgroundLoadThreads 0" work-around described here:
    https://support.microsoft.com/en-us/kb/3100465

    ?
    If so, did you first remove this work-around (change it back to 1), before applying KB3145126? I ask because the failures you describe concern me, so I would like to know what other possible factors (other than the patch itself) led to them, lest they appear later in my environment. We do not want to remove KB3145126, because we have already applied it to fix the failure to load DNS zones that was caused by KB3100465. It did fix that failure, and it has caused no other issues (so far).
    Currently, we have both KB3100465 and KB3145126 applied without any problems. We did not apply the work-around that was recommended before the release of KB3145126. We just waited until KB3145126 was released, and then applied KB3100465, rebooted, and then applied KB3145126. Please let me know what, if anything you did differently.

    

    JCSunday

    Monday, May 9, 2016 5:36 PM
  • Hi Learned_Individual,

    We applied updates last week and noticed the same behavior.  The DNS.exe would start and stop giving us the error you referenced.  We found that KB3145126 was installed.  I uninstalled it and DNS started working again.

    • Proposed as answer by snokohn Saturday, May 21, 2016 4:30 PM
    Tuesday, May 10, 2016 12:12 PM
  • Exact same problem. Thank for dropping the Ball MS, you knocked my enterprise offline. Thank god someone over at spiceworks is paying attention. . . smh...
    Wednesday, May 11, 2016 11:30 AM
  • Exact same problem. Thank for dropping the Ball MS, you knocked my enterprise offline. Thank god someone over at spiceworks is paying attention. . . smh...
    Not trying to side with Microsoft here but are you saying that you don't TEST your patches before you apply them to a critical piece of your infrastructure?  It is pretty naive to assume that you will never have an issue when you patch a machine, so you should be patching to a test environment...best practices and such  :). (again, Not trying to defend Microsoft, but ultimately you are responsible for your network, at least IMHO)
    Thursday, May 12, 2016 6:30 PM
  • Personally, I've always found that a difficult concept to grasp. We look after dozens of SME networks - from single server environments to muli-sites - but nothing more than 100 seats. How is it possible to have a test environment for each? Sure we have a test server that we patch first at out office, but our clients often have completely different setups. Some have server 2008, some 2012. Hell some still have SBS2011. Some run DNS servers, some don't. Some have 3rd party software installed on the DC. Some don't.

    This bug is a perfect example. It knocked out one of our clients, but not others. MS also released a patch in April that knocked out a couple of servers with Storage Craft installed. Wouldn't even boot!

    I would hope MS do extensive testing before releasing patches. If they can't find a bug in their tests, how can we?

    I must admit, I'm noticing a lot more buggy updates and patches being released lately. Probably seen about 4 or 5 this year so far...

    My only way to try and manage this so far is to hold off on updates for a few weeks and keep my eye on the forums. But even that sometimes fails...

    Thursday, May 12, 2016 10:04 PM
  • Had the same issue....we have 56 DC's in multiple domains and forests (porduction environment)

    issue - KB3145126 caused issues on most of our DC's, but not all.  we had 10 Dc's that had no issue with the patch (all DC's are built from same process, and patches etc).

    there where no issues in testing in our test environment (18 DC's).

    Friday, May 13, 2016 1:28 PM
  • Regarding patching testing, I'm somewhere in between what Michael_Goff and TheSpook said.  It can be challenging to have a truly equivalent test environment for ADDS/DNS, but it is essential to mitigate the impact of mal-formed Windows Updates.  I think there is a middle ground...

    I'm certain that most people have multiple domain controllers and multiple DNS servers for HA's sake.  The technique I use is to stagger the installation of new Windows Updates (Wednesdays & Sundays) to my domain controllers/DNS Servers.  This bad update only affected 50% of my DNS servers, and the impact to the my dual-homed DNS Clients was actually negligible.

    Friday, May 13, 2016 2:15 PM
  • That's a smart call, Tony. We have a number of smaller clients with just 2 boxes - a DC and a file / email / SQL server etc. I've often debated with myself whether I should make the second box a backup DC, just in case. Traditionally I like to keep DC functions on their own server. I guess I've always felt that is best practice. But if it has the resources, I guess there is no reason not to go down that path. For the really small single server environments that still isn't an option though.

    I tell you what though, that's where I have come to loooooove hyper-v. These days we never go bare metal, even in single server environments. Just add the hyper-v role, make a VM and make that the DC. Now we do a 2 minute snapshot before updates and if they stuff up, revert the snapshot. Makes life so much easier!

    Friday, May 13, 2016 10:36 PM
  • Same symptoms.

    KB3145126 broke DNS on Win2008r2 DCs.

    Uninstalled this patch and DNS working again.

    How did this pass Microsoft QA?

    Saturday, May 14, 2016 5:25 PM
  • This worked for me! DNS.exe did not start crashing until after I installed May 2016 Patches and rebooted.  After hours and hours of troubleshooting (thanks Microsoft), if finally found your post. Thank you! 
    Sunday, May 15, 2016 12:45 PM
  • Same exact symptoms here after applying this patch. Unfortunately, while we stagger our patch installs for DCs, the window is too small and all got patched before anyone answered the alarm about DNS being broken on the first system.

    How exactly did this patch (KB3145126) get marked as "recommended" in WSUS? A Google search reveals reports of it breaking things frequently go back at least a full month when it was only an optional hotfix?!

    Sunday, May 15, 2016 3:15 PM
  • We have the same problem on all of our DC's. Any updates?
    Tuesday, May 17, 2016 3:09 PM
  • Day and 1/2 trying to figure out why DNS kept crashing on this machine.  Remove and put back DNS, no bueno.  Tried 10 other things, nada.  Removed this update and DNS starts right up on reboot.  Thanks Learned_Individual for the troubleshooting and resolution.

    Dude, you Rock!


    Wednesday, May 18, 2016 1:45 PM
  • Thanks!! we just had the same issue today removing The patch 3145126 Fixed it
    Saturday, May 21, 2016 4:29 PM
  • had the same issue at a customer today on 2 Server 2008 R2 DCs - uninstalling KB3145126 solved the issue and DNS service started working again !
    Monday, May 23, 2016 3:20 PM
  • Andy_Pan,

    Have you discovered the solution? KB3145126 clearly causes DNS.exe to crash and disables critical domain functionality. This has got be known to Microsoft as a very serious problem. What is Microsoft's solution? 

    George


    George Perkins

    Monday, May 23, 2016 7:05 PM
  • Hi all,

    Thanks you for your attention.

    I confirmed that the Product Group is currently working on getting a permanent fix released.

    As of now, the workaround is to get the KB uninstalled (or to disable Background Zone Loading).

    Thanks again for all of your understanding.

    Best regards,


    Andy_Pan

    Tuesday, May 24, 2016 2:10 AM
  • Hi Everyone, take a look at this blog:

    https://blogs.technet.microsoft.com/supportingwindows/2016/04/25/fix-loading-dns-zones-takes-a-long-time-on-a-windows-server-2008-r2-based-dns-server/

    See the top section under Update for your specific scenario.

    Wednesday, May 25, 2016 3:29 PM
  • Hi Everyone, take a look at this blog:

    https://blogs.technet.microsoft.com/supportingwindows/2016/04/25/fix-loading-dns-zones-takes-a-long-time-on-a-windows-server-2008-r2-based-dns-server/

    See the top section under Update for your specific scenario.


    Didn't work for me. The script says "No zones found with the issue" and dns.exe still crashes.
    Thursday, May 26, 2016 11:04 AM
  • AJAYPS,

    Since you are proposing this as an answer, can you perhaps help out a bit with some more details? The link you reference "under Update for your specific scenario" contains another link to MS12-017 from April 2012: https://support.microsoft.com/en-us/kb/2647170 which contains a section "Known issues with this security update" that contains this description: "Cause: This issue may occur if DNS is configured to have a CNAME and a SOA record that both exist for the "@" record. The "@" record identifies the root of a DNS zone. This can frequently be identified in the DNS Manager as a record with the "(same as parent folder)" name. The SOA and NS records are allowed in this folder. RFC 2181 describes name uniqueness checks for CNAME records. According to RFC 2181, the CNAME may not exist in the “same as parent folder" ("@") of a zone." and then provides a sample PowerShell which indicates it will identify zones containing the described invalid CNAME records. However, when I run the script it produces the result "No zones found with the issue". Therefore, I don't think your proposed answer to this thread applies, as I don't think the KB from 2012 matches our environment.

    Will Microsoft address the quality issue of KB3145126 causing DNS to crash?


    George Perkins

    Thursday, May 26, 2016 1:39 PM
  • Hi George Perkins,

    That's interesting as the known issue described in KB 2647170 is the reason I have seen so far the DNS service does not start.

    Would you mind posting the DNS.EXE version before and after patch?

    Friday, May 27, 2016 12:12 PM
  • I'm confused why you would even ask such a question.  The before version is anything before 6.1.7601.23375.  The after version is 6.1.7601.23375.  It has nothing to do with the price of rice in China.
    Friday, May 27, 2016 12:45 PM
  • I don't have access to the "after" version because we removed KB3145126. However I can tell you the "before" version of DNS.exe:  6.1.7601.19046 10/21/2015

    George Perkins

    Friday, May 27, 2016 1:49 PM
  • Andy_Pan,

    Any update on this? Does Microsoft DNS internals acknowledge this is an error in KB3145126 and is now providing new guidance on the resolution to the issue KB3145126 is designed to address? Has KB3145126 been withdrawn? Is there an updated binary update replacement? Thanks.

    George


    George Perkins

    Wednesday, June 1, 2016 3:44 PM
  • It is horrible how Microsoft is (not) handling this issue
    Thursday, June 2, 2016 7:29 AM
  • Hi George Perkins,

    We have already published the gudiance from the root cause we investigated.
    I am not sure on the sequence of steps you followed to detect the @ cname record, but below are the steps that are needed:

    1) Ensure the DNS service is started before you run the script (DNS service will not be in running state if you have the patch KB3145126 installed and have a @ cname record)
    2) Run the script to detect the @ cname record
    3) Remove as mentioned (DNSCMD /recorddelete DNS zone name @ cname)

    If the steps have been followed as per the above with DNS service running, and you still see "no zone found with this issue" with DNS service crashing, I request you to open a Microsoft support case. If you open a support case, please ask the support engineer to contact me (My name in the forum is what they need to find me) 

    Thank you!
    • Proposed as answer by mtimley Thursday, June 9, 2016 9:53 AM
    Thursday, June 2, 2016 11:31 AM
  • As i had indicated earlier in this thread, when I run the script the result is "No zones found with the issue" (ran with KB3145126 uninstalled; and the version of DNS.exe is 6.1.7601.19046 10/21/2015).

    George Perkins

    Thursday, June 2, 2016 2:05 PM
  • Hi Ajay & Andy,

    I would like to know why this optional hotfix from April became a recommended patch for May that broke both of our DNS servers last night & caused us a lot of grief?!  Why would it be pushed out in this patch group when there was already so much fuss about it as an optional hotfix?  It's getting hard to trust Microsoft with these updates.  We already hold off taking them by 2 weeks (or more in this case) expecting/hoping that issues will be discovered & corrected by the time we take the patches.  We look for info on grief with patches beforehand & didn't catch anything on this one.  Clearly we'll have to put more time into that going forward!

    Friday, June 3, 2016 5:18 PM
  • Bump.

    George Perkins

    Wednesday, June 8, 2016 9:14 PM
  • Should we all open a "Microsoft support case" ?
    Thursday, June 9, 2016 9:20 AM
  • Thanks, removing CNAME for @ solved my issue with KB315126.
    Thursday, June 9, 2016 9:57 AM
  • mtimley,

    Before removing "CNAME for @" did you run the PowerShell script in the article https://support.microsoft.com/en-us/kb/2647170 (which contains a section "Known issues with this security update")? And doing so, did the script return a result, or like my organization, gave a message "No zones found with the issue"?

    I ask because it would seem my organization does not seem to have the same data causing symptom.

    Frankly, I am very disappointed with this response from Microsoft. Any good programming should not crash a service simply because of some "invalid" value found in a database. At worst the program might treat the invalid data (in this case "CNAME for @") in some unexpected way, but at best should write an error message to a log and handle the invalid data in some elegant non-disruptive way. Clearly, without KB3145126 installed, our DNS is behaving normally. With KB3145126 installed DNS.exe is crashing the DNS service and disabling all functionality of the domain. (No DNS, almost nothing works). 

    This is simply too critical of a system component for the light treatment Microsoft is giving it.


    George Perkins

    Thursday, June 9, 2016 1:34 PM
  • Before removing "CNAME for @" did you run the PowerShell script in the article https://support.microsoft.com/en-us/kb/2647170 (which contains a section "Known issues with this security update")? And doing so, did the script return a result, or like my organization, gave a message "No zones found with the issue"?

    Yes, the script returned a result when I run it second time without KB3145126 installed (and with working dns server).

    I totally agree how critical this issue, seems that Microsoft don't care quality of their products even for such crucial services.
    Thursday, June 9, 2016 4:03 PM
  • Hi Learned,

    Were you able to run the script from here to get CNAME results?

    Rick

    Tuesday, June 14, 2016 4:51 AM
  • What am I doing wrong?

    http://prntscr.com/bgnbu4

    Wednesday, June 15, 2016 11:50 AM
  • The script has an error on the first line.  It should be split into two lines like this:

    $count = 0

    $var = get-wmiobject -query "select * from win32_service where name = 'dns'"

    Wednesday, July 13, 2016 4:07 PM
  • Script returns:

    No zones returned
    No zones found with the issue

    Still the same issue

    Monday, July 18, 2016 6:29 AM
  • Jason224,

    When I look at the script in https://support.microsoft.com/en-us/kb/2647170, it contains the same two lines of code you propose has having an error (I don't see any difference). But looking at the example code in KB2647170 I see some issues. I wonder if Microsoft can respond to their intent in KB2647170 (I know, it has a disclaimer that Microsoft does not support example code) but what is provided does not contain enough information and it doesn't work to solve this issue.

    1. Where should this code be run? Looking at the code I presume that it needs to run on a DNS server (domain controller). Correct?

    2. Looking further at the code, it doesn't work. I ran the WMI queries in the code and they failed. 

    3. What is the meaning of the namespace "root\microsoftdns" and select "microsoftdns_zone" values in the code? Should these values be replaced with customer-specific locally-unique values?

    Thanks for bringing this up Jason224. Now if Microsoft will respond that will be nice.


    George Perkins


    Monday, July 18, 2016 3:24 PM
  • ngiii,

    I also get that result. But I think it may be only because the example code provided by Microsoft needs to be modified before it does anything useful.


    George Perkins

    Monday, July 18, 2016 3:25 PM
  • Microsoft has just updated the Known Issue section of KB ("https://support.microsoft.com/en-us/kb/3145126") with a working copy of the PowerShell script that detects the problem root CNAME record. It now describes the problem exactly that we were having.

    We will remove the offending CNAME and then re-apply KB3145126 update and confirm.

    P.S. It pains me to observe that whoever wrote the code in KB3145126 has a poor grasp of exception handling. Instead of causing DNS.exe to fail, a graceful error-recovery and error message would have been a much better solution. Sigh.


    George Perkins

    Thursday, August 11, 2016 8:26 PM
  • The script identified my offending article too. Removed that, reapplied 3145126 and everything seems fine today, so looks like they've finally cracked it. Quite a saga...
    Monday, August 15, 2016 6:55 AM
  • I don't see any updated script in the KB - its the same version I had on the DC and it returns the same error. Can you post the new version here?
    Monday, August 15, 2016 3:06 PM
  • Reporting my results as promised. After removing the @ CNAME issue from one of our AD-integrated DNS zones, and re-applying the KB3145126 update, DNS.exe no longer is crashing. This is finally fixed.

    As far as whether the PowerShell script "works", I can't say for certain: But now at least Microsoft has included it as part of the description of a known issue in KB3145126. When I ran this script on one of our Windows Server 2008 R2 domain controllers, it reported an invalid CNAME in one of our AD-integrated DNS zones. Per the instructions, I removed the offending CNAME entry - in my case, the entire DNS zone was determined to be obsolete, so that entire zone was deleted. This solved the problem.  (However, a similar script contained in KB2647170 had previously reported no issue!! I think the script was updated in KB3145126 or perhaps there are some unknown circumstances which do not give consistent script output?) I'd suggest try re-running the script from several of your domain controllers, or all of your domain controllers, and see if you get consistent results.


    George Perkins


    • Proposed as answer by George Perkins Monday, August 15, 2016 3:26 PM
    • Edited by George Perkins Monday, August 15, 2016 3:32 PM additional info
    Monday, August 15, 2016 3:25 PM
  • I did open a Microsoft support case. Initially Microsoft was going to close the case as 'solved' after I uninstalled KB3145126, but that was not a good enough answer for me. :) I kept the pressure on and insisted Microsoft acknowledge the defect. They did finally accept the defect and it is acknowledged now as a "known issue" and the description of KB3145126 was updated accordingly. The known issue applied to our environment and I closed the case with Microsoft.

    George Perkins

    Monday, August 15, 2016 3:36 PM
  • I used the current script from https://support.microsoft.com/en-us/kb/3145126 and it just throws an exception and the result is something linke "No zones found".
    Monday, August 15, 2016 4:42 PM
  • Had the same problema. Uninstalling KB3145126 worked!  (no reboot required)

    Thanks

    Thursday, September 15, 2016 5:37 PM
  • Same thing happened to me today and removing the hotfix resolved the issue.
    Wednesday, December 14, 2016 10:36 AM