none
KB4457129 breaks MSNLB used by VMs

    General discussion

  • When installing KB4457129 on our Hyper-V Nodes (Server 2012 R2 DataCenter with 2012 R2 Hyper-V Role), MSNLB (unicast in separate VLAN) breaks. We use it for VMs for Basic Network Loadbalancing our Exchange 2010 CAS Servers. After uninstalling KB4457129 from the Hyper-V Nodes and rebooting the VMs, everything goes back to normal and NLB is working again.

    The Hyper-V`s SwitchExtension changes from 6.3.9600.18937 to 6.3.9600.19123.

    It looks as if the Hyper-V virtual switch (logical switch) drops the unicast “flood” to the VM´s (members of the NLB Cluster) when KB4457129 is installed on Hyper-V Nodes.

    Any help appreciated…

    Regards

    Juergen

    Thursday, September 13, 2018 2:49 PM

All replies

  • Hi,

    Due to KB4457129 came recently, Microsoft is not currently aware of any issues reported with this update. 

    I'll also report your feedback to our product team. There might be some time delay. If any update, I'll post to you as soon as possible. 

    I totally understand the inconvenience this issue has brought to you, and your current feeling about this issue. If the issue is urgent, I would also suggest you contact Microsoft Professional tech support service so that a dedicated support professional can give more rapid support. You can find the phone number in the following link:

    https://support.microsoft.com/en-us/gp/customer-service-phone-numbers

    Besides, we can also click on the following link to give some suggestions and any suggestion will be appreciated.

    https://windowsserver.uservoice.com/forums/295047-general-feedback

    Hope above information can help you. If you have any question or concern, please feel free to let me know.

    Best regards,

    Michael


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com

    Friday, September 14, 2018 2:46 AM
  • Hi Michael,

    thanks for the reply.

    I did some testing today in our Test-Environment and I was able to reproduce the issue. We have a 4 Hyper-V nodes cluster in this environment, 2 of those nodes are fully patched with all patches delivered in August 2018, 2 of them are fully patched with patch level September 2018 including KB4457129.

    There are two VMs in our NLB-“Cluster” (Exchange 2010 Sp3 UR22 CAS-Server, Windows Server 2008 R2 Standard with MSNLB configured on the VMs on a separate NIC in a dedicated VLAN, fully patched with September 2018 Patches including KB4457129).

    As soon as one of those two VMs is migrated to a Hyper-V node with patch level September 2018 and the VM is rebooted on this node, the VM doesn´t respond to IP Packets sent to it from outside the NLB Cluster. If the VM is migrated back to a Hyper-V node with patch level August 2018 and rebooted on this node, it does respond to NLB IP packets again.

    If both VMs are migrated to Hyper-V nodes with patch level September 2018 and are rebooted on this node, both of them do not respond to incoming NLB Cluster traffic anymore. When they are moved back to Hyper-V nodes with patch level August 2018 and are rebooted on this nodes, NLB Cluster works again like expected.

    I also tested a scenario with a standalone Hyper-V node, patch level August 2018. I migrated all VMs (same as described above) to this Node, MSNLB for our two CAS-Servers worked as expected. Then I installed KB4457129 on this Hyper-V node, and rebooted it. As soon as the VMs were up again, they didn´t respond to any incoming NLB-Traffic. I removed KB4457129 only from the Hyper-V node, rebooted Hyper-V and the VMs again, the VMs came up and NLB worked as expected…

    In short, removing KB4457129 only from Hyper-V nodes and rebooting the VMs solved this issue for us...

    Maybe MAC-Address-Spoofing configured on the VMs dedicated NLB NIC breaks with KB4457129 installed on Hyper-V Nodes… or unicast NLB Packets are dropped by the virtual Hyper-V switch...?

    Best regards,

    Juergen

    Friday, September 14, 2018 1:28 PM
  • KB4457131 exhibits the same exact problem on Windows Server 2016 Hyper-V. We got burned by this Friday and finding this post allowed us to fix our problem.

    Microsoft, who is QA-ing these updates?

    Saturday, September 15, 2018 5:46 PM
  • KB4457131 caused a multi-hour outage for my company this week.  We are a B2B service provider, and we received numerous calls from furious customers that could not access websites we host on NLB servers.

    We are still calculating how many thousands of dollars we lost and caused our customers to lose.  The damage to our reputation is incalculable.

    Nearly every update this summer has been plagued by serious, serious problems.  My question is the same as banduraj's.  Who, if anyone, is quality checking the updates?


    -Tony

    Sunday, September 16, 2018 1:55 AM
  • Was uninstalling KB4457131 enough to resolve the issue or did you have to take other action?   I'm asking because I've got a server that's impacted and uninstalling KB4457131 has not resolved the issue.    The VMs that have Unicast NLB configured are not able to communicate outside themselves.  
    Sunday, September 16, 2018 8:49 PM
  • Was uninstalling KB4457131 enough to resolve the issue or did you have to take other action?   I'm asking because I've got a server that's impacted and uninstalling KB4457131 has not resolved the issue.    The VMs that have Unicast NLB configured are not able to communicate outside themselves.  

    Hey James,

    We were experiencing this issue as well on our Hyper-V Server 2012 R2 hosts. We have successfully resolved the issue by uninstalling the two KB's mentioned in this post and restarting the host:

    - KB4457143

    - KB4457129

    We believe that uninstalling these KB's from the Hyper-V Host is what ultimately resolved our issue. Although in troubleshooting we also uninstalled these two KB's from our ADFS / ADFP servers that were using NLB. We didn't uninstall these from our RDS environment that uses NLB for internal applications, and so far internally the issue is resolved. But, externally to the RDS NLB it's not working properly. This may be due to not uninstalling the KB's from the VMs as well. I plan to troubleshoot this further tomorrow. Hopefully this is helpful to resolve your issue!

    Sunday, September 16, 2018 9:32 PM
  •  

    Hey James,

    We were experiencing this issue as well on our Hyper-V Server 2012 R2 hosts. We have successfully resolved the issue by uninstalling the two KB's mentioned in this post and restarting the host:

    - KB4457143

    - KB4457129

    We believe that uninstalling these KB's from the Hyper-V Host is what ultimately resolved our issue. Although in troubleshooting we also uninstalled these two KB's from our ADFS / ADFP servers that were using NLB. We didn't uninstall these from our RDS environment that uses NLB for internal applications, and so far internally the issue is resolved. But, externally to the RDS NLB it's not working properly. This may be due to not uninstalling the KB's from the VMs as well. I plan to troubleshoot this further tomorrow. Hopefully this is helpful to resolve your issue!

    Bryan,

    Thanks for the response.  

    I was able to resolve the issue by deleting and recreating the NLB cluster / configuration from the VMs that had it configured.  I had done that earlier, with KB4457131 in place, before I found this post.   Maybe that was why simply removing KB4457131 did not fix it for me.  

    For what it's worth, I do have KB4457131 on all the VMs - I'm only running 2016 servers.   I don't have KB4457129 or KB4457143 as those only apply to 2012r2 machines.  

    Sunday, September 16, 2018 11:10 PM
  • James,

    I appreciate the update, I'll definitely try rebuilding the NLB cluster / configuration if I experience additional issues. Just as an update here, I started investigating the RDS NLB issue. To resolve the NLB on the RDS environment I uninstalled the same KB articles (KB4457143 and KB4457129) from the VMs that used NLB. Restarted them, and afterwards external access to the NLB was restored. So based on my testing today I am assuming that the updates have to be uninstalled from both the VM and the Hosts that utilize NLB.

    Thanks,

    Bryan

    Sunday, September 16, 2018 11:58 PM
  • Was uninstalling KB4457131 enough to resolve the issue or did you have to take other action?   I'm asking because I've got a server that's impacted and uninstalling KB4457131 has not resolved the issue.    The VMs that have Unicast NLB configured are not able to communicate outside themselves.  

    We simply had to uninstall KB4457131 from our Windows 2016 Hyper-V hosts, and reboot of course.  All the VMs (Windows 2012R2 & 2016) have all the updates installed.  You need to reboot the NLB VMs after the hosts have been downgraded.

    -Tony

    Monday, September 17, 2018 6:15 AM
  • I can confirm that, in my environment, it's only the update on the host that matters.  All the VMs have KB4457131.   I did have to reconfigure NLB on the VMs after removing KB4457131 from the host (and rebooting the host and the VMs running NLB) but that might be because I had previously reconfigured NLB, with KB4457131 installed, as part of my troubleshooting.
    Monday, September 17, 2018 2:31 PM
  • You know, it would be really helpful if there was some sort of description as to what these updates actually do on the change history pages. It might have saved us some time in identifying the problem to begin with.

    https://support.microsoft.com/en-us/help/4457129
    https://support.microsoft.com/en-us/help/4457131

    One bullet point that says "Security updates to Windows media, Windows Shell, Windows Hyper-V, Windows datacenter networking, Windows kernel, Windows virtualization and kernel, Microsoft JET Database Engine, Windows MSXML, and Windows Server." is obviously not at all useful for the IT pros in the field that have to deal with these issues when applying patches.

    Monday, September 17, 2018 7:08 PM
  • I had the same issues. Although removing the patches

    - KB4457143

    - KB4457129

    did not help completely, for some users it worked, others were still facing the same problem.

    I also tried to recreate the OL Profile, which always ended with not finding the users account.
    This led me to think about autodiscover. Trying to run the Outlook "Test E-mail AutoConfiguration" always failed.
    Trying to reach the autodiscover.xml file by a Web Call failed with a Server Error and this error is a known issue. 

    Finally the following article helped to circumvent the problem:

    https://support.microsoft.com/en-za/help/2020789/you-receive-a-server-error-while-browsing-the-exchange-ews-or-autodisc

    HTH


    Regards, Jim

    Tuesday, September 18, 2018 7:42 AM
  • We also had our NLB fail after this patch installed.  We attempted to re-create our NLB setup and took a look at our switch configs but nothing worked.  Hoping this issue gets resolved soon
    36 minutes ago