locked
IPv6 Network Flood with Wireless Hyper-V Virtual Network RRS feed

  • Question

  • Recently, on our core switch (also our layer 3 router) we regularly see a high cpu load. This cpu load is the cause of packet loss and bad connections. This is happening for almost two weeks now.

    Network inspection shows us that there is a flood of IPv6 Neighbor Advertisements originating from what seems the same IPv6 Address but from different Hardware (MAC) addresses. In each second there are up to a 1500 packets per second advertising the same IPv6 address. See below for more details about these packets.

    Tracing the MAC addresses to their owner learns that all clients are using Windows 8 with Hyper-V and that they have an External Virtual Network bound to their Wireless/WiFi Network Adapter. Removing this virtual network stops sending those packets. Because multiple hosts are involved it may be necessary to remove this network on all of these hosts.

    As is visible in the following packet dump this is an unsolicited neighbor advertisements, resulting in an update in the neighbor table on our IPv6 router. These updates are probably the course of the high cpu load.

      Frame: Number = 188594, Captured Frame Length = 86, MediaType = ETHERNET

    + Ethernet: Etype = IPv6,DestinationAddress:[33-33-00-00-00-01],SourceAddress:[00-24-D7-76-C4-68]

    - Ipv6: Next Protocol = ICMPv6, Payload Length = 32

      + Versions: IPv6, Internet Protocol, DSCP 0

        PayloadLength: 32 (0x20)

        NextProtocol: ICMPv6, 58(0x3a)

        HopLimit: 255 (0xFF)

        SourceAddress: FE80:0:0:0:6F7:E4FF:FE4A:2267

        DestinationAddress: FF02:0:0:0:0:0:0:1

    - Icmpv6: Neighbor Advertisement, Target = FD00:4953:4E4C:20:6F7:E4FF:FE4A:2267

        MessageType: Neighbor Advertisement, 136(0x88)

        Code: 0 (0x0)

        Checksum: 3593 (0xE09)

      - NeighborAdvertisementFlag: 536870912 (0x20000000)

         R:   (0...............................) Not router

         S:   (.0..............................) Not solicited

         O:   (..1.............................) Override

         Rsv: (...00000000000000000000000000000)

        TargetAddress: FD00:4953:4E4C:20:6F7:E4FF:FE4A:2267

      - TargetLinkLayerAddress:

         Type: Target Link-Layer Address, 2(0x2)

         Length: 1, in unit of 8 octets

         Address: 00-24-D7-76-C4-68

    Because the target address is the same in all packets but the MAC address changes, it seems that there is something on the hosts that is forwarding/proxying these packets (and only those packets) back to the network it came from, but only after changing the targetlinklayeraddress in the advertisement. This process is described in RFC4389.

    One host with this problem on the network will not resolve in a flood. Only when more hosts with this issue are active the flood occurs. It seems that two or more hosts are required to magnify the packets send by the others.

    Hyper-V on Windows 8 makes use of a Network Bridge in software to enable the use of wireless network adapters. Is this the cause of all those packets?

    Some of the hosts also had Windows Phone 8 SDK installed. The installation of this SDK enables some networks in Hyper-V. Maybe some configuration change is made enabling the proxying of those packets back to the network they came from? After removing the SDK the hosts are still transmitting too much packets.

    The first occurrence of this flood was on the day after Microsoft’s update Tuesday. Is it possible that one of the updates may be the cause of this? There is one mayor update KB2770917 which includes updates to DHCPv6 and NDIS library’s. Again, removing this update does not solves the problem.

    Updating our switch/router to a higher level of firmware may lessen the impact, this is something I can try. But it shall not solve the problem, the switch is not the source of the packets.

    Hopefully somebody recognizes this issue. All ideas are welcome.

    Regards,

    Martijn

    Monday, November 26, 2012 11:44 AM

Answers

All replies

  • Hi Martijn

    We are experiencing the exakt same thing.

    It seems that some computers with Win8 and Hyper-v is flooding the wireless network.

    The only "fix" so far is to uninstall their virtual switch created by Hyper-v.

    The strange thing is that on my computer it works without any problems but for some it doesn´t.

    Maybe a driver issue?

    Could it be fixed by disabling the Ipv6 protocol on the virtual switch?

    I hope to find a working solution for this otherwise we have to disable the feature.

    Regards

    Fredrik

    Monday, December 3, 2012 9:54 AM
  • Hello Fredrik,

    it could be a driver issue, what network card are you using? Most of the notebooks on my network have an Intel Ultimate-N 6300
    (802.11 a/b/g/n).

    Regards,

    Martin

    Monday, December 3, 2012 3:28 PM
  • Hello,

    We have the same issue. When we enable wifi sharing in HyperV almost instantaneously the ping to the Accesspoints goes from 1ms to ~300ms and all wired ports broadcast about 2000-3000 kbits of bandwidth and more then 3000 packets per second. We monitor this with PRTG on our netgear and cisco switches.The strange thing is that it doesn't broadcast all the time. When we enable it on the laptop it can go bad (broadcasting) or not bad (no issues).

    When things go bad wireshark displays thousands of: ICMPv6 "86 Neighbor Solicitation" from one or more sources.

    Last week we even had a Mac Book and an iPhone as a souce of these packets according to wireshark.

    Regards,

    Stefan


    Monday, December 3, 2012 3:51 PM
  • Hello today we have a little breakthrough. Our Accesspoints are connected to a Cisco C2960S Switch. We have enabled storm control on the switches. This blocks only the multicast packets that were flooding the wireless and the wired network.

    We determined this using PRTG snmpv2 connection to the Cisco. Whenever the network flooding was occurring the packets per seconds went in the thousands. We now have put the following code on the Gbit interface: (The last line with the storm-control setting fixed flooding the network at least partially because the multicast packets cannot leave the Accesspoint to travel to other accesspoints or to wired devices.)

    Hope this helps others as well.

    interface GigabitEthernet1/0/7
     description AP06
     switchport trunk allowed vlan 1-99,101-4094
     switchport mode trunk
     speed 1000
     duplex full
     storm-control multicast level pps 200 50
    end


    Tuesday, December 4, 2012 9:29 AM
  • Yes that was our guess too...

    Most of our computers have Intel Centrino Advanced-N6200 AGN but also a few with Intel Ultimate-N 6300.

    Tried different drivers without any luck.

    Do your users have VPN clients installed?

    I enabled hyper-v and tried to replicate the issue but there was no problem until the user installed a few diffrent VPN clients...that´s when the flood began again....

    I really hope there is a solution for this, don´t want to block this great feature on the client computers...

    /Fredrik

    Thursday, December 13, 2012 3:46 PM
  • We recently endure such problem at work.

    We found that the problem was indeed the bridge created when launching the Windows Phone emulator in Visual Studio. It causes the device to flood ICMPv6 packet to multicast FF02::1 address.

    All that mess causes the Wi-FI network to be really slow !

    If anyone has a fix to that problem I would enjoy it =).

    Monday, December 17, 2012 4:27 PM
  • We recently endure such problem at work.

    We found that the problem was indeed the bridge created when launching the Windows Phone emulator in Visual Studio. It causes the device to flood ICMPv6 packet to multicast FF02::1 address.

    All that mess causes the Wi-FI network to be really slow !

    If anyone has a fix to that problem I would enjoy it =).

    Monday, December 17, 2012 4:27 PM
  • We recently endure such problem at work.

    We found that the problem was indeed the bridge created when launching the Windows Phone emulator in Visual Studio. It causes the device to flood ICMPv6 packet to multicast FF02::1 address.

    All that mess causes the Wi-FI network to be really slow !

    If anyone has a fix to that problem I would enjoy it =).

    Monday, December 17, 2012 4:27 PM
  • Well we could not relate this to VPN clients. Most Windows 8 PCs don't have third party VPN clients. We beleive it only occurs if there are more IPv6 devices in the network. We have had an MacBook and iPhone being the source (or at least its MAC Address) of the flood.

    I think the Wireless HyperV bridge act as an Ethernet loop without SpanningTree :-) and when some IPv6 devices start broad/mutlicasting the Wifibridge replicates/loops the packets arround. So the source may look like a MacBook, iPhone or something but the wifibridge is doing the bad stuff.

    The setting on the Cisco is keeping our users happy for the last few weeks, also we urged everyone to not use the Wifibridge, but instead use the ethernet bridge with wires.

    In my opinion we need a fix from Microsoft to fix this.

    Wednesday, December 19, 2012 1:58 PM
  • I also experience this problem. As said, it could be any ipv6 device that triggers the packet storm.
    I tried the following to resolve this issue, but to no avail;

    • Killing ipv6 on the hyper-v host with a registery entry
    • Blocking various sorts of ipv6 traffic with the windows firewall
    • Configuring ipv6 correctly on the cisco switches/routers and hosts.

    Only removing the bridge solved the issue. So this is definitely a bug.

    Regards, Koen


    Wednesday, January 9, 2013 10:19 AM
  • We have this same issue. We opened up a ticket with Microsoft.
    Friday, April 19, 2013 6:09 PM
  • I hope Microsoft can help you. For now Dell has not been able to help me.

    Monday, April 22, 2013 3:03 PM
  • We have done the same with the Microsoft tech. No fix yet...  They have been trying to play the blame game. We have a stock windows 8 laptop that is having the same issues.
    Thursday, May 2, 2013 6:17 PM
  • It also appears this issue is capable of hard crashing any Samsung GS4 devices on the network. 
    Thursday, July 18, 2013 8:45 PM
  • We are experiencing this as well on our network. I can't seem to block these advertisement packets... I have tried everything. Has anyone else had any luck?
    Monday, October 7, 2013 2:09 PM
  • We have had no luck with Microsoft whatsoever. We even disabled ipv6 through the registry and it still floods. Also seems to happen in 8.1. We still haven an open ticket with them.
    Monday, October 7, 2013 2:11 PM
  • Thanks for the update. We're working with Dell on it. Still no luck either.

    Monday, October 7, 2013 2:13 PM
  • What wifi cards do you have in your laptops? We have Dells too. Intel Centrino Ultimate-N 6300AGN
    Monday, October 7, 2013 2:15 PM
  • Anyone had any luck resolving this or word from Microsoft on a coming fix?  We are trying to move our dev vms over from VMWare but this is blocking our move as the network gets completely saturated when more than 2 people get on the network with these faulty bridges.
    Thursday, December 5, 2013 6:33 PM
  • We have gotten no resolution from Microsoft. We are on the verge of migrating everyone back to VMware.

    What kind of computers do you have? What wireless card?

    This floods my coworkers home network too on the Linksys router.
    Thursday, December 5, 2013 6:35 PM
  • We all have Dells, some laptops but mostly XPS desktops.  We run a Realtek PCIe GBE wired NIC and a Dell Wireless 1703 802.11b|g|n wireless NIC.  We are using the wireless card for the external switch we associate to the VMs since we were having some 802.1X problems using the wired card.  The flood goes out the wireless and eventually loops back in to try to flood our wired connections (since its a gig card it can't saturate that link with 4 people at least).  We have Aerohives that the wireless cards connect to.
    Thursday, December 5, 2013 7:28 PM
  • Interesting. I wonder if anyone is able to reproduce this issue with other Wireless cards. All of ours are the Dell branded Intel Centrinos for our laptops. We have no desktops. Unfortunately we have no way to replicate this issue with non-Dell laptops. 

    Thursday, December 5, 2013 7:36 PM
  • We have tried many network cards. At first we had one with a Broadcom chip that worked fine. About a month a go this card began to give problems too, probably after an windows update or a driver update pushed by windows update.

    Now I've checked with many cards and all come with this issue.

    We use cisco access points.

    Friday, December 6, 2013 9:14 AM
  • So we may have found a workaround.  Since we didn't have an issue connected to the secondary wireless network that was not on the same VLAN, we tried connecting to our main wireless network and unplugging the wired connections which stopped the IPv6 neighbor advertisements.  The current theory is that since the main wireless network and the wired cards were both on the same VLAN, it kept looping traffic since the virtual adapters didn't know how to stop the advertisements.  

    So I think Microsoft still needs to figure out their virtual adapter driver, but just using the wireless without wired fixed our issues.

    Edit:

    Except we forgot that the 802.1X doesn't work then on the wireless since its using a bridge instead of NAT....so 2 options for Microsoft to fix this, allow NAT instead of bridge or else fix the neighbor advertisements not getting stopped.

    • Edited by DanJohnson12 Monday, December 16, 2013 7:17 PM Update
    Monday, December 16, 2013 7:09 PM
  • Glad to hear you have found a workaround.

    Sadly this workaround is not working for me. The wifi is on a separate vlan already, the issue also occurs when there is no wired connection available.

    Tuesday, December 17, 2013 8:32 AM
  • Same behavior here, with Intel Centrino Ultimate-N 6300 AGN.
    Wednesday, January 15, 2014 11:25 AM
  • i recall seeing this issue in few other forums as well. its definitely gotta be the NIC itself. Suggest contacting the NIC vendor, ie Intel or Broadcom similar. have u tried  disabling IPv6 and using IPv4 only?
    Wednesday, January 15, 2014 11:55 AM
  • Hi,

    Did you ever manage to resolve this issue? We're experiencing the exact same thing over here...

    Wednesday, February 26, 2014 9:32 AM
  • Still an open case with Microsoft. We just reproduced it on a Surface Pro 2. Right out of the box.
    Wednesday, February 26, 2014 2:33 PM
  • Thanks for the quick reply Chris.

    I just don't get it. The OP reported this issue back in November 2012 and Chris had an open case with Microsoft since April 2013. How are you coping with this in your day to day work while waiting for a solution, seeing as there is no obvious workaround other than not using Wi-Fi with the external switch?


    Wednesday, February 26, 2014 2:40 PM
  • Half of our staff are on VMware still... some are on VirtualBox. Some are just told to use Gigabit instead of the convenient Wifi. We were also told directly from a Microsoft staffer that technet posts don't matter. So all these responses in this thread are moot apparently. This has been extremely frustrating on our end. That same Microsoft staffer hung up on my boss too. I'm going to roll the dice here and say this will never get resolved.

    Wednesday, February 26, 2014 2:46 PM
  • Fantastic. Doesn't seem like Intel is too keen on looking into this on their end either given the (lack of) response in their own support forums.

    I guess we'll just have to make sure not to combine Intel (Centrino?) WLAN adapters and Hyper-V switches.

    sigh.

    Wednesday, February 26, 2014 2:55 PM
  • The Surface Pro 2 that I have is a Marvell AVASTAR 350N. So it's not just Intel.
    • Proposed as answer by Mark D _NL_ Thursday, May 8, 2014 1:01 PM
    • Unproposed as answer by Mark D _NL_ Thursday, May 8, 2014 1:02 PM
    Wednesday, February 26, 2014 2:57 PM
  • We are still having this issue. Its definitely not based on the wifi card. Since we have tested this with different types.

    Wish Microsoft had a fix, since our executives are not happy about this issue. Currently we have a wire shark station, and block people as soon as they flood. Then we inform them to delete their wifi bridge. We can't disable this feature since our users need to be able to bridge at client sites.

    Wednesday, May 7, 2014 2:35 PM
  • Just an update post. Our ticket with Microsoft is still open and they are still looking at it.
    Thursday, May 8, 2014 1:36 PM
  • Still no solution. I've been working with Microsoft and they have been able to reproduce the issue.

    Please let them know when you also have this problem, this may speed up te resolution time.

    Friday, May 9, 2014 7:36 AM
  • What is the best way to let them know that we are having the same issue? Call support?
    Friday, May 9, 2014 12:28 PM
  • Contact me at the address in http://1drv.ms/1mJnBPe and we can share support information in private.
    Friday, May 9, 2014 12:40 PM
  • Today I found out that monitoring with Microsoft Network Monitor 3.4 does not show any ICMPv6 neighbor advertisements. However, when using WireShark they were visible!

    This means that Microsoft Network Monitor is not a suitable tool to troubleshoot this problem.


    Kind regards, Mark

    Monday, June 2, 2014 11:25 AM
  • I just wanted to let everyone know I heard back from Microsoft and they said they have determined the problematic code and are working on a way to fix it.  I will keep you posted.

    I work with Chris Erickson1212.  If you have any questions please let me know.


    Jeff Johnson

    Tuesday, June 24, 2014 8:26 PM
  • Thank you Jeff. I received the same message. I hope the hotfix will be avaible soon.

    Kind regards, Mark

    Friday, June 27, 2014 7:52 AM
  • Great news!

    Hotfix available at http://support.microsoft.com/kb/2982348.

    • Marked as answer by M. de Vries Wednesday, August 27, 2014 12:47 PM
    Wednesday, August 27, 2014 12:46 PM
  • Anyone successfully installed the hotfix? I'm getting "This update is not applicable to your computer" on my Windows 8.1u1 x64 Enterprise. 2919355 is installed.
    Wednesday, August 27, 2014 1:46 PM
  • i have 8.1u1 x64 enterprise but the kbarticle i downloaded is kb2982348 not the one you have listed in your comment

    Jeff Johnson

    Wednesday, August 27, 2014 4:05 PM
  • I Just wanted to let everyone know I tested the patch today with 4 laptops (3 usually caused it)  and it did flood for a second and than it went away.  We are running windows 8.1 U! on all the computers.  It looks like this is fixed.  :).  It sucks that it took almost 2 years.  I can now finish replacing VMWARE with hyper-v.

    Jeff Johnson

    Wednesday, August 27, 2014 10:13 PM