none
hyper-v cluster 2016, event id 25, Hyper-V-VmSwitch RRS feed

  • Question

  • I have a cluster, 2 nodes. There are repeating events id 25, every few second:

    The MAC address 00-15-5D-AE-25-19 has moved from port 53A6AB28-EBD3-4F72-B690-69406077B8A7 (Friendly Name: vSwitchFN_External) to port 6D5ABBEF-0A8E-4F5D-8CC2-0871846CEF82 (Friendly Name: BCK).

    The MAC address 00-15-5D-AE-25-18 has moved from port 53A6AB28-EBD3-4F72-B690-69406077B8A7 (Friendly Name: vSwitchFN_External) to port 9A8780F9-D931-4F85-B087-397A2F718C01 (Friendly Name: HB).

    The MAC address 00-15-5D-AE-25-18 has moved from port 9A8780F9-D931-4F85-B087-397A2F718C01 (Friendly Name: HB) to port 53A6AB28-EBD3-4F72-B690-69406077B8A7 (Friendly Name: vSwitchFN_External).

    The MAC address 00-15-5D-AE-25-19 has moved from port 6D5ABBEF-0A8E-4F5D-8CC2-0871846CEF82 (Friendly Name: BCK) to port 53A6AB28-EBD3-4F72-B690-69406077B8A7 (Friendly Name: vSwitchFN_External).

    Config:

    No NLB (MAC spoofing) on VMs.

    PS C:\Users\pero>  Get-NetLbfoTeam


    Name                   : mgmt
    Members                : {M2, M1}
    TeamNics               : mgmt-624
    TeamingMode            : Lacp
    LoadBalancingAlgorithm : Dynamic
    LacpTimer              : Fast
    Status                 : Up

    Name                   : vm
    Members                : {K2L1, K1L1}
    TeamNics               : vm
    TeamingMode            : Lacp
    LoadBalancingAlgorithm : Dynamic
    LacpTimer              : Fast
    Status                 : Up


    PS C:\Users\pero> Get-VMNetworkAdapter -ManagementOS

    Name          IsManagementOs VMName SwitchName MacAddress   Status IPAddresses
    ----          -------------- ------ ---------- ----------   ------ -----------
    BCK           True                  vSwitchFN  00155DAE2519 {Ok}
    HB            True                  vSwitchFN  00155DAE2518 {Ok}
    SETswitchSMB2 True                  SETswitch  00155DAE250C {Ok}
    SETswitchSMB1 True                  SETswitch  48DF371EFC2D {Ok}
    LM            True                  SETswitch  00155DAE250D {Ok}

    PS C:\Users\pero> Get-VMSwitch

    Name      SwitchType NetAdapterInterfaceDescription
    ----      ---------- ------------------------------
    vSwitchFN External   Microsoft Network Adapter Multiplexor Driver #2
    SETswitch External   Teamed-Interface


    PS C:\Users\pero> Get-VMNetworkAdapterVlan -ManagementOS

    VMName VMNetworkAdapterName Mode   VlanList
    ------ -------------------- ----   --------
           SETswitchSMB1        Access 1980
           LM                   Access 1981
           HB                   Access 1044
           BCK                  Access 887
           SETswitchSMB2        Access 1979


    PS C:\Users\pero> Get-NetAdapter

    Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
    ----                      --------------------                    ------- ------       ----------             ---------
    mgmt-624                  Microsoft Network Adapter Multiplexo...       6 Up           F4-03-43-5C-75-91         2 Gbps
    K1L2                      HPE Ethernet 10Gb 2-port 561T Adapter        32 Up           48-DF-37-1F-0F-09        10 Gbps
    K1L1                      HPE Ethernet 10Gb 2-port 561T Adap...#4      30 Up           48-DF-37-1F-0F-08        10 Gbps
    SMB2                      Hyper-V Virtual Ethernet Adapter #5          35 Up           00-15-5D-AE-25-0C        20 Gbps
    SMB1                      Hyper-V Virtual Ethernet Adapter #4          11 Up           48-DF-37-1E-FC-2D        20 Gbps
    LM1                       Hyper-V Virtual Ethernet Adapter #3          16 Up           00-15-5D-AE-25-0D        20 Gbps
    BCK1                      Hyper-V Virtual Ethernet Adapter #2          27 Up           00-15-5D-AE-25-19        10 Gbps
    HB1                       Hyper-V Virtual Ethernet Adapter             19 Up           00-15-5D-AE-25-18        10 Gbps
    M2                        Broadcom NetXtreme Gigabit Ethernet #3       20 Up           F4-03-43-5C-75-91         1 Gbps
    M1                        Broadcom NetXtreme Gigabit Ethernet          24 Up           F4-03-43-5C-75-90         1 Gbps
    vm                        Microsoft Network Adapter Multiple...#2      17 Up           48-DF-37-1E-FC-2C        20 Gbps
    K2L2                      HPE Ethernet 10Gb 2-port 561T Adap...#2      15 Up           48-DF-37-1E-FC-2D        10 Gbps
    K2L1                      HPE Ethernet 10Gb 2-port 561T Adap...#3      18 Up           48-DF-37-1E-FC-2C        10 Gbps

    The events intensively fill the log and sometimes network connectivity for these virtual adapters (BCK, HB) is temporary lost.

    What is problem?

    Wednesday, April 18, 2018 1:51 PM

Answers

  • I am also having this error occur on certain 2016 servers we support.  Have looked into the things others have tried on these threads, along with recreating the vSwitch, etc.  I tried changing the macaddress tonight of the Hyper-V Virtual Ethernet adapter, in ncpa.cpl, properties, configure, advanced, Network Address, setting a static MAC that is one higher than the others, as it was identical to the NIC team MAC.  So far so good, the errors have not resumed.

    Thursday, May 2, 2019 2:02 AM

All replies

  • Hi ,

    This is a quick note to let you know that I am currently performing research on this issue and will get back to you as soon as possible. I appreciate your patience.
    If you have any updates during this process, please feel free to let me know.

    Best Regards,

    Candy


    Please remember to mark the replies as an answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com   

    Thursday, April 19, 2018 9:57 AM
  • Any update on this problem?
    Thursday, July 12, 2018 12:49 PM
  • Any new info on this?  Ever since I started seeing all these Hyper-V VmSwitch 25 events all the time, server 2016 crashes at least every 12 hours
    Monday, November 5, 2018 11:42 PM
  • I am in the same boat, also seeing multiple event with ID 25.

    Example:

    The MAC address 00-15-5D-D4-23-00 has moved from port 958DFE2E-7078-4D1D-828F-8C59139F35EA (Friendly Name: vswitch10G_External) to port E3DF345B-7C4C-436B-9EF1-5432B21A2F07 (Friendly Name: vNICMGMT).

    Config:
    WS2016 with Hyper-V.

    TEAM with 2x10Gig Intel x710, LACP Dynamic mode. Cisco switches.

    PS C:\> Get-VMNetworkAdapterVlan -ManagementOS
    
    VMName VMNetworkAdapterName Mode   VlanList
    ------ -------------------- ----   --------
           LiveMigration        Access 7
           Heartbeat            Access 7
           vNICMGMT             Access 7
    
    
    PS C:\> Get-VMNetworkAdapter -ManagementOS
    
    Name          IsManagementOs VMName SwitchName MacAddress   Status IPAddresses
    ----          -------------- ------ ---------- ----------   ------ -----------
    Heartbeat     True                  vSWITCH1G  00155DD42302 {Ok}
    LiveMigration True                  vswitch10G 00155DD42301 {Ok}
    vNICMGMT      True                  vswitch10G 00155DD42300 {Ok}
    
    
    PS C:\> Get-VMSwitch
    
    Name       SwitchType NetAdapterInterfaceDescription
    ----       ---------- ------------------------------
    vSWITCH1G  External   Microsoft Network Adapter Multiplexor Driver #2
    vswitch10G External   Microsoft Network Adapter Multiplexor Driver
    


    PS C:\> get-netadapter
    
    Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
    ----                      --------------------                    ------- ------       ----------             ---------
    TEAM1G                    Microsoft Network Adapter Multiple...#2       9 Up           E4-43-4B-1A-C8-A9         2 Gbps
    vEthernet (Heartbeat)     Hyper-V Virtual Ethernet Adapter #3          10 Up           00-15-5D-D4-23-02         1 Gbps
    vEthernet (LiveMigration) Hyper-V Virtual Ethernet Adapter #2           7 Up           00-15-5D-D4-23-01        10 Gbps
    vEthernet (vNICMGMT)      Hyper-V Virtual Ethernet Adapter             12 Up           00-15-5D-D4-23-00        10 Gbps
    NIC2                      Intel(R) Ethernet 10G X710 rNDC               4 Up           E4-43-4B-1A-C8-8A        10 Gbps
    TEAM10G                   Microsoft Network Adapter Multiplexo...      18 Up           E4-43-4B-1A-C8-8A        20 Gbps
    NIC4                      Intel(R) Gigabit 4P X710/I350 rNDC #2        11 Up           E4-43-4B-1A-C8-A9         1 Gbps
    NIC3                      Intel(R) Gigabit 4P X710/I350 rNDC           15 Up           E4-43-4B-1A-C8-A8         1 Gbps
    NIC1                      Intel(R) Ethernet 10G 4P X710/I350 rNDC      13 Up           E4-43-4B-1A-C8-88        10 Gbps
    

    Any help or guidance as to where to look for answers would be appreciated.


    Marek ITPro

    Tuesday, November 27, 2018 11:56 AM
  • ...same problem here - any update or info would be highly appreciated

    Wednesday, January 2, 2019 4:52 PM
  • I am getting this issue as well, it is causing intermittent connection issues as well as false alerts on our monitoring software telling us a Hyper-V server is down.   I have 4 identical installs at 4 different sites and only one Hyper-V server is showing this issue.  The Team 1 Mac address is the one that shows up in the logs over and over.  This happens over and over about 20 times every 10 minutes.  

    The MAC address EC-EB-B8-97-AB-BF has moved from port 8A69D434-AC4B-4E50-B42B-9A18DDDC51FD (Friendly Name: Microsoft Network Adapter Multiplexor Driver - Virtual Switch_External) to port 3BBEFC3E-861E-41FD-A4D0-27CCA74B9D21 (Friendly Name: Microsoft Network Adapter Multiplexor Driver - Virtual Switch).

    PS C:\Users\administrator.PCS> Get-NetAdapter
    
    Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
    ----                      --------------------                    ------- ------       ----------             ---------
    vEthernet (Microsoft N... Hyper-V Virtual Ethernet Adapter              9 Up           EC-EB-B8-97-AB-BF         1 Gbps
    Embedded LOM 1 Port 4     HPE Ethernet 1Gb 4-port 331i Adapter #2       4 Up           EC-EB-B8-97-AB-BF         1 Gbps
    Embedded LOM 1 Port 3     HPE Ethernet 1Gb 4-port 331i Adapter          2 Up           EC-EB-B8-97-AB-BE         1 Gbps
    Embedded LOM 1 Port 2     HPE Ethernet 1Gb 4-port 331i Adapter #4       7 Up           EC-EB-B8-97-AB-BD         1 Gbps
    Embedded LOM 1 Port 1     HPE Ethernet 1Gb 4-port 331i Adapter #3      10 Up           EC-EB-B8-97-AB-BC         1 Gbps
    Team 1                    Microsoft Network Adapter Multiplexo...       6 Up           EC-EB-B8-97-AB-BF         4 Gbps
    

    I 2nd the any update or info would be highly apprecaited.

    Chris Deluca

    Tuesday, January 8, 2019 2:35 PM
  • Hi Chris,

    I have the same issue on Supermicro servers with onboard Intel X540-AT2 10Gb ethernet.  Driver version 4.1.76.0. Profile: Virtualization Server (Hyper-V). Windows 2016 (14393.2670) Datacenter Hyper-V 3-node Cluster with S2D and 2018-12 CU(KB4483229) installed.  SET teaming was configured at one time but later removed because of frequent disconnects. Another standalone 2016 Hyper-V server with the same hardware and no SET teaming does not have any ID 25 events.

    All cluster nodes get about 1500 ID 25 events per hour.  In my case the event flood stops immediately after updating the VlandID of the problem ManagementOS VMNetworkAdapter.  It is a temporary fix.  The event 25 flood usually comes back a few days or weeks later.

    Get-VMNetworkAdapterVlan -ManagementOS

    Set-VMNetworkAdapterVlan -VMNetworkAdapterName vEthernet* -ManagementOS -Access -VlanID ##

    Hope that helps.

    -Jeremy


    • Edited by jeremy-crw Tuesday, January 8, 2019 8:19 PM
    Tuesday, January 8, 2019 8:16 PM
  • Also having the same issue on 1 of 2 severs. Both servers are specced out the same, same OS (Server 2016), but only one of the servers is having the issue.

    I believe the server with the issue at one point had some of it's NICs teamed - however, that is no longer the case.

    Moved all the running VMs off the server with the issue. Still 3 test VMs on the server, but they aren't running.

    Deleted the HyperV Switch and the issue seemed to go away (duh) - but after re-creating the HyperV Switch the warnings start coming back - even though all the VMs on the system are turned off.

    Have upgraded all NIC (Intel X550/I350) drivers on the system to whatever the latest is.

    In the process up installing 2019-01 Cumulative Update to see if it's any help.

    Whatever the issue is, it doesn't SEEM to impart performance - at least not to the point that's it is noticeable.

    Friday, January 11, 2019 11:32 PM
  • Having the same errors logged with Broadcom 57800 10G in Dell 730xd. Any update?

    Only 1 of 9 servers logging the error and that server loses network connectivity. All interfaces show up but no traffic passes. No errors or information found on the switch side.

    Discovered that disabling the domain firewall allows traffic to flow. Re-enabling the firewall, the traffic continues to flow. Compared the firewall settings to the other working servers, no difference.

    Thursday, February 14, 2019 8:21 PM
  • I am getting this issue as well, it is causing intermittent connection issues as well as false alerts on our monitoring software telling us a Hyper-V server is down.   I have 4 identical installs at 4 different sites and only one Hyper-V server is showing this issue.  The Team 1 Mac address is the one that shows up in the logs over and over.  This happens over and over about 20 times every 10 minutes.  

    The MAC address EC-EB-B8-97-AB-BF has moved from port 8A69D434-AC4B-4E50-B42B-9A18DDDC51FD (Friendly Name: Microsoft Network Adapter Multiplexor Driver - Virtual Switch_External) to port 3BBEFC3E-861E-41FD-A4D0-27CCA74B9D21 (Friendly Name: Microsoft Network Adapter Multiplexor Driver - Virtual Switch).

    PS C:\Users\administrator.PCS> Get-NetAdapter
    
    Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
    ----                      --------------------                    ------- ------       ----------             ---------
    vEthernet (Microsoft N... Hyper-V Virtual Ethernet Adapter              9 Up           EC-EB-B8-97-AB-BF         1 Gbps
    Embedded LOM 1 Port 4     HPE Ethernet 1Gb 4-port 331i Adapter #2       4 Up           EC-EB-B8-97-AB-BF         1 Gbps
    Embedded LOM 1 Port 3     HPE Ethernet 1Gb 4-port 331i Adapter          2 Up           EC-EB-B8-97-AB-BE         1 Gbps
    Embedded LOM 1 Port 2     HPE Ethernet 1Gb 4-port 331i Adapter #4       7 Up           EC-EB-B8-97-AB-BD         1 Gbps
    Embedded LOM 1 Port 1     HPE Ethernet 1Gb 4-port 331i Adapter #3      10 Up           EC-EB-B8-97-AB-BC         1 Gbps
    Team 1                    Microsoft Network Adapter Multiplexo...       6 Up           EC-EB-B8-97-AB-BF         4 Gbps

    We're having an identical issue here.  We've recently rolled out around 20 servers all running Windows Server 2019 Datacenter and are all Hyper-V hosts with 4 gigabit NICs in a switch independent team (all 4 NICs in the team; none in another team or standalone), the team bound to the Hyper-V vSwitch, and -AllowManagementOS set to $true so the host can use the vSwitch.  We see the exact same problem as others in this thread; repeat Event ID 25's (sometimes lots in the span of an hour and every few hours, sometimes none for a day or 2), and momentary disconnects from the host when this happens.

    We'll momentarily lose a connection to the host server (can't ping, can't WMI query from our monitoring system, and can't Remote Desktop), but meanwhile, we never once drop a ping to the guest VMs residing on the server.  So the actual users at these locations never notice a problem as they're using the guest VM servers, meanwhile our monitoring software watching the host OS is going nuts with servers "down" all over the place and then coming back up and going down again a little while later, seemingly at random.



    • Edited by link470 Tuesday, April 9, 2019 8:22 PM
    Tuesday, April 9, 2019 8:20 PM
  • I am also having this error occur on certain 2016 servers we support.  Have looked into the things others have tried on these threads, along with recreating the vSwitch, etc.  I tried changing the macaddress tonight of the Hyper-V Virtual Ethernet adapter, in ncpa.cpl, properties, configure, advanced, Network Address, setting a static MAC that is one higher than the others, as it was identical to the NIC team MAC.  So far so good, the errors have not resumed.

    Thursday, May 2, 2019 2:02 AM
  • I tried changing the macaddress tonight of the Hyper-V Virtual Ethernet adapter, in ncpa.cpl, properties, configure, advanced, Network Address, setting a static MAC that is one higher than the others, as it was identical to the NIC team MAC.  So far so good, the errors have not resumed.

    Interesting; part of me was curious if it had something to do with that MAC address showing as the same.  Powershell's output for Get-NetAdapter shows 6 adapters when using a 4 NIC team; 4 are the actual NICs, one is the Team (Microsoft Multiplexor driver), and the last one is the Hyper-V virtual adapter for the host, and typically, three of the MAC addresses match (one of the 4 NICs, the Hyper-V virtual adapter, and the multiplexor Team adapter).

    I'm very interested AlexDatsko, were you experiencing the same issues we were with regards to dropped pings and RDP/other access to the host when this occurs?  Then you'd gain access for a bit, and it would cut out, and the timing would match the appearance of the Event 25s in the logs? 

    Just to be double clear, if this was indeed the symptoms, what you did by bumping up the MAC on the virtual adapter by one in the last character fixed the issue?

    I should also note to add additional info: all of our servers are HP G7 and Gen8 servers that are having this issue.  Both are currently using Microsoft supplied drivers with Windows Server 2019, as the HP QLogic network adapter drivers for the G7 servers are older than the Microsoft supplied ones and don't install, and the Gen8 HP drivers just were never installed (that was going to be my next troubleshooting step, but knowing that if that worked for the Gen8s, that's a bummer for all the G7s that will most likely not see another network adapter driver update past the one Microsoft provides).

    To mitigate the issue for now, I ended up just splitting port 4 off of the team on each virtual host server.  So now we have a NIC team of 3 on each server, set -AllowManagementOS to $false, and port 4 is the management/OS port now.  This immediately stopped the problems we were having with intermittent connectivity issues to the host, but of course, is a solution that shouldn't have to be needed.


    • Edited by link470 Thursday, May 2, 2019 3:58 AM
    • Proposed as answer by Martin H. Stark Thursday, May 23, 2019 12:06 PM
    • Unproposed as answer by Martin H. Stark Thursday, May 23, 2019 12:07 PM
    Thursday, May 2, 2019 3:56 AM
  • No solution but a hint: I could stop this event-25-spam after deactivating the specific "Hyper-V Virtual Ethernet Adapter #2" (configured with dynamic ip setting <dhcp>) on my machine. This vm-adapter has obviously the same IP address as the physical adapter with fix ip settings "Intel(R) Ethernet Connection I217-LM". So me and you imho have to investigate in more skillfully network settings at all.
    Thursday, May 23, 2019 12:15 PM
  • Hi everyone,

    I think I actually found the problem of this... It seems to be related to HP G8/G9 Server with Windows Server 2016 running on it (not sure about that but I only got this on these specific hardware).

    The problem actually comes from VMQ. Run following command on your server:

    Get-NetAdapterVmq

    You should see that the vSwitch and the NIC participating in the teaming are set to "True", which means VMQ is enabled. The solution consist of disabling VMQ on these interfaces. VMQ is not really needed, as it comes into consideration only if you have NIC with 10Gbps or more, but even with that you actually got more troubles with it... To disable VMQ on the interface use following command:

    Set-NetAdapterVmq -Name "NIC_NAME" -Enabled $false
    (This will automatically disable it on the vSwitch, you just need to apply it on the physical NICs)

    Notice if you do this on a Hyper-V Node, a short connectivity interruption will occur (5-10 seconds). So maybe don't do this during work hours :) This change does not need a reboot or other stuff and after this the event ID 25 should disappear!

    HTH,
    Sebastien


    Sunday, June 30, 2019 10:55 PM
  • We had the same problem here with a 2 node 2019 S2D Cluster and Intel X722 and later also X540 NICs. First we configured a SET Team with -"AllowManagementOS set to $true" and an own VLAN for the management network. After some days one node is crashing with a black screen and the unmanaged switch gets completely locked up and did not allow any connection to any device on it. This did not happen with a managed HP switch but the node still crashes on that switch. We also noticed that about every 10th ping was dropped with the X722 NICs (this did not happen with the X540 NICs).

    To avoid problems with the SET team we separated the ports and use one port for management with "AllowManagementOS set to $true" to put a management VM to the same VLAN. The problem still persists and we had lots of ID 25 messages in the event log. Looks like the MAC address from the physical port of the management NIC is flapping to the virtual management NIC and vice versa. This crashes the nodes after some hours. We tried the VMQ tip but it did not help.

    The solution was to set "AllowManagementOS  to $false" on all NICs and use the management NIC only as a physical NIC without creating a virtual switch with it. 




    • Edited by AlexNimda Saturday, August 24, 2019 10:34 PM
    Saturday, August 24, 2019 10:08 PM