none
Intel I350 SR-IOV Suddenly Stopped Working While Hyper-V Still Not Reporting as Network as Degraded

    Question

  • After working for almost a year without issues, SR-IOV functionality suddenly quit working. The NIC is an Intel I350-T4 and the OS is Windows Server 2012 R2.

    Looking at the event logs in the parent, here's an example of the message indicating failure:

    Log Name:      Microsoft-Windows-Hyper-V-SynthNic-Admin
    Source:        Microsoft-Windows-Hyper-V-SynthNic
    Date:          3/21/2014 2:12:18 PM
    Event ID:      12585
    Task Category: None
    Level:         Warning
    Keywords:      
    User:          NT VIRTUAL MACHINE\5AEE01AC-A4BC-4C66-B83A-54F9A0FF87C7
    Computer:      [removed from public post]
    Description:
    'ADDS01' Network Adapter (5aee01ac-a4bc-4c66-b83a-54f9a0ff87c7--ac9784d3-e52c-4334-ac68-465a9a853454) failed to allocate a virtual function: The request is not supported. (0x80070032): IOV networking might be incompatible with other configured networking features. (Virtual Machine ID 5AEE01AC-A4BC-4C66-B83A-54F9A0FF87C7)

    Looking in one of the affected Windows VMs, the following log entry is reported:

    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="v1q" />
        <EventID Qualifiers="40964">27</EventID>
        <Level>3</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2014-03-21T18:12:23.582053900Z" />
        <EventRecordID>203556</EventRecordID>
        <Channel>System</Channel>
        <Computer>[removed from public post]</Computer>
        <Security />
      </System>
      <EventData>
        <Data>
        </Data>
        <Data>Intel(R) I350 Virtual Function</Data>
        <Binary>0000040002003000000000001B0004A00000000000000000000000000000000000000000000000001B0004A0</Binary>
      </EventData>
    </Event>

    I've looked at John Howard's detailed, "Everything you wanted to know about SR-IOV in Hyper-V" series and followed the steps in part 8 (http://blogs.technet.com/b/jhoward/archive/2012/03/21/everything-you-wanted-to-know-about-sr-iov-in-hyper-v-part-8.aspx) to see if that would lead me to the resolution. Unfortunately, most of the error conditions that he mentions are paired with a 'Degraded' status and status messages with suggested reasons.

    Unfortunately, the network connections in my case are listed as 'OK', and so I'm not sure where to go next to find out what changed and why SR-IOV is enabled on the NIC, virtual switch, and virtual NIC... and yet still not functional and not reporting as degraded.

    Any help would be greatly appreciated.

    Thanks!
    Paul

    Saturday, March 22, 2014 7:19 PM

Answers

  • I would try removing and readding the virtual NICs on the VMs that aren't working.  I have no idea why they would have stopped working.

    .:|:.:|:. tim

    • Marked as answer by Paul R Davis Monday, March 24, 2014 11:21 PM
    Monday, March 24, 2014 9:41 PM

All replies

  • Elton,

    Thank you so much for your response! :)

    All of the existing VMs are unable to use SR-IOV. I created a new VM and it IS showing SR-IOV as active.

    So, since SR-IOV used to work on all of the VMs and now it's not, but it is still working on a new VM, I guess my question now is: How do I get SR-IOV working on the existing VMs again (which it used to work just fine on)?

    I've tried restarting the existing VMs as well as the host server... and yet all the existing VMs are still not taking advantage of SR-IOV?

    Thanks so much!
    Paul


    • Edited by Paul R Davis Monday, March 24, 2014 1:44 PM Typos
    Monday, March 24, 2014 1:43 PM
  • I would try removing and readding the virtual NICs on the VMs that aren't working.  I have no idea why they would have stopped working.

    .:|:.:|:. tim

    • Marked as answer by Paul R Davis Monday, March 24, 2014 11:21 PM
    Monday, March 24, 2014 9:41 PM
  • Elton and Tim,

    I want to thank you both so much for your time and assistance with troubleshooting this issue. Thanks to your quick and accurate responses, I am happy to report that SR-IOV is once again working correctly on all the VMs hosted on the affected server.

    Once again, thank you so much!!!

    Paul

    Monday, March 24, 2014 11:21 PM
  • Same issue here with same setup. Hopefully this solution will work for me too, but question still arises as to what has gone wrong in the first place.

    Thanks for reporting this Paul!

    Wednesday, April 30, 2014 12:35 PM
  • Andy,

    In my case, I was finally able to track down what was causing the issue: http://blogs.msdn.com/b/virtual_pc_guy/archive/2014/03/21/my-daily-hyper-v-status-email-part-5-of-5.aspx

    If you check my comment at the end of that post (which was a series by Ben Armstrong about a daily Hyper-V status report script), you'll find that I discovered the problem started when I rolled out this script in my pre-production environment and the issue stopped happening when I disabled the scheduled, daily run of the script.

    I'm not sure why querying the status (which should be a read-only operation) of Hyper-V through PowerShell would cause SR-IOV to break, but that's exactly what was happening in my case.

    I hope that helps.

    Paul

    Wednesday, April 30, 2014 2:31 PM
  • Paul,

    Great feed back: thanks.  I don't script against the host, but I have recently installed Veeam v7 (including the Veeam One monitor services). This polls the host performance counters regularly via remote registry and other mechanisms, so I wonder if this might trigger a similar response on the SR-IOV?

    Thanks again


    Andy

    Thursday, May 01, 2014 7:34 AM