none
Windows Server 2012 R2 runs out of ephemeral ports, though it shouldn't RRS feed

  • Question

  • We are regularly experiencing strange issues with networking on our dedicated server.

    It runs Windows Server 2012 R2 x64 on Xeon E5620 with 16 Gb RAM and Intel 82575EB network adapter.

    Please note that we've already tuned `HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters` key values `TcpTimedWaitDelay` and `MaxUserPort` to 30 and 65530 respectively.

    At a random point of time our websites stop responding, the reason being they cannot connect to a local database.
    It's approximately 2 weeks uptime when this issues start taking place.
    The system log starts getting TCPIP warnings 4227 and 4231.
    It states "A request to allocate an ephemeral port number from the global TCP port space has failed due to all such ports being in use.".
    If I run

    Get-Counter -Counter \TCPv4\*
    or
    Get-Counter -Counter \TCPv6\*
    or
    netstat -abn | find /c ":"


    I always get reasonable value of 500-1500 connections, which is not even close to 65K limit.
    Additionally, "localhost" stops resolving to ::1 locally, reverting to 127.0.0.1

    Only a forced machine restart can resolve the situation.

    Could it be a network adapter problem?


    • Edited by BeHappy-IT Monday, December 1, 2014 2:17 PM clarified OS version
    Monday, December 1, 2014 2:16 PM

All replies

  • Hi,

    This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection.

    Please try to use the command below to set the dynamic port range for TCP/IP,

    • netsh int ipv4 set dynamicport tcp start=10000 num=50000
    • netsh int ipv4 set dynamicport udp start=10000 num=50000
    • netsh int ipv6 set dynamicport tcp start=10000 num=50000
    • netsh int ipv6 set dynamicport udp start=10000 num=50000

    To verify if the settings are configured properly, please run the command below,

    • netsh int ipv4 show dynamicport tcp
    • netsh int ipv4 show dynamicport udp
    • netsh int ipv6 show dynamicport tcp
    • netsh int ipv6 show dynamicport udp

    Best Regards.



    Steven Lee

    TechNet Community Support

    Tuesday, December 2, 2014 8:39 AM
    Moderator
  • As I said, it has already been done long ago

    C:\Users\Administrator>netsh int ipv4 show dynamicport tcp
    
    Protocol tcp Dynamic Port Range
    ---------------------------------
    Start Port      : 1025
    Number of Ports : 64506
    
    
    C:\Users\Administrator>netsh int ipv6 show dynamicport tcp
    
    Protocol tcp Dynamic Port Range
    ---------------------------------
    Start Port      : 1025
    Number of Ports : 64506

    Tuesday, December 2, 2014 12:46 PM
  • I can second this issue, did you ever get a resolution? I starting seeing this after deploying an OPSMGR 2012 R2 agent.

    I have two servers with this, both are cluster nodes for different clusters.

    • One is a file server cluster node
    • One is a SQL server cluster node
    • 2012R2, patched monthly
    • VMs under 2012 R2 Hyper-V
    • Both have OPSMGR 2012R2 CU5 and DPM 2012R2 CU4 agents
    • Happens only to nodes with active clustered roles (that is, they are in use)

    Happens about 3-4 weeks of operations (your millage may vary). Reboot is the only fix.

    No idea on even how to start to track this one down...

    Log Name:      System
    Source:        Tcpip
    Date:          19/06/2015 1:43:44 AM
    Event ID:      4227
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      ...
    Description:
    TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Tcpip" />
        <EventID Qualifiers="32768">4227</EventID>
        <Level>3</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2015-06-18T15:43:44.215413800Z" />
        <EventRecordID>278517</EventRecordID>
        <Channel>System</Channel>
        <Computer>...</Computer>
        <Security />
      </System>
      <EventData>
        <Data>
        </Data>
        <Binary>00000000010000000000000083100080000000000000000000000000000000000000000000000000</Binary>
      </EventData>
    </Event>
    Log Name:      System
    Source:        Tcpip
    Date:          19/06/2015 1:45:02 AM
    Event ID:      4231
    Task Category: None
    Level:         Warning
    Keywords:      Classic
    User:          N/A
    Computer:      ...
    Description:
    A request to allocate an ephemeral port number from the global TCP port space has failed due to all such ports being in use.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Tcpip" />
        <EventID Qualifiers="32768">4231</EventID>
        <Level>3</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2015-06-18T15:45:02.253958200Z" />
        <EventRecordID>278522</EventRecordID>
        <Channel>System</Channel>
        <Computer>...</Computer>
        <Security />
      </System>
      <EventData>
        <Data>
        </Data>
        <Binary>00000000010000000000000087100080000000000000000000000000000000000000000000000000</Binary>
      </EventData>
    </Event>




    Friday, June 19, 2015 8:24 AM
  • Any chance there's been any progress on this one?

    It's a serious issue many of us have seen with Server 2012 R2 and one that to date remains unfixed for many with the above fixes.

    Monday, July 20, 2015 9:06 PM
  • Sorry mate, no fix from my end, we still suffer from the same issue.

    Cheers.

    Tuesday, July 21, 2015 9:50 AM
  • Guys, I am dying for an answer to this issue also. This is a persistent issue for us in our environment as well. Same symptoms across the same timeline (3-4 weeks from the looks of it), in a similar architecture.  Aside from rebooting the server periodically when it occurs, I would love to know of a more permanent solution.

    This is the kind of thing that has a pretty high impact when it occurs.

    Friday, October 23, 2015 3:37 PM
  • Would anyone have a found a solution for this already?

    Has anyone tried the solution on this page yet?

    http://serverfault.com/questions/648424/windows-server-2012-r2-runs-out-of-ephemeral-ports-though-it-shouldnt

    Friday, January 8, 2016 9:38 AM
  • yes ,unfortunately it didn't work...


    tomer


    Sunday, March 6, 2016 12:50 PM
  • Try disabling MPP.

    You can find more information here:

    https://support.microsoft.com/en-us/kb/974288

    Wednesday, July 13, 2016 6:20 PM
  • How did you solve the problem?
    • Edited by ivldenis1 Thursday, November 10, 2016 2:04 PM
    Thursday, November 10, 2016 2:04 PM
  • did anyone ever find an answer to this? we have two separate domain environments running into this issue. The servers reporting this issue all talk to SQL and WEB Servers.
    Thursday, July 6, 2017 2:25 PM
  • Hey, this is causing me headaches the last few days too...
    Friday, July 7, 2017 2:27 PM
  • I have a 2012 R2 server with SQL 2016 running into this same thing, but it takes ~24-30 hours for the symptoms to occur after a reboot.  Today I was unable to connect via RDP until I stopped a service that closed several connections.  Once I got connected, things seemed fine for a minute, but Tcpip 4227 came back and now I can't access my NAS from the server.

    Sr System Engineer | Vision One IT Consulting | www.v1corp.com

    Monday, July 10, 2017 6:42 PM
  • Same, running sam os with Eaglesoft sql, carbonite, and splashtop in a workgroup.
    Sunday, July 16, 2017 6:40 PM
  • Do you have any iSCSI disks connected to your server by any chance? Because I did and was suffering from same issue as described. Check all your iSCSI connection and remove the targets that do not exist (anymore).

    Take a look at this post and scroll down. Uninstall specified KB or manually download the KB that fixes it.

    https://social.technet.microsoft.com/Forums/windowsserver/en-US/8d29f351-6bc4-4ad8-9fcc-02d827e05536/ephemeral-port-exhaustion-event-id-4231?forum=winserver8gen

    I had this same issue on a 2012 R2 server with some connection iscsi disks and the specified KB update is clearly causing this issue. For me the issue was a removed iscsi target (and windows keeps trying to establish connection which causes the symptoms as described).

    Of course remove the old unused iscsi targets at your host.


    • Edited by LootiLootie Friday, August 11, 2017 10:27 AM
    Friday, August 11, 2017 10:27 AM
  • Hi Derek,

    i face exactly the same problem as you ...PLS did you find a solution?Kind Regards

    Tuesday, April 10, 2018 4:39 PM
  • Whoops, guess I never posted an update on my issue.  LootiLootie suggested iSCSI, and this ended up being my issue.  My SQL server was a Hyper-V guest that was using the Microsoft iSCSI initiator to backup to a Netgear ReadyNAS.  Someone (not me) decided to switch backup platforms, so they installed the new agent on the SQL VM and removed the LUN from the ReadyNAS.  They did NOT remove the iSCSI target details from the initiator.  Stopping the Microsoft iSCSI service confirmed that was the source of my problem (and I didn't have any other iSCSI disks on this VM, so it was safe to stop it).  I ended up starting the service back up and removing the orphaned iSCSI connections and all was well in the world again.  

    Sr System Engineer | Vision One IT Consulting | www.v1corp.com

    Tuesday, April 10, 2018 8:44 PM
  • I thought I would update this thread, as I have been researching this issue for some time.

    It appears that by default, and unlike Server 2008 and 2008 R2, Server 2012 and Server 2012 R2 do not attempt to re-use ports that are in a TIMED-WAIT state, and therefore are much more prone to port exhaustion.

    There is a hotfix to correct this behaviour and force re-use of ports available here:

    https://support.microsoft.com/en-us/help/3014399/various-network-and-computer-issues-occur-when-tcp-ephemeral-ports-are

    This issue is fixed in Server 2016, which correctly attempts to re-use half-open and TIMED-WAIT ports.


    • Edited by Alisterg Friday, October 12, 2018 3:27 PM
    Friday, October 12, 2018 3:23 PM