none
Firewall may drop packet when talking to old server if timestamp is randomized

    Question

  • I have encountered a problem with Windows Firewall and would like to report it to Microsoft. It requires some specific circumstances to trigger, but I actually ran into this several times. It forces me to use a less secure network configuration (because windows tcp timestamps leak system uptime and allow an observer to distinguish individual hosts behind nat).

    Client: Windows 7 with Windows Firewall enabled, rfc1323 timestamps enabled, ECN enabled
    Router: FreeBSD with firewall that performs timestamp randomization ("modulation")
    Server: Windows XP(?) running IIS6, rfc1323 timestamps enabled, ECN disabled(?)

    In this scenario, there is a 50% chance that the tcp connection to the server will stall after the initial handshake. That's because Windows Firewall will drop all packets on any connection to the server where the timestamp is above 2^31 (INT_MAX).

    Original packet, before randomization:
    C -> S [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 TSval=29507066 TSecr=0

    The packet exchange itself, as recorded on the router, can then look like this:
    C -> S [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 TSval=743741 TSecr=0
    S -> C [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 TSval=0 TSecr=0
    C -> S [ACK] Seq=1 Ack=1 Win=66560 Len=0 TSval=743744 TSecr=0
    C -> S GET / HTTP/1.0
    S -> C ... answer which gets acked ... ✓

    Or it may turn out like this:
    C -> S [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 TSval=2344120535 TSecr=0
    S -> C [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 TSval=0 TSecr=0
    C -> S [ACK] Seq=1 Ack=1 Win=66560 Len=0 TSval=2344120537 TSecr=0
    C -> S GET / HTTP/1.0
    S -> C ... all received packets are dropped by client firewall... ✗

    If client timestamps are disabled, obviously this isn't an issue. If client ECN is disabled, WF doesn't malfunction. If router timestamp modulation is disabled, this won't happen until the client uptime is 16 years and the real timestamp crosses over. And if the server runs an up-to-date tcp stack, this also doesn't happen.

    How to set this up:
    win7: netsh interface tcp set global ecncapability=enabled timestamps=enabled
    bsd: pf.conf: scrub on $wan all reassemble tcp
    server: https://www.shodan.io/search?query=Server%3A+Microsoft-IIS%2F6.0

    For reference, here's what talking to a well-behaved host looks like:
    C -> S [SYN, ECN, CWR] Seq=0 Win=8192 Len=0 TSval=4027614977 TSecr=0
    S -> C [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 TSval=2860014714 TSecr=4027614977
    C -> S [ACK] Seq=1 Ack=1 Win=66560 Len=0 TSval=4027614977 TSecr=2860014714

    When comparing old and the new captures, I see that the old server does not fill in a proper TSval until the handshake is complete. This clearly is not a problem since in most cases the communication can still proceed. However, for some values of serverside TSval, Windows Firewall will trip up. I suggest that a Microsoft dev take a peek and either find and fix the malfunctioning code, or add support for this quirk.







    Sunday, August 20, 2017 8:30 AM

All replies

  • Hi,

    Did you set NAT on your network, router or firework?

    When using Per-host paws mechanism, it is not guaranteed that timestamp is linearly incremented. Because of the two real machines using the same NAT address, their timestamp is not guaranteed to be synchronized.
    Capture network packets, including TCPIP ETL Trace ,at both client and server to determine what caused the timestamp exception occurred.

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, August 25, 2017 7:00 AM
  • NAT is used. Timestamp change is done on purpose:

    timestamp modulation
    Modern TCP stacks will send a timestamp on every TCP packet and echo the other endpoint's timestamp back to them. Many operating systems will merely start the timestamp at zero when first booted, and increment it several times a second. The uptime of the host can be deduced by reading the timestamp and multiplying by a constant. Also observing several different timestamps can be used to count hosts behind a NAT device. And spoofing TCP packets into a connection requires knowing or guessing valid timestamps. Timestamps merely need to be monotonically increasing and not derived off a guessable base time. reassemble tcp will cause scrub to modulate the TCP timestamps with a random number.

    I have adjusted the first post to maybe make it more clear what my problem is.



    Friday, August 25, 2017 7:10 AM
  • Hi,

    For further research, we need to capture the network traffic on both client and server. Also, the event trace logs of firewall on the client and tcp/ip are also needed. Here are some steps that helps you to use Network Monitor to capture the network trace.

    Step 1: Install network monitor (make sure you choose the file correspond to your server architecture, X86 or X64), install with default settings

    http://www.microsoft.com/downloads/details.aspx?FamilyID=983b941d-06cb-4658-b7f6-3088333d062f&displaylang=en

    Step 2: Right click on the "shortcut" of Network Monitor -> "run as administrator";

    Step 3: Use "Select Networks" on "start pages" to choose the related connections to monitor;

    Step 4: Click the "New Capture" to start a new capture and then click the "Capture"->"Start" button to capture;

     

    Step 5: Start tracing:

    Run cmd as administrator, netsh trace start capture=yes overwrite=yes maxsize=2048 tracefile=c:\minio_sockets.etl provider="Microsoft-Windows-Winsock-AFD" keywords=0x800000000000003f level=0x5 provider={EB004A05-9B1A-11D4-9123-0050047759BC} keywords=0x3ffff level=0x5 provider="Microsoft-Windows-TCPIP" keywords=0x80007fff000000ff level=0x5 provider={B40AEF77-892A-46F9-9109-438E399BB894} keywords=0xffffffffffffffff level=0xff provider="Microsoft-Windows-WFP" keywords=0xffffffffffffffff level=0xff provider={106B464A-8043-46B1-8CB8-E92A0CD7A560} keywords=0xffffffffffffffff level=0xff

     

    Step 6: please "repro symptom";

     

    Step 7: If you want to stop it, just click "Capture"->"stop", please click "File"->"Save as", save file with extension "cap”, then upload to workspace.

     

    Step 8: Stop tracing:

    netsh trace stop


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, August 25, 2017 9:51 AM
  • I already have pcap (wireshark/tcpdump) recordings from 2 years ago, on both the client and router, for both the positive and negative outcomes. I don't feel comfortable posting them in the public. Can provide privately. Does this thing have private messaging?

    I think I posted all the needed info related to the tcp headers. But if you need to do a live test, I can provide an ip:port that NATs to one of the problematic servers.

    Friday, August 25, 2017 10:10 AM
  • Hi,

    We cannot provide e-mail communication to help solve the problem, if you want to one-on-one service, I suggest you can contact Support For Business, which can deal with the problem by phone, e-mail or remote assistance. 

    http://support.microsoft.com/select/default.aspx?target=assistance

    Best regards,

    Vivian


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Monday, August 28, 2017 9:17 AM
  • Talking to Microsoft support requires a paid support license, which I don't have. I'm just a normal user, that's why I'm posting here as a last resort. I have no other way of reporting this.
    Monday, August 28, 2017 11:31 AM
  • Hi,

    It is a personal workspace that you can upload your log.  Please help us collect the log by captring the network traffic on both client and server. Also, the event trace logs of firewall on the client and tcp/ip are also needed.

    here


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Wednesday, August 30, 2017 9:15 AM
  • Hi,

    From the network package, I found that the client can't communicate with server because server didn't get the response from client ack (frame 8). Instead server keep sending HTTP: Response #7 and #9, and client seems continue to send ack (frame 8), which leads to communicate failure.

    However, I don’t find any packages dropped by firewall on the client. The issue still can’t be researched further unless you post the etl message both on the server and router. Because we can’t find how server dealt with the ack (frame 8), it is also possible that it is dropped by the router.

    Based on my knowledge, I did everything to help you on the issue. Since there are still many gaps during the communication, I suggest you can contact Support For Business. They have best resource and knowledge to help you and you can talk to them on time, which brings a more effective approach.

    http://support.microsoft.com/select/default.aspx?target=assistance

    ,Best regards,

    Vivian


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, August 31, 2017 7:57 AM
  • I think you are reading the trace wrong. The request clearly reached the server and the server is sending data back, and the replies are reaching the client host, but not the wget software at the endpoint. Windows Firewall is eating all of them, because is mis-interpreting them based on an integer overflow in its timestamp processing. If I disable Windows Firewall, the transfer succeeds even when the timestamps are randomized.

    I think you have all the info needed to research this; the router recording clearly shows that the fault is on the client side. I cannot give you an ETL trace from the router since it's not a windows device. And I do not have access to the server, nor do I know its exact specs; I am guessing that it's w2k3, but I do not have access to such a server instance for testing.

    I am not sure where on that page I can find business support. When clicking through Win7 enterprise, I am asked to provide payment or a support contract to proceed.

    Thursday, August 31, 2017 8:56 AM
  • Hi,

    The following picture shows the communication between server and client as I said above. For further research, I consider it is beyond the forum support here. And the business support really need you to have a payment for service.


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Thursday, August 31, 2017 9:29 AM
  • Well, good news! The issue can be reproduced by using a simple Windows XP machine with the IIS feature installed as the server. And I was also able to confirm that it happens when the client is win2008r2 or win2012r2. I set up a testing environment and gave out a link, unfortunately I got zero participation, so I cannot confirm if it still happens on win10. My guess is 'yes'.

    Thursday, August 31, 2017 5:05 PM
  • Hi,

    I have feedback the problem in our internal channel.

    Please keep your system up to date and I wish the bug may be fixed by installing the released updates.


    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    Friday, September 1, 2017 6:44 AM
  • I have done an extensive re-test. I used Win7, Win2008r2, Win2012r2 and Win10. For Win10 I used the latest preview image from https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/ .

    Here is a chart of the timestamp values used by various systems I tested. In all cases, there was a timestamp randomizing router inbetween. The first number is the timestamp of the sending party, the second is supposed to echo the previous timestamp of the other party.

    Legend: C = client timestamp, S = server timestamp, WX=Win10, LI=GNU/Linux

                XP-XP   XP-LI   W7-XP   W7-LI   WX-XP   WX-LI
    -> SYN 0 0 0 0 C 0 C 0 C 0 C 0
    <- SYN,ACK 0 0 S 0 0 0 S C 0 0 S C
    -> ACK C 0 C S C 0 C S C 0 C S
    -> GET C 0 C S C 0 C S C 0 C S
    <- HTTP S C S C S C S C S C S C
    ok ok fail ok fail ok

    Conclusions:
     * I can still reproduce this on Win10.
     * Simply turning windows firewall off no longer works on my PC. It did when I initially tested this 2 years ago. Clients running Symantec's firewall do not seem to have this issue. I will re-test.
     * WinXP client sends '0' instead of client timestamp in initial SYN.
     * WinXP server sends '0' instead of server timestamp in initial SYN,ACK.
     * WinXP client and server both don't care if the timestamp value is over 2^31.
     * Newer Windows as client will drop all packets with timestamp over 2^31 when talking to WinXP.

    Friday, September 1, 2017 11:24 AM
  • I can still reproduce this on latest Win7 and Win10.
    I'm forced to keep ecncapability and timestamps disabled to work around this.

    Tuesday, January 30, 2018 5:01 PM