none
Slow network performance between ISA and internal network

    Discussione generale

  • Config: Win2k3 SP2, ISA 2004 EE SP3, 2 Array Members, 3-Leg Perimiter (Networks: External, Internal, Perimeter, Array Communication), NLB enabled on all but intra Array Communication. Each ISA is connected to a separate managed switch (connected to each other).

    Issue: Transferring files such as web logs and web publishing (reverse proxy) by way of either ISA box to the Perimeter network works fine but there is considerable lag and slowness with any activity over the Internal network.

    I discovered this while debugging performance issues with a published web server on the Internal network and found this to be the case for all activity over the Internal NIC of either ISA server. The Internal network is unique to the others as it was using a Broadcom BCM5708C, the others resided on a quad port Intel NIC.

    So initially, I focused on firmware/drivers for the network adapter, but that had no impact. I then looked at wiring; the NIC wire test utilities didn’t report any issues. Then to switch and port configurations it all looked right, transfer between two other servers on the internal VLAN was fine. Forcing speeds and settings had an interesting result on one server, where 100Mbps preformed twice as fast as a Gig!?

    I was working with the theory that the Broadcom NIC was somehow different that the Intel with its TOE features, that led me to the impact of Scalable Networking Pack for win2k3 and NLB but that proved to be fruitless, nothing that I tried in these articles helped. http://support.microsoft.com/kb/948496/ and http://blogs.technet.com/b/exchange/archive/2007/07/18/3403490.aspx

    To prove that there was nothing wrong with the wiring or switch, I shut down ISA services and removed the NLB configuration and network performance was excellent.

    My last resort option was to move the Internal network to the Intel NIC, but unfortunately, the problem still presented. And remember, over my Perimeter network (via the Intel) I still have no network performance problems.

    Based on all my diagnostics thus far and the fact that this is a problem on both servers and on both NIC’s, I’m thinking that there’s got to be some sort of setting/config/bug in ISA that is causing this trouble and possibly related to NLB/BDA. I don’t have Diffserv enabled for any networks and I’m unaware of any settings specific to the Internal network that would cause this.

    This should be simple, I’m transferring files from the ISA server to another server on the same LAN and subnet, no routing involved. Does anyone know of any built in differences between “Internal” and “Perimeter”?

    Then again, maybe I overlooked something major... please let me know, thanks.

    domenica 12 febbraio 2012 23:38

Tutte le risposte

  • Hi,

    Thats strange, looks for me issue at NLB and switch side issue. Port setting, NLB mode/settings. What if you use in your test client a static route to use the Private Lan ip as DFGW from 1 node in the NLB,instead of the VIP. What happens then?


    Best regards,
    Mark Scholman.
    Infrastructure Engineer
    Follow me on Twitter
    My Blog:TechMark's Blog

    Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.

    mercoledì 22 febbraio 2012 19:38
  • What operating mode is NLB using?

    Jason Jones | Forefront MVP | Silversands Ltd | My Blogs: http://blog.msedge.org.uk and http://blog.msfirewall.org.uk

    mercoledì 22 febbraio 2012 21:39
    Moderatore
  • Hi Mark and Jason, thanks for the reply.

    Apparently I didn't have alerts enabled so I didn't know about these replies.

    I agree that it probably does have something to do with NLB because when I removed the NLB config from the server there were no problems however I'm only seeing this NLB issue on one network and not the other, they both have the same NLB setup (which is managed by ISA server). The NLB cluster mode is Unicast.

    Mark, with regard to your idea of a static route, I don't know what this test would accomplish because the client is not using the default gateway to get to the ISA server. If I open a file share on the private LAN from the ISA server, I can see clearly in a netstat that the connection is going from the ISA dedicated IP and not the VIP. The same is true when I run the netstat from the file server.

    On ISA
    TCP    10.10.2.5:38425        10.10.2.180:445        ESTABLISHED

    On File Server
    TCP    10.10.2.180:445        10.10.2.5:38425        ESTABLISHED
    Where 10.10.2.5 is the ISA dedicated IP and 10.10.2.180 is a LAN server, the VIP is 10.10.2.1 for which there are no connections.
    lunedì 12 marzo 2012 14:29
  • was it working fine before and now it doest not work?

    what is the type of switch your NLB internal NICs are connecting to, layer2 or layer3?

    windowss\ISA integrated NLB works with layer 2 sitch, if you have a layer3 switch you need to configure a VLAN on it that is configured as layer2. 


    Thanks and Regards Suraj Singh My blog: http://blogs.technet.com/b/sooraj-sec/

    lunedì 12 marzo 2012 17:59
  • To be honest, I can’t say with certainty when this problem began but it may have begun after installing Win2k3 SP3 and in theory is connected to Scalable Networking Pack which has known to have problems with NLB but disabling SNP didn’t help.

    We’re using Layer 3 switches and the External, Internal and Perimeter networks are on separate VLan’s all configured the same and I don’t have this problem with the External or Perimeter network, only the Internal network. But what do you mean ‘configured as layer2’?
    lunedì 12 marzo 2012 18:19
  • have a look at following

    from http://technet.microsoft.com/en-us/library/cc783135(v=ws.10).aspx

    . How Do I Configure NLB with Layer 2 Switches?

    A. If you are connecting NLB hosts to a switch rather than a
    hub, you need to make sure that the switch does not associate the cluster MAC
    address with a particular switch port. Knowledge Base article Configuration Options for
    WLBS Hosts Connected to a Layer 2 Switches

    (http://go.microsoft.com/fwlink/?LinkId=18367) explains how to configure NLB
    with Layer 2 switches.


    Q. How Do I Configure NLB with Layer 3 Switches?


    A. Layer 3 switches need to be specially configured to work
    with NLB. A VLAN must be established for the hosts in the cluster, and this VLAN
    must be configured to operate in Layer 2 mode. All Layer 3 switches may not
    support this capability, and when they do, the mechanism to setup the Layer-2
    VLAN is specific to the particular make and model. Consult the documentation for
    the switch before attempting to configure such a system.


    Thanks and Regards Suraj Singh My blog: http://blogs.technet.com/b/sooraj-sec/

    lunedì 12 marzo 2012 20:43
  • Thanks for the suggestions.

    I confirmed that the switches are not tying the cluster MAC to a particular port and in general NLB works, it's just slow. i.e. I can load balance and drain/stop etc.

    The Dell PC 5448 switches are Layer 3 capable but the VLANs function at Layer 2 and they do support Unicast NLB. I use these same switches for a web NLB cluster and there is no problem there and as noted before, the Perimiter network does not have this performance issue which also has NLB enabled.

    martedì 13 marzo 2012 03:25
  • As per your statement

    "To prove that there was nothing wrong with the wiring or switch, I shut down ISA services and removed the NLB configuration and network performance was excellent."

    how did you test the performance? file access?

    when you say slow, are you comparing it with other working networks as baseline?

    are EnableRSS and ENableTCPA keys set to 0?

    i guess you are using ISA for reverse proxy role as well. do you have by any chance a internal website on port 80 published through ISA? its much easier to read http traffic in network monitor and see the delays (to see where is delay). You can try to open one of these http websites from ISA server and take a network monitor trace on the internal NIC and filter this http traffic and see where is this delay. if you see delay in response from ISA. then ISA data packager is going to answer your questions but normaly that can happen through MS support i.e. you might have to open a support case , as i wont suggest uploading ISA datapackager output file here as it can be accessed by any one .


    Thanks and Regards Suraj Singh My blog: http://blogs.technet.com/b/sooraj-sec/

    martedì 13 marzo 2012 13:41
  • any updates?

    Thanks and Regards Suraj Singh My blog: http://blogs.technet.com/b/sooraj-sec/

    domenica 18 marzo 2012 17:05
  • Sorry, I got tied up with a major project... back onto this issue now.

    "To prove that there was nothing wrong with the wiring or switch, I shut down ISA services and removed the NLB configuration and network performance was excellent."

    how did you test the performance? file access?

    Yes, transfering files from the ISA server to another LAN server over the Internal network. ISA -> switch -> FileServerA. When NLB was enabled, it was slow (26MB in 6 minutes), when it was disabled, it was fine.

    when you say slow, are you comparing it with other working networks as baseline?

    Yes, I was able to test between ServerB -> switch -> FileServerA and that same file copied instantly.

    are EnableRSS and ENableTCPA keys set to 0?

    Yes, both set to 0 and server was reset since that was applied.

    i guess you are using ISA for reverse proxy role as well. do you have by any chance a internal website on port 80 published through ISA? its much easier to read http traffic in network monitor and see the delays (to see where is delay). You can try to open one of these http websites from ISA server and take a network monitor trace on the internal NIC and filter this http traffic and see where is this delay. if you see delay in response from ISA. then ISA data packager is going to answer your questions but normaly that can happen through MS support i.e. you might have to open a support case , as i wont suggest uploading ISA datapackager output file here as it can be accessed by any one .

    Yes again, we are using reverse proxy and that's exactly how I discovered this problem, I was publishing a server that lies on the Internal network and it was performing slow, the servers that I publish which lie on the External network work fine. So I then published that same server over the External network and it resolved the page load performance.

    I also tried loading the website from within the ISA RDP session and saw the same performance problem over one network and not the other, consistent with other testing. So it really has nothing to do with the web publishing rule.

    I've ruled out the NIC, Drivers, Firmware, Wire, Switch, LAN servers and Publishing Rules

    NLB on the other two ISA networks work, but not on this 'network'.

    giovedì 22 marzo 2012 05:45
  • I've looked into this some more but haven't come up with any solutions, anyone have any ideas?

    Thanks.

    martedì 3 aprile 2012 04:25