locked
Multicast NACK/Retries RRS feed

  • Question

  • Hi all,

    I have another thread open regarding a specific returnc code, but here is anothe issue I am noticing on another network:
    When multicasting to just one machine (obviously the master multicast client), there are many packet retries due to NACK responses form this master.
    If I check performance logs I can see a ration of at least 1:5 (sends:resends)!! This is obviously slowing down multicasting.

    This happens on multiple clients (individually) with different NICs (they are only out of box supported by intel at the moment).
    These switches are set up for multicasting based on our previous Ghost multicasting (IGMP configured).
    No collisions/errors on the switch.
    There are no errors reported in the Admin/Ops event logs.


    I set up two different servers on different areas and get the same result where ever I image.
    I tried changing the multicast adress ranges, but no success.


    Any advice? Perhaps I am missing a step in the WDS role setup?

    Friday, December 11, 2009 4:28 PM

Answers

  • Modifying apBlockSize appears to fix the issue.

     

    The size is set as default to something like 8582. This is far to big for us, and I would have thought for others too?

    We changed the block size to 1472.

    This compliments our client MTU size and router block size = That is 1472 data + 28 headers = 1500.

    Anything above this size creates fragmentation in the data, hence the issues.

     

    We now get no NACKs/Retries whatsoever and speeds are acceptable:

    • -6GB image in about 12 mins.
    • -Average 80Mb/s
    • -99 % NIC Utilisation on clients.

     

    I hope this information helps others.

    • Marked as answer by MrBeatnik Monday, March 22, 2010 10:19 AM
    Monday, March 22, 2010 10:19 AM

All replies

  • *bump*
    No takers? Nobody experiencing NACKS/RETRIES at all out there?
    Thursday, January 7, 2010 5:03 PM
  • Do you have any clients that are in a sleep state on the local network?
    Thursday, January 7, 2010 11:33 PM
  • None at all.
    Just to be sure of results we created a closed network:

    1 DS Server running AD/DNS/DHCP (Windows 2003)
    1 WDS Server running Windows 2008 R2
    1 Client Machine to PXE boot to WDS.

    If we set this up with one switch, all is fine - speeds are great (all 3 devices on one switch).
    If we have two switches uplinked, then we get the problem (client on downlink).
    It doesn't matter which switch we use and uplink.

    We tried this with brand new cisco switches right out of the box.
    We have also tried settings:
    - Both IGMP and CGMP enabled.
    - Only IGMP or CGMP enabled.
    - No xGMP enabled (!)
    - Installed the latest IOS ensuring the IP tools are available.

    If they are connected to the same switch, NIC utilization on the client is around 97% and flying.
    If connected on different switches, the utilization is about 15% and awful.

    I just can't figure it out, neither can our newtork guys. It doesn't make any sense that the uplink is slowing it down..
    Friday, January 8, 2010 9:42 AM
  • Do you have flow control turned on on your uplink ports? You might try turning that off.
    Tuesday, January 12, 2010 12:02 AM
  • Do you have flow control turned on on your uplink ports? You might try turning that off.

    No flow control used.
    I can't understand where the issue is coming from!!
    Still performing various tests over here, but no joy as yet. Very strange.

    Would appreciate any more feedback from anyone.

    Thanks
    Wednesday, January 13, 2010 11:59 AM
  • Some intersting results here, I was wondering if anyone might be able to advise further with this info.

    Problem is occuring:
    Server is sitting on 1Gb set at auto.
    Client is sitting on 100Mb set at auto. Client shows about ~37% NIC usage.


    Setting server explicitly to 100Mb/Full results in almost no NACKs/Retries (although still a few).
    Client gets 99% NIC usage.

    Setting server explicitly to 10Mb/Full results in NO NACKS/Retries (obviously slow multicasting).
    Client shows 9% NIC usage (sounds right from the 100 to 10 Mb change).


    An ideas why the server being at 1Gb causes issues?
    We would ideally want to stay at 1Gb due to different switch speeds at different sites - some clients may be at 1Gb, and fibre/edge switches would take advantage of these speeds anyway.

    Thanks
    Sunday, January 17, 2010 10:18 PM
  • Modifying apBlockSize appears to fix the issue.

     

    The size is set as default to something like 8582. This is far to big for us, and I would have thought for others too?

    We changed the block size to 1472.

    This compliments our client MTU size and router block size = That is 1472 data + 28 headers = 1500.

    Anything above this size creates fragmentation in the data, hence the issues.

     

    We now get no NACKs/Retries whatsoever and speeds are acceptable:

    • -6GB image in about 12 mins.
    • -Average 80Mb/s
    • -99 % NIC Utilisation on clients.

     

    I hope this information helps others.

    • Marked as answer by MrBeatnik Monday, March 22, 2010 10:19 AM
    Monday, March 22, 2010 10:19 AM
  • I'm happy that you have resolved your issue.

    However, I'm somewhat surprised that making the block size smaller resolved the issue. Did the fragemented packets cause high CPU usage on your routers or switches?

    On the WDS side, it's generally significantly faster to send out larger, fragmented packets as we have to perform hashing on a per packet basis. By decreasing the block size, you're increasing the number of hashes we must perform by 6x.

    Monday, March 22, 2010 5:27 PM
  • However, I'm somewhat surprised that making the block size smaller resolved the issue. Did the fragemented packets cause high CPU usage on your routers or switches?

    Yes - Our core switches CPU were maxed out and sending out warnings to our network team.

    Does this suggest anything in particular to you? Sounds like you may have had experience down this route before?

     

    I'll pass on the information about hashing to our network team to see if this gives any ideas to them at all. I'm unsure if this could increase imaging time though since the NIC utilization on the client is around 95% and we get an average of 80Mbs+ on a 100Mbs network (which realistically is near the top speed)?

    Monday, March 22, 2010 10:08 PM
  • The hashing here is being performed by the WDS server process, and faster in this contex means less CPU usages on your WDS server. On older servers (single core, not much memory, etc) it's possible that we can max out the CPU performing hashing before we saturate the network, which will limit the multicast transmission speed.

    However, if you're already saturating your client's NICs, you're not hitting this issue so I wouldn't be concerend about it.

    Tuesday, March 23, 2010 8:09 PM
  • Hi MrBeatnik,

    Thankyou for the information you have provided in my thread http://social.technet.microsoft.com/Forums/en-GB/winserversetup/thread/69b43c14-e4ed-43de-b2da-60e372b3565f#9da253da-c2ac-4428-8329-9041d7784494 

    Where I too was having similar problems.

    I have also set the ApBlockSize to be 1482  and can confirm that after doing this I was getting multicast deployment speeds with the server connected at 1Gbps and the client 100Mbps of around 80Mbps.

    I have however since set the value to be 1024 as advised here (It is recommended to set this value in 4-KB increments-for example, 1 KB (1024 bytes), 4 KB (4096), 8 KB (8192), 16 KB (16384)).

    I can now confirm that I am now getting multicast deployment speeds of around 95-100Mbps (100% network utilization on the clients) which I assume is the maximum speed I will ever get.

    Thankyou for your assistance and I hope Microsoft acknowledge this as a problem and fix this.

    Cheers, Matt

    Wednesday, March 24, 2010 1:36 PM
  • The article you've linked to is part of the book "Introducing Windows Server 2008", and is outdated information. When configuring the multicast block size, the most important thing to consider is the MTU of your network.

    For most ethernet networks, the MTU is 1500 bytes. Since WDS uses IPv4 UDP multicast, we need to subtract the size of those headers as well. WDS also uses a packet header to identify it's protocol, so we must include this also.

    • IPv4 header = 20 bytes
    • UDP header = 8 bytes
    • WDS header = 87 bytes in default configuration

    Please note that ApBlockSize represents the size of the data block being sent to the client, not the overall size of the packet that WDS is sending.

    1500 - (20 + 8 + 87) = 1385. This is the best size to use for ApBlockSize if you do not want packet fragmentation to occur.

    From this, you can also see that by using a value of 1482 fragmentation will occur. The first packet will contain the majority of the payload (1385 bytes out of 1482), and the second packet will contain very little (97 bytes). This is why you're seeing a performance increase by going to a block size of 1024 from a block size of 1482.

    From this, you can also see where the default block size comes from. With 5 packet fragments plus the original packet, you can send (1385 + (5 * 1480)) = 8785 bytes.

    Wednesday, March 24, 2010 11:33 PM
  • Hi Aaron,

    Thankyou for your response. However I seem to have another issue and wandered if you could shed any light on this for me?

    If I start a multicast session on the server the image is now being deployed at around 80Mbps. But if I try to start another session or even try to load a boot image image on another client it is extremely slow until the 1st multicast session has finished.

    The server is connected to the network with a 1Gbps connection so I would have thought that in theory I would be able to run multiple session at once as each session is only using around 8% of the available network bandwidth on the server. But this does not seem to be the case as if I am running a multicast session I do not seem to be able to do anything else until it has finished i.e. load boot images over the network, start unicast, join another multicast session etc.

    It as if WDS can only use 100MB of bandwith and no more???

    Cheers, Matt

    Thursday, March 25, 2010 3:36 PM
  • Well, Can I just say a HUGE thank you for this post! I have been fighting with this at a number of locations. Yet other locations tend to be just fine with the default settings. Looks like the schools that are fine are on Cysco switches that have more memory to handle the fragmentation and the schools that have issues are HP's with less memory to buffer the fragmentation. We are seeing night/day improvements after setting the ApBlockSize. Thank you thank you!
    Tuesday, July 6, 2010 8:06 PM
  • What version of Windows Server is this on? In Server 2008, WDS had a setting called "Network Profile" which by default was set to 100Mbps. If your server is connected at 1Gbps, you need to change the profile to the 1Gbps profile.

    For Server2008 R2, we figured out a way to dynamically calculate the optimal network settings on the fly, so the concept of Network Profiles was removed.

    Tuesday, August 17, 2010 5:08 PM
  • Modifying apBlockSize appears to fix the issue.

     

    The size is set as default to something like 8582. This is far to big for us, and I would have thought for others too?

    We changed the block size to 1472.

    This compliments our client MTU size and router block size = That is 1472 data + 28 headers = 1500.

    Anything above this size creates fragmentation in the data, hence the issues.

     

    We now get no NACKs/Retries whatsoever and speeds are acceptable:

    • -6GB image in about 12 mins.
    • -Average 80Mb/s
    • -99 % NIC Utilisation on clients.

     

    I hope this information helps others.


    Very helpful indeed, you've saved me a lot of work. By changing the ApBlockSize, the multicast stream really gets stable, and stays at a higher but most of all, a stable transfer speed. For me, I don't get it higher than ~30000KBps, but that's on the other hand quite high, enables me to multicast a 76GB image to a whole classroom in 20 minutes!
    Thursday, April 18, 2013 12:21 PM
  • I have been fighting with multicasting on SCCM 2012 (Server 2012) for weeks now. The max I can get per stream is about 8000KBps on a 99-client stream. On a single client stream, I've maxed out at 10000KBps. My ApBlockSize is set to 1385. Will not setting it above that result in more fragmentation (of which we have a TON). Our router and switches are at least 1GB speeds. My image is 80GB and takes about an hour via Unicast and four hours or more via multicast. Will not setting the ApBockSize to 1472 create more fragmentation? What am I doing wrong here?
    Wednesday, July 22, 2015 3:14 PM