none
Detecting the amount of UDP packet loss

    Question

  • Hello,

    we're connecting several hundred hardware video camera devices to a Windows 2003 Server system. The hardware devices speak the RTP protocol using UDP multicast. I'm currently writing the receiving application that has to handle several 100 mbit/s of incoming video data.

    I'm using .NET sockets (that perform quite fast as we measured, maybe because MS did a good by using IOCP etc. :). When I use a very small UDP-Socket-Buffer size I see lots of packet losses. When I use a big UDP-Socket-Buffer I see no packet losses. So the small UDP-Socket-Version that looses packets should loose the packets in the UCP-Socket-Buffer and nowhere else, right ? So I can use the small UDP-Socket-Version as a test-tool to enforce packet-loss in the UDP-Buffer.

    When I use "netstat -s" I don't see any of the packet loss I enforced with the small-UDP-buffer-application. It is important for us to have a tool handy for our customers (airports, banks, subways ...) to analyze in case of trouble if there's an overflow in the sockets UDP-buffer or if anything else drops too much packets (Camera, Switch, NIC, Server-Application whatever ...).

    I tried not only "netstat -s", I also tried almost any good-sounding perfmon counter ... but I didn't see the packet loss I enforced.

    Question:
    How can I see wether there's packet loss in the UDP-socket-buffer ? Is "netstat -s" the right command ? Is it reliable ? Does it depend on features of the NIC-driver ? Why don't I see the dropped packets created with my test-tool ?

    Thanks a lot,
    Roelof Berg

    (Note: We can see the same behaviour with a native c++ alternative that uses berkley-sockets and blocking select(). So this is not .NET related ...)
    Monday, October 13, 2008 12:26 PM

All replies

  • Hello RoelofBerg4BoschDe,
        If you want to monitor the amount of UDP packet loss you are experiencing I would use the performance logs and alerts utility.  The counter I believe you would be interested in is under the UDP performance object, counter being "Datagrams Received Errors".  Also, you can query any of these counters via WMI.

    Hope this helps!
    A+, Network+, MCSE (2000 \ 2003), MCTS, CISSP LinkedIn
    Tuesday, October 14, 2008 2:52 PM
  • Hello TampaITGuy,

    thank you very much for the answer. Unfortunately this counter also does not measure the UDP-Loss in our environment. I tried everything in perfmon.exe that sounded like being capable of measuring anything interesting regarding the UDP-Loss (static counters, NIC-bound counters etc.) - but every counter I tried failed to report the UDP loss.

    I really wonder wether measuring the UDP-Loss must be supported by the NIC driver. Maybe I will install a high-quality NIC into our test-server during the next days and try to reproduce the issue with a (hopefully) better NIC driver.

    Thank you,
    Roelof Berg
    Wednesday, October 15, 2008 9:39 AM
  • One thing here if I may.  I'm wondering if you can truly measure packet loss as UDP is a connectionless protocol.  Here's why I say that:
     

    Note that since it is a connectionless protocol there is no serialization so there would be no way of knowing that a packet did not get there unless the data (which would need to be tracked at the application level) serialized them.
    A+, Network+, MCSE (2000 \ 2003), MCTS, CISSP LinkedIn
    Wednesday, October 15, 2008 10:24 AM
  • Hello TampaITGuy,

    thank you for pointing that out and getting into detail. The protocol we use (RTP) uses a sequence-counter in each packet. We're tracking that counter on application level. So in our case we can 100%ly determine each single missing packet.

    (Note: That packets are missing doesn't necessarily mean that the missing packets are lost in the UDP buffer. To track this down we compiled two versions of our RTP-receiver. One with a big UDP-Buffer and one with a small UDP-Buffer. The application with the big UDP-Buffer looses only 15 packets in 24 hours (on 200 mbit network-load). But the application with the small UDP-Buffer constantly looses many, many packets each second. I hope our way to enforce UDP-Loss really does enforce UDP-Loss in the UDP-Socket-Buffer and nowhere else.)

    Thank you,
    Roelof Berg
    Wednesday, October 15, 2008 11:34 AM
  • Hello alltogether,

    we opened a support case at MS regarding this question which was finally answered by the MS developers:

    Netstat displays the statistics from the transport layer. Winsock layer sits on top of the transport layer. Datagrams may be dropped at various layers within the networking stack. The netstat statistics do not account for datagram drops happening at Winsock layer due to exceeding the socket receive buffer limit.

     

    Unfortunately, there’s no tool currently that shows Winsock layer datagram drop statistics. The closest thing on Vista and later is Winsock ETW tracing which would log a trace entry for each drop. Disadvantage of the ETW tracing that only the support personal can convert it to readable format (and this is for internal use only).

     

    So this issue is probably not solveable. If anyone sees any solution (on Windows 2003 Server and without writing a network sniffer :) I'd be glad to hear about it.

    The MS support also told me about a Winsock performance monitoring tool fow windows CE. Maybe this can help, I haven't looked at it yet: http://msdn.microsoft.com/en-us/library/ms895049.aspx

    Thanks for the answers in this forum, thanks to the MS developers and MS supporters who processed our support call. I hope this thread will also help anyone else.

    Roelof Berg

    www.bosch.de
    www.berg-solutions.de

    Wednesday, November 12, 2008 2:09 PM