Re: Slow disk write speeds over network

From: Terry Lambert (tlambert2_at_mindspring.com)
Date: 06/11/03

  • Next message: Terry Lambert: "Re: Slow disk write speeds over network" 
    Date: Wed, 11 Jun 2003 00:46:03 -0700
To: Jason Stone <freebsd-performance@dfmm.org>

    Jason Stone wrote:
    > > You haven't said if you were using UDP or TCP for the mounts; you
    > > should definitely use TCP with FreeBSD NFS servers; it's also just
    > > generally a good idea, since UDP frags act as a fixed non-sliding
    > > window: NFS over UDP sucks.
    >
    > Huh. I thought that the conventional wisdom was that on a local network
    > with no packet loss (and therefore no re-transmission penalties), udp was
    > way faster because the overhead was so much less.
    >
    > Sorry if this seems like a pretty basic question, but can you explain
    > this?

    Sure:

    1) There is no such thing as no packet loss.

    2) The UDP packets are reassembled in a reassembly queue
            on the receiver. While this is happening, you can only
            have one datagram outstanding at a time. With TCP, you
            get a sliding window; with UDP, you stall waiting for
            the reassembly, effectively giving you a non-sliding
            window (request/response, with round trip latencies per
            packet, instead of two of them amortized across a 100M
            file transfer).

    3) When a packet is lost, the UDP retransmit code is rather
            crufty. It resends the whole series of packets, and you
            eat the overhead for that. TCP, on the other hand, can
            do selective acknowledgement, or, if it's not supported
            by both ends, it can at least acknowledge the packets
            that did get through, saving you a retransmit.

    4) FreeBSD's UDP fragment reassembly buffer code is well
            known to pretty much suck. This is true of most UDP
            fragment reassembly code in the universe, however, and
            is not that specific to FreeBSD. So sending UDP packets
            that get fragged because they're larger than your MTU is
            not a very clever way of achieving a fixed window size
            larger than the MTU (see also #2, above, for why you do
            not want to used an effectively fixed window protocol
            anyway).

    Even if there were no packet loss at all with UDP, unless all
    your data is around the size of one rsize/wsize/packet, the
    combined RTT overhead for even a moderately large number of
    packets in a single run is enough to trigger the amortized cost
    of the additional TCP overhead being lower than the UDP overhead
    from the latency. Depending on your hardware (switch latency,
    half duplex, etc.), you could also be talking about a significant
    combined bandwidth delay product.

    Now add to all this that you have to send explicit ACKs with UDP,
    while you can use piggy-back ACKs on the return payloads for TCP.

    I think the idea that UDP was OK for nearly-lossless short-haul
    came about from people who couldn't code a working TCP NFS client.
    8-).

    -- Terry
    _______________________________________________
    freebsd-performance@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-performance
    To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org"

  • Next message: Terry Lambert: "Re: Slow disk write speeds over network"

    Relevant Pages

    • Re: Log Out Issues
      ... which possible with UDP. ... Here is game example but in NYSE broadcasting used ... your perormance may be worse than TCP. ... > would be if your protocol anticipates packet loss and does not ...
      (microsoft.public.win32.programmer.networks)
    • Re: Kerberos UDP vs TCP
      ... The main reason for using UDP by default is that it's lightweight compared ... UDP starts having problem if the network is not reliable, i.e. busy network, ... slow links, or has packet loss. ... The advantage of using TCP is that it uses ...
      (microsoft.public.security)
    • Re: TCP or UDP?
      ... So far it seems you favor TCP with the robustness ... it has 12 bytes greater overhead per packet. ... It has another hidden overhead in the sense it does not ... needed in intranet it's no problem if I would use UDP?). ...
      (microsoft.public.win32.programmer.networks)
    • Re: Net Meter [radio station bit rate] tutorial please? I want to nail this????????
      ... addition to the audio stream (UDP doesn't, ... headers etc and, in any event, lots of audio streams still use TCP) ... UDP doesn't have what? ... TCP has overhead because it includes extensive error correction, ...
      (alt.radio.digital)
    • Re: TCP or UDP?
      ... With no packet loss that makes no difference. ... UDP too if guaranteed delivery is needed. ... tune it to a specific usage model, which you can't do for TCP. ...
      (microsoft.public.win32.programmer.networks)