Re: sendfile(2) SF_NOPUSH flag proposal

From: Terry Lambert (tlambert2_at_mindspring.com)
Date: 05/27/03

  • Next message: Terry Lambert: "Re: sendfile(2) SF_NOPUSH flag proposal" 
    Date: Tue, 27 May 2003 08:29:19 -0700
To: Peter Jeremy <peterjeremy@optushome.com.au>

    Peter Jeremy wrote:
    > On Tue, May 27, 2003 at 11:57:20AM +0400, Igor Sysoev wrote:
    > >I thought about it more and I agree with you. TF_NOPUSH should be turned on
    > >at the start of a transaction and turned off at the end of a transaction.
    > >
    > >So I think there should be two flags:
    > >SF_NOPUSH - it turns TF_NOPUSH on before the sending. It's cheap:
    > >SF_PUSH - it turns TF_NOPUSH off after the sending has been completed.
    >
    > I agree that the code appears trivial but in order to justify its
    > inclusion, you will need to demonstrate that there is some benefit to
    > FreeBSD to implement this code. Good justification would be:
    >
    > 1) The same API is implemented somewhere else (or there is agreement
    > between multiple groups to implement it). I don't believe this
    > functionality is implemented anywhere else and you've not provided
    > any evidence that any other groups are considering such functionality.

    Actually, the functionality can be implemented *without* going
    and implementing the API. It should really be contrlled already
    by the TCP_NODELAY option *not* having been set by the user, and,
    for last-block next-first-block coelescing, by TCP_NOPUSH *having*
    been set.

    Basically, the stack is minorly misbehaving on us in the sendfile
    case; effectively, it's unintentionally fragging up to one packet
    between the user supplied header (if any) and the file content,
    and the file content and the user-supplied trailer (if any).

    It's nothing to be terrifically concerned about, unless you are
    paying by the packet, you keep you connections open a very long
    time (e.g. HTTP/1.1), such that the amortized packet count is
    relatively high, and your files, headers, and trailers are tiny,
    enough that the frags constitute a significant portion of your
    packet traffic.

    In other words, you have to win the lottery. 8-).

    > 2) The new feature provides significant performance benefit. In this
    > case, I believe the overhead of calling setsockopt(2) is negligible
    > so the performance gain would be negligible.

    The overhead of toggling it would be costly. However, I really
    don't understand why he isn't just not setting TCP_NODELAY in
    the first place, since it's an affirmative option, and then
    leaaving the socket alone to act like it's supposed to act.

    > 3) The new feature provides novel functionality that cannot be
    > achieved using the existing API (eg kqueue(2)). The functionality
    > is already available via setsockopt(2) so this isn't applicable.

    Heck; I'd argue that it can be achieved with sendfile(2), if
    you leave the TCP options alone, and are willing to accept not
    setting TCP_NOPUSH for back-to-back potentially one packet
    worth of overhead, just by reorganizing the sendfile(2)
    implementation to comply with existing default conditionals.

    > At this stage, I would suggest that you need to do better than "the
    > change is cheap" to justify adding this feature. Can you quantify
    > the performance benefits, or provide some other justification?

    I'd also like to see a performance comparison; the issue is
    probably that, without a testbed that can drive traffic at
    full Gigabit speeds, he's probably not going to be able to
    show anything of statistical significance from this; at full
    Gigabit speed, he could probably show CPU copy overhead that's
    high enough to impact total top-end throughput, as he runs
    out of CPU to do the copies. IMO, that'd only be true if his
    data set was small enough to fit in cache after the first one
    or two sends. The mbuf allocator overhead shows the same
    level of overhead, though, and you could reclaim performance
    there, instead, if you were looking for low-hanging fruit.

    -- Terry
    _______________________________________________
    freebsd-arch@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-arch
    To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"

  • Next message: Terry Lambert: "Re: sendfile(2) SF_NOPUSH flag proposal"

    Relevant Pages

    • Re: bites, bytes, packets, TCP, and DSL speed
      ... there will be 18 bytes of overhead per frame (source and destination MAC ... of overhead (version, header length, type of service, packet length, ... assuming the network can carry full-size Ethernet frames end-to-end, ... Packets larger than your line's MTU size will be fragmented at the IP ...
      (comp.sys.mac.comm)
    • Re: FreeBSD handles leapsecond correctly
      ... :never ever passes a packet again until a down/up on the receiving interface. ... considering that we haven't removed the MP lock from the network ... The single biggest overhead we have right now is that we have not ...
      (freebsd-current)
    • Re: FreeBSD handles leapsecond correctly
      ... >:I've been testing network and routing performance over the past two weeks ... >:never ever passes a packet again until a down/up on the receiving interface. ... > between 3 uS and 1.2 uS per packet of overhead. ... > serialized down to effectively 1 cpu due to the MP lock. ...
      (freebsd-current)
    • Re: CAsyncSocket and performance issues
      ... The Nagel Algorithm is one reason the OP can't depend on reading 8 bytes and getting all ... feeding data in to me, I saturated at 1300 packets/min, with an average packet size of 80. ... ever call to Receivedraws on a buffer. ... >There's extra overhead in making extra copies. ...
      (microsoft.public.vc.mfc)
    • Re: Cleaning Silvia with B-Brite or Cleancaf??
      ... The reality is that any transaction has an overhead to it. ... loss irrespective of margin. ... The shipping and handling just adds another overhead component. ...
      (alt.coffee)