Re: sendfile(2) SF_NOPUSH flag proposal

From: Igor Sysoev (is_at_rambler-co.ru)
Date: 05/27/03

  • Next message: Bruce A. Mah: "Re: New bootloader!" 
    Date: Tue, 27 May 2003 21:46:54 +0400 (MSD)
To: Terry Lambert <tlambert2@mindspring.com>

    On Tue, 27 May 2003, Terry Lambert wrote:

    > Igor Sysoev wrote:

    > > I mean that if you have 230 bytes header then sendfile() will send it
    > > in separate packet nevertheless the size of header and of the file.
    > > Something like this - 230, 1460, 1460, ...
    >
    > Again, see other post: this is arguably a sendfile(2) bug,
    > though a reall minor one; one which should be addressed in
    > the sendfile(2) implementation, and doesn't need options
    > added to the API in order to address it.

    How do suppose to coelesce the file pages ? Wire two or more pages
    to mbuf's at once ?

    BTW I did not see how sendfile() work over jumbo ethernet. I suspect
    that without TCP_NOPUSH it sometimes sends 4096 or 8192 bytes packets
    instead of 9000.

    > > > > it will return me 230 bytes:
    > > >
    > > > The "HEAD" is atypical, compared to the "GET"; the full Google
    > > > front page is larger than that, and consists of multiple files;
    > > > assuming you support HTTP/1.1 and pipelining, it's going to be
    > > > a back-to-back transfer involving multiple sendfile() calls.
    > >
    > > I use HEAD to show you the size of the HTTP header.
    > > The HEAD is atypical but such small HTTP header is typical.
    >
    > Here is my problem: you are arguing both amortized cost and
    > total cost, depending on which is more supportive of your
    > main thesis. These arguments are seperate and orthogonal to
    > each other: they don't support each other. You can argue
    > tiny files, and a relatively high total cost, or you can argue
    > large files and pipelining, and a relatively high amortized
    > cost, but you can't argue both time and large files and
    > many connections and one connection at the same time.

    Terry, I do not understand you.
    My argument is simple - I want to avoid the partial packets because it
    decreases the number of packets. That's all. There's nothing about
    amortized cost or total cost. I do not even know what they are.

    > Personally, I'd step back and get the arguments straight,
    > and get an implementation that demonstrates statistically
    > significant performance differences, and then come back, if
    > I wanted to press the case for additional option flags. I
    > have done this several times in the past, e.g. with my soft
    > interrupt coelescing implementation that's now part of most
    > of the ethernet drivers people care about.
    >
    > Actually, in this case, I'd just try to fix sendfile(2) to
    > do the packet coelescing I'd expect, given the relative
    > state of the TCP_NODELAY and TCP_NOPUSH options flags.

    Actually, sendfile() already works according to TCP_NOPUSH flag.
    I do not know about TCP_NODELAY - I do not work with it.
    But if you turn TCP_NOPUSH on then sendfile() will send the full packets.
    If you turn TCP_NOPUSH off then sendfile() will send some packets partially
    filled. It's correct.

    > BTW: I'm still wary of the initial fault on the file data, if
    > it's not already in cache: arguably, it's better to start
    > sending the headers, and avoid the startup latency of delaying
    > sending the headers until the fault is satisfied: part of the
    > thing that's going to be eating your PCI bandwidth is the
    > disk I/O, and your disks are going to be the slowest data
    > sources/sinks in the whole equation.

    I agree but after all it's 20ms or so delay.

    > In any case, I expect that this should be handled in the
    > context of TCP_NODELAY and TCP_NOPUSH, rather than by adding
    > options to work around an arguably broken sendfile(2).

    sendfile() already works nice with TCP_NOPUSH. I propose only the flags
    that allow to turn TCP_NOPUSH (actually TF_NOPUSH) on/off inside sendfile().
    Then in one syscall you can turn TCP_NOPUSH on, send the HTTP header, the file
    pages and turn TCP_NOPUSH off if all file pages are wired to mbuf's.
    And this TCP_NOPUSH state is not bound by sendfile() internals, you
    can control it via setsockopt/getsockopt(TCP_NOPUSH).

    Igor Sysoev
    http://sysoev.ru/en/

    _______________________________________________
    freebsd-arch@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-arch
    To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org"

  • Next message: Bruce A. Mah: "Re: New bootloader!"

    Relevant Pages

    • Re: [git pull] FireWire fixes and documentation update
      ... to receive the following IEEE 1394/ FireWire subsystem update. ... firewire: cdev: require quadlet-aligned headers for transmit packets ... firewire: cdev: change license of exported header files to MIT license ...
      (Linux-Kernel)
    • core dump on kldload atapicam
      ... Dump header from device /dev/ad4s4b ... Panic String: page fault ... data packets ... connections established ...
      (freebsd-current)
    • Re: Improving FreeBSD NFS performance (esp. directory updates)
      ... >> I don't think the network is at fault, nor is the server really going ... 155645171 data packets ... discarded for bad header offset fields ... 790 connections established ...
      (freebsd-questions)
    • Re: Advice on a multithreaded netisr patch?
      ... cache, and the link layer will take a cache miss on the front of the ... netstat packets per second and my application's TCP ... header flowid field, and use that instead of the hash for work placement. ...
      (freebsd-net)
    • Re: sendfile(2) SF_NOPUSH flag proposal
      ... > decreases the number of packets. ... > amortized cost or total cost. ... For a small amount of data, ... > But if you turn TCP_NOPUSH on then sendfile() will send the full packets. ...
      (freebsd-arch)