Re: Much improved sendfile(2) kernel implementation



Between TSO and your sendfile changes, things are looking up!

Here are some Myri10GbE 1500 byte results from a 1.8GHz UP
FreeBSD/amd64 machine (AMD Athlon(tm) 64 Processor 3000+) sending to a
2.0GHz SMP Linux/x86_64 machine (AMD Athlon(tm) 64 X2 Dual Core Processor
3800+) running 26.17.7smp and our 1.1.0 Myri10GE driver (with LRO).
I used a linux receiver because LRO is the only way to receive
standard frames at line rate (without a TOE).

These tests are all for sendfile of a 10MB file in /var/tmp:

% netperf242 -Hrome-my -tTCP_SENDFILE -F /var//tmp/zot -T,1 -c -C -- -s393216

The -T,1 is required to force the netserver to use a different core
than the interrupt handler is bound to on the linux machine. BTW,
it would be really nice if FreeBSD supported CPU affinity for processes
and interrupt handlers..

I did a number of runs with TSO and the patch applied and found that
setting the send-side socket buffer size to 393216 gave the best
performance in that case. I used this size for all tests, but it is
possible there is a different sweet spot for other configurations.
Note that linux auto-tunes socket buffer sizes, so I omitted the --
-s393216 for linux.


Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB

Without patch:
87380 393216 393216 10.00 2163.08 100.00 19.35 3.787 1.466
Without patch + TSO:
87380 393216 393216 10.00 4367.18 71.54 42.07 1.342 1.578
With patch:
87380 393216 393216 10.01 1882.73 86.15 18.43 3.749 1.604
With patch + TSO:
87380 393216 393216 10.00 6961.08 47.69 60.11 0.561 1.415

For comparision, if I reboot the sender into RHEL (Linux 2.6.9-11.EL x86_64):
87380 65536 65536 10.01 9333.00 28.98 75.23 0.254 1.321



The above results are the median result for 5 runs at each setting.


Drew
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Much improved sendfile(2) kernel implementation
    ... Between TSO and your sendfile changes, ... I used a linux receiver because LRO is the only way to receive ... Without patch: ...
    (freebsd-net)
  • Re: The ultimate TOE design
    ... > Plus we have things like TSO too but that doesn't require a full Linux ... If people ask about how to support TOE in Linux, you can say "sure, we ... send the line "unsubscribe linux-kernel" in ...
    (Linux-Kernel)
  • Re: Easy way to check whether a job is still running (without using TSO)
    ... From Linux, From Unix, from Solaris? ... Is there an easy way to check whether a previously submitted job is still running (without using TSO)? ...
    (bit.listserv.ibm-main)
  • Re: can the ENTER key be simulated in an ISPF panel
    ... You need TSO to be a multi-tasking environment, where a parent process can kill another after a specific interval. ... I've done this in UNIX, GCOS, LINUX, and an experimental OS, but TSO is a single task. ... For IBM-MAIN subscribe / signoff / archive access instructions, ... send email to listserv@xxxxxxxxxxx with the message: GET IBM-MAIN INFO ...
    (bit.listserv.ibm-main)
  • [PATCH] mmu notifiers #v2
    ... In short when the linux VM decides to free a page, ... This patch allows the shadow pagetables to be dropped and the page to ... behavior of the KVM gphysical memory. ...
    (Linux-Kernel)