Re: New optimized soreceive_stream() for TCP sockets, proof of concept


On Mon, 5 Mar 2007, Andrew Gallatin wrote:

Robert Watson writes:
> On Mon, 5 Mar 2007, Andrew Gallatin wrote:
>
> > With the patch, we finally seem to be performance competative on the receive
> > side with Linux x86_64 and Solaris/amd64 on this same hardware. Both of
> > those OSes do much better (saturate the link with jumbos) when CPU affinity
> > is used to bind the interrupt handler and netserver process to different
> > cores on the same socket. I imagine FreeBSD may be able to do even better
> > if it ever grows CPU affinity support for both interrupt handlers and
> > processes. With the patch, it performs at least as well, if not better
> > than, Solaris and Linux do without CPU affinity.
>
> I don't have numbers in front of me, and am currently packing for a trip > to Tokyo so won't find them before traveling, but my experience has been > that binding the ithread to a specific CPU is very helpful in improving > receive performance. You can slap a sched_bind(0) into the interrupt > handler the first time it runs and it should stick appropriately, and add > a sysctl to sched_bind() for a user process as a hack to test it out.

You lost me at adding the sysctl for the user process.. Does FreeBSD have, or plan to have, an interface to bind threads to CPUs?

Right now we have a facility to bind kernel threads to specific CPUs (sched_bind(9)), but not an API or mature facility to bind user threads (accept in as much as the scheduling of the user thread reflects the scheduling of the kernel thread, which works but requires adding a system call or sysctl as a hack). There was a recent discussion on arch@ about the relative merits of various such user APIs floating around. Adding a basic model for user thread CPU binding would be fairly easy, but I'm not sure anyone owns the task currently.

The quick hack model for testing is to sched_bind() the user thread using a sysctl added for testing purposes, which I believe should work to demonstrate the general point, and can be done in about 15 minutes. Since I'm running out the door now, it will have to be someone else's 15 minutes. :-)

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-net@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: New optimized soreceive_stream() for TCP sockets, proof of concept
    ... >> those OSes do much better when CPU affinity ... You can slap a sched_bindinto the interrupt> handler the first time it runs and it should stick appropriately, and add> a sysctl to sched_bindfor a user process as a hack to test it out. ... Right now we have a facility to bind kernel threads to specific CPUs ), but not an API or mature facility to bind user threads. ...
    (freebsd-current)
  • Re: CONFIG_PREEMPT and server workloads
    ... I've been meaning to do another round of latency tuneups for ages, ... but the other CPU is hanging on the lock for ages ... ALSA drivers which would cause a dump_stackto be triggered if the audio ... of underruns we can just ask them to turn on the sysctl and we get a trace ...
    (Linux-Kernel)
  • Re: [PATCH] avoid cpu removal if busy revisited
    ... added sysctl ... cpu hot remove migrates all tasks on target cpu by force. ... +#ifdef CONFIG_HOTPLUG_CPU ...
    (Linux-Kernel)
  • Re: cpufreq probs dual core intel
    ... leroy# sysctl hw.acpi.acline ... Whats odd to me is the temp the cpu is running at. ... <Enhanced SpeedStep Frequency Control> on cpu0 ...
    (freebsd-current)
  • Re: Setting priority for exe built in DFf6.6
    ... Because its the only USER process and as such it gets the CPU scheduler's ... Windows scheduler is VERY sophisticated with its multi-thread ... setting capability and still does the RIGHT thing for your unwashed user ...
    (comp.lang.fortran)