Re: a proposed callout API


On Thu, 30 Nov 2006, Ivan Voras wrote:

No trying to take sides here, but for us willing to learn here, what exactly are the problems in Matt Dillon's suggestions? From a novice's POV, having per-cpu queues looks (emphasis: looks) very scalable and performant.

The implications of adopting the model Matt proposes are quite far-reaching: callouts don't exist in isolation, but occur in the context of data structures and work occuring in many threads. If callouts are pinned to a particular CPU, and can only be scheduled, rescheduled, and cancelled from that CPU, that implies either that all work associated with that callout is also pinned to the CPU, or that migration or message-passing be involved if the requirement comes up in a thread on another CPU.

Consider the case of TCP timers: a number of TCP timers get regularly rescheduled (delack, retransmit, etc). If they can only be manipulated from cpu0 (i.e., protected by a synchronization primitive that can't be acquired from another CPU -- i.e., critical sections instead of mutexes), how do you handle the case where the a TCP packet for that connection is processed on cpu1 and needs to change the scheduling of the timer? In a strict work/data structure pinning model, you would pin the TCP connection to cpu0, and only process any data leading to timer changes on that CPU. Alternatively, you might pass a message from cpu1 to cpu0 to change the scheduling.

The idea of processing timers in multiple threads and pinning them to multiple CPUs clearly isn't a bad idea: we could likely benefit from parallelism (and generally, concurrency) in timer processing. One of the things we discussed at the recent developer summit was subsystem callout threads (introducing the opportunity for parallism without committing to a particular CPU scheduling model), as well as per-CPU callout threads but protected using mutexes so that reschedule/cancel/etc can be performed form other CPUs still. Changing the API so that scheduling/rescheduling/etc activities themselves must occur on a particular CPU has serious implications and commits us to an architectural approach for which there is little concensus. If the goal is simply parallelism, it's possible to accomplish that without embedding assumptions about the synchronization model at this point. Take a look at the USENIX paper by Paul Willmann (et al) at Rice for some rather interesting experimentation, measurement, and discussion precisely along these lines:

http://www.ece.rice.edu/~willmann/pubs/paranet_tr06-872.pdf

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-arch@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-arch
To unsubscribe, send any mail to "freebsd-arch-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: posix-cpu-timers revamp
    ... Where "interesting tick" means "tick in which a process timer has ... The thread CPU timers track the thread CPU clocks, ... The process CPU timers track the process CPU clocks. ...
    (Linux-Kernel)
  • Re: [PATCH] Migration of Standard Timers
    ... standard timers from one cpu to another. ... time critical threads with minimal timer induced latency ... Note that migrating timers will not, by itself, keep new ...
    (Linux-Kernel)
  • [PATCH] Migration of Standard Timers
    ... This patch allows the user to migrate currently queued ... standard timers from one cpu to another. ... Note that migrating timers will not, by itself, keep new ...
    (Linux-Kernel)
  • [PATCH] Migration of standard timers
    ... standard timers from one cpu to another, ... timer induced latency on the chosen cpu. ... +int move_timers ...
    (Linux-Kernel)
  • Re: [patch 1/2] round_jiffies infrastructure
    ... This avoids multiple timers to fire within the second for no real ... The exact wakeup moment is skewed by the cpu number, ...
    (Linux-Kernel)