Re: [HEADS UP!] IPFW Ideas: possible SoC 2008 candidate

On Wed, 26 Mar 2008, Julian Elischer wrote:

it wouldn't.. you'd add them together before presenting them. but every time a packet changes a counter that is shared, there is a chance that it is being altered by another processor, so if you have fine grained locking in ipfw, you really should use atomic adds, which are slow, or accept possibl collisions (which might be ok) but still cause a lot of cross cpu TLB flushing.

In malloc(9) and uma(9), we maintain per-CPU stats, coalescing only for presentation, relying on soft critical sections rather than locks to pretect consistency. What's worth remembering, however, is that recent multicore machines have significantly optimized the cost of atomic operations on cache lines held for write by the current CPU, and so the cost of locking has dramatically fallen in the last few years. This re-emphasizes the importance of careful cacheline management for per-CPU data structures (particularly, don't put data written by multiple CPUs in the same cacheline if you want the benefits of per-CPU access).

Where read-write locking is the best model, Stephan's recent work on rmlocks looks quite promising. In my micro-benchmarks, on recent hardware it performs extremely well on SMP for read locks, but still requires optimization for UP-compiled kernels. For stats and writable structures, such as per-CPU caches, rmlocks aren't very helpful, but when compared with replicating infrequently written data structures across many CPUs, rwlocks/rmlocks offer a much simpler and less error-prone programming model. We need to see more optimization and measurement done on rmlocks for 8.x, and the lack of full priority propagation for rwlocks has to be kept in mind.

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-hackers@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: [PATCH 5/7] Use %gs for per-cpu sections in kernel
    ... I'll produce a patch, so we can see what we're talking about... ... This patch actually uses the gs register to implement the per-cpu ... per-cpu offset for the particular cpu ... movl %eax,%cr0 ...
    (Linux-Kernel)
  • Re: route finder
    ... cpu. ... computer science data structures course books would mention the problem. ... Paul T. ... I believe that the only solution is to build alone me the algorithm. ...
    (microsoft.public.dotnet.framework.compactframework)
  • Re: [PATCH 2/4] i386 GDT cleanups: Use per-cpu GDT immediately upon boot
    ... it doesn't mirror any GDT at the moment. ... The boot CPU uses the per-cpu var directly, ... extern void enable_sep_cpu; ...
    (Linux-Kernel)
  • Re: [PATCH 24/25] r/o bind mounts: track number of mount writers
    ... superblock counters in XFS as they've been debugged quite well ... It also handles cases like hotplug cpu nicely that this code ... I'll take a look at xfs to see what I can get out of it. ... Where the per-cpu writer count underflows. ...
    (Linux-Kernel)
  • [PATCH] Fix hotplug cpu on x86_64
    ... > (still pointing to the original per-cpu data, ... Initialize cpu_possible_map properly in the hotplug cpu case. ...
    (Linux-Kernel)