Re: Locking etc. (Long, boring, redundant, newbie questions)

2007/3/29, Yar Tikhiy <yar@xxxxxxxxxxxxxxxx>:
On Wed, Mar 28, 2007 at 10:40:58AM +0100, Robert Watson wrote:
>
> Spin locks are, FYI, slower than default mutexes. The reason is that they
> have to do more work: they not only perform an atomic operation/memory
> barrier to set the cross-CPU lock state, but they also have to disable
> interrupts to synchronize with fast interrupt handlers. In general, you
> are right: you should only use a spin mutex if you are running in a fast
> handler, or synchronizing with a fast handler. The one general exception
> is the scheduler itself, which must protect its data structures with spin
> locks in order to implement sleeping primitives. As such, the scheduler
> lock and various other low level locks (such as in turnstiles) are
> implemented with spin locks and not default mutexes. Since default mutexes
> spin adaptively, the reduced overhead of contention experienced with spin
> locks (i.e., no scheduler overhead for simple contention cases) is also
> experienced with default mutexes.

By the way, could the following observation of mine be related to
the high cost of spin mutexes?

While doing some measurements of the performance of our vlan driver
in a router, I found that with RELENG_6 the pps rate through the
router degraded considerably (by ~100Kpps) if its physical interfaces
used hardware vlan tagging. I attributed that to the overhead of
allocating and freeing a mbuf tag for each packet as it entered and
then left the router. I used hwpmc and pmcstat to see which kernel
functions took most time and found that critical_exit() made top 5
in the list of CPU time eaters if the network interfaces were using
hardware vlan tagging.

The machine was an UP amd64, but it ran FreeBSD/i386, with an UP
kernel.

As I can see from the code, critical_exit() may grab and later
release a spin mutex. I've got a hazy recollection that our kernel
memory allocator uses critical sections to protect its per-CPU
structures. That's why I suspect that the effect I observed may
have its roots in the behaviour of spin mutexes. Could it be so?

This is not entirely true.
What happens is that if you enable preemption in your kernel,
critical_exit() holds sched_lock just beacause it needs to perform a
mi_switch(), so just another thread will be scheduled to run currently
(and, please, note that sched_lock will be dropped by internal
functions when context switch will be completed). Otherwise you don't
pay the penalty of the sched_lock acquisition.

Attilio


--
Peace can only be achieved by understanding - A. Einstein
_______________________________________________
freebsd-hackers@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Locking etc. (Long, boring, redundant, newbie questions)
    ... Spin locks are, FYI, slower than default mutexes. ... I used hwpmc and pmcstat to see which kernel ...
    (freebsd-hackers)
  • Re: [PATCH][RFC]: mutex: adaptive spin
    ... It's not the typical spin-mutex i think. ... long should we spin" question elegantly: we spin until the owner runs on a ... well-known locking behavior in the important boundary situations. ... I think a lot of stuff there is bit locks. ...
    (Linux-Kernel)
  • Re: Btrfs for mainline
    ... So I don't really know if -rt justifies adaptive locks in mainline/btrfs. ... The spinning does make a big ... In general, we want to spin as long as the lock holder is doing btree searches, ...
    (Linux-Kernel)
  • Re: [PATCH] OsdSynch.c modernization
    ... Spin mutex is too restrictive (e.g., it cannot be used with other ... disabling interrupts while you block on other locks is just as ... If the malloc is using M_NOWAIT you will ...
    (freebsd-current)
  • Re: all mutexes -> read-write locks?
    ... locking primitives it has come to be that mutexes and exclusively ... acquired reader-writer locks are almost the same in terms of overhead ... by all means change it to use rwlocks. ... changing it would allow a slow transition to using rw locks. ...
    (freebsd-arch)