Re: FreeBSD 5.2.1: Mutex/Spinlock starvation?

From: Robert Watson (rwatson_at_freebsd.org)
Date: 06/04/04

  • Next message: Steven Hartland: "Re: FreeBSD 5.2.1: Mutex/Spinlock starvation?" 
    Date: Fri, 4 Jun 2004 16:08:21 -0400 (EDT)
To: Ali Niknam <ali@transip.nl>

    On Thu, 3 Jun 2004, Ali Niknam wrote:

    > First of all: this is my first posting in this group so please be gentil
    > :)

    Welcome :-).

    <snip>
    > Now i unfortunately do not know enough about the internals of BSD to do a
    > very estimated guess, but i'll give a shot nevertheless: my estimate is that
    > due to the tremendous amount of 'locked' processes the system simply starves
    > of CPU to do anything. My guess is the Locking mechanism probably uses
    > some kind of 'spin' to wait until the resource is unlocked (whichever
    > resource it is, probably something network related, though).
    </snip>

    Actually, by default, most mutexes in the system are sleep mutexes, so
    they sleep on contention rather than spinning. In some cases, this
    actually hurts more than spinning, because if the mutex is released
    quickly by the holder, then you pay the context switches which cost more
    than spinning for the short period of time.

    You might want to try adding "options ADAPTIVE_MUTEXES" to your kernel
    configuration, which will cause mutexes to spin briefly on SMP systems
    before sleeping, and has been observed to improve performance quite a bit.

    > I would be very interested to hear what this problem could be; perhaps i
    > can test a little if someone has solutions (i cant test much
    > unfortunately, it's a production system).

    As you may or may not be aware, improving locking and parallelism in
    FreeBSD 5.x is a big on-going task, with a lot of activity. A moderate
    quantity of recent locking work has occurred since 5.2.1 release, so
    depending on your tolerance for experimentation on this system, you might
    wish to give 5-CURRENT a try. Be warned that 5-CURRENT, while having a
    number of performance enhancements, also has some stability regressions,
    more recent ACPI code, etc. I'm using older snapshots of 5-CURRENT in
    production today, but generally not newer than about April or early May.
    If you do try -CURRENT, take a look at UPDATING, and make sure to disable
    a lot of the debugging features present if you're interested specifically
    in performance.

    If you have a lower tolerance for instability, there are a number of minor
    performance tweaks that can be easily back-ported to 5.2.1, such as the
    change to proc.h to make grabbing and releasing the proc lock conditional
    on p_stops having events defined. This removes several mutex operations
    from each system call, and I've observed the difference in a pretty
    measurable way on micro-benchmarks. It's also pretty low risk. The
    change is src/sys/sys/proc.h:1.366. There are some other related changes
    that can probably be dug up, including changes to improve the performance
    of the scheduler in the presence of threads, etc.

    Robert N M Watson FreeBSD Core Team, TrustedBSD Projects
    robert@fledge.watson.org Senior Research Scientist, McAfee Research

    _______________________________________________
    freebsd-hackers@freebsd.org mailing list
    http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
    To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

  • Next message: Steven Hartland: "Re: FreeBSD 5.2.1: Mutex/Spinlock starvation?"

    Relevant Pages

    • Re: FreeBSD 5.2.1: Mutex/Spinlock starvation?
      ... >> Actually, by default, most mutexes in the system are sleep mutexes, so ... >> they sleep on contention rather than spinning. ... > timeout to a quite high value. ... the lock spins so long as the current owning ...
      (freebsd-hackers)
    • Re: msleep() on recursivly locked mutexes
      ... global strategy when we write device drivers, so that various modules can be connected together without races and locking order reversals. ... In my view there are two categories of mutexes. ... 1a) The global mutex protects interrupts/callbacks into a hardware driver. ... But it does not make sense to make lock #2 lock shorter than lock #1. ...
      (freebsd-hackers)
    • Re: Solaris adaptive mutex in userland?
      ... >> There is no interface for using adaptive mutexes. ... > If the mutex is already locked, ... unsuccessful in acquiring the mutex while spinning, ... diving into the kernel). ...
      (comp.unix.solaris)
    • Re: Solaris adaptive mutex in userland?
      ... >> There is no interface for using adaptive mutexes. ... > If the mutex is already locked, ... unsuccessful in acquiring the mutex while spinning, ... diving into the kernel). ...
      (comp.unix.programmer)
    • Re: The Home Of Super Slinky Helical Antennas -by- KB0XA
      ... If I recall, your coax wrapped around the spinning ... hub, locking it solid. ...
      (rec.radio.shortwave)