Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 05:04:26PM -0400, Kris Kennaway wrote:

477 23472709 2810986 8 5671248 1900047 kern/kern_synch.c:220 (process lock)

The top 10 heavily contended mutexes are very different (but note the
number of mutex acquisitions, column 3, is about the same).

There is not much contention on unp_global_mtx any longer, but there
is a lot more on some of the other mutexes, especially the process
lock via msleep(). Off-hand I don't know what is the cause of this
bottleneck (note: libthr is used as threading library and libpthread
is not ported to sparc64).

i.e. the same number of msleep calls in both cases, but the patch
shuffles them from sbwait onto everything else.

Oops, they're the same number because that was the size of my ktr
buffer, duh :) Nevertheless it's a sample from the end of the same
workload, so the distribution should be comparable.

sigwait +27%

This appears to be the only relevant msleep wait channel that is using
the process lock.

static int
kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi,
struct timespec *timeout)
{
...
td->td_sigmask = savedmask;
SIGSETNAND(td->td_sigmask, waitset);
signotify(td);
error = msleep(&ps, &p->p_mtx, PPAUSE|PCATCH, "sigwait", hz);

i.e. several threads in the process are all sleeping for 1 tick and
then doing a thundering herd on the same proc lock when they wake up.

To confirm this I added a KTR tracepoint before the msleep call. It's
indeed being called from the various threads in mysqld. It's hard to
say for sure how many CPUs are contending at once, but plotting a
histogram of the time distribution of the calls indicates that several
CPUs are frequently executing this code in close proximity.

Kris

Attachment: pgpsp5iuWjehk.pgp
Description: PGP signature

Relevant Pages

  • Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)
    ... The top 10 heavily contended mutexes are very different (but note the ... There is not much contention on unp_global_mtx any longer, ... lock via msleep. ... This appears to be the only relevant msleep wait channel that is using ...
    (freebsd-current)
  • Re: race on multi-processor solaris
    ... More correctly, CONTENTION is not good. ... While CONTENTION on a lock is bad, so are contended "lock free" queues, or ... so called "lock free" data -- sparingly and only when necessary. ... be one or more bottlenecks further down the system stack; ...
    (comp.unix.solaris)
  • [PATCH 0/4] lock stat for 2.6.19-rt1
    ... On a contention, it increments a field atomically that's pointed in a variable ... is contented against during the lifetime of that lock. ... kernel and therefore is more meaningful. ... in line with the function initializer to avoid the binary tree lookup. ...
    (Linux-Kernel)
  • Lock profiling results on TCP and an 8.x project
    ... I've started to look at decomposing tcbinfo, and preparatory to that, I ran some lock profiling on simple TCP workloads to explore where and how contention is arising. ... These results appear to confirm that we need to look at breaking out the tcbinfo lock as a key goal for 8.0, with serious thoughts about an MFC once stabilized and if ABI-safe--these two workloads represent the extremes of TCP workloads, and real world configurations will fall in between, but as the number of cores goes up and the desire to spread work over CPUs goes up, so will contention on a single global lock. ...
    (freebsd-arch)
  • Re: [RFC PATCH] Fair rwlock
    ... Tweakable contention behavior seems interesting, ... comes in and asks for a read lock, it has to get it on the spot. ... ok, there's a pending write lock holder on _my_ CPU, so I need to just ... interrupts obviously have to be disabled here, because any reader can ...
    (Linux-Kernel)