Re: kernel performance issue - spins on readers/writer locks

From: Gavin Maltby (G_a_v_i_n.M_a_l_t_b_y_at_sun.com)
Date: 10/07/03 

Date: Tue, 07 Oct 2003 10:38:19 +0100

Ciaran Harron wrote:
> We have a billing application running on Solaris 8 against Oracle 8.1.7. We
> run 36 parallel processes. Recently we have experienced a deterioration in
> performance of around 50%.

The obvious question is: did anything change recently? Did you increase
load, add another database, add/remove hardware, change storage, add
users, upgrade application, reindex etc.

> Having checked the performance of Oracle we have concluded that the
> deterioration is not related to Oracle. Actually the problem is quite
> strange as there are no obvious bottlenecks. We have 24 CPUs installed but
> the system is 60 idle. The remaining split between user, system and iowait
> looks normal.
>
> The only thing I found strange was the output of mpstat. Most of the figures
> were around half of what they were one month ago, with the exception of the
> srw figure which increased. The manpage says that this is the spins on
> readers/writer locks.

Use lockstat(1M) to gather statistics on kernel locks, including rwlocks.
Easiest may be to run 'lockstat sleep 30' or similar, which will gather
lock statistics for 30 seconds (make sure Oracle etc is running and
experiencing slowdown during this time). Obviously most locks will
experience some contention (there'd be no need to lock if they didn't)
but you need to analyze the output to decide of that's where you're
spending your time.

Do you have datafiles in UFS? I have vague recollections of single writer
locks to ufs files getting in the way of databases in the past. I think
that locking mechanism was revised in Solaris 9.

> Can the srw be the problem and if so what can cause this? It is a semaphore
> problem and do we need to reconfigure the kernel to fix?

It's possible, but you need to get more detailed performance info before
we can jump to this conclusion.

rwlocks are separate to semaphores and there's no specific tuning for them.
They allow multiple readers or one writer to a resource. If a writer is
waiting (for readers to complete) then further readers are blocked etc.
They're all a bit delicate as you need to prevent readers starving writers
and writers starving readers. If you've got to the point of throughput
limitation because of throttling on some resource that is rwlock protected,
then you've either got to find a way of backing off (eg, tuning application,
looking for ways in which the application load aggravates underlying kernel
implementation etc) or the locking strategy for that resource needs to be
revised (ie, in a patch or upgrade of the OS) to no longer limit throughput.

Gavin

Relevant Pages

  • [RFC PATCH] Fair low-latency rwlock v3
    ... same CPU. ... Using a writer subscription to the rwlock to make sure the readers stop taking ... periodical interrupt readers on 7/8 cpus. ...
    (Linux-Kernel)
  • Re: [PATCH 0/3] 64-bit futexes: Intro
    ... i suspect _any_ abstract locking functionality around a data structure ... For example, some loads are almost entirely read-read locks, with only ... rwlock situation is the "tons of readers, occasional writer". ...
    (Linux-Kernel)
  • [RT PATCH v2] seqlock: serialize against writers
    ... This fixes a nasty deadlock on my systems under ... Seqlocks have always advertised that readers do not "block", ... the readers to serialize with the writers under contention. ... and it will pi-boost the writer to prevent inversion. ...
    (Linux-Kernel)
  • [RT PATCH v3] seqlock: serialize against writers
    ... Problem 1) If A spins in seqbegin due to writer contention retries, ... Problem 3) Lower priority tasks such as C may continually acquire the write ... a rwlock for the basic spinlock previously used, and forces the readers ... unsigned ret; ...
    (Linux-Kernel)
  • Re: [RT PATCH v3] seqlock: serialize against writers
    ... Problem 1) If A spins in seqbegin due to writer contention retries, ... Problem 3) Lower priority tasks such as C may continually acquire the write ... a rwlock for the basic spinlock previously used, and forces the readers ... unsigned ret; ...
    (Linux-Kernel)