Re: panic: sleeping thread owns a non-sleepable lock



On Wednesday 15 August 2007, Rong-en Fan wrote:
On 8/12/07, Max Laier <max@xxxxxxxxxxxxxx> wrote:
On Saturday 11 August 2007, Kris Kennaway wrote:
On Sun, Aug 12, 2007 at 02:22:35AM +0800, Rong-en Fan wrote:
I'm running 7.0-CURRENT as of yesterday, and it's very easy
to make it panic:

Sleeping thread (tid 100065, pid 1066) owns a non-sleepable lock
sched_switch(c50a1600,0,1,1c7a7e4,4217e5,...) at
sched_switch+0x190 mi_switch(1,0) at mi_switch+0x13f
sleepq_switch(c50a1600,0,c078a4e2,21b,c07e3820,...) at
sleepq_switch+0x87 sleepq_wait(c07e3820,0,c0770b7e,3,0,...) at
sleepq_wait+0x36 _sx_xlock_hard(c07e3820,c50a1600,0,0,0,...) at
_sx_xlock_hard+0x21d
fr_checknatout(f9c7a8d0,f9c7a8cc,64,c57ad900,c4de7400,...) at
fr_checknatout+0x29d
fr_check(c8cc4644,14,c4de7400,1,f9c7a9b4,...) at fr_check+0x9b1
fr_check_wrapper(0,f9c7a9b4,c4de7400,2,c54dab28,...) at
fr_check_wrapper+0x3f
pfil_run_hooks(c08057c0,f9c7aa4c,c4de7400,2,c54dab28,...) at
pfil_run_hooks+0x74 ip_output(c8cc4600,0,f9c7aa10,0,0,...) at
ip_output+0x913
tcp_output(cae322d0,cb277200,0,0,0,...) at tcp_output+0x1106
tcp_usr_send(c51e7318,0,cb277200,0,0,...) at tcp_usr_send+0x240
kern_sendfile(c50a1600,f9c7acfc,0,0,0,...) at
kern_sendfile+0x1037
sendfile(c50a1600,f9c7acfc,20,16,f9c7ad2c,...) at sendfile+0xa8
syscall(f9c7ad38) at syscall+0x315
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (393, FreeBSD ELF32, sendfile), eip = 0x28290bff, esp
= 0xbfbfc6ac, ebp = 0xbfbfe718 ---

What is the lock it holds, and where is it acquired?

My bet is on the pfil rwlock - accquired in pfil_run_hooks and
tcbinfo / inp mtxs from tcp_output. Nothing in the transmission path
must use sx locks. I keep on telling that.

I compiled kernel with WITNESS and INVARIANTS as in GENERIC.
After boot, without doing anything (I even set ipnat_enable="NO", i.e.
only ipfw is running, but ipf/ipnat is compiled in). I got panic.
I don't have serial console attached, but the console screen is at

http://www.rafan.org/tmp/pfil_panic.jpg

You can find 'show locks', 'show allpcpu', and 'show lock' for each
lock. Also two trace for thread that are found in 'show allpcpu'.

If necessary, I can setup serial console, but it will take few days.

The problem is understood and has been reported to the ipfilter
maintainer. One sollution might be to use rwlocks instead, but if there
is a path that sleeps with the lock held another sollution might be
required. Most likely the ioctl path (on copyin/copyout) exercises that
problem.

--
/"\ Best regards, | mlaier@xxxxxxxxxxx
\ / Max Laier | ICQ #67774661
X http://pf4freebsd.love2party.net/ | mlaier@EFnet
/ \ ASCII Ribbon Campaign | Against HTML Mail and News

Attachment: signature.asc
Description: This is a digitally signed message part.



Relevant Pages

  • Re: panic: spin lock held too long (RELENG_8 from today)
    ... I'm also getting similar panics on 8.2-STABLE. ... Sleeping thread owns a non-sleepable lock ... doing UFS snapshots and having this problem ...
    (freebsd-stable)
  • Re: panic: sleeping thread owns a non-sleepable lock
    ... Sleeping thread owns a non-sleepable lock ... If necessary, I can setup serial console, but it will take few days. ... One sollution might be to use rwlocks instead, ...
    (freebsd-current)
  • Re: panic: sleeping thread owns a non-sleepable lock
    ... Sleeping thread owns a non-sleepable lock ... Also two trace for thread that are found in 'show allpcpu'. ... If necessary, I can setup serial console, but it will take few days. ...
    (freebsd-current)
  • Re: panic: sleeping thread owns a non-sleepable lock
    ... Sleeping thread owns a non-sleepable lock ... It looks like a whole lot of complex code can be run with pfil rwlock ... More complex code - harder to avoid sleeping. ...
    (freebsd-current)
  • Re: hard lock-up writing to tape
    ... Num lock doesn't come on. ... you will need to set up a serial console with some special ... Enable the serial console and boot the system. ... If you get the db> prompt, then your hang is probably due to a Giant ...
    (freebsd-current)