Re: panic: Lock so_rcv_sx not exclusively locked

On Mon, Jan 28, 2008 at 08:49:41PM +0000, Robert Watson wrote:
On Mon, 28 Jan 2008, Jos Backus wrote:

Recent -current kernels can reliably be made to panic by killing
npviewer.bin. Right before killing, npviewer.bin shows up like this in
top:

6288 jos 1 45 5 62844K 36788K so_rcv 0:08 0.00% npviewer.bin

Is this a multithreaded app? If so, could you look and see if you can find
the other threads in npviewer.bin in the kernel (info thread, etc) and get
stack traces for those threads as well?

It's a multithreaded Linux app, installed through ports; see the attached
file. Please let me know if you'd like me to gather more information.

Thanks!

--
Jos Backus
jos at catnook.com
lizzy:~# pkg_which /usr/local/lib/nspluginwrapper/i386/linux/npviewer.bin
nspluginwrapper-0.9.91.5
lizzy:~% ldd /usr/local/lib/nspluginwrapper/i386/linux/npviewer.bin
/usr/local/lib/nspluginwrapper/i386/linux/npviewer.bin:
libgtk-x11-2.0.so.0 => /usr/lib/libgtk-x11-2.0.so.0 (0x2807e000)
libgdk-x11-2.0.so.0 => /usr/lib/libgdk-x11-2.0.so.0 (0x28376000)
libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x283fa000)
libdl.so.2 => /lib/libdl.so.2 (0x28436000)
libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x2843a000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x284c3000)
libXt.so.6 => /usr/X11R6/lib/libXt.so.6 (0x28596000)
libpthread.so.0 => /lib/obsolete/linuxthreads/libpthread.so.0 (0x285ec000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x2863f000)
libc.so.6 => /lib/obsolete/linuxthreads/libc.so.6 (0x28649000)
libgdk_pixbuf-2.0.so.0 => /usr/lib/libgdk_pixbuf-2.0.so.0 (0x28768000)
libpangoxft-1.0.so.0 => /usr/lib/libpangoxft-1.0.so.0 (0x2877e000)
libpangox-1.0.so.0 => /usr/lib/libpangox-1.0.so.0 (0x28785000)
libpango-1.0.so.0 => /usr/lib/libpango-1.0.so.0 (0x28791000)
libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0 (0x287cc000)
libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x287e3000)
libm.so.6 => /lib/obsolete/linuxthreads/libm.so.6 (0x287e7000)
libXrandr.so.2 => /usr/X11R6/lib/libXrandr.so.2 (0x2880d000)
libXi.so.6 => /usr/X11R6/lib/libXi.so.6 (0x28812000)
libXinerama.so.1 => /usr/X11R6/lib/libXinerama.so.1 (0x2881a000)
libXft.so.2 => /usr/X11R6/lib/libXft.so.2 (0x2881d000)
libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x28830000)
libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x28898000)
libXfixes.so.3 => /usr/X11R6/lib/libXfixes.so.3 (0x288c2000)
libXcursor.so.1 => /usr/X11R6/lib/libXcursor.so.1 (0x288c7000)
libXrender.so.1 => /usr/X11R6/lib/libXrender.so.1 (0x288d2000)
libXext.so.6 => /usr/X11R6/lib/libXext.so.6 (0x288da000)
/lib/ld-linux.so.2 (0x2805f000)
libSM.so.6 => /usr/X11R6/lib/libSM.so.6 (0x288e9000)
libICE.so.6 => /usr/X11R6/lib/libICE.so.6 (0x288f2000)
libpangoft2-1.0.so.0 => /usr/lib/libpangoft2-1.0.so.0 (0x2890c000)
libexpat.so.0 => /usr/lib/libexpat.so.0 (0x28934000)
libz.so.1 => /usr/lib/libz.so.1 (0x28953000)
lizzy:~%

191 Thread 100197 (PID=3595: npviewer.bin) sched_switch (td=0xc5092440, newtd=Variable "newtd" is not available.
190 Thread 100201 (PID=3594: npviewer.bin) sched_switch (td=0xc5092aa0, newtd=Variable "newtd" is not available.
189 Thread 100200 (PID=3592: npviewer.bin) sched_switch (td=0xc5092880, newtd=Variable "newtd" is not available.
188 Thread 100183 (PID=3591: npviewer.bin) sched_switch (td=0xc5028440, newtd=Variable "newtd" is not available.
187 Thread 100163 (PID=3588: npviewer.bin) sched_switch (td=0xc4faf880, newtd=Variable "newtd" is not available.
* 186 Thread 100134 (PID=3561: npviewer.bin) doadump () at pcpu.h:195

(kgdb) thread
[Current thread is 186 (Thread 100134)]
(kgdb) bt
#0 doadump () at pcpu.h:195
#1 0xc05435ef in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:417
#2 0xc0543867 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:571
#3 0xc054a427 in _sx_assert (sx=0xc4fb89a0, what=4, file=0xc06d49ca "/usr/src/sys/kern/uipc_sockbuf.c", line=157) at /usr/src/sys/kern/kern_sx.c:931
#4 0xc054a8da in _sx_xunlock (sx=0xc4fb89a0, file=0xc06d49ca "/usr/src/sys/kern/uipc_sockbuf.c", line=157) at /usr/src/sys/kern/kern_sx.c:316
#5 0xc0595444 in sbunlock (sb=0xc4fb8968) at /usr/src/sys/kern/uipc_sockbuf.c:157
#6 0xc0598245 in sorflush (so=0xc4fb8918) at /usr/src/sys/kern/uipc_socket.c:1902
#7 0xc05982df in soshutdown (so=0xc4fb8918, how=2) at /usr/src/sys/kern/uipc_socket.c:1866
#8 0xc059ce86 in shutdown (td=0xc4b31880, uap=0xe7219c58) at /usr/src/sys/kern/uipc_syscalls.c:1248
#9 0xc07ffa80 in ?? ()
#10 0xc4b31880 in ?? ()
#11 0xe7219c58 in ?? ()
#12 0x00000008 in ?? ()
#13 0x00000000 in ?? ()
#14 0x00000000 in ?? ()
#15 0xc103d800 in ?? ()
#16 0x00000c1e in ?? ()
#17 0x00000000 in ?? ()
#18 0xc4e25488 in ?? ()
#19 0xc5007e9c in ?? ()
#20 0x00000000 in ?? ()
#21 0x00000000 in ?? ()
#22 0xe7219c72 in ?? ()
#23 0x00000004 in ?? ()
#24 0x80000000 in ?? ()
#25 0xe7219c8c in ?? ()
#26 0xc06a1a44 in __qdivrem (uq=Unhandled dwarf expression opcode 0x93
) at /usr/src/sys/libkern/qdivrem.c:186
#27 0xc0697b93 in syscall (frame=0xe7219d38) at /usr/src/sys/i386/i386/trap.c:1034
#28 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#29 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread 187
[Switching to thread 187 (Thread 100163)]
#0 sched_switch (td=0xc4faf880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc4faf880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc46caaa8) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc46caaa8) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572a77 in sleepq_timedwait_sig (wchan=0xc46caaa8) at /usr/src/sys/kern/subr_sleepqueue.c:612
#5 0xc050ffc8 in _cv_timedwait_sig (cvp=0xc46caaa8, lock=0xc46caa90, timo=2001) at /usr/src/sys/kern/kern_condvar.c:369
#6 0xc05787ca in seltdwait (td=Variable "td" is not available.
) at /usr/src/sys/kern/sys_generic.c:1416
#7 0xc0578d7a in poll (td=0xc4faf880, uap=0xe728ccfc) at /usr/src/sys/kern/sys_generic.c:1060
#8 0xc0697b93 in syscall (frame=0xe728cd38) at /usr/src/sys/i386/i386/trap.c:1034
#9 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#10 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread 188
[Switching to thread 188 (Thread 100183)]
#0 sched_switch (td=0xc5028440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5028440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572a77 in sleepq_timedwait_sig (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:612
#5 0xc054bb91 in _sleep (ident=0xc073b664, lock=0x0, priority=348, wmesg=0xc06ce3dc "nanslp", timo=21) at /usr/src/sys/kern/kern_synch.c:222
#6 0xc0552891 in kern_nanosleep (td=0xc5028440, rqt=0xe72ddc5c, rmt=0x0) at /usr/src/sys/kern/kern_time.c:379
#7 0xc0802dd3 in ?? ()
#8 0xc5028440 in ?? ()
#9 0xe72ddc5c in ?? ()
#10 0x00000000 in ?? ()
#11 0xe72ddcfc in ?? ()
#12 0xbefffa28 in ?? ()
#13 0x00000000 in ?? ()
#14 0x01312d00 in ?? ()
#15 0xc5028440 in ?? ()
#16 0x0020dd38 in ?? ()
#17 0x00000000 in ?? ()
#18 0x01312d00 in ?? ()
#19 0xbefff9f0 in ?? ()
#20 0xc5028440 in ?? ()
#21 0xc4fa0804 in ?? ()
#22 0xe72ddd2c in ?? ()
#23 0xc0697b93 in syscall (frame=0xc5028440) at /usr/src/sys/i386/i386/trap.c:1034
Previous frame identical to this frame (corrupt stack?)
(kgdb) thread 189
[Switching to thread 189 (Thread 100200)]
#0 sched_switch (td=0xc5092880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5092880, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc4fb89d0) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc4fb89d0) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572b84 in sleepq_wait_sig (wchan=0xc4fb89d0) at /usr/src/sys/kern/subr_sleepqueue.c:575
#5 0xc054bbb9 in _sleep (ident=0xc4fb89d0, lock=0xc4fb8988, priority=344, wmesg=0xc06d4a16 "sbwait", timo=0) at /usr/src/sys/kern/kern_synch.c:226
#6 0xc0595556 in sbwait (sb=0xc4fb8968) at /usr/src/sys/kern/uipc_sockbuf.c:131
#7 0xc059a69f in soreceive_generic (so=0xc4fb8918, psa=0xe7313b34, uio=0xe7313b40, mp0=0x0, controlp=0x0, flagsp=0xe7313bc4)
at /usr/src/sys/kern/uipc_socket.c:1485
#8 0xc059670d in soreceive (so=0xc4fb8918, psa=0xe7313b34, uio=0xe7313b40, mp0=0x0, controlp=0x0, flagsp=0xe7313bc4)
at /usr/src/sys/kern/uipc_socket.c:1853
#9 0xc059bdfd in kern_recvit (td=0xc5092880, s=63, mp=0xe7313bac, fromseg=UIO_USERSPACE, controlp=0x0) at /usr/src/sys/kern/uipc_syscalls.c:968
#10 0xc059bfe1 in recvit (td=Variable "td" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:1075
#11 0xc059c156 in recvfrom (td=0xc5092880, uap=0xe7313c28) at /usr/src/sys/kern/uipc_syscalls.c:1119
#12 0xc07ff8ad in ?? ()
#13 0xc5092880 in ?? ()
#14 0xe7313c28 in ?? ()
#15 0x00000010 in ?? ()
#16 0x00000000 in ?? ()
#17 0x00000000 in ?? ()
#18 0xc5055680 in ?? ()
#19 0x0000001e in ?? ()
#20 0x00000000 in ?? ()
#21 0xc4e25488 in ?? ()
#22 0xc730b2a8 in ?? ()
#23 0x00000000 in ?? ()
#24 0x00000000 in ?? ()
#25 0x00000000 in ?? ()
#26 0xba7b9cd6 in ?? ()
#27 0xe7313c14 in ?? ()
#28 0xc068cd18 in spinlock_enter () at /usr/src/sys/i386/i386/machdep.c:2380
#29 0xc0697b93 in syscall (frame=0xe7313d38) at /usr/src/sys/i386/i386/trap.c:1034
#30 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#31 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) thread 190
[Switching to thread 190 (Thread 100201)]
#0 sched_switch (td=0xc5092aa0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5092aa0, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572a77 in sleepq_timedwait_sig (wchan=0xc073b664) at /usr/src/sys/kern/subr_sleepqueue.c:612
#5 0xc054bb91 in _sleep (ident=0xc073b664, lock=0x0, priority=348, wmesg=0xc06ce3dc "nanslp", timo=21) at /usr/src/sys/kern/kern_synch.c:222
#6 0xc0552891 in kern_nanosleep (td=0xc5092aa0, rqt=0xe7316c5c, rmt=0x0) at /usr/src/sys/kern/kern_time.c:379
#7 0xc0802dd3 in ?? ()
#8 0xc5092aa0 in ?? ()
#9 0xe7316c5c in ?? ()
#10 0x00000000 in ?? ()
#11 0xe7316cfc in ?? ()
#12 0xbe9ffa28 in ?? ()
#13 0x00000000 in ?? ()
#14 0x01312d00 in ?? ()
#15 0xc5092aa0 in ?? ()
#16 0x40242aa0 in ?? ()
#17 0x00000000 in ?? ()
#18 0x01312d00 in ?? ()
#19 0xbe9ff9f0 in ?? ()
#20 0xc5092aa0 in ?? ()
#21 0xc505b2ac in ?? ()
#22 0xe7316d2c in ?? ()
#23 0xc0697b93 in syscall (frame=0xc5092aa0) at /usr/src/sys/i386/i386/trap.c:1034
Previous frame identical to this frame (corrupt stack?)
(kgdb) thread 191
[Switching to thread 191 (Thread 100197)]
#0 sched_switch (td=0xc5092440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
1905 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xc5092440, newtd=Variable "newtd" is not available.
) at /usr/src/sys/kern/sched_ule.c:1905
#1 0xc054b6d3 in mi_switch (flags=Variable "flags" is not available.
) at /usr/src/sys/kern/kern_synch.c:444
#2 0xc05720a0 in sleepq_switch (wchan=0xc44bd84c) at /usr/src/sys/kern/subr_sleepqueue.c:478
#3 0xc05722ee in sleepq_catch_signals (wchan=0xc44bd84c) at /usr/src/sys/kern/subr_sleepqueue.c:414
#4 0xc0572b84 in sleepq_wait_sig (wchan=0xc44bd84c) at /usr/src/sys/kern/subr_sleepqueue.c:575
#5 0xc054bbb9 in _sleep (ident=0xc44bd84c, lock=0xc44bd804, priority=344, wmesg=0xc06d4a16 "sbwait", timo=0) at /usr/src/sys/kern/kern_synch.c:226
#6 0xc0595556 in sbwait (sb=0xc44bd7e4) at /usr/src/sys/kern/uipc_sockbuf.c:131
#7 0xc059a69f in soreceive_generic (so=0xc44bd794, psa=0xe730ab34, uio=0xe730ab40, mp0=0x0, controlp=0x0, flagsp=0xe730abc4)
at /usr/src/sys/kern/uipc_socket.c:1485
#8 0xc059670d in soreceive (so=0xc44bd794, psa=0xe730ab34, uio=0xe730ab40, mp0=0x0, controlp=0x0, flagsp=0xe730abc4)
at /usr/src/sys/kern/uipc_socket.c:1853
#9 0xc059bdfd in kern_recvit (td=0xc5092440, s=65, mp=0xe730abac, fromseg=UIO_USERSPACE, controlp=0x0) at /usr/src/sys/kern/uipc_syscalls.c:968
#10 0xc059bfe1 in recvit (td=Variable "td" is not available.
) at /usr/src/sys/kern/uipc_syscalls.c:1075
#11 0xc059c156 in recvfrom (td=0xc5092440, uap=0xe730ac28) at /usr/src/sys/kern/uipc_syscalls.c:1119
#12 0xc07ff8ad in ?? ()
#13 0xc5092440 in ?? ()
#14 0xe730ac28 in ?? ()
#15 0x00000010 in ?? ()
#16 0x00000000 in ?? ()
#17 0x00000000 in ?? ()
#18 0xc5055680 in ?? ()
#19 0x00000034 in ?? ()
#20 0x00000000 in ?? ()
#21 0xc4e25488 in ?? ()
#22 0xc730b2a8 in ?? ()
#23 0x00000000 in ?? ()
#24 0x00000000 in ?? ()
#25 0x00000000 in ?? ()
#26 0xdd313180 in ?? ()
#27 0xe730ac14 in ?? ()
#28 0xc068cd18 in spinlock_enter () at /usr/src/sys/i386/i386/machdep.c:2380
#29 0xc0697b93 in syscall (frame=0xe730ad38) at /usr/src/sys/i386/i386/trap.c:1034
#30 0xc06818b0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:203
#31 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) q
_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages