RE: Ufs dead-locks on freebsd 6.2

I've upgraded to 6-stable, added the kernel options as per the kernel
handbook. After about 5 hours of they system in a deadlock it panic'd.
Here's the backtrace, and show pcpu, show allpcpu, show locks, show
alllocks, show lockedvnods and alltrace.

I will have the system down for approx another 15-20mins if there's
anything else someone would like while I'm in the debugger.


db> bt
Tracing pid 46784 tid 105112 td 0xd44a8000
kdb_enter(c0785f13) at kdb_enter+0x2b
vfs_badlock(c0785f2c,c0786051,ccd47984) at vfs_badlock+0x47
assert_vop_locked(ccd47984,c0786051) at assert_vop_locked+0x4a
vop_lock_post(f9f709dc,0,1002,ccd47984,f9f709f8,...) at
vop_lock_post+0x2a
VOP_LOCK_APV(c07dc2e0,f9f709dc) at VOP_LOCK_APV+0xa0
vn_lock(ccd47984,1002,d44a8000) at vn_lock+0xac
lookup(f9f70c08) at lookup+0xde
namei(f9f70c08) at namei+0x39a
unp_connect(d44b2de8,d44dc380,d44a8000,d44b2de8,25,...) at
unp_connect+0xf0
uipc_connect(d44b2de8,d44dc380,d44a8000) at uipc_connect+0x66
soconnect(d44b2de8,d44dc380,d44a8000) at soconnect+0x4e
kern_connect(d44a8000,7,d44dc380,d44dc380,0,...) at kern_connect+0x74
connect(d44a8000,f9f70d04) at connect+0x2f
syscall(3b,805003b,bfbf003b,bfbfd920,bfbfd922,...) at syscall+0x25b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (98, FreeBSD ELF32, connect), eip = 0x6812d553, esp =
0xbfbfd90c, ebp = 0xbfbfd9a8 ---
db> show pcpu
cpuid = 1
curthread = 0xd44a8000: pid 46784 "cron"
curpcb = 0xf9f70d90
fpcurthread = none
idlethread = 0xccb5c600: pid 10 "idle: cpu1"
APIC ID = 6
currentldt = 0x50
spin locks held:
db> show allpcpu
Current CPU: 1

cpuid = 0
curthread = 0xd4490d80: pid 46779 "cron"
curpcb = 0xf9f13d90
fpcurthread = none
idlethread = 0xccb5c780: pid 11 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
spin locks held:

cpuid = 1
curthread = 0xd44a8000: pid 46784 "cron"
curpcb = 0xf9f70d90
fpcurthread = none
idlethread = 0xccb5c600: pid 10 "idle: cpu1"
APIC ID = 6
currentldt = 0x50
spin locks held:

db> show locks
db> show alllocks
db> show lockedvnods
Locked vnodes

0xd12d16cc: tag ufs, type VREG
usecount 1, writecount 1, refcount 4 mountedhere 0
flags ()
v_object 0xcd64ac60 ref 0 pages 48
lock type ufs: EXCL (count 1) by thread 0xcd6a6180 (pid 22025)#0
0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06015c4 at vn_write+0x138
#5 0xc05c4544 at dofilewrite+0x7c
#6 0xc05c43e3 at kern_writev+0x3b
#7 0xc05c4309 at write+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f

ino 494646, on dev amrd0s1d

0xccf9f2b8: tag ufs, type VDIR
usecount 1, writecount 0, refcount 2495 mountedhere 0
flags (VV_ROOT)
v_object 0xd23c39cc ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xcd23fd80 (pid 40153) with
2493 pending#0 0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc05f5d12 at vget+0xbe
#5 0xc05ed9f9 at vfs_hash_get+0x8d
#6 0xc06b7b8f at ffs_vget+0x27
#7 0xc06c1435 at ufs_root+0x19
#8 0xc05eef1c at lookup+0x7c8
#9 0xc05ee4b2 at namei+0x39a
#10 0xc0600a13 at vn_open_cred+0x5b
#11 0xc06009b6 at vn_open+0x1e
#12 0xc05fa126 at kern_open+0xb6
#13 0xc05fa03a at open+0x1a
#14 0xc0723e2b at syscall+0x25b
#15 0xc070ee0f at Xint0x80_syscall+0x1f

ino 2, on dev amrd0s1d

0xcd030984: tag ufs, type VDIR
usecount 7, writecount 0, refcount 9 mountedhere 0
flags ()
v_object 0xcffb2108 ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xcd9bc780 (pid 29680) with
1 pending#0 0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc05f5d12 at vget+0xbe
#5 0xc05ea48e at cache_lookup+0x34a
#6 0xc05ea9c2 at vfs_cache_lookup+0x92
#7 0xc0737847 at VOP_LOOKUP_APV+0x87
#8 0xc05eebf8 at lookup+0x4a4
#9 0xc05ee4b2 at namei+0x39a
#10 0xc0600a13 at vn_open_cred+0x5b
#11 0xc06009b6 at vn_open+0x1e
#12 0xc05fa126 at kern_open+0xb6
#13 0xc05fa03a at open+0x1a
#14 0xc0723e2b at syscall+0x25b
#15 0xc070ee0f at Xint0x80_syscall+0x1f

ino 494592, on dev amrd0s1d

0xd1bad828: tag ufs, type VREG
usecount 2, writecount 2, refcount 6 mountedhere 0
flags ()
v_object 0xd08b28c4 ref 0 pages 82
lock type ufs: EXCL (count 1) by thread 0xcd5b0780 (pid 36375) with
2 pending#0 0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06015c4 at vn_write+0x138
#5 0xc05c4544 at dofilewrite+0x7c
#6 0xc05c43e3 at kern_writev+0x3b
#7 0xc05c4309 at write+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f

ino 494775, on dev amrd0s1d

0xcfed0984: tag ufs, type VREG
usecount 1, writecount 1, refcount 4 mountedhere 0
flags ()
v_object 0xd0dcf39c ref 0 pages 7
lock type ufs: EXCL (count 1) by thread 0xccfa9000 (pid 653)#0
0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc05fd950 at fsync+0x9c
#5 0xc0723e2b at syscall+0x25b
#6 0xc070ee0f at Xint0x80_syscall+0x1f

ino 188447, on dev amrd0s1d

0xd2e3b828: tag ufs, type VREG
usecount 2, writecount 2, refcount 1612 mountedhere 0
flags ()
v_object 0xd2dee630 ref 0 pages 6436
lock type ufs: SHARED (count 1) with 1 pending#0 0xc0593b80 at
lockmgr+0x160
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06012fe at vn_read+0x132
#5 0xc05c4269 at dofileread+0x85
#6 0xc05c4102 at kern_readv+0x36
#7 0xc05c402d at read+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f

ino 130807917, on dev amrd1s1d

0xd157f570: tag ufs, type VREG
usecount 1, writecount 1, refcount 2713 mountedhere 0
flags ()
v_object 0xd2d5118c ref 0 pages 245644
lock type ufs: EXCL (count 1) by thread 0xcd6a6300 (pid 26383)#0
0xc0593f0d at lockmgr+0x4ed
#1 0xc06b8e0e at ffs_lock+0x76
#2 0xc0739787 at VOP_LOCK_APV+0x87
#3 0xc0601c28 at vn_lock+0xac
#4 0xc06015c4 at vn_write+0x138
#5 0xc05c4544 at dofilewrite+0x7c
#6 0xc05c43e3 at kern_writev+0x3b
#7 0xc05c4309 at write+0x45
#8 0xc0723e2b at syscall+0x25b
#9 0xc070ee0f at Xint0x80_syscall+0x1f

ino 116230408, on dev amrd1s1d
db> alltrace

Tracing command pid 46787 tid 105110 td 0xd44a8300
*** error reading from address 72d76109 ***
db>




-----Original Message-----
From: owner-freebsd-performance@xxxxxxxxxxx
[mailto:owner-freebsd-performance@xxxxxxxxxxx] On Behalf Of
Andrew Edwards
Sent: Wednesday, May 16, 2007 1:16 PM
To: freebsd-performance@xxxxxxxxxxx
Cc: freebsd-fs@xxxxxxxxxxx
Subject: RE: Ufs dead-locks on freebsd 6.2

Here's the backtrace from the last crash along with the
output from show alllocks when the system was deadlocked. I
have been running 6.2-release and compliled with makeoptions
debug=-g, invariants, invariant_support and witness. I will
update to 6-STABLE add diagnositc, debug_locks and
debug_vfs_locks as per the handbook recommendation and retry.

Yes, when the system was un-usable I was still able to ping
it. I have the serial console setup as the default console
so I can remotely access the box and break into the debugger etc.

(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc059b480 in boot (howto=260) at
/usr/src/sys/kern/kern_shutdown.c:409
#2 0xc059b795 in panic (fmt=0xc0787b04 "Most recently used by %s\n")
at /usr/src/sys/kern/kern_shutdown.c:565
#3 0xc06c4a4d in mtrash_ctor (mem=0xce74fa00, size=0, arg=0x0,
flags=258)
at /usr/src/sys/vm/uma_dbg.c:137
#4 0xc06c2c07 in uma_zalloc_arg (zone=0xc10615a0, udata=0x0,
flags=258)
at /usr/src/sys/vm/uma_core.c:1850
#5 0xc0591416 in malloc (size=272, mtp=0xc07c32c0, flags=258) at
uma.h:275
#6 0xc05edfab in __mnt_vnode_first (mvp=0xf3741c48, mp=0xcaa14cf8)
at /usr/src/sys/kern/vfs_mount.c:1813
#7 0xc05f2467 in vfs_msync (mp=0xcaa14cf8, flags=2)
at /usr/src/sys/kern/vfs_subr.c:2874
#8 0xc05f2bbd in sync_fsync (ap=0x0) at
/usr/src/sys/kern/vfs_subr.c:3119
#9 0xc072f4ee in VOP_FSYNC_APV (vop=0x0, a=0xf3741cbc) at
vnode_if.c:1020 #10 0xc05f097c in sync_vnode (bo=0xca854e90,
td=0xca435000) at
vnode_if.h:537
#11 0xc05f0bf1 in sched_sync () at /usr/src/sys/kern/vfs_subr.c:1698
#12 0xc0587248 in fork_exit (callout=0xc05f0a04 <sched_sync>, arg=0x0,
frame=0xf3741d38) at /usr/src/sys/kern/kern_fork.c:821
#13 0xc070712c in fork_trampoline () at
/usr/src/sys/i386/i386/exception.s:208


db> show alllocks
Process 36596 (sshd) thread 0xd1238c00 (102406) exclusive
sleep mutex vm object (standard object) r = 0 (0xce2c87bc)
locked @ /usr/src/sys/vm/vm_object.c:446 exclusive sx user
map r = 0 (0xd128060c) locked @
/usr/src/sys/vm/vm_map.c:307
Process 887 (sshd) thread 0xca7d2000 (100056) exclusive sleep
mutex vm object (standard object) r = 0 (0xcb713ad4) locked @
/usr/src/sys/vm/vm_fault.c:297 exclusive sx user map r = 0
(0xcaae4734) locked @
/usr/src/sys/vm/vm_map.c:3074
db> show lockedvnods
Locked vnodes

0xcaa78660: tag ufs, type VREG
usecount 2, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xc1046738 ref 0 pages 1596
lock type ufs: EXCL (count 1) by thread 0xca689c00 (pid
536) with 1 pending
ino 494620, on dev amrd0s1d

0xcaa86110: tag ufs, type VREG
usecount 1, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xca85f738 ref 0 pages 44
lock type ufs: EXCL (count 1) by thread 0xca7d2780 (pid 715)
ino 494633, on dev amrd0s1d

0xcabe4110: tag ufs, type VDIR
usecount 12, writecount 0, refcount 14 mountedhere 0
flags ()
v_object 0xcab28840 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xca9bbd80 (pid
14253) with
3 pending
ino 423947, on dev amrd0s1d

0xcb437990: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb3c3dec ref 0 pages 4100
lock type ufs: EXCL (count 1) by thread 0xcaffac00 (pid 20868)
ino 282640, on dev amrd0s1d

0xcb99e550: tag ufs, type VDIR
usecount 2, writecount 0, refcount 4 mountedhere 0
flags ()
v_object 0xcef979cc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xca9bb600 (pid
881) with 1 pending
ino 423987, on dev amrd0s1d

0xcfc97dd0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcd4975ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb748900 (pid 2518)
ino 424275, on dev amrd0s1d

0xccad9aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xcf0c539c ref 0 pages 5
lock type ufs: EXCL (count 1) by thread 0xca7d1c00 (pid 600)
ino 188446, on dev amrd0s1d

0xccb0f110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb4609cc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf480 (pid 11054)
ino 424100, on dev amrd0s1d

0xcc501bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 3 mountedhere 0
flags ()
v_object 0xcc7d19cc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb76d600 (pid
13743) with
1 pending
ino 424279, on dev amrd0s1d

0xcf96b220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf135c60 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf900 (pid 29458)
ino 424374, on dev amrd0s1d

0xcc5bbbb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcdd5b318 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaad9900 (pid 50782)
ino 424276, on dev amrd0s1d

0xcec1d000: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcd3d7108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb76dc00 (pid 59514)
ino 424500, on dev amrd0s1d

0xcebe5110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xccee95ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb650780 (pid 59975)
ino 424509, on dev amrd0s1d

0xce0c1880: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xca8b64a4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb768a80 (pid 69466)
ino 424555, on dev amrd0s1d

0xcf652110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf4a318c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8600 (pid 75577)
ino 424579, on dev amrd0s1d

0xce282550: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf261318 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0235a80 (pid 81734)
ino 424927, on dev amrd0s1d

0xcc1d4dd0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcd4a6630 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaccb900 (pid 81772)
ino 424928, on dev amrd0s1d

0xcb820bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb251ad4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadaf480 (pid 84037)
ino 424935, on dev amrd0s1d

0xced5aaa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb784210 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0236000 (pid 202)
ino 425039, on dev amrd0s1d

0xcbe45220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce55de70 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaad9a80 (pid 230)
ino 425043, on dev amrd0s1d

0xcc098220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce4a9dec ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbfd80 (pid 9902)
ino 425093, on dev amrd0s1d

0xcd585110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf8c1e70 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaccba80 (pid 24017)
ino 425144, on dev amrd0s1d

0xceeac000: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb1225ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff5a80 (pid 24775)
ino 425149, on dev amrd0s1d

0xcc549aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcab2d318 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8a80 (pid 42358)
ino 425227, on dev amrd0s1d

0xcc6f7000: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfd4139c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf600 (pid 43117)
ino 425230, on dev amrd0s1d

0xccc44bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfd18d68 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadafd80 (pid 42859)
ino 425234, on dev amrd0s1d

0xcc7a7220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfedf420 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb76c600 (pid 48968)
ino 425264, on dev amrd0s1d

0xcc693aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf92f738 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb655300 (pid 55381)
ino 425286, on dev amrd0s1d

0xcbabf220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfe297bc ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0234a80 (pid 63802)
ino 425322, on dev amrd0s1d

0xcd760220: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcb11ac60 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadb0480 (pid 69938)
ino 425348, on dev amrd0s1d

0xcc044990: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcaaff084 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadeda80 (pid 70418)
ino 425360, on dev amrd0s1d

0xcc190660: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfed9108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xca7d2900 (pid 76803)
ino 425378, on dev amrd0s1d

0xcc676330: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf8c14a4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8900 (pid 76841)
ino 425384, on dev amrd0s1d

0xcf0ad110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce53b5ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadb0900 (pid 79849)
ino 425394, on dev amrd0s1d

0xce4f6aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce3cc4a4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb767300 (pid 79620)
ino 425402, on dev amrd0s1d

0xce80d110: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf502108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff5d80 (pid 98225)
ino 425478, on dev amrd0s1d

0xcd218990: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf08e18c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff5000 (pid 98241)
ino 425482, on dev amrd0s1d

0xcbcb3440: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf8a8738 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaff8180 (pid 1341)
ino 425505, on dev amrd0s1d

0xcf6fe440: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf88f39c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcaad7900 (pid 4512)
ino 425512, on dev amrd0s1d

0xcdd07aa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf7e9a50 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadaf600 (pid 4464)
ino 425513, on dev amrd0s1d

0xcc18eaa0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xce2fa108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcafbf300 (pid 13669)
ino 425549, on dev amrd0s1d

0xcb9e1440: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcfbea39c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb64e480 (pid 13656)
ino 425555, on dev amrd0s1d

0xcb8a8bb0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf6565ac ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcadb0180 (pid 22845)
ino 425596, on dev amrd0s1d

0xccc47660: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcf551b58 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xcb64e180 (pid 22870)
ino 425597, on dev amrd0s1d

0xcec8ecc0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2 mountedhere 0
flags ()
v_object 0xcab61948 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xd0234600 (pid 32036)
ino 425633, on dev amrd0s1d

0xcdca1bb0: tag ufs, type VREG
usecount 2, writecount 2, refcount 888 mountedhere 0
flags ()
v_object 0xcaf23294 ref 0 pages 41308
lock type ufs: EXCL (count 1) by thread 0xcadb0a80 (pid 5541)
ino 130855246, on dev amrd1s1d

-----Original Message-----
From: Kris Kennaway [mailto:kris@xxxxxxxxxxxxxx]
Sent: Wednesday, May 16, 2007 12:33 PM
To: Andrew Edwards
Cc: freebsd-performance@xxxxxxxxxxx
Subject: Re: Ufs dead-locks on freebsd 6.2

On Wed, May 16, 2007 at 12:08:24PM -0400, Andrew Edwards wrote:
I have a system running a dual intel zeon 2.8Ghz with 4G
of ram and
using an intel raid controller model SRCU42X which uses the
amr driver.
I have had this server running 5.4 upgraded to 6.2 and
was running
fine for several months and then after a normal reboot
I've started
having all sorts of problems with what appears to be
dead-locks in the
filesystem. This server is my backup server and I rsync
files from
various servers onto this one fairly non-stop. If I stop
the rsync's
the system appears to be stable although I did have a
kernel core just
last night.

When I have been able to observe the problem I ususally see one
filesystem become inaccessible, perhaps var but I'm not
sure, and then
in a short period of time the whole system is inaccessible.
Usually
if I startup just one of the rsync's within a couple of hours the
system will be un-usable.

I did find this thread which seems to describe similar
issues but this
is a different driver.

http://lists.freebsd.org/pipermail/freebsd-questions/2006-Augu
st/127835.
html

Probably not relevant then. Deadlocks come in many varieties, all
different.

Currently I'm running with debug.mpsafevfs=0,
debug.mpsafenet=1and
debug.mpsafevm=0 but this doesn't seem to help.

On perahps a related issue I have two other nearly
identical systems
which were going to be upgrading to 6.2 as on 5.4 I am
experiencing
deadlocks and when I hit ctrl-t I see the system is
either stuck in
ufs or zoneinfo and I have not found very much information
about zoneinfo.

Does anyone have any suggestions on what I can look for or other
tuning options or had similar experiences?

See the chapter on kernel debugging in the developers
handbook for the
information you need to provide before we can begin to debug your
problem.

Kris


_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to
"freebsd-performance-unsubscribe@xxxxxxxxxxx"

_______________________________________________
freebsd-performance@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@xxxxxxxxxxx"

Relevant Pages