[RELENG_6] NFS panic on locking against myself

From: Xin LI (delphij_at_frontfree.net)
Date: 10/28/05

  • Next message: Julian Elischer: "Re: Timers and timing, was: MySQL Performance 6.0rc1"
    Date: Sat, 29 Oct 2005 00:53:14 +0800
    To: freebsd-current@FreeBSD.org
    
    
    

    Hi,

    On a production CVS server of ours we got panics when there is some wrong
    data was injected to the NFS TCP connection. This may indicate some error
    in our error handling code of NFS client.

    However, the issue happens only when the gateway between the CVS server and
    the NFS server is heavily loaded, therefore reproducing the issue is somewhat
    hard. I have enabled DEBUG_VFS_LOCK to see if I can catch something.

    The backtrace goes here:

    GNU gdb 6.1.1 [FreeBSD]
    Copyright 2004 Free Software Foundation, Inc.
    GDB is free software, covered by the GNU General Public License, and you are
    welcome to change it and/or distribute copies of it under certain conditions.
    Type "show copying" to see the conditions.
    There is absolutely no warranty for GDB. Type "show warranty" for details.
    This GDB was configured as "i386-marcel-freebsd".

    Unread portion of the kernel message buffer:
    <3>impossible packet length (745074944) from nfs server 10.88.15.238:/data0/vhost/wiki/vol/APPLE/matrixdata/docroot
    panic: lockmgr: locking against myself
    KDB: enter: panic
    Dumping 1022 MB (2 chunks)
      chunk 0: 1MB (159 pages) ... ok
      chunk 1: 1022MB (261600 pages) 1006 990 974 958 942 926 910 894 878 862 846 830 814 798 782 766 750 734 718 702 686 670 654 638 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46 30 14

    #0 doadump () at pcpu.h:165
            in pcpu.h
    (kgdb) bt full
    #0 doadump () at pcpu.h:165
    No locals.
    #1 0xc047f373 in db_fncall (dummy1=-1066385920, dummy2=0, dummy3=-1067193049,
        dummy4=0xe775d7a0 "Ì×uç\224ÔcÀ¸×uç¼×uç\220\a") at /usr/src/sys/ddb/db_command.c:492
            fn_addr = -1068348316
            args = {1, 0, 544593784, -1067199340, -1066463456, -1066463680, 0, -411707512, 2, -1066737952}
            nargs = 0
            retval = 0
            t = 0
    #2 0xc047f178 in db_command (last_cmdp=0xc06dc4c4, cmd_table=0x0, aux_cmd_tablep=0xc06a83f4,
        aux_cmd_tablep_end=0xc06a8410) at /usr/src/sys/ddb/db_command.c:350
            cmd = (struct command *) 0xc06ae080
            t = 0
            modif = "Ì×uç\224ÔcÀ¸×uç¼×uç\220\a\000\000\220\a\000\000Ï\a\000\000\000\000\000\000\000>pÀ\r\000\000\000\000>pÀ\000>pÀ\r\000\000\000\001\000\000\000ø×uçOÎcÀø×uçhÎcÀ@\016oÀ`rnÀx\000\000\000ÀÍmÀ\000\000\000\000\030Øuçð\021HÀ\000$iÀà\016HÀ\000\000\000\000ÀÍmÀ\222\006H?
            addr = -1066385920
            count = -1067193049
            have_addr = 0
            result = 0
    #3 0xc047f240 in db_command_loop () at /usr/src/sys/ddb/db_command.c:458
    No locals.
    #4 0xc0480e4d in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:221
            jb = {{_jb = {-411707304, -411707324, -411707252, -1006365520, 0, -1069019674, -1068274507, -1066851157,
          -1066845781, -1066851596, -411707248, -1068273655}}}
            prev_jb = (void *) 0x0
            bkpt = 0
    #5 0xc053e2af in kdb_trap (type=3, code=0, tf=0xe775d8e0) at /usr/src/sys/kern/subr_kdb.c:473
            handled = -411707168
    #6 0xc0659578 in trap (frame=
          {tf_fs = -411762680, tf_es = -1068302296, tf_ds = -1066860504, tf_edi = 1, tf_esi = -1066857605, tf_ebp = -411707104, tf_isp = -411707124, tf_ebx = -411707060, tf_edx = 0, tf_ecx = -1061072896, tf_eax = 18, tf_trapno = 3, tf_err = 0, tf_eip = -1068244941, tf_cs = 32, tf_eflags = 658, tf_esp = -411707072, tf_ss = -1068346465})
        at /usr/src/sys/i386/i386/trap.c:591
            td = (struct thread *) 0xc40414b0
            p = (struct proc *) 0xc4044418
            sticks = 17104896
            i = 0
            ucode = 0
            type = 3
            code = 0
            eva = 0
    #7 0xc06498aa in calltrap () at /usr/src/sys/i386/i386/exception.s:139
    No locals.
    #8 0xc053e033 in kdb_enter (msg=0x12 <Address 0x12 out of bounds>) at cpufunc.h:60
    No locals.
    #9 0xc052539f in panic (fmt=0xc0690b7b "lockmgr: locking against myself") at /usr/src/sys/kern/kern_shutdown.c:539
            td = (struct thread *) 0xc40414b0
            bootopt = 256
            newpanic = 1
            ap = 0xe775d94c "°\024\004?
            buf = "lockmgr: locking against myself", '\0' <repeats 224 times>
    #10 0xc0518966 in lockmgr (lkp=0xc2d109e8, flags=8194, interlkp=0x80, td=0xc40414b0) at /usr/src/sys/kern/kern_lock.c:330
            error = 0
            thr = (struct thread *) 0xc40414b0
            extflags = 128
            lockflags = 18
    #11 0xc0573246 in vop_stdlock (ap=0x0) at /usr/src/sys/kern/vfs_default.c:258
            vp = (struct vnode *) 0xc0c15000
    #12 0xc0669583 in VOP_LOCK_APV (vop=0xc06c2c60, a=0xe775d9b0) at vnode_if.c:1642
            rc = -1066652576
    #13 0xc0587e78 in vn_lock (vp=0xc2d10990, flags=8194, td=0xc40414b0) at vnode_if.h:844
            error = 18
    #14 0xc057be9a in vrele (vp=0xc2d10990) at /usr/src/sys/kern/vfs_subr.c:2050
            td = (struct thread *) 0xc40414b0
    #15 0xc05cbe2c in nfs_lookup (ap=0x12) at /usr/src/sys/nfsclient/nfs_vnops.c:893
            cnp = (struct componentname *) 0xe775dc90
            dvp = (struct vnode *) 0xc43ab110
            vpp = (struct vnode **) 0xe775dc7c
            flags = 16814096
            newvp = (struct vnode *) 0xc2d10990
            bpos = 0xc511d150 "h"
            dpos = 0xc44e0ab0 ""
            mreq = (struct mbuf *) 0xc511d100
            mrep = (struct mbuf *) 0x0
            md = (struct mbuf *) 0xc44e0a00
            mb = (struct mbuf *) 0xc511d100
            len = 72
            fhp = (nfsfh_t *) 0xc44e0a38
            np = (struct nfsnode *) 0xc44ee564
            error = 72
            attrflag = 0
            fhsize = 28
            v3 = 512
            td = (struct thread *) 0xc40414b0
    #16 0xc06689a7 in VOP_LOOKUP_APV (vop=0xc06c8820, a=0xe775db3c) at vnode_if.c:99
            rc = -1066629088
    #17 0xc0575389 in lookup (ndp=0xe775dc68) at vnode_if.h:56
            cp = 0xc2a2805b ""
            dp = (struct vnode *) 0xc43ab110
            tdp = (struct vnode *) 0xc2290bb0
            mp = (struct mount *) 0xc2a2805b
            docache = 0
            wantparent = 16
            rdonly = 0
            trailing_slash = 0
            error = 0
            dpunlocked = 0
            cnp = (struct componentname *) 0xe775dc90
            td = (struct thread *) 0xc40414b0
            vfslocked = 1
            tvfslocked = 1
    #18 0xc0574cca in namei (ndp=0xe775dc68) at /usr/src/sys/kern/vfs_lookup.c:203
            fdp = (struct filedesc *) 0xc2ba2000
            cp = 0xc2ba2000 "d ºÂ?ºÂ ÂiÂPe\035ÂPe\035Â\024"
            dp = (struct vnode *) 0xc21d6550
            aiov = {iov_base = 0xc0582a7b, iov_len = 8194}
            auio = {uio_iov = 0xe775dbb4, uio_iovcnt = 128, uio_offset = -4322306996204929024, uio_resid = 0,
      uio_segflg = 3228314720, uio_rw = 3883260924, uio_td = 0x4}
            error = -1038260912
            linklen = -1038260912
            cnp = (struct componentname *) 0xe775dc90
            td = (struct thread *) 0xc40414b0
            p = (struct proc *) 0x0
            vfslocked = 0
    #19 0xc0583d90 in kern_rename (td=0xc40414b0, from=0x12 <Address 0x12 out of bounds>,
        to=0x12 <Address 0x12 out of bounds>, pathseg=UIO_USERSPACE) at /usr/src/sys/kern/vfs_syscalls.c:3188
            mp = (struct mount *) 0x0
            tvp = (struct vnode *) 0x2002
            fvp = (struct vnode *) 0x0
            tdvp = (struct vnode *) 0x0
            fromnd = {ni_dirp = 0x82435dc <Address 0x82435dc out of bounds>, ni_segflg = UIO_USERSPACE, ni_startdir = 0x0,
      ni_rootdir = 0xc21d6550, ni_topdir = 0xc21d6550, ni_vp = 0x0, ni_dvp = 0xc43ab110, ni_pathlen = 1,
      ni_next = 0xc2a2805b "", ni_loopcnt = 0, ni_cnd = {cn_nameiop = 2, cn_flags = 16814096, cn_thread = 0xc40414b0,
        cn_cred = 0xc2757680, cn_lkflags = 2,
        cn_pnbuf = 0xc2a28000 "/usr/local/share/docroot/bkup/cvs/mailtech/FooApp1/myapp10/stuff/nconf/#cvs.cvsup-2172.6595",
        cn_nameptr = 0xc2a28047 "#cvs.cvsup-2172.6595", cn_namelen = 20, cn_consume = 0}}
            tond = {ni_dirp = 0xc057c1f2 "\203Ä\004d\213\025", ni_segflg = 3268479376, ni_startdir = 0xc2d10990,
      ni_rootdir = 0xe775dc48, ni_topdir = 0xc057bf36, ni_vp = 0xc2d10990, ni_dvp = 0xc06d7940, ni_pathlen = 3268479376,
      ni_next = 0x0, ni_loopcnt = 3288601776, ni_cnd = {cn_nameiop = 1, cn_flags = 0, cn_thread = 0xe775dcc4,
        cn_cred = 0xc0582b05, cn_lkflags = -1026487920, cn_pnbuf = 0xc40414b0 "\030D\004Äà\004N?,
        cn_nameptr = 0xc2d10990 "\001", cn_namelen = 493, cn_consume = -411706264}}
            tvfslocked = -411706372
            fvfslocked = -1067018852
            error = -1006353384
    #20 0xc0583d49 in rename (td=0xc40414b0, uap=0x12) at /usr/src/sys/kern/vfs_syscalls.c:3167
    No locals.
    #21 0xc0659dcb in syscall (frame=
          {tf_fs = 1858994235, tf_es = -1078001605, tf_ds = 136249403, tf_edi = 1859007112, tf_esi = -1077940604, tf_ebp = 136256060, tf_isp = -411706012, tf_ebx = 3, tf_edx = 32768, tf_ecx = 0, tf_eax = 128, tf_trapno = 22, tf_err = 2, tf_eip = 1859694163, tf_cs = 51, tf_eflags = 530, tf_esp = 136255664, tf_ss = 59}) at /usr/src/sys/i386/i386/trap.c:976
            params = 0x81f18b4 <Address 0x81f18b4 out of bounds>
            callp = (struct sysent *) 0xc06b74c0
            td = (struct thread *) 0xc40414b0
            p = (struct proc *) 0xc4044418
            orig_tf_eflags = 530
            sticks = 688
            error = 0
            narg = 2
            args = {136590812, 136590216, 80, 0, 0, 0, 688, -1006353384}
            code = 128
    #22 0xc06498ff in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:200
    No locals.
    #23 0x00000033 in ?? ()
    No symbol table info available.
    (kgdb)

    Cheers,

    -- 
    Xin LI <delphij frontfree net>	http://www.delphij.net/
    See complete headers for GPG key and other information.
    
    



  • Next message: Julian Elischer: "Re: Timers and timing, was: MySQL Performance 6.0rc1"

    Relevant Pages

    • Re: [RELENG_6] NFS panic on locking against myself
      ... > On a production CVS server of ours we got panics when there is some wrong ... > data was injected to the NFS TCP connection. ... > Copyright 2004 Free Software Foundation, ... > No locals. ...
      (freebsd-current)
    • RE: panic: lockmgr: locking against myself (kern/64206)
      ... Copyright 2002 Free Software Foundation, ... GDB is free software, covered by the GNU General Public License, and you are ... No locals. ... > Panics are being brought on by heavy building of various ports. ...
      (freebsd-current)
    • panic every few hours, pmap related?
      ... since this weekend my highly loaded newsserver panics every few hours with ... GNU gdb 5.2.1 ... Fatal trap 3: breakpoint instruction fault while in kernel mode ... No locals. ...
      (freebsd-current)
    • Re: panic every few hours, pmap related?
      ... >since this weekend my highly loaded newsserver panics every few hours with ... >panic: from debugger ... breakpoint instruction fault while in kernel mode ... >No locals. ...
      (freebsd-current)
    • panic: lockmgr: locking against myself (was Panic on very recent -CURRENT)
      ... Panics are being brought on by heavy building of various ports. ... GDB is free software, covered by the GNU General Public License, and you are ... No locals. ...
      (freebsd-current)