Re: panic in rt_check_fib()


On Sat, 13 Sep 2008, Robert Watson wrote:

On Fri, 5 Sep 2008, Giorgos Keramidas wrote:

A kernel that I built last night to test Ed's "packet mode" for ptys included all the changes up to 182743 panics with:

I had an identical panic on 7-STABLE last night:

db> bt
Tracing pid 782 tid 100091 td 0xc4496440
kdb_enter_why(c0b25ea1,c0b25ea1,c0b24c19,e6772978,0,...) at kdb_enter_why+0x3a
panic(c0b24c19,c0b32d59,c0b32d7a,633,c436c9b0,...) at panic+0x12c
_mtx_lock_sleep(c436ddf4,c4496440,0,c0b32d7a,633,...) at _mtx_lock_sleep+0x4a
_mtx_lock_flags(c436ddf4,0,c0b32d7a,633,c436ca14,...) at _mtx_lock_flags+0xd1

This is actually from i386/machine/pcpu.h, line 194:

static __inline struct thread *
__curthread(void)
{
struct thread *td;

__asm __volatile("movl %%fs:0,%0" : "=r" (td));
return (td);
}

rt_check_fib(e6772a0c,e6772a28,c424ea90,0,e6772a1c,...) at rt_check_fib+0x2b4

(kgdb) l *rt_check_fib+0x2b4
0xc0827104 is in rt_check_fib (/usr/src/sys/net/route.c:1587).
warning: Source file is more recent than executable.

1582 if ((rt->rt_flags & RTF_UP) == 0) {
1583 RTFREE_LOCKED(rt); /* unlock gwroute */
1584 rt = rt0;
1585 rt0->rt_gwroute = NULL;
1586 lookup:
1587 RT_UNLOCK(rt0);
1588 /* XXX MRT link level looked up in table 0 */
1589 rt = rtalloc1_fib(rt->rt_gateway, 1, 0UL, 0);
1590 if (rt == rt0) {
1591 RT_REMREF(rt0);

in_rt_check(e6772a0c,e6772a28,c424ea90,0,0,...) at in_rt_check+0x26

(kgdb) l *in_rt_check+0x26
0xc08576f6 is in in_rt_check (/usr/src/sys/netinet/in_rmx.c:473).
468 int
469 in_rt_check(struct rtentry **lrt, struct rtentry **lrt0,
470 struct sockaddr *dst, u_int fibnum)
471 {
472 return (rt_check_fib(lrt, lrt0, dst, fibnum));
473 }
474
475 void
476 in_rtredirect(struct sockaddr *dst,
477 struct sockaddr *gateway,

arpresolve(c4040000,c436c9b0,c4240800,c424ea90,e6772a42,...) at arpresolve+0xb9

(kgdb) l *arpresolve+0xb9
0xc084ddc9 is in arpresolve (/usr/src/sys/netinet/if_ether.c:379).
warning: Source file is more recent than executable.

374 ETHER_MAP_IP_MULTICAST(&SIN(dst)->sin_addr, desten);
375 return (0);
376 }
377 fibnum = M_GETFIB(m);
378 }
379
380 if (rt0 != NULL) {
381 /* Look for a cached arp (ll) entry. */
382 error = in_rt_check(&rt, &rt0, dst, fibnum);
383 if (error) {

Looks like I'm using an older version of if_ether.c than I have checked out.

ether_output(c4040000,c4240800,c424ea90,c436c9b0,c450b9d8,...) at ether_output+0x7e

(kgdb) l *ether_output+0x7e
0xc081774e is in ether_output (/usr/src/sys/net/if_ethersubr.c:175).
warning: Source file is more recent than executable.

170
171 hlen = ETHER_HDR_LEN;
172 switch (dst->sa_family) {
173 #ifdef INET
174 case AF_INET:
175 error = arpresolve(ifp, rt0, m, dst, edst);
176 if (error)
177 return (error == EWOULDBLOCK ? 0 : error);
178 type = htons(ETHERTYPE_IP);
179 break;

Robert N M Watson
Computer Laboratory
University of Cambridge

ip_output(c4240800,0,e6772ab0,0,0,...) at ip_output+0xa34
udp_send(c44f74b0,0,c4240800,c4514ac0,0,...) at udp_send+0x58b
sosend_dgram(c44f74b0,c4514ac0,e6772bd4,c4240800,0,...) at sosend_dgram+0x352
sosend(c44f74b0,c4514ac0,e6772bd4,0,0,...) at sosend+0x54
kern_sendit(c4496440,20,e6772c58,0,0,...) at kern_sendit+0x106
sendit(0,1,e6772c54,28,c426a090,...) at sendit+0x162
sendmsg(c4496440,e6772cfc,c,c4496630,c0bd53c0,...) at sendmsg+0x78
syscall(e6772d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20

Unfortunately, I was unable to successfully get a crashdump -- not entirely sure why as it seemed to go to disk ok.

Robert N M Watson
Computer Laboratory
University of Cambridge



========================================================================

root@kobe:/var/crash# kgdb /boot/kernel/kernel vmcore.5
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: _mtx_lock_sleep: recursed on non-recursive mutex rtentry @ /home/build/src/sys/net/route.c:1742

cpuid = 0
Uptime: 5m26s
Physical memory: 2026 MB
Dumping 80 MB: 65 49 33 17 1

Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from /boot/kernel/snd_hda.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/snd_hda.ko
Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/sound.ko
Reading symbols from /boot/kernel/if_iwn.ko...Reading symbols from /boot/kernel/if_iwn.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/if_iwn.ko
Reading symbols from /boot/kernel/acpi.ko...Reading symbols from /boot/kernel/acpi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/snake_saver.ko...Reading symbols from /boot/kernel/snake_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/snake_saver.ko
#0 doadump () at pcpu.h:221
221 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) list
216 in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:221
#1 0xc05e13ac in boot (howto=260) at /home/build/src/sys/kern/kern_shutdown.c:418
#2 0xc05e1678 in panic (fmt=Variable "fmt" is not available.
) at /home/build/src/sys/kern/kern_shutdown.c:572
#3 0xc05d3fda in _mtx_lock_sleep (m=0xc573eba4, tid=3314466816, opts=0, file=0xc08f457a "/home/build/src/sys/net/route.c", line=1742) at /home/build/src/sys/kern/kern_mutex.c:310
#4 0xc05d422f in _mtx_lock_flags (m=0xc573eba4, opts=0, file=0xc08f457a "/home/build/src/sys/net/route.c", line=1742) at /home/build/src/sys/kern/kern_mutex.c:182
#5 0xc0694ad8 in rt_check_fib (lrt=0xe7c299ec, lrt0=0xe7c29a08, dst=0xc5550710, fibnum=0) at /home/build/src/sys/net/route.c:1742
#6 0xc06caf36 in in_rt_check (lrt=0xe7c299ec, lrt0=0xe7c29a08, dst=0xc5550710, fibnum=0) at /home/build/src/sys/netinet/in_rmx.c:472
#7 0xc06c0ecd in arpresolve (ifp=0xc51fd800, rt0=0xc573eca8, m=0xc59c2200, dst=0xc5550710, desten=0xe7c29a22 "") at /home/build/src/sys/netinet/if_ether.c:388
#8 0xc0689a9e in ether_output (ifp=0xc51fd800, m=0xc59c2200, dst=0xc5550710, rt0=0xc573eca8) at /home/build/src/sys/net/if_ethersubr.c:183
#9 0xc06d1bf1 in ip_output (m=0xc59c2200, opt=0x0, ro=0xe7c29aa8, flags=Variable "flags" is not available.
) at /home/build/src/sys/netinet/ip_output.c:563
#10 0xc073ecfb in udp_send (so=0xc573b498, flags=0, m=0xc59c2200, addr=0xc597e2f0, control=0x0, td=0xc58ec000) at /home/build/src/sys/netinet/udp_usrreq.c:1060
#11 0xc064530f in sosend_dgram (so=0xc573b498, addr=0xc597e2f0, uio=0xe7c29bd4, top=0xc59c2200, control=0x0, flags=Variable "flags" is not available.
) at /home/build/src/sys/kern/uipc_socket.c:1059
#12 0xc0643054 in sosend (so=0xc573b498, addr=0xc597e2f0, uio=0xe7c29bd4, top=0x0, control=0x0, flags=0, td=0xc58ec000) at /home/build/src/sys/kern/uipc_socket.c:1292
#13 0xc064bf15 in kern_sendit (td=0xc58ec000, s=516, mp=0xe7c29c54, flags=0, control=0x0, segflg=UIO_USERSPACE) at /home/build/src/sys/kern/uipc_syscalls.c:782
#14 0xc064c121 in sendit (td=0xc58ec000, s=516, mp=0xe7c29c54, flags=0) at /home/build/src/sys/kern/uipc_syscalls.c:719
#15 0xc064c1d1 in sendmsg (td=0xc58ec000, uap=0xe7c29cf8) at /home/build/src/sys/kern/uipc_syscalls.c:915
#16 0xc0884d13 in syscall (frame=0xe7c29d38) at /home/build/src/sys/i386/i386/trap.c:1090
#17 0xc0869020 in Xint0x80_syscall () at /home/build/src/sys/i386/i386/exception.s:261
#18 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)

========================================================================

From the limited testing I could do today it seems that the following
changes might be useful to track down why this is happening:

/head@182698 -> ok so far
/head@182743 -> panic

I don't see any rt_check_fib() changes in this commit range, so it may
be false that /head@182698 is ok. It just doesn't panic immediately
when I try to bring up my re0 interface and set the default route.

- Giorgos


_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

_______________________________________________
freebsd-current@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@xxxxxxxxxxx"

Relevant Pages