Weird memory exhaustion with FreeBSD 4.10-STABLE

From: Pekka Savola (pekkas_at_netcore.fi)
Date: 09/23/04

  • Next message: Paul Schenkeveld: "Re: question on tunnels (VPN)" 
    Date: Thu, 23 Sep 2004 15:03:47 +0300 (EEST)
To: snap-users@kame.net

    Hi,

    I've been suffering from a weird memory exhaustion problem (or so it
    looks?!) on FreeBSD 4.10 -STABLE for a year or two now (obviously,
    then it was something like 4.5 or the like :).

    After the system has been up for a while (a couple of weeks) and/or
    the ethernet link has gone up/down for some time (I suspect this is
    related to the Neighbor Discovery cache contents), I get dmesg errors
    like:

    nd6_lookup: failed to add route for a neighbor(fe80:0001::0290:6900:4276:4d7b), errno=55
    nd6_lookup: failed to add route for a neighbor(2001:0708:0:0001::0625), errno=55

    (55 is 'No buffer space available'.)

    (However, if you look at the 'top' output below, the whole swap is
    still pretty much unused..)

    Some other commands then also cease to work, e.g., 'ifconfig' displays
    nothing, and gives something like 'interface stf0 does not exist'
    for interfaces that are up and running. The system still works
    otherwise.

    There's nothing really fancy about this system, it's just a 6to4 relay
    router. The "weirdest" thing is probably that it has address
    2001:0708:0:1::624/125 [ie, not /64] configured on its xl0 interface.

    I can reproduce this on this particular box, so I can do some tests or
    debugging if needed.

    Ideas?

    ...

    A few outputs (when there is a problem) which might help:

    sixpack# netstat -m
    129/2048/6144 mbufs in use (current/peak/max):
            129 mbufs allocated to data
    128/232/1536 mbuf clusters in use (current/peak/max)
    976 Kbytes allocated to network (21% of mb_map in use)
    0 requests for memory denied
    0 requests for memory delayed
    0 calls to protocol drain routines

    sixpack# top

    last pid: 78282; load averages: 0.00, 0.00, 0.00
                                                                up 33+09:44:56 17:22:47
    20 processes: 1 running, 19 sleeping
    CPU states: % user, % nice, % system, % interrupt, % idle
    Mem: 12M Active, 20M Inact, 24M Wired, 3424K Cache, 14M Buf, 780K Free
    Swap: 132M Total, 112K Used, 132M Free
                                                                                                                                             
      PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
    34357 root 2 0 5288K 1612K select 0:00 0.05% 0.05% sshd
      104 root 2 0 984K 548K select 426:12 0.00% 0.00% syslogd
      146 root 2 0 1896K 944K select 404:35 0.00% 0.00% zebra
      148 root 2 0 3644K 2512K select 17:30 0.00% 0.00% bgpd
      121 root 2 0 3048K 1628K select 7:06 0.00% 0.00% sendmail
      116 root 2 0 2588K 1364K select 1:33 0.00% 0.00% sshd
      114 root 10 0 1000K 600K nanslp 1:02 0.00% 0.00% cron
    77556 root 3 0 956K 532K ttyin 0:57 0.00% 0.00% getty
      124 smmsp 18 0 2928K 1412K pause 0:08 0.00% 0.00% sendmail
      112 root 2 0 1076K 616K select 0:08 0.00% 0.00% inetd
    34359 root 18 0 1368K 872K pause 0:00 0.00% 0.00% csh
    78282 root 31 0 1892K 888K RUN 0:00 0.00% 0.00% top
      197 root 3 0 956K 532K ttyin 0:00 0.00% 0.00% getty
      198 root 3 0 956K 532K ttyin 0:00 0.00% 0.00% getty
      200 root 3 0 956K 532K ttyin 0:00 0.00% 0.00% getty
      201 root 3 0 956K 532K ttyin 0:00 0.00% 0.00% getty
      199 root 3 0 956K 532K ttyin 0:00 0.00% 0.00% getty
      195 root 3 0 956K 532K ttyin 0:00 0.00% 0.00% getty
      196 root 3 0 956K 532K ttyin 0:00 0.00% 0.00% getty
       24 root 18 0 212K 68K pause 0:00 0.00% 0.00% adjkerntz
                                                                                                                                             
    sixpack# dmesg
    nd6_lookup: failed to add route for a neighbor(fe80:0001::0290:6900:4276:4d7b), errno=55
    nd6_lookup: failed to add route for a neighbor(fe80:0001::0290:6900:4276:4d7b), errno=55
    nd6_lookup: failed to add route for a neighbor(2001:0708:0:0001::0625), errno=55
    nd6_lookup: failed to add route for a neighbor(fe80:0001::0290:6900:4276:4d7b), errno=55
    nd6_lookup: failed to add route for a neighbor(fe80:0001::0290:6900:4276:4d7b), errno=55
    nd6_lookup: failed to add route for a neighbor(2001:0708:0:0001::0625), errno=55
    [snipped like 50-100 other such messages]
    sixpack# ifconfig
    sixpack# ps auxwww
    USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
    root 0 0.0 0.0 0 0 ?? DLs 19Aug04 0:00.00 (swapper)
    root 77556 0.0 0.9 956 532 v0 Ss+ 1:52PM 0:56.89 /usr/libexec/getty Pc ttyv0
    root 34359 0.0 1.4 1368 872 p1 Ss 13Sep04 0:00.22 -csh (csh)
    root 34357 0.0 2.6 5288 1612 ?? S 13Sep04 0:00.44 sshd: root@ttyp1 (sshd)
    root 201 0.0 0.9 956 532 v7 Is+ 19Aug04 0:00.02 /usr/libexec/getty Pc ttyv7
    root 200 0.0 0.9 956 532 v6 Is+ 19Aug04 0:00.02 /usr/libexec/getty Pc ttyv6
    root 199 0.0 0.9 956 532 v5 Is+ 19Aug04 0:00.02 /usr/libexec/getty Pc ttyv5
    root 198 0.0 0.9 956 532 v4 Is+ 19Aug04 0:00.02 /usr/libexec/getty Pc ttyv4
    root 197 0.0 0.9 956 532 v3 Is+ 19Aug04 0:00.02 /usr/libexec/getty Pc ttyv3
    root 196 0.0 0.9 956 532 v2 Is+ 19Aug04 0:00.02 /usr/libexec/getty Pc ttyv2
    root 195 0.0 0.9 956 532 v1 Is+ 19Aug04 0:00.02 /usr/libexec/getty Pc ttyv1
    quagga 148 0.0 4.1 3644 2512 ?? Ss 19Aug04 17:29.51 /usr/local/sbin/bgpd -d
    quagga 146 0.0 1.6 1896 944 ?? Ss 19Aug04 404:34.68 /usr/local/sbin/zebra -d
    smmsp 124 0.0 2.3 2928 1412 ?? Is 19Aug04 0:08.22 sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
    root 121 0.0 2.7 3048 1628 ?? Ss 19Aug04 7:05.86 sendmail: accepting connections (sendmail)
    root 116 0.0 2.2 2588 1364 ?? Is 19Aug04 1:32.52 /usr/sbin/sshd
    root 114 0.0 1.0 1000 600 ?? Ss 19Aug04 1:01.90 /usr/sbin/cron
    root 112 0.0 1.0 1076 616 ?? Is 19Aug04 0:08.14 /usr/sbin/inetd -wW
    root 104 0.0 0.9 984 548 ?? Ss 19Aug04 426:12.11 /usr/sbin/syslogd -ss
    root 24 0.0 0.1 212 68 ?? Is 19Aug04 0:00.00 adjkerntz -i
    root 7 0.0 0.0 0 0 ?? DL 19Aug04 24:49.48 (syncer)
    root 6 0.0 0.0 0 0 ?? DL 19Aug04 1:08.24 (vnlru)
    root 5 0.0 0.0 0 0 ?? DL 19Aug04 0:47.40 (bufdaemon)
    root 4 0.0 0.0 0 0 ?? DL 19Aug04 0:00.00 (vmdaemon)
    root 3 0.0 0.0 0 0 ?? DL 19Aug04 0:32.02 (pagedaemon)
    root 2 0.0 0.0 0 0 ?? DL 19Aug04 0:00.00 (taskqueue)
    root 1 0.0 0.3 544 200 ?? ILs 19Aug04 0:01.09 /sbin/init --
    root 78285 0.0 0.4 420 216 p1 R+ 5:23PM 0:00.00 ps auxwww 

    -- 
Pekka Savola                 "You each name yourselves king, yet the
Netcore Oy                    kingdom bleeds."
Systems. Networks. Security. -- George R.R. Martin: A Clash of Kings
_______________________________________________
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
  • Next message: Paul Schenkeveld: "Re: question on tunnels (VPN)"