Re: syslogd spinning the CPU, not logging...

On Wed, May 04, 2011 at 02:58:20PM -0400, Mikhail T. wrote:
On FreeBSD/amd64 8.2-STABLE as of Sun Feb 27. I dismissed this
problem the first time (a week or two ago), but just saw it again.
Something triggers syslogd into spinning all available CPU -- while
not logging anything.

Attempts to log anything -- such as by invoking logger(1) -- simply
hang. sendmail hangs too -- now new e-mail is coming. It is not
pretty.

ktrace-ing the process yields empty file -- it is not doing
system-calls, while spinning.

Attaching the debugger several times shows the same stack:

#0 0x00000008007cb1d6 in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7
#1 0x00000008007cd9fb in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7
#2 0x00000008007d4825 in free () from /lib/libc.so.7
#3 0x00000008007c5aab in catopen () from /lib/libc.so.7
#4 0x00000008007c5410 in strerror_r () from /lib/libc.so.7
#5 0x00000008007c556d in strerror () from /lib/libc.so.7
#6 0x0000000000403c10 in ?? ()
...

Among the logging-destinations in my syslog.conf there is a program,
that exits sometimes -- but never "too fast". Normally, syslogd
simply restarts it without an issue...

Any ideas? Thanks!

If ktrace doesn't show any syscalls (did you use "ktrace -i -t+"?), then
I'm not sure how the process is actually spinning (100% CPU). "procstat
-kk" on the PID of syslogd might be more useful in this case. The
output is mainly useful for kernel folks. You may end up having to drop
to ddb to examine what's going on within the pid.

I'm not sure if attaching gdb to a running process on FreeBSD works with
things like threads, etc.. Meaning I'm not sure if the above stack
trace is indicative of a problem or not. No offence to anyone intended,
but I try to avoid gdb on FreeBSD wherever possible (it's so old that I
don't know how useful it is for extreme debugging).

I've never seen syslogd spiral out of control on any of our systems, but
all logging destinations in syslog.conf on our systems are files on
local disks.

If you can find a way to reliably reproduce the issue that would be
useful, otherwise this sounds like one of those things that's going to
be a problem to track down. :-(

--
| Jeremy Chadwick jdc@xxxxxxxxxxxx |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP 4BD6C0CB |

_______________________________________________
freebsd-stable@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • RE: Logging Stops after few minutes
    ... are you sure that syslogd stops working after only 2 ... Syslogd only chews on the processor when it has something to do. ... If you're certain that it isn't logging events that it should: ... If you look at my FTP command it should ...
    (freebsd-questions)
  • Re: syslogd question (modifying syslogd.c)
    ... Solaris seems to have more feature-full logging facilities. ... And in 'syslogd': ... by an identifier in the following format: ... In syslog, the logging priority ...
    (RedHat)
  • Re: Logging TCP events
    ... sending those events to syslogd. ... syslogd is a relatively slow logging mechanism, ... Syslog-NG can use TCP connectivity (I use it to SSH tunnel logging ... either the IPTABLES definitions or if doing the TCPDUMP, ...
    (comp.security.firewalls)
  • Re: remote syslog
    ... To enable this [network logging on the server] you have to specify the ... The default behavior is that syslogd ...
    (Fedora)