Re: DDB scripting, output capture, and textdumps

On Tuesday 18 December 2007 14:10:46 Robert Watson wrote:
Dear all:

I've been hacking on-and-off for a while on a side project to improve
our kernel debugging facilities. Primarily, my concern has been to
address three problems:

- The complications of employing kernel core dumps for debugging,
including the large size of dumps making them unwieldy to distribute
or store for any extended period (even with minidumps), the requirement
to have relatively synchronized kernel source in order to use the dumps,
the need to have a kernel with debugging symbols, and the problems with
fsck causing sufficient swap use to invalidate dumps before they can be
extracted.

- The decreasing likelihood that notebooks will ship with serial ports
that can be used for interactive debugging using DDB. Making
end-users type in stack traces is cruel, photos are a pain, and X11
rules out both.

- The fact that a great many problems are most easily diagnosed using
utility routines present in DDB, but not as easily using kgdb for
offline analysis. I find that for many bugs I analyze, simply
looking at the DDB output is sufficient to identify the source of the
problem.

An idea I punted around a bit at BSDCan earlier this year (or perhaps it
was at EuroBSDCon the previous year) was an idea of a "textdump" -- that
is, a new type of kernel dump based on capturing automatically extracted
debugging information generated by DDB. The result would be an ASCII
text file that could be filed as a bug report, perhaps even
automatically.

To this end, I have implemented three new facilities for use with DDB:

(1) DDB output capture. The output of DDB is stored in a memory buffer,
and can be extracted using a sysctl or textdumps (see below). This
can be turned on and off, both for use manually ("I'll want this
later, but not that") and as part of scripts (see below).

(2) DDB scripting. A limited number of named scripts can be defined to
run a series of DDB commands. No loops, etc, just simple command
lists. These can be caused to run automatically on entering DDB
for various scenarios, including WITNESS violations and kernel panics.
They can also be run by hand in order to save a bit of typing if you use
DDB in a repetitive way (as I do).

(3) Textdumps. A new dump type that stores a series of data files
containing various pieces of information, including the DDB capture
buffer, kernel message buffer, kernel configuration (if compiled
into the kernel), panic message, and kernel version string. These are
stored in the ustar format inside the dump partition (aligned to the
end) so can be easily extended, and savecore(8) requires almost no new
logic to deal with them (it just drops numbered tar files in
/var/crash). This makes it straight forward to extend the textdump
format to include new types of information and avoids the issue of how
to safely simultaneously represent information in many different formats
in the same file.

These are pretty flexible tools, and you can imagine doing the following
sorts of things:

- Setting the kdb.enter.panic script to automatically turn on output
capture, do full backtraces of all threads, show open file
information, dump UMA stats, and save it all to a textdump and then
reboot.

- Setting the kdb.enter.witness script to show lock information,
generate a coredump, and reboot. Or, just to automatically do "show
allocks" and drop to the DDB prompt.

- Adding a flag to rc.conf to automatically submit textdumps via e-mail
to a specific address, perhaps including GNATS or an automated bug
system. These could be unpacked and automatically analyzed, and do to
the compact size, kept for long-term trend analysis or to identify when
a problem started occuring.

I've produced an initial snapshot of the above, which can be found here:

http://www.watson.org/~robert/freebsd/20071218-ddb.tgz

This adds three files to DDB, patches quite a few kernel files (to pass
more information into KDB about why it's being entered, in order to
trigger the right script), enhancements to savecore(8) to know how to
extract textdumps, adds a ddb(8) command line tool so that userspace can
manage DDB scripts from outside the debugger, extensions to the ddb(4)
man page, and a new textdump(4) man page.

There are a number of known limitations; I've tried to document them at
the top of the pertinent files where I am aware of them. I also regret
to say that to date I've been able to test only on i386, and not other
platforms. I'd welcome any feedback -- I'd like to get these changes
into CVS in the next week or two.

It looks like some files is not included in the patch.
I have this error:

make -V CFILES -V SYSTEM_CFILES -V GEN_CFILES | MKDEP_CPP="cc -E" CC="cc"
xargs
mkdep -a -f .newdep -O -pipe -std=c99 -g -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -nostdinc -I. -I/usr/src/sys -I/usr/src/sys/contrib/altq -I/usr/src/sys/contrib/ipfilter -I/usr/src/sys/contrib/pf -I/usr/src/sys/dev/ath -I/usr/src/sys/contrib/ngatm -I/usr/src/sys/dev/twa -I/usr/src/sys/gnu/fs/xfs/FreeBSD -I/usr/src/sys/gnu/fs/xfs/FreeBSD/support -I/usr/src/sys/gnu/fs/xfs -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include
opt_global.h -fno-common -finline-limit=8000 --param
inline-unit-growth=100 --param
large-function-growth=1000 -mno-align-long-strings -mpreferred-stack-boundary=2 -mno-mmx -mno-3dnow -mno-sse -mno-sse2 -mno-sse3 -ffreestanding
cc: /usr/src/sys/ddb/db_capture.c: No such file or directory
cc: /usr/src/sys/ddb/db_script.c: No such file or directory
cc: /usr/src/sys/ddb/db_textdump.c: No such file or directory
mkdep: compile failed
*** Error code 1

Stop in /usr/obj/usr/src/sys/GENERIC.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.

root@orion-vm:/usr/src# ls -la /usr/src/sys/ddb/
total 424
drwxr-xr-x 2 root wheel 1024 Dec 19 16:49 ./
drwxr-xr-x 53 root wheel 1024 Oct 29 08:24 ../
-rw-r--r-- 1 root wheel 2591 Dec 4 2005 db_access.c
-rw-r--r-- 1 root wheel 1431 Jan 6 2005 db_access.h
-rw-r--r-- 1 root wheel 7737 Jan 6 2005 db_break.c
-rw-r--r-- 1 root wheel 2098 Jan 6 2005 db_break.h
-rw-r--r-- 1 root wheel 16579 Dec 19 16:49 db_command.c
-rw-r--r-- 1 root wheel 15705 Jan 17 2007 db_command.c.orig
-rw-r--r-- 1 root wheel 1633 Dec 19 16:49 db_command.h
-rw-r--r-- 1 root wheel 1588 Jan 6 2005 db_command.h.orig
-rw-r--r-- 1 root wheel 7270 Oct 27 20:19 db_examine.c
-rw-r--r-- 1 root wheel 4811 Jan 6 2005 db_expr.c
-rw-r--r-- 1 root wheel 7981 Dec 19 16:49 db_input.c
-rw-r--r-- 1 root wheel 7931 Jan 6 2005 db_input.c.orig
-rw-r--r-- 1 root wheel 5978 Dec 19 16:49 db_lex.c
-rw-r--r-- 1 root wheel 5304 Jan 6 2005 db_lex.c.orig
-rw-r--r-- 1 root wheel 1951 Dec 19 16:49 db_lex.h
-rw-r--r-- 1 root wheel 1861 Jan 6 2005 db_lex.h.orig
-rw-r--r-- 1 root wheel 5976 Dec 19 16:49 db_main.c
-rw-r--r-- 1 root wheel 5787 Nov 6 2006 db_main.c.orig
-rw-r--r-- 1 root wheel 6889 Dec 19 16:49 db_output.c
-rw-r--r-- 1 root wheel 6639 Oct 10 2006 db_output.c.orig
-rw-r--r-- 1 root wheel 1444 Oct 8 2006 db_output.h
-rw-r--r-- 1 root wheel 2013 Apr 14 2005 db_print.c
-rw-r--r-- 1 root wheel 10956 Nov 13 13:43 db_ps.c
-rw-r--r-- 1 root wheel 8810 Apr 14 2005 db_run.c
-rw-r--r-- 1 root wheel 7815 Jun 16 2006 db_sym.c
-rw-r--r-- 1 root wheel 3591 Jan 6 2005 db_sym.h
-rw-r--r-- 1 root wheel 5143 Jan 17 2007 db_thread.c
-rw-r--r-- 1 root wheel 3402 Jan 6 2005 db_variables.c
-rw-r--r-- 1 root wheel 1854 Jan 6 2005 db_variables.h
-rw-r--r-- 1 root wheel 7218 Nov 17 2006 db_watch.c
-rw-r--r-- 1 root wheel 1509 Jan 6 2005 db_watch.h
-rw-r--r-- 1 root wheel 2264 Jan 6 2005 db_write_cmd.c
-rw-r--r-- 1 root wheel 7467 Dec 19 16:49 ddb.h
-rw-r--r-- 1 root wheel 5776 Jul 12 2006 ddb.h.orig


--
======================================================================
- Best regards, Nikolay Pavlov. <<<-----------------------------------
======================================================================

Attachment: signature.asc
Description: This is a digitally signed message part.

Relevant Pages

  • HEADSUP: KDB framework
    ... A new debugging framework has been committed. ... for kernel threads to both DDB and remote GDB and other nice features, ...
    (freebsd-current)
  • Re: DDB scripting, output capture, and textdumps
    ... The complications of employing kernel core dumps for debugging, ... that can be used for interactive debugging using DDB. ...
    (freebsd-current)
  • DDB scripting, output capture, and textdumps
    ... I've been hacking on-and-off for a while on a side project to improve our kernel debugging facilities. ... The complications of employing kernel core dumps for debugging, ... that can be used for interactive debugging using DDB. ...
    (freebsd-arch)
  • DDB scripting, output capture, and textdumps
    ... I've been hacking on-and-off for a while on a side project to improve our kernel debugging facilities. ... The complications of employing kernel core dumps for debugging, ... that can be used for interactive debugging using DDB. ...
    (freebsd-current)
  • Re: 5.1-CURRENT hangs on disk i/o? sysctl_old_user() non-sleepable locks
    ... I checked in a fix for this problem today so you should get ... > I'm a newbie to ddb. ... >> kernel config to see if that turns up anything. ... If you're getting a lot of VFS lock violation reports, ...
    (freebsd-current)