using scodb for various purposes, Re: 5.0.6 grinds to a complete halt

From: Bela Lubkin (belal_at_sco.com)
Date: 05/20/04 

Date: Thu, 20 May 2004 08:34:07 GMT
To: scomsc@xenitec.ca

Barry Swane wrote:

> Some questions re the debugger- which I have now configured.
> If the disk has stopped-- am I likely to get anything back from the
> debugger?

Yes. Oh, there may be some cases where a disk hang would stop the
kernel as a whole, but that would be unusual. You've already described
symptoms where the kernel is clearly still running (you can type and
edit an input line); under those conditions, scodb will definitely be
helpful.

> I assume this can only be run from the system console-- I can't do it
> remotely?

OpenServer includes two different items named "scodb".

There's the kernel debugger driver, enabled by putting 'Y' in
/etc/conf/sdevice.d/scodb, relinking the kernel and rebooting. To get
into that, you must be sitting at the system console. Note that the
console can be either the local video card multiscreens, or a serial
port. I like to use something I call the "buddy system", described on:

  http://groups.google.com/groups?selm=9804061301.aa02092@vagabond.armory.com

scodb only supports standard ASCII over a serial port, so it has no way
to observe a sequence like Ctrl-Alt-D; the default debugger key on a
serial port is Ctrl-X.

There's also the scodb(ADM) command (/etc/scodb). This is basically a
recompilation of the scodb driver as a user-level program. It can read
and write kernel memory, but cannot affect flow of control (for instance
it can't set breakpoints nor be invoked when a breakpoint -- or panic --
is hit). It can also be run to examine a saved crash dump.

> I imagine that, in order to get info from the debugger, root must
> already be logged in, and sitting at # prompt?

If it's linked in, the in-kernel debugger can be invoked at any time
from the console, regardless of who's logged in (including nobody).
Exception: you can invoke it from a graphical screen like X, but it
doesn't do anything to put the screen back in text mode, nor display
itself in graphics. You can do simple interactions (like "whoops, I
just hit Ctrl-Alt-D and the whole system hung ... type 'q' Return ...
ah, we're back). There may be a similar situation on a serial port, but
it's much more obscure: OSR5 supports a "scan code mode" where, with a
suitable terminal, programs _can_ see things like Ctrl and Alt being
pressed and released. I've never tried scodb under those circumstances,
but I bet it would go wrong. Hardly anything uses scancode mode.

In order to run user-level scodb, you must be logged in as root. This
would be impossible if the disk subsystem had already hung. However, if
you were _already running_ scodb by the time of the hang, you might get
away with it. So experiment with that -- open an ssh session to the
machine, run `/etc/scodb -w`, leave it that way until the hang happens.
See if you can interact. Two possible impediments: (1) if memory is
overcommitted, pages of your idle process will have been pushed out to
swap and it'll hang trying to retrieve them; (2) scodb may not have
faulted in all the pages of its own executable image. #2 could be a big
problem. To stack the deck in your favor, when you first start the
session, briefly use all the scodb commands you're likely to use later.

> I am trying to experiment with the debugger in advance of the
> freeze-up, to try to get a little bit familiar with it:
> i) if I hold CTRL-ALT-D - it just logs me out, as if I had pressed
> CTRL-D

You mentioned that you were coming in remotely, so that's not going to
work. If you use the "buddy system" you'll be coming in (ssh or telnet)
to a machine _next to_ the problem machine, then running a serial comm
program to the problem machine. Then you'll be able to pop up the real
in-kernel debugger (but the key will be Ctrl-X, not Ctrl-Alt-D).

> ii) I can load scodb, from shell prompt
> If I enter "stack" command, I get
> When operating on /dev/mem, you cannot examine the stack of the
> current process. The "stack" command must be used with the "-p"
> argument.
> If I enter "stack -p", I get the same message

Use "?" for scodb's internal help. On the "stack" command it says:

  stack [stack_addr|-p pid|-p proc_addr] Stack backtrace

Try "stack -p 1".

The notion of "current process" doesn't make sense in the context of
user-level scodb. If you break into in-kernel scodb, you've (usually)
interrupted some process that was doing its thing. Commands like
"stack" can then show you what that process was doing. In user-level
scodb, at all times when scodb can read information out of the kernel,
scodb itself is the running process. Looking at the workings of the
debugger process itself might be somewhat interesting, but it's almost
never what you set out to do. So you have to tell it what process you
care about. (It could, probably should, be enhanced to have a way to
inform it: "nevermind all that, the process I am generally interested in
is #1234; pretend I've specified that in all commands that care". But
that doesn't exist in the current /etc/scodb.)

> Can someone point me to documentation on scodb? man scodb makes
> reference to the SCODB User's Guide. I thought I had a complete set
> of manuals- but I don't have that one.

Again, this is a good question which I will respond to in a third
separate message so it can more easily be found in the future.

>Bela<

Relevant Pages