Re: 5.0.7 machine locks up!
From: Bela Lubkin (belal_at_sco.com)
Date: 09/06/03
- Next message: bill: "SCO emulator for Linux"
- Previous message: Bela Lubkin: ""pci_debug" symbol conflict, Re: Is 5.0.7 ready for production?"
- In reply to: Brian Lavender: "Re: 5.0.7 machine locks up!"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Sat, 6 Sep 2003 09:41:40 GMT To: scomsc@xenitec.ca
Brian Lavender wrote:
> >> I just put into production my new OSR 5.0.7 machine running an
> >> Orthodontist application, and I have experienced several times where
> >> the system became unresponsive and I had to do power cycle on it
> >> (Ouch). I tried a terminal login, a virtual console, and even a telnet
> >> login. All I had was one shell that responded, but when I tried to
> >> switch to the root user, it would become unresponsive. Where can I
> >> look to find out what caused this?
> Here's what the machine has attached to it.
>
> 2 telnet logins
> 2 serial based terminal logins
> 1 login is through the Digi Terminal Server
> 1 login is through the tty1a
> The console with three virtual consoles
>
> The telnet and serial based terminals became totally unresponsive. I
> had one console on tty02 in an existing shell where I could type
> $ w
> and it would respond. I could type a few other commands as well. On
> the other virtual consoles, if I logged out, I would get a login:
> prompt. I could type in the user name, but then I would get no
> password: prompt. I do believe that after a waiting a long time, I was
> able to get a password prompt. Then I did get a # prompt. I typed
> # init 6
> but it wouldn't go into reboot. The result of the w command showed
> zero load. I did a power cycle on the box, and after rebooting, I
> checked syslog and messages. I couldn't see nothing that resulted in
> the problem.
This could very well be a disk hang. Next time it happens, watch the
disk light very carefully while provoking as much response as you can
still get (e.g. running `w`, getting to login prompts on multiscreens,
typing username, hitting return, no password prompt; etc.) If there is
no disk light activity (remains stuck off or on), it's probably a disk
hang. Then, when rebooting, watch the light again just to be sure that
it works under normal conditions (it wouldn't do to blame the disk when
actually the cable to the LED wasn't connected...)
Then, check all disk cabling and related areas. If it's SCSI, go into
the HBA's BIOS setup, make sure it's configured sanely.
> The Digi Terminal Server doesn't connect via PCI or ISA. It sits on
> the network and uses a driver to make the serial ports look as if they
> are local.
>
> A friend suggested I look at ps and see if there is a process that has
> a blocked or waiting interrupt. He also suggested looking at lsof.
Something that causes all terminal processes on the system to become
unresponsive is unlikely to be caused by a process. But do look at `ps
-ef` if you can. Column 4 (labeled "C") is an indicator of recent CPU
usage. If you see one or more processes that show 80 (the max value),
they're probably spinning consuming CPU. But normally that would only
slow down a system, not stop it.
> The one thing I do know is that one of the serial based terminals
> shows names of patients who are scheduled to arrive. The patients
> normally check themselves in. If the receptionist checks in a person
> in a person instead, the program normally removes the patient, and
> updates the patient login screen. The tty for the patient checkin is
> writeable by other users. I am thinking that maybe there is some
> process that has or is waiting for something to come available, and it
> is causing the system to block. There doesn't seem to be any specific
> conditions that causes this lockup.
This doesn't sound like a probable cause. Though I must say I still
don't particularly understand the description.
> Any suggestions on how to troubleshoot this?
Look into the disk stuff. Observe the disk lights; also run the
commands I asked about earlier (dparam etc.).
Link scodb into the kernel, break into the kernel debugger when the
system hangs, look around. If you set up a serial console (easy, you've
already got tty1a active), you can capture scodb sessions nicely. To do
a temporary serial console, get to the boot prompt and type "systty=1".
The boot prompt will move to tty1a == COM1 (9600/8/N/1). Have a PC
waiting there with "capture" turned on. Control-X is the "enter scodb"
command on a serial console. Once the machine hangs, hit ^X on the
serial console, then:
scodb> stack
scodb> ps()
There's much more you can do, but that's a decent start. One other
thing you can do is capture a system image:
scodb> sysdump()
scodb> reboot()
(the machine reboots as if you'd hit the power switch). Now go to
single-user mode and run:
sysdump -i /dev/swap -fu -o - | bzip2 > /tmp/dump.bz2
Save that, I might want to look at it.
>Bela<
- Next message: bill: "SCO emulator for Linux"
- Previous message: Bela Lubkin: ""pci_debug" symbol conflict, Re: Is 5.0.7 ready for production?"
- In reply to: Brian Lavender: "Re: 5.0.7 machine locks up!"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
|
