Re: SCO OSR5.0.6 System Crashed during LoneTAR Verify

tony_at_aplawrence.com
Date: 06/13/03 

Date: Fri, 13 Jun 2003 16:38:40 +0000 (UTC)

Lucky Leavell <scomisc@unixpress.com> wrote:
: On Fri, 13 Jun 2003 tony@aplawrence.com wrote:

:> Lucky Leavell <lucky@unixpress.com> wrote:
:> : OS: OSR5.0.6a Enterprise
:> : Lone-TAR: v3.2.4.1
:>
:> : The last two nights, a client's system has died while in the LT verify pass.
:> : I do not know if this is related or a coincidence.
:>
:> : I find nothing at all in the /usr/adm/messages or syslog to indicate any
:> : problem. I have both AUTOBOOT and PANICBOOT in /etc/default/boot set to YES
:> : but when they come in the system is down and must be manually rebooted.
:>
:> So what does it show on the screen?
:>
: It was asking if they wanted to save the dump; they didn't. I asked them
: to call me next time and I will talk them through saving the dump to tape.

You don't want to save the dump. You want to know what the line that
begins with k_trap says. That tells you what the panic is from. Most
usually, it's a trap E, which is an invalid memory reference, but
this can also be from lack of patches.

At http://aplawrence.com/Unixart/trape.html is an article that
explains this, but since my site is still down:

Trap 0x0000000E

   Don't panic. That's the kernel's job - Jeff Liebermann

   More Articles

   Of all the possible reasons for a system panic, a "trap 0x0000000E" is
   the one most often seen (see SCO's "What are Traps, Interrupts and
   Exceptions?" for other reasons). Technically, an E trap is a page
   fault that referenced an impossible page: the CPU tries to access an
   address that does not exist and can't be accessed. As page references
   are normally very carefully managed, the usual cause for this is bad
   (defective) RAM; scrambled bits point the CPU toward disaster and it
   blindly follows. Therefor, if you have a Trap E panic on a machine
   that otherwise has been running along for months or years, bad RAM is
   the most likely suspect.

   You can't expect that the so called "memory test" that runs when your
   computer starts is going to catch bad ram. That testing is very
   superficial, and really can only find ram that's totally screwed up-
   subtle problems just will not be seen by that test. There are tests
   available that can really stress memory, but the best ones need to run
   a very long time, so if the suspect machine is critical, you probably
   don't have the time to do this.

   RAM is pretty cheap nowadays, so you may just want to replace all of
   it, or you can pull individual sticks and swap things around until you
   determine where the problem is. In taking this approach, try booting
   with as little RAM as possible; the bad chips can be found more easily
   that way (see Memory).

   Of course, there are other possibilities. A bus card that uses shared
   memory can mess up the CPU by misaddressing itself into an area that
   the CPU doesn't expect it to be in. If it writes its own patterns into
   some of that shared memory, your CPU can once again be presented with
   an insane memory reference and it will react accordingly. So, when you
   open the machine to see what you can do about the memory, try pulling
   all non-essential cards that use any shared memory (multiport cards,
   etc. use shared memory. Any card that does will usually show that in
   "hwconfig"). If you aren't sure, just pull anything that you don't
   need to boot- if the problem goes away, one of those cards is the
   problem; put them back one at a time until you know which one.

   It's also possible that you just need patches- some of these crashes
   are caused by problems in the OS- be sure to search the TA database
   for symptoms and messages that match yours, and be sure that you do
   have the recommended minimum patches for your OS version.

   You can do a little panic analysis yourself:
   http://stage.caldera.com/cgi-bin/ssl_reference?106181 explains how to
   determine if panics are consistent, that is, are they happening in the
   same kernel routine. If they are, your problem still could be
   hardware, but now you will have more info to narrow it down. If the
   panics are inconsistent, it's surely hardware.

   It's also possible that a bad CPU will cause the same symptoms because
   it gets the info it read from RAM scrambled internally. That's much
   more rare: the CPU's self-test and usually would halt long before
   you'd get to boot. Of course, a bad motherboard can cause good RAM to
   deliver bad bits to a good CPU also, but again the POST (Power On Self
   Tests) will usually catch this sort of thing unless it is very subtle.

   A bad driver can also do this by trampling flowerbeds (stepping on RAM
   that the CPU needs for its own sanity). If the system has been running
   up to this point you can usually discount that, but if you've just
   installed a new driver, this could be the cause. Try booting
   "unix.old"; if that works, the new driver could very well be at fault.

   Finally, disk corruption can cause otherwise good code to be read
   incorrectly from disk, which ends up being the same as a bad driver:
   misread bits send the kernel off on a rampage where it ultimately
   steps on its own tail and panics. Using "unix.old" or other kernels
   you may have can sometimes get around this at least long enough to
   save the data. Emergency Boot Floppies can also help here, and if you
   don't have those, you can break into the original install disks boot
   and get at the drive with that. The method for doing that varies with
   the release of SCO. With modern SCO, just type "tools" at the install
   floppy boot prompt. See SCO FAQ's for a listing of different methods
   for older versions.

   If it is disk corruption, and can't be gotten around with alternate
   kernels or boot floppies, the disk recovery guys can suck your data
   down to cdrom or other media:
     * Data Recovery Labs
     * Ontrack
     * Excalibur

   It's unlikely that you'd need to go to this extent for a trap E
   problem, though:if the disk isn't obviously trashed in other ways, a
   local problem that happens to be in the kernel tracks should be able
   to be gotten around with one of the other methods suggested above.

   colin@rudaz.co.uk added this:
     _________________________________________________________________

   The problem with the above article though is that it doesn't take into
   account ECC protected memory as you commonly find on Intel servers;
   Single bit failures are corrected by hardware and are invisible to the
   operating system, when multiple bit failures occur then a hardware NMI
   should be raised and subsequently caught by the OpenServer nmi kernel
   driver, resulting in a PANIC message that contains "FATAL:Parity error
   address unknown".

   To date I have managed to get Caldera to include a second note on this
   on this subject,

   http://stage.caldera.com/cgi-bin/ssl_reference?111621

   rudaz
     _________________________________________________________________

   Publish your articles, comments, book reviews or opinions here!

--
tony@aplawrence.com Unix/Linux resources: http://aplawrence.com
Inexpensive phone/email support
Download Free SCO Skills Test: http://pcunix.com/skilltests.html

Relevant Pages

  • Re: Slow performance
    ... I realize that the pagefile usage is excessive and I need to reduce memory ... I've never messed with those settings before. ... Peak are greatly in excess of the installed RAM. ... The Disk Defragmenter report shows you have minimal free disk space. ...
    (microsoft.public.windowsxp.perform_maintain)
  • Re: What can I check to fix system performance?
    ... it seems you have plenty of memory available: ... copies of files you have read of written lately, in a cache, in case ... processes per CPU, or 40 in all. ... Consider the disk structure. ...
    (comp.os.linux.setup)
  • Re: How does the ZIP chip work?
    ... the orig 6502 has a 500ns cycle to access memory. ... the zipchip has internal ram. ... Is there a possibility that 6502 CPU chip and 65C02 CPU chip can run at 14MHz without using accelerator? ... Any time they interface with the Apple II, they do so at the 1MHz speed. ...
    (comp.sys.apple2)
  • Re: IPAQ RX1955 or IPAQ 4150
    ... I purchased a 1GB high speed SD card for memory expansion, ... so the low amount of memory included ... Windows Media Player 10 running in the background, the CPU was idling ... only about 15 MB of RAM left. ...
    (microsoft.public.pocketpc)
  • Re: Diagnosing extreme fps drop
    ... It's probably the game loading from disk all kinds of new textures on ... or competition for RAM by other memory hog apps like MS ... what the current and peak commit size shows in relation to the physical ...
    (alt.games.warcraft)