Solaris 9 will not Boot following disk failure

From: Samurai-AL (alind_at_joho.com)
Date: 03/25/05 

Date: 24 Mar 2005 20:46:53 -0800

Firstly, excuse my ignorance as I am fairly new to the Unix World.

I have a SUN Fire V240 with four 36GB internal hot plugguble SCSI hard
drives installed.

I used Solaris Volume Manager to create the mirroring.
When I remove DISK0 and reboot the server will not boot (see error
below).

I have the latest 9_Recommended.zip patches installed

I ran this command:
installboot /usr/platform/`uname -i`/lib/fs/ufs/bootblk
/dev/rdsk/c1t1d0s0

And set the monitor ROM to boot off DISK1:

ok printenv
Variable Name Value Default Value
boot-device disk0 disk1 disk net
use-nvramrc? false false

Disk0 and Disk1 are partitioned like this:

     0 / 1000
     1 /var 16627
     2 overlap 34730
     3 swap 4000
     4 /opt 6000
     5 /usr 6000
     6 /export/home 1000
     7 100

Disk2 and Disk3 are partitioned like this:

     0 33820
     1 100

I have 2 Metadatabase copies on each disk in the spare 100 Mb
partitions (total of 8 databases).

bash-2.05# metastat -p
d107 -m d20 d30 1
d20 1 1 c1t2d0s0
d30 1 1 c1t3d0s0
d106 -m d6 d16 1
d6 1 1 c1t0d0s6
d16 1 1 c1t1d0s6
d105 -m d5 d15 1
d5 1 1 c1t0d0s5
d15 1 1 c1t1d0s5
d104 -m d4 d14 1
d4 1 1 c1t0d0s4
d14 1 1 c1t1d0s4
d103 -m d3 d13 1
d3 1 1 c1t0d0s3
d13 1 1 c1t1d0s3
d100 -m d0 d10 1
d0 1 1 c1t0d0s0
d10 1 1 c1t1d0s0
d101 -m d1 d11 1
d1 1 1 c1t0d0s1
d11 1 1 c1t1d0s1
bash-2.05#

contents of /etc/vfstab
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/md/dsk/d103 - - swap - no -
/dev/md/dsk/d100 /dev/md/rdsk/d100 / ufs 1
no -
/dev/md/dsk/d105 /dev/md/rdsk/d105 /usr ufs 1
no -
/dev/md/dsk/d101 /dev/md/rdsk/d101 /var ufs 1
no -
/dev/md/dsk/d106 /dev/md/rdsk/d106 /export/home ufs
2 yes -
/dev/md/dsk/d104 /dev/md/rdsk/d104 /opt ufs 2
yes -
/dev/md/dsk/d107 /dev/md/rdsk/d107 /var/CommuniGate ufs
    2 yes -
swap - /tmp tmpfs - yes -

bash-2.05#
bash-2.05# Mar 25 12:48:55 cgp rmclomv: DISK @ HDD0 has been removed.

SC Alert: DISK @ HDD0 has been removed.

bash-2.05# reboot
Mar 25 12:48:55 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:48:55 cgp SCSI transport failed: reason 'incomplete':
retrying command
Mar 25 12:48:57 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:48:57 cgp disk not responding to selection
Mar 25 12:48:59 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:48:59 cgp disk not responding to selection
Mar 25 12:49:02 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:02 cgp disk not responding to selection
Mar 25 12:49:04 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:04 cgp disk not responding to selection
Mar 25 12:49:07 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:07 cgp disk not responding to selection
Mar 25 12:49:07 cgp md_stripe: WARNING: md: d0: write error on
/dev/dsk/c1t0d0s0
Mar 25 12:49:09 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:09 cgp disk not responding to selection
Mar 25 12:49:11 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:11 cgp disk not responding to selection
Mar 25 12:49:11 cgp md_stripe: WARNING: md: d0: write error on
/dev/dsk/c1t0d0s0
Mar 25 12:49:13 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:13 cgp disk not responding to selection
Mar 25 12:49:13 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:13 cgp offline
Mar 25 12:49:15 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:15 cgp disk not responding to selection
Mar 25 12:49:17 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:17 cgp disk not responding to selection
Mar 25 12:49:19 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:19 cgp disk not responding to selection
Mar 25 12:49:21 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:21 cgp disk not responding to selection
Mar 25 12:49:24 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:24 cgp disk not responding to selection
Mar 25 12:49:26 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:26 cgp disk not responding to selection
Mar 25 12:49:28 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:28 cgp disk not responding to selection
Mar 25 12:49:30 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:30 cgp disk not responding to selection
Mar 25 12:49:32 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:32 cgp disk not responding to selection
Mar 25 12:49:34 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:34 cgp disk not responding to selection
Mar 25 12:49:36 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:36 cgp disk not responding to selection
Mar 25 12:49:36 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:36 cgp offline
Mar 25 12:49:36 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:36 cgp i/o to invalid geometry
Mar 25 12:49:36 cgp md_stripe: WARNING: md: d0: write error on
/dev/dsk/c1t0d0s0
Mar 25 12:49:38 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:38 cgp disk not responding to selection
Mar 25 12:49:38 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:38 cgp offline
Mar 25 12:49:40 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:40 cgp disk not responding to selection
Mar 25 12:49:42 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:42 cgp disk not responding to selection
Mar 25 12:49:44 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:44 cgp disk not responding to selection
Mar 25 12:49:44 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:44 cgp offline
Mar 25 12:49:44 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:44 cgp i/o to invalid geometry
Mar 25 12:49:44 cgp md_stripe: WARNING: md: d0: write error on
/dev/dsk/c1t0d0s0
Mar 25 12:49:44 cgp ufs: WARNING: Error writing ufs log
Mar 25 12:49:44 cgp ufs: WARNING: ufs log for / changed state to Error
Mar 25 12:49:44 cgp ufs: WARNING: Please umount(1M) / and run fsck(1M)
Mar 25 12:49:46 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:46 cgp disk not responding to selection
Mar 25 12:49:46 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:46 cgp offline
Mar 25 12:49:48 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:48 cgp disk not responding to selection
Mar 25 12:49:50 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:50 cgp disk not responding to selection
Mar 25 12:49:53 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:53 cgp disk not responding to selection
Mar 25 12:49:55 cgp scsi: WARNING: /pci@1c,600000/scsi@2/sd@0,0 (sd30):
Mar 25 12:49:55 cgp disk not responding to selection
reboot: Cannot find /usr/lib/ld.so.1
Killed
bash-2.05#
bash-2.05# init 0
bash: init: command not found
bash-2.05#
bash-2.05# reboot
bash: /usr/sbin/reboot: I/O error
bash-2.05# Debugging requested; hardware watchdog disabled; reboot to
re-enable.
Type 'go' to resume
ok
ok boot

SC Alert: SC Request to send Break to host.

SC Alert: Host System has Reset

Sun Fire V240, No Keyboard
Copyright 1998-2003 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.11.4, 2048 MB memory installed, Serial #56796109.
Ethernet address 0:3:ba:62:a3:cd, Host ID: 8362a3cd.

Rebooting with command: boot
Boot device: disk1 File and args:
SunOS Release 5.9 Version Generic_118558-04 64-bit
Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
e_ddi_get_dev_info: Illegal major device number <-1>e_ddi_get_dev_info:
Illegal major device number <-1>
panic[cpu0]/thread=140a000: BAD TRAP: type=31 rp=14090b0
addr=30800063500 mmu_fsr=0

: trap type = 0x31
addr=0x30800063500
pid=0, pc=0x1092da0, sp=0x1408951, tstate=0x4480001602, context=0x0
g1-g7: 130ef70, 7fffffff8, 30000063508, 0, 30000282fe0, 10, 140a000

0000000001408de0 unix:die+a4 (31, 14090b0, 30800063500, 0, 0, 0)
  %l0-3: 0000000000000000 0000030000063508 00000000014090b0
0000000001408fa8
  %l4-7: 0000000000000031 0000000000000000 ffffffffffffffff
ffffffffffffffff
0000000001408ec0 unix:trap+874 (14090b0, 0, 10000, 10200, 308, 0)
  %l0-3: 0000000000000001 0000000000000000 0000000001438a78
0000000000000031
  %l4-7: 0000000000000006 0000000000000001 0000000000000000
0000000000000000
0000000001409000 unix:ktl0+48 (0, ffffffff, 1409208, ffffffffffffffff,
0, 130ec4c)
  %l0-3: 0000000000000003 0000000000001400 0000004480001602
000000000102cd2c
  %l4-7: 00000300003e00e0 000003000046d428 0000000000000000
00000000014090b0
0000000001409150 md:md_call_strategy+58 (300010abea8, 1, 0,
ffffffffffffffff, 10, 130ec4c)
  %l0-3: 0000000001444368 0000000000000000 0000000000002000
0000000000000000
  %l4-7: 0000030000283000 ffffffffffffffff 0000000000000010
00000300010abe68
0000000001409220 md_stripe:md_stripe_strategy+368 (1, 30000282f78,
300010abe68, 0, 10, 300010adf80)
  %l0-3: 00000300010abea8 0000000000000000 00000300010adf28
0000000000002000
  %l4-7: 0000000000000010 0000000000000000 00000300010afee0
0000000000000000
00000000014092f0 md_mirror:mirror_read_strategy+5a8 (0, 1316bf4,
300010b1f40, 1, 30000272ec3, 14d9400)
  %l0-3: 00000300010afea8 00000300010b1ea8 0000000000000000
0000000000000010
  %l4-7: 0000000000000010 00000300010afee0 0000000000000000
00000300010d6000
00000000014093c0 md:mdstrategy+d0 (300010d6000, 140a000, 1409528, 9758,
10, 8)
  %l0-3: 0000030000069508 0000000000000001 0000000000000000
0000000000002000
  %l4-7: 00000300010d6000 00000300010d60b0 00000300004db1c0
00000300004a1d90
0000000001409470 genunix:bdev_strategy+90 (300010d6000, 1443558, 1, 0,
55, 30000041348)
  %l0-3: 00000000011c2ad4 0000030000d62ca0 0000000000000010
0000005500000064
  %l4-7: 00000300010d6000 0000030000d62cd0 00000300000148c0
0000000000001000
0000000001409540 genunix:bread_common+138 (30000041348, 5500000064, 10,
2000, 300004d3f28, 300004a1d98)
  %l0-3: 000000000142f030 0000030000041348 00000300010d6000
0000000000010000
  %l4-7: 00000300000146c8 00000300004d3f28 00000300004a1eb0
00000300004a1d98
0000000001409600 ufs:mountfs+14c (0, 1, 5500000064, 0, 300004d3f28, 1)
  %l0-3: 000000000118531c 0000030000041348 0000000000000000
0000000000000010
  %l4-7: 000000000146c188 0000005500000064 0000000000000000
0000000000000000
0000000001409740 ufs:ufs_mountroot+370 (146c188, 0, 0, 0, 0, 8)
  %l0-3: 0000000001183d78 0000000000000001 0000005500000064
0000000000000000
  %l4-7: 00000000000000b0 0000000000000000 0000000000000000
0000000000000000
0000000001409810 swapgeneric:rootconf+268 (0, 0, 4, 0, 0, 0)
  %l0-3: 00000000011998ec 0000000001438820 0000000000000000
000000000144bdf8
  %l4-7: 000000000144bc00 00000000014a4800 000003000000bf40
000003000029b000
00000000014098c0 unix:stubs_common_code+70 (3000029b000, 0, 4, 0, 0, 0)
  %l0-3: 000003000029b000 0000000000000000 000003000029b000
ffffffffffffffff
  %l4-7: 00000000000000b0 0000000001410dc8 0000000000000000
000000000144ed18
0000000001409970 genunix:vfs_mountroot+54 (0, 0, 0, 200, 14581b0, 0)
  %l0-3: 000000000144bc00 0000000001444400 0000000000002000
00000000014957a8
  %l4-7: 000000000149b400 0000000001411e68 000000000144c400
000000000144f400
0000000001409a20 genunix:main+90 (1409ba0, f0059c40, 1409ec0, 3987b8,
2000, 500)
  %l0-3: 0000000000000001 000000000140a000 0000000001412fd8
0000000000000000
  %l4-7: 0000000078002000 000000000039a000 00000000014a3c28
0000000001066878

skipping system dump - no dump device configured
rebooting...

Sun Fire V240, No Keyboard
Copyright 1998-2003 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.11.4, 2048 MB memory installed, Serial #56796109.
Ethernet address 0:3:ba:62:a3:cd, Host ID: 8362a3cd.

Rebooting with command: boot
Boot device: disk1 File and args:
SunOS Release 5.9 Version Generic_118558-04 64-bit
Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
e_ddi_get_dev_info: Illegal major device number <-1>e_ddi_get_dev_info:
Illegal major device number <-1>
panic[cpu0]/thread=140a000: BAD TRAP: type=31 rp=14090b0
addr=30800063500 mmu_fsr=0

: trap type = 0x31
addr=0x30800063500

... and so on continuously

When I re-insert DISK0 I get a normal boot:

ok boot
Boot device: disk0 File and args:
SunOS Release 5.9 Version Generic_118558-04 64-bit
Copyright 1983-2003 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed
WARNING: forceload of misc/md_sp failed
Hardware watchdog enabled
SC unretrieved msg MAR 25 03:50:35 2005 UTC [Host System has Reset]
configuring IPv4 interfaces: bge0.
Hostname: cgp
The system is coming up. Please wait.
checking ufs filesystems
/dev/md/rdsk/d104: is logging.
/dev/md/rdsk/d106: is logging.
/dev/md/rdsk/d107: is logging.
starting rpc services: rpcbind done.
Setting netmask of bge0 to 255.255.255.0
Setting default IPv4 interface for multicast: add net 224.0/4: gateway
cgp
syslog service starting.
Starting CommuniGate Pro
Mar 25 13:38:15 cgp htt[292]: Error : Another IIIMP Server might be
running.
volume management starting.
The system is ready.

cgp console login:

Any help appreciated!
Regards,
Allan