Strange case of root filesystem corruption

• Previous message: Aaron: "Re: Configuring multiple monitors"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

To: freebsd-questions@freebsd.org
Date: Sat, 17 Sep 2005 21:54:14 +0200

First of all, all low level diagnostic tools I've used on my drive show
no signs of hardware error. The shutdown prior to when the problems
started went smooth, without any error messages.

Yesterday GRUB would suddenly not display the boot menu anymore. After
some investigation I discovered that all files in /boot/grub had turned
into symlinks to locale files, as seen in the following ufs2tools dump:

Jun 22,2005 01:16:24 <DIR> .
Jul 16,2005 00:41:53 <DIR> ..
May 11,2005 18:23:10 30 menu.lst -> ../it_CH.ISO8859-1/LC_MONETARY
Jun 22,2005 01:16:04 201 menu.lst~
May 11,2005 18:23:10 30 stage1 -> ../it_IT.ISO8859-1/LC_MONETARY
May 11,2005 18:23:10 30 stage2 -> ../nl_BE.ISO8859-1/LC_MONETARY
May 11,2005 18:23:10 30 ufs2_stage1_5 ->
../nl_NL.ISO8859-1/LC_MONETARY

I could no longer log in via any of my accounts, so I tried adding a new
account in single-user mode. This gave an error message about "vnodes"
(or perhaps inodes, my system won't boot anymore, so I can't reproduce
the error) and a kernel panic followed by a reboot.

I then ran fsck, which reported a whole bunch of errors: wrong ref
wounts, something about things being bad with the superblock, and about
bad/missing things in general. Perhaps the strangest errors were about
extraneous links to files directly under / (more on this later). I
answered 'yes' to all fix/salvage questions from fsck.

After that FreeBSD would not boot at all, causing a reboot before the
beastie menu.

Looking at filesystems on the disk with the free ufs2tools program, only
the root filesystem seems to be affected by the corruption. Here's what
/ looks like (seems fsck was right about those extraneous links):

Jan 01,1970 01:00:00 0 .
Sep 16,2005 01:07:25 <DIR> .
Jan 01,1970 01:00:00 0 ..
Sep 16,2005 01:07:25 <DIR> ..
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Jan 01,1970 01:00:00 0
Nov 05,2004 02:27:17 801 .cshrc
Nov 05,2004 02:27:17 801 .cshrc
Nov 05,2004 02:27:17 251 .profile
Nov 05,2004 02:27:17 251 .profile
Feb 17,2005 18:21:41 <DIR> .snap
Jan 01,1970 01:00:00 0 .snap
May 11,2005 18:31:27 6192 COPYRIGHT
May 11,2005 18:31:27 6192 COPYRIGHT
May 11,2005 18:21:44 <DIR> bin
Jan 01,1970 01:00:00 0 bin
Jan 01,1970 01:00:00 0 boot
Jul 16,2005 00:41:53 <DIR> boot
Feb 17,2005 18:21:48 <DIR> cdrom
Jan 01,1970 01:00:00 0 cdrom
Feb 17,2005 18:29:51 10 compat -> usr/compat
Feb 17,2005 18:29:51 10 compat -> usr/compat
Jan 01,1970 01:00:00 0 dev
Feb 17,2005 18:21:41 <DIR> dev
Jan 01,1970 01:00:00 0 dist
Feb 17,2005 18:21:50 <DIR> dist
Sep 16,2005 01:07:22 4096 entropy
Sep 15,2005 21:27:31 <DIR> etc
Jan 01,1970 01:00:00 0 etc
Mar 10,2005 21:53:07 655360 festival-synthesis-.core
Mar 10,2005 21:53:07 655360 festival-synthesis-.core
Feb 17,2005 18:21:42 <DIR> home
Jan 01,1970 01:00:00 0 home
Mar 23,2005 22:03:23 <DIR> ipod
Jan 01,1970 01:00:00 0 ipod
Jan 01,1970 01:00:00 0 lib
Jul 16,2005 04:50:21 <DIR> lib
May 11,2005 18:21:39 <DIR> libexec
Jan 01,1970 01:00:00 0 libexec
Sep 16,2005 00:05:35 <DIR> lost+found
Nov 05,2004 02:23:32 <DIR> mnt
Jan 01,1970 01:00:00 0 mnt
Jan 01,1970 01:00:00 0 pappas
May 22,2005 14:50:23 <DIR> pappas
Apr 10,2005 21:10:07 <DIR> proc
Jan 01,1970 01:00:00 0 proc
May 11,2005 18:22:29 <DIR> rescue
Jan 01,1970 01:00:00 0 rescue
Sep 10,2005 13:08:14 <DIR> root
Jan 01,1970 01:00:00 0 root
May 29,2005 13:26:54 <DIR> s
Jan 01,1970 01:00:00 0 s
Jan 01,1970 01:00:00 0 sbin
May 11,2005 18:23:56 <DIR> sbin
Feb 17,2005 18:21:48 <DIR> stand
Jan 01,1970 01:00:00 0 stand
Jun 15,2005 18:10:54 11 sys -> usr/src/sys
Jun 15,2005 18:10:54 11 sys -> usr/src/sys
Sep 15,2005 21:27:32 <DIR> tmp
Jan 01,1970 01:00:00 0 tmp
Jan 01,1970 01:00:00 0 usr
Feb 17,2005 18:21:45 <DIR> usr
Jan 01,1970 01:00:00 0 var
Feb 17,2005 18:21:46 <DIR> var
Mar 06,2005 15:16:20 <DIR> vcdrom
Jan 01,1970 01:00:00 0 vcdrom
Jan 01,1970 01:00:00 0 win
Aug 27,2005 12:12:03 <DIR> win

Subfolders of / on the same filesystem are affected as well. I can't see
any corruption inside files though.

My problems wouldn't be so alarming if not for the fact that two friends
of mine have had very similar problems (but unfortunately never told
anyone about it). Their root filesystem suddenly went haywire on
seemingly fine disks for no apparent reason.

Some general info:

I had other filesystems (stored on the same and other disks) mounted on
/usr, /var, /home, and /win.

I'm running FreeBSD 5.4. My friends can't remember which version they
ran at the time, but they both think it was 5.x.

In case they might be helpful, I'll include the contents of /etc and
/boot (both show signs of corruption), as well as the output of the
bsdlabel program (part of the ufs2tools package) when ran on the disk in
question:

/etc:

Sep 15,2005 21:27:31 <DIR> .
Sep 16,2005 01:07:25 <DIR> ..
Jun 06,2005 12:49:27 0 COPYRIGHT
Aug 16,2005 13:54:44 <DIR> X11
Feb 17,2005 18:22:04 12 aliases -> mail/aliases
Mar 30,2005 15:01:01 65536 aliases.db
Nov 05,2004 02:27:16 209 amd.map
Nov 05,2004 02:27:16 1234 apmd.conf
Mar 12,2005 16:33:49 225 auth.conf
Nov 05,2004 02:27:16 231 auth.conf~
Mar 12,2005 22:12:43 0 blocks.fb
Feb 17,2005 18:22:02 <DIR> bluetooth
Apr 28,2005 21:09:42 7 console
Nov 05,2004 02:27:16 737 crontab
Nov 05,2004 02:27:16 108 csh.cshrc
Nov 05,2004 02:27:16 481 csh.login
Nov 05,2004 02:27:16 110 csh.logout
May 19,2005 18:25:36 <DIR> defaults
May 11,2005 18:29:51 4846 devd.conf
Nov 05,2004 02:27:16 2071 devfs.conf
Aug 20,2005 18:55:00 23 dhclient-enter-hooks
Aug 20,2005 18:54:57 23 dhclient-enter-hooks~
Nov 05,2004 02:27:16 267 dhclient.conf
Nov 05,2004 02:27:16 6621 disktab
Nov 05,2004 02:27:17 0 dumpdates
Nov 05,2004 02:27:16 142 fbtab
Sep 03,2005 22:43:21 509 fstab
Aug 27,2005 11:26:13 511 fstab~
Nov 05,2004 02:27:16 245 ftpusers
Nov 05,2004 02:27:16 5904 gettytab
May 11,2005 18:26:29 <DIR> gnats
Aug 04,2005 16:42:42 487 group
Feb 23,2005 18:52:40 443 group~
May 07,2005 20:08:15 25 host.aliases
May 07,2005 20:04:04 21 host.aliases~
Feb 17,2005 19:57:39 60 host.conf
Feb 18,2005 20:07:35 1107 hosts
Nov 05,2004 02:27:16 3186 hosts.allow
Nov 05,2004 02:27:16 111 hosts.equiv
Nov 05,2004 02:27:16 99 hosts.lpd
Feb 17,2005 22:51:26 1106 hosts~
Apr 10,2005 22:15:42 5378 inetd.conf
Apr 10,2005 22:12:07 5380 inetd.conf~
Feb 17,2005 18:22:02 <DIR> isdn
May 11,2005 18:23:10 30 libfstab -> ../en_AU.ISO8859-1/LC_MONETARY
Aug 27,2005 23:08:06 511 libfstab~
Feb 27,2005 22:23:18 390 libmap.conf
Feb 17,2005 20:24:09 675 localtime
Nov 05,2004 02:27:16 619 locate.rc
Nov 05,2004 02:27:16 1847 login.access
Mar 12,2005 16:29:14 6522 login.conf
Mar 12,2005 16:30:16 65536 login.conf.db
Nov 05,2004 02:27:16 6522 login.conf~
Nov 05,2004 02:27:16 564 mac.conf
Jul 31,2005 00:36:29 <DIR> mail
Nov 05,2004 02:27:16 106 mail.rc
Mar 23,2005 12:59:50 117 mailcap
Mar 23,2005 12:57:00 117 mailcap~
Jul 20,2005 13:19:42 13206 make.conf
Jul 16,2005 04:08:09 13132 make.conf.bak
Mar 19,2005 11:40:53 13132 make.conf2~
Jul 20,2005 13:19:14 13206 make.conf~
Jul 20,2005 13:19:14 1087 manpath.config
Jul 16,2005 04:08:09 946 manpath.config.bak
Aug 04,2005 16:42:42 1884 master.passwd
Mar 12,2005 17:07:09 1750 master.passwd~
Jun 06,2005 12:45:59 0 motd
May 11,2005 18:16:36 1111 motd.old
May 11,2005 18:27:42 <DIR> mtree
May 02,2005 19:44:26 153 named.conf~
Sep 15,2005 21:27:31 21 namedb -> /var/named/etc/namedb
Nov 05,2004 02:27:16 783 netconfig
Nov 05,2004 02:27:16 2362 netstart
May 11,2005 18:30:09 13696 network.subr
Nov 05,2004 02:27:16 365 networks
Mar 09,2005 21:33:05 1932 newsyslog.conf
Mar 09,2005 21:28:46 1935 newsyslog.conf~
Nov 05,2004 02:27:16 1701 nsmb.conf
Feb 17,2005 19:57:39 113 nsswitch.conf
Nov 05,2004 02:23:32 <DIR> ntp
Nov 05,2004 02:27:16 432 opieaccess
Feb 17,2005 20:01:00 0 opiekeys
Feb 17,2005 18:22:02 <DIR> pam.d
Aug 04,2005 16:42:42 1582 passwd
Feb 22,2005 19:50:54 1394 passwd~
May 11,2005 18:30:56 5814 pccard_ether
Feb 17,2005 18:22:03 <DIR> periodic
Mar 21,2005 11:50:51 137 periodic.conf
Nov 05,2004 02:27:16 3567 pf.conf
Nov 05,2004 02:27:16 22556 pf.os
Nov 05,2004 02:27:16 293 phones
Feb 17,2005 18:22:03 <DIR> ppp
Apr 23,2005 16:34:38 2152 printcap
Apr 23,2005 16:32:27 2152 printcap~
May 07,2005 20:07:03 842 profile
May 07,2005 20:06:58 841 profile~
May 11,2005 18:30:11 5807 protocols
Aug 04,2005 16:42:42 40960 pwd.db
Nov 05,2004 02:27:16 2716 rc
May 11,2005 18:30:16 5298 rc.bsdextended
Aug 20,2005 19:09:38 710 rc.conf
Jun 18,2005 19:06:23 693 rc.conf.orig
Aug 20,2005 19:09:34 710 rc.conf~
May 11,2005 18:29:47 <DIR> rc.d
Nov 05,2004 02:27:16 9569 rc.firewall
Nov 05,2004 02:27:16 8905 rc.firewall6
Nov 05,2004 02:27:16 2165 rc.resume
Nov 05,2004 02:27:16 5785 rc.sendmail
May 11,2005 18:30:27 3211 rc.shutdown
May 11,2005 18:30:31 30905 rc.subr
Nov 05,2004 02:27:16 2209 rc.suspend
Nov 05,2004 02:27:16 2392 remote
Sep 15,2005 21:27:31 52 resolv.conf
Aug 20,2005 18:32:44 23 resolv.conf.no
Jul 15,2005 21:07:06 69 resolv.conf~
Feb 17,2005 18:22:04 13 rmt -> /usr/sbin/rmt
Nov 05,2004 02:27:16 1674 rpc
Nov 05,2004 02:27:16 73410 services
Jun 02,2005 12:40:40 276 shells
Nov 05,2004 02:23:32 <DIR> skel
May 11,2005 18:23:10 30 spwd.db -> ../pt_PT.ISO8859-1/LC_MONETARY
Feb 17,2005 20:00:07 <DIR> ssh
May 11,2005 18:29:49 <DIR> ssl
Nov 05,2004 02:27:16 367 sysctl.conf
Jun 06,2005 12:37:53 1352 syslog.conf
May 02,2005 20:24:16 1344 syslog.conf~
Feb 17,2005 18:22:04 23 termcap -> /usr/share/misc/termcap
Apr 28,2005 21:09:29 7 tty
Mar 12,2005 17:18:17 7670 ttys
Nov 05,2004 02:27:16 7658 ttys~
May 11,2005 18:30:52 2032 usbd.conf
Jun 16,2005 22:31:19 0 wall_cmos_clock

/boot:

ul 16,2005 00:41:53 <DIR> .
Sep 16,2005 01:07:25 <DIR> ..
May 11,2005 18:23:16 7007 beastie.4th
Mar 11,2005 23:53:02 6757 beastie.4th~
May 11,2005 18:23:15 8192 boot
May 11,2005 18:23:15 512 boot0
May 11,2005 18:23:15 512 boot0sio
May 11,2005 18:23:15 512 boot1
May 11,2005 18:23:15 7680 boot2
May 11,2005 18:23:15 1184 cdboot
May 11,2005 18:23:16 <DIR> defaults
Nov 05,2004 02:27:17 1982 device.hints
May 11,2005 18:23:16 2249 frames.4th
Jun 22,2005 01:16:24 <DIR> grub
Jun 15,2005 18:11:05 <DIR> kernel
Jun 15,2005 15:59:52 <DIR> kernel.old
May 11,2005 18:23:16 262144 loader
May 11,2005 18:23:16 7772 loader.4th
Jul 16,2005 00:41:53 58 loader.conf
May 11,2005 18:23:10 30 loader.conf.orig ->
../no_NO.ISO8859-1/LC_MONETARY
Feb 17,2005 18:29:51 0 loader.conf~
May 11,2005 18:23:16 13893 loader.help
Nov 05,2004 02:26:31 212992 loader.old
Nov 05,2004 02:26:31 370 loader.rc
May 11,2005 18:23:15 512 mbr
Jul 16,2005 00:41:48 <DIR> modules
May 11,2005 18:23:16 264192 pxeboot
May 11,2005 18:23:16 692 screen.4th
May 11,2005 18:23:16 36440 support.4th

bsdlabel output:

# Drive 0, Slice 1:
type: ESDI
disk:
label:
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 4865
sectors/unit: 78165360
rpm: 3600
interleave: 1
trackskew: 0
cylinderskew: 0
headswitch: 0 # milliseconds
track-to-track seek: 0 # milliseconds
drivedata: 0

8 partitions:
# size offset fstype [fsize bsize bps/cpg]
  a: 1048576 0 4.2BSD 0 0 0
  b: 3145728 1048576 swap
  c: 78164624 0 unused 0 0 # "raw" part,
don't edit
  d: 2097152 4194304 4.2BSD 0 0 0
  e: 62914560 6291456 4.2BSD 0 0 0
  f: 8957952 69206016 4.2BSD 0 0 0

I haven't lost any important data because of these problems, but others
might..
/Ulf
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"

• Previous message: Aaron: "Re: Configuring multiple monitors"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]