summary: filesystem grows out of control

From: Carl Ma (carl_ma_at_scotiacapital.com)
Date: 06/27/03

  • Next message: Andreas Hoeschler: "Summary: Booting from disk mirror with Solstice DiskSuite"
    Date: Fri, 27 Jun 2003 11:45:34 -0400 (EDT)
    To: sunmanagers@sunmanagers.org
    
    

    Hi,

    Thanks for the replies from:

    Darren Dunham
    Hichael Morton
    Stephen Cawley
    Alex Ugolini
    Alan Pae

    Finally, I have to bring the machine to single user mode with basic filesystem mounted. There
    is one new, big file under one automount point. I still have no idea how such file created
    since the consensus is it is impossible to create file under mounted filesystem.

    thanks & have a good long weekend,

    carl

    :> Message: 10
    :> Date: Wed, 25 Jun 2003 17:52:46 -0400 (EDT)
    :> From: Carl Ma <carl_ma@scotiacapital.com>
    :> Subject: filesystem grows out of control
    :> To: sunmanagers@sunmanagers.org
    :> Message-ID: <200306252155.h5PLt4q05582@sbtormlr2.ScotiaCapital.com>
    :> Content-Type: TEXT/plain; charset=us-ascii
    :>
    :> Hello all,
    :>
    :> We have one production server - E4500 with T3+ diskarray running
    :> solaris 8 with
    :> latest patch level. It used to be very stable till today.
    :>
    :> %df -k -F ufs
    :> Filesystem kbytes used avail capacity Mounted on
    :> /dev/md/dsk/d0 6191949 3293302 2836728 54% /
    :> /proc 0 0 0 0% /proc
    :> fd 0 0 0 0% /dev/fd
    :> mnttab 0 0 0 0% /etc/mnttab
    :> swap 670536 0 670536 0% /var/run
    :> swap 671664 1128 670536 1% /tmp
    :> /dev/md/dsk/d1 400089861 200071534 200018327 50% /space
    :> /dev/md/dsk/d2 200149000 20987800 179161200 10% /space/data1
    :> /dev/md/dsk/d3 ... ... ... ... /space/data2
    :> ...
    :> /dev/md/dsk/dN .. ... ... ... /space/dataN
    :>
    :> We can see there are lots of local filesystems mounted under /space.Also
    :> hundreds of files,directories and automount/nfs mount points are
    :> also under
    :> /space directory.
    :>
    :> Today, the /space usage grows continuously,slowly but constantly.
    :> It grows about
    :> 10% since noon,which is very abnormal.
    :>
    :> I did the following check:
    :>
    :> 1."du -d -sk /space" every 5 seconds,which shows the usage is growing
    :> 2."du -d -sk /space/*" every 5 seconds,which gives me all the directories
    :> capacity information including nfs.(-d doesn't work??)
    :> 3."find /space -xdev -type f -mtime -1 -ls | sort -n -k7", it
    :> does give me the
    :> list of changed files on /space. There are a lot but none has big size.
    :> 4. failed to use tnfdump to analyze since this filesystem has
    :> lots of I/O. I am
    :> lost at certain stage. :-)
    :>
    :> My questions are:
    :> 1.may I just display the size of every directories under
    :> /space(d1),excluding
    :> NFS and directories crossing filesytem boundaries(du -dsk
    :> /space/* doesn't
    :> work)?
    :> 2.is it possible that someone hides data under mount points? how
    :> can I find it?
    :>
    :> Right now, I am working on a dirty script and would like to hear
    :> from others. I
    :> will summarize if solution is available.
    :>
    :> thanks a lot,
    :>
    :> carl
    :>
    :
    _______________________________________________
    sunmanagers mailing list
    sunmanagers@sunmanagers.org
    http://www.sunmanagers.org/mailman/listinfo/sunmanagers


  • Next message: Andreas Hoeschler: "Summary: Booting from disk mirror with Solstice DiskSuite"

    Relevant Pages

    • Re: fsck: broken file system with background check remains broken after bootup
      ... > to boot and then fsck your disk partitions. ... If the system ends with a bad filesystem, the background check may leave the ... (single user mode, rescue floppies, live CDs bootup etc.). ... single user mode), but the background check left the system broken..... ...
      (freebsd-stable)
    • Re: Startup problem
      ... When the kernel boots, it mounts the root filesystem (normally ... init that boots us into single user mode. ...
      (Fedora)
    • Re: Startup problem
      ... it mounts the root filesystem (normally ... > still do an fsck on the filesystem (cannot do it on a filesystem that is ... > init that boots us into single user mode. ... Of course, if fsck fails, we end ...
      (Fedora)
    • Re: fsck
      ... (and rebooting into single user mode again doesn't help). ... You cannot fsck a read-write mounted filesystem. ...
      (freebsd-stable)
    • Re: Large filesystem woes
      ... it appears it uses devfs in single user mode as ... Hmmm, in multiuser mode, your root filesystem is mounted as read-write ... so fdisk -u should work. ...
      (freebsd-questions)