Re: /var mysteriously full - only 7M used - df reports 252M used

From: Bill Vermillion (bv_at_wjv.com)
Date: 03/24/05

  • Next message: Pawel Jakub Dawidek: "Re: Gmirror on boot" 
    Date: Thu, 24 Mar 2005 17:55:01 GMT

    In article <IDvAIy.1Bp9@wjv.com>, Bill Vermillion <bv@wjv.com> wrote:
    >In article <1111640524.586382.311690@z14g2000cwz.googlegroups.com>,
    >Royce <mroyce@gmail.com> wrote:
    >>Hi all,
    >>
    >>I've got a server running 4.10-release and its /var is mysteriously
    >>full. See below - there's not much on it... du reports 7M used, and
    >>I've double checked there isn't much there (pretty pristine server, not
    >>yet in prod'n). system definitely thinks it's full - won't let me copy
    >>things onto it and postfix is choking.
    >>
    >>Appreciate any tips you have before I go to the trouble of blindly
    >>moving /var to another disk. fsck results look suspicious ... maybe
    >>it's time to drive to the colo.
    >>
    >># du -ks /var
    >>7566 /var
    >>
    >># df -h /var
    >>Filesystem Size Used Avail Capacity Mounted on
    >>/dev/ad0s1e 252M 252M -20.1M 109% /var
    >>
    >># df -h
    >>Filesystem Size Used Avail Capacity Mounted on
    >>/dev/ad0s1a 126M 57M 59M 49% /
    >>/dev/ad0s1f 252M 20M 212M 8% /tmp
    >>/dev/ad0s1g 72G 1.8G 64G 3% /usr
    >>/dev/ad0s1e 252M 252M -20.1M 109% /var
    >>/dev/ad2s1c 73G 2.0K 67G 0% /disk1
    >>procfs 4.0K 4.0K 0B 100% /proc
    >>
    >># fsck /var
    >>root@zoe root# fsck /var
    >>** /dev/ad0s1e (NO WRITE)
    >>** Last Mounted on /var
    >>** Phase 1 - Check Blocks and Sizes
    >>** Phase 2 - Check Pathnames
    >>** Phase 3 - Check Connectivity
    >>** Phase 4 - Check Reference Counts
    >>UNREF FILE I=345 OWNER=root MODE=100644
    >>SIZE=119308258 MTIME=Mar 23 20:59 2005
    > ^^^^^^^^^^^^^
    >
    >That one file is using 1/2 of your var. Since it shows unref the
    >typical result of this is removing something like a log file, which
    >an application has open. So rm'ing the file only removes the
    >visible link.
    >
    >>CLEAR? no
    >
    >You should have said "YES". But since I see NO WRITE above
    >that means you did the fsck while /var was mounted. You need
    >to unmount /var and run fsck and then your problems will go away.
    >
    >So - now the question - did you ever manually remove something like
    >a log file to get free space - and did not stop and restart the ap
    >that had it opened.
    >
    >Stoping and restarting syslog >might< cure this too, if it's a
    >program run under syslogd
    >
    >>UNREF FILE I=348 OWNER=root MODE=100644
    >>SIZE=136970238 MTIME=Mar 23 20:59 2005
    >>CLEAR? no

    >That file is 136MB.

    >So between the two you have used up all of /var

    One short note I meant to add before.

    Learn to read and understand the output of your diagnostics and
    you'll be a giant step ahead.

    This is NOT meant as a criticism but as general advice. I'm still
    learning things about Unix all the time - and I've first made Unix
    my home system back in 1983. It's a lifetime learning experience.

    I've walked into Unix systems that I've never seen before, and
    looking at error messages, and then going to man pages that seemed
    appropriate, I've usually been able to get things going. Learning
    to read man pages is almost a black art at times, but it is well
    worth the effort.

    Bill 

    -- 
Bill Vermillion - bv @ wjv . com
  • Next message: Pawel Jakub Dawidek: "Re: Gmirror on boot"