Re: Phantom /var full messages

From: Bill Moran (wmoran_at_potentialtech.com)
Date: 09/09/04

  • Next message: Bart Silverstrim: "Re: Tar pitting automated attacks" 
    Date: Thu, 9 Sep 2004 13:03:33 -0400
To: Paul Schmehl <pauls@utdallas.edu>

    Paul Schmehl <pauls@utdallas.edu> wrote:
    > I'm running snort 2.1.3 and mysql 3.23.58 on FreeBSD 4.9 RELEASE. All
    > applications are built from ports.
    >
    > Periodically I get /var full messages and everything comes to a grinding
    > halt. The problem is, /var isn't full.
    >
    > df -h will show /var at 104%, but du -h /var shows /var at 40% (for
    > example).

    This typically happens because a file has been deleted, but some program
    still has it open. The filesystem can't actually free up the data blocks
    until no other programs are using them. But du doesn't see the usage
    because there's no longer a file there to attribute it to.

    > If I shut down snort and mysql, wait for a minute and then start
    > them back up, df agrees with du again.

    You've already done the first diagnostic step. You know that either snort
    or MySQL is keeping a file handle after a file is deleted.

    > The system works fine because only /var is full (although things can get
    > squirrelly if I let it go long enough because the system can't write to the
    > logs or the mail spool), so I can still ssh in and run utilities.

    Yup.

    > I suspect this is some sort of filehandle not being released issue, but I'm
    > not sure how to track it down. I've got lsof installed, but I'm not an
    > expert on it yet.
    >
    > Any hints would be welcomed. What's the best way to troubleshoot this
    > problem?

    First, if you could isolate it to just snort or just MySQL.

    Typically, folks have this problem because they try to rotate log files
    without restarting the program that's logging to them. The rotate program
    compresses the current log file into a new file, then deletes the original
    file ... but the program is still logging to it. Thus the space fills up,
    but there is no file to see the space in. Restarting the program doing
    the logging causes the old file to disappear, and a new log file to be
    created.

    On a guess, Snort would be the first thing I'd look at. However, MySQL
    can create a TON of data if logging is enabled, so you may want to look
    closely at it as well.

    HTH 

    -- 
Bill Moran
Potential Technologies
http://www.potentialtech.com
_______________________________________________
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
  • Next message: Bart Silverstrim: "Re: Tar pitting automated attacks"

    Relevant Pages

    • Re: Phantom /var full messages
      ... if you could isolate it to just snort or just MySQL. ... > without restarting the program that's logging to them. ... > compresses the current log file into a new file, ... I am using newsyslog.conf which *should* HUP processes when ...
      (freebsd-questions)
    • Re: Phantom /var full messages
      ... if you could isolate it to just snort or just MySQL. ... >> compresses the current log file into a new file, ... open a file and then delete it in order to have anonymous scratch space. ...
      (freebsd-questions)
    • Re: Logging components?
      ... I think Logging is pretty much individual, so that usualy everyone has their ... {Write Logged exception into the Log file? ... {Write exception with a short string description into the Global App Log ... > By logging components, I specifically DON'T MEAN components for catching ...
      (borland.public.delphi.thirdpartytools.general)
    • Re: [Snort-users] Snort DoS Fallacies
      ... If "a lot of people" are logging in ASCII mode then nobody is reading ... NO PRODUCTION SNORT DEPLOYMENT SHOULD EVER ... then that's no longer a production sensor, ...
      (Bugtraq)
    • Re: Annoying 1202 errors
      ... Enabling this logging tracks all changes and settings applied to the machine ... Name: UserEnvDebugLevel ... I recommend that you remove any existing log file and then use gpupdate ... feature GUEST. ...
      (microsoft.public.win2000.group_policy)