Summary: File size issues

Rick.Brashear_at_ercgroup.com
Date: 10/23/03

  • Next message: Milan Matesin: "SUMMARY: v240 sol 9/08 install problem" 
    To: sunmanagers@sunmanagers.org
Date: Thu, 23 Oct 2003 09:46:55 -0500

    Thanks to the following repondents:
    Darren Dunham [ddunham@taos.com]
    Casper *** [casper@holland.sun.com]
    Rich Kulawiec [rsk@gsp.org]
    Thiagarajan, Karthikeyan [kthiagarajan@doitt.nyc.gov]
    Dave Gagliardi [dgagliardi@zonecms.net]
    Chandramohan Kowdlex , Tidel Park - Chennai [ChandramohanW@msdc.hcltech.com]
    Bill Voight [Bill.Voight@fcc.gov]

    Darren Dunham provided the most definitive information with the following:

    The application itself maintains a filehandle and a current pointer
    within it... After some time, you've got this..

    application: FH=4, FP=1234568
    0 1M
    [xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] size=1234567 length=1234567

    The application has written 1.2MB to the file, and knows that the next
    byte goes to position 1234568. You nuke the file out from under it and
    it writes another 10 bytes, but starting at the same position...

    application: FH=4, FP=1234578
    0 1M
    [ x] size=10 length=1234577

    The size of the file is small (just whatever blocks are necessary to
    hold the 10 bytes), but the length is big. You can't change that
    without signalling the application somehow to close and then reopen a
    new file. Usually by restarting the app, or sometimes it's written to
    rotate the logs on signal like HUP.

    Casper *** offered this response that proved to be true:

    An open file descriptor has an offset associated with it; the process will
    continue to write at that offset; the first write after the truncation will
    cause a large hole to appear at the start of the file, filled with NULLs.

    If you copy the file then you run out of disk space but the large
    files themselves do not take space as they have holes int hem which read
    as zeros.

    The application which writes the files needs to be changed to use O_APPEND
    when opening the file; this will cause all writes to do a seek to the
    actual end of the file.

    The Netegrity support staff (siteminder application) confirmed that Casper
    ***'s suggestion has indeed been
    applied in their 5.5 release to circumvent this problem in that version.

    The issue we are dealing with is that the applications utilizing this
    product is a global insurance app for which
    there is not a convenient window to cycle the app and rotate the logs.

    The solution for our monitoring concerns is that I wrote a perl script that
    utilizes du to determine the actual disk
    usage we have used to replace the original sitescope script that used df.

    Original inquiry:
    I have a file issue I do not understand.

    O/S: Solaris 8 Generic_108528-19
    Pltfrm: E10K domain

    We have an actively running web application whose siteminder webagent is
    logging to siteminder.log. I was under the impression that you could do
    something like:

    cp siteminder.log otherlocation.log
    cat /dev/null > siteminder.log and free up disk space.

    It seems that shortly after I do this the filesystem reports that the log
    file is at the size it was when I began and still growing. df reflects this
    as well.

    I wrote a script that captures a particular case:

    + ls -l siteminder.log
    -rwxrwxr-x 1 kintana iplanet 109625128 Oct 20 14:37 siteminder.log
    + wc -l siteminder.log
       49312 siteminder.log
    + wc -c siteminder.log
     109629860 siteminder.log
    + cp -p siteminder.log /var/tmp/kintana
    + cat /dev/null
    + 1> siteminder.log
    + ls -l siteminder.log
    -rwxrwxr-x 1 kintana iplanet 0 Oct 20 14:38 siteminder.log
    + wc -l siteminder.log
           0 siteminder.log
    + wc -c siteminder.log
           0 siteminder.log
    + sleep 60
    + ls -l siteminder.log
    -rwxrwxr-x 1 kintana iplanet 109635695 Oct 20 14:39 siteminder.log
    + wc -l siteminder.log
          65 siteminder.log
    + wc -c siteminder.log
    109647732 siteminder.log

    You can see that the line count appears that it may be correct but the
    character count shows the file to still be very large. We have often dealt
    with this in the past successfully. But now it seems to be ineffective.
    Can someone help? We are running out of disk space fast....

    g ERC
    _______________________________
    Rick Brashear
    Server Systems
    Information Technology Department
    Employers Reinsurance Corporation
    5200 Metcalf
    Overland Park, Kansas 66201
    * 913 676-6418
    * rick.brashear@ercgroup.com
    _______________________________________________
    sunmanagers mailing list
    sunmanagers@sunmanagers.org
    http://www.sunmanagers.org/mailman/listinfo/sunmanagers

  • Next message: Milan Matesin: "SUMMARY: v240 sol 9/08 install problem"