Re: Performance problem (I/O)

From: Bob Harris (harris_at_zk3.dec.com)
Date: 03/22/04 

Date: Mon, 22 Mar 2004 18:25:05 GMT

In article <c3n2v0$1etc$2@jeeves.eng.abbnm.com>,
 peter@abbnm.com (Peter da Silva) wrote:

> In article <harris-779E4E.23395720032004@cacnews.cac.cpqcorp.net>,
> Bob Harris <harris@zk3.dec.com> wrote:
> > I'm not sure I understand the question. I think, based on what you have
> > extracted above, you are wondering if AdvFS defers the 8K allocation
> > until after the frag has been created, and just what I/O really occurs.
> > Is this correct?
>
> Yes, and thanks for the detailed and interesting explanation. I think I
> followed it all, though of course I might be completely misinterpreting
> something so if I go ahead and turn off fragging and it makes things
> worse I won't blame you. :)

The most important thing to know about turning off small file frag'ing
is that it can consume a lot more disk space. If you have the disk
space, then it doesn't matter.

Also remember that turning off small file frag'ing only affects new
files or existing files that are extended. All other small files with
frag will keep their frags.

Attached is a script that may give you an estimate of how much extra
space will be needed if all the small files on the file system were
created without frags. It is brand new code, so there could be bugs in
it.

                                        Bob Harris

#!/bin/ksh
#
# estimate_extra_storage_needed_if_no_small_file_frags.ksh /mount_point
#

find $1 -xdev -type f -ls 2>/dev/null |\
awk '
    BEGIN {
        ONE_K = 1024
        SEVEN_K = (ONE_K * 7)
        EIGHT_K = (ONE_K * 8)
        FRAG_PERCENT = 5
        SIZE_LIMIT = ((EIGHT_K - 1) * 100) / FRAG_PERCENT
    }
    $7 !~ /[0-9]+/ { size = $6 }
    $7 ~ /[0-9]+/ { size = $7 }
    {
        fragsize = size % EIGHT_K
        if (fragsize == 0 || fragsize > SEVEN_K || size >= SIZE_LIMIT) {
            next # not a canidate
        }
        frag_storage = int((fragsize + (ONE_K-1)) / ONE_K)
        increased_storage = 8 - frag_storage
        total_storage_increase += increased_storage
    }
    END {
        print "Disabling small file fragging will need approx"
        if ( total_storage_increase < (10*1024) ) {
            print total_storage_increase, "Kilobytes more storage"
        }
        else if ( total_storage_increase < (10*1024*1024) ) {
            total_storage_increase /= 1024
            print total_storage_increase, "Megabytes more storage"
        }
        else {
            total_storage_increase /= (1024*1024)
            print total_storage_increase, "Gigabytes more storage"
        }
    }
'
exit