Re: generating huge file

From: Janis Papanagnou (Janis_Papanagnou_at_hotmail.com)
Date: 11/06/05 

Date: Sun, 06 Nov 2005 04:02:37 +0100

Robert Katz wrote:
> Janis Papanagnou wrote:
>
>> Robert Katz wrote:
>>
>>> I'm trying to generate as large a file as possible, with no newline
>>> characters, in a file system that has FREE free bytes. Any temporary
>>> files must come from the same file system. I came up with the
>>> following, but I wonder if it's reliable? Is it system memory
>>> dependent? Will I get any broken pipes if FREE is of the order of
>>> several hundred gigabytes? Is there anything more efficient?
>>>
>>> ####
>>> PATH=/bin:/usr/bin:/usr/local/bin
>>> : ${FREE:=1000000000}
>>> typeset -i FREE
>>> ((FREE < 0)) && exit 2
>>> trap 'rm -rf file_h; exit 3' HUP INT QUIT TERM
>>> if ((FREE == 0))
>>> then
>>> : >file_h
>>> exit
>>> fi
>>> yes X | sed "${FREE}"q | tr -d '\n' >file_h
>>> ####
>>>
>>
>> More efficient? For example...
>>
>> dd if=/dev/zero of=hugefile bs=10000 count=100000
>>
>> Define block size (bs) and count in terms of FREE.
>
>
> I liked the idea of dd, but I must have had a broken one, because it
> insisted on making `bs' some nontrivial power of two, times some power
> of 10.
>

You may want to split that operation up in two dd calls; one that makes
the huge part in multiples of these "nontrivial power of two", and one
that adds the rest. Such handling seems to be necessary in any case since
you want the size to be choosen freely and performance demands that you
define a blocksize huge enough. The two dd calls will be significantly
faster than the pipe expression. (For the second dd call you'll need the
"seek=" argument; hope that's available with your dd.)

Janis