Re: Print last bulk of data in a file - "non-greedy" line adressing

From: Loki Harfagr (lars.hummigeret_at_yahuu.no)
Date: 05/31/05 

Date: Tue, 31 May 2005 16:28:21 +0200

Le Tue, 31 May 2005 12:44:35 +0000, Stein Arne Storslett a écrit :

> <lars.hummigeret@yahuu.no> wrote in <429c4db4$0$6504$636a15ce@news.free.fr>:
>> Le Tue, 31 May 2005 09:58:08 +0000, Stein Arne Storslett a écrit :
>>
> <snip>
>>>
>>> Examples of how I produced the wanted output follows. The draw backs
>>> are, of course that I have to read the whole file.
>>>
>>> $ tail -r logfile | sed -n '1,/^=/p' | tail -r
>>> # tail -r seems to work on Solaris, but not HP-UX
>>> # HP-UX have the rev command though
>>> .
>>> $ perl -ne '{/^=/&&{$a=$_}||{$a.=$_}}END{print$a}' logfile
>>> .
>>> $ awk '/^=/{a=$0} /^[^=]/{ a=a"\n"$0 } END { print a }' logfile
>>>
>>> Etc. All of which read the whole file. For a large file this would
>>> probably be quicker, but...:
>>> $ tail -200 logfile | awk '/^=/{a=$0}/^[^=]/{a=a"\n"$0}END{print a}'
>>>
>>>
>>> How do you solve this? Any thoughts?
>>> Thank you.
>>
>> OK, and now, for something completely different ;-)
>>
>> Here recipe :
>>
>> $ rm /tmp/boo ; mkfifo /tmp/boo && tac toappend.other.huge.flat > /tmp/boo &
>> $ awk '/^=========/{nextfile}; 1' /tmp/boo 2> /dev/null | tac
>
> M'kay....
> So, the fifo will break when awk asks for the next file.

I used 'nextfile' just for a generic solution, if you know
you *want* only *one* file, just use 'exit' :-)

> Well, a test on a 180MB file proved your sollution way faster than the
> perl and awk sollution above anyway (on HP-UX 11i (11.11)).
> awk: real 7:11.8, perl: real: 8:22.0
>
> Yours I timed with a "date" command, and it took about 20 seconds on the
> same file.

 Man, I don't know where it went in these 20 seconds !
It's thousand time slower than here !?
 You're sure that you didn't wait 20 seconds before pressing enter
to release the background return message ???

> What tics me is how "tail -200" can be so much faster.

Well, it may also be related to your Unix, your machine, your filesystem,
your settings on these filesystems, etc.

On my main desktop it gives the same fast time whatever the file size,
I especially tested on thes 2 :
$ ls -lah toappend.other.huge.flat*
-rw-r--r-- 1 rpascal users 201M May 31 16:16 toappend.other.huge.flat
-rw-r--r-- 1 rpascal users 51M May 31 16:15 toappend.other.huge.flat.2

And had this average response, (in 10 tries)
real 0m0.028s
user 0m0.010s
sys 0m0.008s

I guess I's have better times on my servers but i won't test it at
the time, anyway they run the same system and filesystems types, thus
I's expect them to be in the line.

> I guess I have to
> start learning C ...

That could be, if you really want to deal with massive files and
fast access on records using the toolbox is safe but can be slow
and cagier than a few lines of C ;-)