buffer/vm cache, turn off for a specific file ?

From: Andrew Torda (andrew.torda_at_anu.edu.au)
Date: 03/31/05 

Date: 31 Mar 2005 14:57:17 +0200

My program reads 10000 files in turn (fopen(), fread(), ... fclose())

Regardless of how low its priority (nice'ness), it makes a
machine very very unresponsive if it is running in the
background.
What seems to happen is that the OS, linux, caches each
file. After a while, everything else that was cached gets wiped
out. When a user hits a keystroke, the pages from the user's
application have been paged out and have to be found again.

There is no point in caching the 10000 files read by my
program. They are only read once.
Is there any way to ask the system to *not* to cache a file after
fopen()ing it ?
The 10000 files are NFS mounted, but I imagine the problem would
still happen with a local file system.

Pretty much everything in the code is ANSI C, but this is so
desperate, I would gladly put in linux specific, lower level code
if it helped. I do not want to replace the fopen(), but I am
happy to use fileno() to get the file descriptor and play with
it. I certainly have not found anything useful amongst the calls
to fcntl().
This code has to run in the background on machines maintained by
others, so it is not really practical to play with kernel options
which manipulate the split of vm / file buffer cache.

I would be grateful for any advice.
Andrew

Relevant Pages

  • Re: Sysinstall automatic filesystem size generation.
    ... much safer with the cache _enabled_, on most drives except the most ... >>advocates, typically against Linux users with journalled fs, on web ... points, even on consumer-grade hardware. ...
    (freebsd-stable)
  • Re: [00/17] Large Blocksize Support V3
    ... with this form of block aggregation - this is pretty much what is ... linux we can't say how nasty it would be. ... what Linux had for a buffer cache. ... Given that small block sizes give us better storage efficiency, ...
    (Linux-Kernel)
  • Re: Improvements to fsck performance in -current ...?
    ... > conjunction with the performance impact it had on Postgre. ... > The tests don't 100% apply, since he was testing with Linux and XFS, ... I doubt seriously that it is the disk caching which is to be blamed here, ... The cache sized on disks ...
    (freebsd-current)
  • Re: Memory usage - Evoultion and Mozilla
    ... > Yep - that's basically a subtotal of the total less buffers and cache. ... do know that each should show an identical amount of active RAM use, ... If you see 6 threads of Mozilla, each showing 50MB, it's actually 50MB ... Also, as was hinted at earlier in this message thread, Linux makes more ...
    (alt.os.linux.suse)
  • Re: Are Linux users all childish fools? What better reason to use Windows.
    ... As a Linux user here, ... denser drives with 8MB cache should really ... Windows does not support the 6410 chipset. ... Microsoft does not support hardware. ...
    (alt.os.linux.suse)
Loading