Re: Compressing file system, NFS
From: notformail (notformail_at_jlab.org)
Date: 10/30/03
- Next message: Robert Downes: "Re: Apache core dumps when started with php4_module on FreeBSD 5.1"
- Previous message: Jim: "Re: "No /boot/loader" on new install"
- In reply to: Nick Hilliard: "Re: Compressing file system, NFS"
- Next in thread: Joe Doupnik: "Re: Compressing file system, NFS"
- Reply: Joe Doupnik: "Re: Compressing file system, NFS"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Date: Thu, 30 Oct 2003 14:48:49 -0500
Nick Hilliard wrote:
>> Is there any compressing file system available for FreeBSD 5.1-RELEASE
>> (or
>> later; i386)?
>
>
> At the moment, no.
>
>> The systems are running as file servers (NFS). Therefore the file system
>> needs to be NFS exportable. A userland NFS server or a modified kernel
>> based NFS server would do the trick, too. All I need is on-the-fly
>> decompression. Right now the files are already compressed with gzip. The
>> idea is to decompress the files on the servers so that the clients see
>> only
>> the original files. The clients are busy with analyzing the data and must
>> not waste CPU time with decompression. The NICs are Gigabit cards. The
>> network transfer rate is not an issue here.
>
>
> Really, you're talking about moving the compression / decompression
> processing to the NFS server. If there were a compressed filesystem
> available, this would almost certainly cause really serious performance
> problems, because while decompression uses a fair amount of cpu,
> compressing stuff really chews it up. So if there were any sort of i/o
> load on the system, cpu usage would rocket, and performance would fall
> through the floor.
Decompressing with GZIP gives me a transfer rate of 25 Mbyte/s. 6 (servers)
times 25 Mbyte/s is 150 Mbyte/s, more than I need. Compressing the data is
in my case no big deal since the data already came from a tape silo, were
once compressed on the servers and stay there compressed.
Beside, the servers have sufficient CPU power to handle any additional load
for decompression and could even do compression (which I don't need at the
moment) without slowing down the clients. They are fast enough to serve the
disks for the clients which have to read and analyse the data and then
write back the result.
>
>> Before anybody tells me that disk space is cheap: the current set
>> contains
>> 6*8*200 Gbytes. GZIP gives me a factor 2.5, which I really need.
>> Actually I
>> need more. ;-) Adding the difference in disks costs some 10,000 bucks.
>
>
> For a 10 terabyte system, that's not a large amount of money.
But for me it is a lot.
>
> Nick
I found cfs, an encrypting userland NFS server. Unfortunately this is only
a single-threaded server. Each of my clients (all double CPU systems) may
read as many as two or more files.
Does anybody know about a modified nullfs or a modified NFS server that
could do transparent decompression, and for now no compression, on the
servers?
Perhaps someone who knows the details of the NFS server could point me to
the best place where to insert a decompression routine? The compressed
files already have the file extension .gz which identifies them nicely.
The server would also have to strip the extension. Any idea?
- Next message: Robert Downes: "Re: Apache core dumps when started with php4_module on FreeBSD 5.1"
- Previous message: Jim: "Re: "No /boot/loader" on new install"
- In reply to: Nick Hilliard: "Re: Compressing file system, NFS"
- Next in thread: Joe Doupnik: "Re: Compressing file system, NFS"
- Reply: Joe Doupnik: "Re: Compressing file system, NFS"
- Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
Relevant Pages
|
