Re: Compressing file system, NFS

From: Nick Hilliard (nick_at_foobar#delete2email#.org)
Date: 10/30/03 

Date: Thu, 30 Oct 2003 10:43:27 +0000

> Is there any compressing file system available for FreeBSD 5.1-RELEASE (or
> later; i386)?

At the moment, no.

> The systems are running as file servers (NFS). Therefore the file system
> needs to be NFS exportable. A userland NFS server or a modified kernel
> based NFS server would do the trick, too. All I need is on-the-fly
> decompression. Right now the files are already compressed with gzip. The
> idea is to decompress the files on the servers so that the clients see only
> the original files. The clients are busy with analyzing the data and must
> not waste CPU time with decompression. The NICs are Gigabit cards. The
> network transfer rate is not an issue here.

Really, you're talking about moving the compression / decompression
processing to the NFS server. If there were a compressed filesystem
available, this would almost certainly cause really serious performance
problems, because while decompression uses a fair amount of cpu,
compressing stuff really chews it up. So if there were any sort of i/o
load on the system, cpu usage would rocket, and performance would fall
through the floor.

> Before anybody tells me that disk space is cheap: the current set contains
> 6*8*200 Gbytes. GZIP gives me a factor 2.5, which I really need. Actually I
> need more. ;-) Adding the difference in disks costs some 10,000 bucks.

For a 10 terabyte system, that's not a large amount of money.

Nick

Relevant Pages

  • Re: target: Parallel Compresion/decompresion
    ... Actually even my solid state disk can't keep up. ... For gzip the CPU will in most cases be faster at decompressing than the incoming data rate. ... BWT-based compression formats are well-suited to parallel decompression since the input is broken up into independent blocks. ...
    (comp.compression)
  • Re: NSA and crypto
    ... David - where are your test results? ... was to run a compression then a decompression to see if file matches. ... For example the compressor actually only works with strings. ...
    (sci.crypt)
  • Re: NSA and crypto
    ... David - where are your test results? ... | was to run a compression then a decompression to see if file matches. ... For example the compressor actually only works with strings. ...
    (sci.crypt)
  • Re: [RFC] LZO de/compression support - take 6
    ... Compression: 42.4646 usec ... Decompression: 42.4646 usec ... early finish compressed data buffer to less than the full size ...
    (Linux-Kernel)
  • Re: A basic cryptanalysis question
    ... >> keys. ... >> the above is for bijective file compression. ... unique bitstream weather doing compression or decompression. ... based on second order english. ...
    (sci.crypt)