memory pool, rfc

Hello:

I have some free time and want to do an memory pool. The idea is to have a memory zone of N KB (or several MB) compressed in memory. I have fast compression algorithms now that can release under BSD licence that are faster than hd i/o, so it take less compress/decompress a memory zone than read/write it to disk. I don't know if it already exist for FreeBSD, so if it's already done i'll try to improve it.

- Each memory chunk is compressed separately, so i can decompress and use one without decompress anyother one.
- In 4KB chunks of text i get 50-60 % compression (avg 2 - 1.6 KB result)
- In 4KB chunks of binary (application code) i get 30-40 % compression (avg 2.8 - 2.4 KB result)
- In both cases, results may vary depending on data type and chunk size, greater implies better compression
- Speed once implemented will be very fast. Current speed hogs a PATA 133 disk.

For what can be used?

- Memory pools in applications (like malloc)
- Ram disks
- Disk Cache (permit bigger disk cache)
- 'On the fly' filesystem compression (and it takes less read/write compressed data than non-compressed)
- Perhaps add it as Virtual Memory swap cache?
- Other

Don't point me to zlib or libbzip2, they are on another league and are much slower than my code.

Thanks

_______________________________________________
freebsd-questions@xxxxxxxxxxx mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscribe@xxxxxxxxxxx"

Relevant Pages

  • Re: Recording Endlessly w/o Memory Overflow?
    ... the AVI compression was the issue. ... > is buffered in memory before being written to disk. ... and much faster than the Indeo5 codec. ...
    (comp.soft-sys.matlab)
  • Re: Reiser4. BEST FILESYSTEM EVER.
    ... I guess you have also not been hit by out of memory ... one with Reiser3 and the other with Reiser4. ... Seriously, disk storage is getting less expensive all the time, you can ... multiple files because of the compression in the file system. ...
    (Linux-Kernel)
  • memory pool, rfc
    ... The idea is to have a memory zone of N KB compressed in memory. ... I have fast compression algorithms now that can release under BSD licence that are faster than hd i/o, so it take less compress/decompress a memory zone than read/write it to disk. ...
    (freebsd-hackers)
  • Re: Quick Huffman theory question
    ... application compresses and decompresses, compression appears to ... categories as context. ... good text compression requires gigabytes of memory. ... don't bother with Huffman codes. ...
    (comp.compression)
  • Re: view menu
    ... When the file is opened into any graphical viewer or editor, the data is transferred from disk to memory so that the program can work with the data. ... We know that png uses data compression, so we can expect the uncompressed size to be much more. ... The image is 1080 pixels wide and 902 pixels high for a total of 1080x902 = 974160 pixels. ...
    (comp.graphics.apps.gimp)