Compression Plugin

Compression Plugin

[Clay Barnes]

Forgive me if this has been answered recently, but I haven't gotten my
last two dozen e-mails for today yet.

Regarding the compression plugin, what sort of compression can one
expect from it?  I know that compression of files like bz2 will vary (in
that they'll be essentially uncompressed) from source/text files
(excellent compression).  A rough estimate from someone's experience
using on mixed files it would help.

Also:  Is the algorithm set in stone?  If so what is it?  If not, what
is it now/expected to be?

Thanks,
Clay Barnes

Re: Compression Plugin

[David Masover]

Clay Barnes wrote:

> Also:  Is the algorithm set in stone?  If so what is it?  If not, what
> is it now/expected to be?

Probably lzo, which is already used for other things like network
connections (ssh, openvpn, and so on).  The nice thing about lzo is that
it's fast, faster than gzip or bzip2, and gets decent compression -- not
great, but decent.  I don't usually get gzip or bzip2 to compress at
disk speed, but then, I usually crank the compression way up, so YMMV.
The point of using a fast algorithm is that you not only save space, but
when you apply it to things like text files, it can actually make things
go faster.

But I imagine it will be settable per-file.  Files can be both encrypted
and compressed, and I think (I hope) it could be with a choice of
crypto/compression algorithms.

Re: Compression Plugin

[Gregory Maxwell]

On 9/20/05, Clay Barnes <clay.barnes@gmail.com> wrote:
> Forgive me if this has been answered recently, but I haven't gotten my
> last two dozen e-mails for today yet.
> 
> Regarding the compression plugin, what sort of compression can one
> expect from it? 
[snip]

Just a general, no reiser4 specifc answer since because the
compression isn't done yet I don't know the details on reiser4's
performance....

It is generally the case the disk based compression performs somewhat
worse than normal file based compressors. This is because every block
of data must be compressed alone in order to preserve the random
access semantics of the file system.

This also means that there is less to be gained by using alternative
compression systems (such as bzip2 or better LZMA) because most only
pick up their impressive performance as a result of having a much
larger context and at a greatly increased cost in memory usage.

For disk based compression Lz77 is pretty good and is so widely used
that people feel comfortable implimenting it in kernel space. Another
interesting player is LZO because decompression requires very little
memory and is VERY VERY fast (I think someting like 8x faster than
gzip in my testing). This means that decompression  is effectively
free. However compression is perhaps 10% worse than LZ77 ... on most
hardware the disk is so slow that the decrease in compression
outweighs the improvement in decompression performance. But on systems
with a fast disk array, LZO may be a welcome tradeoff.

Re: Compression Plugin

[Gregory Maxwell]

On 9/20/05, David Masover <ninja@slaphack.com> wrote:
> Probably lzo, which is already used for other things like network
> connections (ssh, openvpn, and so on).  The nice thing about lzo is that
> it's fast, faster than gzip or bzip2, and gets decent compression -- not
> great, but decent.  I don't usually get gzip or bzip2 to compress at
> disk speed, but then, I usually crank the compression way up, so YMMV.
> The point of using a fast algorithm is that you not only save space, but
> when you apply it to things like text files, it can actually make things
> go faster.
> 
> But I imagine it will be settable per-file.  Files can be both encrypted
> and compressed, and I think (I hope) it could be with a choice of
> crypto/compression algorithms.

I didn't know SSH supported LZO.  Rsync does though...

Gzip compression is pretty darn quick at lower levels, though
depending on the lz77 implimentaiton it can be fairly slow at higher
compression levels.

An interesting idea:  select the algo and a range of compression
levels per file, but select the actual compression level at flush time
based on some estimate of how loaded the system is.. :)
Probably not worth it even though the amount of compression and the
speed differ greatly from -1 to -9... I hope no one wastes their time
on it until the more important things are done.. but perhaps a nice
touch.

Re: Compression Plugin

[michael chang]

On 9/20/05, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> An interesting idea:  select the algo and a range of compression
> levels per file, but select the actual compression level at flush time
> based on some estimate of how loaded the system is.. :)
> Probably not worth it even though the amount of compression and the
> speed differ greatly from -1 to -9... I hope no one wastes their time
> on it until the more important things are done.. but perhaps a nice
> touch.

Kinda makes it sound like in addition to the "repacker" that repacks
blocks in order and squeezes them, (which should allow for filesystem
size reduction), there'll be a compression repacker too that allows
you to repack files to a higher or lower compression level (anyone
remember the Compression Agent that came with DoubleSpace and the like
in Windows 9x?).

-- 
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: Compression Plugin

[Tomasz Chmielewski]

michael chang schrieb:
> On 9/20/05, Gregory Maxwell <gmaxwell@gmail.com> wrote:
> 
>>An interesting idea:  select the algo and a range of compression
>>levels per file, but select the actual compression level at flush time
>>based on some estimate of how loaded the system is.. :)
>>Probably not worth it even though the amount of compression and the
>>speed differ greatly from -1 to -9... I hope no one wastes their time
>>on it until the more important things are done.. but perhaps a nice
>>touch.
> 
> 
> Kinda makes it sound like in addition to the "repacker" that repacks
> blocks in order and squeezes them, (which should allow for filesystem
> size reduction), there'll be a compression repacker too that allows
> you to repack files to a higher or lower compression level (anyone
> remember the Compression Agent that came with DoubleSpace and the like
> in Windows 9x?).

AFAIR e2compr patch to ext2 (which enables compression on a ex2 
filesystem) could use many packing methods, including gzip and bzip2 
(!), with a range from 1 to 9...

I certainly could imagine cases where compression level is more 
important than speed.


-- 
Tomek
http://wpkg.org

Re: Compression Plugin

[PFC]

>>> An interesting idea:  select the algo and a range of compression
>>> levels per file,

	A simple check on wether it's an already compressed file (using file  
extension and magic number) should be quite easy to do and cheap.

	Now, intrigued by this lzo thingie, I ran a little benchmark on my emails  
; I have a huge mail spool and my mail client is always slow to launch...
	Seems that on this laptop LZO can compress about 40 MB/s and decompress  
about 200 (!) MB/s. On the mail spool, compression ratio was about 1/2 ;  
gzip was better, although 5-6 times slower. So considering the disk  
throughput is only 15 MB/s, this could make it twice as fast with CPU to  
spare. Yowza !

>>> but select the actual compression level at flush time
>>> based on some estimate of how loaded the system is.. :)
>>> Probably not worth it even though the amount of compression and the
>>> speed differ greatly from -1 to -9... I hope no one wastes their time
>>> on it until the more important things are done.. but perhaps a nice
>>> touch.
>>   Kinda makes it sound like in addition to the "repacker" that repacks
>> blocks in order and squeezes them, (which should allow for filesystem
>> size reduction), there'll be a compression repacker too that allows
>> you to repack files to a higher or lower compression level (anyone
>> remember the Compression Agent that came with DoubleSpace and the like
>> in Windows 9x?).
>
> AFAIR e2compr patch to ext2 (which enables compression on a ex2  
> filesystem) could use many packing methods, including gzip and bzip2  
> (!), with a range from 1 to 9...
>
> I certainly could imagine cases where compression level is more  
> important than speed.
>
>

Re: Compression Plugin

[Edward Shishkin]

Gregory Maxwell wrote:

>On 9/20/05, David Masover <ninja@slaphack.com> wrote:
>  
>
>>Probably lzo, which is already used for other things like network
>>connections (ssh, openvpn, and so on).  The nice thing about lzo is that
>>it's fast, faster than gzip or bzip2, and gets decent compression -- not
>>great, but decent.  I don't usually get gzip or bzip2 to compress at
>>disk speed, but then, I usually crank the compression way up, so YMMV.
>>The point of using a fast algorithm is that you not only save space, but
>>when you apply it to things like text files, it can actually make things
>>go faster.
>>
>>But I imagine it will be settable per-file.  Files can be both encrypted
>>and compressed, and I think (I hope) it could be with a choice of
>>crypto/compression algorithms.
>>    
>>
>
>I didn't know SSH supported LZO.  Rsync does though...
>
>Gzip compression is pretty darn quick at lower levels, though
>depending on the lz77 implimentaiton it can be fairly slow at higher
>compression levels.
>
>An interesting idea:  select the algo and a range of compression
>levels per file, but select the actual compression level at flush time
>based on some estimate of how loaded the system is.. :)
>  
>

Compression level usually means a separate algorithm of some family and
should be represented by separate transform plugin. It is possible to assign
_any_ compression transform plugin id once in flush time when the file 
is newly
created, but I am not sure if it should depend on the current snapshot 
of system
monitoring.
  However, it will be possible to turn a compression off for a file in 
any time
you want. Reiser4 deflate manager will call the compression mode plugin 
which
represents some policy in such switching. The default policy will be the one
suggested by Hans: if the first cluster of a file is incompressible, 
then turn
it off. One can invent and add new policies.

>Probably not worth it even though the amount of compression and the
>speed differ greatly from -1 to -9... I hope no one wastes their time
>on it until the more important things are done.. but perhaps a nice
>touch.
>  
>

Re: Compression Plugin

[Hans Reiser]

PFC wrote:

>
>>>> An interesting idea:  select the algo and a range of compression
>>>> levels per file,
>>>
>
>     A simple check on wether it's an already compressed file (using
> file  extension and magic number) should be quite easy to do and cheap.
>
>     Now, intrigued by this lzo thingie, I ran a little benchmark on my
> emails  ; I have a huge mail spool and my mail client is always slow
> to launch...
>     Seems that on this laptop LZO can compress about 40 MB/s and
> decompress  about 200 (!) MB/s. On the mail spool, compression ratio
> was about 1/2 ;  gzip was better, although 5-6 times slower. So
> considering the disk  throughput is only 15 MB/s, this could make it
> twice as fast with CPU to  spare. Yowza !

Yup.  Yowza is what we want.;-)  LZO is what Edward selected for us to use.

>
>>>> but select the actual compression level at flush time
>>>> based on some estimate of how loaded the system is.. :)
>>>> Probably not worth it even though the amount of compression and the
>>>> speed differ greatly from -1 to -9... I hope no one wastes their time
>>>> on it until the more important things are done.. but perhaps a nice
>>>> touch.
>>>
>>>   Kinda makes it sound like in addition to the "repacker" that repacks
>>> blocks in order and squeezes them, (which should allow for filesystem
>>> size reduction), there'll be a compression repacker too that allows
>>> you to repack files to a higher or lower compression level (anyone
>>> remember the Compression Agent that came with DoubleSpace and the like
>>> in Windows 9x?).
>>
>>
>> AFAIR e2compr patch to ext2 (which enables compression on a ex2 
>> filesystem) could use many packing methods, including gzip and bzip2 
>> (!), with a range from 1 to 9...
>>
>> I certainly could imagine cases where compression level is more 
>> important than speed.
>>
>>
>
>
>
>

Compression Plugin

[Hönsch Roland]

What is the state of the compression plugin?  Is it likely to appear in 
mm-sources version of reiser4 any time soon?

Sincerely,

Roland

Re: Compression Plugin

[Hans Reiser]

Clay Barnes wrote:

>Forgive me if this has been answered recently, but I haven't gotten my
>last two dozen e-mails for today yet.
>
>Regarding the compression plugin, what sort of compression can one
>expect from it?  I know that compression of files like bz2 will vary (in
>that they'll be essentially uncompressed) from source/text files
>(excellent compression).  A rough estimate from someone's experience
>using on mixed files it would help.
>
>Also:  Is the algorithm set in stone?  If so what is it?  If not, what
>is it now/expected to be?
>  
>
It is lzo1.  There are multiple compression algorithms to choose from,
one of them is gzip (more cpu but better compression).  lzo1 gets > 50%
on text and executables and nothing on mp3s and video.

>Thanks,
>Clay Barnes
>
>
>  
>