Re: Reiser4 und LZO compression

[Edward Shishkin <edward@namesys.com>]

Nigel Cunningham wrote:
> Hi.
> 
> On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote:
> 
>>Nigel Cunningham wrote:
>>
>>>Hi.
>>>
>>>On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote:
>>>
>>>>>>>>Hmm.  LZO is the best compression algorithm for the task as measured by
>>>>>>>>the objectives of good compression effectiveness while still having very
>>>>>>>>low CPU usage (the best of those written and GPL'd, there is a slightly
>>>>>>>>better one which is proprietary and uses more CPU, LZRW if I remember
>>>>>>>>right.  The gzip code base uses too much CPU, though I think Edward made
>>>>>>>
>>>>>>>I don't think that LZO beats LZF in both speed and compression ratio.
>>>>>>>
>>>>>>>LZF is also available under GPL (dual-licensed BSD) and was choosen in favor
>>>>>>>of LZO for the next generation suspend-to-disk code of the Linux kernel.
>>>>>>>
>>>>>>>see: http://www.goof.com/pcg/marc/liblzf.html
>>>>>>
>>>>>>thanks for the info, we will compare them
>>>>>
>>>>>For Suspend2, we ended up converting the LZF support to a cryptoapi
>>>>>plugin. Is there any chance that you could use cryptoapi modules? We
>>>>>could then have a hope of sharing the support.
>>>>
>>>>I am throwing in gzip: would it be meaningful to use that instead? The 
>>>>decoder (inflate.c) is already there.
>>>>
>>>>06:04 shanghai:~/liblzf-1.6 > l configure*
>>>>-rwxr-xr-x  1 jengelh users 154894 Mar  3  2005 configure
>>>>-rwxr-xr-x  1 jengelh users  26810 Mar  3  2005 configure.bz2
>>>>-rw-r--r--  1 jengelh users  30611 Aug 28 20:32 configure.gz-z9
>>>>-rw-r--r--  1 jengelh users  30693 Aug 28 20:32 configure.gz-z6
>>>>-rw-r--r--  1 jengelh users  53077 Aug 28 20:32 configure.lzf
>>>
>>>We used gzip when we first implemented compression support, and found it
>>>to be far too slow. Even with the fastest compression options, we were
>>>only getting a few megabytes per second. Perhaps I did something wrong
>>>in configuring it, but there's not that many things to get wrong!
>>
>>All that comes to mind is the speed/quality setting -- the number from 1 
>>to 9.  Recently, I backed up someone's hard drive using -1, and I 
>>believe I was still able to saturate... the _network_.  Definitely try 
>>again if you haven't changed this, but I can't imagine I'm the first 
>>persson to think of it.
>>
>> From what I remember, gzip -1 wasn't faster than the disk.  But at 
>>least for (very) repetitive data, I was wrong:
>>
>>eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync'
>>10+0 records in
>>10+0 records out
>>104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec)
>>
>>real    0m3.746s
>>user    0m0.005s
>>sys     0m0.627s
>>eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 > 
>>test; sync'
>>10+0 records in
>>10+0 records out
>>104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec)
>>  99.5%
>>
>>real    0m2.558s
>>user    0m1.554s
>>sys     0m0.680s
>>eve:~ sanity$
>>
>>
>>
>>This was on OS X, but I think it's still valid -- this is a slightly 
>>older Powerbook, with a 5400 RPM drive, 1.6 ghz G4.
>>
>>-1 is still worlds better than nothing.  The backup was over 15 gigs, 
>>down to about 6 -- loads of repetitive data, I'm sure, but that's where 
>>you win with compression anyway.
> 
> 
> Wow. That's a lot better; I guess I did get something wrong in trying to
> tune deflate. That was pre-cryptoapi though; looking at
> cryptoapi/deflate.c, I don't see any way of controlling the compression
> level. Am I missing anything?
> 

zlib is tunable, not cryptoapi's deflate.
look at zlib_deflateInit2()

> 
>>Well, you use cryptoapi anyway, so it should be easy to just let the 
>>user pick a plugin, right?
> 
> 
> Right. They can already pick deflate if they want to.
> 
> Regards,
> 
> Nigel
> 
> 
>