* Re: Snappy compression algorithm for Reiser4.
2013-09-25 19:52 ` Edward Shishkin
@ 2013-09-25 21:04 ` Evgeniy
2013-09-25 21:10 ` Chris Gentile
2013-09-25 21:12 ` Chris Gentile
2 siblings, 0 replies; 6+ messages in thread
From: Evgeniy @ 2013-09-25 21:04 UTC (permalink / raw)
To: Edward Shishkin; +Cc: reiserfs-devel
On 25 September 2013 22:52, Edward Shishkin <edward.shishkin@gmail.com> wrote:
> On 09/22/2013 01:50 AM, Edward Shishkin wrote:
>>
>> 21.09.2013 23:12, Evgeniy пишет:
>>>
>>> Hello all,
>>>
>>> Last few days I was playing a little bit around implementation Snappy
>>> algo in Reiser4. My code is dirty and should be optimized. But it's
>>> enough for testing.
>>> According to my tests Snappy shows near the same prefomance as LZO. In
>>> some kind of situations Snappy shows better compression with the less
>>> CPU usage.
>>> LZO has a few advantages over Snappy:
>>> 1) It's in kernel by default
>>> 2) It's optimized better then Snappy.
>>> Also a very significant fact is we're plaing with small blocks of
>>> data. In this case results of all algorithms looks near the same.
>>>
>>> Maybe in the future I'll reqrite a little bit BratSinot's LZ4 patch
>>> (http://sourceforge.net/p/reiser4/discussion/general/thread/780facb4/)
>>> to make it use LZ4 library which is build-in kernel. Friendly
>>> speaking, I'm not expect any big difference in comparison with LZO.
>>> But in any case, it funny :)
>>
>>
>>
>> Support of a new compression algorithm means a format change.
>
>
>
> To be precise, format _upgrade_ (not change). Format change is a very
> bad thing inherent to systems, which don't possess any development
> model.
>
>
>
>>
>> So, people with roots on reiser4 will be forced to fsck their partitions.
>> So, not so funny. A visible advantage in some benchmarks is highly
>> desirable..
>
>
>
> It seems, it is hard to invent something more interesting than currently
> supported algorithms (lzo1 and gzip1). At least for file systems, where
> we can not afford a luxury to comress/decompress big chunks of data.
Well, I think it depends only of compression algo and its
implementation in the kernel. As I described above, LZO currently has
an advantage over Snappy. That's why Snappy wins in speed of
(de)compression only on 1-2% over LZO which is almost nothing. Maybe
sutuation will be better with curent implementation LZ4 in kernel.
> Instead, I would suggest to take a look at the "intelligent" compression
> modes implemented by plugins of COMPRESSION_MODE interface. This is
> a set of hooks arranged at different levels, which decide, if compression
> should be turned off/on.
>
> The default mode ("conv") was suggested long ago by Hans. It's idea is
> that first, we try to compress the first 64K of the file and look at the
> result. If it is incompressible, then we turn compression off forever (pass
> management to unix-file plugin with formatting policy "extents only").
> Otherwise, stay with cryptcompress plugin, and perform "selective"
> compression on the "dynamic" lattice.
>
> I suspect that such heuristic doesn't work well for all kind of files. At
> the
> same time, we have never experimented in this area. The common idea is
> that it would be nice to understand by the beginning of a file, whether the
> whole file is incompressible. For example, badly compressible binary
> executables have special magics at the beginning, etc. Also, I think, it
> doesn't make sense to compress ISO images (no?)
It's a difficult question what is faster: to check first 64K of the
file and look at the result or read the file header, find type in DB
and deside to turn of or off compression.
To "understand" the file type we need:
1) to include some kind of libmediainfo with file types database.
2) to have our own DB with data like: "images?"->off,
"binaries?"->off,"text?"->on.
Also we can implement "stupid" compression compression mode: on/off
compression use only file extension. If the file is without extension
then use "conv" compression (to test if first 64K compresses well). I
thnks this kind of compression mode will work faster on partition with
SMB file share.
Maybe I'm wrong somewhere above. So please correct me.
> Anyway, a sanity check that file represents a binary executable won't be
> superfluous. If someone has other ideas, we'll be happy to discuss/encode
> them.
Is there any documentation regarding file system design? It's quite
difficult to understand from source comments how does it work. For
example, when I copy file to reiser4 partition, first of all it goes
to file cache. In some time (when?) VFS call SYNC function it call R4
hook (where is it?) and run all chains of funtion and finally write
file to disk. When I was debugging compression function I was that FS
doesn't call compress function until I run "sync" in the console.
Maybe these questions not regarding reiser4. Could you please suggest
documentation about such things I wrote above?
P.S: In my opinion to make fsck work faster is the much important task
than compression modes. Sometimes fsck is terribly slow :(
> Thanks,
> Edward.
--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Snappy compression algorithm for Reiser4.
2013-09-25 19:52 ` Edward Shishkin
2013-09-25 21:04 ` Evgeniy
@ 2013-09-25 21:10 ` Chris Gentile
2013-09-25 21:12 ` Chris Gentile
2 siblings, 0 replies; 6+ messages in thread
From: Chris Gentile @ 2013-09-25 21:10 UTC (permalink / raw)
To: Edward Shishkin, Evgeniy; +Cc: reiserfs-devel
Edward,
I would agree on most points, though the LZ4 algorithm is gaining some
interest. Not to say we know if it works well or not ...
ISO images can be compressed as they are just a concatenation of files,
as well as DD created images (raw block)
If we were to take a vote, I would test LZ4 out in the general sense,
and if it's as good as they say (and fast enough for real-time
compression) I would vote that way.
Chris
On 09/25/13 15:52, Edward Shishkin wrote:
> On 09/22/2013 01:50 AM, Edward Shishkin wrote:
>> 21.09.2013 23:12, Evgeniy пишет:
>>> Hello all,
>>>
>>> Last few days I was playing a little bit around implementation Snappy
>>> algo in Reiser4. My code is dirty and should be optimized. But it's
>>> enough for testing.
>>> According to my tests Snappy shows near the same prefomance as LZO. In
>>> some kind of situations Snappy shows better compression with the less
>>> CPU usage.
>>> LZO has a few advantages over Snappy:
>>> 1) It's in kernel by default
>>> 2) It's optimized better then Snappy.
>>> Also a very significant fact is we're plaing with small blocks of
>>> data. In this case results of all algorithms looks near the same.
>>>
>>> Maybe in the future I'll reqrite a little bit BratSinot's LZ4 patch
>>> (http://sourceforge.net/p/reiser4/discussion/general/thread/780facb4/)
>>> to make it use LZ4 library which is build-in kernel. Friendly
>>> speaking, I'm not expect any big difference in comparison with LZO.
>>> But in any case, it funny :)
>>
>>
>> Support of a new compression algorithm means a format change.
>
>
> To be precise, format _upgrade_ (not change). Format change is a very
> bad thing inherent to systems, which don't possess any development
> model.
>
>
>>
>> So, people with roots on reiser4 will be forced to fsck their
>> partitions.
>> So, not so funny. A visible advantage in some benchmarks is highly
>> desirable..
>
>
> It seems, it is hard to invent something more interesting than currently
> supported algorithms (lzo1 and gzip1). At least for file systems, where
> we can not afford a luxury to comress/decompress big chunks of data.
>
> Instead, I would suggest to take a look at the "intelligent" compression
> modes implemented by plugins of COMPRESSION_MODE interface. This is
> a set of hooks arranged at different levels, which decide, if compression
> should be turned off/on.
>
> The default mode ("conv") was suggested long ago by Hans. It's idea is
> that first, we try to compress the first 64K of the file and look at the
> result. If it is incompressible, then we turn compression off forever
> (pass
> management to unix-file plugin with formatting policy "extents only").
> Otherwise, stay with cryptcompress plugin, and perform "selective"
> compression on the "dynamic" lattice.
>
> I suspect that such heuristic doesn't work well for all kind of files.
> At the
> same time, we have never experimented in this area. The common idea is
> that it would be nice to understand by the beginning of a file,
> whether the
> whole file is incompressible. For example, badly compressible binary
> executables have special magics at the beginning, etc. Also, I think, it
> doesn't make sense to compress ISO images (no?)
>
> Anyway, a sanity check that file represents a binary executable won't be
> superfluous. If someone has other ideas, we'll be happy to discuss/encode
> them.
>
> Thanks,
> Edward.
> --
> To unsubscribe from this list: send the line "unsubscribe
> reiserfs-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: Snappy compression algorithm for Reiser4.
2013-09-25 19:52 ` Edward Shishkin
2013-09-25 21:04 ` Evgeniy
2013-09-25 21:10 ` Chris Gentile
@ 2013-09-25 21:12 ` Chris Gentile
2 siblings, 0 replies; 6+ messages in thread
From: Chris Gentile @ 2013-09-25 21:12 UTC (permalink / raw)
To: Edward Shishkin, Evgeniy; +Cc: reiserfs-devel
Edward,
I would agree on most points, though the LZ4 algorithm is gaining some
interest. Not to say we know if it works well or not ...
ISO images can be compressed as they are just a concatenation of files,
as well as DD created images (raw block)
If we were to take a vote, I would test LZ4 out in the general sense,
and if it's as good as they say (and fast enough for real-time
compression) I would vote that way.
Chris
On 09/25/13 15:52, Edward Shishkin wrote:
> On 09/22/2013 01:50 AM, Edward Shishkin wrote:
>> 21.09.2013 23:12, Evgeniy пишет:
>>> Hello all,
>>>
>>> Last few days I was playing a little bit around implementation Snappy
>>> algo in Reiser4. My code is dirty and should be optimized. But it's
>>> enough for testing.
>>> According to my tests Snappy shows near the same prefomance as LZO. In
>>> some kind of situations Snappy shows better compression with the less
>>> CPU usage.
>>> LZO has a few advantages over Snappy:
>>> 1) It's in kernel by default
>>> 2) It's optimized better then Snappy.
>>> Also a very significant fact is we're plaing with small blocks of
>>> data. In this case results of all algorithms looks near the same.
>>>
>>> Maybe in the future I'll reqrite a little bit BratSinot's LZ4 patch
>>> (http://sourceforge.net/p/reiser4/discussion/general/thread/780facb4/)
>>> to make it use LZ4 library which is build-in kernel. Friendly
>>> speaking, I'm not expect any big difference in comparison with LZO.
>>> But in any case, it funny :)
>>
>>
>> Support of a new compression algorithm means a format change.
>
>
> To be precise, format _upgrade_ (not change). Format change is a very
> bad thing inherent to systems, which don't possess any development
> model.
>
>
>>
>> So, people with roots on reiser4 will be forced to fsck their
>> partitions.
>> So, not so funny. A visible advantage in some benchmarks is highly
>> desirable..
>
>
> It seems, it is hard to invent something more interesting than currently
> supported algorithms (lzo1 and gzip1). At least for file systems, where
> we can not afford a luxury to comress/decompress big chunks of data.
>
> Instead, I would suggest to take a look at the "intelligent" compression
> modes implemented by plugins of COMPRESSION_MODE interface. This is
> a set of hooks arranged at different levels, which decide, if compression
> should be turned off/on.
>
> The default mode ("conv") was suggested long ago by Hans. It's idea is
> that first, we try to compress the first 64K of the file and look at the
> result. If it is incompressible, then we turn compression off forever
> (pass
> management to unix-file plugin with formatting policy "extents only").
> Otherwise, stay with cryptcompress plugin, and perform "selective"
> compression on the "dynamic" lattice.
>
> I suspect that such heuristic doesn't work well for all kind of files.
> At the
> same time, we have never experimented in this area. The common idea is
> that it would be nice to understand by the beginning of a file,
> whether the
> whole file is incompressible. For example, badly compressible binary
> executables have special magics at the beginning, etc. Also, I think, it
> doesn't make sense to compress ISO images (no?)
>
> Anyway, a sanity check that file represents a binary executable won't be
> superfluous. If someone has other ideas, we'll be happy to discuss/encode
> them.
>
> Thanks,
> Edward.
> --
> To unsubscribe from this list: send the line "unsubscribe
> reiserfs-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe reiserfs-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 6+ messages in thread