Will I need to re-format my partition for using the compression plugin?

Will I need to re-format my partition for using the compression plugin?

[Clemens Eisserer]

Hello again,

I am currently re-installing my linux system and I would like to use
reiser4 since I think its really powerful and great (and is well
written too ;) ).

However, will it be possible to convert a non-compressed reiser4
partition to a compressed one withought the need of deleting data or
reformat anything?

Thanks, lg Clemens

Re: Will I need to re-format my partition for using the compression plugin?

[Edward Shishkin]

Clemens Eisserer wrote:

>Hello again,
>
>I am currently re-installing my linux system and I would like to use
>reiser4 since I think its really powerful and great (and is well
>written too ;) ).
>
>However, will it be possible to convert a non-compressed reiser4
>partition to a compressed one withought the need of deleting data or
>reformat anything?
>
>Thanks, lg Clemens
>  
>

No need in such reformat actually. It may require special config options
to build the kernel, and maybe special mount option to mount your partition.

Edward.

Re: Will I need to re-format my partition for using the compression plugin?

[Vladimir V. Saveliev]

Hello

Clemens Eisserer wrote:
> Hello again,
> 
> I am currently re-installing my linux system and I would like to use
> reiser4 since I think its really powerful and great (and is well
> written too ;) ).
> 
> However, will it be possible to convert a non-compressed reiser4
> partition to a compressed one withought the need of deleting data or
> reformat anything?
> 

reiser4 compression is based on a file. That means that when it will work you will be able to have new files compressed.
I guess there will be no way to compress existing files. Experts will correct me if I am wrong.

> Thanks, lg Clemens
> 
> 

Re: Will I need to re-format my partition for using the compression plugin?

[Edward Shishkin]

Vladimir V. Saveliev wrote:

>Hello
>
>Clemens Eisserer wrote:
>  
>
>>Hello again,
>>
>>I am currently re-installing my linux system and I would like to use
>>reiser4 since I think its really powerful and great (and is well
>>written too ;) ).
>>
>>However, will it be possible to convert a non-compressed reiser4
>>partition to a compressed one withought the need of deleting data or
>>reformat anything?
>>
>>    
>>
>
>reiser4 compression is based on a file. That means that when it will work you will be able to have new files compressed.
>I guess there will be no way to compress existing files. Experts will correct me if I am wrong.
>
>  
>

Precisely speaking the subject and message body contain different
questions. The answer for the first one is no, and Vladimir answered
to the second one correctly.

Re: Will I need to re-format my partition for using the compression plugin?

[Clemens Eisserer]

So yes, hmm sad that there is not compress-option - it could be easily
doable with the files in the pseudo directory of every single file.
This way one could select which files to compress and which not -
however thats not that important for me. I am just happy that there is
at least one serious filesystem with compression available.

Thanks a lot, I waited for Raiser4 the last 8 years ;-)

Thanks, lg Clemens

Re: Will I need to re-format my partition for using the compression plugin?

[Tomasz Chmielewski]

Clemens Eisserer schrieb:
> So yes, hmm sad that there is not compress-option - it could be easily
> doable with the files in the pseudo directory of every single file.
> This way one could select which files to compress and which not -
> however thats not that important for me. I am just happy that there is
> at least one serious filesystem with compression available.

does it mean reiser4 has a compression plugin available now?

-- 
Tomek
http://wpkg.org

Re: Will I need to re-format my partition for using the compression plugin?

[Edward Shishkin]

Tomasz Chmielewski wrote:

> Clemens Eisserer schrieb:
>
>> So yes, hmm sad that there is not compress-option - it could be easily
>> doable with the files in the pseudo directory of every single file.
>> This way one could select which files to compress and which not -
>> however thats not that important for me. I am just happy that there is
>> at least one serious filesystem with compression available.
>
>
> does it mean reiser4 has a compression plugin available now?
>

Guys, it is not available. It still requires a work and the creation 
interface
is on a discussion level.

Edward.
 
 

Re: Will I need to re-format my partition for using the compression plugin?

[Grzegorz Jaśkiewicz]

I think there should be file attribute that says whether it is
compressed/whatever/ or not. This way, it would be possible to
compress already existing files. At least, if filesystem is so
flexible as everyone say, it would be possible.

On 9/20/05, Edward Shishkin <edward@namesys.com> wrote:

> Guys, it is not available. It still requires a work and the creation
> interface
> is on a discussion level.


--
GJ

Re: Will I need to re-format my partition for using the compression plugin?

[Edward Shishkin]

Grzegorz Jaśkiewicz wrote:

>I think there should be file attribute that says whether it is
>compressed/whatever/ or not.
>

Right. All the plugins responsible for compression, etc.. will be observed
via pseudo-file interface

> This way, it would be possible to
>compress already existing files. At least, if filesystem is so
>flexible as everyone say, it would be possible.
>  
>

It will be possible to turn on/off compression at any
time if a file is powered by cryptcompress plugin id.
AFAIK changing file plugin id will be impossible (at least in reiser4)

Edward.

>On 9/20/05, Edward Shishkin <edward@namesys.com> wrote:
>
>  
>
>>Guys, it is not available. It still requires a work and the creation
>>interface
>>is on a discussion level.
>>    
>>
>
>
>--
>GJ
>  
>

Re: Will I need to re-format my partition for using the compression plugin?

[Fred Schaettgen]

On Thursday, 22. September 2005 18:41, Edward Shishkin wrote:
> Grzegorz Jaśkiewicz wrote:
> >I think there should be file attribute that says whether it is
> >compressed/whatever/ or not.
>
> Right. All the plugins responsible for compression, etc.. will be observed
> via pseudo-file interface
>
> > This way, it would be possible to
> >compress already existing files. At least, if filesystem is so
> >flexible as everyone say, it would be possible.
>
> It will be possible to turn on/off compression at any
> time if a file is powered by cryptcompress plugin id.
> AFAIK changing file plugin id will be impossible (at least in reiser4)

I don't quite understand how the file plugin concept scales once we get more 
of them. For instance if I want to have an additional attribute attached to 
my files, which contains a checksum that is kept up to date whenever the file 
is changed. I'd have to use a special file plugin for those files then, is 
this correct? 
In that case it's always an exclusive decision if I want my files checksummed, 
compressed, encrypted, monitored for changes or whatever.

Apparently the encryption and the compression is realized with one single file 
plugin, so it seems that there is indeed no way to have both features 
implemented as separate plugins. Or is there another reason why you packed 
both things into one plugin? If most new file features have to be implemented 
as part of one single plugin, then the whole concept of using file plugin 
starts looking a bit less promising in my eyes. 
I hope I'm just wrong or the plugin interface is not set in stone yet ;)

Fred

Re: Will I need to re-format my partition for using the compression plugin?

[Edward Shishkin]

Fred Schaettgen wrote:

>On Thursday, 22. September 2005 18:41, Edward Shishkin wrote:
>  
>
>>Grzegorz Jaśkiewicz wrote:
>>    
>>
>>>I think there should be file attribute that says whether it is
>>>compressed/whatever/ or not.
>>>      
>>>
>>Right. All the plugins responsible for compression, etc.. will be observed
>>via pseudo-file interface
>>
>>    
>>
>>>This way, it would be possible to
>>>compress already existing files. At least, if filesystem is so
>>>flexible as everyone say, it would be possible.
>>>      
>>>
>>It will be possible to turn on/off compression at any
>>time if a file is powered by cryptcompress plugin id.
>>AFAIK changing file plugin id will be impossible (at least in reiser4)
>>    
>>
>
>I don't quite understand how the file plugin concept scales once we get more 
>of them. For instance if I want to have an additional attribute attached to 
>my files, which contains a checksum that is kept up to date whenever the file 
>is changed. I'd have to use a special file plugin for those files then, is 
>this correct?
>
Yes. It is impossible to implement all features in one file plugin. 
Checksuming means a low
performance: in order to read some bytes of such file you will need 
first to read the whole file
to check a checksum (isnt it?). So it will be suitable for a small 
number of special files.
To write this new file plugin you will want to use already existing 
cipher and compression
transform plugins (dont mix it with cryptcompress file plugin which also 
calls those plugins)
to compress and encrypt your checksumed file.
 

>In that case it's always an exclusive decision if I want my files checksummed, 
>compressed, encrypted, monitored for changes or whatever.
>
>Apparently the encryption and the compression is realized with one single file 
>plugin, so it seems that there is indeed no way to have both features 
>implemented as separate plugins.
>

Why do you need separate ones? Having only a cryptcompress file plugin 
you will be able
to create files which are either only encrypted or only compressed, just 
set the transform
plugins properly.

> Or is there another reason why you packed 
>both things into one plugin?
>

Because sometimes it is useful to compress data before encryption since 
compression
destroys vulnerable regular structure of some special files (like *.html)

>If most new file features have to be implemented 
>as part of one single plugin,
>

Then this plugin will look like a monster ;)
Not necessarily. On the oher hand, creating a separate file plugin for 
each feature
imho is not a good idea.

> then the whole concept of using file plugin
>starts looking a bit less promising in my eyes. 
>I hope I'm just wrong or the plugin interface is not set in stone yet ;)
>
>Fred
>
>
>  
>

Re: Will I need to re-format my partition for using the compression plugin?

[David Masover]

Edward Shishkin wrote:
> Fred Schaettgen wrote:
> 
>> On Thursday, 22. September 2005 18:41, Edward Shishkin wrote:
>>  
>>
>>> Grzegorz Jaśkiewicz wrote:


>> Or is there another reason why you packed both things into one plugin?
>>
> 
> Because sometimes it is useful to compress data before encryption since 
> compression
> destroys vulnerable regular structure of some special files (like *.html)

Although I'd imagine some algorithms are fairly resistant against that 
(RSA, maybe?), the main reason is simple -- encryption tends to 
introduce randomness.  If the crypto is any good at all, you won't be 
able to compress very well after you've encrypted.

Anyway, can't you stack plugins?  Or at least create one new plugin out 
of two existing ones?

>> If most new file features have to be implemented as part of one single 
>> plugin,
>>
> 
> Then this plugin will look like a monster ;)
> Not necessarily. On the oher hand, creating a separate file plugin for 
> each feature
> imho is not a good idea.

How about some other examples of things to group together and things to 
not group together?

I wonder if it would make sense to try to optimize for a huge number of 
plugins?  It'd be harder than optimizing for a huge number of files per 
directory, but would it be possible?

Re: Will I need to re-format my partition for using the compression plugin?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Fri, 23 Sep 2005 00:03:32 +0400, Edward Shishkin said:

> Checksuming means a low
> performance: in order to read some bytes of such file you will need 
> first to read the whole file
> to check a checksum (isnt it?).

No.  Almost all modern networking gear is *perfectly* able to do incremental
updates of the checksum.  See this RFC:

1141 Incremental updating of the Internet checksum. T. Mallory, A.
     Kullberg. Jan-01-1990. (Format: TXT=3587 bytes) (Updates RFC1071)
     (Updated by RFC1624) (Status: INFORMATIONAL)
http://www.ietf.org/rfc/rfc1141.txt

The method is trivially extensible to other CRC schemes - and in fact, the
triviality is the entire reason why cryptographically strong hashes like MD5 or
the SHA family are interesting at all.  (I've seen more than one definition of
"cryptographically strong hash" as being basically a CRC function that does
*not* permit incremental updating....)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Will I need to re-format my partition for using the compression plugin?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

On Thu, 22 Sep 2005 15:11:59 CDT, David Masover said:

> > Because sometimes it is useful to compress data before encryption since compression
> > destroys vulnerable regular structure of some special files (like *.html)
> 
> Although I'd imagine some algorithms are fairly resistant against that 
> (RSA, maybe?), the main reason is simple -- encryption tends to 
> introduce randomness.  If the crypto is any good at all, you won't be 
> able to compress very well after you've encrypted.

1) RSA is useless for this - you really need a symmetric block cipher of some
sort.  Almost all block ciphers are best used with maximum-entropy input - if
the attacker can lop out a large part of the keyspace, a brute force attack
becomes a lot easier.  This is somewhat related to the concept of "Hamming
Distance". If the attacker tries a brute force attack, and the first 8 bytes of
the output look like valid HTML, or English text, or anything else
recognizable, he's almost certainly found found the correct key.  On the other
hand, well-compressed data has very high entropy - as a result, it becomes
harder to tell if a correct key has been found.  If it's English text, but
3 of the first 8 bytes have the high bit set, it's probably not a correct key.
If it's compressed, 3 flipped bits in the first 8 bytes will probably still
represent a valid compressed stream - just of something else wildly different.

2) Even though most modern block ciphers are designed to be fast, it's still
faster to apply a reasonably quick compression scheme to whomp 16K of data
down to 5-6K and encrypt/decrypt 5-6k than it is to encrypt/decrypt 16K.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Will I need to re-format my partition for using the compression plugin?

[michael chang]

On 9/22/05, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote:
> 2) Even though most modern block ciphers are designed to be fast, it's still
> faster to apply a reasonably quick compression scheme to whomp 16K of data
> down to 5-6K and encrypt/decrypt 5-6k than it is to encrypt/decrypt 16K.

Two questions.  One, does this mean that compression will usually be
performed before encryption (which to me, sounds like it appears to be
what would be the best method here)?

Second, will plugins be layerable, e.g. can I use a compression plugin
and a _seperate_ encryption plugin on the _same_ file, and then if so,
can I choose which order in which they take effect?

--
~Mike
 - Just my two cents
 - No man is an island, and no man is unable.

Re: Will I need to re-format my partition for using the compression plugin?

[Gregory Maxwell]

On 9/22/05, Edward Shishkin <edward@namesys.com> wrote:
> Yes. It is impossible to implement all features in one file plugin.
> Checksuming means a low
> performance: in order to read some bytes of such file you will need
> first to read the whole file
> to check a checksum (isnt it?). So it will be suitable for a small
> number of special files.
> To write this new file plugin you will want to use already existing
> cipher and compression
> transform plugins (dont mix it with cryptcompress file plugin which also
> calls those plugins)
> to compress and encrypt your checksumed file.

For file data integrity it would actually be more useful to have a per
block hash or checksum. This solves the update problem.  It would be
useful if the file offset and some file identifier were also mixed
into the calculation so that a misplaced block will fail as well. This
would fit quite nicely into the existing actions of the cryptocompress
plugin, and could be accomplished as just another compression algo..
one that always adds the 64-256 bits of check data per block.. At
least as long as the error handling in the FS is robust enough to be
able to treat a decompression failure as an IO error.  ...

If it were desirable to produce a cryptographically strong checksum
which can be handed to the user, what you would do is perform a hash
per block, and store that with each block, then a hash of the hashes
which is returned to the user. This is called a tree hash (google it),
usually you have a deeper hierarchy than two, depends on the
application. This makes incremental updates cheap enough (just hash
the block, then ripple the changes up the tree).  This would remove
the ability to include the file id and offsets directly in the hash,
but i would argue that they should still be used: for example you
could xor the hash value with them before writing it to disk and on
reading it from disk. This would still allow you to detect a misplaced
block but would not make the tree value differ for multiple copies of
the same file.

Re: Will I need to re-format my partition for using the compression plugin?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 926 bytes --]

On Thu, 22 Sep 2005 16:54:12 EDT, michael chang said:
> On 9/22/05, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote:
> > 2) Even though most modern block ciphers are designed to be fast, it's still
> > faster to apply a reasonably quick compression scheme to whomp 16K of data
> > down to 5-6K and encrypt/decrypt 5-6k than it is to encrypt/decrypt 16K.
> 
> Two questions.  One, does this mean that compression will usually be
> performed before encryption (which to me, sounds like it appears to be
> what would be the best method here)?

Yes, the general rule is "compress, then encrypt" and "Decrypt then decompress".

The corner cases where it should be the other way around are so few and far
between that it's not worth worrying about - they basically center around those
few times when compression causes a *larger* payload, and can be dealt with
a simple "Don't compress if output is bigger than input" rule.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Will I need to re-format my partition for using the compression plugin?

[PFC]

> Why do you need separate ones? Having only a cryptcompress file plugin  
> you will be able
> to create files which are either only encrypted or only compressed, just  
> set the transform
> plugins properly.

	It also make sense to have compression and crypto "close" to each other,  
which lets the data stay in the processor cache between the two...

Re: Will I need to re-format my partition for using the compression plugin?

[Fred Schaettgen]

On Thursday, 22. September 2005 22:03, Edward Shishkin wrote:
> Fred Schaettgen wrote:
> >I don't quite understand how the file plugin concept scales once we get
> > more of them. For instance if I want to have an additional attribute
> > attached to my files, which contains a checksum that is kept up to date
> > whenever the file is changed. I'd have to use a special file plugin for
> > those files then, is this correct?
>
> Yes. It is impossible to implement all features in one file plugin.
> Checksuming means a low performance: 

It was just meant as an example for another possible feature of a file plugin, 
so let's forget about if it's slow or not for the moment or if it even makes 
no sense at all.
Replace it by any other file plugin you can imagine. Like a virus-scanning 
file plugin, a change log plugin, a versioned-file plugin, whatever.

The question was more about combining new features for single files in 
general. At some point I might want to have a file which is both compressed 
and which supports feature X at the same time.

But I can't do that (at least I think so). I can either patch the 
cryptcompress plugin, which is certainly not the right way(tm).
Or I can make up another file plugin, but then it can't be compressed at the 
same time. If we have as few as five different file plugins, such a 
limitation might hurt already.

There seems to be no way to create sort of a pipeline of plugins, it uses 
either one file plugin or another one.

...
> To write this new file plugin you will want to use already existing
> cipher and compression
> transform plugins (dont mix it with cryptcompress file plugin which also
> calls those plugins)
> to compress and encrypt your checksumed file.
...
> >Apparently the encryption and the compression is realized with one single
> > file plugin, so it seems that there is indeed no way to have both
> > features implemented as separate plugins.
>
> Why do you need separate ones? Having only a cryptcompress file plugin
> you will be able
> to create files which are either only encrypted or only compressed, just
> set the transform
> plugins properly.

I haven't heard about transform plugins at all so far. Are they meant to be 
used by the cryptcompress plugin only or is this a broader concept that can 
they be used by regular files too?

Sounds like it's a cryptcompress special, but "transform plugin" sounds like 
what I'm missing. It would be very heplful if I could attach any number of 
transform plugins to any single file.
Like that: VFS -> virus scanner -> change monitor -> versioning -> compression 
-> encryption -> checksum -> DISK

Let's hope that no one takes the the virus scanner part literally, it has the 
potential of feeding the reiser-eating trolls.
It's not about being able to create endless transformation pipelines and 
hundreds of plugins, but just the mere possibility to create an short 
arbitrary pipeline whenever it makes sense.

> > Or is there another reason why you packed
> >both things into one plugin?
>
> Because sometimes it is useful to compress data before encryption since
> compression
> destroys vulnerable regular structure of some special files (like *.html)
>
> >If most new file features have to be implemented
> >as part of one single plugin,
>
> Then this plugin will look like a monster ;)
> Not necessarily. On the oher hand, creating a separate file plugin for
> each feature
> imho is not a good idea.

Right, it isn't, because it's not possible to combine them. 
But if I do want to combine them, then what?

Hm, on second thought... forget about it for the moment. I shouldn't waste 
your time while reiser4 is trying so hard to find it's way into the mainline 
kernel. Let's hope that this fight will be over soon.

Fred

Re: Will I need to re-format my partition for using the compression plugin?

[Gregory Maxwell]

On 9/22/05, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote:
> 1) RSA is useless for this - you really need a symmetric block cipher of some
> sort.  Almost all block ciphers are best used with maximum-entropy input - if
> the attacker can lop out a large part of the keyspace, a brute force attack
> becomes a lot easier.  This is somewhat related to the concept of "Hamming
> Distance". If the attacker tries a brute force attack, and the first 8 bytes of
> the output look like valid HTML, or English text, or anything else
> recognizable, he's almost certainly found found the correct key.  On the other
> hand, well-compressed data has very high entropy - as a result, it becomes
> harder to tell if a correct key has been found.  If it's English text, but
> 3 of the first 8 bytes have the high bit set, it's probably not a correct key.
> If it's compressed, 3 flipped bits in the first 8 bytes will probably still
> represent a valid compressed stream - just of something else wildly different.

It would normally seem silly to use RSA for disk encryption... but
there might be applications, although you'd still never use RSA
directly on user controlled data.  For example, RSA could be used on a
multi user server to append mail to a mail file so that once written
the data is only accessible once the user logs on.  The reiser4 crypto
system will use the kernel keyring api, so it would be quite
reasonable to tie encryption to user accounts. 'write only' files and
'read only' files would be a simple logical extension, and would
require asymetric cryptography.

Although for most compression algorithms not all inputs are valid
outputs, so this may not work for you... It would be ideal (for disk
encryption) if it were not possible to tell if you have the right key
without decrypting an entire sector. This requires careful selection
in compression and chaining mode.  Alternatively, it may be possible
to develop a good large block cipher which while being much slower
than a single block of a small-block cipher, is faster for a disk
block.  For example, mercy is about 4x faster than AES on my system
but is still 16x slower for the smallest unit of decryption than AES.
Unfortunately mercy has security problems.

> 2) Even though most modern block ciphers are designed to be fast, it's still
> faster to apply a reasonably quick compression scheme to whomp 16K of data
> down to 5-6K and encrypt/decrypt 5-6k than it is to encrypt/decrypt 16K.

Depends on the compression mode and the cipher. A good AES
implementation is around the same speed as an aggressive gzip. In
general this is correct.

Re: Will I need to re-format my partition for using the compression plugin?

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 4385 bytes --]

On Thu, 22 Sep 2005 18:13:23 EDT, Gregory Maxwell said:

> It would normally seem silly to use RSA for disk encryption... but
> there might be applications, although you'd still never use RSA
> directly on user controlled data.  For example, RSA could be used on a
> multi user server to append mail to a mail file so that once written
> the data is only accessible once the user logs on.  The reiser4 crypto
> system will use the kernel keyring api, so it would be quite
> reasonable to tie encryption to user accounts. 'write only' files and
> 'read only' files would be a simple logical extension, and would
> require asymetric cryptography.

In fact, RSA would *still* be a poor choice there - the CPU costs go up
exponentially with the size of the object encrypted.  And if you have a 64K
sized files, that means if you use RSA directly, you get to do mathematics with
524,288 bit numbers.  Yep, multiply a 524,288 bit number by a 1024 bit number
and then compute the remainder when divided by another 1024 bit number. Lather,
rinse, repeat. ;)

You know how sites that do a lot of SSL buy special hardware accelerators?
The only *real* benefit they give you is offloading the CPU cost of doing
RSA over a 128-bit or so session key.

OK. Got that?  Doing RSA over a 16 byte file "costs" as much as opening a
standard 128-bit encryption SSL connection (because it's basically the same
thing).  And a 17 byte file costs you a lot more than 8 times as much.  And a
32 byte file isn't 16 times worse, it's *hundreds* of times worse.

That's why *nobody* uses RSA for anything other than securing a good-sized
symmetric session key.  So for this use, you'd use RSA to secure the file's
actual symmetric key (and possibly things like the initialization vectors).

(Note to designers - those pesky IV's are a *lot* trickier to get right
than you might think.  For instance, there's a known watermarking attack
against the current cryptoloop implementation in the kernel that allows
an attacker to prove the existence of data on the disk even without the
key - so a DRM scheme could find watermarked data even *after* encryption).

> Although for most compression algorithms not all inputs are valid
> outputs, so this may not work for you... It would be ideal (for disk
> encryption) if it were not possible to tell if you have the right key
> without decrypting an entire sector. This requires careful selection
> in compression and chaining mode.

In fact, Hamming distance considerations imply that usually you don't
need to decrypt more than 1 or 2 (*maybe* 3) blocks the size of the
symmetric cypher's blocksize.  For something like AES-256, you can probably
be sure in 32 bytes (1 block), very sure in 64 bytes, and totally sure in 128
bytes (unless the attacker has the misfortune to be trying to decrypt a file
that has actual structure on the same order as /dev/random output).

>                                   Alternatively, it may be possible
> to develop a good large block cipher which while being much slower
> than a single block of a small-block cipher, is faster for a disk
> block.  For example, mercy is about 4x faster than AES on my system
> but is still 16x slower for the smallest unit of decryption than AES.
> Unfortunately mercy has security problems.

Tough design challenge there.

The problem is that if you have a cipher that can handle 512-*byte* input
blocks, it's going to probably stomp on a *lot* of L1 and L2 cache lines.
And you can't even rely on the usual pre-expansion tricks because that adds
even *more* to cache pressure.

Another desirable property of symmetric ciphers is that they tend to change
about half the output bits for a single-bit input change, and in an unpredictable
manner.  This ends up meaning that you'll probably need O(log2 N) rounds, and
more likely closer to O(N) rounds, to mix the pool.  Gonna be a *lot* of rounds
for a 512-byte block. ;)

> > 2) Even though most modern block ciphers are designed to be fast, it's still
> > faster to apply a reasonably quick compression scheme to whomp 16K of data
> > down to 5-6K and encrypt/decrypt 5-6k than it is to encrypt/decrypt 16K.
> 
> Depends on the compression mode and the cipher. A good AES
> implementation is around the same speed as an aggressive gzip. In
> general this is correct.

That's why you don't use an *aggressive* gzip, but use 'gzip -3' instead. :)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Will I need to re-format my partition for using the compression plugin?

[Edward Shishkin]

michael chang wrote:

>Second, will plugins be layerable, e.g. can I use a compression plugin
>and a _seperate_ encryption plugin on the _same_ file,
>

There is a number of specific attributes including compression and 
encryption
transform plugin ids, so user can assign it separately when creating a file.
If you dont want some transform to be present, you should assign a special
"none" plugin id for this transform.

> and then if so,
>can I choose which order in which they take effect?
>  
>

What effect do you mean?
The order is hardcoded and can not be changed: compression is first.

Edward.

>--
>~Mike
> - Just my two cents
> - No man is an island, and no man is unable.
>
>
>  
>

Re: Will I need to re-format my partition for using the compression plugin?

[Edward Shishkin]

Fred Schaettgen wrote:

>On Thursday, 22. September 2005 22:03, Edward Shishkin wrote:
>  
>
>>Fred Schaettgen wrote:
>>    
>>
>>>I don't quite understand how the file plugin concept scales once we get
>>>more of them. For instance if I want to have an additional attribute
>>>attached to my files, which contains a checksum that is kept up to date
>>>whenever the file is changed. I'd have to use a special file plugin for
>>>those files then, is this correct?
>>>      
>>>
>>Yes. It is impossible to implement all features in one file plugin.
>>Checksuming means a low performance: 
>>    
>>
>
>It was just meant as an example for another possible feature of a file plugin, 
>so let's forget about if it's slow or not for the moment or if it even makes 
>no sense at all.
>Replace it by any other file plugin you can imagine. Like a virus-scanning 
>file plugin, a change log plugin, a versioned-file plugin, whatever.
>
>The question was more about combining new features for single files in 
>general. At some point I might want to have a file which is both compressed 
>and which supports feature X at the same time.
>  
>

Usually new features are incompartible with the disign of existing
objects, so I think there is no any general policy.

>But I can't do that (at least I think so). I can either patch the 
>cryptcompress plugin, which is certainly not the right way(tm).
>Or I can make up another file plugin, but then it can't be compressed at the 
>same time. If we have as few as five different file plugins, such a 
>limitation might hurt already.
>
>There seems to be no way to create sort of a pipeline of plugins, it uses 
>either one file plugin or another one.
>
>...
>  
>
>>To write this new file plugin you will want to use already existing
>>cipher and compression
>>transform plugins (dont mix it with cryptcompress file plugin which also
>>calls those plugins)
>>to compress and encrypt your checksumed file.
>>    
>>
>...
>  
>
>>>Apparently the encryption and the compression is realized with one single
>>>file plugin, so it seems that there is indeed no way to have both
>>>features implemented as separate plugins.
>>>      
>>>
>>Why do you need separate ones? Having only a cryptcompress file plugin
>>you will be able
>>to create files which are either only encrypted or only compressed, just
>>set the transform
>>plugins properly.
>>    
>>
>
>I haven't heard about transform plugins at all so far. Are they meant to be 
>used by the cryptcompress plugin only or is this a broader concept that can 
>they be used by regular files too?
>
>Sounds like it's a cryptcompress special, but "transform plugin" sounds like 
>what I'm missing. It would be very heplful if I could attach any number of 
>transform plugins to any single file.
>  
>

Each file has a so-called plugin set, so there will be a common mechanism of
assignment/inheritance which is under development

>Like that: VFS -> virus scanner -> change monitor -> versioning -> compression 
>-> encryption -> checksum -> DISK
>
>Let's hope that no one takes the the virus scanner part literally, it has the 
>potential of feeding the reiser-eating trolls.
>It's not about being able to create endless transformation pipelines and 
>hundreds of plugins, but just the mere possibility to create an short 
>arbitrary pipeline whenever it makes sense.
>
>  
>
>>>Or is there another reason why you packed
>>>both things into one plugin?
>>>      
>>>
>>Because sometimes it is useful to compress data before encryption since
>>compression
>>destroys vulnerable regular structure of some special files (like *.html)
>>
>>    
>>
>>>If most new file features have to be implemented
>>>as part of one single plugin,
>>>      
>>>
>>Then this plugin will look like a monster ;)
>>Not necessarily. On the oher hand, creating a separate file plugin for
>>each feature
>>imho is not a good idea.
>>    
>>
>
>Right, it isn't, because it's not possible to combine them. 
>But if I do want to combine them, then what?
>
>Hm, on second thought... forget about it for the moment. I shouldn't waste 
>your time while reiser4 is trying so hard to find it's way into the mainline 
>kernel. Let's hope that this fight will be over soon.
>
>Fred
>
>
>  
>