Reiser4 documentation & specifically current status of repacker, compression, and semantics

Reiser4 documentation & specifically current status of repacker, compression, and semantics

[John Gilmore]

There is a 'compress' directory in in my reiser4 source tree. So is the 
compression plugin done? Is my disk/files compressed? And if it was/wasn't 
how would I know the difference, and change it?

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[John Gilmore]

On Monday 24 October 2005 14:44, Edward Shishkin wrote:
> John Gilmore wrote:
> >There is a 'compress' directory in in my reiser4 source tree. So is the
> >compression plugin done? Is my disk/files compressed?
>
> not yet,
> this is under construction..
>
> > And if it was/wasn't
> >how would I know the difference, and change it?
>
> I guess reiser4progs will be able to dump the attributes then
>
> Edward.
I thought the ability to muck around with the plugins assigned to  a 
particular file was implemented as file-as-directory? I've tried to re-enable 
that ability, but haven't been able to get it to work. How are plugin/file 
relationships handled without it?

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[John Gilmore]

On Monday 24 October 2005 19:08, Edward Shishkin wrote:
> John Gilmore wrote:
> > How are plugin/file
> >relationships handled without it?
>
> For the first time there will be an option in mkfs to assign a file
> plugin for regular files per superblock.
> Then (if everything will be okay) we will granulate the relationship.
>
> Edward.

So the first beta of compression will only have the option to control it at 
mkfs time? And then later on, compression control and status information will 
be available on a per-file (or per-directory) basis via a user-space tool 
operating on mounted filesystems? Which tool will also be extendable to 
control and give status information (where applicable) on other types of 
plugins?

Is there a page that I could go to - some sort of "Major feature changes" page 
- to see what the current status of the filesystem is? To answer such 
questions as "Is compression ready for beta?" and "what the heck happened to 
file-as-directory?" and "I saw mention of a test of the reiser4 repacker, and 
also mention of compile problems with repacker.c, which definitely doesn't 
exist in MY version of resier4 - what gives?" (though I've concluded that 
that last one must be somebodies copy of alpha software...)

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Edward Shishkin]

John Gilmore wrote:

>There is a 'compress' directory in in my reiser4 source tree. So is the 
>compression plugin done? Is my disk/files compressed?
>

not yet,
this is under construction..

> And if it was/wasn't 
>how would I know the difference, and change it?
>  
>

I guess reiser4progs will be able to dump the attributes then

Edward.

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Hans Reiser]

John Gilmore wrote:

>There is a 'compress' directory in in my reiser4 source tree. So is the 
>compression plugin done?
>
No, it is in an "any couple of weeks now"state.....

> Is my disk/files compressed? And if it was/wasn't 
>how would I know the difference, and change it?
>
>
>  
>

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Edward Shishkin]

John Gilmore wrote:

>On Monday 24 October 2005 14:44, Edward Shishkin wrote:
>  
>
>>John Gilmore wrote:
>>    
>>
>>>There is a 'compress' directory in in my reiser4 source tree. So is the
>>>compression plugin done? Is my disk/files compressed?
>>>      
>>>
>>not yet,
>>this is under construction..
>>
>>    
>>
>>>And if it was/wasn't
>>>how would I know the difference, and change it?
>>>      
>>>
>>I guess reiser4progs will be able to dump the attributes then
>>
>>Edward.
>>    
>>
>I thought the ability to muck around with the plugins assigned to  a 
>particular file was implemented as file-as-directory? I've tried to re-enable 
>that ability, but haven't been able to get it to work.
>

This code (pseudo plugin) has been removed as mainline folks required.
It will be supported as an optional patch a bit later

> How are plugin/file 
>relationships handled without it?
>
>  
>

For the first time there will be an option in mkfs to assign a file 
plugin for regular files per superblock.
Then (if everything will be okay) we will granulate the relationship.

Edward.

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Hans Reiser]

John Gilmore wrote:

>On Monday 24 October 2005 19:08, Edward Shishkin wrote:
>  
>
>>John Gilmore wrote:
>>    
>>
>>>How are plugin/file
>>>relationships handled without it?
>>>      
>>>
>>For the first time there will be an option in mkfs to assign a file
>>plugin for regular files per superblock.
>>Then (if everything will be okay) we will granulate the relationship.
>>
>>Edward.
>>    
>>
>
>So the first beta of compression will only have the option to control it at 
>mkfs time?
>
It should be explained that on the first flush to disk of a new file the
default compress plugin will test to see if the first 64k is
compressable, and if it is not then it converts the file to a regular
file plugin.

Edward, confirm that you coded it that way, as I think I expressed
things more precisely this time.;-)

I believe most users will find most of their files get the right
treatment using this heuristic.

> And then later on, compression control and status information will 
>be available on a per-file (or per-directory) basis via a user-space tool 
>operating on mounted filesystems? Which tool will also be extendable to 
>control and give status information (where applicable) on other types of 
>plugins?
>
>Is there a page that I could go to - some sort of "Major feature changes" page 
>- to see what the current status of the filesystem is?
>
If you pay for the tech writer.:-/  We have less funding than most users
imagine we have.

> To answer such 
>questions as "Is compression ready for beta?" and "what the heck happened to 
>file-as-directory?" and "I saw mention of a test of the reiser4 repacker, and 
>also mention of compile problems with repacker.c, which definitely doesn't 
>exist in MY version of resier4 - what gives?" (though I've concluded that 
>that last one must be somebodies copy of alpha software...)
>
>
>  
>

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Edward Shishkin]

Hans Reiser wrote:

>John Gilmore wrote:
>
>  
>
>>On Monday 24 October 2005 19:08, Edward Shishkin wrote:
>> 
>>
>>    
>>
>>>John Gilmore wrote:
>>>   
>>>
>>>      
>>>
>>>>How are plugin/file
>>>>relationships handled without it?
>>>>     
>>>>
>>>>        
>>>>
>>>For the first time there will be an option in mkfs to assign a file
>>>plugin for regular files per superblock.
>>>Then (if everything will be okay) we will granulate the relationship.
>>>
>>>Edward.
>>>   
>>>
>>>      
>>>
>>So the first beta of compression will only have the option to control it at 
>>mkfs time?
>>
>>    
>>
>It should be explained that on the first flush to disk of a new file the
>default compress plugin will test to see if the first 64k is
>compressable, and if it is not then it converts the file to a regular
>file plugin.
>
>  
>

a little correction:
this is a compression transform plugin that is converted at flush time, 
not file plugin

>Edward, confirm that you coded it that way, as I think I expressed
>things more precisely this time.;-)
>
>I believe most users will find most of their files get the right
>treatment using this heuristic.
>
>  
>
>>And then later on, compression control and status information will 
>>be available on a per-file (or per-directory) basis via a user-space tool 
>>operating on mounted filesystems? Which tool will also be extendable to 
>>control and give status information (where applicable) on other types of 
>>plugins?
>>
>>Is there a page that I could go to - some sort of "Major feature changes" page 
>>- to see what the current status of the filesystem is?
>>
>>    
>>
>If you pay for the tech writer.:-/  We have less funding than most users
>imagine we have.
>
>  
>
>>To answer such 
>>questions as "Is compression ready for beta?" and "what the heck happened to 
>>file-as-directory?" and "I saw mention of a test of the reiser4 repacker, and 
>>also mention of compile problems with repacker.c, which definitely doesn't 
>>exist in MY version of resier4 - what gives?" (though I've concluded that 
>>that last one must be somebodies copy of alpha software...)
>>
>>
>> 
>>
>>    
>>
>
>
>
>  
>

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Hans Reiser]

Edward Shishkin wrote:

> Hans Reiser wrote:
>
>> John Gilmore wrote:
>>
>>> So the first beta of compression will only have the option to
>>> control it at mkfs time?
>>>
>>>   
>>
>> It should be explained that on the first flush to disk of a new file the
>> default compress plugin will test to see if the first 64k is
>> compressable, and if it is not then it converts the file to a regular
>> file plugin.
>>
>>  
>>
>
> a little correction:
> this is a compression transform plugin that is converted at flush
> time, not file plugin
>
Please define in more detail please.  Particularly, define whether large
files that cannot be compressed get stored using extents and everything
else that regular files use.

Also, I thought we convert at first flush time, not at every flush
time.  Did you write it to check at every flush?  Not entirely sure we
want to do that for large multimedia files that are compressed by
hardware, it could add a bunch of CPU, and I would think it would be
more complex code, because now you have to convert extents, etc.  I
would be concerned that if you always have to check for compression with
every little write to a file, the cost of that check will be substantial
for many purposes.

Hans

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Edward Shishkin]

Hans Reiser wrote:

>Edward Shishkin wrote:
>
>  
>
>>Hans Reiser wrote:
>>
>>    
>>
>>>John Gilmore wrote:
>>>
>>>      
>>>
>>>>So the first beta of compression will only have the option to
>>>>control it at mkfs time?
>>>>
>>>>  
>>>>        
>>>>
>>>It should be explained that on the first flush to disk of a new file the
>>>default compress plugin will test to see if the first 64k is
>>>compressable, and if it is not then it converts the file to a regular
>>>file plugin.
>>>
>>> 
>>>
>>>      
>>>
>>a little correction:
>>this is a compression transform plugin that is converted at flush
>>time, not file plugin
>>
>>    
>>
>Please define in more detail please.
>

http://www.namesys.com/cryptcompress_design.html
"Handling incompressible data"
and
http://www.namesys.com/cryptcompress-related_plugins.html
"Compression mode plugin"

>  Particularly, define whether large
>files that cannot be compressed get stored using extents and everything
>else that regular files use.
>
>  
>

Currently files powered by cryptcompress file plugin have its data 
stored only in the
items of CTAIL_ID

>Also, I thought we convert at first flush time, not at every flush
>time.  Did you write it to check at every flush? 
>

no, just turning off and no anymore check, but it can be easily changed

> Not entirely sure we
>want to do that for large multimedia files that are compressed by
>hardware, it could add a bunch of CPU, and I would think it would be
>more complex code, because now you have to convert extents, etc.
>

Yes, such converting seems to be a complicated workaround. We will need
something like "clustered extents"


>  I
>would be concerned that if you always have to check for compression with
>every little write to a file, the cost of that check will be substantial
>for many purposes.
>
>  
>

Since compression is going per cluster, we can consider a set of nominated
clusters as a pattern to check if we need to turn compression off[on] 
for the
whole file. Say, if the clusters of indexes #(N*k), k=0, 1, 2, ...  are
incompressible[compressible], then turn compression off[on].
The smaller N - the more substantial cost of check in the case when 
compression is off.

>Hans
>
>
>  
>

Re: Reiser4 documentation & specifically current status of repacker, compression, and semantics

[Edward Shishkin]

Edward Shishkin wrote:

> Hans Reiser wrote:
>
>>  Particularly, define whether large
>> files that cannot be compressed get stored using extents and everything
>> else that regular files use.
>>
>>  
>>
>
> Currently files powered by cryptcompress file plugin have its data 
> stored only in the
> items of CTAIL_ID
>
>> Also, I thought we convert at first flush time, not at every flush
>> time.  Did you write it to check at every flush?
>
>
> no, just turning off and no anymore check, but it can be easily changed
>
>> Not entirely sure we
>> want to do that for large multimedia files that are compressed by
>> hardware, it could add a bunch of CPU, and I would think it would be
>> more complex code, because now you have to convert extents, etc.
>>
>
> Yes, such converting seems to be a complicated workaround. We will need
> something like "clustered extents"
>

So having such "cextents" instead of incompressible ctails on leaf level
will reduce (/=2 times) memory usage and response to emergency flush
(currently there are no any chances for cryptcompress files to be 
eflushed).

Edward.