how do plugins work?

how do plugins work?

[Gregor Zeitlinger]

Hi,

I'm new to this list. I'm wondering how the file plugins work. Are they
just a means of accessing the file contents or are they also designed to
save the file differently.

For example, could you create a plugin that automatically zips each
text/plain file, which is totally transparent to the user?
If so, can those plugins be hirachical. Like: when I save a .tex file its
automatically zipped and encrypted before writtten to disk.
Or even better: When I save a file, its automatically zipped, encrypted
and written over a ssh or ftp connection.

More precisely, I'm thinking about a plugin that stores xml files in an
efficient manner. Somewhat like a btree. I'm not sure wheater you've
already been through this issue, but I didn't find anything in the
archives. The issue is somewhat difficult, because I'd love to be able to
use xpath (possibly dom or xquery) to retreive/update xml data
EFFICIENTY.

I've gotten some ideas from a university project I'm working on. There
we're trying to do a whole xml database management system. Although
progress is terribly slow there are some good algos that might be
reusable.

-- 
Gregor Zeitlinger
gregor@zeitlinger.de

Re: how do plugins work?

[Edward Shishkin]

Gregor Zeitlinger wrote:
> 
> Hi,
> 
> I'm new to this list. I'm wondering how the file plugins work. Are they
> just a means of accessing the file contents or are they also designed to
> save the file differently.
> 
> For example, could you create a plugin that automatically zips each
> text/plain file, which is totally transparent to the user?

Yeah, we could. Although transparentness means a bit worse compression..
Edward.  

> If so, can those plugins be hirachical. Like: when I save a .tex file its
> automatically zipped and encrypted before writtten to disk.

It means this file is controlled by primary plugin of compressed object,
and there is installed subordinate compression plugin in its stat-data,
which provides zip-compression.  
Edward.


> Or even better: When I save a file, its automatically zipped, encrypted
> and written over a ssh or ftp connection.
> 
> More precisely, I'm thinking about a plugin that stores xml files in an
> efficient manner. Somewhat like a btree. I'm not sure wheater you've
> already been through this issue, but I didn't find anything in the
> archives. The issue is somewhat difficult, because I'd love to be able to
> use xpath (possibly dom or xquery) to retreive/update xml data
> EFFICIENTY.
> 
> I've gotten some ideas from a university project I'm working on. There
> we're trying to do a whole xml database management system. Although
> progress is terribly slow there are some good algos that might be
> reusable.
> 
> --
> Gregor Zeitlinger
> gregor@zeitlinger.de

Re: how do plugins work?

[Hans Reiser]

Edward Shishkin wrote:

>Gregor Zeitlinger wrote:
>  
>
>>Hi,
>>
>>I'm new to this list. I'm wondering how the file plugins work. Are they
>>just a means of accessing the file contents or are they also designed to
>>save the file differently.
>>
>>For example, could you create a plugin that automatically zips each
>>text/plain file, which is totally transparent to the user?
>>    
>>
>
>Yeah, we could. Although transparentness means a bit worse compression..
>
?

You mean because of choosing to use compression atoms that are node 
sized?  Maybe it might be more precise to say that efficient random 
access motivates small compression atoms which are less efficient? 
 (Assuming that I understood you.)

Re: how do plugins work?

[Edward Shishkin]

Hans Reiser wrote:
> 
> Edward Shishkin wrote:
> 
> >Gregor Zeitlinger wrote:
> >
> >
> >>Hi,
> >>
> >>I'm new to this list. I'm wondering how the file plugins work. Are they
> >>just a means of accessing the file contents or are they also designed to
> >>save the file differently.
> >>
> >>For example, could you create a plugin that automatically zips each
> >>text/plain file, which is totally transparent to the user?
> >>
> >>
> >
> >Yeah, we could. Although transparentness means a bit worse compression..
> >
> ?
> 
> You mean because of choosing to use compression atoms

Yes, but it is not the only reason.

> that are node sized?

No, compression atoms are limited by common sense ;)
Obviously, it doesn't make sense to choose it less then node size, if you
want to compress a file presented by tails. On the other hand, I am not
sure if you will be delighted from transparent compression supported by 256K
clusters even for larger files.

>  Maybe it might be more precise to say that efficient random
> access motivates small compression atoms which are less efficient?
>  (Assuming that I understood you.)

There is one more obscured reason that makes transparent compression worser.
I have to remind that its primary assignment is the same: to provide more 
efficient disk space usage. Let's take a look how transparent compression 
and the package using the same compression algorithm will provide it:
Suppose we want to compress a 8S-byte file, and both transparent compression
and package is supported by 4-blocks clusters, S - block size.

 File system:
divides the file into 2 clusters, suppose each cluster was compressed up to
(2S+1) bytes, so compressed file occupies 6 blocks.
 Package:
makes compressed file which occupies 2*(2S+1) = 4S+2 bytes and passes it
to the file system, which places it (aieee!) into 5 blocks..

  So we do have the actual compression ratio produced by the file system is
6/8 = 25% against actual ones (5/8 = 37,5%) produced by the package. The user
can observe this effect only by using "du" (when he does "df", he see that
file system and package produce the same compression ratio 37,5%).
This is another effect caused by the specific of packing policy (for the
ext2 files, reiserfs files presented by indirect items, etc..) Obviously
we can avoid this by using bigger cluster (8 blocks). 

Edward.

Re: how do plugins work?

[Hans Reiser]

Edward Shishkin wrote:

>Hans Reiser wrote:
>  
>
>>Edward Shishkin wrote:
>>
>>    
>>
>>>Gregor Zeitlinger wrote:
>>>
>>>
>>>      
>>>
>>>>Hi,
>>>>
>>>>I'm new to this list. I'm wondering how the file plugins work. Are they
>>>>just a means of accessing the file contents or are they also designed to
>>>>save the file differently.
>>>>
>>>>For example, could you create a plugin that automatically zips each
>>>>text/plain file, which is totally transparent to the user?
>>>>
>>>>
>>>>        
>>>>
>>>Yeah, we could. Although transparentness means a bit worse compression..
>>>
>>>      
>>>
>>?
>>
>>You mean because of choosing to use compression atoms
>>    
>>
>
>Yes, but it is not the only reason.
>
>  
>
>>that are node sized?
>>    
>>
>
>No, compression atoms are limited by common sense ;)
>Obviously, it doesn't make sense to choose it less then node size, if you
>want to compress a file presented by tails. On the other hand, I am not
>sure if you will be delighted from transparent compression supported by 256K
>clusters even for larger files.
>
>  
>
>> Maybe it might be more precise to say that efficient random
>>access motivates small compression atoms which are less efficient?
>> (Assuming that I understood you.)
>>    
>>
>
>There is one more obscured reason that makes transparent compression worser.
>I have to remind that its primary assignment is the same: to provide more 
>efficient disk space usage. Let's take a look how transparent compression 
>and the package using the same compression algorithm will provide it:
>Suppose we want to compress a 8S-byte file, and both transparent compression
>and package is supported by 4-blocks clusters, S - block size.
>
> File system:
>divides the file into 2 clusters, suppose each cluster was compressed up to
>(2S+1) bytes, so compressed file occupies 6 blocks.
> Package:
>makes compressed file which occupies 2*(2S+1) = 4S+2 bytes and passes it
>to the file system, which places it (aieee!) into 5 blocks..
>
>  So we do have the actual compression ratio produced by the file system is
>6/8 = 25% against actual ones (5/8 = 37,5%) produced by the package. The user
>can observe this effect only by using "du" (when he does "df", he see that
>file system and package produce the same compression ratio 37,5%).
>This is another effect caused by the specific of packing policy (for the
>ext2 files, reiserfs files presented by indirect items, etc..) Obviously
>we can avoid this by using bigger cluster (8 blocks). 
>
>Edward.
>
>
>  
>
Or by not block aligning the clusters, which might be the right 
solution, and might allow for compression atoms larger than nodes. 
 Please consider that approach.

Hans

Re: how do plugins work?

[Edward Shishkin]

Hans Reiser wrote:
> 
> Edward Shishkin wrote:
> 
> >Hans Reiser wrote:
> >
> >
> >>Edward Shishkin wrote:
> >>
> >>
> >>
> >>>Gregor Zeitlinger wrote:
> >>>
> >>>
> >>>
> >>>
> >>>>Hi,
> >>>>
> >>>>I'm new to this list. I'm wondering how the file plugins work. Are they
> >>>>just a means of accessing the file contents or are they also designed to
> >>>>save the file differently.
> >>>>
> >>>>For example, could you create a plugin that automatically zips each
> >>>>text/plain file, which is totally transparent to the user?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>Yeah, we could. Although transparentness means a bit worse compression..
> >>>
> >>>
> >>>
> >>?
> >>
> >>You mean because of choosing to use compression atoms
> >>
> >>
> >
> >Yes, but it is not the only reason.
> >
> >
> >
> >>that are node sized?
> >>
> >>
> >
> >No, compression atoms are limited by common sense ;)
> >Obviously, it doesn't make sense to choose it less then node size, if you
> >want to compress a file presented by tails. On the other hand, I am not
> >sure if you will be delighted from transparent compression supported by 256K
> >clusters even for larger files.
> >
> >
> >
> >> Maybe it might be more precise to say that efficient random
> >>access motivates small compression atoms which are less efficient?
> >> (Assuming that I understood you.)
> >>
> >>
> >
> >There is one more obscured reason that makes transparent compression worser.
> >I have to remind that its primary assignment is the same: to provide more
> >efficient disk space usage. Let's take a look how transparent compression
> >and the package using the same compression algorithm will provide it:
> >Suppose we want to compress a 8S-byte file, and both transparent compression
> >and package is supported by 4-blocks clusters, S - block size.
> >
> > File system:
> >divides the file into 2 clusters, suppose each cluster was compressed up to
> >(2S+1) bytes, so compressed file occupies 6 blocks.
> > Package:
> >makes compressed file which occupies 2*(2S+1) = 4S+2 bytes and passes it
> >to the file system, which places it (aieee!) into 5 blocks..
> >
> >  So we do have the actual compression ratio produced by the file system is
> >6/8 = 25% against actual ones (5/8 = 37,5%) produced by the package. The user
> >can observe this effect only by using "du" (when he does "df", he see that
> >file system and package produce the same compression ratio 37,5%).
> >This is another effect caused by the specific of packing policy (for the
> >ext2 files, reiserfs files presented by indirect items, etc..) Obviously
> >we can avoid this by using bigger cluster (8 blocks).
> >
> >Edward.
> >
> >
> >
> >
> Or by not block aligning the clusters, which might be the right
> solution,

It is impossible for the files presented by indirect items, since
it means you'll place dead space in memory per file.
 
> and might allow for compression atoms larger than nodes.
>  Please consider that approach.
> 
> Hans

Re: how do plugins work?

[Hans Reiser]

Edward Shishkin wrote:

>Hans Reiser wrote:
>  
>
>>Edward Shishkin wrote:
>>
>>    
>>
>>>Hans Reiser wrote:
>>>
>>>
>>>      
>>>
>>>>Edward Shishkin wrote:
>>>>
>>>>
>>>>
>>>>        
>>>>
>>>>>Gregor Zeitlinger wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>>>Hi,
>>>>>>
>>>>>>I'm new to this list. I'm wondering how the file plugins work. Are they
>>>>>>just a means of accessing the file contents or are they also designed to
>>>>>>save the file differently.
>>>>>>
>>>>>>For example, could you create a plugin that automatically zips each
>>>>>>text/plain file, which is totally transparent to the user?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>            
>>>>>>
>>>>>Yeah, we could. Although transparentness means a bit worse compression..
>>>>>
>>>>>
>>>>>
>>>>>          
>>>>>
>>>>?
>>>>
>>>>You mean because of choosing to use compression atoms
>>>>
>>>>
>>>>        
>>>>
>>>Yes, but it is not the only reason.
>>>
>>>
>>>
>>>      
>>>
>>>>that are node sized?
>>>>
>>>>
>>>>        
>>>>
>>>No, compression atoms are limited by common sense ;)
>>>Obviously, it doesn't make sense to choose it less then node size, if you
>>>want to compress a file presented by tails. On the other hand, I am not
>>>sure if you will be delighted from transparent compression supported by 256K
>>>clusters even for larger files.
>>>
>>>
>>>
>>>      
>>>
>>>>Maybe it might be more precise to say that efficient random
>>>>access motivates small compression atoms which are less efficient?
>>>>(Assuming that I understood you.)
>>>>
>>>>
>>>>        
>>>>
>>>There is one more obscured reason that makes transparent compression worser.
>>>I have to remind that its primary assignment is the same: to provide more
>>>efficient disk space usage. Let's take a look how transparent compression
>>>and the package using the same compression algorithm will provide it:
>>>Suppose we want to compress a 8S-byte file, and both transparent compression
>>>and package is supported by 4-blocks clusters, S - block size.
>>>
>>>File system:
>>>divides the file into 2 clusters, suppose each cluster was compressed up to
>>>(2S+1) bytes, so compressed file occupies 6 blocks.
>>>Package:
>>>makes compressed file which occupies 2*(2S+1) = 4S+2 bytes and passes it
>>>to the file system, which places it (aieee!) into 5 blocks..
>>>
>>> So we do have the actual compression ratio produced by the file system is
>>>6/8 = 25% against actual ones (5/8 = 37,5%) produced by the package. The user
>>>can observe this effect only by using "du" (when he does "df", he see that
>>>file system and package produce the same compression ratio 37,5%).
>>>This is another effect caused by the specific of packing policy (for the
>>>ext2 files, reiserfs files presented by indirect items, etc..) Obviously
>>>we can avoid this by using bigger cluster (8 blocks).
>>>
>>>Edward.
>>>
>>>
>>>
>>>
>>>      
>>>
>>Or by not block aligning the clusters, which might be the right
>>solution,
>>    
>>
>
>It is impossible for the files presented by indirect items, since
>it means you'll place dead space in memory per file.
>
We don't use indirect items.  It is not impossible in any event, the 
problem is with being efficient in reads while also ensuring that we 
read whole compression atoms at a time.

> 
>  
>
>>and might allow for compression atoms larger than nodes.
>> Please consider that approach.
>>
>>Hans
>>    
>>
>
>
>  
>

Re: how do plugins work?

[Hans Reiser]

Gregor Zeitlinger wrote:

>Hi,
>
>I'm new to this list. I'm wondering how the file plugins work. Are they
>just a means of accessing the file contents or are they also designed to
>save the file differently.
>
>For example, could you create a plugin that automatically zips each
>text/plain file, which is totally transparent to the user?
>
Yes, and this is in our plan.

>If so, can those plugins be hirachical. Like: when I save a .tex file its
>automatically zipped and encrypted before writtten to disk.
>
Yes, planned.

>Or even better: When I save a file, its automatically zipped, encrypted
>and written over a ssh or ftp connection.
>
Yes, not planned but not rejected.;-)

>
>More precisely, I'm thinking about a plugin that stores xml files in an
>efficient manner. Somewhat like a btree. I'm not sure wheater you've
>already been through this issue, but I didn't find anything in the
>archives. The issue is somewhat difficult, because I'd love to be able to
>use xpath (possibly dom or xquery) to retreive/update xml data
>EFFICIENTY.
>
>I've gotten some ideas from a university project I'm working on. There
>we're trying to do a whole xml database management system. Although
>progress is terribly slow there are some good algos that might be
>reusable.
>
>  
>

reiserfs is said to be the most efficient way to store xml.