Implementing an attribute directory.

Implementing an attribute directory.

[Peter van Hardenberg]

We have an implementation plan for the attribute plugin. We plan to base it 
around the plugin.c so that it can be available for all files, directory or 
otherwise.

Every file which has a pseudo will gain a new user-attributes pseudofile. This 
pseudofile will essentially wrap a standard directory. In order to provide 
support for this directory, this plugin will include a new field on the inode 
which points to a standard dir_file.

When a new file is created, this inode will be created as well, and when a 
file is deleted, this one must be removed (with children) as well. Future 
versions should provide the optimization of only creating the inode once it 
becomes needed and otherwise simulating an empty directory.

Namesys folks, does this seem like a good course of action?

-pvh

-- 
Peter van Hardenberg (pvh@pvh.ca)
Victoria, BC, Canada

Re: Implementing an attribute directory.

[Hans Reiser]

Peter van Hardenberg wrote:

>We have an implementation plan for the attribute plugin. We plan to base it 
>around the plugin.c so that it can be available for all files, directory or 
>otherwise.
>
>Every file which has a pseudo will gain a new user-attributes pseudofile. This 
>pseudofile will essentially wrap a standard directory. 
>
Usage examples please.

>In order to provide 
>support for this directory, this plugin will include a new field on the inode 
>which points to a standard dir_file.
>
>When a new file is created, this inode will be created as well, and when a 
>file is deleted, this one must be removed (with children) as well. Future 
>versions should provide the optimization of only creating the inode once it 
>becomes needed and otherwise simulating an empty directory.
>
>Namesys folks, does this seem like a good course of action?
>
>-pvh
>
>  
>

Two usage examples for attribute directories.

[Peter van Hardenberg]

On November 20, 2005 11:47 pm, Hans Reiser wrote:
> Peter van Hardenberg wrote:
> >We have an implementation plan for the attribute plugin. We plan to base
> > it around the plugin.c so that it can be available for all files,
> > directory or otherwise.
> >
> >Every file which has a pseudo will gain a new user-attributes pseudofile.
> > This pseudofile will essentially wrap a standard directory.
>
> Usage examples please.
>
---------------------------------
Simple moves first:

$ echo "Matmos" >> track01.mp3/..@/artist
$ echo "A Chance to Cut is a Chance to Cure" >> track01.mp3/..@/album
$ grep Matmos track01.mp3/..@/*
artist: Matmos

---------------------------------
Some hypothetical pseudocode from a music player:

playcount = open('track01.mp3/..@/amarok/playcount').read;
playcount = playcount + 1;
open('track01.mp3/..@/amarok/playcount', 'w').write(playcount);

---------------------------------

-- 
Peter van Hardenberg (pvh@pvh.ca)
Victoria, BC, Canada

Re: Two usage examples for attribute directories.

[Peter Foldiak]

Peter van Hardenberg wrote:
> On November 20, 2005 11:47 pm, Hans Reiser wrote:
>   
>> Peter van Hardenberg wrote:
>>     
>>> We have an implementation plan for the attribute plugin. We plan to base
>>> it around the plugin.c so that it can be available for all files,
>>> directory or otherwise.
>>>
>>> Every file which has a pseudo will gain a new user-attributes pseudofile.
>>> This pseudofile will essentially wrap a standard directory.
>>>       
>> Usage examples please.
>>
>>     
> ---------------------------------
> Simple moves first:
>
> $ echo "Matmos" >> track01.mp3/..@/artist
> $ echo "A Chance to Cut is a Chance to Cure" >> track01.mp3/..@/album
> $ grep Matmos track01.mp3/..@/*
> artist: Matmos
>
> ---------------------------------
> Some hypothetical pseudocode from a music player:
>
> playcount = open('track01.mp3/..@/amarok/playcount').read;
> playcount = playcount + 1;
> open('track01.mp3/..@/amarok/playcount', 'w').write(playcount);
>
> ---------------------------------
>   

PvH,

I am really worried about introducing this new concept of "attribute 
directory" and new syntax. Maybe I am just missing the point (in which 
case please explain), but is there anything you can do with this that 
there is no way you could do without it.
Hans' favourite topic (quite rightly) is namespace unification, which 
involves keeping the syntax as simple as possible, having only one type 
of expression for one kind of logical structure. If you could possibly 
do this without the "..@", do it without it.

So why not just do (to take your example):

$ echo "Matmos" >> track01.mp3/artist


  Peter F

Re: Two usage examples for attribute directories.

[Peter van Hardenberg]

> PvH,
>
> I am really worried about introducing this new concept of "attribute
> directory" and new syntax. Maybe I am just missing the point (in which
> case please explain), but is there anything you can do with this that
> there is no way you could do without it.
> Hans' favourite topic (quite rightly) is namespace unification, which
> involves keeping the syntax as simple as possible, having only one type
> of expression for one kind of logical structure. If you could possibly
> do this without the "..@", do it without it.
>
> So why not just do (to take your example):
>
> $ echo "Matmos" >> track01.mp3/artist
>
>
>   Peter F

This is not new syntax, it is a new type of File (well, tecnically, it would 
be implemented as a new pseudoplugin.)

From a user's point of view, these files are equivalent to an XML attribute, 
whereas normal files are child elements. The semantic difference is, I 
contend, significant.

From an implementation point of view, this would allow us to provide guidance 
to, or eventually even full FS-level support for indexed attributes. 

These attribute directories, like "...." would also be available for 
directories, not just files. 

Also, important to me is that they could provide Reiser4 xattr compatibility.  
(The xattr('artist') for a file would be file/@/artist.)

Personally, I think "..@" is a keyboard-full. I would prefer "@" for improved 
consistency with xpath, but worrying about the actual string to access with 
is worrying about what colour the shed will be before you put it up.

I think resource directories in the style of OSX have a seperate value to them 
and also deserve implementation, but are another (related) project.

Dinner beckons,
-pvh


-- 
Peter van Hardenberg (pvh@pvh.ca)
Victoria, BC, Canada

Re: Two usage examples for attribute directories.

[Peter Foldiak]

Peter van Hardenberg wrote:
>> PvH,
>>
>> I am really worried about introducing this new concept of "attribute
>> directory" and new syntax. Maybe I am just missing the point (in which
>> case please explain), but is there anything you can do with this that
>> there is no way you could do without it.
>> Hans' favourite topic (quite rightly) is namespace unification, which
>> involves keeping the syntax as simple as possible, having only one type
>> of expression for one kind of logical structure. If you could possibly
>> do this without the "..@", do it without it.
>>
>> So why not just do (to take your example):
>>
>> $ echo "Matmos" >> track01.mp3/artist
>>
>>
>>   Peter F
>>     
>
> This is not new syntax, it is a new type of File (well, tecnically, it would 
> be implemented as a new pseudoplugin.)
>   
But why do we need a new type of file? What can you do with it that you 
absolutely couldn't without?
What is wrong with my simplification of your example? (track01.mp3/artist)
> From a user's point of view, these files are equivalent to an XML attribute, 
> whereas normal files are child elements.
I never quite understood the need for XML attributes either. You could 
easily survive without them, right?
>  The semantic difference is, I contend, significant.
>   
Could you explain this difference.

> From an implementation point of view, this would allow us to provide guidance 
> to, or eventually even full FS-level support for indexed attributes.
>   
Sorry for my ignorance, what is an "indexed attribute" and what's good 
about it.
Are you sure the simple "/" is not a more elegant and simple way to do 
all this.    Peter

Re: Two usage examples for attribute directories.

[Hans Reiser]

Peter Foldiak wrote:

> Peter van Hardenberg wrote:
>
>>> PvH,
>>>
>>> I am really worried about introducing this new concept of "attribute
>>> directory" and new syntax. Maybe I am just missing the point (in which
>>> case please explain), but is there anything you can do with this that
>>> there is no way you could do without it.
>>> Hans' favourite topic (quite rightly) is namespace unification, which
>>> involves keeping the syntax as simple as possible, having only one type
>>> of expression for one kind of logical structure. If you could possibly
>>> do this without the "..@", do it without it.
>>>
>>> So why not just do (to take your example):
>>>
>>> $ echo "Matmos" >> track01.mp3/artist
>>>
>>>
>>>   Peter F
>>>     
>>
>>
>> This is not new syntax, it is a new type of File (well, tecnically,
>> it would be implemented as a new pseudoplugin.)
>>   
>
> But why do we need a new type of file? What can you do with it that
> you absolutely couldn't without?
> What is wrong with my simplification of your example?
> (track01.mp3/artist)
>
>> From a user's point of view, these files are equivalent to an XML
>> attribute, whereas normal files are child elements.
>
> I never quite understood the need for XML attributes either. You could
> easily survive without them, right?
>
>>  The semantic difference is, I contend, significant.
>>   
>
> Could you explain this difference.
>
>> From an implementation point of view, this would allow us to provide
>> guidance to, or eventually even full FS-level support for indexed
>> attributes.
>>   
>
> Sorry for my ignorance, what is an "indexed attribute" and what's good
> about it.
> Are you sure the simple "/" is not a more elegant and simple way to do
> all this.    Peter
>
>
I agree with Peter's questions.

Re: Two usage examples for attribute directories.

[Alexander G. M. Smith]

Peter Foldiak wrote on Sat, 03 Dec 2005 10:54:31 +0000:
> But why do we need a new type of file? What can you do with it that you 
> absolutely couldn't without?

That's how I would do it, keep the namespace simpler.  The new file/directory
for attributes would only perhaps be useful for backwards compatibility with
operating systems that can't handle an item that's simultaneously both a
file and a directory.

As for mixing in the XML view, how about having it in a totally separate
directory tree?  Maybe a separate mount point.  Or since it's virtual,
have file/XML as the root of the XML tree that represents file and all its
contents and its nested children's contents.  Child files would also have a
child/XML tree just showing their contents and sub-children's representation,
which does duplicate part of the parent file/XML tree, but since it's virtual,
that's free.

>> From an implementation point of view, this would allow us to provide guidance 
>> to, or eventually even full FS-level support for indexed attributes.

For indexing, I'd add a file type byte or two to every file and use that to
help decide (along with file name) what things should be indexed or not.
BeOS decided based on the attribute name, and required that attributes with
that name all have the same datatype (otherwise adding to the index wouldn't
work too well - keys all have to be the same type for sorting reasons).

- Alex

Re: Two usage examples for attribute directories.

[Peter van Hardenberg]

Hi Peter F, Hans, and all,

we had a great group meeting today and I think managed to come up with some 
reasonable explanations. I will break the threading a bit to answer the 
questions coherently, but I believe this answers all questions asked, so if 
I've missed any, please let me know.

Let me begin by assuming that attributes are not in any way different from 
normal files and show how that leads to some limitations. I will imagine a 
more detailed use case.

Bundled apps. I would love to be able to distribute applications as 
single-file bundles with all their resources in them, like this:

----------------------

$ # look at the resource files contained in app/
$ ls app/
bin
lib
src
var
Makefile
icon.png
install.sh

$ # run app
$ ./app
This version of app is not built for your platform.
Please run "cd app/; ./configure; make;" to update the build for your 
platform.

$ # okay, we'll recompile app for our system
$ cd app/; ./configure; make;
Compiling app...
Done.

$ # let's see if it works
$ ./app
Starting app...

$ # yep, looks good! let's install it.
$ sudo app/install.sh --prefix=/usr/local
Installing app into /usr/local...

# Note that app is now installed so the ./ is no longer needed.
$ app 
Starting app...

$ # done.

----------------------

That demonstrated using file-directories to contain resources. Any file-dir 
application would add itself to its own path on startup. This would be 
supremely useful for projects like Klik, and implements (I would say) ideally 
the things that OS X is currently emulating with their "prog.app" 
directories. 

Still, say we'd like to be able to also specify some attributes for app, let's 
say "author", "version", "name", "description", "contact", "category" -- 
things that would be useful for creating a menu entry.

Now the directory begins to get ugly...

$ ls app/
author
bin
category
contact
description
lib
name
src
var
version
Makefile
icon.png
install.sh

But which are attributes? If only our namespace provided us a way to seperate 
one group of files from another... Aha! Why not a directory?

$ ls app/@/
author
category
contact
description
name
version
$ ls app/
bin
lib
src
var
Makefile
icon.png
install.sh

Ahh, much better. It also ensures we can preserve Hans' goal of not enforcing 
unnatural structure.  What if I want to find all the files Hans is in any way 
associated with, whether it is as "author", "contact", "inspiration", 
whatever.

I don't want to look into EVERY file on the disk if I don't have to. Instead, 
I can just 

$ grep "Hans Reiser" */@/* 

without having to grep through gig upon gig of other data. Of course, if 
that's what I want to do, the option is still there...

SO! Why privilege the @ directory with its own plugin? Why not just use it as 
a convention?

Efficiency. Files governed by the @ plug-in could have size restrictions, and 
should be packed tightly with their parents. They might have special format 
to fit more into a single disk read. They could be indexed by the operating 
system to make searching for particular values fast.

Control. They might have strict data types or controlled grammars. The @ 
directory might not actually be displayed in a standard "ls" of the directory 
(like "...." isn't). 

Compatibility. The @ directory plugin should definitely have compatibility 
with xattr syntax so that existing software would be able to use Reiser 
attributes "out of the box" without any source changes. That would be a big 
win for driving adoption.

Thus, I conclude that while file-directories are valuable and closely related 
to attribute directories, but the two serve different roles and have 
different design goals, so should be served by different plugins.

On December 3, 2005 02:54 am, Peter Foldiak wrote:
> Are you sure the simple "/" is not a more elegant and simple way to do
> all this.    Peter

No, I'm not sure, but I'm hoping we'll find one by talking about it.

-- 
Peter van Hardenberg (pvh@pvh.ca)
Victoria, BC, Canada

Re: Two usage examples for attribute directories.

[Peter Foldiak]

Peter van Hardenberg wrote:
> Still, say we'd like to be able to also specify some attributes for app, let's 
> say "author", "version", "name", "description", "contact", "category" -- 
> things that would be useful for creating a menu entry.
>
> Now the directory begins to get ugly...
>
> $ ls app/
> author
> bin
> category
> contact
> description
> lib
> name
> src
> var
> version
> Makefile
> icon.png
> install.sh
>
> But which are attributes? If only our namespace provided us a way to seperate 
> one group of files from another... Aha! Why not a directory?
>   

You say "author" should be an attribute but "icon.png" is not an 
attribute. I am not convinced they should be treated differently.
The listing doesn't look ugly to me at all (just because it is long).

> Ahh, much better. It also ensures we can preserve Hans' goal of not enforcing 
> unnatural structure.  What if I want to find all the files Hans is in any way 
> associated with, whether it is as "author", "contact", "inspiration", 
> whatever.
>
> I don't want to look into EVERY file on the disk if I don't have to. Instead, 
> I can just 
>
> $ grep "Hans Reiser" */@/* 
>   

I don't think this is a nice solution.
The nice solution is what is suggested in Hans' whitepaper:

[HansReiser]

which would give you all objects associated with Hans, and this would be 
efficient (unlike */@/*).
(Read
http://www.namesys.com/whitepaper.html
again.)

> without having to grep through gig upon gig of other data. Of course, if 
> that's what I want to do, the option is still there...
>
> SO! Why privilege the @ directory with its own plugin?

Quite. You shouldn't.

> Why not just use it as 
> a convention?
>
> Efficiency. Files governed by the @ plug-in could have size restrictions,

Arbitrary limits are disliked in Linux, with good reason.

> should be packed tightly with their parents. They might have special format 
> to fit more into a single disk read. They could be indexed by the operating 
> system to make searching for particular values fast.
>   
An efficient implementation of "[ ]" would do the indexing, or would 
have the same effect.

> Thus, I conclude that while file-directories are valuable and closely related 
> to attribute directories, but the two serve different roles and have 
> different design goals, so should be served by different plugins.
>   

Where does the term "attribute directory" come from? Did you invent it?
Are they your "@" directories, or is that referring to something more?

Thanks, Peter

Re: Two usage examples for attribute directories.

[Hubert Chan]

On Thu, 08 Dec 2005 14:12:46 +0000, Peter Foldiak said:

[...]

> You say "author" should be an attribute but "icon.png" is not an 
> attribute. I am not convinced they should be treated differently.  The
> listing doesn't look ugly to me at all (just because it is long).

IMHO, putting attributes in foo/ instead of foo/@/ increases the risk of
collisions.  e.g. let's say I have a directory that contains a file
named icon.png, but I want my file manager to display its icon as some
other file.  If all the attributes are in foo/, then the file manager
will read foo/icon.png as the directory's icon, which is not what I
want.  It would be better if I could have a directory that contains
icon.png, and uses foo/@/icon.png as its icon in the file manager.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Two usage examples for attribute directories.

[Nate Diller]

[-- Attachment #1: Type: text/plain, Size: 1253 bytes --]

On 12/8/05, Hubert Chan <hubert@uhoreg.ca> wrote:
>
> On Thu, 08 Dec 2005 14:12:46 +0000, Peter Foldiak said:
>
> [...]
>
> > You say "author" should be an attribute but "icon.png" is not an
> > attribute. I am not convinced they should be treated differently.  The
> > listing doesn't look ugly to me at all (just because it is long).
>
> IMHO, putting attributes in foo/ instead of foo/@/ increases the risk of
> collisions.  e.g. let's say I have a directory that contains a file
> named icon.png, but I want my file manager to display its icon as some
> other file.  If all the attributes are in foo/, then the file manager
> will read foo/icon.png as the directory's icon, which is not what I
> want.  It would be better if I could have a directory that contains
> icon.png, and uses foo/@/icon.png as its icon in the file manager.
>

This problem is already solved.

$ ls /home/nate
icon.png

$ ls -a /home/nate
.
..
.bash_rc
icon.png

so...

$ls -aa /home/nate
.
..
..icon.png
.bash_rc
icon.png

this works for PvH's example too:
$ ls -aa app/
.
..
..author
..category
..contact
..description
..name
..version
.config
.default_options
bin
lib
src
var
Makefile
icon.png
install.sh

NATE

[-- Attachment #2: Type: text/html, Size: 1755 bytes --]

Re: Two usage examples for attribute directories.

[Hans Reiser]

Nate Diller wrote:

> On 12/8/05, *Hubert Chan* <hubert@uhoreg.ca <mailto:hubert@uhoreg.ca>>
> wrote:
>
>     On Thu, 08 Dec 2005 14:12:46 +0000, Peter Foldiak said:
>
>     [...]
>
>     > You say "author" should be an attribute but "icon.png" is not an
>     > attribute. I am not convinced they should be treated
>     differently.  The
>     > listing doesn't look ugly to me at all (just because it is long).
>
>     IMHO, putting attributes in foo/ instead of foo/@/ increases the
>     risk of
>     collisions.  e.g. let's say I have a directory that contains a file
>     named icon.png, but I want my file manager to display its icon as some
>     other file.  If all the attributes are in foo/, then the file manager
>     will read foo/icon.png as the directory's icon, which is not what I
>     want.  It would be better if I could have a directory that contains
>     icon.png, and uses foo/@/icon.png as its icon in the file manager.
>
>
> This problem is already solved.
>
> $ ls /home/nate
> icon.png
>
> $ ls -a /home/nate
> .
> ..
> .bash_rc
> icon.png
>
> so...
>
> $ls -aa /home/nate
> .
> ..
> ..icon.png
> .bash_rc
> icon.png
>
> this works for PvH's example too:
> $ ls -aa app/
> .
> ..
> ..author
> ..category
> ..contact
> ..description
> ..name
> ..version
> .config
> .default_options
> bin
> lib
> src
> var
> Makefile
> icon.png
> install.sh
>
> NATE

yup

Re: Two usage examples for attribute directories.

[Hubert Chan]

On Fri, 9 Dec 2005 17:11:55 -0800, Nate Diller <nate.diller@gmail.com> said:

> This problem is already solved.

[...]

> so...

> $ls -aa /home/nate
> .
> ..
> ..icon.png
> .bash_rc
> icon.png

Dotfiles are horrible, IMHO, but that's just me.  I would prefer
foo/@/icon.png over foo/..icon.png.  But yes, that does solve the
collision problem.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Two usage examples for attribute directories.

[Hans Reiser]

Peter Foldiak wrote:

Thanks for helping with this Peter Foldiak.