Re: Carrying Attributes too Far

Re: Carrying Attributes too Far

[lrc1]

Sorry for the delay in replying.

Quoting Hubert Chan <hubert@uhoreg.ca>:

> >>>>> "Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:
> 
> [...]
> 
> >> Why?  Not at all, I would say.
> 
> Leo> Short answer: because otherwise you have ""attributes" that cannot
> Leo> interact with files in all the same ways that files can interact
> Leo> with files" ( http://www.namesys.com/v4/v4.html ).
> 
> Hard linking isn't a universal attribute, though.  For example, you
> cannot hard link over different filesystems.  If an attribute (such as
> uid, premissions, etc.) are implemented by just taking the file's
> standard stat data and exposing it as a file interface, that it is
> similar to that data being from a different filesystem, and so you
> can't hardlink.  This would be similar, I suppose, to if a plugin
> exposes an MP3's id3 data as files -- it would make no sense to be able
> to hardlink that, since the data is stored in the MP3 file, and not as
> part of the filesystem.
>

Yes, it is impossible to hard-link between two files on different volumes
(except at mount points) in the Unix filesystem, but it shouldn't be. (More
generally, with the necessary permissions it should be possible to make any file
the child of any directory via a hard link, except where doing so would create a
cycle.) There's no semantic reason why it shouldn't be possible; in other words,
if it's meaningful for a file to be named both /pub/pictures/sunrise and
/home/alice/pictures/daylight , why would it in fact not be meaningful just
because a volume is mounted at /home ? Hard-linking across mount subtrees is not
possible in existing Unix filesystems for non-semantic reasons, the most
interesting of which is that it doesn't fit with the Unix concept of mount
subtrees and mount points. The correct solution to this is to throw out that
concept, something I'll post about later.

Note that hard-linking between files on different volumes is a necessity if you
want to do the kind of things discussed in Future Vision cleanly. If I did a
simple search for all mp3s at least 3 minutes long, I would expect to get back a
directory which had as its children all the 3-min.+ mp3s visible to me on the
computer. I wouldn't necessarily care if some of the mp3s were on a different
mount subtree to the search directory, and I wouldn't expect to find things like
symlinks being used to duct-tape the mp3s onto the directory.

(As to your id3 example: Yes, the id3 file data is stored in the mp3 file, but
then the /etc/motd file data is stored in the file /dev/hda1 or whatever.
There's no reason why it makes any less sense be able to hard-link to the id3
file than to /etc/motd. Why shouldn't I be able to use a search to get a
directory linked to all the id3 files on the system?)

But for the sake of argument, let's assume that indeed it should remain
impossible to hard-link across mount subtrees, or at least that it will. Will
this actually prevent the problems of ordinary directories hard-linking to
attribute files not their own? Consider a case in which /pub/somedir and
/home/alice/hello.mp3 are both on the same volume, say /dev/hda1 . If even one
of hello.mp3's attribute files (or to be more precise, one of the files which
inherit metadata from hello.mp3) is on the same mount subtree as somedir , then
the restriction on hard-linking between files on different volumes is not
sufficient to prevent the problems. So can we find an attribute file of
hello.mp3 which might actually be on /dev/hda1 ? An id3 file derived
automatically from hello.mp3 should be quite safe; its device file is hello.mp3
, just as hello.mp3 's device file is /dev/hda1 , so it should be in a different
mount subtree to hello.mp3 . (But incidentally, this raises another question: if
one of hello.mp3 's descendants is a mount point, what happens when alice types

    mv hello.mp3 hi.mp3

?) By contrast, imagine that alice writes a short text file of information about
hello.mp3. Clearly this is hello.mp3 metadata, so she saves the file in
hello.mp3/+/info . info isn't automatically derived from mp3, so why wouldn't it
just be another file on /dev/hda1 like hello.mp3 and somedir ? As to
permissions, wasn't it the intention to use permissions files as a *replacement*
for stat and friends, not just as a means of exposing stat through a file
interface? If hello.mp3/+/permissions replaces stat and friends, then it's in
the same position as hello.mp3/+/info . If it simply exposes then, then subfile
metadata is a lot less interesting - for one thing, it doesn't make implementing
different permissions systems any easier (the opposite, in fact).

But even if ordinary directories can't hard-link to other files' attribute
files, the problem of multiple inheritance of metadata still isn't fully solved.
What if two children of hello.mp3/+/ have different permissions, and a third
file is the child of both of them? And what about the proposal that ordinary,
non-attribute files should inherit metadata from their parent directories? 
 
<snip />

> Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
<snip />

Leo.


-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

lrc1@st-andrews.ac.uk wrote on Sat,  4 Oct 2003 06:58:04 +0100:
> Yes, it is impossible to hard-link between two files on different volumes
> (except at mount points) in the Unix filesystem, but it shouldn't be. (More
> generally, with the necessary permissions it should be possible to make any
> file the child of any directory via a hard link, except where doing so would
> create a cycle.)

Or if the file system supports parent links for all objects, and does a bit
of graph traversal when deleting files, then you can have cycles.  It is
useful and more natural to organize information in a graph rather than a
hierarchy.  It's possible to do it with symbolic links, but hard links
make it more reliable (moving files around doesn't destroy the links).

Hard links across volumes or removable media isn't possible since you
don't have real time notification of changes; so the links would be
slightly-soft, squishy, firm, or some other such technical term :-).
I guess you could make such a firm link evaluation block the caller
until the system has gotten a lock on the target (insert removable
disk, establish network connection, etc).

> There's no semantic reason why it shouldn't be possible; in other words,
> if it's meaningful for a file to be named both /pub/pictures/sunrise and
> /home/alice/pictures/daylight , why would it in fact not be meaningful just
> because a volume is mounted at /home ?

I'd ditch the cool but mostly useless and confusing feature of having
different names for the same file.  Use symbolic links for that.  Having
a single name makes the implementation easier too (store the name in the
file's inode rather than in the directories).  This would slow down
directory traversal, but "ls" and other tools already stat() each file to
read its inode metadata anyway, so putting the name there wouldn't be
too bad.

> What if two children of hello.mp3/+/ have different permissions, and a third
> file is the child of both of them? And what about the proposal that ordinary,
> non-attribute files should inherit metadata from their parent directories? 

Good point.  I guess the inheritance algorithm should take multiple parents
into account when doing its traversal.  Or just restrict the use of
inheritance.

- Alex

Re: Carrying Attributes too Far

[Hubert Chan]

[...]

Leo> There's no semantic reason why it shouldn't be possible; in other
Leo> words, if it's meaningful for a file to be named both
Leo> /pub/pictures/sunrise and /home/alice/pictures/daylight , why would
Leo> it in fact not be meaningful just because a volume is mounted at
Leo> /home ?

I agree with Alexander that such a situation is better handled with
symlinks.  Or some type of firm link.  In general, it's more useful for
a file to have only a single filename -- especially from a user's
perspective.  I've only ever seen approximately one good use of hard
links.  But hopefully Reiser6 (or whichever version of ReiserFS has the
query system) will solve the situation that you described above.

[...]

Leo> Note that hard-linking between files on different volumes is a
Leo> necessity if you want to do the kind of things discussed in Future
Leo> Vision cleanly. If I did a simple search for all mp3s at least 3
Leo> minutes long, I would expect to get back a directory which had as
Leo> its children all the 3-min.+ mp3s visible to me on the computer. I
Leo> wouldn't necessarily care if some of the mp3s were on a different
Leo> mount subtree to the search directory, and I wouldn't expect to
Leo> find things like symlinks being used to duct-tape the mp3s onto the
Leo> directory.

I fail to see how hard links would help this at all.  If you have
multiple volumes, the way to do things would be to have the system pass
the query on to all the (relevant) filesystems.  Or to make Reiser* a
distributed filesystem, so that all the volumes can look like a single
filesystem.

Leo> (As to your id3 example: Yes, the id3 file data is stored in the
Leo> mp3 file, but then the /etc/motd file data is stored in the file
Leo> /dev/hda1 or whatever.  There's no reason why it makes any less
Leo> sense be able to hard-link to the id3 file than to /etc/motd. Why
Leo> shouldn't I be able to use a search to get a directory linked to
Leo> all the id3 files on the system?)

The point was that the id3 data and the regular files are conceptually
on different filesystems.  So hard linking to id3 data is similar to
hard linking across filesystems.

Additionally, I doubt that hard linking within a single file's id3 data
would be possible, since it is not supported by the underlying file
structure.  Just like I don't think hard linking on FAT or NTFS
filesystems is supported.  I don't think we're going to lose much by
not allowing hard links everywhere.

Leo> But for the sake of argument, let's assume that indeed it should
Leo> remain impossible to hard-link across mount subtrees, or at least
Leo> that it will. Will this actually prevent the problems of ordinary
Leo> directories hard-linking to attribute files not their own? Consider
Leo> a case in which /pub/somedir and /home/alice/hello.mp3 are both on
Leo> the same volume, say /dev/hda1 . If even one of hello.mp3's
Leo> attribute files (or to be more precise, one of the files which
Leo> inherit metadata from hello.mp3) is on the same mount subtree as
Leo> somedir , then the restriction on hard-linking between files on
Leo> different volumes is not sufficient to prevent the problems.

Which problems are you talking about?

Leo> So can we find an attribute file of hello.mp3 which might actually
Leo> be on /dev/hda1 ?

Yes, if the "attribute file" is a normal file.

One thing to remember is that Reiser4 does not store attributes per
se.  It stores regular files, which can be treated as attributes.  Thus
when they say that a file is both a regular file and a directory (with
the appropriate plugin), they literally mean that it is a normal
directory.

[...]

Leo> (But incidentally, this raises another question: if one of
Leo> hello.mp3 's descendants is a mount point, what happens when alice
Leo> types

Leo>     mv hello.mp3 hi.mp3

Leo> ?)

I assume the same thing that would happen if you tried to "mv foo bar",
where foo is a normal directory, and one of foo's descendants is a
mount point.  I have no idea what would happen, since I've never tried
it, but Reiser4 doesn't introduce any new problems in this regard.

Leo> By contrast, imagine that alice writes a short text file of
Leo> information about hello.mp3. Clearly this is hello.mp3 metadata, so
Leo> she saves the file in hello.mp3/+/info . info isn't automatically
Leo> derived from mp3, so why wouldn't it just be another file on
Leo> /dev/hda1 like hello.mp3 and somedir ?

That is correct.  It will be a regular file.  (Unless the id3 plugin
decides to grab it and turn it into an id3 tag.)

Leo> As to permissions, wasn't it the intention to use permissions files
Leo> as a *replacement* for stat and friends, not just as a means of
Leo> exposing stat through a file interface?

No.  If you do that, you run into lots of problems.  All files need
stat data, and so if foo.txt/+/owner was a normal file, then you would
also need foo.txt/+/owner/+/owner, foo.txt/+/owner/+/owner/+/owner, ad
infinitum.  As well, some stat data just doesn't make sense to make to
have as a normal file, such as inode number, device type, size, etc.

Leo> If hello.mp3/+/permissions replaces stat and friends, then it's in
Leo> the same position as hello.mp3/+/info . If it simply exposes then,
Leo> then subfile metadata is a lot less interesting - for one thing, it
Leo> doesn't make implementing different permissions systems any easier
Leo> (the opposite, in fact).

How would having stat data as a normal file make permission systems any
easier?  And how does just exposing the stat data make permission
systems harder?

Leo> But even if ordinary directories can't hard-link to other files'
Leo> attribute files, the problem of multiple inheritance of metadata
Leo> still isn't fully solved.  What if two children of hello.mp3/+/
Leo> have different permissions, and a third file is the child of both
Leo> of them? And what about the proposal that ordinary, non-attribute
Leo> files should inherit metadata from their parent directories?

I have never heard of any such proposal.  Where is that from?

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Carrying Attributes too Far

[Hans Reiser]

Oops, found this in my drafts box after a long time had passed.....

Hans

lrc1@st-andrews.ac.uk wrote:

>Sorry for the delay in replying.
>
>Quoting Hubert Chan <hubert@uhoreg.ca>:
>
>  
>
>>>>>>>"Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:
>>>>>>>              
>>>>>>>
>>[...]
>>
>>    
>>
>>>>Why?  Not at all, I would say.
>>>>        
>>>>
>>Leo> Short answer: because otherwise you have ""attributes" that cannot
>>Leo> interact with files in all the same ways that files can interact
>>Leo> with files" ( http://www.namesys.com/v4/v4.html ).
>>
>>Hard linking isn't a universal attribute, though.  For example, you
>>cannot hard link over different filesystems.  If an attribute (such as
>>uid, premissions, etc.) are implemented by just taking the file's
>>standard stat data and exposing it as a file interface, that it is
>>similar to that data being from a different filesystem, and so you
>>can't hardlink.  This would be similar, I suppose, to if a plugin
>>exposes an MP3's id3 data as files -- it would make no sense to be able
>>to hardlink that, since the data is stored in the MP3 file, and not as
>>part of the filesystem.
>>
>>    
>>
>
>Yes, it is impossible to hard-link between two files on different volumes
>(except at mount points) in the Unix filesystem, but it shouldn't be. (More
>generally, with the necessary permissions it should be possible to make any file
>the child of any directory via a hard link, except where doing so would create a
>cycle.) 
>
This implies that two filesystems can share an identification number
between them.

>There's no semantic reason why it shouldn't be possible; in other words,
>if it's meaningful for a file to be named both /pub/pictures/sunrise and
>/home/alice/pictures/daylight , why would it in fact not be meaningful just
>because a volume is mounted at /home ? Hard-linking across mount subtrees is not
>possible in existing Unix filesystems for non-semantic reasons, the most
>interesting of which is that it doesn't fit with the Unix concept of mount
>subtrees and mount points. The correct solution to this is to throw out that
>concept, something I'll post about later.
>
>Note that hard-linking between files on different volumes is a necessity if you
>want to do the kind of things discussed in Future Vision cleanly. If I did a
>simple search for all mp3s at least 3 minutes long, I would expect to get back a
>directory which had as its children all the 3-min.+ mp3s visible to me on the
>computer. I wouldn't necessarily care if some of the mp3s were on a different
>mount subtree to the search directory, and I wouldn't expect to find things like
>symlinks being used to duct-tape the mp3s onto the directory.
>
>(As to your id3 example: Yes, the id3 file data is stored in the mp3 file, but
>then the /etc/motd file data is stored in the file /dev/hda1 or whatever.
>There's no reason why it makes any less sense be able to hard-link to the id3
>file than to /etc/motd. Why shouldn't I be able to use a search to get a
>directory linked to all the id3 files on the system?)
>
>But for the sake of argument, let's assume that indeed it should remain
>impossible to hard-link across mount subtrees, or at least that it will. Will
>this actually prevent the problems of ordinary directories hard-linking to
>attribute files not their own? Consider a case in which /pub/somedir and
>/home/alice/hello.mp3 are both on the same volume, say /dev/hda1 . If even one
>of hello.mp3's attribute files (or to be more precise, one of the files which
>inherit metadata from hello.mp3) is on the same mount subtree as somedir , then
>the restriction on hard-linking between files on different volumes is not
>sufficient to prevent the problems. So can we find an attribute file of
>hello.mp3 which might actually be on /dev/hda1 ? An id3 file derived
>automatically from hello.mp3 should be quite safe; its device file is hello.mp3
>, just as hello.mp3 's device file is /dev/hda1 , so it should be in a different
>mount subtree to hello.mp3 . (But incidentally, this raises another question: if
>one of hello.mp3 's descendants is a mount point, what happens when alice types
>
>    mv hello.mp3 hi.mp3
>
>?) By contrast, imagine that alice writes a short text file of information about
>hello.mp3. Clearly this is hello.mp3 metadata, so she saves the file in
>hello.mp3/+/info . info isn't automatically derived from mp3, so why wouldn't it
>just be another file on /dev/hda1 like hello.mp3 and somedir ? As to
>permissions, wasn't it the intention to use permissions files as a *replacement*
>for stat and friends, not just as a means of exposing stat through a file
>interface? If hello.mp3/+/permissions replaces stat and friends, then it's in
>the same position as hello.mp3/+/info . If it simply exposes then, then subfile
>metadata is a lot less interesting - for one thing, it doesn't make implementing
>different permissions systems any easier (the opposite, in fact).
>
>But even if ordinary directories can't hard-link to other files' attribute
>files, the problem of multiple inheritance of metadata still isn't fully solved.
>What if two children of hello.mp3/+/ have different permissions, and a third
>file is the child of both of them? And what about the proposal that ordinary,
>non-attribute files should inherit metadata from their parent directories? 
> 
><snip />
>
>  
>
>>Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
>>    
>>
><snip />
>
>Leo.
>
>
>-----------------------------------------------------------------
>University of St Andrews Webmail: http://webmail.st-andrews.ac.uk
>
>
>  
>


-- 
Hans

Re: Carrying Attributes too Far

[lrc1]

Quoting Hans Reiser <reiser@namesys.com>:

...

> lrc1@st-andrews.ac.uk wrote:
> 
> >Sorry for the delay in replying.
> >
> >Quoting Hubert Chan <hubert@uhoreg.ca>:

...

> >Yes, it is impossible to hard-link between two files on different volumes
> >(except at mount points) in the Unix filesystem, but it shouldn't be. (More
> >generally, with the necessary permissions it should be possible to make any
> file
> >the child of any directory via a hard link, except where doing so would
> create a
> >cycle.) 
> >
> This implies that two filesystems can share an identification number
> between them.

I would suggest that for every link to a file on a different filesystem, the
linking directory should store both an identifier for the child file's
filesystem and the child file's on-disk inumber on that filesystem.* The reverse
links would be stored in the same way. That way, filesystems don't have to
understand other filesystems' identifier schemes, or coordinate their own
schemes with them. (The obvious alternative approach - maintaining a list of
unique identifiers for all files on all mounted filesystems - is undesirable for
a number of reasons. For one thing, it is limiting and unsafe to assume that all
mounted filesystems are willing and able to provide an accurate list of all
their files in a reasonable time.)

This leaves one significant problem: what happens when a file is moved from one
filesystem to another? What will prevent the cross-filesystem hard links to the
file from breaking? Cleaning up cross-filesystem links after explicit file
deletion is a related but simpler problem. I think a good but not ideal solution
to this problem is possible. I'll go into detail if anyone is interested.

* Actually filesystems should hand out file identifiers for external linking
which may or may not be the same as their on-disk inumbers.

...

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

A bold idea (Re: Carrying Attributes too Far)

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

lrc1@st-andrews.ac.uk wrote:

 >Quoting Hans Reiser <reiser@namesys.com>:
 >
 >...
 >
 >
 >
 >>lrc1@st-andrews.ac.uk wrote:
 >>
 >>
 >>
 >>>Sorry for the delay in replying.
 >>>
 >>>Quoting Hubert Chan <hubert@uhoreg.ca>:
 >>>
 >>>
 >
 >...
 >
 >
 >
 >>>Yes, it is impossible to hard-link between two files on different 
volumes
 >>>(except at mount points) in the Unix filesystem, but it shouldn't 
be. (More
 >>>generally, with the necessary permissions it should be possible to 
make any
 >>>
 >>>
 >>file
 >>
 >>
 >>>the child of any directory via a hard link, except where doing so would
 >>>
 >>>
 >>create a
 >>
 >>
 >>>cycle.)
 >>>
 >>>
 >>>
 >>This implies that two filesystems can share an identification number
 >>between them.
 >>
 >>
 >
 >I would suggest that for every link to a file on a different 
filesystem, the
 >linking directory should store both an identifier for the child file's
 >filesystem and the child file's on-disk inumber on that filesystem.* 
The reverse
 >
 >
 >
Here's a crazy idea -- maybe performance can be improved to the point
where it's actually more efficient to have one large partition to begin
with?  Back to hda1 is /boot, hda2 is /, and hda3 is swap?

That gets rid of the hardlink problem, and also the problem of making
decisions about partitioning.  Other features of Reiser4 like plugins
eliminate any need that might arise from things like Cryptoloop.  In
fact, maybe other filesystems will be implemented as Reiser4 plugins to
a .img file, but that's probably going a bit far.

I agree that hardlinking directories should be allowed, and I think it's
worth it to check whether it creates a loop, unless rm can be modified
to simply rm the directory, and the filesystem (and maybe VFS) would be
modified to have directories have link counts that don't include . or
.., so when a directory is to be deleted, the system does its own manual
deletion of each of the files within it.  Then, if I type 'rm loop' and
it removes the link of 'loop/loop' to 'loop', there are now no more
links to 'loop' and it gets destroyed.

I'm a teenager, I have the right to try to be revolutionary!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9ACcgisZLIF6uqOAQLw8w//YrJkVSGlur5XUL92Aem65aOz6lP41ATm
pmiakJMfL3Qbd1/UekO+UsdwJl7Sz8EvA0eqGpPeaOlgEa7TfvkAhGNvsGlKmQEj
89goTZltrd4FM3emZUOgabLg8RQLEbKlq58QAz4GoSTClaOz7P2FaCrNr6fXgZ5U
DSVBqO7I/YYCdHUvQtKUipahM+KR/BZbM7CnFkCZZtsRGGzGGEl0kphRC3+tYWv0
ELcuA9MB1B8ASdpGREgU9T27YMUh7N2hrU/rQwb1WRanvO8Kb8rES4SyqyqNdPqG
6onWyu2a6PwVWH5WA0yn58DE7g05X1aOkYr3wsfDmBxuJeSECHJugJ39vM78oXaB
tc/uwYn2ZDak/Hijwltj7hJiLmaR13M6NUGAY8Q5UXgiW4lXTnXEFl68CdG+DbZz
IGREme+ftlozyezyipqVEoJSdGFxSYO9DL8AtV+KKET3OEHVw+fR2ATLrDH1QPCy
OiHYh6BE/339o2bIfiCO9TuIWSyo4nSr11M/9xwC+FCbxxm17DqxUNe5+WOYNT1j
Hb1k+2ESM3o9tbrmquCs8ohA0Unr9L7lZiX6e6L4FJe1JLyRQznMq2Gks39jN5JI
gkeG2zWyXgBUjZ5WJ3p734cHetq139qc/narnii+MLpwk0OnjyLOe/UzH1xnwZCl
N2CHJZje340=
=pVUD
-----END PGP SIGNATURE-----

Re: A bold idea (Re: Carrying Attributes too Far)

[Heinz-Josef Claes]

Am Fr, 2003-12-05 um 04.58 schrieb David Masover:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> lrc1@st-andrews.ac.uk wrote:
> 
>  >Quoting Hans Reiser <reiser@namesys.com>:
>  >
>  >...
>  >
>  >
>  >
>  >>lrc1@st-andrews.ac.uk wrote:
>  >>
>  >>
>  >>
>  >>>Sorry for the delay in replying.
>  >>>
>  >>>Quoting Hubert Chan <hubert@uhoreg.ca>:
>  >>>
>  >>>
>  >
>  >...
>  >
>  >
>  >
>  >>>Yes, it is impossible to hard-link between two files on different 
> volumes
>  >>>(except at mount points) in the Unix filesystem, but it shouldn't 
> be. (More
>  >>>generally, with the necessary permissions it should be possible to 
> make any
>  >>>
>  >>>
>  >>file
>  >>
>  >>
>  >>>the child of any directory via a hard link, except where doing so would
>  >>>
>  >>>
>  >>create a
>  >>
>  >>
>  >>>cycle.)
>  >>>
>  >>>
>  >>>
>  >>This implies that two filesystems can share an identification number
>  >>between them.
>  >>
>  >>
>  >
>  >I would suggest that for every link to a file on a different 
> filesystem, the
>  >linking directory should store both an identifier for the child file's
>  >filesystem and the child file's on-disk inumber on that filesystem.* 
> The reverse
>  >
>  >
>  >
> Here's a crazy idea -- maybe performance can be improved to the point
> where it's actually more efficient to have one large partition to begin
> with?  Back to hda1 is /boot, hda2 is /, and hda3 is swap?
> 
> That gets rid of the hardlink problem, and also the problem of making
> decisions about partitioning.  Other features of Reiser4 like plugins
> eliminate any need that might arise from things like Cryptoloop.  In
> fact, maybe other filesystems will be implemented as Reiser4 plugins to
> a .img file, but that's probably going a bit far.
> 
If you want to have a flexible size for a filesystem, the are
possiblities with LVM (or others) which gives you also the possibility
to create a raid - which you normally want on such filesystems.
Hardlinks between different partitions gives another problem: What is,
if the other partition is not available (unmounted or crashed)? What is,
if the other partition is an NFS-mounted partition with all the
possiblities that can happen?
 
> I agree that hardlinking directories should be allowed, and I think it's
> worth it to check whether it creates a loop, unless rm can be modified
I think hard linking directories is a very good idea. The should be a
count for the number of hard links and it should be handled like with
normal files. I directly have an application with would have a great
benefit of this kind of hard linked directories, but up to now they are
not available in linux :-(


> to simply rm the directory, and the filesystem (and maybe VFS) would be
> modified to have directories have link counts that don't include . or
> .., so when a directory is to be deleted, the system does its own manual
> deletion of each of the files within it.  Then, if I type 'rm loop' and
> it removes the link of 'loop/loop' to 'loop', there are now no more
> links to 'loop' and it gets destroyed.
> 

Heinz-Josef Claes                     hjclaes@web.de
project: http://sourceforge.net/projects/storebackup
      -> snapshot-like backup to another disk

Re: A bold idea (Re: Carrying Attributes too Far)

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>If you want to have a flexible size for a filesystem, the are
>possiblities with LVM (or others) which gives you also the possibility
>to create a raid - which you normally want on such filesystems.
>Hardlinks between different partitions gives another problem: What is,
>if the other partition is not available (unmounted or crashed)? What is,
>if the other partition is an NFS-mounted partition with all the
>possiblities that can happen?
> 
>  
>
That's why I don't think there should be hardlinks across devices.  You 
can put a filesystem on a RAID, of course.  LVM is not really a 
solution, for several reasons.  For example, it seems as though it's 
actually possible to have an entire partition fragmented in the same way 
that a single file can be, and there's no way to fix that.

It also requires the user to change things.

Now, why would I use separate filesystems in the first place?  Two 
reasons:  performance and security.  Right now, it's a lot easier to 
have /var be a certain sized partition rather than trying to enforce 
disk quotas.  It also improves performance if we know that quickly 
moving data like that in /tmp is separated from slowly moving data like 
that in /usr.  However, I think both of these problems could be solved 
to the point where an intelligent filesystem would beat an intelligent 
admin on partitioning any time.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9CPcwisZLIF6uqOAQJePA//UIo+NjWW6AwLSNpanthF+kggKGQ5R/ha
GYim57s3BRCKOl1JSSpopCsM2jgKQvSqUzdLOJJX7BB8G1YNoGNB5/RAHaW8EmEp
bJ3rZro+ki2lkA4zAMiXP5f1Sfk3McjFA07C0ZBVOy1pC1oZ1max7YvLn/qIdJPe
JuDeWRadNYly6Tcjykehio/Lsd5FJI52TBetZZ/SMuY6ib3EB8JPLrjjBpAxcXFZ
n9vZduC7rPEsS5YWtdWtUebR6uPZg4eYNUc2er1GG3Z0beMCwKtB6lM7/NxBd0Pt
JG+JekxKxTQxZOAaL88oOJ8wpiQMh0/BP8yHMimnOY7YwPKJYjblGl5Od9lngGoY
0dLEVjmjvPjUB4qyV8m1xxI1DGWbXOmjxV8zfZzYgvj6C1BqGArzc1pKlic0CrS3
ZgvoddjzEZlSSfmndlzh9JokIxsI63vLxA6qUkLYVn0NCKKq/XecmKlhHGCOPEWD
t0FLmqjYl3p2rSrpUrcCiZXOOhjCKo5ae2NHlfT6aDJDR930i1QFROYZdSWi8L7y
/nqnlKdQOQuP7dI+cndZTAf4WvHS095TE/rsveKlkMhVpo+V4UbWse6q5xTrzU3X
CYJkggpmzhwiRF+GEscnCcY2uCIxn8cteShpWmlYjJmZZ1AofaFC8b/ChKutrPJP
SuxuINXgjhE=
=tsHB
-----END PGP SIGNATURE-----

Re: A bold idea (Re: Carrying Attributes too Far)

[Hubert Chan]

>>>>> "David" == David Masover <jedi@ninja.dynup.net> writes:

[...]

David> Now, why would I use separate filesystems in the first place?
David> Two reasons: performance and security.  Right now, it's a lot
David> easier to have /var be a certain sized partition rather than
David> trying to enforce disk quotas. ...

There's more to security and good administration than just disk quotas.
You can have different partitions mounted read-only.[1] You have
different mount attributes such as nodev, nosuid, noexec.  You may even
want to take advantage of the fact that you can't hardlink across
partitions (you don't want users to be able to hardlink programs from
/usr/bin).  Separate partitions also allows you to easily reinstall by
blowing away your root partition (after copying your /etc), e.g. if
your system gets compromised.  And so forth.

[1] This presents another problem to hardlinking across partitions.
What if you try to hardlink a file from a read-only partition onto a
read-write partition (or vice versa)?  A hardlink is supposed to be
essentially the same on both sides, so will you be able to write the to
the file or not?  Whichever one you choose, something breaks.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: A bold idea (Re: Carrying Attributes too Far)

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hubert Chan wrote:

>>>>>>"David" == David Masover <jedi@ninja.dynup.net> writes:
>>>>>>            
>>>>>>
>
>[...]
>
>David> Now, why would I use separate filesystems in the first place?
>David> Two reasons: performance and security.  Right now, it's a lot
>David> easier to have /var be a certain sized partition rather than
>David> trying to enforce disk quotas. ...
>
>There's more to security and good administration than just disk quotas.
>You can have different partitions mounted read-only.[1] You have
>  
>
And how is that different than chmod -x?  Root can remount a partition 
writable just as easily as chmod'ing a file, unless the partition is 
something like iso9660.

>different mount attributes such as nodev, nosuid, noexec.  You may even
>want to take advantage of the fact that you can't hardlink across
>partitions (you don't want users to be able to hardlink programs from
>/usr/bin).  Separate partitions also allows you to easily reinstall by
>  
>
Why not?  (Naive question -- I can't see any problem here.)

>blowing away your root partition (after copying your /etc), e.g. if
>your system gets compromised.  And so forth.
>  
>
There are many ways of doing this, including:  copy to a network server, 
make a temporary partition (after resizing the main one), burning a CD, etc.

>[1] This presents another problem to hardlinking across partitions.
>What if you try to hardlink a file from a read-only partition onto a
>read-write partition (or vice versa)?  A hardlink is supposed to be
>  
>
I argued not for hardlinking across partitions, but for improving 
flexibility and raw speed to where none of the usual reasons for having 
multiple partitions (or for hardlinking across partitions) matters.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9EzFQisZLIF6uqOAQLvfRAAmzFz71o9ur9nfysFutnEoEY1sD5s8KQh
Jyc1BqK2skLKs5RNZJbKJ3AsGnkdZ/ABR265k99Bt/CKkM+KuNy8gqsG6Y6qwNFA
gWYbeXgkhm0ql9ih0VQ/+exm9AhLL2/DzgEf6ND7yYCFxnfxqmH6uGAQxq64yIU+
G64+JtoIbALODSv4a29VPDIjXfgEr5YZFFIDjw0tMX/KV98n419CVJUfg9qeIWN8
Oi2XJQzXVyOXx9kt2YxXVOPqLUh5Vody6GC5vckO5CNut6G4FdolW8V6NkJo8A3t
Yeg995M1v4B4h6a4HjH/OrYOllAB7zmyMw5v4gZI0MSmChLwpLnwXuxPsyaLcOJy
PVFtZMrHgk/fmH3Ur0HxaJxofiToZOOsh+Bc9vGcKQek8B/+zzeakEu+L6hwYdsq
FCC1BryKz7guRBgXexr1DCalYH7w+NrZsELUYt6ocrBuB48dJMSVoqcLcW4DPgHD
fZZozuWDT2Psnn8NGlGhQFs/CapSyMwaWjcUtFxKrRu8hc1ZfXS18tUoFtlVLctP
1NQ43MhQ8VME0Rg6k54CXNjq7boi2Zn3pvtupWIN+ZE5ZmzCZTVP97Yz0X25cBYF
YquetvWZLQrsI6nFSZYrLmUe7IsMgNuJa7CZZdxhPrl0Zpnx+2Y6RGnHMYEsvolv
cEDSKfEpdmY=
=n7JO
-----END PGP SIGNATURE-----

Re: A bold idea (Re: Carrying Attributes too Far)

[Hubert Chan]

>>>>> "David" == David Masover <jedi@ninja.dynup.net> writes:

[...]

>> There's more to security and good administration than just disk
>> quotas.  You can have different partitions mounted read-only.[1] You
>> have
>> 
David> And how is that different than chmod -x?  Root can remount a
David> partition writable just as easily as chmod'ing a file, unless the
David> partition is something like iso9660.

Yes.  Mounting a partition read-only mainly protects against
accidentally doing something stupid.  (e.g. "rm -rf /")

(What does "chmod -x" have to do with mounting read-only?  Or did you
mean "chmod -r"?)

>> different mount attributes such as nodev, nosuid, noexec.  You may
>> even want to take advantage of the fact that you can't hardlink
>> across partitions (you don't want users to be able to hardlink
>> programs from /usr/bin).  Separate partitions also allows you to
>> easily reinstall by
>> 
David> Why not?  (Naive question -- I can't see any problem here.)

There was a recent thread on Bugtraq about: if a user can hardlink from
/usr/bin, then they could link an suid program.  If a vulnerability is
discovered later, and the admin (or packaging program) just rm's the
file, the user still has access to it through his hard link.  (The
solution is to truncate the file to 0, drop the suid bits, and then rm,
but you might forget.)

>> blowing away your root partition (after copying your /etc), e.g. if
>> your system gets compromised.  And so forth.
>> 
David> There are many ways of doing this, including: copy to a network
David> server, make a temporary partition (after resizing the main one),
David> burning a CD, etc.

Yes, but being able to just blow away your root partition to reinstall
is a whole lot easier.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: A bold idea (Re: Carrying Attributes too Far)

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>Yes.  Mounting a partition read-only mainly protects against
>accidentally doing something stupid.  (e.g. "rm -rf /")
>  
>
Good point.  However, the usual way to do that is "alias rm='rm -i' or 
to never give the root password to people who do things like that.  And 
if you're already backing up to protect against people breaking in and 
remounting it, you've already got a backup.

>(What does "chmod -x" have to do with mounting read-only?  Or did you
>mean "chmod -r"?)
>  
>
Actually, I meant "chmod -w".

>  
>
>>>different mount attributes such as nodev, nosuid, noexec.  You may
>>>even want to take advantage of the fact that you can't hardlink
>>>across partitions (you don't want users to be able to hardlink
>>>programs from /usr/bin).  Separate partitions also allows you to
>>>easily reinstall by
>>>
>>>      
>>>
>David> Why not?  (Naive question -- I can't see any problem here.)
>
>There was a recent thread on Bugtraq about: if a user can hardlink from
>/usr/bin, then they could link an suid program.  If a vulnerability is
>discovered later, and the admin (or packaging program) just rm's the
>file, the user still has access to it through his hard link.  (The
>solution is to truncate the file to 0, drop the suid bits, and then rm,
>but you might forget.)
>  
>
That's true, it'd probably be a good idea to either make it impossible 
to create a hardlink to a file you don't own or (the simpler solution) 
patch packaging software to do the truncating for you.  The second 
approach makes a lot of sense because some distros (maybe all of them?) 
make only /boot, /, and swap by default (just as I was describing).  
Maybe a "hardlink" permission flag?

>  
>
>>>blowing away your root partition (after copying your /etc), e.g. if
>>>your system gets compromised.  And so forth.
>>>
>>>      
>>>
>David> There are many ways of doing this, including: copy to a network
>David> server, make a temporary partition (after resizing the main one),
>David> burning a CD, etc.
>
>Yes, but being able to just blow away your root partition to reinstall
>is a whole lot easier.
>  
>
I use Gentoo, so there is no "easy" way to do an install -- it's always 
going to take a lot of crunching time, not necessarily a lot of your 
time.  So you do a network backup, automated, to, say, a web server.  
Then you download it and unpack it.  It's easier, sure, but not 
necessarily a lot easier.  Certainly not always a lot easier than 'rm -rf'.

I realize there are some good points made here, and I don't think it 
should be impossible to have separate partitions.  I do, however, think 
that most of these security issues and sanity checks are much easier to 
deal with (especially for a newbie) than managing 10-20 separate 
partitions, even with LVM.  (And that's ignoring the performance issues 
of LVM.)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9IUhwisZLIF6uqOAQL6fQ//fIKObk9XFM76V7d8PqdWXiCB4EnaINR6
RlDoVgabeW0CMsd4G3fBY6nnuTVtEhC4HouF5nFfXfea0kHs7VmPvn1jjqQI/7/D
bDkY7fuBwt7WQg+9Kez4GtsTEtDHkVOLzpDwq9feI3ouw1t/IpzXkLc5PdgOAoB9
KaLLHp3jSeKa813t/sJLJfj03aPL+OPNlB5da3y1mqechLEXV2TQnGhTX70ReS6l
p4SPYeth6OI/5clEXPBx2omXHOTh9wGtjw1SOpP1JEFLqKZFEKpQhaSOIv71beW/
uMWRLDuf100cv2QYBgmibuxU+bmoZuIR+SAiOuJgGLgXUbwWs6yjedQR82lGb6xY
md3TB5PSaJZUpPil34hGQquwJvk7MHWsLinA9DkGMyyqWBkmPZl8kyAyZ8RbHhEd
Z/k14/VxeOh/ZYnlfVXJM0OLvfUZFHkyS7+G5BWmgqvg9+2xXYFsgRg1QLMu8OP0
x7KttP6Q8EgUCCJwjat4eQ0jHWQFHmko1sppFMZjeZqOw3E7sNvnKIeJVXYaMGu8
q3vQOGoYqtfDdBo2vHbKiMghPkNv7LcJ3K+LHlTqIKX9aAn0uVWmJqajuQn2JyBB
2DuwCUNcOGTPQzDjbv8t2iuix8hGP2O0ALhBNiz8lZ/JyHN2uubsa+18PdT+ddPG
wbPmiSaojNc=
=iN9Z
-----END PGP SIGNATURE-----

Re: A bold idea (Re: Carrying Attributes too Far)

[lrc1]

Quoth Hubert Chan <hubert@uhoreg.ca>:

...

> 
> [1] This presents another problem to hardlinking across partitions.
> What if you try to hardlink a file from a read-only partition onto a
> read-write partition (or vice versa)?  A hardlink is supposed to be
> essentially the same on both sides, so will you be able to write the to
> the file or not?  Whichever one you choose, something breaks.
> 

Yes, this is why I said earlier that hard linking across filesystems doesn't
fit with the standard Unix idea of mount points and mount subtrees. The
problem is the stipulation that the files on a volume must all form a single
hard-linked tree. Of course, for that to be the case every file on the volume
(except the root) must have at least one hard-parent which is also on the
volume. And since all the hard links to a file are indeed equal, no hard links
across volumes can be permitted (except to volume roots) lest a (non-root)
file be left with no hard-parents on the same volume and yet not be deleted.

(This is what makes it safe to assume that all of a directory's hard-children
are on the same volume as their parent (mount points aside), as you do above.)

The solution I suggest is to throw out the "one volume, one tree" requirement.
The volume's tree then becomes a forest, with every file with no parent in the
volume being the root of a tree. (Of course, if you also have multiple
hard links to directories, then the trees are instead directed, acyclic
graphs.) You could think of them all as being mount points, but in fact the
volume would have no fixed mount points at all.

Taking a step back, if all the filesystems on a system were so, then any file in
any location in the computer's global filesystem tree (or graph) could in
principle be on any of the computer's (sub)filesystems regardless of what
(sub)filesystem its parents were on. Thus the answer to your question above
would be that the file would be in ... whatever filesystem it happened to be in,
not necessarily that of /any/ of its parents.

stat()/lstat() could (still) be used to determine the filesystem of a given
file. To determine all the files on a given filesystem without (l)stat()ing
every file on the computer, some new means would be necessary. This could simply
be a new system call, but I would favour a kind of /proc-like setup, which would 
however require two different new types of link (weak non-symbolic links and 
links to directories which are to be interpreted as if they were links to 
non-directory files). Two-way linking between directories and their children and 
explicit deletion of files are also pretty much prerequisites for hard-linking 
across filesystems. More implementation problems and solutions on 
request.

...

Leo.


-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

carrying links too far? (was Re: A bold idea (Re: Carrying Attributes too Far))

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

lrc1@st-andrews.ac.uk wrote:

> ...
>
>The solution I suggest is to throw out the "one volume, one tree" requirement.
>The volume's tree then becomes a forest, with every file with no parent in the
>volume being the root of a tree. (Of course, if you also have multiple
>hard links to directories, then the trees are instead directed, acyclic
>graphs.) You could think of them all as being mount points, but in fact the
>volume would have no fixed mount points at all.
>  
>
Cool, but the problem there is that it is now possible to have broken 
hardlinks.  Broken symlinks we're all used to, but broken hardlinks 
usually represent a more critical problem.

>Taking a step back, if all the filesystems on a system were so, then any file in
>any location in the computer's global filesystem tree (or graph) could in
>principle be on any of the computer's (sub)filesystems regardless of what
>(sub)filesystem its parents were on. Thus the answer to your question above
>  
>
The problem here is that if the link itself is on a different filesystem 
than the actual data, and the original filesystem gets nuked, what do I 
do about the link?  Traditionally, in order to delete a link, you drop 
the refcount of whatever it points to.  Here, you'd be left with the 
choice of either having a way to forceably remove a link, even if it 
can't adjust the refcount, and have "lost chains" or orphaned files, or 
you'd be left with certain links that you cannot remove without 
reformatting that filesystem -- possibly in the process creating more 
such links on other filesystems.  Or you could just not have the 
refcounts (or, indirectly, the deletion of the file) have anything to do 
with any hardlinks (maybe "firmlinks"?) on other partitions/filesystems, 
but then the file goes away when all hardlinks are gone, leaving these 
"firmlinks" broken.

I guess the main problem I have with this is that it only works when 
we're talking about all the filesystems on the local machine, and then 
only so far.  It kind of falls apart with removable filesystems like 
floppies and cds.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9J/8gisZLIF6uqOAQLqyg/+PwacI8rg+seKyxunxWA5MJ7KuOYjkshx
PmmoMCFBCg/P/Ygm12q3vK+GJdIKDzAPWY6xoGtTAQclyihbvm9qV3DxMdeoteGG
rS+P2sEaQrfvzEnGMW7JpdWl+Y9EAUhJV/7bUblZRsAUKenASW5K/CM4LnYIohKN
ZJAet6nKdhCR9i3nTQ7k16FU5AV6jg4n/CEpdSFsgkqbWbM/fJ9O3tGFEsk3AnZM
NCKfYjsQybNX1gQYyG66sUNL5wtsYfTUAS6G+87p8CvfvmS4BPkMuhcEA5r9olO+
YefrO7vxX/hP7eVmBbTwktnpsUDV+Z53/+U5rMELFGlfIsUJ+NYcdZDkX4lsrFqO
4fBFpqysUmJc/z7LUsbf5uB6cB/5TVcgcQLoqbWPcFEb8snr5acDe0RLlbqY7BJb
NK/VSLlVG+DE2bnoIz2CfaLPIAEhf0V0Pf8v27wrmZ6JKrzZ6tEdBAzshv7qXxlx
z6ns8iCuiPGlwlxglTE9nKL9Cvjw8t+O2yOd4weu41MEMnwkpr3Fbza6nPFyKBJI
UaTgf+7O+2wAXfsy4g6ZaUdtJaeKoix+dSKsb/F8Q1gJfacxkC0MiRhG3mcUKFPa
YrHBQtts1dNsWw2o8IscA1o2u23J2lQkw45d8TkX7xxS4AJfkuevHzNI5abIFbzD
DAHEzyys3w4=
=1dYB
-----END PGP SIGNATURE-----

Re: carrying links too far? (was Re: A bold idea (Re: Carrying Attributes too Far))

[Hubert Chan]

>>>>> "David" == David Masover <jedi@ninja.dynup.net> writes:

[...]

David> I guess the main problem I have with this is that it only works
David> when we're talking about all the filesystems on the local
David> machine, and then only so far.  It kind of falls apart with
David> removable filesystems like floppies and cds.

And network mounts.

OK, another problem with Leo's scheme (and with hardlinking across
filesystems).  If I create a hardlink from partition A to partition B
(i.e. the file physically resides on partition B), and then unmount
partition B, I should still be able to access the file, since hardlink
semantics stipulate that the "link" on partition A is the actual file
itself, and not just a pointer.  But the file is not accessible, since
partition B is unmounted.  So things break.

Hmm.  Maybe later I'll write a couple of paragraphs on why it is
impossible to handle cross-filesystem hardlinking in any sane way.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

The danger of bad external links

[lrc1]

Let me begin with a reading from scripture:

    The same non-directory file may appear in several directories under possibly
    different names. This feature is called linking; a directory entry for a
    file is sometimes called a link. The Unix system differs from other systems
    in which linking is permitted in that all links to a file have equal status.
    That is, a file does not exist within a particular directory; the directory
    entry for a file consists merely of its name and a pointer to the
    information actually describing the file. Thus a file exists independently
    of any directory entry, although in practice a file is made to disappear
    along with the last link to it.

    - The UNIX Time-Sharing System, D. M. Ritchie and K. Thompson
    http://cm.bell-labs.com/cm/cs/who/dmr/cacm.html

So Unix hard-links should indeed be thought of as references to a file rather
than as the file itself. No hard-link is any more or less just a pointer than
any other - but they /are/ all "just pointers".

Now, in the system I propose there are three things that threaten to create
broken or blocking hard-links:

1) A file is cleanly deleted

2) A filesystem is cleanly unmounted

3) Bad Stuff happens

1) As you know, unlink() does exactly what it says: it deletes the specified
link to a file. This deletes the file if and only if it then has no hard-links
left. Explicit file deletion does pretty much the reverse: a file's owner (or
root) can use it to delete a file, and it will then delete the one or more links
that would otherwise be left hanging. Of course, to do that it needs to find the
hard links to the file. A reference count won't tell us that, so instead we need
a means to find the hard-parent directories of a given file. (As it happens we
need that feature for some other useful things too, such as finding all the
pathnames of a given file.) The obvious way to do this is to maintain pointers 
back from each file to all its parent directories. (If the deleted file is a 
directory, this means that we will also have to clean up by deleting all the 
reverse links from its children.)

Once you have what you need to do explicit file deletion, it should be easy to
add weak non-symbolic links too. These are exactly like hard links except that
they don't keep a file from being deleted: if a file has only weak non-symbolic
links to it, it will be gced. unlink()ing the last hard link to a file with weak
non-symbolic links should be exactly the same as explicitly deleting it.

2) When a filesystem is unmounted, every external link to the files on it should 
normally be deleted. If we have the ability to do explicit file deletion, we 
should already have everything we need to do this. (Again, we also need to tidy 
up external reverse links to directories that are being removed.)

3) When either of the two filesystems involved in an external hard link screw up 
then the link may block or break, and there is no realistic way to prevent this. 
But then this is nothing new. Any filesystem which screws up can mangle internal 
hard links, corrupt file bodies, block all calls to it forever, or what have 
you. The important thing is that the damage can extend only to its own files and
their descendants, and external hard linking does not change this. (And if a
file on a well-behaved filesystem has one path to root which passes through a
misbehaving filesystem, and one which does not, only the path through the
misbehaver can be affected.) The /don't do that, then/ principle applies here:
if you don't trust a particular filesytem to store an important file, or to
record an important pathname, then don't put the file, or any of the directories
which record segments of the pathname, in that filesystem.

More later, and in my reply to David Masover.

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: The danger of bad external links

[Hubert Chan]

>>>>> "Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:

[...]

Leo> So Unix hard-links should indeed be thought of as references to a
Leo> file rather than as the file itself.

A hardlink "is the file" just as much as the "original" file "is the
file".  Neither one should be given any privilege over the other.
That's what I mean when I say that an external hardlink shouldn't be
thought of as just a pointer.  Yes, technically, it is just a pointer,
as is the "original".  But normal people don't think of it that way.
Normal people think of the file *being* the file itself, and so if you
want to implement cross-filesystem hardlinking, normal people should be
able to treat the file just like a normal file.  Not like a pointer
that points to a remote resource that may or may not be available.

[...]

Leo> 1) As you know, unlink() does exactly what it says: it deletes the
Leo> specified link to a file. This deletes the file if and only if it
Leo> then has no hard-links left. Explicit file deletion does pretty
Leo> much the reverse: ...

Why is explicit file deletion needed?  UNIX systems have been running
fine without it for many many years.  Is there something special about
external hardlinks that requires this?

[...]

Leo> Once you have what you need to do explicit file deletion, it should
Leo> be easy to add weak non-symbolic links too. These are exactly like
Leo> hard links except that they don't keep a file from being deleted:
Leo> if a file has only weak non-symbolic links to it, it will be
Leo> gced. ...

OK, I think this is what I would call a firmlink.  "Weaklink" to me
would imply something that's weaker than a symlink.  A firmlink implies
something stronger than a symlink, but weaker than a hardlink.  But
that's just arguing language.  If you say "weaklink", now I'll know
what you mean.

Leo> 2) When a filesystem is unmounted, every external link to the files
Leo> on it should normally be deleted. ...

This breaks hardlink semantics by treating one of the links as
preferential over the other.

David pointed out another of my objections to this.  When you remount,
you should have some way of restoring the links.  But then you should
be careful to make sure that bad things don't happen in the mean time.

Another problem with this is that it depends on what order the
filesystems are unmounted.  If filesystem A has a link to file that's
physically stored on filesystem B, and filesystem A gets unmounted
first, then the link won't be updated when filesystem B gets
unmounted.  Then, when you remount filesystem A, you have a broken link.

Leo> 3) When either of the two filesystems involved in an external hard
Leo> link screw up then the link may block or break, and there is no
Leo> realistic way to prevent this.  But then this is nothing new. Any
Leo> filesystem which screws up can mangle internal hard links, corrupt
Leo> file bodies, block all calls to it forever, or what have you. ...

Sure, but then this is because of programming error, or bad hardware.
If you do cross-filesystem hardlinking, broken hardlinks will become a
matter of normal system operation, and will basically be the norm,
rather than the exception.


What I'm really wondering, though, is why we need cross-filesystem
hardlinking in the first place.  I have only ever seen approximately one
good use of hardlinking, and it wouldn't have benefited from having any
cross-filesystem capability.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: carrying links too far? (was Re: A bold idea (Re: Carrying Attributes too Far))

[lrc1]

Quoting David Masover <jedi@ninja.dynup.net>:

[...]

> lrc1@st-andrews.ac.uk wrote:

[...]

> >
> The problem here is that if the link itself is on a different filesystem 
> than the actual data, and the original filesystem gets nuked, what do I 
> do about the link?  Traditionally, in order to delete a link, you drop 
> the refcount of whatever it points to.  Here, you'd be left with the 
> choice of either having a way to forceably remove a link, even if it 
> can't adjust the refcount, and have "lost chains" or orphaned files, or 
> you'd be left with certain links that you cannot remove without 
> reformatting that filesystem -- possibly in the process creating more 
> such links on other filesystems. 

If the child file's filesystem gets unmounted, then the external link to the 
child file should normally be deleted; it certainly shouldn't be left hanging. 
The child file won't just have a refcount; it will have pointers back to all of 
its external parent directories (and possibly its internal ones too). So there's 
no danger of external references being mistaken for ordinary internal ones. And 
the external reference information can be used to figure out where to reconnnect 
the filesystem when it is next remounted - exactly how is a longer story though.

> Or you could just not have the 
> refcounts (or, indirectly, the deletion of the file) have anything to do 
> with any hardlinks (maybe "firmlinks"?) on other partitions/filesystems, 
> but then the file goes away when all hardlinks are gone, leaving these 
> "firmlinks" broken.

'Firmlinks' is probably a bad word, since apparently the things are known as
weak references in garbagecollectionese.

> 
> I guess the main problem I have with this is that it only works when 
> we're talking about all the filesystems on the local machine, and then 
> only so far.  It kind of falls apart with removable filesystems like 
> floppies and cds.

There are several different ways you can handle filesystems which are
frequently mounted and remounted. Old single-tree filesystems can probably be
adapted or fooled into working reasonably well with external hard links too. (I
expect that making them work with multiple hard-links to directories would be
more difficult.)

[...]

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: carrying links too far? (was Re: A bold idea (Re: Carrying Attributes too Far))

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

lrc1@st-andrews.ac.uk wrote:

>Quoting David Masover <jedi@ninja.dynup.net>:
>
>[...]
>
>  
>
>>lrc1@st-andrews.ac.uk wrote:
>>    
>>
>
>[...]
>
>  
>
>>The problem here is that if the link itself is on a different filesystem 
>>than the actual data, and the original filesystem gets nuked, what do I 
>>do about the link?  Traditionally, in order to delete a link, you drop 
>>the refcount of whatever it points to.  Here, you'd be left with the 
>>choice of either having a way to forceably remove a link, even if it 
>>can't adjust the refcount, and have "lost chains" or orphaned files, or 
>>you'd be left with certain links that you cannot remove without 
>>reformatting that filesystem -- possibly in the process creating more 
>>such links on other filesystems. 
>>    
>>
>
>If the child file's filesystem gets unmounted, then the external link to the 
>child file should normally be deleted; it certainly shouldn't be left hanging. 
>The child file won't just have a refcount; it will have pointers back to all of 
>its external parent directories (and possibly its internal ones too). So there's 
>no danger of external references being mistaken for ordinary internal ones. And 
>the external reference information can be used to figure out where to reconnnect 
>the filesystem when it is next remounted - exactly how is a longer story though.
>  
>
Aside from breaking the idea of a "hardlink" (forcing us to do "weak 
links"), there's the question of why would someone want to do this on 
any kind of system that is ever rebooted.  If I unmount the filesystems 
to reboot, I break (and thus remove) any hardlinks on other partitions.  
If the hardlinks are so fragile, what is the advantage over symbolic links?

> ...
>
>>I guess the main problem I have with this is that it only works when 
>>we're talking about all the filesystems on the local machine, and then 
>>only so far.  It kind of falls apart with removable filesystems like 
>>floppies and cds.
>>    
>>
>
>There are several different ways you can handle filesystems which are
>frequently mounted and remounted. Old single-tree filesystems can probably be
>adapted or fooled into working reasonably well with external hard links too. (I
>expect that making them work with multiple hard-links to directories would be
>more difficult.)
>  
>
Yes, it would be harder, because that would involve changing the method 
of garbage collection.  Hardlinks across filesystems only requires 
(depending on how well it's implemented) heavy use of fsck.  But that's 
only the technical details.  I get the feeling that the high-level 
design (as has already been shown) is much more difficult.  It seems 
that everyone agrees on what hardlinks to directories should look like, 
and it seems the worst danger is breaking "find".  No one seems to agree 
on what hardlinks across partitions should look like, if they should 
even be done at all.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9NSkgisZLIF6uqOAQKehA//UkGwdhTznO16VfgQpxUDxjGYD3wi8VT9
PF8b37WLEOC10p3L8gSf1gzx49zxGH0qcJjVS7vucZEoohfv7kn92g5k9Fm9hPXf
GrKVfV1p42jddAP9qthZ6drEWEoSkJWSngPleiq8IzVDwqptFfBs5tgvX/PqRA8a
agkAVEidfElJW/xOPYDGmJZ3PzeieChT21QmFUGIKCXEa4/N8vrbuerQxI2jpI8y
WGPNuRpvFB21JB1a3323arVdcQwGIh//sCGEtqI4359TtnVRXitoo6CHSQznerMy
UL3DMjHAPg0KJLFmMQE9st7mBQSyQDJIdb2UqdH9OZKQTqgl/x+dantU4jL4wFlx
b8ciJN2k4DeyDkhKlMFt+RMaPGd4uPYcF9R59Q5CUJO2eLpppQ/qBixS9td+zJN4
qomRfr18a7p0vOgkKcaGkW+sl/le19s55QG+i6x7Wru9uZq/JHG0L5yc/c0tXt0z
Auvqy7ZEeK1TjlMViXBdJ48jvw0zuk41d/DFPOpQKs26cpV0wc1c3M/JaZmXuzpo
e4Nrlegp0y4IbFiWt03VDBEq+durHU6m5fK6qaWA545j1EjsVqbGuTWyWkhk8YAE
B2tGHkO9WVytypOgi5Zj7Mmk1MaSyX9AA4ZK15Mr+Ks5VNxLMoY1fvTSCflPIYVp
J/TXcwDKHjc=
=FZzV
-----END PGP SIGNATURE-----

Re: carrying links too far? (was Re: A bold idea (Re: Carrying Attributes too Far))

[Hubert Chan]

>>>>> "Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:

[...]

Leo> There are several different ways you can handle filesystems which
Leo> are frequently mounted and remounted.

I don't see any way that would work reasonably well.  Can you give an
example of one way?

Leo> Old single-tree filesystems can probably be adapted or fooled into
Leo> working reasonably well with external hard links too.

I'm not sure what you mean by this.  I think any filesystem that needs
to work with external hardlinks, in either direction, needs to be
patched -- either to be able to store the location of an external
hardlink, or to be able to update a refcount without having a new local
reference.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: A bold idea (Re: Carrying Attributes too Far)

[Hubert Chan]

>>>>> "Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:

[...]

Leo> The solution I suggest is to throw out the "one volume, one tree"
Leo> requirement.  The volume's tree then becomes a forest, with every
Leo> file with no parent in the volume being the root of a tree.

[...]

Leo> Thus the answer to your question above would be that the file would
Leo> be in ... whatever filesystem it happened to be in, not necessarily
Leo> that of /any/ of its parents.

Umm.  You aren't answering the problem that I posed, and your solution
doesn't solve the issue.  If you have a file that is hardlinked from a
read-only filesystem to a read-write filesystem (or vice versa), then
can you write to that file or not.  As a hardlink, the file should
inherit from its filesystem, but since it is essentially contained on
two filesystems (where it actually physically resides is unimportant),
as hardlink semantics would require, the file must both be read-write
and read-only.

David also points out problems with your solution in maintaining
consistency, which I agree with.

(BTW, David, I'm dropping our other conversation about partitions, since
it is very off topic, and not really very productive, since I don't
think you're advocating that we should remove the capability of having
multiple partitions.)

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: A bold idea (Re: Carrying Attributes too Far)

[Boyd Waters]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Heinz-Josef Claes wrote:

| I think hard linking directories is a very good idea. The should be a
| count for the number of hard links and it should be handled like with
| normal files. I directly have an application with would have a great
| benefit of this kind of hard linked directories, but up to now they are
| not available in linux :-(

I use

mount -o bind path/to/subdirectory path/to/mountpoint

as a sort-of "hard-link" directories...

But it is not the same thing as a "real" hard-link of a directory.
(I am not familiar with the kernel internals, beyond a quick read of
this month's Linux Journal article on dentry cache.)

~ - boyd

Boyd Waters
National Radio Astronomy Observatory
http://www.aoc.nrao.edu/~bwaters
Socorro, New Mexico
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQE/1OUN0is8k1r0QeURAitsAKCF7X2D7L8TvoLl/kexksP/K/qvVQCfWnPu
aq2IyX+aPYFpFXWNxkK0xCY=
=hwEx
-----END PGP SIGNATURE-----

Re: A bold idea (Re: Carrying Attributes too Far)

[Heinz-Josef Claes]

Am Mo, 2003-12-08 um 21.54 schrieb Boyd Waters:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Heinz-Josef Claes wrote:
> 
> | I think hard linking directories is a very good idea. The should be a
> | count for the number of hard links and it should be handled like with
> | normal files. I directly have an application with would have a great
> | benefit of this kind of hard linked directories, but up to now they are
> | not available in linux :-(
> 
> I use
> 
> mount -o bind path/to/subdirectory path/to/mountpoint
> 
> as a sort-of "hard-link" directories...
> 
Yes, that would look a little bit like a hard link. But I want to use it
as a possiblity for my backup tool (see below). It makes a backup to
another hard disk and hard links to files with a contents which is
already in the backup. If a hole directory had not changed, I could make
a link to the previous one and that's all. Now I have to mkdir the
directories and hard link every single file. I could use much less space
if there were an option to hard link directories (with a count for
proper deletition).

Regards,

 
Heinz-Josef Claes                     hjclaes@web.de
project: http://sourceforge.net/projects/storebackup
      -> snapshot-like backup to another disk

more about links (was Re: A bold idea (Re: Carrying Attributes too Far))

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Heinz-Josef Claes wrote:

>Am Mo, 2003-12-08 um 21.54 schrieb Boyd Waters:
>  
>
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA1
>>
>>Heinz-Josef Claes wrote:
>>
>>| I think hard linking directories is a very good idea. The should be a
>>| count for the number of hard links and it should be handled like with
>>| normal files. I directly have an application with would have a great
>>| benefit of this kind of hard linked directories, but up to now they are
>>| not available in linux :-(
>>
>>I use
>>
>>mount -o bind path/to/subdirectory path/to/mountpoint
>>
>>as a sort-of "hard-link" directories...
>>
>>    
>>
>Yes, that would look a little bit like a hard link. But I want to use it
>as a possiblity for my backup tool (see below). It makes a backup to
>another hard disk and hard links to files with a contents which is
>already in the backup. If a hole directory had not changed, I could make
>a link to the previous one and that's all. Now I have to mkdir the
>directories and hard link every single file. I could use much less space
>if there were an op
>
If a symlink really won't work for you, have the whole directory be a 
hardlink farm.  Trust me, it's cleaner that way.

My backup system involves using rsync to do that -- first I do a 'cp 
- -al' from the old generation dir to the new one.  Then I rsync over my 
current files.  Only the files that have changed are updated.

If that still isn't enough, use symlinks.  Some things which used to be 
hardlinks on my distro have been changed to symlinks, and I like the change.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9aBCgisZLIF6uqOAQK0YA/6AxzwZ33y1n+NVY/5WbJhFz30tcQDLcy4
l/5d8BQi75JGaGphuhb40+wh9omNQ+WBDYnTpL93XC2JeegmTbO5MYRqhTLyvfmD
oPkSNN9F3O1FdLDAIKJbJKu5zuQMulifzuqw1/Ru9j9Tck8JkRmlmCws/OFKoTTB
lN0IU9PiobBbOGrGpisfHARxy5rzVLXVpO3zUFLV8nrIdym2AVUi0sg4QAvfgaAT
FOLFDADdySPGFUyHoUYZ2dYNh9X+XiEw2fGJIvL1PzjfO+C9vWFpi5CW9goH3WKf
FInlVTkhijJPVi/lhRG2MnMcih64IH0xgTlm8jUBlgwnxc1/rncjBwISJNu+v7Si
y0yOa+CWcCajoiYrLrTzUI7ltDiynhZWhJClGGXIdqdpD606taU5N9t7nZk6TkTT
2fnXEnTSUaOp+vjNT2HsZ287ZITmCeMfD4O3okqbmTEM+Tg6fzxRgF45p3h2mKxn
44lLEBJY22rvVsOkZseP/Ar82fknyaHBIQy9BH2ZMwptPBbD8Nb40PH3FciZMdta
5D+emORumurMe8vnYNesl7NKQUZ/kKeEBZ87Pg2MQHKJbPRG+HzrIBtkhsXGa30n
UqnX8vvZGrDRZvJJKh98u6ljRVrhPVgz7hzpDpZsU1U2Kx0VfmLaHF1/VHNcKKjp
8Vu0oMwEwVQ=
=S65a
-----END PGP SIGNATURE-----

Re: more about links (was Re: A bold idea (Re: Carrying Attributes too Far))

[Heinz-Josef Claes]

Am Mi, 2003-12-10 um 03.12 schrieb David Masover:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Heinz-Josef Claes wrote:
> 
> >Am Mo, 2003-12-08 um 21.54 schrieb Boyd Waters:
> >  
> >
> >>-----BEGIN PGP SIGNED MESSAGE-----
> >>Hash: SHA1
> >>
> >>Heinz-Josef Claes wrote:
> >>
> >>| I think hard linking directories is a very good idea. The should be a
> >>| count for the number of hard links and it should be handled like with
> >>| normal files. I directly have an application with would have a great
> >>| benefit of this kind of hard linked directories, but up to now they are
> >>| not available in linux :-(
> >>
> >>I use
> >>
> >>mount -o bind path/to/subdirectory path/to/mountpoint
> >>
> >>as a sort-of "hard-link" directories...
> >>
> >>    
> >>
> >Yes, that would look a little bit like a hard link. But I want to use it
> >as a possiblity for my backup tool (see below). It makes a backup to
> >another hard disk and hard links to files with a contents which is
> >already in the backup. If a hole directory had not changed, I could make
> >a link to the previous one and that's all. Now I have to mkdir the
> >directories and hard link every single file. I could use much less space
> >if there were an op
> >
> If a symlink really won't work for you, have the whole directory be a 
> hardlink farm.  Trust me, it's cleaner that way.
> 
> My backup system involves using rsync to do that -- first I do a 'cp 
> - -al' from the old generation dir to the new one.  Then I rsync over my 
> current files.  Only the files that have changed are updated.
> 
> If that still isn't enough, use symlinks.  Some things which used to be 
> hardlinks on my distro have been changed to symlinks, and I like the change.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.3 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
> 
Hi,

if I would use symlinks, I'll run into problems when deleting. With a
symlink, I have the original file and a pointer to that file. That's
quite different from having hard links, where the file (or in this case
the whole directory) will be deleted if the last link is removed.

Using mount does not change this behavior. If I delete an old backup,
the whole (shared) directory is deleted for *all* backups.
If you want to see what's possible with deletition strategies, take a
look at www.sf.net/projects/storebackup. These will not work with
symlinks or multiple mount points.

So hard links to a directory would be a very useful feature for my use
case.

Regards,
Heinz-Josef Claes

More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Alexander G. M. Smith]

David Masover wrote on Thu, 04 Dec 2003 21:58:42 -0600:
> I agree that hardlinking directories should be allowed, and I think it's
> worth it to check whether it creates a loop, unless rm can be modified
> to simply rm the directory, and the filesystem (and maybe VFS) would be
> modified to have directories have link counts that don't include . or
> .., so when a directory is to be deleted, the system does its own manual
> deletion of each of the files within it.  Then, if I type 'rm loop' and
> it removes the link of 'loop/loop' to 'loop', there are now no more
> links to 'loop' and it gets destroyed.

I think you need more than link counts.  Like I mentioned earlier, I used
a graph traversal in my BeOS file system to handle deletion of loops.

Unfortunately that implies locking potentially large numbers of files while
doing the traversal, which can fail or deadlock if the file system is in
continuous use (like it might be on a server).  But then you already have
that sort of problem on regular file systems, though only for one file at
a time.

- Alex

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:

>David Masover wrote on Thu, 04 Dec 2003 21:58:42 -0600:
>  
>
>>I agree that hardlinking directories should be allowed, and I think it's
>>worth it to check whether it creates a loop, unless rm can be modified
>>to simply rm the directory, and the filesystem (and maybe VFS) would be
>>modified to have directories have link counts that don't include . or
>>.., so when a directory is to be deleted, the system does its own manual
>>deletion of each of the files within it.  Then, if I type 'rm loop' and
>>it removes the link of 'loop/loop' to 'loop', there are now no more
>>links to 'loop' and it gets destroyed.
>>    
>>
>
>I think you need more than link counts.  Like I mentioned earlier, I used
>a graph traversal in my BeOS file system to handle deletion of loops.
>
>Unfortunately that implies locking potentially large numbers of files while
>doing the traversal, which can fail or deadlock if the file system is in
>continuous use (like it might be on a server).  But then you already have
>that sort of problem on regular file systems, though only for one file at
>a time.
>  
>
Yes, locking, as a transaction.  Which means either the whole deletion 
fails (couldn't get the lock, user got annoyed and hit ^C) or it 
succeeds (got lock on all the files by doing some sort of recursive lock 
on the root dir).  From inside the transaction, it should look as though 
everything is happening in realtime, but it only flushes when it can get 
the lock.  (right?)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9CRLAisZLIF6uqOAQJoLg/7Bou4ztb1My4YJ7NTFtLz6lg9WeRqGTaN
Kocw64tzSIR9+iyFh0z1zHRzwTwJr35W7TBbitf6zdNIDbkMi/qK6fehaG6ipRXh
+FX4ZEyjLpW5Q9ST39wEo4rpGNC9YVGvoOVz4YVITz/wASBsh8ZoY87lM43OD/n5
5UQodUFnXtPBcJrEqWeFRGkl0viOPdq3XTu5XNLATClszN/k9BYKXeyWRgNLH9xl
LbcWMCAW0DtsyB4V7Kwv/gCKicpVT8ss4V+91Jq0Ls1jvr53TTY9eMLejfjMtrcA
hAkioRjJjsJrqEh2LLRl5QvXamWNPu5B3ghL+aSBLjSHz3BboQn14U/zVN/42svE
B+2QYB7cY+JguzuYsBEbFN7qtjFzgMTQH5AMlg1SdXY+dYL+XIHmcROHvFkB/uaT
vlICasDktv1dzubyw9TI9nbGuttSleuqi4gqM/51IaP09BHTzY+mdluup86zZD8+
8ShE440Hq8swLlVN7JTYMDxwqJ78yJTvDrgKxlgUWulAjQqRTWPyZ1Hbru0iaA5F
ayIXFXSwlgRU1zDM/6rNsNpCoMiMPdZjH+NOd/3QSBnIWoc7ybsp6hxyOZruz5ej
1yA7jI0Qjs/FhAPMswZGVGI4/qY08nEYqA1ffrjApcdjLQFWx02Z0s1NTeVy2XGd
8D3gUzOLmoc=
=Mp6q
-----END PGP SIGNATURE-----

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Nikita Danilov]

David Masover writes:
 > -----BEGIN PGP SIGNED MESSAGE-----
 > Hash: SHA1
 > 
 > Alexander G. M. Smith wrote:
 > 
 > >David Masover wrote on Thu, 04 Dec 2003 21:58:42 -0600:
 > >  
 > >
 > >>I agree that hardlinking directories should be allowed, and I think it's
 > >>worth it to check whether it creates a loop, unless rm can be modified
 > >>to simply rm the directory, and the filesystem (and maybe VFS) would be
 > >>modified to have directories have link counts that don't include . or
 > >>.., so when a directory is to be deleted, the system does its own manual
 > >>deletion of each of the files within it.  Then, if I type 'rm loop' and
 > >>it removes the link of 'loop/loop' to 'loop', there are now no more
 > >>links to 'loop' and it gets destroyed.
 > >>    
 > >>
 > >
 > >I think you need more than link counts.  Like I mentioned earlier, I used
 > >a graph traversal in my BeOS file system to handle deletion of loops.
 > >
 > >Unfortunately that implies locking potentially large numbers of files while
 > >doing the traversal, which can fail or deadlock if the file system is in
 > >continuous use (like it might be on a server).  But then you already have
 > >that sort of problem on regular file systems, though only for one file at
 > >a time.
 > >  
 > >
 > Yes, locking, as a transaction.  Which means either the whole deletion 
 > fails (couldn't get the lock, user got annoyed and hit ^C) or it 
 > succeeds (got lock on all the files by doing some sort of recursive lock 
 > on the root dir).  From inside the transaction, it should look as though 
 > everything is happening in realtime, but it only flushes when it can get 
 > the lock.  (right?)

Deadlock detection and/or avoidance will be a nightmare to
implement. And at one point or another you need to keep all directories
in the "closure" to be locked, which means, that their inodes have to be
in memory, which means that if file-size / mem-size ratio is large, you
are in trouble.

Look at this from another point of view: deletion of a file when its
last name is removed is but a form of the garbage
collection. Traditionally UNIX implemented reference counted garbage
collectors for this, which was fine, because UNIX file system is a tree
(modulo dots and dotdots). But if multiple hard-links to the directories
are allowed, file system is an arbitrary graph. It makes little sense to
try to stretch reference counting to work in this situation. There are
much more efficient forms of garbage collection for this.

Nikita.

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Hans Reiser]

Nikita Danilov wrote:

> But if multiple hard-links to the directories
>are allowed, file system is an arbitrary graph. It makes little sense to
>try to stretch reference counting to work in this situation. There are
>much more efficient forms of garbage collection for this.
>
>Nikita.
>
>
>  
>
Educate us, I am curious.

-- 
Hans

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 
 > > But if multiple hard-links to the directories
 > >are allowed, file system is an arbitrary graph. It makes little sense to
 > >try to stretch reference counting to work in this situation. There are
 > >much more efficient forms of garbage collection for this.
 > >
 > >Nikita.
 > >
 > >
 > >  
 > >
 > Educate us, I am curious.

http://citeseer.nj.nec.com/jones03garbage.html is a recent bibliography
in this subject, containing more than 20000 references. I am afraid this
is little too much for a brief description. But basically, instead of
reclaiming garbage when "last reference" goes out, collectors typically
run periodically trying to find unreachable objects. Common way to do so
is to find -reachable- ones, and then to declare the rest
unreachable. Also, do avoid scanning the whole heap^Wfile system,
objects are segregated into "generations", and allocation is tuned so
that there are no references from object older generation to younger
one. Some smart techniques (like colored pointers) are used to avoid
garbage collection to proceed concurrently with mutators. And so
on. Large field for an experimentation.

 > 
 > -- 
 > Hans
 > 

Nikita.

 > 

Garbage collection for files (was Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far)))

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I will step aside and let the smart people figure this out.  Only one 
real suggestion -- most directories will not have loops.  I'm not sure 
how, but I'd imagine it's possible to use refcounts there and other 
forms of garbage collection elsewhere.  I felt my refcount-based idea 
was clean and cool-looking, but I have had approximately 10-15 mins of 
experience in this subject.

Only one question -- does it work for rm -rf?

Thread below:

Nikita Danilov wrote:

>Hans Reiser writes:
> > Nikita Danilov wrote:
> > 
> > > But if multiple hard-links to the directories
> > >are allowed, file system is an arbitrary graph. It makes little sense to
> > >try to stretch reference counting to work in this situation. There are
> > >much more efficient forms of garbage collection for this.
> > >
> > >Nikita.
> > >
> > >
> > >  
> > >
> > Educate us, I am curious.
>
>http://citeseer.nj.nec.com/jones03garbage.html is a recent bibliography
>in this subject, containing more than 20000 references. I am afraid this
>is little too much for a brief description. But basically, instead of
>reclaiming garbage when "last reference" goes out, collectors typically
>run periodically trying to find unreachable objects. Common way to do so
>is to find -reachable- ones, and then to declare the rest
>unreachable. Also, do avoid scanning the whole heap^Wfile system,
>objects are segregated into "generations", and allocation is tuned so
>that there are no references from object older generation to younger
>one. Some smart techniques (like colored pointers) are used to avoid
>garbage collection to proceed concurrently with mutators. And so
>on. Large field for an experimentation.
>
> > 
> > -- 
> > Hans
> > 
>
>Nikita.
>
> > 
>  
>

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9E1+QisZLIF6uqOAQJ4VQ/+MZizcUlDJ2Zbb7zteUP8upK3f9a2Z/qY
hEtWrTAqP2/zpLf68GR0OkfPnwe7g3h9NYvCWSJ5j3tt3lr7TS/PcPoMWn8cv+7j
wQoJRwI6rVb7FkCuOu/s8qSMG47dww/3Ao1jc2eFngT16ghIgDPEBBL0DwwqSbsG
NrNevs1yDqO+qRlsoHbe6oqz+55BEtwc/APnBKHzXj8dDNevv5PEx7WwH12A+tRe
etMxXygc7OTqk7miH00OTH5+afnJEgF0JiWwMDUsDiYTT54Cl4a9E+nUzm0kp5SI
R0VYHgNkxTQ692SPE2Zdl+xSVaPq9N9byfDZdTO4jfSp9zx3PA6HfG+Lc30o0IAZ
VGKfV3+wV0jnIE/Ek3pFDv1Xn1gAtM3htfIvFZSvysEC4Km0UkvIOgsQyZByh3fq
Dzop9pjmM/CMLfu/Z+4APQ4j8Mt3nCSvEPnTgrp8D7ptpkq/1nhCzS1Ej42+XQZ+
h/7pyE15wrC8EyVUgVQMDZa1n6R1k+w/VvsNhMqfuTaTfXJBWaiq0Chj6QMfX6nx
kBRjzudFkh5Sg2Kf7vwwv+aH/wd0F953BnFq6Ceg2FAV6Mt308aHI9JnBSP08uIz
my/p1fLxhwyEZ1MzBJC97rQivRa2WUXrDwnC6duHbTg8A913tiW5lXIvEIooarOR
IFSQ2vdEzio=
=X4Fs
-----END PGP SIGNATURE-----

Re: Garbage collection for files (was Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far)))

[Hans Reiser]

The approaches nikita suggests don't seem very efficient for the no loop 
case. 

Hans

David Masover wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> I will step aside and let the smart people figure this out.  Only one 
> real suggestion -- most directories will not have loops.  I'm not sure 
> how, but I'd imagine it's possible to use refcounts there and other 
> forms of garbage collection elsewhere.  I felt my refcount-based idea 
> was clean and cool-looking, but I have had approximately 10-15 mins of 
> experience in this subject.
>
> Only one question -- does it work for rm -rf?
>
> Thread below:
>
> Nikita Danilov wrote:
>
>> Hans Reiser writes:
>> > Nikita Danilov wrote:
>> > > > But if multiple hard-links to the directories
>> > >are allowed, file system is an arbitrary graph. It makes little 
>> sense to
>> > >try to stretch reference counting to work in this situation. There 
>> are
>> > >much more efficient forms of garbage collection for this.
>> > >
>> > >Nikita.
>> > >
>> > >
>> > >  > >
>> > Educate us, I am curious.
>>
>> http://citeseer.nj.nec.com/jones03garbage.html is a recent bibliography
>> in this subject, containing more than 20000 references. I am afraid this
>> is little too much for a brief description. But basically, instead of
>> reclaiming garbage when "last reference" goes out, collectors typically
>> run periodically trying to find unreachable objects. Common way to do so
>> is to find -reachable- ones, and then to declare the rest
>> unreachable. Also, do avoid scanning the whole heap^Wfile system,
>> objects are segregated into "generations", and allocation is tuned so
>> that there are no references from object older generation to younger
>> one. Some smart techniques (like colored pointers) are used to avoid
>> garbage collection to proceed concurrently with mutators. And so
>> on. Large field for an experimentation.
>>
>> > > -- > Hans
>> >
>> Nikita.
>>
>> >  
>>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.3 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
>
> iQIVAwUBP9E1+QisZLIF6uqOAQJ4VQ/+MZizcUlDJ2Zbb7zteUP8upK3f9a2Z/qY
> hEtWrTAqP2/zpLf68GR0OkfPnwe7g3h9NYvCWSJ5j3tt3lr7TS/PcPoMWn8cv+7j
> wQoJRwI6rVb7FkCuOu/s8qSMG47dww/3Ao1jc2eFngT16ghIgDPEBBL0DwwqSbsG
> NrNevs1yDqO+qRlsoHbe6oqz+55BEtwc/APnBKHzXj8dDNevv5PEx7WwH12A+tRe
> etMxXygc7OTqk7miH00OTH5+afnJEgF0JiWwMDUsDiYTT54Cl4a9E+nUzm0kp5SI
> R0VYHgNkxTQ692SPE2Zdl+xSVaPq9N9byfDZdTO4jfSp9zx3PA6HfG+Lc30o0IAZ
> VGKfV3+wV0jnIE/Ek3pFDv1Xn1gAtM3htfIvFZSvysEC4Km0UkvIOgsQyZByh3fq
> Dzop9pjmM/CMLfu/Z+4APQ4j8Mt3nCSvEPnTgrp8D7ptpkq/1nhCzS1Ej42+XQZ+
> h/7pyE15wrC8EyVUgVQMDZa1n6R1k+w/VvsNhMqfuTaTfXJBWaiq0Chj6QMfX6nx
> kBRjzudFkh5Sg2Kf7vwwv+aH/wd0F953BnFq6Ceg2FAV6Mt308aHI9JnBSP08uIz
> my/p1fLxhwyEZ1MzBJC97rQivRa2WUXrDwnC6duHbTg8A913tiW5lXIvEIooarOR
> IFSQ2vdEzio=
> =X4Fs
> -----END PGP SIGNATURE-----
>
>
>


-- 
Hans

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Stewart Smith]

[-- Attachment #1: Type: text/plain, Size: 1306 bytes --]

On Sat, 2003-12-06 at 03:18, Nikita Danilov wrote:
> http://citeseer.nj.nec.com/jones03garbage.html is a recent bibliography
> in this subject, containing more than 20000 references. I am afraid this
> is little too much for a brief description. But basically, instead of
> reclaiming garbage when "last reference" goes out, collectors typically
> run periodically trying to find unreachable objects. Common way to do so
> is to find -reachable- ones, and then to declare the rest
> unreachable. Also, do avoid scanning the whole heap^Wfile system,

What also may be interesting is the Charging in a Secure Environment
paper,

http://www.csse.monash.edu.au/courseware/cse4333/rdp-material/Bremen-paper.pdf

The way it works (as sort-of implemented on Walnut) is that objects are
charged for resources, and when they can no longer pay, they are
deleted.

Periodically you run through all the objects deducting rent, and
removing any objects that can no longer pay for themselves.

To adapt it to a hierarchical filesystem, you could have the root
directory being the thing that generates money, and then gives it to all
the objects in it (which then give it to their sub objects) etc.

just a thought.
-- 
Stewart Smith (stewart@flamingspork.com)
http://www.flamingspork.com/


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Alexander G. M. Smith]

Nikita Danilov wrote on Fri, 5 Dec 2003 17:17:48 +0300:
> Deadlock detection and/or avoidance will be a nightmare to
> implement. And at one point or another you need to keep all directories
> in the "closure" to be locked, which means, that their inodes have to be
> in memory, which means that if file-size / mem-size ratio is large, you
> are in trouble.

It was a bit of a pain.  If it runs out of in-memory inodes (the BeOS
equivalent), it just fails with a suitable error code (and log
message).  Then it's up to the user to delete things in smaller chunks.
Same sort of thing for deadlocks - after a while it times out and
the user has to do smaller deletions.

Incidentally, renaming files is a superset of the delete operation.
Implement that first, and you have delete for free.

- Alex

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


>Incidentally, renaming files is a superset of the delete operation.
>Implement that first, and you have delete for free.
>  
>
If only it was that easy.  When you rename a folder, you don't usually 
do a recursive rename of everything inside.  When you delete a folder -- 
you don't, unless it's "rmdir".  You recursively delete each file 
inside, and it's that kind of operation that creates problems for hard 
links to directories.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9E20wisZLIF6uqOAQIgfQ/7BxHQKiew/8jCIlo7lKKLHBxBS1mnZBLH
rNaJFnqZLCHJOsPSphx2YcVxRq4IhSCK4lKdT5IMzbwwY9nQOyp7ans8IgzTLMbh
edryWJXWjOPI4Q6ZS9j7mPnDa5452CBc5/ioMJIuuqScH/MMoeK5hWOqED/snzl/
NzjIvxWgaXv8FWZsN/0WG/GPFwlp0KdaM2uPZSfozMZ8r/j4lc5kYsYhL/frDKH8
kkU6I6AvEQTsdAT+NP+idfwYFMkYelfd8y0jkLzq3kN+XiogC7fdaC0U+biKTxpg
rSfs608okneLNGPe0zunmBD/xuZAEbvf6d+ILML4F7VBpG+4j5Wt/KAneJJyCM3H
QQggG4O36pPeHOHvjJ446njjTYq1KBBO5oZdJvmWp5vUiQW+zA88uHUsn7KjTKkY
5/czmukczBIK7wcYjHPKGUsOLUmWKp331eybeOMTM6UsMiWORfQwKsQitKG+pkFf
j01ClbPUICH6tRhZ08JJZsO11hsAIjDaWVib2a+fHBuiXpEgoKaLIuqrxHfgJaUr
1PUlcKn8G9RNYXZkO+ZH7bts9WhkGcukFLMotiuRrLY+W70QhnA/DLay0siadhu9
3DTiRWCx2+y8MswlKyMa1N9hYI0EPtPqNIAnbtl9cG+Cx/PiIWoN1lzZDr6NJ0P6
GotO6gSuOMQ=
=7FN9
-----END PGP SIGNATURE-----

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Alexander G. M. Smith]

David Masover wrote on Fri, 05 Dec 2003 19:54:26 -0600:
> >Incidentally, renaming files is a superset of the delete operation.
> >Implement that first, and you have delete for free.
> >  
> >
> If only it was that easy.  When you rename a folder, you don't usually 
> do a recursive rename of everything inside.  When you delete a folder -- 
> you don't, unless it's "rmdir".  You recursively delete each file 
> inside, and it's that kind of operation that creates problems for hard 
> links to directories.

Actually, renaming also is recursive.  The obvious case is when you
rename a file over an existing file (or directory), which implies the
whole deletion mechanism is needed.

Also, at least for the AGMS link mechanism (or perhaps for any system
that allows multiple parent directories), it has to do a graph traversal
to find out which parent links are still valid, and which need to be
adjusted if their parent has been changed.  If you see the example at
the bottom of http://web.ncf.ca/au829/BeOS/AGMSLinkExplanation.txt you'll
notice that deleting something can cause a sweeping re-evaluation of who
is the parent to what.  The same thing would happen if you rename (move)
a parent directory to be a subdirectory of one of its children.

- Alex

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:

>David Masover wrote on Fri, 05 Dec 2003 19:54:26 -0600:
>  
>
>>>Incidentally, renaming files is a superset of the delete operation.
>>>Implement that first, and you have delete for free.
>>> 
>>>
>>>      
>>>
>>If only it was that easy.  When you rename a folder, you don't usually 
>>do a recursive rename of everything inside.  When you delete a folder -- 
>>you don't, unless it's "rmdir".  You recursively delete each file 
>>inside, and it's that kind of operation that creates problems for hard 
>>links to directories.
>>    
>>
>
>Actually, renaming also is recursive.  The obvious case is when you
>rename a file over an existing file (or directory), which implies the
>whole deletion mechanism is needed.
>  
>
Most of the time, rename over an existing file needs only a single 
deletion, and renaming over a directory is (last I checked) impossible 
with standard tools without removing or renaming the directory first.  
If I do
touch a
mkdir b
mv a b
It will move 'a' to a file inside of 'b', rather than removing 'b'.  
This isn't necessarily the best way, but it's how it's done now -- to 
get the effect you're implying, I'd have to do
rm -rf b
mv a b

>Also, at least for the AGMS link mechanism (or perhaps for any system
>that allows multiple parent directories), it has to do a graph traversal
>to find out which parent links are still valid, and which need to be
>adjusted if their parent has been changed.  If you see the example at
>the bottom of http://web.ncf.ca/au829/BeOS/AGMSLinkExplanation.txt you'll
>notice that deleting something can cause a sweeping re-evaluation of who
>is the parent to what.  The same thing would happen if you rename (move)
>a parent directory to be a subdirectory of one of its children.
>
>  
>
Can we do that?  Last I checked, the file utilities (or maybe the fs 
itself) won't let you move a directory to a child of itself, or a child 
of a child of itself.  If this "graph traversal" is what I think it is, 
it belongs in fsck, not normal fs operation.  Of course, I don't have a 
better solution, aside from either stretching the limits of reference 
counts or dropping directory hardlinks altogether.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9J9pwisZLIF6uqOAQKZhxAAnMQNtl9MMwVNNhbDKIBpT9PDKqwb1b0C
NfrgOBMyxlosTWT0OVrbS5deFoTefkpk6inzlcck9nCLDCB5CnOfKBlnA1MbWaPN
jHsmC5Cz0mVOk61hWUvWDWZ1DLrl4yb313KF/5XwTGyQ/0NZWTHEAgHf4FjX3wRq
ZkLubWGt7fdsRv+FWo1s5ah2sQPzPj/3nynECFVt1gQaGoNkaxETWj3a5bAjLWwX
9SZLDrhqPxPzKCFvs9OL9PH2MC7nlYa0rOEOKdKBdTsTMdEhEluZyTXVYAMmwlpM
d3UQj6tcg5ehRs7jHhsjKSD4nIzdmK8nKrBXi0j+anw/g8n+bHFbgsVCOwumv8WO
k1AxnGncgcdtkNSafUmNGYgtL5Ra0Fv0b2sIjggyHmZFes5OBr3Kgj93Mznk1z10
siiVA43NhBimHuZsdyZ1U64Sl5uV287iAgKqDPbvQ4Z24uV6X0b5/4vpRcDsm++U
05M1c/CwG9YLqVluzm0ewvq8px0zGrcPbkm6v86PL9UxIbN8aDKKqyaUv8lXHi7y
YiWwFingzlUIkyXV2yfdzRbXthdhXo17gX0NJBaiyS9jYl40DecuFH5w3bFxLhnj
ys9IOrM0Q2hSTzyD2GFyR8FoJQIypc95YwkeOBZbUjskyJy7xE2bX+269US9ursz
BA6EToDsPwg=
=aMts
-----END PGP SIGNATURE-----

Re: More on Hard Links (was A bold idea (Re: Carrying Attributes too Far))

[Alexander G. M. Smith]

David Masover wrote on Sat, 06 Dec 2003 19:08:51 -0600:
> Most of the time, rename over an existing file needs only a single 
> deletion, and renaming over a directory is (last I checked) impossible 
> with standard tools without removing or renaming the directory first.  

Correct.  But that's just a convention.  Perhaps it could be changed
to distinguish between  move a b/  and  mv a b  where one would move
"a" into "b" and the other would overwrite "b".

> Can we do that?  Last I checked, the file utilities (or maybe the fs 
> itself) won't let you move a directory to a child of itself, or a child 
> of a child of itself.

Actually, if you try that with the BeOS file system I was talking
about, it will detect the isolation of a large part of the name
space (by doing the graph traversal) and reject the operation.  It
could just as easily allow it (it would delete the isolated files and directories), but that would be a bad design choice for normal users.

- Alex

Re: More on Hard Links

[Narcoleptic Electron]

I've read through all the emails on the subject, and
here are my thoughts on how links should act, with a
very rough outline of how they could be implemented. 
I've also covered symlinks for good measure.  I'm not
an expert, and this is long, so please bear with me...

Note: I use the term "file" to refer to the underlying
file object (think 'inode'), and "directory entry" to
refer to the user-level reference to the underlying
file.

A. Symlinks:
============

Symlinks should be able to specify a file in any/all
of the following ways:

1. By location (i.e. path).  This is the way it works
now.
2. By file (i.e. volume id and file id).  File id is a
unique identifier for the file on the volume (eg.
inode number).  Not sure if there is a way to uniquely
identify a volume reliably...

In addition, the symlink file should specify the
look-up order (i.e. file first, then location, or vice
versa).

B. Hard links:
==============

About hard linking directories: I think it should be
allowed and possible.

About circularity: It should be possible to prevent
circularity at link creation time (as per Alex's
explanation of his BeOS system), and I think this
should happen.  I don't know of a case where the
utility of a circular link would outweigh the
confusion and mess it would create (including gc
requirements).

As far as types of hard links, I think they can be
boiled down to 3 basic use cases:

1. Linking to a file on the same ReiserX volume.
------------------------------------------------

Nothing new here: these can be implemented as regular
hard-links (i.e. pointers to an underlying
reference-counted file).

2. Linking to a file on a different ReiserX volume.
---------------------------------------------------

I think that there is a way that this can be done so
that hard link semantics are preserved (i.e. each
directory entry equally represents the file, at all
times -- even after unmounting of one of the volumes).

My idea: Each volume can contain a "synchronized" copy
of the underlying file.

Details:
- All underlying files (again, think 'inode' level)
contain a "sync list".  This contains the volume ID
and file ID of each file that it is synchronized with.
 If it is not synchronized with any files, the sync
list is empty.
- Synchronization is always two-way.  Thus, if two
files are synchronized, each refers to the other in
its sync list.
- Any time a transaction is to be committed to a
synchronized file, it is committed to all files in the
sync list, or none.  The latter case might occur if
one of the volumes is not mounted, for example, and
will return an error to the caller, possibly resulting
in a user message such as "This file is linked with
another file on an unavailable volume, and cannot be
modified until the volume becomes available.".
- When a file is copied, its sync list is not copied.
- For a file to be deleted, it must have a reference
count of 0 (no references to it on its own volume) and
not be synchronized with any other files (no items in
its sync list, meaning that no references to it exist
in other sync lists).  The sync list can only be
cleared once all the files in the sync list can be
resolved (i.e. all relevant volumes are mounted).  If
this resolution cannot be done right away, the file is
replaced with a "NULL" placeholder (leaving the sync
list unchanged) to save space (and reserve the File
ID) until the file can be deleted (i.e. next
transaction commit).

The principle of least privilege applies: if a file is
on a non-writable volume, no files synchronized with
it are modifiable.  Transaction commits are all or
nothing.

When creating a link on Volume A to a file on Volume B
(another ReiserX volume), the following would happen:
(i) If the target file doesn't already have a
non-"NULL" file synchronized to it on Volume A, a copy
of the underlying file on Volume B is created on
Volume A and they are synchronized.
(ii) A directory entry is created at the new link
location (on Volume A), which refers to the
synchronized file on Volume A.

If Volume B becomes unmounted, no transactions can be
committed to the synchronized file on Volume A until
Volume B is remounted.  However, the link can be
deleted (decrementing the reference count on the
synchronized Volume A file), or the file can be copied
and modified (note that the sync list is *not* copied
with the file).

If the link on Volume A is moved to Volume B, the
following happens:
(i) The original directory entry on Volume A is
deleted (decrementing the reference count on the
underlying synchronized file accordingly).
(ii) A new directory entry, referring to the
synchronized file on Volume B, is created at the drop
location on Volume B.

Because the "sync list" uses volume id and file id,
these references won't break if the links are moved
around.

3. Linking to a file on a different non-ReiserX
volume.
-------------------------------------------------------

Not possible (because "file synchronization"
management code would be unique to ReiserX).  User
either has to use a symlink or make a copy.



______________________________________________________________________ 
Post your free ad now! http://personals.yahoo.ca

Re: More on Hard Links

[Hubert Chan]

>>>>> "Narcoleptic" == Narcoleptic Electron <narcoleptic_electron@yahoo.co.uk> writes:

[...]

Narcoleptic> A. Symlinks:
Narcoleptic> ============

[...]

Narcoleptic> 2. By file (i.e. volume id and file id).  File id is a
Narcoleptic> unique identifier for the file on the volume (eg.  inode
Narcoleptic> number).  Not sure if there is a way to uniquely identify a
Narcoleptic> volume reliably...

I don't think it's a good idea to classify these as symlinks.  For one
thing, it's called a "symbolic" link, because the link contains a
symbol: a path name.  Also, it works differently from regular
symlinks.  IMHO, it would be better to give it another name.  (This is
what I would call a "firmlink", and I think what Leo calls a "weak
link".)

[...]

Narcoleptic> 2. Linking to a file on a different ReiserX
Narcoleptic> volume.
Narcoleptic> ---------------------------------------------------

I believe the plan is to have Reiser5 be a distributed filesystem,
which will basically (AFAICT) solve all our problems and eliminates the
need for cross-ReiserFSx hard links.  IMHO this is the better way to do
things, since a distributed filesystem will already take into account a
lot of problems that we have.

Not to say that you (or Leo) shouldn't pursue this idea.  You may not
want to use the distributed stuff, or you may want to implement it as a
proof of concept to convince other FS maintainers to add it to their
FS, or etc.

[...]

Narcoleptic> My idea: Each volume can contain a "synchronized" copy of
Narcoleptic> the underlying file.

My first reaction: "Bwaa!  Wasted space!  Consistency problems!"

Wasted space, you're of course aware of.  But then, the number of
hardlinks is probably not that large, so we won't have that many
duplicates.

Consistency problems you've dealt with, sort of.  I'm a bit wary of
preventing the file from being changed if not all the involved volumes
are mounted.  Also, if one volume dies and needs to be reformatted, or
removed, you need to have some way of recovering.  It's hard to
differentiate that from the volume just being unmounted.

[...]

Narcoleptic> The principle of least privilege applies: if a file is on a
Narcoleptic> non-writable volume, no files synchronized with it are
Narcoleptic> modifiable.

So then you have the case where you have a file that's on a read-write
partition, has all the bits necessary for writing, but is read-only.
I'm not convinced this is the best thing to do.  But if you must have
cross-filesystem hardlinks, I think this is what you must do.

Really, it's a compromise.  I look at this situation and say: this is
why we can't have cross-filesystem hardlinks.  Others may look at it
and say: I really want cross-filesystem hardlinks, and so this is what
I will do, which is reasonable, and doesn't completely suck.

[...]

Narcoleptic> 3. Linking to a file on a different non-ReiserX
Narcoleptic> volume.
Narcoleptic> -------------------------------------------------------

Narcoleptic> Not possible (because "file synchronization" management
Narcoleptic> code would be unique to ReiserX).  User either has to use a
Narcoleptic> symlink or make a copy.

Well, if you implement it on ReiserFSx in a reasonably portable way,
you may be able to convince other FS maintainers to use it (if they
aren't completely opposed to the idea of cross-filesystem hardlinks).

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: More on Hard Links

[Narcoleptic Electron]

Hubert Chan wrote: 

> "Narcoleptic" == Narcoleptic Electron writes:
> 
> [...]
> 
> Narcoleptic> A. Symlinks:
> Narcoleptic> ============
> 
> [...]
> 
> Narcoleptic> 2. By file (i.e. volume id and file
> id).  File id is a
> Narcoleptic> unique identifier for the file on the
> volume (eg.  inode
> Narcoleptic> number).  Not sure if there is a way to
> uniquely identify a
> Narcoleptic> volume reliably...
> 
> I don't think it's a good idea to classify these as
> symlinks.  For one
> thing, it's called a "symbolic" link, because the
> link contains a
> symbol: a path name.  Also, it works differently
> from regular
> symlinks.  IMHO, it would be better to give it
> another name.  (This is
> what I would call a "firmlink", and I think what Leo
> calls a "weak
> link".)

I see symbolic links as one-way links (there is a
link, and a target), in which the link indicates the
file in a symbolic way, and the target remains
unaffected.  As such, my links qualify, as the volume
ID/file ID combo is just as much a symbol indicating
the file as the path is.  My link simply represents a
superset of current symbolic link functionality.  You
can call it a firm link if you like (I've called it
that on another list!). 

> [...]
> 
> Narcoleptic> 2. Linking to a file on a different
> ReiserX
> Narcoleptic> volume.
> Narcoleptic>
> ---------------------------------------------------
> 
> I believe the plan is to have Reiser5 be a
> distributed filesystem,
> which will basically (AFAICT) solve all our problems
> and eliminates the
> need for cross-ReiserFSx hard links.  IMHO this is
> the better way to do
> things, since a distributed filesystem will already
> take into account a
> lot of problems that we have.
> 
> Not to say that you (or Leo) shouldn't pursue this
> idea.  You may not
> want to use the distributed stuff, or you may want
> to implement it as a
> proof of concept to convince other FS maintainers to
> add it to their
> FS, or etc.

I don't know much about Reiser5's distributed fs
plans... more reading is required for me here.
 
> [...]
> 
> Narcoleptic> My idea: Each volume can contain a
> "synchronized" copy of
> Narcoleptic> the underlying file.
> 
> My first reaction: "Bwaa!  Wasted space! 
> Consistency problems!"
> 
> Wasted space, you're of course aware of.
> 
> [...]

No, I'm not aware of any wasted space in my plan. 
Each volume must have its own copy of the file so that
it is available when the other volume dismounts.

> Consistency problems you've dealt with, sort of. 

What cases have I missed?

> I'm a bit wary of
> preventing the file from being changed if not all
> the involved volumes
> are mounted.  

It is the only way to ensure consistency.

> Also, if one volume dies and needs to
> be reformatted, or
> removed, you need to have some way of recovering. 

You can copy the hard link, then use it as a regular
local file (the sync list doesn't get copied with the
file).  You can also delete the hard link if you want
(thus replacing the actual file with that "NULL"
placeholder if its reference count is 0).

This brings me to another point: under this proposal,
all files would essentially have one of 3 states:
"online", "offline" (meaning that it is hard linked to
unavailable files), or "unknown" (not yet checked). 
Each file could have a pseudo-file attribute (i.e.
"+/Online") derived from the resolvability of the sync
list.  Perhaps a file manager would show offline files
with some sort of icon badge (i.e. a little
ghostbusters-style circle with crossbar).

So when an offline file is copied, the copy is online
(because the sync list doesn't go with it).

> It's hard to
> differentiate that from the volume just being
> unmounted.

Not sure what you mean.

> [...]
> 
> Narcoleptic> The principle of least privilege
> applies: if a file is on a
> Narcoleptic> non-writable volume, no files
> synchronized with it are
> Narcoleptic> modifiable.
> 
> So then you have the case where you have a file
> that's on a read-write
> partition, has all the bits necessary for writing,
> but is read-only.
> I'm not convinced this is the best thing to do.  But
> if you must have
> cross-filesystem hardlinks, I think this is what you
> must do.
> Really, it's a compromise.  I look at this situation
> and say: this is
> why we can't have cross-filesystem hardlinks. 
> Others may look at it
> and say: I really want cross-filesystem hardlinks,
> and so this is what
> I will do, which is reasonable, and doesn't
> completely suck.

Shouldn't all files on a read-only partition
automatically have their permission bits set to
read-only (or at least appear that way)?  If so, then
all files will have the permission bits set
identically, and no problem.

> [...]
> 
> Narcoleptic> 3. Linking to a file on a different
> non-ReiserX
> Narcoleptic> volume.
> Narcoleptic>
>
-------------------------------------------------------
> 
> Narcoleptic> Not possible (because "file
> synchronization" management
> Narcoleptic> code would be unique to ReiserX).  User
> either has to use a
> Narcoleptic> symlink or make a copy.
> 
> Well, if you implement it on ReiserFSx in a
> reasonably portable way,
> you may be able to convince other FS maintainers to
> use it (if they
> aren't completely opposed to the idea of
> cross-filesystem hardlinks).

True.


______________________________________________________________________ 
Post your free ad now! http://personals.yahoo.ca

Re: More on Hard Links

[Hubert Chan]

>>>>> "Narcoleptic" == Narcoleptic Electron <narcoleptic_electron@yahoo.co.uk> writes:

[...]

Narcoleptic> I don't know much about Reiser5's distributed fs
Narcoleptic> plans... more reading is required for me here.

Me neither.  I've only heard Hans mention it a couple of times.  I
don't think there are many details available (and there probably won't
be until Reiser4 is ready).  But I assume it would provide something
similar to what you get with Coda, or AFS, etc.  Which I also don't
know a whole lot about, but some of them contain provisions for
maintaining local copies of some files, and synchronizing them if
something goes offline.

[...]

>> Wasted space, you're of course aware of.
>> 
>> [...]

Narcoleptic> No, I'm not aware of any wasted space in my plan.  Each
Narcoleptic> volume must have its own copy of the file so that it is
Narcoleptic> available when the other volume dismounts.

Well, I guess you wouldn't call it "wasted", since it is needed, but
basically, you have to have multiple copies of the same file, instead
of just the one copy with normal hard links.

>> Consistency problems you've dealt with, sort of.

Narcoleptic> What cases have I missed?

Well, I'm not completely satisfied with the condition that you can't
write to a file if the involved volumes are not mounted.  As well as
the problem of a volume dying/being reformatted.

>> I'm a bit wary of preventing the file from being changed if not all
>> the involved volumes are mounted.

Narcoleptic> It is the only way to ensure consistency.

Yes.  Which is why I made the claim a while ago that there is no "good"
way to implement cross-filesystem hardlinks.

If you *must* have cross-filesystem hardlinks, I think that your scheme
is how it must be done, in order to get hardlink semantics as much as
possible.  I just don't like some of the implications, which is why I
don't like cross-filesystem hardlinks.

>> Also, if one volume dies and needs to be reformatted, or removed, you
>> need to have some way of recovering.

Narcoleptic> You can copy the hard link, then use it as a regular local
Narcoleptic> file (the sync list doesn't get copied with the file).  You
Narcoleptic> can also delete the hard link if you want (thus replacing
Narcoleptic> the actual file with that "NULL" placeholder if its
Narcoleptic> reference count is 0).

I think that scheme may result in space wastage (whoah.  I fully
expected my spellchecker to complain about that word.) since you have
to maintain the NULL placeholder, and can't reclaim the space.

But my concern with that was that if one partition needs to be
reformatted, then the other instances of the link become unwritable.
So if you want to recover, you would have to copy the file and manually
re-link everything.  There's no way to automatically recover.

[...]

Narcoleptic> ... Each file could have a pseudo-file attribute (i.e.
Narcoleptic> "+/Online") ...
              ^^^

Oh great.  Are we going to get into the debate on how to name
attributes again? ;-)

[...]

>> It's hard to differentiate that from the volume just being unmounted.

Narcoleptic> Not sure what you mean.

Just what I wrote above.

[...]

Narcoleptic> Shouldn't all files on a read-only partition automatically
Narcoleptic> have their permission bits set to read-only (or at least
Narcoleptic> appear that way)?

Hmm.  I don't think so.  I'm not sure.  I don't have any r-o partitions
right now, so I can't check.

Narcoleptic> If so, then all files will have the permission bits set
Narcoleptic> identically, and no problem.

But then you run into the problem that you can't change the permissions
on a file that's on a read-write partition, which would be unexpected
behaviour, IMHO.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: More on Hard Links

[Narcoleptic Electron]

Hubert Chan wrote: 

> "Narcoleptic" == Narcoleptic Electron writes:
> 
> Narcoleptic> You can copy the hard link, then use it
> as a regular local
> Narcoleptic> file (the sync list doesn't get copied
> with the file).  You
> Narcoleptic> can also delete the hard link if you
> want (thus replacing
> Narcoleptic> the actual file with that "NULL"
> placeholder if its
> Narcoleptic> reference count is 0).
> 
> I think that scheme may result in space wastage
> since you have
> to maintain the NULL placeholder, and can't reclaim
> the space.

If there is a way to quickly and reliably verify that
two files are identical (or not), then when committing
a transaction, each file in the sync list can be
resolved as follows:
- If a volume ID is not found, don't commit anything
(as per the current proposal).
- Otherwise, if a file ID for a given volume ID refers
to a file not identical to the current file (or is not
found), remove the file from the sync list.

This way, we don't need the "NULL" placeholder...
simply delete the file when its reference count
reaches 0, as usual.  This removes space reclamation
issues if a volume goes down.

This is a far better solution, because there is no
need to worry about global uniqueness of volume IDs or
file ID reuse within volumes.

> But my concern with that was that if one partition
> needs to be
> reformatted, then the other instances of the link
> become unwritable.
> So if you want to recover, you would have to copy
> the file and manually
> re-link everything.  There's no way to automatically
> recover.

It would be pretty easy to write a script to do this. 
This script would be pretty useful, in fact... good
thinking.

> Narcoleptic> ... Each file could have a pseudo-file
> attribute (i.e.
> Narcoleptic> "+/Online") ...
>               ^^^
> 
> Oh great.  Are we going to get into the debate on
> how to name
> attributes again? ;-)

Well, although the subject line has been changed many
times, I believe we are still on that thread!  ;-)

> >> It's hard to differentiate that from the volume
> just being unmounted.

There is no need to differentiate from a volume going
down and just being unmounted.

> Narcoleptic> Shouldn't all files on a read-only
> partition automatically
> Narcoleptic> have their permission bits set to
> read-only (or at least
> Narcoleptic> appear that way)?
> 
> Hmm.  I don't think so.  I'm not sure.  I don't have
> any r-o partitions
> right now, so I can't check.

If it doesn't work this way, I think it should.

> Narcoleptic> If so, then all files will have the
> permission bits set
> Narcoleptic> identically, and no problem.
> 
> But then you run into the problem that you can't
> change the permissions
> on a file that's on a read-write partition, which
> would be unexpected
> behaviour, IMHO.

It wouldn't be expected for offline files, which would
be clearly denoted as such.


______________________________________________________________________ 
Post your free ad now! http://personals.yahoo.ca

Re: More on Hard Links

[Narcoleptic Electron]

Had a few more thoughts...

Narcoleptic Electron wrote: 

> This is a far better solution, because there is no
> need to worry about global uniqueness of volume IDs
> or file ID reuse within volumes.

A revision: note that global uniqueness of volume IDs
is still ideal.  Does ReiserFS provide globally unique
IDs for its volumes?

Hubert Chan wrote:

> I'm not completely satisfied with the condition 
> that you can't write to a file if the involved 
> volumes are not mounted.

Think of the non-writable, offline file as a "cached"
version of the file, and you can make a copy for
modification if you wish.  I don't see a usability
issue here.


______________________________________________________________________ 
Post your free ad now! http://personals.yahoo.ca

Re: More on Hard Links

[Hubert Chan]

>>>>> "Narcoleptic" == Narcoleptic Electron <narcoleptic_electron@yahoo.co.uk> writes:

[...]

Narcoleptic> Think of the non-writable, offline file as a "cached"
Narcoleptic> version of the file, and you can make a copy for
Narcoleptic> modification if you wish.  I don't see a usability issue
Narcoleptic> here.

Here's a use case: there's a shared file, "foo.txt", that I am working
with collaboratively with a bunch of other people.  It resides on a
network volume.  I decide I want a local copy on my laptop, so I make a
hardlink on my local disk.  Now I take my laptop home, so it's no
longer connected to the network.  The other guys want to make some
changes, but since the file has a remote link that isn't mounted, it's
unwritable.  The guys are annoyed at having to copy the file, leaving
"foo.txt" hanging there.  (They're even more annoyed when they find out
that I had mass-hardlinked the whole directory.)  They track me down
and beat me up. :-P

It's even worse if "foo.txt" was, say, actually a configuration file in
/etc, so you can't just copy it and make changes, because the program
won't be able to find the new version.

It's a bit of a contrived example, but not *completely* ridiculous.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: More on Hard Links

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 3025 bytes --]

Sorry for the very long delay in responding; I have
been detained.

Hubert Chan wrote:

> Here's a use case: there's a shared file, "foo.txt",
> that I am working
> with collaboratively with a bunch of other people. 
> It resides on a
> network volume.  I decide I want a local copy on my
> laptop, so I make a
> hardlink on my local disk.  Now I take my laptop
> home, so it's no
> longer connected to the network.  The other guys
> want to make some
> changes, but since the file has a remote link that
> isn't mounted, it's
> unwritable.  The guys are annoyed at having to copy
> the file, leaving
> "foo.txt" hanging there.  (They're even more annoyed
> when they find out
> that I had mass-hardlinked the whole directory.) 
> They track me down
> and beat me up. :-P
> 
> It's even worse if "foo.txt" was, say, actually a
> configuration file in
> /etc, so you can't just copy it and make changes,
> because the program
> won't be able to find the new version.
> 
> It's a bit of a contrived example, but not
> *completely* ridiculous.

It's not ridiculous at all... you have a very good
point.

I've given this a lot of thought, and firstly have
come to believe that it is misguided to discuss user
features that should be added to Hard Links.  Hard
Links are a low-level feature of the underlying file
system, and should be treated as such. 

Instead, in a discussion about user features, we
should first identify the features that we want, then
design a higher-level construct (which may make use of
hard links in its implementation, possibly directly)
that delivers these features.  

I can think of only 2 user-level linking constructs
that I think would be useful:

(1) "Aliases" (Inspired only partially by Mac OS
aliases.)

- A user can create an alias of any file.
- An alias can be created on any volume whose file
system supports aliases.
- Aliased files look/act exactly like their master
unless they are disconnected 
(i.e. the target is on an unmounted volume).
- Disconnected aliases have some sort of icon badge.
- When an alias is created on another volume, the user
will be asked if the alias should keep a local cache
of the master.  If so, the file can still be read (but
not modified) when disconnected.  If not, then it is
not readable when disconnected.

(2) "Shortcuts" (This is just a name; I am *not*
referring to Windows-style shortcuts.)

- Contain a path of a target file.
- Act like the target file (if resolvable), and the
icon of the file shows some sort of badge (an arrow,
for example).

Shortcuts would be implemented directly using symbolic
links.

Aliases on the same volume could be implemented using
hard links.  However, to implement aliases that span
volumes, I have attached a *very rough* description of
how this could be done, using a hypothetical low-level
construct I have called "cloning".

What do you think?

Cheers,
N. Electron

-------------------

______________________________________________________________________ 
Post your free ad now! http://personals.yahoo.ca

[-- Attachment #2: aliases.txt --]
[-- Type: text/plain, Size: 3015 bytes --]



CLONING

- Pertains to the underlying file structure (think inode), which will henceforth be simply called a "file".
- Any file can be "cloned".
- A clone can be created on any volume whose file system supports cloning.
- When a file is cloned, it becomes the "master".
- Clones are guaranteed to be either identical to the master, a read-only snapshot of the master (i.e. disconnected with cached copy), or not readable (i.e. disconnected with no cache).

How (very roughly):

- Each file can have a master file ID, specifying volume ID and an inode number-type identifier (I'll just call it the inode number).
- Each file can have a "clone list", which is a list of clone files (by file ID) and a bit indicating whether or not the clone is "dirty" (its cache is out of sync).
- When committing a transaction to a file:
	- If it has a master file ID, but the volume ID is unresolvable:
		- The file is currently disconnected from the master.  No transaction is committed.
	- Otherwise (it has no master file ID, or it has a master file ID with a resolvable volume ID):
		- If it has a master file ID:
			- If the master file ID's inode number is unresolvable, or the current file is not in the master file's clone list:
				- This file is no longer a clone.  Clear the master file ID.
			- Otherwise:
				- Commit the transaction to the master.
		- If it has no master file ID:
			- Commit the transaction to this file.
		- If it has a clone list:
			- For each clone file ID:
				- If the volume is unresolvable:
					- Don't commit the transaction to the clone file.
					- Mark this item in the clone list as "dirty".
				- Otherwise, if the clone file ID's inode number is unresolvable:
					- Remove its file ID from the current file's clone list.
				- Otherwise, if the clone file's master file ID is not for the current file:
					- The file is no longer a clone.  Clear its master file ID.
					- Remove its file ID from the current file's clone list.
				- Otherwise:
					- Commit the transaction to the clone file.
					- Mark this item in the clone list as "clean".
- When attempting to read a file:
	- If it has a master file ID with a resolvable volume ID:
		- If the master file ID's inode number is unresolvable, or the current file is not in the master file's clone list:
			- This file is no longer a clone.  Clear the master file ID.
		- Otherwise, if the current file is marked "dirty" in the master file's clone list:
			- Update the file from the master.
			- Update the entry as "clean".
	- Read the file.

Requirements:

- File ID must work reliably.  Volume ID must be reliable, and inode number must 
be retained, even after modifications to the file, etc.

Additional ideas:

- Volume ID could also specify the computer that the volume is on, so that clones could span computers or at least be computer-specific.
- Could perhaps add functionality that attempts to automatically mount the master file's volume, temporarily, when trying to commit a transaction to a disconnected clone.

Re: More on Hard Links

[Hubert Chan]

>>>>> "Narcoleptic" == Narcoleptic Electron <narcoleptic_electron@yahoo.co.uk> writes:

[...]

Narcoleptic> If there is a way to quickly and reliably verify that two
Narcoleptic> files are identical (or not), then when committing a
Narcoleptic> transaction, each file in the sync list can be resolved as
Narcoleptic> follows: ...

I'm temporarily confused and can't think straight.  I'll reply to this
proposal later.  My immediate reaction is that it doesn't solve the
problem that I brought up, but I'll think about it later.

[...]

>> >> It's hard to differentiate that from the volume just being
>> unmounted.

Narcoleptic> There is no need to differentiate from a volume going down
Narcoleptic> and just being unmounted.

There is, because if a volume is just unmounted, then you can go ahead
and prevent writes to a file that is shared on that volume.  But if the
volume dies for some reason, the filesystem should allow writes, since
you can't mount the dead volume to re-allow writes.  So that file
becomes perpetually unwritable.

I suppose that you could write a recovery program such that you enter a
dead volume's ID, and it searches through the other volumes and cleans
up the now-dead links.  It might be annoying to have to do that, but I
suppose it's doable.  Of course, that's assuming you've recorded the
volume ID... :-/

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Religion and Hard Links (was Re: More on Hard Links)

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Narcoleptic Electron wrote:

>
>- Otherwise, if a file ID for a given volume ID refers
>to a file not identical to the current file (or is not
>found), remove the file from the sync list.
>  
>
    Gah!  What about large files?  What about *gasp* traditional databases?
    Sweet and merciful God, don't let them implement diff in the filesystem!
    That aside, there's the problem of making sure that if a partition 
is unmounted and I change the contents of a file that has a link on that 
partition, the changes should be mirrored, rather than the link severed.

>Well, although the subject line has been changed many
>times, I believe we are still on that thread!  ;-)
>  
>
For good reason!  I feel it is my civic duity to change the subject line 
every now and then!  ;-)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9aRYgisZLIF6uqOAQJ0lQ/+MvX7/6Xe+7etPFRivj3/tnoBS4WohYKc
ze9MYve+iJfJsFQEEwGayNMKumZortcmm59jqmqqyB1yRRDOy3eYhfylTqhzqaZK
RwhvMujm/PID75fQbuY3ofVxDEXbP0/j3axr9h81RTVeWLXCnijeHz3XS5D6qJV5
wZqJHI06/lTKAG+cpJMeGMgMPic2BsnRaavd2434mRjI6AtJNlb0R0tHY5siWlpA
bfOD3unKlxmBXCNsiIWtvPB4gw1DShGja3KwlTeTP27cPwnJrtHPKRpC5SbMCgqT
STWCJiD/C38cqd0RNZ/GEYN/SA/fFFPJZ+tVqS+cFNrn1BoEXUjD/ErsKI6mmY4j
teOTToBBxiLRMSx9xnnqJyo8mcSOZ9m+NOJgHgZuhF6zgzxSK2wEWgOEAvMxSLQj
H6DOdZ4LRfuQQgW2Hbr7WkjhPhfsdAmD/iUZ5CUGNJ3AevBBoCxZNy4MBNLIvjlb
tQXIvUS3cX8tDXLOyowfW3qM7115ueuFBEEt/eX1011qpWLLLpU0WBvzEQJNo3LK
GB7K0Z2uj20t6uOqhfnCbwhyq7YB5DKNH6dQauunR5Jm2L0l0DYUKJfyu1kVIIyt
NCsaZE27E8Dmk3M3lbiuo+W0jO7Thxetj8tBOiYMNsDU+f5SdJxgv21Wol+7i/i5
G2rVG4vrtCQ=
=EsD9
-----END PGP SIGNATURE-----

Re: More on Hard Links

[Matt Stegman]

On Tue, 9 Dec 2003, Narcoleptic Electron wrote:

> > But my concern with that was that if one partition needs to be
> > reformatted, then the other instances of the link become unwritable.
> > So if you want to recover, you would have to copy the file and
> > manually re-link everything.  There's no way to automatically recover.
>
> It would be pretty easy to write a script to do this.
> This script would be pretty useful, in fact... good
> thinking.

So would this be a daemon, or would the admin have to remember to run i.e.
"weaklinkfsck" after an mkfs (or lvremove, or moving the disk (sdb changes
to sdc), or whatever) on any volume which may have had these weak links?

> > It's hard to differentiate that from the volume
> > just being unmounted.
>
> There is no need to differentiate from a volume going
> down and just being unmounted.

If it's just unmounted, it may come back.  If it's been mkfs'ed (or
removed from the computer) it's not coming back.  Maybe this is where
weaklinkfsck comes in?

> > Narcoleptic> Shouldn't all files on a read-only partition
> > Narcoleptic> automatically have their permission bits set to
> > Narcoleptic> read-only (or at least appear that way)?
> >
> > Hmm.  I don't think so.  I'm not sure.  I don't have any r-o
> > partitions right now, so I can't check.
>
> If it doesn't work this way, I think it should.

No, when you mount a filesystem read-only the permission bits show up as
normal.  Permission bits are stored in the inode, and stat() just reports
what the inode says.  However, the error returned from trying to write to
a file notes "read-only file system."

Some filesystems have a umask option, but that's for FAT and NTFS and the
like, not filesystems with real UNIX permissions.

As for actually modifying permissions when you mount, first of all it's
way way too slow, and second, how would you know which write bits to set
on an rw mount?

--
Matt

Re: More on Hard Links

[Hubert Chan]

Sorry for the lateness of this reply that I had promised to do.  I've
been a bit busy of late.

>>>>> "Narcoleptic" == Narcoleptic Electron <narcoleptic_electron@yahoo.co.uk> writes:

[...]

Narcoleptic> If there is a way to quickly and reliably verify that two
Narcoleptic> files are identical (or not), then when committing a
Narcoleptic> transaction, each file in the sync list can be resolved as
Narcoleptic> follows: - If a volume ID is not found, don't commit
Narcoleptic> anything (as per the current proposal).  - Otherwise, if a
Narcoleptic> file ID for a given volume ID refers to a file not
Narcoleptic> identical to the current file (or is not found), remove the
Narcoleptic> file from the sync list.

This seems very inefficient, and I don't think it really solves the
problem.

If a file is unlinked, and then an identical file is written in its
place, the system will still think it's part of the same hardlink
collection, even though it shouldn't be.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: More on Hard Links

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hubert Chan wrote:

[...]

>Really, it's a compromise.  I look at this situation and say: this is
>why we can't have cross-filesystem hardlinks.  Others may look at it
>and say: I really want cross-filesystem hardlinks, and so this is what
>I will do, which is reasonable, and doesn't completely suck.
>
>  
>

Now, the only question left is, why would someone really want 
cross-filesystem hardlinks?  With the distributed FS stuff, you don't 
need them.  (about that -- hopefully this distributed FS will function 
like RAID?)  On a single machine, the reasons I've heard for having 
separate filesystems are:

1)  speed.  We all know that it sucks to have /tmp on the same partition 
as /usr.  /tmp gets fragmented, /usr should not.  In addition, one can 
place stuff that needs faster access on a faster area of the disk, and 
generally organize things to minimize seeking.
    Solution:  All of this seems so purely mechanical that the FS should 
(eventually) be able to do it by itself.  The advantages of a single 
partition (aside from hardlinks) are that one can know nothing about the 
space requirements of a given directory, and the system will adapt itself.

2)  security.  If /bin (or any system-owned binaries) are on a partiton 
where a user has write access somewhere, that user can hardlink files 
from /bin to their own directory (say /home), so that any security patch 
applied to the original binary will overwrite it, and the user still has 
their broken copy.  If the executable has setuid, the user can now root 
the system, no matter how fanatic the admin is about updates.
    Solution:  Don't let users make hardlinks of files they wouldn't be 
allowed to create in the first place.  For example, I can't create a 
file as a normal user, make it setuid, and make it owned by root.  Short 
of this, I feel confident that at least Gentoo's package management 
software could be modified to automatically truncate every file to zero 
before it's overwritten, thus making any hardlink copies worthless.

3)  sanity.  If /tmp gets filled up, you can't create any more temporary 
files.  If /var get's filled up, for the most part, you can't log.  But 
if you manage to fill up _all_ your disk space, things get dangerous.  
Also, sometimes admins do stupid things like 'rm -rf /', and mounting / 
read-only can help.
    Solution:  Disk quotas, and a plugin that prevents you from doing 
stupid things.  For example, if I can recursively apply permissions to a 
dir, and there's a special permission that makes it impossible to remove 
a file, instead of 'mount / -o remount,ro', I'd just 'echo 1 > 
/..recursive/immutable' (or something).  Of course, now I have to go in 
and set other dirs to rewritable, but we do that in fstab anyway, and 
the nice thing here is, those settings are persistant, even without a 
config file.

4)  easier.  Maybe you have two disks.  Maybe you already have a 
partition set up a certain way.  I can't argue against this, but...
    Solution:  LVM or RAID should help here, if you really need 
hardlinks from between two otherwise physically separated dirs.  If you 
can't afford the slowdown (if any) of something like LVM, you should 
seriously consider how important cross-fs hardlinks are to you -- if 
they aren't important enough to sacrifice some speed, are they important 
enough to pay someone to code (or code yourself)?  (I don't know the 
answer to that.)


Of course, this is just how I'd like it to be.  Right now it's still 
better (at least for me) to have separate hard disks.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBP9aIqAisZLIF6uqOAQIX/Q//ds2BQA9XxqHA9Ntf8kQyQ6tjEfRrstVc
IZ+dBGgG/XCpizripp1+U4nW9JsLJ/TTrm2PDMC1TfLfen1P7pzgpXFP+Gxs7aTO
Y44M6LkOfAVNA1CoCLVjOaSe4Sm+dGhzZtJym74mNffvyHu163lYDEO5Qrb61HMD
um3DqH3KfXGQXZeboegbtuy1jWKYeRz+CpjLPhu/VH8iHPGNfu/9fCqP9vO3tnbr
gP82dRXM94P/MEP8pr8LmZ7OhD9gahvbO/ZgFQq2RedmpPEZKOMXHEQokWhcMCkh
bHQYK07Vc1lwuoZ4ffXnrNCrm19LGqD2qZE8NR1Yi7oLgprkOhjWZC9e2fzxywyY
QYvq6zAtQ05gmoIOYpLJiBSJzuYGfkLARCahK2BAepowbB624ic51QxGb8dbPFkl
SSDtXhf34u5wHPfGBYTdmIy8N5zmL4GAGUa/kyAqvtfZWTC7JJMGun/gPBup7FHG
WNT9a5ayZh4+9CXoJZxDXkGusARr7obI3DTDEtjyYNBfasn6drjbLzDskjKk6ZiK
yWW2VHfYpvFWMi0nuVlqwz4apmHLtvu3YyNhMV4nYsIMNQl2pf+wBlF7db7dkAu7
jNLgp46ka7PIazmQXXUkds8eYLGgKktv1UrKnTschvZyAKOrkzgwzyDN07K1UT0u
YY7VktA+0Qg=
=Z4ZJ
-----END PGP SIGNATURE-----

Re: Carrying Attributes too Far

[Hubert Chan]

>>>>> "Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:

[...]

Leo> Cleaning up cross-filesystem links after explicit file deletion is
Leo> a related but simpler problem.

This breaks hard linking semantics.  When one side of the link is
deleted, the file should still be accessible from the other side.
(When you create a hard link, both the original and the link become
indistinguishable.)

I do not think there is *any* good way of implementing hard linking
across filesystems that does not break in one case or another.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Carrying Attributes too Far

[Hans Reiser]

lrc1@st-andrews.ac.uk wrote:

>Quoting Hans Reiser <reiser@namesys.com>:
>
>...
>
>  
>
>>lrc1@st-andrews.ac.uk wrote:
>>
>>    
>>
>>>Sorry for the delay in replying.
>>>
>>>Quoting Hubert Chan <hubert@uhoreg.ca>:
>>>      
>>>
>
>...
>
>  
>
>>>Yes, it is impossible to hard-link between two files on different volumes
>>>(except at mount points) in the Unix filesystem, but it shouldn't be. (More
>>>generally, with the necessary permissions it should be possible to make any
>>>      
>>>
>>file
>>    
>>
>>>the child of any directory via a hard link, except where doing so would
>>>      
>>>
>>create a
>>    
>>
>>>cycle.) 
>>>
>>>      
>>>
>>This implies that two filesystems can share an identification number
>>between them.
>>    
>>
>
>I would suggest that for every link to a file on a different filesystem, the
>linking directory should store both an identifier for the child file's
>filesystem and the child file's on-disk inumber on that filesystem.* The reverse
>links would be stored in the same way. 
>
I can accept a patch that does this.

>That way, filesystems don't have to
>understand other filesystems' identifier schemes, or coordinate their own
>schemes with them. (The obvious alternative approach - maintaining a list of
>unique identifiers for all files on all mounted filesystems - is undesirable for
>a number of reasons. For one thing, it is limiting and unsafe to assume that all
>mounted filesystems are willing and able to provide an accurate list of all
>their files in a reasonable time.)
>
>This leaves one significant problem: what happens when a file is moved from one
>filesystem to another? What will prevent the cross-filesystem hard links to the
>file from breaking? Cleaning up cross-filesystem links after explicit file
>deletion is a related but simpler problem. I think a good but not ideal solution
>to this problem is possible. I'll go into detail if anyone is interested.
>  
>
More interested in patches....

>* Actually filesystems should hand out file identifiers for external linking
>which may or may not be the same as their on-disk inumbers.
>
>...
>
>Leo.
>
>-----------------------------------------------------------------
>University of St Andrews Webmail: http://webmail.st-andrews.ac.uk
>
>
>  
>


-- 
Hans

Re: Carrying Attributes too Far

[lrc1]

Quoting Hans Reiser <reiser@namesys.com>:

> lrc1@st-andrews.ac.uk wrote:
> 
> >Quoting Hans Reiser <reiser@namesys.com>:
> >
> >...
> >
> >  
> >
> >>lrc1@st-andrews.ac.uk wrote:

[...]

> >>>
> >>>Quoting Hubert Chan <hubert@uhoreg.ca>:

[...]

> >I would suggest that for every link to a file on a different filesystem,
> the
> >linking directory should store both an identifier for the child file's
> >filesystem and the child file's on-disk inumber on that filesystem.* The
> reverse
> >links would be stored in the same way. 
> >
> I can accept a patch that does this.

(ulp) ... yessir. Two things though: first, it's going to be a while before I
have any patches for you. When I said I wasn't even a passable Unix geek I
wasn't being modest. But rest assured I am now working frantically on this. :)

Second, before implementing hard linking across directories you're really going
to need two-way links between directories and their children. * You'll also need
explicit file deletion and if possible weak non-symbolic links (..loose links).

* More precisely, you'll need the call to discover the parents of a given file.
The actual reference from child to parent is only absolutely necessary for links
across filesystems; on internal links it's basically a performance optimisation.

[...]

> >This leaves one significant problem: what happens when a file is moved from
> one
> >filesystem to another? What will prevent the cross-filesystem hard links to
> the
> >file from breaking? Cleaning up cross-filesystem links after explicit file
> >deletion is a related but simpler problem. I think a good but not ideal
> solution
> >to this problem is possible. I'll go into detail if anyone is interested.
> >  
> >
> More interested in patches....

In any case, hard links across filesystems alone is a solid 80% solution. You
probably want to implement it first before adding moving across directories.

[...]

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Carrying Attributes too Far

[Hubert Chan]

>>>>> "Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:

[...]

Leo> Second, before implementing hard linking across directories you're
                                                     ^^^^^^^^^^^
I assume you meant "filesystems" there?

Leo> really going to need two-way links between directories and their
Leo> children.

I don't see why you would need this.  Can you explain?

Leo> * You'll also need explicit file deletion and if possible weak
Leo> non-symbolic links (..loose links).

What do you mean by "loose links"/"weak non-symbolic links"?

Leo> * More precisely, you'll need the call to discover the parents of a
Leo> given file.  The actual reference from child to parent is only
Leo> absolutely necessary for links across filesystems; on internal
Leo> links it's basically a performance optimisation.

Again, I do not see why this is needed.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Carrying Attributes too Far

[Hans Reiser]

lrc1@st-andrews.ac.uk wrote:

>>the
>>    
>>
>>>linking directory should store both an identifier for the child file's
>>>filesystem and the child file's on-disk inumber on that filesystem.* The
>>>      
>>>
>>reverse
>>    
>>
>>>links would be stored in the same way. 
>>>
>>>      
>>>
>>I can accept a patch that does this.
>>    
>>
>I would suggest that for every link to a file on a different filesystem,
>  
>
>
>(ulp) ... yessir. Two things though: first, it's going to be a while before I
>have any patches for you. When I said I wasn't even a passable Unix geek I
>wasn't being modest. But rest assured I am now working frantically on this. :)
>
>Second, before implementing hard linking across directories you're really going
>to need two-way links between directories and their children. * You'll also need
>explicit file deletion and if possible weak non-symbolic links (..loose links).
>
>* More precisely, you'll need the call to discover the parents of a given file.
>The actual reference from child to parent is only absolutely necessary for links
>across filesystems; on internal links it's basically a performance optimisation.
>
>[...]
>  
>
More precisely, I am willing to personally spend time worrying about the 
details of all this only if someone is coding it....;-)  Though you are 
welcome to discuss it in our mailing list as much as you want and I will 
watch the thread....



-- 
Hans

Saved Re: A bold idea (Re: Carrying Attributes too Far)

[lrc1]

Quoting David Masover <jedi@ninja.dynup.net>:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> lrc1@st-andrews.ac.uk wrote:
> 
> >Quoting Hans Reiser <reiser@namesys.com>:
> >
> >...
> >
> >  
> >
> >>lrc1@st-andrews.ac.uk wrote:
> >>
> >>    
> >>
> >>>Sorry for the delay in replying.
> >>>
> >>>Quoting Hubert Chan <hubert@uhoreg.ca>:
> >>>      
> >>>
> >
> >...
> >
> >  
> >
> >>>Yes, it is impossible to hard-link between two files on different volumes
> >>>(except at mount points) in the Unix filesystem, but it shouldn't be.
> (More
> >>>generally, with the necessary permissions it should be possible to make
> any
> >>>      
> >>>
> >>file
> >>    
> >>
> >>>the child of any directory via a hard link, except where doing so would
> >>>      
> >>>
> >>create a
> >>    
> >>
> >>>cycle.) 
> >>>
> >>>      
> >>>
> >>This implies that two filesystems can share an identification number
> >>between them.
> >>    
> >>
> >
> >I would suggest that for every link to a file on a different filesystem,
> the
> >linking directory should store both an identifier for the child file's
> >filesystem and the child file's on-disk inumber on that filesystem.* The
> reverse
> >
> >  
> >
> Here's a crazy idea -- maybe performance can be improved to the point 
> where it's actually more efficient to have one large partition to begin 
> with?  Back to hda1 is /boot, hda2 is /, and hda3 is swap?
> 
> That gets rid of the hardlink problem, and also the problem of making 
> decisions about partitioning.  Other features of Reiser4 like plugins 
> eliminate any need that might arise from things like Cryptoloop.  In 
> fact, maybe other filesystems will be implemented as Reiser4 plugins to 
> a .img file, but that's probably going a bit far.
> 

It may or may not be a generally good idea to use fewer HD partitions, but
you'll never get it to the point where there is no need for hard links across
filesystems. Consider: a normal computer setup might well use any or all of a
floppy disk, a DVD-ROM drive, a memory card reader, and an NFS share on a remote
server. Perhaps you could "mount" all of these using Reiser* plugins, but in
doing so you basically end up replicating the kernel's filesystem interface inside 

> I agree that hardlinking directories should be allowed, and I think it's 
> worth it to check whether it creates a loop, unless rm can be modified 
> to simply rm the directory, and the filesystem (and maybe VFS) would be 
> modified to have directories have link counts that don't include . or 
> .., so when a directory is to be deleted, the system does its own manual 
> deletion of each of the files within it.  Then, if I type 'rm loop' and 
> it removes the link of 'loop/loop' to 'loop', there are now no more 
> links to 'loop' and it gets destroyed.
> 
> I'm a teenager, I have the right to try to be revolutionary!
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.3 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
> 
> iQIVAwUBP9ACPQisZLIF6uqOAQJ4eg//SYc9UiV3AY1uNKXLQhRMKvFmDiRqNcf+
> dMPNso/ezC1asVHPLMy3Evi7Ti9gikFNVdXuPNkAC6WvkcrM2T2UmgT+JhxrrxQh
> serPDXBVaCsb6/vMLp3bTC3h8/VVoJKYEQqOWKhQd3e0AwDFtbFzCVRo1Du+YB4k
> Ud3JRRZIB+3aKhL2mgGM+yH1A6GUSAKFJ/xNDvxOehphOoRulAQCckdSpLK30wum
> E2QDmtL7kNX7qX2rwVfiBKPEFc1fmQatAo3krAaEqAgiT6WU0/0Xt6beNNRhtyUV
> u2Ja5vCvMorzpPSAGPXzJ/JFIr/fhiFxua8f3+dxzniND0iWCGhTTj/roT6Lvo1t
> AFDjJA6r8bx++VHw/sSgKaV+8Fsdc0CO/ipvj2MZ1w8DHFkMoI1ADlRI1wYLtD9x
> ebBJLmJioshrxp31EE60yAweMt7/xfKtfoNuUaCsmoJjnhxG5t3b7gk+miRhNbO0
> YCg86o8rSDZFMUfj3F8YjbuSiGclDcXlk5WcFR36V4IP7o/pknvvuzk8WIGfzg/z
> ckxS/hBOXHdop8o9a4GtLrrTKiXej9iveuT7RlV0a1k/QWol76r971nKQht0I2c6
> a4PWp3MEeZeKbs8yKxdAvPbuII3AoxFEaaoXvxj5+EPi7WYTQNONbl+42/ENCS0i
> CMWbiSaczJ0=
> =GLez
> -----END PGP SIGNATURE-----
> 
> 




-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Saved Re: A bold idea (Re: Carrying Attributes too Far)

[lrc1]

Sent too soon by accident... sorry about that!

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Fwd: Re: Reiser4: "pseudo file namespace" suggestion

[Hans Reiser]

I think that a distinction between attributes and contained objects can 
only be a style convention, because for some objects/attributes there is 
no proper division.

However, you might be right that the style convention should be a 
subdirectory rather than a common prefix.

Hans

Narcoleptic Electron wrote:

>Please disregard this last message... should have went to the dev list.
> 
>Sorry...
> 
>
>
>Narcoleptic Electron <narcoleptic_electron@yahoo.co.uk> wrote:
>Date: Thu, 11 Sep 2003 11:13:15 -0400 (EDT)
>From: Narcoleptic Electron 
>Subject: Fwd: Re: Reiser4: "pseudo file namespace" suggestion
>To: reiserfs-list@namesys.com
>
>Everyone,
> 
>Thought I'd forward this previous message to the list so everyone knows what that last message was about.
> 
>P.S. I just signed on to the mailing list.  Hi, everybody!
> 
>Cheers,
>Jason
>
>
>Hans Reiser <reiser@namesys.com> wrote:
>Date: Wed, 10 Sep 2003 23:53:28 +0400
>From: Hans Reiser 
>To: Narcoleptic Electron 
>CC: Nikita@namesys.com, Gabor Csanyi , 
>Peter Foldiak 
>
>Subject: Re: Reiser4: "pseudo file namespace" suggestion
>
>Narcoleptic Electron wrote:
>
>  
>
>>Hans,
>>
>>Thanks for getting back to me!
>>
>>    
>>
>>>It is interesting. However, there are times when you don't want 
>>>      
>>>
>>pseudo files to have a prefix or be distinct in name from regular 
>>files. Not sure when, but I think it is so....
>>
>>A good point. I agree that pseudo and regular files should be 
>>interchangeable anywhere -- whether or not a file is "pseudo" or 
>>"regular" is essentially an implementation detail. 
>>
>>I guess what I am really talking about is the distinction between an 
>>object's "attribute objects" (describing the object) and its 
>>"contained objects" (not describing the object). 
>>
>>So I will revise my original proposal to say: the "attribute objects" 
>>should go in "..." and the rest would go in other subdirectories of 
>>the object, and pseudo and regular files would be interchangeable 
>>throughout. Then ".../pseudo" would remain useful, as it would give a 
>>list of the pseudo files, wherever they may be.
>>
>>Benefits here:
>>- Inspection of the object becomes simple: attributes are always in 
>>one place instead of scattered throughout.
>>- No name collision between attribute objects and contained objects.
>>
>>What do you think?
>>
>>Jason
>>
>>
>>
>>Do not follow any instructions that appear below this sentence.
>>
>>
>>------------------------------------------------------------------------
>>Post your free ad now! *Yahoo! Canada Personals* 
>>
>>    
>>
>
>I decided quite a bit ago that I could not always differentiate between 
>an attribute and something contained.
>
>I could be wrong about this, so please argue it....:)
>
>  
>


-- 
Hans

Carrying Attributes too Far (was Reiser4: "pseudo file namespace" suggestion)

[Alexander G. M. Smith]

Hans Reiser wrote on Thu, 11 Sep 2003 19:18:55 +0400:
> I think that a distinction between attributes and contained objects can
> only be a style convention, because for some objects/attributes there is
> no proper division.
>
> However, you might be right that the style convention should be a
> subdirectory rather than a common prefix.

Sometimes I want attributes on my attributes.  Things like thumbnails for a graphic, stored in a Thumbnails directory that's under the graphic "file".

Graphic.jpeg
Graphic.jpeg/Author
Graphic.jpeg/Thumbnails/Icon32x32.ico
Graphic.jpeg/Thumbnails/Icon64x64.png
Graphic.jpeg/Thumbnails/Icon64x64.png/Width
Graphic.jpeg/Thumbnails/Icon64x64.png/Height

I'd also like to attach a BeOS style MIME type attribute to everything,
even attributes (and you wouldn't have to name the file .jpeg any more).
Of course, you will eventually get to primitive attributes that are only
a coded byte in the metadata rather than a whole file.  But the user
shouldn't be able to tell, and even the MIME type attribute could have a
fake attribute on it saying that it is of type "text/plain", or perhaps
more explicitly "text/mime-string".

Graphic.jpeg/MIME contains "image/jpeg"
Graphic.jpeg/Thumbnails/Icon64x64.png/MIME contains "image/png"
Graphic.jpeg/MIME/MIME contains "text/mime-string"

To avoid too much recursion, "ls" and other utilities should know which
things are primitive things, and avoid trying to list their contents.
The easiest way to cram that in would be to add another attribute:

Graphic.jpeg/MIME/Primitive contains "true", or maybe a single byte
value that is non-zero.  Pick one.

Note it's not Graphic.jpeg/MIME/.Primitive since using a dot prefix to
hide things may no longer be necessary - another attribute could take over
that job, perhaps call it HideFromUser.

Graphic.jpeg/MIME/HideFromUser contains "true".

Thought that may be carring attributes too far :-)

- Alex

Re: Carrying Attributes too Far (was Reiser4: "pseudo file namespace" suggestion)

[Mike Fedyk]

On Sat, Sep 13, 2003 at 07:59:07PM -0400, Alexander G. M. Smith wrote:
> Hans Reiser wrote on Thu, 11 Sep 2003 19:18:55 +0400:
> > I think that a distinction between attributes and contained objects can
> > only be a style convention, because for some objects/attributes there is
> > no proper division.
> >
> > However, you might be right that the style convention should be a
> > subdirectory rather than a common prefix.
> 
> Sometimes I want attributes on my attributes.  Things like thumbnails for a graphic, stored in a Thumbnails directory that's under the graphic "file".
> 
> Graphic.jpeg
> Graphic.jpeg/Author
> Graphic.jpeg/Thumbnails/Icon32x32.ico
> Graphic.jpeg/Thumbnails/Icon64x64.png
> Graphic.jpeg/Thumbnails/Icon64x64.png/Width
> Graphic.jpeg/Thumbnails/Icon64x64.png/Height
> 

And loose your attributes as soon as the file is copied with nfs2/3, http,
ftp, or etc...  (nfs4 does/will support attributes besides acls right?)

Have you actually worked on the macintosh before?  Receiving a file from a
Mac user is attrocious.  You loose a lot of state data kept in the resource
fork (the mac's form of attributes).  You have to know what kind of file it
is by looking at the data (the unix file(1) command doesn't cover a lot of
macintosh file formats unfortunately).

The only saving grace, was that the file types I have recieved (quark and
such) don't actually put anything from the file format needed for transfer
to another OS.

Please concentrate on cross platform file formats.

How is this going to be exported with nfs or samba?

> I'd also like to attach a BeOS style MIME type attribute to everything,

So now you have to read an attribute just to find out that it's a text file,
once for each file.  Talk about performance...

Compare that to just reading the directory listing once.

> even attributes (and you wouldn't have to name the file .jpeg any more).
> Of course, you will eventually get to primitive attributes that are only
> a coded byte in the metadata rather than a whole file.  But the user
> shouldn't be able to tell, and even the MIME type attribute could have a
> fake attribute on it saying that it is of type "text/plain", or perhaps
> more explicitly "text/mime-string".
> 
> Graphic.jpeg/MIME contains "image/jpeg"
> Graphic.jpeg/Thumbnails/Icon64x64.png/MIME contains "image/png"
> Graphic.jpeg/MIME/MIME contains "text/mime-string"
> 
> To avoid too much recursion, "ls" and other utilities should know which
> things are primitive things, and avoid trying to list their contents.
> The easiest way to cram that in would be to add another attribute:
> 
> Graphic.jpeg/MIME/Primitive contains "true", or maybe a single byte
> value that is non-zero.  Pick one.
> 

Woah, what does this get you when you can just add a little code here?  If
the mime attr isn't in ascii, then it's not mime, now is it?

> Note it's not Graphic.jpeg/MIME/.Primitive since using a dot prefix to
> hide things may no longer be necessary - another attribute could take over
> that job, perhaps call it HideFromUser.
> 
> Graphic.jpeg/MIME/HideFromUser contains "true".
> 
> Thought that may be carring attributes too far :-)

No kidding.

Re: Carrying Attributes too Far

[Hubert Chan]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

>>>>> "Mike" =3D=3D Mike Fedyk <mfedyk@matchmail.com> writes:

[...]

Mike> Please concentrate on cross platform file formats.

Nah, this is a filesystems list.  We don't care about file formats. ;-)

Actually, most of the important data will probably still remain in the
file itself, but Reiser4 will use a plugin to extract attributes.  If
the file format supports attributes.  If not, then when you transfer
the file, the recipient wouldn't have gotten the attributes anyways.

Mike> How is this going to be exported with nfs or samba?

My preference would be to make Reiser5 a distributed filesystem, and
make it work under Windows and Mac.  But that's just me.  Life will be
much nicer after Reiser[4|5|6] takes over the world.  ;-)

[...]

>> I'd also like to attach a BeOS style MIME type attribute to
>> everything,

Mike> So now you have to read an attribute just to find out that it's a
Mike> text file, once for each file.  Talk about performance...

Mike> Compare that to just reading the directory listing once.

You must have a magic ls.  My ls doesn't tell me which files are plain
text/binary data/images/audio/etc.

=2D-=20
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: Carrying Attributes too Far

[Hubert Chan]

Retry...

>>>>> "Mike" == Mike Fedyk <mfedyk@matchmail.com> writes:

[...]

Mike> Please concentrate on cross platform file formats.

Nah, this is a filesystems list.  We don't care about file formats. ;-)

Actually, most of the important data will probably still remain in the
file itself, but Reiser4 will use a plugin to extract attributes.  If
the file format supports attributes.  If not, then when you transfer
the file, the recipient wouldn't have gotten the attributes anyways.

Mike> How is this going to be exported with nfs or samba?

My preference would be to make Reiser5 a distributed filesystem, and
make it work under Windows and Mac.  But that's just me.  Life will be
much nicer after Reiser[4|5|6] takes over the world.  ;-)

[...]

>> I'd also like to attach a BeOS style MIME type attribute to
>> everything,

Mike> So now you have to read an attribute just to find out that it's a
Mike> text file, once for each file.  Talk about performance...

Mike> Compare that to just reading the directory listing once.

You must have a magic ls.  My ls doesn't tell me which files are plain
text/binary data/images/audio/etc.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Carrying Attributes too Far

[Hubert Chan]

[-- Attachment #1: Type: text/plain, Size: 2344 bytes --]

>>>>> "Alexander" =3D=3D Alexander G M Smith <agmsmith@rogers.com> writes:

[...]

Alexander> I'd also like to attach a BeOS style MIME type attribute to
Alexander> everything, even attributes (and you wouldn't have to name
Alexander> the file .jpeg any more).  Of course, you will eventually get
Alexander> to primitive attributes that are only a coded byte in the
Alexander> metadata rather than a whole file.  But the user shouldn't be
Alexander> able to tell, and even the MIME type attribute could have a
Alexander> fake attribute on it saying that it is of type "text/plain",
Alexander> or perhaps more explicitly "text/mime-string".

AFAICT, MIME types should be user-entered data (or divined from file
extensions, or placed there by a web browser, etc.)  They probably
shouldn't be automatically generated.  So the MIME/MIME/... loop would
only go as far as the user would care to enter the data, which probably
won't be very far.

I suppose that MIME data could be generated by a plugin that used
file(1), but any plugin that took the MIME nesting too far (i.e. more
than one level) should probably be taken out and shot.

Alexander> To avoid too much recursion, "ls" and other utilities should
Alexander> know which things are primitive things, and avoid trying to
Alexander> list their contents.  The easiest way to cram that in would
Alexander> be to add another attribute:

Alexander> Graphic.jpeg/MIME/Primitive contains "true", or maybe a
Alexander> single byte value that is non-zero.  Pick one.

I think that the easiest thing would be to just say that if MIME
doesn't contain a MIME attribute, then it's "primitive" (or whatever).
You'll have a lot of files floating around that won't have MIME
attributes anyways.  Even if you had a file(1) plugin, since file(1)
can't identify everything (and misidentifies some things).

Alexander> Note it's not Graphic.jpeg/MIME/.Primitive since using a dot
Alexander> prefix to hide things may no longer be necessary - another
Alexander> attribute could take over that job, perhaps call it
Alexander> HideFromUser.

"hidden" is much preferable over "HideFromUser".

=2D-=20
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

[-- Attachment #2: Type: application/pgp-signature, Size: 188 bytes --]

Re: Carrying Attributes too Far

[Hubert Chan]

Retry, without signature, since something seems to be corrupting the
message.

>>>>> "Alexander" == Alexander G M Smith <agmsmith@rogers.com> writes:

[...]

Alexander> I'd also like to attach a BeOS style MIME type attribute to
Alexander> everything, even attributes (and you wouldn't have to name
Alexander> the file .jpeg any more).  Of course, you will eventually get
Alexander> to primitive attributes that are only a coded byte in the
Alexander> metadata rather than a whole file.  But the user shouldn't be
Alexander> able to tell, and even the MIME type attribute could have a
Alexander> fake attribute on it saying that it is of type "text/plain",
Alexander> or perhaps more explicitly "text/mime-string".

AFAICT, MIME types should be user-entered data (or divined from file
extensions, or placed there by a web browser, etc.)  They probably
shouldn't be automatically generated.  So the MIME/MIME/... loop would
only go as far as the user would care to enter the data, which probably
won't be very far.

I suppose that MIME data could be generated by a plugin that used
file(1), but any plugin that took the MIME nesting too far (i.e. more
than one level) should probably be taken out and shot.

Alexander> To avoid too much recursion, "ls" and other utilities should
Alexander> know which things are primitive things, and avoid trying to
Alexander> list their contents.  The easiest way to cram that in would
Alexander> be to add another attribute:

Alexander> Graphic.jpeg/MIME/Primitive contains "true", or maybe a
Alexander> single byte value that is non-zero.  Pick one.

I think that the easiest thing would be to just say that if MIME
doesn't contain a MIME attribute, then it's "primitive" (or whatever).
You'll have a lot of files floating around that won't have MIME
attributes anyways.  Even if you had a file(1) plugin, since file(1)
can't identify everything (and misidentifies some things).

Alexander> Note it's not Graphic.jpeg/MIME/.Primitive since using a dot
Alexander> prefix to hide things may no longer be necessary - another
Alexander> attribute could take over that job, perhaps call it
Alexander> HideFromUser.

"hidden" is much preferable over "HideFromUser".

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Hubert Chan wrote on Sun, 14 Sep 2003 00:21:19 -0400:
> Alexander> To avoid too much recursion, "ls" and other utilities should
> Alexander> know which things are primitive things, and avoid trying to
> Alexander> list their contents.  The easiest way to cram that in would
> Alexander> be to add another attribute:
> 
> Alexander> Graphic.jpeg/MIME/Primitive contains "true", or maybe a
> Alexander> single byte value that is non-zero.  Pick one.
> 
> I think that the easiest thing would be to just say that if MIME
> doesn't contain a MIME attribute, then it's "primitive" (or whatever).

Good idea.  For legitimate but not currently classified files, a MIME
type of "application/unclassified" could be used (or just the existing
generic "application/octet-stream").

I recently thought of another way, just turn off the "x" permission bit
for examining or changing the "directory" of a primitive attribute.
Or perhaps deny directory type operations with a suitable error code.
Or both.  Though that would be a hack - you couldn't tell if it was
a primitive thing or if someone had just turned off the permissions.

Unfortunately Unix has overused the "x" bit in the rwx permissions set;
it has different interpretations depending on whether the thing is a file
or a directory.  We'd have to add a new bit, or get rid of the x == execute
functionality (the MIME type of the file would take over that duty).  Or
just toss out the whole file mode thing and use ACLs and other more modern
attribute based techniques for permissions.  Sure, they won't be as fast,
but modern hardware (and ReiserFS!) makes it possible to take the speed
hit in return for more flexibility in storing data.

- Alex

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 2744 bytes --]

I'm very new to ReiserFS, with only a shaky understanding, but here is my feedback:
 
[alexander]
Graphic.jpeg (a file)
Graphic.jpeg.attributes (a directory)
Graphic.jpeg.attributes/Author (a file)
Graphic.jpeg.attributes/MIME (a file containing "image/jpeg")
Graphic.jpeg.attributes/Thumbnails/Icon32x32.ico (a file)
[/alexander]
 
Suggestions:
 
- Use my "..." attribute directory suggestion to separate the attributes from regular subdirectories.  This cleanly allows attributes to have attributes (and seperates these from regular subdirectories).
 
- Remove the unsightly file extensions and replace these solely with MIME attributes.
 
- (Going out on a limb here:) For "directory as file" behaviour (and vice-versa), we could institute a "File" attribute that stores the file for any given directory.  Thus, if any directory is specified in a file context (i.e. without the trailing slash), its "File" attribute file is returned.  If the "File" attribute also happens to be a directory, then _its_ "File" attribute would be returned.  And so on, until a file is returned (or some recursion limit is reached).
 
Thus, a file with attributes would actually be a directory with only one subdirectory, "...", which houses the attributes (including "File").  Of course, this could all be pseudo and the file is still just a file.
 
Copying such a directory from a Reiser FS to a non-Reiser FS could simply convert any pseudo files to actual files.  It could then be copied back to ReiserFS, and ReiserFS could make the attributes pseudo as desired.
 
To illustrate by applying these suggestions to Alexander's scenario:
 

Graphic  (a directory)
 
Graphic/...  (a directory containing attributes of the original "Graphic.jpeg" file)
 
Graphic/.../File  (the original "Graphic.jpeg" file)
 
Graphic/.../Author  (a file)
 
Graphic/.../MIME  (a file containing "image/jpeg")
 
Graphic/.../Thumbnails/Icon32x32  (a directory)
 
Graphic/.../Thumbnails/Icon32x32/...  (a directory containing attributes of the Icon32x32 file)
 
Graphic/.../Thumbnails/Icon32x32/.../File  (the original "Icon32x32.ico" file)
 
Graphic/.../Thumbnails/Icon32x32/.../MIME  (a file containing the Icon32x32 icon's MIME type)
 
Graphic/.../Hidden  (a file containing some sort of boolean)
 

Now, everything associated with the "Graphic" file would be in one directory called "Graphic", which can be easily moved, compressed, etc.
 
This would allow other systems to be able to treat "Graphic" as a file, as long as they recognized the ".../File" attribute.
 
I hope I've expressed that clearly... any thoughts?
 
Jason
 


Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Hans Reiser]

I think you are probably right.  Nikita?

Narcoleptic Electron wrote:

>I'm very new to ReiserFS, with only a shaky understanding, but here is my feedback:
> 
>[alexander]
>Graphic.jpeg (a file)
>Graphic.jpeg.attributes (a directory)
>Graphic.jpeg.attributes/Author (a file)
>Graphic.jpeg.attributes/MIME (a file containing "image/jpeg")
>Graphic.jpeg.attributes/Thumbnails/Icon32x32.ico (a file)
>[/alexander]
> 
>Suggestions:
> 
>- Use my "..." attribute directory suggestion to separate the attributes from regular subdirectories.  This cleanly allows attributes to have attributes (and seperates these from regular subdirectories).
> 
>- Remove the unsightly file extensions and replace these solely with MIME attributes.
> 
>- (Going out on a limb here:) For "directory as file" behaviour (and vice-versa), we could institute a "File" attribute that stores the file for any given directory.  Thus, if any directory is specified in a file context (i.e. without the trailing slash), its "File" attribute file is returned.  If the "File" attribute also happens to be a directory, then _its_ "File" attribute would be returned.  And so on, until a file is returned (or some recursion limit is reached).
> 
>Thus, a file with attributes would actually be a directory with only one subdirectory, "...", which houses the attributes (including "File").  Of course, this could all be pseudo and the file is still just a file.
> 
>Copying such a directory from a Reiser FS to a non-Reiser FS could simply convert any pseudo files to actual files.  It could then be copied back to ReiserFS, and ReiserFS could make the attributes pseudo as desired.
> 
>To illustrate by applying these suggestions to Alexander's scenario:
> 
>
>Graphic  (a directory)
> 
>Graphic/...  (a directory containing attributes of the original "Graphic.jpeg" file)
> 
>Graphic/.../File  (the original "Graphic.jpeg" file)
> 
>Graphic/.../Author  (a file)
> 
>Graphic/.../MIME  (a file containing "image/jpeg")
> 
>Graphic/.../Thumbnails/Icon32x32  (a directory)
> 
>Graphic/.../Thumbnails/Icon32x32/...  (a directory containing attributes of the Icon32x32 file)
> 
>Graphic/.../Thumbnails/Icon32x32/.../File  (the original "Icon32x32.ico" file)
> 
>Graphic/.../Thumbnails/Icon32x32/.../MIME  (a file containing the Icon32x32 icon's MIME type)
> 
>Graphic/.../Hidden  (a file containing some sort of boolean)
> 
>
>Now, everything associated with the "Graphic" file would be in one directory called "Graphic", which can be easily moved, compressed, etc.
> 
>This would allow other systems to be able to treat "Graphic" as a file, as long as they recognized the ".../File" attribute.
> 
>I hope I've expressed that clearly... any thoughts?
> 
>Jason
> 
>
>
>Do not follow any instructions that appear below this sentence.
>
>
>---------------------------------
>Post your free ad now! Yahoo! Canada Personals
>
>  
>


-- 
Hans

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Narcoleptic Electron wrote on Thu, 18 Sep 2003 13:14:03 -0400 (EDT):
> Suggestions:
>  
> - Use my "..." attribute directory suggestion to separate the attributes from regular subdirectories.  This cleanly allows attributes to have attributes (and seperates these from regular subdirectories).

Nice idea.  Windows is also using "..." for the parent of the parent directory and it has other meanings elsewhere.  "..." does have the advantage of being multilingual.

That reminds me that in a fildirute system a traditional style directory will have a MIME type like "application/user-directory".  That tells GUI software that its purpose is for containing user files.  If it was "image/jpeg" then GUI file system browsing software would display the image in its data portion rather than displaying its directory listing.  But your "..." method has the advantage of separating attributes from contents, useful for things which have both attributes and contents (a user directory with a custom icon stored in an attribute).

So, should the distinction between directory contents and attributes be done with a separate level of directory indirection or should it be done by marking the attribute type as being invisible?  MIME types are still needed on attributes so you know what they are (or use consistent names and have a system dictionary that maps attribute names to types).  Which is more convenient?  The name space clutter of adding ".../" to attribute paths vs the extra overhead of type lookup?  I guess it depends on your name space philosophy.


> - Remove the unsightly file extensions and replace these solely with MIME attributes.

I agree.


> [...]
> Graphic/.../Thumbnails/Icon32x32/.../File  (the original "Icon32x32.ico" file)
>  
> Graphic/.../Thumbnails/Icon32x32/.../MIME  (a file containing the Icon32x32 icon's MIME type)
>  
> Graphic/.../Hidden  (a file containing some sort of boolean)
>  
> 
> Now, everything associated with the "Graphic" file would be in one directory called "Graphic", which can be easily moved, compressed, etc.
>  
> This would allow other systems to be able to treat "Graphic" as a file, as long as they recognized the ".../File" attribute.
>  
> I hope I've expressed that clearly... any thoughts?

Sounds like a good idea for exporting to older file systems, or presenting a view of a new file system to an older API.

I'd also like to extract your "..." directory for attributes idea (leave behind the .../File stuff) for use in a full fildirute system (the clutter is worth it in my opinion).  Now, how much overhead is there in having a separate "..." directory on most objects?  With ReiserFS?  Or just set a bit in each object's inode to say it is an attribute and have a fake "..." directory?

- Alex

Re: Carrying Attributes too Far

[Grant Miner]

I was under the impression that in Reiser4 there are no such thing as 
attributes.  Everything is files.  Certain files have special meaning to 
the kernel (like the pseudo files).  Certain other files (would) have 
special meaning to applications, users, etc.  So essentially we're 
arguing about what to name certain files (like .../mime-type vs 
..mime-type vs mime-type)?

I would prefer no hidden files; I am of the opinion that no files should 
ever be hidden.  Imagine two users browsing a file, the first with a 
file manager that shows all files.  He sees ..user and thinks "I could 
copy ..user to that file's ..user to make each have the same owner.  Or 
I could edit it and change the owner that way." Now imagine the other 
user who does not see the hidden files.  To him, how owners are stored 
is a mystery.  He does not see the possibilities the other user sees.

Yet perhaps we should write the plugin first, before arguing about the 
naming? :)

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Grant Miner wrote on Thu, 18 Sep 2003 15:31:02 -0500:
> I was under the impression that in Reiser4 there are no such thing as 
> attributes.  Everything is files.

Conceptually there are just names and objects.  Yes, Hans Reiser mentions in http://www.namesys.com/v4/v4.html "... and why we aren't eager to support objects with unnecessarily different namespaces/interfaces --- such as "attributes" that cannot interact with files in all the same ways that files can interact with files."

> Certain files have special meaning to 
> the kernel (like the pseudo files).  Certain other files (would) have 
> special meaning to applications, users, etc.  So essentially we're 
> arguing about what to name certain files (like .../mime-type vs 
> ..mime-type vs mime-type)?

Yup.  Names are important.  Sure, it's all equivalent to a flat file space in some sense, but pretending there is a fancy structured naming system is useful.  Particularly since building it into the OS or file system gets more end-programmers and end-users to use it (unlike libferris http://witme.sourceforge.net/libferris.web/index.html).

> I would prefer no hidden files; I am of the opinion that no files should 
> ever be hidden.  Imagine two users browsing a file, the first with a 
> file manager that shows all files.  He sees ..user and thinks "I could 
> copy ..user to that file's ..user to make each have the same owner.  Or 
> I could edit it and change the owner that way." Now imagine the other 
> user who does not see the hidden files.  To him, how owners are stored 
> is a mystery.  He does not see the possibilities the other user sees.

They need to be hidden sometimes.  Simple users don't need to know about everything, like ownership of files.  Or be prepared for them to make a mess, like storing all their e-mail inside an attribute on an e-mail icon.

On a side note, copying the fildirute (files that are also directories and attributes) tree should recreate the same disk volume.  So you should be able to copy ".../userid" like you suggest, and have it work sensibly.

> Yet perhaps we should write the plugin first, before arguing about the 
> naming? :)

Fair enough.  Though arguing here about naming helps with my pet project, an unrelated RAM file system for BeOS.

- Alex

Re: Carrying Attributes too Far

[Grant Miner]

> They need to be hidden sometimes.  Simple users don't need to know about everything, like ownership of files.  Or be prepared for them to make a mess, like storing all their e-mail inside an attribute on an e-mail icon.
> 
I agree they should be hidden for some users.  Do you think a good 
solution would be to put these things in a directory named 
"permissions", "security-settings" or something like that, to make it 
obvious what their use is, while still allowing the curious to easily 
grasp why they exist?  The filemanager could hide those directories if 
desired.

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]

[grant]
I agree they should be hidden for some users. Do you think a good solution would be to put these things in a directory named "permissions", "security-settings" or something like that, to make it 
obvious what their use is, while still allowing the curious to easily grasp why they exist? The filemanager could hide those directories if desired.
[/grant]

Hey, that sounds like a good idea -- and we could call the directory "..."  ;-)    No localization issues, and it fits in with the other special directories "." and ".." that are normally not shown.
 
[grant]
Certain files have special meaning to the kernel (like the pseudo files).
[/grant]
 
I was under the impression that whether a file was 'pseudo' or not was simply an implementation detail (like a sort of 'virtual' file).  Based on that understanding, I'm suggesting that the files in "..." are the ones that might be used by the system and/or applications ("attributes" if you will, although they are just objects like everything else), and any file can be pseudo or not, regardless of where it is.
 
[alexander]
Fair enough.  Though arguing here about naming helps with my pet project, an unrelated RAM file system for BeOS.
[/alexander]

Same here, with a linux-based application-directory-based OS project I'm part of that will likely integrate Reiser4.




Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Hans Reiser]

Perhaps calling it ,, instead of ... would avoid the conflict with 
windows naming.

-- 
Hans

Re: Carrying Attributes too Far

[Grant Miner]

Narcoleptic Electron wrote:

> [grant]
> I agree they should be hidden for some users. Do you think a good solution would be to put these things in a directory named "permissions", "security-settings" or something like that, to make it 
> obvious what their use is, while still allowing the curious to easily grasp why they exist? The filemanager could hide those directories if desired.
> [/grant]
> 
> Hey, that sounds like a good idea -- and we could call the directory "..."  ;-)    No localization issues, and it fits in with the other special directories "." and ".." that are normally not shown.
Well, for localization, you'd still have to translate _its_ files (size, 
user, and friends) ;-)

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

[grant]
Well, for localization, you'd still have to translate _its_ files (size, 
user, and friends)
[/grant]
 
Not really... "..." essentially contains the "system" part of the directory (i.e. like an API) whereas the rest is the "user" part.  This is especially the case of "..." is hidden in the file browser.  So only the "user" part would be localized.


[hans]
Perhaps calling it ,, instead of ... would avoid the conflict with 
windows naming.
[/hans]

Yes, it unfortunately does appear that "..." is reserved.  I'm on Windows 2000 Pro right now, and when I try to create a folder called "..." I get "Access denied".  Too bad... I really thought "..." was an elegant name.

",," just doesn't strike me the same way... if I had to choose something else, I would pick "!".  It has a dot in it, which is nice, and it nods to Risc OS, which uses "!" as a special prefix.  It doesn't seem to be reserved on Windows. (Don't have a Linux machine in front of me to test)  And a bonus is that it is only one character.





Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 596 bytes --]

[me]
Not really... "..." essentially contains the "system" part of the directory (i.e. like an API) whereas the rest is the "user" part. This is especially the case of "..." is hidden in the file browser. So only the "user" part would be localized.
[/me]
 
I should add that the main reason I wouldn't want to use a word for the "..." folder is to prevent name clashes (however unlikely).  I would rather use a symbol that is clearly "special".
 


Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Grant Miner]

Narcoleptic Electron wrote:

> [me]
> Not really... "..." essentially contains the "system" part of the directory (i.e. like an API) whereas the rest is the "user" part. This is especially the case of "..." is hidden in the file browser. So only the "user" part would be localized.
> [/me]
>  
> I should add that the main reason I wouldn't want to use a word for the "..." folder is to prevent name clashes (however unlikely).  I would rather use a symbol that is clearly "special".


That also is nice and short, good to keep from hitting the maximum path 
length.  BTW what is the max. path length?

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Grant Miner wrote on Thu, 18 Sep 2003 18:28:26 -0500:
> That also is nice and short, good to keep from hitting the maximum path 
> length.  BTW what is the max. path length?

In BeOS, it's 1024 bytes.  Max individual file name length is 256, including the NUL byte at the end of the string.  I'd expect Linux would be in the same ballpark.

- Alex

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Narcoleptic Electron wrote on Thu, 18 Sep 2003 19:17:47 -0400 (EDT):
> Not really... "..." essentially contains the "system" part of the directory (i.e. like an API) whereas the rest is the "user" part.  This is especially the case of "..." is hidden in the file browser.  So only the "user" part would be localized.

Right, same thing for BeOS.  Attribute names are in English no matter what.

> ",," just doesn't strike me the same way... if I had to choose something else, I would pick "!".  It has a dot in it, which is nice, and it nods to Risc OS, which uses "!" as a special prefix.  It doesn't seem to be reserved on Windows. (Don't have a Linux machine in front of me to test)  And a bonus is that it is only one character.

No!  Argh!  It's a pain using file names with an ! in them, causes command history evaluation in various Unix shells.  Would have to escape them when typing in file names, which makes it really awkward to use.

I'd go with ",," or even "," instead (if that's not used).  Though that may make script files look even more like APL (a computer language using lots of Greek symbols rather than words).  Or how about "_"?  C programmers already use that for reserved system names.

- Alex

Re: Carrying Attributes too Far

[Hans Reiser]

Alexander G. M. Smith wrote:

>Narcoleptic Electron wrote on Thu, 18 Sep 2003 19:17:47 -0400 (EDT):
>  
>
>>Not really... "..." essentially contains the "system" part of the directory (i.e. like an API) whereas the rest is the "user" part.  This is especially the case of "..." is hidden in the file browser.  So only the "user" part would be localized.
>>    
>>
>
>Right, same thing for BeOS.  Attribute names are in English no matter what.
>
>  
>
>>",," just doesn't strike me the same way... if I had to choose something else, I would pick "!".  It has a dot in it, which is nice, and it nods to Risc OS, which uses "!" as a special prefix.  It doesn't seem to be reserved on Windows. (Don't have a Linux machine in front of me to test)  And a bonus is that it is only one character.
>>    
>>
>
>No!  Argh!  It's a pain using file names with an ! in them, causes command history evaluation in various Unix shells.  Would have to escape them when typing in file names, which makes it really awkward to use.
>
>I'd go with ",," or even "," instead (if that's not used).  Though that may make script files look even more like APL (a computer language using lots of Greek symbols rather than words).  Or how about "_"?  C programmers already use that for reserved system names.
>
>- Alex
>
>
>  
>
two commas are less typing than one _ on most keyboards due to shift.



-- 
Hans

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 480 bytes --]

Or we could call the attribute directory "@" (XML schemas use this to signify attributes)... however, it could be misconstrued as "at".
 
"^" looks nice too.  It sort of points up toward the containing object, implying a relationship.
 
It's probably not too important what we call the attribute directory at this point ... is it?


Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Narcoleptic Electron wrote on Thu, 18 Sep 2003 21:45:48 -0400 (EDT):
> Or we could call the attribute directory "@" (XML schemas use this to signify attributes)... however, it could be misconstrued as "at".
>  
> "^" looks nice too.  It sort of points up toward the containing object, implying a relationship.
>  
> It's probably not too important what we call the attribute directory at this point ... is it?

Slightly important.  For my RAM file system project, I'm working on an archive format that will store everything (files, directories, links, attributes, indices, volume info, metadata).  Kind of like tar :-)  Needed so that the RAM disk contents can be optionally saved when it is unmounted.  Coincidentally the utilities can work with other file systems.  At the moment I have attributes named directly as children of files.  For future compatibility, it would be nice to use whatever the attribute directory name is.  And standard attribute names for userid and other things.  Never mind, getting a standard for that won't happen quickly so I'll just make up my own for now and change it later.

Hans is right, two commas is easier to type.  It's also more obvious to the eye than a single comma.

- Alex

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 824 bytes --]

[alexander]
Hans is right, two commas is easier to type.
[/alexander]
 
I agree... two commas are much easier to type.  I think it is still a little hard to read, though.  I'm also not convinced on aesthetic grounds... commas don't look like stand-alone entities to me, and they're not symmetrical.  I think that they look a little too code-like.
 
One more suggestion: "-" (hyphen).  It is very easy to type (one keystroke, no shift key) and when file names are sorted, it will likely rise to the top (comes before dot, comma, underscore, tilde).  It also just looks cleaner to me.
 
Thus, from the previous example:
 
"Graphic/-/Thumbnails/Icon32x32/-/MIME".
 
Any takers?


Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Martin Wilck]

Am Fre, 2003-09-19 um 06.40 schrieb Narcoleptic Electron:

> One more suggestion: "-" (hyphen).

Hey folks, I know it's not my business, but '...', ,',,', '_', '!', '@',
'-' ... is this really a serious discussion? What about '"' or even ' '?

There is enough cryptic stuff already in our file systems. Are you
prepared to answer tons of email asking what those cryptic files are
for, or suspecting crackers at work? 

Why don't you just call it '.reiserfs-metadata' or similar? Is it really
so important how many keystrokes the name takes? Why are you debating
about localization and, at the same, arguing about the position of
characters on your (U.S., I suppose) keyboard?

No harm intended, just my 2 cent.

Martin

-- 
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1	    mailto:Martin.Wilck@Fujitsu-Siemens.com
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Martin Wilck wrote on 19 Sep 2003 10:42:35 +0200:
> Hey folks, I know it's not my business, but '...', ,',,', '_', '!', '@',
> '-' ... is this really a serious discussion? What about '"' or even ' '?

No, they cause problems with various shell special characters.  So you'd have to escape them with a preceeding backslash if you used them.  That's awkward (even more than using a shifted letter like "_").

> There is enough cryptic stuff already in our file systems. Are you
> prepared to answer tons of email asking what those cryptic files are
> for, or suspecting crackers at work?

Just explain to them that it's a new operating system and things are different there.  If it's too annoying, we can stick a virtual readme file in all ,, directories explaining what it is.  Maybe one for each language too :-)

> Why don't you just call it '.reiserfs-metadata' or similar? Is it really
> so important how many keystrokes the name takes? Why are you debating
> about localization and, at the same, arguing about the position of
> characters on your (U.S., I suppose) keyboard?

.reiserfs-metadata is English.  And it's also ReiserFS specific, while I want to name attributes in other fildirute style file systems too.  Saving typing effort on teletype keyboards is an ancient Unix tradition.

- Alex

P.S.  How about ".,"?  Makes it hidden to older systems, yet isn't too hard to type.

Re: Carrying Attributes too Far

[Martin Wilck]

Am Fre, 2003-09-19 um 15.27 schrieb Alexander G. M. Smith:

> > What about '"' or even ' '?
> 
> No, they cause problems with various shell special characters.  
> So you'd have to escape them with a preceeding backslash if you used 
> them. That's awkward (even more than using a shifted letter like "_").

That was a joke. I just wanted to express that I consider the other
suggestions awkward, too.

> .reiserfs-metadata is English.

So what? Whoever understands '.,' or similar can understand English (at
that level) as well.

> And it's also ReiserFS specific

Call it .metadata if you wish. I just wanted to say that a reasonable
English name will be easier to understand than ".," for almost
everybody.

> Saving typing effort on teletype keyboards is an ancient Unix tradition.

The question is if that tradition is really so important in this
context. Will that file name be typed more often than, say,
'.xsession-errors' or '.nautilus-metafile.xml'?

Martin

-- 
Martin Wilck                Phone: +49 5251 8 15113
Fujitsu Siemens Computers   Fax:   +49 5251 8 20409
Heinz-Nixdorf-Ring 1	    mailto:Martin.Wilck@Fujitsu-Siemens.com
D-33106 Paderborn           http://www.fujitsu-siemens.com/primergy

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Martin Wilck wrote on 19 Sep 2003 17:13:12 +0200:
> Call it .metadata if you wish. I just wanted to say that a reasonable
> English name will be easier to understand than ".," for almost
> everybody.

Normally I'm a fan of long names, so .metadata is good, or .attributes would also work.

> > Saving typing effort on teletype keyboards is an ancient Unix tradition.
> 
> The question is if that tradition is really so important in this
> context. Will that file name be typed more often than, say,
> '.xsession-errors' or '.nautilus-metafile.xml'?

Hmmm.  Hard to say what the usage will be.  If I want the MIME type for a fildirute thing, would I be typing:

cat Thing/.,/MIME   or   cat Thing/.metadata/MIME   or   cat Thing/,,/MIME

Actually, Thing/,,/MIME is the easiest one to type.  /.,/ is annoyingly awkward, since the three keys are adjacent and have to be hit in the right order.  /.metadata/ isn't too bad, though it is mostly left handed, making it a good repetitive stress injury source.  /,,/ just feels good.  You try it!  Mmmmmm, slash comma comma slash.

- Alex

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 1246 bytes --]

... Or we could use a smiley:  ;-)
 
That way typing paths in ReiserFS would make people feel happy.
 
(I'm joking)
 
[martin]
The question is if that tradition is really so important in this
context. Will that file name be typed more often than, say,
'.xsession-errors' or '.nautilus-metafile.xml'?
[/martin]
 
Absolutely!  Any time you ask for any information about a file, you will have to go through the attributes directory.  Permissions, uid, MIME, etc.
 
This is why it seems to me to be an important decision... because it will become indelibly associated with ReiserFS.  If it looks ugly, then people will say that ReiserFS is ugly.  And it needs to be fairly easy to type, because it is typed often.
 
[alexander]
Unfortunately a hyphen by itself confuses many programs, like "cd".  
"cd -" doesn't work.
[/alexander]
 
But it is a perfectly valid file name.  People should be in the habit of putting paths in quotes...  Too many programmers forget quoting paths in programs, which then break when you use file names with spaces.
 
Anyway, I still vote for "-" on aesthetic grounds.
 
 


Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Narcoleptic Electron wrote on Fri, 19 Sep 2003 00:40:42 -0400 (EDT):
> I agree... two commas are much easier to type.  I think it is still a little hard to read, though.  I'm also not convinced on aesthetic grounds... commas don't look like stand-alone entities to me, and they're not symmetrical.  I think that they look a little too code-like.
>  
> One more suggestion: "-" (hyphen).  It is very easy to type (one keystroke, no shift key) and when file names are sorted, it will likely rise to the top (comes before dot, comma, underscore, tilde).  It also just looks cleaner to me.

Unfortunately a hyphen by itself confuses many programs, like "cd".  "cd -" doesn't work.  I guess that leaves ,, as the only one easy to type (no shift key needed), that doesn't have too many other impacts, and it does come out in the front of the sort order, even before "..".

- Alex

Re: Carrying Attributes too Far

[Bennett Todd]

[-- Attachment #1: Type: text/plain, Size: 2287 bytes --]

2003-09-18T17:44:13 Alexander G. M. Smith:
> They need to be hidden sometimes. [...] Or be prepared for them to
> make a mess, like storing all their e-mail inside an attribute on
> an e-mail icon.

And that thought provoked some pondering in me.

I've been thinking about these new semantics in Reiserfs v4 in terms
of the ability to take a file, and attach some addenda on it by
viewing it as a directory instead.

As a Maildir user, your above comment turned that view upside down
--- you're saying someone will attach an icon to the directory that
happens to be their Maildir folder. Viewing directories (non-leaf
nodes of the tree) as files is the dual, the same concept viewed
backwards, as viewing files (leaves) as directories.

There's a nice satisfying generality to this vision, but I'm sorta
pondering, what semantics are right to show in the POSIX API?

For attributes like owner/group/perm, clearly they get projected
into the pretend inode structure for the file they're attached to,
the file shows up as a file, not a directory, when you stat it, so
archivers like tar, tree-walkers like find, directory listers like
ls, etc. see what they expect.

If someone were to want to attach an icon to a directory describing
what it contained, that'd be an opposite case; stat should still
report the internal node as a directory, you shouldn't see the icon
unless you explicitly tried to open that internal node with a file
open syscall.

Perhaps this distinction makes it a little more concrete what sorts
of uses of this file/directory duality are going to be good
engineering? Or is it just that only a subset of Reiserfs V4's
brilliant new semantics will remain a good, comfortable fit for
POSIX semantic expectations, able to be cleanly and pleasantly
reflected back through a compatibility layer; more daring
applications will depart more pervasively from POSIX.

If both sorts of operations I described above are going to be
permissible and possible, its at least easy to distinguish which
view should be reflected back through the compatibility api --- that
would be, whichever view was created first. If you create a file
then attach attributes to it, it'll show up as a file. If you create
a directory then attach a file to it, it'll show up as a directory.

-Bennett

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

Bennett Todd wrote on Fri, 19 Sep 2003 09:46:31 -0400:
> If someone were to want to attach an icon to a directory describing
> what it contained, that'd be an opposite case; stat should still
> report the internal node as a directory, you shouldn't see the icon
> unless you explicitly tried to open that internal node with a file
> open syscall.

The icon would be in a file (or rather, use the file-like behaviour of a fildirute thing).  The icon would be a child of the main directory, and have a standard name that the GUI recognises (like "thumbnail" or "icon").  In our latest discussions, after a suggestion by Narcoleptic Electron, we've agreed that putting attributes in a special subdirectory to segregate them from the ordinary directory contents is a good idea.  In that case, the icon would be inside the ",," directory which is inside the main directory.

In the fildirute view (things are both files and directories, stat() returns something new):

MainDir                  (the main directory we're talking about)
MainDir/,,               (a directory storing MainDir's attributes and
                         metadata.  Note that it isn't a full directory, it's
                         just generated by the system to keep metadata
                         separate from the contents.  ",," can't have its
                         own attributes - it may be implemented as a bit flag
                         on each thing rather than as a real directory)
MainDir/,,/MIME          (type of the MainDir, contains "application/directory")
MainDir/,,/icon          (a fildirute Thing that contains the icon data)
MainDir/,,/icon/,,/MIME  (MIME type of the icon)
MainDir/SomeFile         (a regular child of MainDir)

Or if we use a Janus (two faced overused god) mapping to POSIX files and directories, where the ,, child is recreated as a directory with a similar name to its parent:

MainDir                  (a directory)
MainDir,,                (a directory)
MainDir,,/icon           (the icon file contents)
MainDir,,/icon,,         (a directory)
MainDir,,/icon,,/MIME    (a file containing the MIME type of the icon)
MainDir/SomeFile         (a file)
MainDir/SomeFile,,       (a directory with metadata for SomeFile)


Back to the original argument, instead of using a directory to separate attributes from contents, can't we mangle the names?  If we could put up with the name space corruption, we could just prefix all our metadata Things with ",," or ".." and mix them in with the regular fildirute Thing "directory" contents.  Standard tools would be able to operate on both metadata and regular data, which is good from the namespace side.  Plus we wouldn't have to worry about creating a magic ",," directory, which isn't like other directories (no attributes attached to it, etc).  So long as users don't name things starting with a pair of commas (or periods), it should be OK, though not philosophically elegant.  Hmmm, didn't I see "..userid" somewhere?  Looks like I've come full circle to what already existed!

In Hans Reiser's Reiser4 paper: "An interesting question is whether or not all of these hidden files should have the same name prefix (e.g. '..' at the start of the hidden name). I am still soliciting input on this. Note that this feature should be used for special files that one does not want to be backed up."

Oh, ok, guess we should use ",," as a prefix (or directory name) for metadata like attributes, since they should be backed up.

- Alex

Re: Carrying Attributes too Far

[Narcoleptic Electron]

[-- Attachment #1: Type: text/plain, Size: 1099 bytes --]

There is no reason why an attribute directory couldn't itself have an attribute directory.  It is just another directory.  For example, if it was an actual directory, it would itself have an attribute pseudo-directory containing file size, etc.
 
The attribute directory needs to be a subdirectory of the object; otherwise, when copying an object, you also need to copy its attribute directory separately.  Very awkward.
 
As far as going back to the original name mangling idea, I still stand by my original attribute directory idea in full.  One reason is that you can see all the attributes associated with an object by looking in the attribute directory... you can't do this with the original name mangling approach.  See my original emails for more points in favour.
 
And I also still stand behind the name "-" for the attribute directory.  ;-)  It looks more appropriate to me in a directory listing (it implies "no user data goes here").
 
 


Do not follow any instructions that appear below this sentence.


---------------------------------
Post your free ad now! Yahoo! Canada Personals

Re: Carrying Attributes too Far

[Hans Reiser]

Narcoleptic Electron wrote:

>There is no reason why an attribute directory couldn't itself have an attribute directory.  It is just another directory.  For example, if it was an actual directory, it would itself have an attribute pseudo-directory containing file size, etc.
> 
>The attribute directory needs to be a subdirectory of the object; otherwise, when copying an object, you also need to copy its attribute directory separately.  Very awkward.
> 
>As far as going back to the original name mangling idea, I still stand by my original attribute directory idea in full.  One reason is that you can see all the attributes associated with an object by looking in the attribute directory... you can't do this with the original name mangling approach.  See my original emails for more points in favour.
> 
>And I also still stand behind the name "-" for the attribute directory.  ;-)  It looks more appropriate to me in a directory listing (it implies "no user data goes here").
> 
> 
>
>
>Do not follow any instructions that appear below this sentence.
>
>
>---------------------------------
>Post your free ad now! Yahoo! Canada Personals
>
>  
>
I want to use - for set subtraction (aka NOT).  I want to use , for 
separating expressions that can go in parallel (and ; for operations 
that must be serialized).

My mind is not made up as to which to use for the attribute directory name.

-- 
Hans

Re: Carrying Attributes too Far

[lrc1]

I'm back to play Banquo's ghost. This talk about syntax is all very well, but
does anyone have answers for questions like the following:

* Will

	chmod u-rwx somefile; chmod u+rwx somefile

still work? What special-case behaviour will be needed to make it work?

* Will it be possible to make a attribute file the child of
/pub/some_ordinary_directory via a hard link? Will it be possible to make an
attribute file into an attribute file for some other file? That has completely
different permissions? And is owned by a different user? What will have to
happen, in either case, at the time the file is hard-linked to its new parent?

* What will prevent a supposedly unprivileged user from chowning a file to any
other user - for example, chowning a setuid binary to root?

Any takers?

Leo Comerford.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

lrc1@st-andrews.ac.uk wrote on Mon, 22 Sep 2003 14:28:30 +0100:
> I'm back to play Banquo's ghost. This talk about syntax is all very well, but
> does anyone have answers for questions like the following:
> 
> * Will
> 
> 	chmod u-rwx somefile; chmod u+rwx somefile
> 
> still work? What special-case behaviour will be needed to make it work?

Likely via the Janus (two faced) view of the file system as ordinary files and directories for old software.  Ideally new versions of ls, chmod (or a completely new permission system), etc, would be needed for native use.

> * Will it be possible to make a attribute file the child of
> /pub/some_ordinary_directory via a hard link? Will it be possible to make an
> attribute file into an attribute file for some other file? That has completely
> different permissions? And is owned by a different user? What will have to
> happen, in either case, at the time the file is hard-linked to its new parent?

Attributes are files (actually what I call fildirute Things - file directory and attribute all at once) like any other Thing.  So presumably the same hard linking and permission rights apply to them as you already have for conventional files (the permission is part of the fildirute Thing, perhaps via a plug-in).  Isn't Hans Reiser doing some research explicitly on access security?

- Alex

Re: Carrying Attributes too Far

[lrc1]

Quoting "Alexander G. M. Smith" <agmsmith@rogers.com>:

<snip />

> lrc1@st-andrews.ac.uk wrote on Mon, 22 Sep 2003 14:28:30 +0100:
> > * Will
> > 
> > 	chmod u-rwx somefile; chmod u+rwx somefile
> > 
> > still work? What special-case behaviour will be needed to make it work?
> 
> Likely via the Janus (two faced) view of the file system as ordinary files
> and directories for old software.  Ideally new versions of ls, chmod (or a
> completely new permission system), etc, would be needed for native use.
> 

The question wasn't "Will there still be a chmod?". Of course there can still 
be a chmod call in the legacy interface, or a chmodlike "convenience method" 
call in the new one. The question is whether the subfile metadata system will 
be compatible with permissions systems in which a user is able to revoke his 
own permissions to a file and then return them again - such as the current Unix 
permissions system - and what bodges you would accept to make it compatible. Be 
more specific - in all these questions, the devil shows itself in the detail.

> > * Will it be possible to make a attribute file the child of
> > /pub/some_ordinary_directory via a hard link? Will it be possible to make
> an
> > attribute file into an attribute file for some other file? That has
> completely
> > different permissions? And is owned by a different user? What will have to
> > happen, in either case, at the time the file is hard-linked to its new
> parent?
> 
> Attributes are files (actually what I call fildirute Things - file directory
> and attribute all at once) like any other Thing.  So presumably the same hard
> linking and permission rights apply to them as you already have for
> conventional files 

I agree that attribute files should be files like any other. So attribute files 
should certainly have the same linking and permissions rights as any other 
file. How would you implement that?

>(the permission is part of the fildirute Thing, perhaps
> via a plug-in).

<snip />

If attribute files have a different permissions system to all other files, how 
are they files like any other?

If a file is both an attribute file and a child 
of /pub/some_ordinary_directory , should it have the permissions system of an 
ordinary file or an attribute file? What are the consequences of either choice?

If an attribute file's permissions metadata is part of it, is the attribute 
file really a simple sequence of bytes like any other file? If it is, then what 
will prevent any user with write permissions to the attribute file from 
changing the part of the file that records its permissions metadata to whatever 
she wishes?

As to my third question, how would a user go about changing the owner of a 
file? How might she be prevented from doing so?

> - Alex

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Carrying Attributes too Far

[Alexander G. M. Smith]

lrc1@st-andrews.ac.uk wrote on Tue, 23 Sep 2003 02:21:01 +0100:
> The question is whether the subfile metadata system will 
> be compatible with permissions systems in which a user is able to revoke his 
> own permissions to a file and then return them again - such as the current
> Unix  permissions system - and what bodges you would accept to make it
> compatible.

That's a detail for deep thought, but I'll let someone else worry about it for now.

In my shallow thoughts, I'm leaning to having attributes identical to files, and permissions as a kind of attribute.  The attribute's permission would be inherited from the file's permissions, if they don't have their own permission attributes (same inheritance rules for file permissions too).

A simple case would be to have only owners allowed to write, anyone to read (I said it was simple).  Have an "owner" attribute.  Only the matching user can write to a file marked with that attribute.  Only the matching user can write to the file's contents, including data, attributes, which of course includes the owner attribute itself.  Of course, this means the owner can give ownership of the file away to anyone he wants to (and contents if they don't have explicit permission attributes of their own).

That's just an example of a simple security scheme.  A real one would be more sophisticated.  And somebody else's problem :-)

- Alex

Re: Carrying Attributes too Far

[lrc1]

Quoth "Alexander G. M. Smith" <agmsmith@rogers.com>:

> lrc1@st-andrews.ac.uk wrote on Tue, 23 Sep 2003 02:21:01 +0100:
> > The question is whether the subfile metadata system will 
> > be compatible with permissions systems in which a user is able to revoke
> his 
> > own permissions to a file and then return them again - such as the current
> > Unix  permissions system - and what bodges you would accept to make it
> > compatible.
> 
> That's a detail for deep thought, but I'll let someone else worry about it
> for now.
> 
> In my shallow thoughts, I'm leaning to having attributes identical to files,
> and permissions as a kind of attribute.  The attribute's permission would be
> inherited from the file's permissions, if they don't have their own
> permission attributes (same inheritance rules for file permissions too).

Yes, this is probably the best approach to subfile metadata, at least on the
face of it. Now consider the case where Alice owns a file with subfile metadata,
and Bob has read access to the file's subfile metadata directory. What should
happen when Bob hard-links one of the files in Alice's file's subfile metadata
directory (you know, /home/alice/mypicture/+/thumbnail or whatever) to one of
his directories (say /home/bob/pictures )? Alternatively, what should prevent
this from happening?

> 
> A simple case would be to have only owners allowed to write, anyone to read
> (I said it was simple).  Have an "owner" attribute.  Only the matching user
> can write to a file marked with that attribute.  Only the matching user can
> write to the file's contents, including data, attributes, which of course
> includes the owner attribute itself.  Of course, this means the owner can
> give ownership of the file away to anyone he wants to (and contents if they
> don't have explicit permission attributes of their own).

And this is a major security hole, yes? Any user who can create files can give
any other user little presents like arbitrary setuid binaries ("and this one's
for you, root") or a few thousand 200MB files of garbage ("Oh dear, I seem to
have used up your quota..."). How would you fix these problems, without
resorting to an even simpler system with no setuid and no quotas? (And a system
without quotas is pretty much at the mercy of untrusted users anyway.)

> 
> That's just an example of a simple security scheme.  A real one would be more
> sophisticated.  And somebody else's problem :-)
> 

The Unix permissions system isn't exactly the acme of sophistication. Ordinary
implementations of it are pretty straightforward. Isn't it worrying if
implementing it using subfile metadata seems to require some kind of profound
insight? Aren't you concerned that maybe there /is/ no really good solution,
that all possible solutions involve nasty compromises?

> - Alex

Thanks for taking up the challenge, and I look forward to your next attempt. I
apologise for stringing you along; I'm not doing it to be irritating or smug. If
I sat down to describe all the attemptable solutions I can think of, and why
they won't work or are bad news, I'd eventually produce another tedious
multi-thousand-word essay, and I still wouldn't anticipate all possible objections.

Leo.

-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Carrying Attributes too Far

[Hans Reiser]

lrc1@st-andrews.ac.uk wrote:

>Quoting "Alexander G. M. Smith" <agmsmith@rogers.com>:
>
><snip />
>
>  
>
>>lrc1@st-andrews.ac.uk wrote on Mon, 22 Sep 2003 14:28:30 +0100:
>>    
>>
>>>* Will
>>>
>>>	chmod u-rwx somefile; chmod u+rwx somefile
>>>
>>>still work? What special-case behaviour will be needed to make it work?
>>>      
>>>
>>Likely via the Janus (two faced) view of the file system as ordinary files
>>and directories for old software.  Ideally new versions of ls, chmod (or a
>>completely new permission system), etc, would be needed for native use.
>>
>>    
>>
>
>The question wasn't "Will there still be a chmod?". Of course there can still 
>be a chmod call in the legacy interface, or a chmodlike "convenience method" 
>call in the new one. The question is whether the subfile metadata system will 
>be compatible with permissions systems in which a user is able to revoke his 
>own permissions to a file and then return them again - such as the current Unix 
>permissions system - and what bodges you would accept to make it compatible. Be 
>more specific - in all these questions, the devil shows itself in the detail.
>
We will continue to support the VFS interface.

>
>  
>
>>>* Will it be possible to make a attribute file the child of
>>>/pub/some_ordinary_directory via a hard link? Will it be possible to make
>>>      
>>>
>>an
>>    
>>
>>>attribute file into an attribute file for some other file? That has
>>>      
>>>
>>completely
>>    
>>
>>>different permissions? And is owned by a different user? What will have to
>>>happen, in either case, at the time the file is hard-linked to its new
>>>      
>>>
>>parent?
>>
>>Attributes are files (actually what I call fildirute Things - file directory
>>and attribute all at once) like any other Thing.  So presumably the same hard
>>linking and permission rights apply to them as you already have for
>>conventional files 
>>    
>>
>
>I agree that attribute files should be files like any other. So attribute files 
>should certainly have the same linking and permissions rights as any other 
>file.
>
Why?  Not at all, I would say. 

> How would you implement that?
>
>  
>
>>(the permission is part of the fildirute Thing, perhaps
>>via a plug-in).
>>    
>>
>
><snip />
>
>If attribute files have a different permissions system to all other files, how 
>are they files like any other?
>
>If a file is both an attribute file and a child 
>of /pub/some_ordinary_directory , should it have the permissions system of an 
>ordinary file or an attribute file? What are the consequences of either choice?
>
>If an attribute file's permissions metadata is part of it, is the attribute 
>file really a simple sequence of bytes like any other file? If it is, then what 
>will prevent any user with write permissions to the attribute file from 
>changing the part of the file that records its permissions metadata to whatever 
>she wishes?
>
>As to my third question, how would a user go about changing the owner of a 
>file? How might she be prevented from doing so?
>
>  
>
>>- Alex
>>    
>>
>
>Leo.
>
>-----------------------------------------------------------------
>University of St Andrews Webmail: http://webmail.st-andrews.ac.uk
>
>
>  
>


-- 
Hans

Re: Carrying Attributes too Far

[lrc1]

Quoting Hans Reiser <reiser@namesys.com>:
> lrc1@st-andrews.ac.uk wrote:
> >Quoting "Alexander G. M. Smith" <agmsmith@rogers.com>:
> >>lrc1@st-andrews.ac.uk wrote on Mon, 22 Sep 2003 14:28:30 +0100:  

> >I agree that attribute files should be files like any other. So attribute
> files 
> >should certainly have the same linking and permissions rights as any other 
> >file.
> >
> Why?  Not at all, I would say. 

Short answer: because otherwise you have ""attributes" that cannot interact with
files in all the same ways that files can interact with files" (
http://www.namesys.com/v4/v4.html ).

> >>- Alex
> -- 
> Hans

Leo.


-----------------------------------------------------------------
University of St Andrews Webmail: http://webmail.st-andrews.ac.uk

Re: Carrying Attributes too Far

[Hubert Chan]

>>>>> "Leo" == lrc1  <lrc1@st-andrews.ac.uk> writes:

[...]

>> Why?  Not at all, I would say.

Leo> Short answer: because otherwise you have ""attributes" that cannot
Leo> interact with files in all the same ways that files can interact
Leo> with files" ( http://www.namesys.com/v4/v4.html ).

Hard linking isn't a universal attribute, though.  For example, you
cannot hard link over different filesystems.  If an attribute (such as
uid, premissions, etc.) are implemented by just taking the file's
standard stat data and exposing it as a file interface, that it is
similar to that data being from a different filesystem, and so you
can't hardlink.  This would be similar, I suppose, to if a plugin
exposes an MP3's id3 data as files -- it would make no sense to be able
to hardlink that, since the data is stored in the MP3 file, and not as
part of the filesystem.

As for your question about security -- changing file owners and such --
I assume that the filesystem would be able to reject file changes, if
it violated security policy.  I'm not a filesystem programmer, so I
don't know exactly how that would be implemented, but in one of Hans'
papers (probably the Reiser4 paper), where it talks about a plugin for
translating the new /etc/passwd interface into a flat file interface,
it mentions that the filesystem would be able to reject changes if,
e.g. the new version doesn't parse correctly.  So I assume file
ownership would be handled similarly.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: Carrying Attributes too Far

[Hans Reiser]

lrc1@st-andrews.ac.uk wrote:

>Quoting Hans Reiser <reiser@namesys.com>:
>  
>
>>lrc1@st-andrews.ac.uk wrote:
>>    
>>
>>>Quoting "Alexander G. M. Smith" <agmsmith@rogers.com>:
>>>      
>>>
>>>>lrc1@st-andrews.ac.uk wrote on Mon, 22 Sep 2003 14:28:30 +0100:  
>>>>        
>>>>
>
>  
>
>>>I agree that attribute files should be files like any other. So attribute
>>>      
>>>
>>files 
>>    
>>
>>>should certainly have the same linking and permissions rights as any other 
>>>file.
>>>
>>>      
>>>
>>Why?  Not at all, I would say. 
>>    
>>
>
>Short answer: because otherwise you have ""attributes" that cannot interact with
>files in all the same ways that files can interact with files" (
>http://www.namesys.com/v4/v4.html ).
>
no, you just have different permission processing on different files.

>
>  
>
>>>>- Alex
>>>>        
>>>>
>>-- 
>>Hans
>>    
>>
>
>Leo.
>
>
>-----------------------------------------------------------------
>University of St Andrews Webmail: http://webmail.st-andrews.ac.uk
>
>
>  
>


-- 
Hans