Fallthrus as full-length symlinks?

Fallthrus as full-length symlinks?

[Valerie Aurora]

Fallthrus were invented as a placeholders for readdir() on a
union-mounted directory - basically, to use the top-level file
system's readdir() cookie mechanism.  Fallthrus are persistent
directory entries and are implemented by the underlying file system -
such as ext2 or tmpfs - in whatever way it sees fit.  We've
implemented them for ext2 in two ways: as a regular directory entry
with a magic inode number, and as a regular directory entry with a
special file type.

Recently, David Woodhouse suggested implementing fallthrus as
full-length symlinks with a special flag.  The interesting thing about
this idea is that it could theoretically let us rename a file from the
low level file system to another place in the low-level file system
without copying the contents of the file up.  Basically, we can
arbitrarily swizzle the namespace of the low-level by maintaining a
set of symlinks above.

Is this useful?  Is it implementable?

Background reading:

http://valerieaurora.org/union/

-VAL

Re: Fallthrus as full-length symlinks?

[Erez Zadok]

In message <20091113174631.GD19656@shell>, Valerie Aurora writes:
> Fallthrus were invented as a placeholders for readdir() on a
> union-mounted directory - basically, to use the top-level file
> system's readdir() cookie mechanism.  Fallthrus are persistent
> directory entries and are implemented by the underlying file system -
> such as ext2 or tmpfs - in whatever way it sees fit.  We've
> implemented them for ext2 in two ways: as a regular directory entry
> with a magic inode number, and as a regular directory entry with a
> special file type.

Other than a possible improvement to ->rename, what's wrong with the idea of
a special dirent flag?  I kinda liked that idea: it's simple and requires
only a small amount of change to lower file systems.  Any idea in which you
have to record the whiteouts using an actual file or inode is more
cumbersome.

> Recently, David Woodhouse suggested implementing fallthrus as
> full-length symlinks with a special flag.

Where does this "special flag" go?  Is it persistent?  Is it new?  Would
that mean having to change lower file systems to teach them about this flag?

Is there a way of doing it w/o having to change lower f/s code at all?
That'll be a major advantage if possible.

> The interesting thing about
> this idea is that it could theoretically let us rename a file from the
> low level file system to another place in the low-level file system
> without copying the contents of the file up.  Basically, we can
> arbitrarily swizzle the namespace of the low-level by maintaining a
> set of symlinks above.

So now you're proposing to allow something like multiple writeable branches,
in that you allow something other than the topmost branch to be modified.
Moreover, it appears that what you're proposing will need to modify two or
more branches, right?

Maybe I don't understand what this symlink idea is all about.  But I do know
(and have documented it in a TOS article), that ->rename is the most
difficult op to implement in a unioning system.

Getting multiple writeable branches to work reliably in Unionfs had been a
major challenge.  In particular, for rename.  We've first tried to get
rename to be optimal: to rename within the branch a file was in already, or
to minimize modifications across too many layers, to reduce copyup.  But we
found it to be hard to do reliably, esp. when you take into account
whiteouts and opaque directories; even worse when you consider multiple
concurrent renames which could touch a subset of your layers.

See, some unioning operations have to proceed from top to bottom (e.g.,
lookups), while others have to proceed from some middle layer going upwards
(e.g., copyups).  This is a recipe for deadlocks and other races, b/c
there's no clear ordering of operations.  After trying this rename
optimization for several years in unionfs, we gave up on it and just went
with a simpler policy: either you can rename within the same branch, or
copyup the destination into its new name (and optionally whiteout the
source).

I'm not saying it can't be done, but I'm concerned that these symlinks may
add too much complexity to the code.  One would need to think very carefully
about every operation that could affect these symlinks, and how they might
interact with concurrent f/s operations, as well as ops that traverse the
stack in different directions.

Also, aren't you worried about the symlinks cluttering the lower namespace
and consuming inodes and data blocks?  (That has been one of the criticisms
of Unionfs.)

> Is this useful?  Is it implementable?

It's Deja Vu all over again. :-)

> Background reading:
> 
> http://valerieaurora.org/union/
> 
> -VAL

Erez.

Re: Fallthrus as full-length symlinks?

[Arnd Bergmann]

On Friday 13 November 2009, Erez Zadok wrote:
> > The interesting thing about
> > this idea is that it could theoretically let us rename a file from the
> > low level file system to another place in the low-level file system
> > without copying the contents of the file up.  Basically, we can
> > arbitrarily swizzle the namespace of the low-level by maintaining a
> > set of symlinks above.
> 
> So now you're proposing to allow something like multiple writeable branches,
> in that you allow something other than the topmost branch to be modified.
> Moreover, it appears that what you're proposing will need to modify two or
> more branches, right?

I think you misunderstood. If the idea is that you can do a
'mv file_on_lower directory_on_higher/', you would just change the
magic symlink on the higher branch, without writing to the lower
branch, but you are able to work as if you can effectively do
operations that would otherwise require write access on the lower
branch.
 
> > Is this useful?  Is it implementable?

I think it sounds very useful and simple, but I may also have missed something.
One issue might be that with a lot of file systems stacked on top of
each other, you could run out of stack space recursing through the
symlinks, but that's something that may already be the case, or that
can be worked around in other ways.

Another idea that I first had when reading the suggestion was to use a
symlink to self (ln -s x x) as the encoding for a fallthrough. It does
not allow renames like what you really describe, but it has another advantage
in that it does not require extensions to the upper file system layout
while not conflicting with any use case I can see.

	Arnd <><

Re: Fallthrus as full-length symlinks?

[Valerie Aurora]

On Fri, Nov 13, 2009 at 08:55:36PM +0100, Arnd Bergmann wrote:
> On Friday 13 November 2009, Erez Zadok wrote:
> > > The interesting thing about
> > > this idea is that it could theoretically let us rename a file from the
> > > low level file system to another place in the low-level file system
> > > without copying the contents of the file up.  Basically, we can
> > > arbitrarily swizzle the namespace of the low-level by maintaining a
> > > set of symlinks above.
> > 
> > So now you're proposing to allow something like multiple writeable branches,
> > in that you allow something other than the topmost branch to be modified.
> > Moreover, it appears that what you're proposing will need to modify two or
> > more branches, right?
> 
> I think you misunderstood. If the idea is that you can do a
> 'mv file_on_lower directory_on_higher/', you would just change the
> magic symlink on the higher branch, without writing to the lower
> branch, but you are able to work as if you can effectively do
> operations that would otherwise require write access on the lower
> branch.

Arnd is right - only the top layer is modified.  Right now, renaming a
file on the lower layer requires copying up the contents of the file
to the top layer, creating the new link to it, and creating a whiteout
over the old name.  This approach would instead leave the contents of
the file on the lower layer, create a new link to it which contains
the full path of the target in the lower layer, and create a whiteout
over the old name.  All of the modifications happen on the top layer
still.

> > > Is this useful?  Is it implementable?
> 
> I think it sounds very useful and simple, but I may also have missed something.
> One issue might be that with a lot of file systems stacked on top of
> each other, you could run out of stack space recursing through the
> symlinks, but that's something that may already be the case, or that
> can be worked around in other ways.

There would be only one layer of these symlinks possible.

> Another idea that I first had when reading the suggestion was to use a
> symlink to self (ln -s x x) as the encoding for a fallthrough. It does
> not allow renames like what you really describe, but it has another advantage
> in that it does not require extensions to the upper file system layout
> while not conflicting with any use case I can see.

That's certainly worth trying!

-VAL

Re: Fallthrus as full-length symlinks?

[Valerie Aurora]

On Fri, Nov 13, 2009 at 01:46:15PM -0500, Erez Zadok wrote:
> In message <20091113174631.GD19656@shell>, Valerie Aurora writes:
> > Fallthrus were invented as a placeholders for readdir() on a
> > union-mounted directory - basically, to use the top-level file
> > system's readdir() cookie mechanism.  Fallthrus are persistent
> > directory entries and are implemented by the underlying file system -
> > such as ext2 or tmpfs - in whatever way it sees fit.  We've
> > implemented them for ext2 in two ways: as a regular directory entry
> > with a magic inode number, and as a regular directory entry with a
> > special file type.
> 
> Other than a possible improvement to ->rename, what's wrong with the idea of
> a special dirent flag?  I kinda liked that idea: it's simple and requires
> only a small amount of change to lower file systems.  Any idea in which you
> have to record the whiteouts using an actual file or inode is more
> cumbersome.

You have it right - the major advantage is a possible simplification
to rename().

> > Recently, David Woodhouse suggested implementing fallthrus as
> > full-length symlinks with a special flag.
> 
> Where does this "special flag" go?  Is it persistent?  Is it new?  Would
> that mean having to change lower file systems to teach them about this flag?
> 
> Is there a way of doing it w/o having to change lower f/s code at all?
> That'll be a major advantage if possible.

I can't think of a way to do it without using up namespace - but
perhaps there is some part of the symlink target namespace that has no
valid meaning that we could use instead, like Arnd's self symlink.

-VAL

Re: Fallthrus as full-length symlinks?

[David Woodhouse]

On Tue, 2009-11-17 at 14:13 -0500, Valerie Aurora wrote:
> 
> I can't think of a way to do it without using up namespace - but
> perhaps there is some part of the symlink target namespace that has no
> valid meaning that we could use instead, like Arnd's self symlink. 

I think it's impossible to declare that, since the symlink namespace is
fair game for userspace do use as it sees fit. 

Emacs and Mozilla lockfiles being but one example...

[dwmw2@macbook ~]$ ls -l .#foo.html .mozilla/firefox/b8v9tyu0.default/lock 
lrwxrwxrwx. 1 dwmw2 dwmw2 44 2009-11-17 19:16 .#foo.html -> dwmw2@macbook.infradead.org.21271:1258450900
lrwxrwxrwx. 1 dwmw2 dwmw2 19 2009-11-17 09:51 .mozilla/firefox/b8v9tyu0.default/lock -> 90.155.92.212:+3756

I wouldn't want to bet that no userspace app has ever come up with the
idea of using a symlink-to-self as a marker for something. I don't think
it's good practice for us to be trying to play games like that.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: Fallthrus as full-length symlinks?

[Valerie Aurora]

On Tue, Nov 17, 2009 at 07:18:26PM +0000, David Woodhouse wrote:
> On Tue, 2009-11-17 at 14:13 -0500, Valerie Aurora wrote:
> > 
> > I can't think of a way to do it without using up namespace - but
> > perhaps there is some part of the symlink target namespace that has no
> > valid meaning that we could use instead, like Arnd's self symlink. 
> 
> I think it's impossible to declare that, since the symlink namespace is
> fair game for userspace do use as it sees fit. 
> 
> Emacs and Mozilla lockfiles being but one example...
> 
> [dwmw2@macbook ~]$ ls -l .#foo.html .mozilla/firefox/b8v9tyu0.default/lock 
> lrwxrwxrwx. 1 dwmw2 dwmw2 44 2009-11-17 19:16 .#foo.html -> dwmw2@macbook.infradead.org.21271:1258450900
> lrwxrwxrwx. 1 dwmw2 dwmw2 19 2009-11-17 09:51 .mozilla/firefox/b8v9tyu0.default/lock -> 90.155.92.212:+3756
> 
> I wouldn't want to bet that no userspace app has ever come up with the
> idea of using a symlink-to-self as a marker for something. I don't think
> it's good practice for us to be trying to play games like that.

That's my feeling too.  I don't see anyway to cleanly implement
fallthrus (or whiteouts) without explicit support from the file system
on the writable layer.  Fortunately it doesn't take much support.

-VAL

Re: Fallthrus as full-length symlinks?

[Erez Zadok]

In message <20091117194300.GD17822@shell>, Valerie Aurora writes:
[...]
> That's my feeling too.  I don't see anyway to cleanly implement
> fallthrus (or whiteouts) without explicit support from the file system
> on the writable layer.  Fortunately it doesn't take much support.
> 
> -VAL

So, if understood you right, these symlinks are an optimization to simplify
and improve the performance of rename()s in r-o layers.  Ok.

And, people prefer to avoid self-referencing symlinks so as to prevent
breaking some userland code that might depend on it.

But, this does *not* then eliminate the need to have whiteouts supported in
every major f/s that can serve as the writeable layer, right?  You still
need those.

BTW, we might try to figure out a way to use these symlinks to optimize any
copyup that's not strictly necessary.  A rename() doesn't change the file's
data, hence this symlink idea is suitable.  But also, there are other
meta-data changes to a file which don't affect its data (chmod, chown,
chgrp, etc.), for which a symlink would be suitable.  This would require
that we could easily change the meta-data of the symlink itself, and return
*that* metadata in the upper inode, while using the lower file's data for
read().

Erez.

Re: Fallthrus as full-length symlinks?

[Valerie Aurora]

On Tue, Nov 17, 2009 at 03:20:58PM -0500, Erez Zadok wrote:
> In message <20091117194300.GD17822@shell>, Valerie Aurora writes:
> [...]
> > That's my feeling too.  I don't see anyway to cleanly implement
> > fallthrus (or whiteouts) without explicit support from the file system
> > on the writable layer.  Fortunately it doesn't take much support.
> > 
> > -VAL
> 
> So, if understood you right, these symlinks are an optimization to simplify
> and improve the performance of rename()s in r-o layers.  Ok.
> 
> And, people prefer to avoid self-referencing symlinks so as to prevent
> breaking some userland code that might depend on it.
> 
> But, this does *not* then eliminate the need to have whiteouts supported in
> every major f/s that can serve as the writeable layer, right?  You still
> need those.

Yes.  The tradeoff is namespace pollution vs. some new code, and the
consensus is pretty firmly against namespace pollution.

> BTW, we might try to figure out a way to use these symlinks to optimize any
> copyup that's not strictly necessary.  A rename() doesn't change the file's
> data, hence this symlink idea is suitable.  But also, there are other
> meta-data changes to a file which don't affect its data (chmod, chown,
> chgrp, etc.), for which a symlink would be suitable.  This would require
> that we could easily change the meta-data of the symlink itself, and return
> *that* metadata in the upper inode, while using the lower file's data for
> read().

I like this idea.  Copying up the file's data in chown(), etc. is an
enormous pain and hard to work into the existing code path.  It might
be possible to do with this with the directory entry-based approach as
well.

-VAL

Re: Fallthrus as full-length symlinks?

[Arnd Bergmann]

On Monday 23 November 2009, Valerie Aurora wrote:
> > BTW, we might try to figure out a way to use these symlinks to optimize any
> > copyup that's not strictly necessary.  A rename() doesn't change the file's
> > data, hence this symlink idea is suitable.  But also, there are other
> > meta-data changes to a file which don't affect its data (chmod, chown,
> > chgrp, etc.), for which a symlink would be suitable.  This would require
> > that we could easily change the meta-data of the symlink itself, and return
> > that metadata in the upper inode, while using the lower file's data for
> > read().
> 
> I like this idea.  Copying up the file's data in chown(), etc. is an
> enormous pain and hard to work into the existing code path.  It might
> be possible to do with this with the directory entry-based approach as
> well.

I guess we can even support strict atime updates with that, which would be
even more painful to do with copyup because they happen more frequently
than other inode changes. AFAIK the consensus for other union
mount implementations was always that strictatime cannot be sanely
done, or not done persistantly.

	Arnd <><

Re: Fallthrus as full-length symlinks?

[Valerie Aurora]

On Mon, Nov 23, 2009 at 07:44:19PM +0100, Arnd Bergmann wrote:
> On Monday 23 November 2009, Valerie Aurora wrote:
> > > BTW, we might try to figure out a way to use these symlinks to optimize any
> > > copyup that's not strictly necessary.  A rename() doesn't change the file's
> > > data, hence this symlink idea is suitable.  But also, there are other
> > > meta-data changes to a file which don't affect its data (chmod, chown,
> > > chgrp, etc.), for which a symlink would be suitable.  This would require
> > > that we could easily change the meta-data of the symlink itself, and return
> > > that metadata in the upper inode, while using the lower file's data for
> > > read().
> > 
> > I like this idea.  Copying up the file's data in chown(), etc. is an
> > enormous pain and hard to work into the existing code path.  It might
> > be possible to do with this with the directory entry-based approach as
> > well.
> 
> I guess we can even support strict atime updates with that, which would be
> even more painful to do with copyup because they happen more frequently
> than other inode changes. AFAIK the consensus for other union
> mount implementations was always that strictatime cannot be sanely
> done, or not done persistantly.

Okay, this seems really worthwhile to try then.

It seems like we can, as before, create a per-fs DT_FALLTHRU file type
(since we're out of bits for the VFS-level file type).  Then, instead
of reusing the file systems' "normal" directory entry code, we reuse
the symlink code.  As an example, with the current code,
ext2_fallthru_dentry() is a lot of copy-n-paste from ext2_add_link();
in the new version it would look a lot like or call ext2_symlink().

-VAL

Re: Fallthrus as full-length symlinks?

[Miklos Szeredi]

On Mon, 23 Nov 2009, Valerie Aurora wrote:
> On Tue, Nov 17, 2009 at 03:20:58PM -0500, Erez Zadok wrote:
> > But, this does *not* then eliminate the need to have whiteouts supported in
> > every major f/s that can serve as the writeable layer, right?  You still
> > need those.
> 
> Yes.  The tradeoff is namespace pollution vs. some new code, and the
> consensus is pretty firmly against namespace pollution.

You could also use the xattr namespace (trusted.*) to hide that
pollution.  Lots of filesystems already support xattrs, so this might
be easier still than adding new fs features.

Thanks,
Miklos

Re: Fallthrus as full-length symlinks?

[hooanon05]

Valerie Aurora:
> Fallthrus were invented as a placeholders for readdir() on a
> union-mounted directory - basically, to use the top-level file
> system's readdir() cookie mechanism.  Fallthrus are persistent
> directory entries and are implemented by the underlying file system -
> such as ext2 or tmpfs - in whatever way it sees fit.  We've
> implemented them for ext2 in two ways: as a regular directory entry
> with a magic inode number, and as a regular directory entry with a
> special file type.
> 
> Recently, David Woodhouse suggested implementing fallthrus as
> full-length symlinks with a special flag.  The interesting thing about
> this idea is that it could theoretically let us rename a file from the
> low level file system to another place in the low-level file system
> without copying the contents of the file up.  Basically, we can
> arbitrarily swizzle the namespace of the low-level by maintaining a
> set of symlinks above.
> 
> Is this useful?  Is it implementable?

I think the idea of fallthru entry is good, even if it is implemented as
a special symlink.
How do you think about the file paths in /proc/pid/maps and
/proc/pid/fd?
They refer the file paths, and some apps depend upon these path. I
remember that the package manager in debian didn't work when the path is
wrong. (But I don't know whether it is true still).

Will FS have to support such case?


J. R. Okajima

Re: Fallthrus as full-length symlinks?

[Valerie Aurora]

On Wed, Nov 18, 2009 at 02:47:15PM +0900, hooanon05@yahoo.co.jp wrote:
> 
> Valerie Aurora:
> > Fallthrus were invented as a placeholders for readdir() on a
> > union-mounted directory - basically, to use the top-level file
> > system's readdir() cookie mechanism.  Fallthrus are persistent
> > directory entries and are implemented by the underlying file system -
> > such as ext2 or tmpfs - in whatever way it sees fit.  We've
> > implemented them for ext2 in two ways: as a regular directory entry
> > with a magic inode number, and as a regular directory entry with a
> > special file type.
> > 
> > Recently, David Woodhouse suggested implementing fallthrus as
> > full-length symlinks with a special flag.  The interesting thing about
> > this idea is that it could theoretically let us rename a file from the
> > low level file system to another place in the low-level file system
> > without copying the contents of the file up.  Basically, we can
> > arbitrarily swizzle the namespace of the low-level by maintaining a
> > set of symlinks above.
> > 
> > Is this useful?  Is it implementable?
> 
> I think the idea of fallthru entry is good, even if it is implemented as
> a special symlink.
> How do you think about the file paths in /proc/pid/maps and
> /proc/pid/fd?
> They refer the file paths, and some apps depend upon these path. I
> remember that the package manager in debian didn't work when the path is
> wrong. (But I don't know whether it is true still).
> 
> Will FS have to support such case?

My current thinking is that an overlay shouldn't be mounted over /proc
or other special file systems.  I think that a writable layer should
only be mounted over exactly one file system, no submounts allowed.

-VAL

Re: Fallthrus as full-length symlinks?

[hooanon05]

Valerie Aurora:
> On Wed, Nov 18, 2009 at 02:47:15PM +0900, hooanon05@yahoo.co.jp wrote:
	:::
> > I think the idea of fallthru entry is good, even if it is implemented as
> > a special symlink.
> > How do you think about the file paths in /proc/pid/maps and
> > /proc/pid/fd?
> > They refer the file paths, and some apps depend upon these path. I
> > remember that the package manager in debian didn't work when the path is
> > wrong. (But I don't know whether it is true still).
> > 
> > Will FS have to support such case?
> 
> My current thinking is that an overlay shouldn't be mounted over /proc
> or other special file systems.  I think that a writable layer should
> only be mounted over exactly one file system, no submounts allowed.

No, that is not what I mean.
When a fallthru entry is implemented as a kind of a symlink, what will
they look like via /proc/pid/{fd,maps,cwd,exe}?

For example,
$ ln -s /bin/sleep /tmp/awake
$ /tmp/awake 100 &
[1] 32524
$ ls -l /proc/32524/exe
lrwxrwxrwx 1 jro jro 0 Nov 25 11:25 exe -> /bin/sleep*

It shows the target path (real filename) even if I execute it as a
symlink name.

When I invoke /UnionMount/fallthru/a.out, and if /proc/pid/exe
shows something like /real/filesystem/a.out, then some app (debian
start-stop-deamon(8) or something) may not work, I am afraid.

Sorry if my poor English made you misunderstood. But I hope I could make
my question clear now.


J. R. Okajima

Re: Fallthrus as full-length symlinks?

[David Woodhouse]

On Wed, 2009-11-25 at 11:36 +0900, hooanon05@yahoo.co.jp wrote:
> 
> No, that is not what I mean.
> When a fallthru entry is implemented as a kind of a symlink, what will
> they look like via /proc/pid/{fd,maps,cwd,exe}?

In that respect, and many others, it would behave more like a
cross-device hardlink than a symlink.

In fact, I was originally thinking that it would have something like an
nfsfh for the target inode, rather than a device (or fsid?) and
pathname.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: Fallthrus as full-length symlinks?

[AYAN TYAGI]

> >Another idea that I first had when reading the suggestion was to use a
> >symlink to self (ln -s x x) as the encoding for a fallthrough. It does
> >not allow renames like what you really describe, but it has another advantage
> >in that it does not require extensions to the upper file system layout
> >while not conflicting with any use case I can see.

It seems to be a great idea to make use of self referenced symlinks .
Could you please describe the whole process u are proposing?

If possible give some example and code .

Re: Fallthrus as full-length symlinks?

[Jamie Lokier]

AYAN TYAGI wrote:
> > >Another idea that I first had when reading the suggestion was to use a
> > >symlink to self (ln -s x x) as the encoding for a fallthrough. It does
> > >not allow renames like what you really describe, but it has another advantage
> > >in that it does not require extensions to the upper file system layout
> > >while not conflicting with any use case I can see.
> 
> It seems to be a great idea to make use of self referenced symlinks .
> Could you please describe the whole process u are proposing?
> 
> If possible give some example and code .

That'll do surprising things when the user _really_ makes a
self-referencing symlink with "ln -s x x", which can happen
unexpectedly, for example by untarring some archive.

If a fallthrough is encoded that way, there should probably be an
error when the user tries to make a self-referencing symlink.

-- Jamie

Re: Fallthrus as full-length symlinks?

[AYAN TYAGI]

On Tue, Nov 17, 2009 at 1:44 AM, Jamie Lokier <jamie@shareable.org> wrote:
> AYAN TYAGI wrote:
>> > >Another idea that I first had when reading the suggestion was to use a
>> > >symlink to self (ln -s x x) as the encoding for a fallthrough. It does
>> > >not allow renames like what you really describe, but it has another advantage
>> > >in that it does not require extensions to the upper file system layout
>> > >while not conflicting with any use case I can see.
>>
>> It seems to be a great idea to make use of self referenced symlinks .
>> Could you please describe the whole process u are proposing?
>>
>> If possible give some example and code .
>
> That'll do surprising things when the user _really_ makes a
> self-referencing symlink with "ln -s x x", which can happen
> unexpectedly, for example by untarring some archive.
>
> If a fallthrough is encoded that way, there should probably be an
> error when the user tries to make a self-referencing symlink.
>
> -- Jamie
>

>> > The interesting thing about
> > >this idea is that it could theoretically let us rename a file from the
> > >low level file system to another place in the low-level file system
> > >without copying the contents of the file up.  Basically, we can
> > >arbitrarily swizzle the namespace of the low-level by maintaining a
> > >set of symlinks above.

So do you mean that the need of copyup in case of renaming is
virtually eliminated ?
If yes, this would mean that by playing somewhat smart we can even
alter the RO file contents without copyup (that seems to be really
impractical)
Also what changes in the existing model would you envision ? Does the
idea of self referencing symlinks do any good or is it just an eye
candy ?

Does anyone know about any filesystem that implements a similar
paradigm ? btw Aufs implements whiteouts and fallthrus by hard links
...

Re: Fallthrus as full-length symlinks?

[Valerie Aurora]

On Tue, Nov 17, 2009 at 03:03:07AM -0500, AYAN TYAGI wrote:
> On Tue, Nov 17, 2009 at 1:44 AM, Jamie Lokier <jamie@shareable.org> wrote:
> > AYAN TYAGI wrote:
> >> > >Another idea that I first had when reading the suggestion was to use a
> >> > >symlink to self (ln -s x x) as the encoding for a fallthrough. It does
> >> > >not allow renames like what you really describe, but it has another advantage
> >> > >in that it does not require extensions to the upper file system layout
> >> > >while not conflicting with any use case I can see.
> >>
> >> It seems to be a great idea to make use of self referenced symlinks .
> >> Could you please describe the whole process u are proposing?
> >>
> >> If possible give some example and code .
> >
> > That'll do surprising things when the user _really_ makes a
> > self-referencing symlink with "ln -s x x", which can happen
> > unexpectedly, for example by untarring some archive.
> >
> > If a fallthrough is encoded that way, there should probably be an
> > error when the user tries to make a self-referencing symlink.
> >
> > -- Jamie
> >
> 
> >> > The interesting thing about
> > > >this idea is that it could theoretically let us rename a file from the
> > > >low level file system to another place in the low-level file system
> > > >without copying the contents of the file up.  Basically, we can
> > > >arbitrarily swizzle the namespace of the low-level by maintaining a
> > > >set of symlinks above.
> 
> So do you mean that the need of copyup in case of renaming is
> virtually eliminated ?

Only in the case of an unaltered file from the read-only layer being
renamed to a directory that exists in the read-only layer as well.  I
suspect this is not a common case, and most rename()s occur on newly
created files.

-VAL