Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[George Spelvin]

Sorry for being so very late to the party, but rather than messing
with xattrs, why not just have a specific file (say, default /.whiteout,
but selectable via a mount option) and links to it are counted as
whiteout entries?

All you need to do is resolve the link (it's probably a good idea
to allow symlinks to avoid hard-link count limits) and compare the
fs and inode number.

It's kind of hackish, but it could be done on pretty much *any* Unix
file system.  VFAT would be SOL, but that's probably acceptable.

(Any security options on mount point crossing or consistent ownership
of symlinks that you want to impose would probably be of general VFS
use, and again, not FS-specific.)

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Al Viro]

On Wed, Mar 20, 2013 at 03:41:08PM -0400, George Spelvin wrote:
> Sorry for being so very late to the party, but rather than messing
> with xattrs, why not just have a specific file (say, default /.whiteout,
> but selectable via a mount option) and links to it are counted as
> whiteout entries?
> 
> All you need to do is resolve the link (it's probably a good idea
> to allow symlinks to avoid hard-link count limits) and compare the
> fs and inode number.
> 
> It's kind of hackish, but it could be done on pretty much *any* Unix
> file system.  VFAT would be SOL, but that's probably acceptable.
> 
> (Any security options on mount point crossing or consistent ownership
> of symlinks that you want to impose would probably be of general VFS
> use, and again, not FS-specific.)

Yeah...  Now, think what rm -rf foo/ would be doing.  We read the
underlying directory.  For each file in it we create a link to that
magical file of yours in covering one.  _Then_ we do rmdir(), and what a joy
it turns out to be - we
	* lock the covering directory
	* read the covering directory and stat everything in it, checking that
it's a link to your file; we also read the underlying directory and verify
that everything in it has a matching whiteout in the covering one.
	* once we are through, we read it *again*, this time doing unlinks
and praying real hard we won't crash in the meanwhile
	* once that is finished, we finally can rmdir the fucker and unlock
its inode.

I'm sorry, but this is insane.

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Rob Landley]

On 03/20/2013 02:41:08 PM, George Spelvin wrote:
> Sorry for being so very late to the party, but rather than messing
> with xattrs, why not just have a specific file (say, default  
> /.whiteout,
> but selectable via a mount option) and links to it are counted as
> whiteout entries?

Don't the ext2 descendants have an array of reserved inodes at the  
beginning (one of which is the journal for ext3, most of which were  
still unused last I checked)?

Maybe ext4 optimized this away, but that would mean it doesn't have to  
be anywhere in the namespace.

(Of course you'd have to teach fsck about it, and probably not update  
the link count or you'll have contention and integer overflow...)

Rob

[PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

Al and Linus,

Please consider overlayfs for inclusion into 3.10.

It's included in Ubuntu and openSUSE, used by OpenWrt and various other
projects.  I regularly get emails asking when it will be included in mainline.

Git tree is here:

  git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git overlayfs.v16

Thanks,
Miklos

---
Andy Whitcroft (3):
      overlayfs-add-statfs-support
      ovl-switch-to-inode_permission
      overlayfs-copy-up-i_uid-i_gid-from-the-underlying-inode

Erez Zadok (1):
      overlayfs-implement-show_options

Miklos Szeredi (6):
      vfs-add-i_op-dentry_open
      vfs-export-do_splice_direct-to-modules
      vfs-introduce-clone_private_mount
      overlay filesystem
      fs-limit-filesystem-stacking-depth
      vfs-export-inode_permission-to-modules

Neil Brown (1):
      overlay-overlay-filesystem-documentation

Robin Dong (2):
      overlayfs-fix-possible-leak-in-ovl_new_inode
      overlayfs-create-new-inode-in-ovl_link

---
 Documentation/filesystems/Locking       |    2 +
 Documentation/filesystems/overlayfs.txt |  199 +++++++++
 Documentation/filesystems/vfs.txt       |    7 +
 MAINTAINERS                             |    7 +
 fs/Kconfig                              |    1 +
 fs/Makefile                             |    1 +
 fs/ecryptfs/main.c                      |    7 +
 fs/internal.h                           |    5 -
 fs/namei.c                              |   10 +-
 fs/namespace.c                          |   18 +
 fs/open.c                               |   23 +-
 fs/overlayfs/Kconfig                    |    4 +
 fs/overlayfs/Makefile                   |    7 +
 fs/overlayfs/copy_up.c                  |  385 +++++++++++++++++
 fs/overlayfs/dir.c                      |  604 +++++++++++++++++++++++++++
 fs/overlayfs/inode.c                    |  372 +++++++++++++++++
 fs/overlayfs/overlayfs.h                |   70 ++++
 fs/overlayfs/readdir.c                  |  566 +++++++++++++++++++++++++
 fs/overlayfs/super.c                    |  685 +++++++++++++++++++++++++++++++
 fs/splice.c                             |    1 +
 include/linux/fs.h                      |   14 +
 include/linux/mount.h                   |    3 +
 22 files changed, 2981 insertions(+), 10 deletions(-)
 create mode 100644 Documentation/filesystems/overlayfs.txt
 create mode 100644 fs/overlayfs/Kconfig
 create mode 100644 fs/overlayfs/Makefile
 create mode 100644 fs/overlayfs/copy_up.c
 create mode 100644 fs/overlayfs/dir.c
 create mode 100644 fs/overlayfs/inode.c
 create mode 100644 fs/overlayfs/overlayfs.h
 create mode 100644 fs/overlayfs/readdir.c
 create mode 100644 fs/overlayfs/super.c

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Sedat Dilek]

On Tue, Mar 12, 2013 at 4:41 PM, Miklos Szeredi <miklos@szeredi.hu> wrote:
> Al and Linus,
>
> Please consider overlayfs for inclusion into 3.10.
>
> It's included in Ubuntu and openSUSE, used by OpenWrt and various other
> projects.  I regularly get emails asking when it will be included in mainline.
>
> Git tree is here:
>
>   git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git overlayfs.v16
>
> Thanks,
> Miklos
>
> ---
> Andy Whitcroft (3):
>       overlayfs-add-statfs-support
>       ovl-switch-to-inode_permission
>       overlayfs-copy-up-i_uid-i_gid-from-the-underlying-inode
>
> Erez Zadok (1):
>       overlayfs-implement-show_options
>
> Miklos Szeredi (6):
>       vfs-add-i_op-dentry_open
>       vfs-export-do_splice_direct-to-modules
>       vfs-introduce-clone_private_mount
>       overlay filesystem
>       fs-limit-filesystem-stacking-depth
>       vfs-export-inode_permission-to-modules
>
> Neil Brown (1):
>       overlay-overlay-filesystem-documentation
>
> Robin Dong (2):
>       overlayfs-fix-possible-leak-in-ovl_new_inode
>       overlayfs-create-new-inode-in-ovl_link
>

What happened to the subject lines (see v15)?

- Sedat -

[1] http://git.kernel.org/cgit/linux/kernel/git/mszeredi/vfs.git/log/?h=overlayfs.v15

> ---
>  Documentation/filesystems/Locking       |    2 +
>  Documentation/filesystems/overlayfs.txt |  199 +++++++++
>  Documentation/filesystems/vfs.txt       |    7 +
>  MAINTAINERS                             |    7 +
>  fs/Kconfig                              |    1 +
>  fs/Makefile                             |    1 +
>  fs/ecryptfs/main.c                      |    7 +
>  fs/internal.h                           |    5 -
>  fs/namei.c                              |   10 +-
>  fs/namespace.c                          |   18 +
>  fs/open.c                               |   23 +-
>  fs/overlayfs/Kconfig                    |    4 +
>  fs/overlayfs/Makefile                   |    7 +
>  fs/overlayfs/copy_up.c                  |  385 +++++++++++++++++
>  fs/overlayfs/dir.c                      |  604 +++++++++++++++++++++++++++
>  fs/overlayfs/inode.c                    |  372 +++++++++++++++++
>  fs/overlayfs/overlayfs.h                |   70 ++++
>  fs/overlayfs/readdir.c                  |  566 +++++++++++++++++++++++++
>  fs/overlayfs/super.c                    |  685 +++++++++++++++++++++++++++++++
>  fs/splice.c                             |    1 +
>  include/linux/fs.h                      |   14 +
>  include/linux/mount.h                   |    3 +
>  22 files changed, 2981 insertions(+), 10 deletions(-)
>  create mode 100644 Documentation/filesystems/overlayfs.txt
>  create mode 100644 fs/overlayfs/Kconfig
>  create mode 100644 fs/overlayfs/Makefile
>  create mode 100644 fs/overlayfs/copy_up.c
>  create mode 100644 fs/overlayfs/dir.c
>  create mode 100644 fs/overlayfs/inode.c
>  create mode 100644 fs/overlayfs/overlayfs.h
>  create mode 100644 fs/overlayfs/readdir.c
>  create mode 100644 fs/overlayfs/super.c
>

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

On Tue, Mar 12, 2013 at 5:49 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> What happened to the subject lines (see v15)?
>

Oops, will fix.  Thanks.

Miklos

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[J. R. Okajima]

Miklos Szeredi:
> Please consider overlayfs for inclusion into 3.10.

Thank you for CCing me.

First, I'd suggest you to follow some recent activities in mainline
kernel such as
- MODULE_ALIAS_FS
- file_inode()
- d_weak_revalidate() which may not be necessary for overlayfs as long
  as it prohibit users the direct-access to layers (bypassing
  overlayfs).


> It's included in Ubuntu and openSUSE, used by OpenWrt and various other
> projects.  I regularly get emails asking when it will be included in mainline.

Such situation is very similar which AUFS had a few years ago. At that
time, AUFS was rejected since UnionMount was chosen. Years passed, the
development of UnionMount seems to stop and linux mainline doesn't have
generic stackable filesytem yet. I had pointed out some defects in
overlayfs (and UnionMount too). They are all based upon the "name-based"
behaviour instead of "inode-based" one. Other than "Non-standard
behavior" in the overlayfs document, the are, for example,
- read(2) may get an obsoleted filedata (fstat(2) for metadata too).
- fcntl(F_SETLK) may be broken by copy-up.
- inotify may not work when it refers to the file before being
  copied-up.
- unnecessary copy-up may happen, for example mmap(MAP_PRIVATE) after
  open(O_RDWR).

AUFS is an "inode-based" stackable filesystem and solved them many years
ago. But I have to admit that AUFS is big. Yes it is grown up.
I don't stop including overlayfs into mainline, but if the development
of UnionMount is really stopped, then I'd ask people to consider merging
aufs as well as overlayfs.


J. R. Okajima

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

On Tue, Mar 12, 2013 at 6:22 PM, J. R. Okajima <hooanon05@yahoo.co.jp> wrote:
>
> Miklos Szeredi:
>> Please consider overlayfs for inclusion into 3.10.
>
> Thank you for CCing me.
>
> First, I'd suggest you to follow some recent activities in mainline
> kernel such as
> - MODULE_ALIAS_FS
> - file_inode()

Okay, thanks for the comments.  Will fix.

Miklos

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Linus Torvalds]

On Tue, Mar 12, 2013 at 8:41 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:
> Al and Linus,
>
> Please consider overlayfs for inclusion into 3.10.

Yes, I think we should just do it. It's in use, it's pretty small, and
the other alternatives are worse. Let's just plan on getting this
thing done with.

Al, I realize you may not love this, but can you please give this a
look? People clearly want to use it. In particular the new interfaces,
like the inode ops open function with dentry passed in or whatever?
The changes outside of overlayfs looked fine to me.

Miklos, I hate how you have patches like 09/13 that fix bugs
introduced in earlier versions. Any particular reasons for that kind
of thing?

                Linus

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Al Viro]

On Tue, Mar 12, 2013 at 02:50:02PM -0700, Linus Torvalds wrote:
> On Tue, Mar 12, 2013 at 8:41 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:
> > Al and Linus,
> >
> > Please consider overlayfs for inclusion into 3.10.
> 
> Yes, I think we should just do it. It's in use, it's pretty small, and
> the other alternatives are worse. Let's just plan on getting this
> thing done with.
> 
> Al, I realize you may not love this, but can you please give this a
> look? People clearly want to use it. In particular the new interfaces,
> like the inode ops open function with dentry passed in or whatever?
> The changes outside of overlayfs looked fine to me.

I'll post a review tonight or tomorrow.  FWIW, I was not too happy with
it the last time I looked, but I'll obviously need to reread the whole
thing.

I *have* looked at unionmount lately, and the recent modifications dhowells
is doing there are closing most of my problems with that; on the other hand,
there's no fundamental reason why both can't get merged.  Hell, might as
well resurrect aufs, while we are at it...

union-like things are actually on top of my "things to deal with this cycle"
list, closely folowed by rework of ->readdir().

Miklos, two points:
	* I would very much prefer to deal with that (as well as unionmount and
aufs) as git branches _expected_ to be reordered/rebased/folded/mutilated/etc.
while we are sorting all that stuff out.  For now, let's base them on -rc1.
I expect that vfs.git will grow common stem, with bits and pieces of those
guys getting gradually pulled into it, at which point(s) the rest will be
rebased.
	* what Linus just said about bisectablity
Oh, and the third one - I still owe you a bottle of your choice for sorting
the atomic_open shite out.  Is there any chance you'll be able to attend
LSFS this year?

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Sedat Dilek]

On Tue, Mar 12, 2013 at 11:23 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Tue, Mar 12, 2013 at 02:50:02PM -0700, Linus Torvalds wrote:
>> On Tue, Mar 12, 2013 at 8:41 AM, Miklos Szeredi <miklos@szeredi.hu> wrote:
>> > Al and Linus,
>> >
>> > Please consider overlayfs for inclusion into 3.10.
>>
>> Yes, I think we should just do it. It's in use, it's pretty small, and
>> the other alternatives are worse. Let's just plan on getting this
>> thing done with.
>>
>> Al, I realize you may not love this, but can you please give this a
>> look? People clearly want to use it. In particular the new interfaces,
>> like the inode ops open function with dentry passed in or whatever?
>> The changes outside of overlayfs looked fine to me.
>
> I'll post a review tonight or tomorrow.  FWIW, I was not too happy with
> it the last time I looked, but I'll obviously need to reread the whole
> thing.
>
> I *have* looked at unionmount lately, and the recent modifications dhowells
> is doing there are closing most of my problems with that; on the other hand,
> there's no fundamental reason why both can't get merged.  Hell, might as
> well resurrect aufs, while we are at it...
>
> union-like things are actually on top of my "things to deal with this cycle"
> list, closely folowed by rework of ->readdir().
>
> Miklos, two points:
>         * I would very much prefer to deal with that (as well as unionmount and
> aufs) as git branches _expected_ to be reordered/rebased/folded/mutilated/etc.
> while we are sorting all that stuff out.  For now, let's base them on -rc1.
> I expect that vfs.git will grow common stem, with bits and pieces of those
> guys getting gradually pulled into it, at which point(s) the rest will be
> rebased.
>         * what Linus just said about bisectablity
> Oh, and the third one - I still owe you a bottle of your choice for sorting
> the atomic_open shite out.  Is there any chance you'll be able to attend
> LSFS this year?
>

Hi all,

I only say: Al-lelujah!

Congrats Miklos!

- Sedat -

> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

On Tue, Mar 12, 2013 at 11:23 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> I'll post a review tonight or tomorrow.  FWIW, I was not too happy with
> it the last time I looked, but I'll obviously need to reread the whole
> thing.
>
> I *have* looked at unionmount lately, and the recent modifications dhowells
> is doing there are closing most of my problems with that; on the other hand,
> there's no fundamental reason why both can't get merged.  Hell, might as
> well resurrect aufs, while we are at it...

I haven't seen the union mount patchset for a long time.   David, do
you plan to send out the series for review sometime?

> Oh, and the third one - I still owe you a bottle of your choice for sorting
> the atomic_open shite out.  Is there any chance you'll be able to attend
> LSFS this year?

Can't make it this year.  But I do plan to go to KS in Edinburgh.

Thanks,
Miklos

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Al Viro]

On Tue, Mar 12, 2013 at 10:23:50PM +0000, Al Viro wrote:

> I'll post a review tonight or tomorrow.  FWIW, I was not too happy with
> it the last time I looked, but I'll obviously need to reread the whole
> thing.

OK...  Here's the first pass at that:

* use of xattrs for whiteouts/opaque is a Bloody Bad Idea(tm).  That's one
thing you definitely can share with unionmount.  In particular, the games
with creds you have to pull off in ovl_do_lookup() are very clear indications
that xattr is simply a wrong interface for that.

* I don't see anything that would protect you from attacker playing silly
buggers with upper layer; mount it r/w elsewhere and do some renames...
Note that your ->lookup() relies on having the result of ovl_lookup_real()
remain the child of dentry we'd passed it as the first argument.  What's
there to guarantee that it will remain such?  The similar question goes for
malicious modifications of xattrs...  For that matter, what's to prevent
the same sucker mounted as upper layer in two places, with two unrelated lower
layers?  AFAICS, things will break rather badly if that happens, and I'm not
sure if you avoid deadlocks in such scenario...  Interfering with copyup
in progress is also possible.

* I think you might have an unpleasant problem in your ->setattr(); suppose
you've got through the checks in notify_change() and ovl_setattr() got called.
With ATTR_SIZE present.  OK, you do a truncated copyup; fair enough.  But
then you do notify_change() on upper layer dentry to do the rest of the job.
What happens if that fails?  Moreover, what's to prevent it being e.g. opened
by another process *before* you get around to that notify_change() part?

* ->follow_link():  Why the hell do you bother with struct ovl_link_data???
Just to avoid calling ovl_dentry_real() in ovl_put_link()?

BTW, a random note:
        if (err)
                return ERR_PTR(err);

        return NULL;
is a weird way to spell return ERR_PTR(err) - ERR_PTR(0) *is* NULL, TYVM,
and we rely on that in a lot of places.

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

On Wed, Mar 13, 2013 at 7:52 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Tue, Mar 12, 2013 at 10:23:50PM +0000, Al Viro wrote:
>
>> I'll post a review tonight or tomorrow.  FWIW, I was not too happy with
>> it the last time I looked, but I'll obviously need to reread the whole
>> thing.
>
> OK...  Here's the first pass at that:
>
> * use of xattrs for whiteouts/opaque is a Bloody Bad Idea(tm).  That's one
> thing you definitely can share with unionmount.  In particular, the games
> with creds you have to pull off in ovl_do_lookup() are very clear indications
> that xattr is simply a wrong interface for that.

I'm not so sure.  The advantages of using xattrs are:

 - already available on lots of filesystems
 - no new, backward incompatible disk-format needed
 - no new userspace interfaces needed to view/manipluate them

Just looked, union mounts supports patchset has whiteout support for
tmpfs, ext2 and jffs2.  That means I can't try it out on my root
filesystem (and couldn't even if ext4 were supported, because of the
incompatible changes required).

>
> * I don't see anything that would protect you from attacker playing silly
> buggers with upper layer; mount it r/w elsewhere and do some renames...
> Note that your ->lookup() relies on having the result of ovl_lookup_real()
> remain the child of dentry we'd passed it as the first argument.  What's
> there to guarantee that it will remain such?  The similar question goes for
> malicious modifications of xattrs...

All of the above are privileged (xattrs are in the "trusted."
namespace).  Behavior is unspecified in those cases, and "attacker"
can do no worse than confuse itself.

BTW. I had plans of adding infrastructure to make sure that only one
instance of "upper" is writable and that instance is only visible
through the overlay.  But then decided to postpone this as it didn't
seem to be very important.

>  For that matter, what's to prevent
> the same sucker mounted as upper layer in two places, with two unrelated lower
> layers?  AFAICS, things will break rather badly if that happens, and I'm not
> sure if you avoid deadlocks in such scenario...  Interfering with copyup
> in progress is also possible.

I don't see how that would deadlock.  We follow VFS locking rules on
upper and lower filesystem and never lock both at the same time.  And
we
only lock overlay first and then upper *or* lower.

As for same upper on unrelated lower: just don't do it.  As I said, we
could enforce this, but I don't think this is top priority.

>
> * I think you might have an unpleasant problem in your ->setattr(); suppose
> you've got through the checks in notify_change() and ovl_setattr() got called.
> With ATTR_SIZE present.  OK, you do a truncated copyup; fair enough.  But
> then you do notify_change() on upper layer dentry to do the rest of the job.
> What happens if that fails?  Moreover, what's to prevent it being e.g. opened
> by another process *before* you get around to that notify_change() part?

Okay, the truncated copyup doesn't seem to be a good idea.  Or rather,
it needs to be done more carefully, only actually revealing the copied
up dentry once the setattr operation has completed successfully.

>
> * ->follow_link():  Why the hell do you bother with struct ovl_link_data???
> Just to avoid calling ovl_dentry_real() in ovl_put_link()?

Yes, a copy-up between ovl_follow_link and ovl_put_link will break that.

Thanks,
Miklos

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Al Viro]

On Wed, Mar 13, 2013 at 11:09:07PM +0100, Miklos Szeredi wrote:
> I don't see how that would deadlock.  We follow VFS locking rules on
> upper and lower filesystem and never lock both at the same time.  And
> we
> only lock overlay first and then upper *or* lower.
> 
> As for same upper on unrelated lower: just don't do it.  As I said, we
> could enforce this, but I don't think this is top priority.

Tell that to container crowd - they seem to be hell-bent on making everything
mount-related non-priveleged ;-/

FWIW, the thing that worries me is the fun involved in situations when
topology of dentry tree of your fs becomes completely unrelated to that
of one (or both) layers; if somebody can start playing with cross-directory
renames in said layers, things can get really nasty.  Another thing is,
you cache some properties of underlying directories in your tree; are you
sure that this cached information becoming stale will _not_ do anything
bad?

> > * ->follow_link():  Why the hell do you bother with struct ovl_link_data???
> > Just to avoid calling ovl_dentry_real() in ovl_put_link()?
> 
> Yes, a copy-up between ovl_follow_link and ovl_put_link will break that.

*blink*

Er...  What's wrong with simply unhashing the sucker on copyup if it's
a symlink?

BTW, looking at your ovl_copy_up() - you do realize that dget_parent(d)
does *not* guarantee that returned dentry will remain the parent of d?
rename() can very well move it away just as dget_parent() is returning
to caller.  As the result, you are not guaranteed that ovl_copy_up_one()
arguments will be anywhere near each other in the tree.  Your locking
and rechecks might be enough to prevent trouble there, but it's not
obvious, to put it mildly.

I'm _very_ sceptical about the idea of delaying copyups that much, BTW;
there's a damn good reason why all implementations starting with Sun's
one in 80s did copy directories up as soon as they got looked up.  Saves
a lot of headache...

As for whiteouts... I think we ought to pull these bits of unionmount
queue into the common stem and add the missing filesystems to them;
ext* and ufs are trivial (keep in mind that FFS derivatives, including
ext*, have d_type in directory entry and type 14 (DT_WHT) is there
precisely for that purpose).  btrfs also has "dir_type" thing - 8bit
field...

Note that upper layer in *any* union would better not change unpredictably
under you - anybody trying to do e.g. NFS as top (either, actually) layer
is welcome to all kinds of PITA when it comes to need to revalidate stale
dentries.  IOW, realistically the upper layer is going to be local,
read-write and not something like sysfs or procfs, where things can disappear
at will.  What does "having xattr is enough" really buy?

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

On Thu, Mar 14, 2013 at 12:19 AM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Wed, Mar 13, 2013 at 11:09:07PM +0100, Miklos Szeredi wrote:

>> As for same upper on unrelated lower: just don't do it.  As I said, we
>> could enforce this, but I don't think this is top priority.
>
> Tell that to container crowd - they seem to be hell-bent on making everything
> mount-related non-priveleged ;-/

Which is good, but it does need some care.  I'm happy to review those changes.

>> > * ->follow_link():  Why the hell do you bother with struct ovl_link_data???
>> > Just to avoid calling ovl_dentry_real() in ovl_put_link()?
>>
>> Yes, a copy-up between ovl_follow_link and ovl_put_link will break that.
>
> *blink*
>
> Er...  What's wrong with simply unhashing the sucker on copyup if it's
> a symlink?

Nothing, so I'll do that.  Actually we can do that for all except
directory dentries and save some worry.

> BTW, looking at your ovl_copy_up() - you do realize that dget_parent(d)
> does *not* guarantee that returned dentry will remain the parent of d?
> rename() can very well move it away just as dget_parent() is returning
> to caller.  As the result, you are not guaranteed that ovl_copy_up_one()
> arguments will be anywhere near each other in the tree.  Your locking
> and rechecks might be enough to prevent trouble there, but it's not
> obvious, to put it mildly.

This issue is documented above ovl_copy_up_one().  It's not all that
complicated, I think.

> I'm _very_ sceptical about the idea of delaying copyups that much, BTW;
> there's a damn good reason why all implementations starting with Sun's
> one in 80s did copy directories up as soon as they got looked up.  Saves
> a lot of headache...

Maybe.   If we find not trivially fixable holes in the current
implementation I'm open to that direction.

Delayed copy up has the advantage of allowing pure read-only overlays.

> As for whiteouts... I think we ought to pull these bits of unionmoun
> queue into the common stem and add the missing filesystems to them;
> ext* and ufs are trivial (keep in mind that FFS derivatives, including
> ext*, have d_type in directory entry and type 14 (DT_WHT) is there
> precisely for that purpose).  btrfs also has "dir_type" thing - 8bit
> field...

What about userspace interfaces?  Are we allowed to extend d_type and
st_mode without breaking things?

Thanks,
Miklos

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Al Viro]

On Thu, Mar 14, 2013 at 11:37:50AM +0100, Miklos Szeredi wrote:

> > As for whiteouts... I think we ought to pull these bits of unionmoun
> > queue into the common stem and add the missing filesystems to them;
> > ext* and ufs are trivial (keep in mind that FFS derivatives, including
> > ext*, have d_type in directory entry and type 14 (DT_WHT) is there
> > precisely for that purpose).  btrfs also has "dir_type" thing - 8bit
> > field...
> 
> What about userspace interfaces?  Are we allowed to extend d_type and
> st_mode without breaking things?

Huh?
	* from st_mode point of view, it's not going to conflict with
anything; FFS "entry type" matches bits 12..15 of mode_t, and the value
picked by whoever had first implemented whiteouts had been chosen so
that it would not clash with any existing values.  We have
#define S_IFMT  00170000
#define S_IFSOCK 0140000
#define S_IFLNK  0120000
#define S_IFREG  0100000
#define S_IFBLK  0060000
#define S_IFDIR  0040000
#define S_IFCHR  0020000
#define S_IFIFO  0010000
and this sucker would've been 0160000; new filesystem object types are not
frequently introduced, to put it mildly, so I wouldn't expect clashes.
Especially since any such clash would hit Solaris and *BSD (including
MacOS X) as well - all of them use that value for whiteouts.
	* for practically all syscalls these guys are treated as file not
being there; e.g. mkdir() on such name results in whiteout silently
replaced with references to newly created subdirectory, etc.
	* BSDs have one major exposure of those guys to userland - their
getdents(2) shows whiteouts (with d_type equal to DT_WHT and d_fileno - to 1).
Other than that (and we obviously do _not_ want that as default behaviour;
maybe a new O_<something> to produce that), there's practically nothing -
undelete(2) removes a whiteout in attempt to expose the object masked by
it, mknod(2) with S_IFWHT as type does create one.  That's it.

IOW, I don't see how we'd get any kind of userland problems with those.  And
this is not a new API - a bunch of BSD-derived Unices starting with, IIRC,
SunOS 3 and 4.3BSD, as well as their bastard progeny (Solaris, MacOS X)
had that since mid-80s...

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

On Thu, Mar 14, 2013 at 11:59 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> Huh?
>         * from st_mode point of view, it's not going to conflict with
> anything; FFS "entry type" matches bits 12..15 of mode_t, and the value
> picked by whoever had first implemented whiteouts had been chosen so
> that it would not clash with any existing values.  We have
> #define S_IFMT  00170000
> #define S_IFSOCK 0140000
> #define S_IFLNK  0120000
> #define S_IFREG  0100000
> #define S_IFBLK  0060000
> #define S_IFDIR  0040000
> #define S_IFCHR  0020000
> #define S_IFIFO  0010000
> and this sucker would've been 0160000; new filesystem object types are not
> frequently introduced, to put it mildly, so I wouldn't expect clashes.

I'm worried exactly because new filetypes are introduced so
infrequently.  No such thing happened during the lifetime of Linux,
AFAICT.  Backup/restore tools are not going to handle it.  File
managers are not going to show anything sane (and quite possibly some
will simply crash).

Yes, all that can be fixed, but it will be a slow and painful process,
since union/overlay type filesystems are themselves quite specialized
and problems are not going to be shaken out quickly with userspace
interaction.  On the other hand if a whiteout is exported to userspace
as a symlink with xattrs, there will be much less of those problems.

BTW I'm not against adding whiteout support to the VFS and possibly to
filesystems to clean up the security uglilness and optimize storage.

But I think the most important aspect is the userspace interface and
I'm far from sure that a new filetype is the best solution here.

Thanks,
Miklos

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Sedat Dilek]

On Tue, Mar 12, 2013 at 4:41 PM, Miklos Szeredi <miklos@szeredi.hu> wrote:
> Al and Linus,
>
> Please consider overlayfs for inclusion into 3.10.
>
> It's included in Ubuntu and openSUSE, used by OpenWrt and various other
> projects.  I regularly get emails asking when it will be included in mainline.
>
> Git tree is here:
>
>   git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git overlayfs.v16
>
> Thanks,
> Miklos
>

Hi Miklos,

Is it possible to get a backport of OverlayFS-v16 for Linux-stable v3.8.y?
Thanks in advance.

- Sedat -

Re: [PATCH 00/13] overlay filesystem: request for inclusion (v16)

[Miklos Szeredi]

On Wed, Mar 13, 2013 at 11:09 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> On Tue, Mar 12, 2013 at 4:41 PM, Miklos Szeredi <miklos@szeredi.hu> wrote:
>> Al and Linus,
>>
>> Please consider overlayfs for inclusion into 3.10.
>>
>> It's included in Ubuntu and openSUSE, used by OpenWrt and various other
>> projects.  I regularly get emails asking when it will be included in mainline.
>>
>> Git tree is here:
>>
>>   git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/vfs.git overlayfs.v16
>>
>> Thanks,
>> Miklos
>>
>
> Hi Miklos,
>
> Is it possible to get a backport of OverlayFS-v16 for Linux-stable v3.8.y?
> Thanks in advance.

v15 is the backport, or rather, v16 is a forward port.  I changed
nothing else.  I'll do a .v17 with the comments addressed.

Thanks,
Miklos