[PATCH 0/4] Support nested overlayfs mounts

[PATCH 0/4] Support nested overlayfs mounts

[Alexander Larsson]

There are cases where you want to use an overlayfs mount as a lowerdir for
another overlayfs mount. For example, if the system rootfs is on overlayfs due
to composefs, or to make it volatile (via tmps), then you cannot currently store
a lowerdir on the rootfs, becasue the inner overlayfs will eat all the whiteouts
and overlay xattrs. This means you can't e.g. store on the rootfs a prepared
container image for use using overlayfs.

This patch series adds support for nesting of overlayfs mounts by escaping the
problematic features on and unescaping them when exposing to the overlayfs user.

This series is also available here:
  https://github.com/alexlarsson/linux/tree/ovl-nesting

And xfstest to test it is available here:
  https://github.com/alexlarsson/xfstests/tree/overlayfs-nesting

Alexander Larsson (4):
  ovl: Add OVL_XATTR_TRUSTED/USER_PREFIX_LEN macros
  ovl: Support escaped overlay.* xattrs
  ovl: Support creation of whiteout files on overlayfs
  ovl: Add documentation on nesting of overlayfs mounts

 Documentation/filesystems/overlayfs.rst | 22 ++++++++++
 fs/overlayfs/dir.c                      | 14 ++++---
 fs/overlayfs/inode.c                    | 31 +++++++++++++--
 fs/overlayfs/namei.c                    | 14 ++++---
 fs/overlayfs/overlayfs.h                | 22 ++++++++++
 fs/overlayfs/readdir.c                  |  7 +++-
 fs/overlayfs/super.c                    | 53 +++++++++++++++++++++++--
 fs/overlayfs/util.c                     | 20 ++++++++++
 8 files changed, 164 insertions(+), 19 deletions(-)

-- 
2.41.0

[PATCH 1/4] ovl: Add OVL_XATTR_TRUSTED/USER_PREFIX_LEN macros

[Alexander Larsson]

These match the ones for e.g. XATTR_TRUSTED_PREFIX_LEN.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
---
 fs/overlayfs/inode.c     | 4 ++--
 fs/overlayfs/overlayfs.h | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/overlayfs/inode.c b/fs/overlayfs/inode.c
index b395cd84bfce..2dccf3f7fcbe 100644
--- a/fs/overlayfs/inode.c
+++ b/fs/overlayfs/inode.c
@@ -345,10 +345,10 @@ bool ovl_is_private_xattr(struct super_block *sb, const char *name)
 
 	if (ofs->config.userxattr)
 		return strncmp(name, OVL_XATTR_USER_PREFIX,
-			       sizeof(OVL_XATTR_USER_PREFIX) - 1) == 0;
+			       OVL_XATTR_USER_PREFIX_LEN) == 0;
 	else
 		return strncmp(name, OVL_XATTR_TRUSTED_PREFIX,
-			       sizeof(OVL_XATTR_TRUSTED_PREFIX) - 1) == 0;
+			       OVL_XATTR_TRUSTED_PREFIX_LEN) == 0;
 }
 
 int ovl_xattr_set(struct dentry *dentry, struct inode *inode, const char *name,
diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
index 72f57d919aa9..33f88b524627 100644
--- a/fs/overlayfs/overlayfs.h
+++ b/fs/overlayfs/overlayfs.h
@@ -28,7 +28,9 @@ enum ovl_path_type {
 
 #define OVL_XATTR_NAMESPACE "overlay."
 #define OVL_XATTR_TRUSTED_PREFIX XATTR_TRUSTED_PREFIX OVL_XATTR_NAMESPACE
+#define OVL_XATTR_TRUSTED_PREFIX_LEN (sizeof(OVL_XATTR_TRUSTED_PREFIX) - 1)
 #define OVL_XATTR_USER_PREFIX XATTR_USER_PREFIX OVL_XATTR_NAMESPACE
+#define OVL_XATTR_USER_PREFIX_LEN (sizeof(OVL_XATTR_USER_PREFIX) - 1)
 
 enum ovl_xattr {
 	OVL_XATTR_OPAQUE,
-- 
2.41.0

[PATCH 2/4] ovl: Support escaped overlay.* xattrs

[Alexander Larsson]

There are cases where you want to use an overlayfs mount as a lowerdir
for another overlayfs mount. For example, if the system rootfs is on
overlayfs due to composefs, or to make it volatile (via tmps), then
you cannot currently store a lowerdir on the rootfs. This means you
can't e.g. store on the rootfs a prepared container image for use
using overlayfs.

To work around this, we introduce an escapment mechanism for overlayfs
xattrs. Whenever the lower/upper dir has a xattr named
`overlay.overlay.XYZ`, we list it as overlay.XYZ in listxattrs, and
when the user calls getxattr or setxattr on `overlay.XYZ`, we apply to
`overlay.overlay.XYZ` in the backing directories.

This allows storing any kind of overlay xattrs in a overlayfs mount
that can be used as a lowerdir in another mount. It is possible to
stack this mechanism multiple times, such that
overlay.overlay.overlay.XYZ will survive two levels of overlay mounts,
however this is not all that useful in practice because of stack depth
limitations of overlayfs mounts.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
---
 fs/overlayfs/inode.c     | 27 +++++++++++++++++++--
 fs/overlayfs/overlayfs.h |  7 ++++++
 fs/overlayfs/super.c     | 51 ++++++++++++++++++++++++++++++++++++++--
 3 files changed, 81 insertions(+), 4 deletions(-)

diff --git a/fs/overlayfs/inode.c b/fs/overlayfs/inode.c
index 2dccf3f7fcbe..743951c11534 100644
--- a/fs/overlayfs/inode.c
+++ b/fs/overlayfs/inode.c
@@ -339,6 +339,18 @@ static const char *ovl_get_link(struct dentry *dentry,
 	return p;
 }
 
+bool ovl_is_escaped_xattr(struct super_block *sb, const char *name)
+{
+	struct ovl_fs *ofs = sb->s_fs_info;
+
+	if (ofs->config.userxattr)
+		return strncmp(name, OVL_XATTR_ESCAPE_USER_PREFIX,
+			       OVL_XATTR_ESCAPE_USER_PREFIX_LEN) == 0;
+	else
+		return strncmp(name, OVL_XATTR_ESCAPE_TRUSTED_PREFIX,
+			       OVL_XATTR_ESCAPE_TRUSTED_PREFIX_LEN - 1) == 0;
+}
+
 bool ovl_is_private_xattr(struct super_block *sb, const char *name)
 {
 	struct ovl_fs *ofs = OVL_FS(sb);
@@ -417,8 +429,8 @@ int ovl_xattr_get(struct dentry *dentry, struct inode *inode, const char *name,
 
 static bool ovl_can_list(struct super_block *sb, const char *s)
 {
-	/* Never list private (.overlay) */
-	if (ovl_is_private_xattr(sb, s))
+	/* Never list non-escaped private (.overlay) */
+	if (ovl_is_private_xattr(sb, s) && !ovl_is_escaped_xattr(sb, s))
 		return false;
 
 	/* List all non-trusted xattrs */
@@ -432,10 +444,12 @@ static bool ovl_can_list(struct super_block *sb, const char *s)
 ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)
 {
 	struct dentry *realdentry = ovl_dentry_real(dentry);
+	struct ovl_fs *ofs = OVL_FS(dentry->d_sb);
 	ssize_t res;
 	size_t len;
 	char *s;
 	const struct cred *old_cred;
+	size_t prefix_len;
 
 	old_cred = ovl_override_creds(dentry->d_sb);
 	res = vfs_listxattr(realdentry, list, size);
@@ -443,6 +457,9 @@ ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)
 	if (res <= 0 || size == 0)
 		return res;
 
+	prefix_len = ofs->config.userxattr ?
+		OVL_XATTR_USER_PREFIX_LEN : OVL_XATTR_TRUSTED_PREFIX_LEN;
+
 	/* filter out private xattrs */
 	for (s = list, len = res; len;) {
 		size_t slen = strnlen(s, len) + 1;
@@ -455,6 +472,12 @@ ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)
 		if (!ovl_can_list(dentry->d_sb, s)) {
 			res -= slen;
 			memmove(s, s + slen, len);
+		} else if (ovl_is_escaped_xattr(dentry->d_sb, s)) {
+			memmove(s + prefix_len,
+				s + prefix_len + OVL_XATTR_ESCAPE_PREFIX_LEN,
+				slen - (prefix_len + OVL_XATTR_ESCAPE_PREFIX_LEN) + len);
+			res -= OVL_XATTR_ESCAPE_PREFIX_LEN;
+			s += slen - OVL_XATTR_ESCAPE_PREFIX_LEN;
 		} else {
 			s += slen;
 		}
diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
index 33f88b524627..1dbd01719f63 100644
--- a/fs/overlayfs/overlayfs.h
+++ b/fs/overlayfs/overlayfs.h
@@ -32,6 +32,13 @@ enum ovl_path_type {
 #define OVL_XATTR_USER_PREFIX XATTR_USER_PREFIX OVL_XATTR_NAMESPACE
 #define OVL_XATTR_USER_PREFIX_LEN (sizeof(OVL_XATTR_USER_PREFIX) - 1)
 
+#define OVL_XATTR_ESCAPE_PREFIX OVL_XATTR_NAMESPACE
+#define OVL_XATTR_ESCAPE_PREFIX_LEN (sizeof(OVL_XATTR_ESCAPE_PREFIX) - 1)
+#define OVL_XATTR_ESCAPE_TRUSTED_PREFIX OVL_XATTR_TRUSTED_PREFIX OVL_XATTR_ESCAPE_PREFIX
+#define OVL_XATTR_ESCAPE_TRUSTED_PREFIX_LEN (sizeof(OVL_XATTR_ESCAPE_TRUSTED_PREFIX) - 1)
+#define OVL_XATTR_ESCAPE_USER_PREFIX OVL_XATTR_USER_PREFIX OVL_XATTR_ESCAPE_PREFIX
+#define OVL_XATTR_ESCAPE_USER_PREFIX_LEN (sizeof(OVL_XATTR_ESCAPE_USER_PREFIX) - 1)
+
 enum ovl_xattr {
 	OVL_XATTR_OPAQUE,
 	OVL_XATTR_REDIRECT,
diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index def266b5e2a3..97bc94459f7a 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -434,11 +434,47 @@ static bool ovl_workdir_ok(struct dentry *workdir, struct dentry *upperdir)
 	return ok;
 }
 
+static char *ovl_xattr_escape_name(const char *prefix, const char *name)
+{
+	size_t prefix_len = strlen(prefix);
+	size_t name_len = strlen(name);
+	size_t escaped_len;
+	char *escaped, *s;
+
+	escaped_len = prefix_len + OVL_XATTR_ESCAPE_PREFIX_LEN + name_len;
+	if (escaped_len > XATTR_NAME_MAX)
+		return ERR_PTR(-EOPNOTSUPP);
+
+	escaped = kmalloc(escaped_len + 1, GFP_KERNEL);
+	if (escaped == NULL)
+		return ERR_PTR(-ENOMEM);
+
+	s = escaped;
+	memcpy(s, prefix, prefix_len);
+	s += prefix_len;
+	memcpy(s, OVL_XATTR_ESCAPE_PREFIX, OVL_XATTR_ESCAPE_PREFIX_LEN);
+	s += OVL_XATTR_ESCAPE_PREFIX_LEN;
+	memcpy(s, name, name_len + 1);
+
+	return escaped;
+}
+
 static int ovl_own_xattr_get(const struct xattr_handler *handler,
 			     struct dentry *dentry, struct inode *inode,
 			     const char *name, void *buffer, size_t size)
 {
-	return -EOPNOTSUPP;
+	char *escaped;
+	int r;
+
+	escaped = ovl_xattr_escape_name(handler->prefix, name);
+	if (IS_ERR(escaped))
+		return PTR_ERR(escaped);
+
+	r = ovl_xattr_get(dentry, inode, escaped, buffer, size);
+
+	kfree(escaped);
+
+	return r;
 }
 
 static int ovl_own_xattr_set(const struct xattr_handler *handler,
@@ -447,7 +483,18 @@ static int ovl_own_xattr_set(const struct xattr_handler *handler,
 			     const char *name, const void *value,
 			     size_t size, int flags)
 {
-	return -EOPNOTSUPP;
+	char *escaped;
+	int r;
+
+	escaped = ovl_xattr_escape_name(handler->prefix, name);
+	if (IS_ERR(escaped))
+		return PTR_ERR(escaped);
+
+	r = ovl_xattr_set(dentry, inode, escaped, value, size, flags);
+
+	kfree(escaped);
+
+	return r;
 }
 
 static int ovl_other_xattr_get(const struct xattr_handler *handler,
-- 
2.41.0

Re: [PATCH 2/4] ovl: Support escaped overlay.* xattrs

[Amir Goldstein]

On Wed, Aug 16, 2023 at 6:29 PM Alexander Larsson <alexl@redhat.com> wrote:
>
> There are cases where you want to use an overlayfs mount as a lowerdir
> for another overlayfs mount. For example, if the system rootfs is on
> overlayfs due to composefs, or to make it volatile (via tmps), then
> you cannot currently store a lowerdir on the rootfs. This means you
> can't e.g. store on the rootfs a prepared container image for use
> using overlayfs.
>
> To work around this, we introduce an escapment mechanism for overlayfs
> xattrs. Whenever the lower/upper dir has a xattr named
> `overlay.overlay.XYZ`, we list it as overlay.XYZ in listxattrs, and
> when the user calls getxattr or setxattr on `overlay.XYZ`, we apply to
> `overlay.overlay.XYZ` in the backing directories.
>
> This allows storing any kind of overlay xattrs in a overlayfs mount
> that can be used as a lowerdir in another mount. It is possible to
> stack this mechanism multiple times, such that
> overlay.overlay.overlay.XYZ will survive two levels of overlay mounts,
> however this is not all that useful in practice because of stack depth
> limitations of overlayfs mounts.
>
> Signed-off-by: Alexander Larsson <alexl@redhat.com>
> ---
>  fs/overlayfs/inode.c     | 27 +++++++++++++++++++--
>  fs/overlayfs/overlayfs.h |  7 ++++++
>  fs/overlayfs/super.c     | 51 ++++++++++++++++++++++++++++++++++++++--
>  3 files changed, 81 insertions(+), 4 deletions(-)
>
> diff --git a/fs/overlayfs/inode.c b/fs/overlayfs/inode.c
> index 2dccf3f7fcbe..743951c11534 100644
> --- a/fs/overlayfs/inode.c
> +++ b/fs/overlayfs/inode.c
> @@ -339,6 +339,18 @@ static const char *ovl_get_link(struct dentry *dentry,
>         return p;
>  }
>
> +bool ovl_is_escaped_xattr(struct super_block *sb, const char *name)
> +{
> +       struct ovl_fs *ofs = sb->s_fs_info;
> +
> +       if (ofs->config.userxattr)
> +               return strncmp(name, OVL_XATTR_ESCAPE_USER_PREFIX,
> +                              OVL_XATTR_ESCAPE_USER_PREFIX_LEN) == 0;
> +       else
> +               return strncmp(name, OVL_XATTR_ESCAPE_TRUSTED_PREFIX,
> +                              OVL_XATTR_ESCAPE_TRUSTED_PREFIX_LEN - 1) == 0;
> +}
> +
>  bool ovl_is_private_xattr(struct super_block *sb, const char *name)
>  {
>         struct ovl_fs *ofs = OVL_FS(sb);
> @@ -417,8 +429,8 @@ int ovl_xattr_get(struct dentry *dentry, struct inode *inode, const char *name,
>
>  static bool ovl_can_list(struct super_block *sb, const char *s)
>  {
> -       /* Never list private (.overlay) */
> -       if (ovl_is_private_xattr(sb, s))
> +       /* Never list non-escaped private (.overlay) */
> +       if (ovl_is_private_xattr(sb, s) && !ovl_is_escaped_xattr(sb, s))
>                 return false;
>
>         /* List all non-trusted xattrs */
> @@ -432,10 +444,12 @@ static bool ovl_can_list(struct super_block *sb, const char *s)
>  ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)
>  {
>         struct dentry *realdentry = ovl_dentry_real(dentry);
> +       struct ovl_fs *ofs = OVL_FS(dentry->d_sb);
>         ssize_t res;
>         size_t len;
>         char *s;
>         const struct cred *old_cred;
> +       size_t prefix_len;
>
>         old_cred = ovl_override_creds(dentry->d_sb);
>         res = vfs_listxattr(realdentry, list, size);
> @@ -443,6 +457,9 @@ ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)
>         if (res <= 0 || size == 0)
>                 return res;
>
> +       prefix_len = ofs->config.userxattr ?
> +               OVL_XATTR_USER_PREFIX_LEN : OVL_XATTR_TRUSTED_PREFIX_LEN;
> +
>         /* filter out private xattrs */
>         for (s = list, len = res; len;) {
>                 size_t slen = strnlen(s, len) + 1;
> @@ -455,6 +472,12 @@ ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)
>                 if (!ovl_can_list(dentry->d_sb, s)) {
>                         res -= slen;
>                         memmove(s, s + slen, len);
> +               } else if (ovl_is_escaped_xattr(dentry->d_sb, s)) {
> +                       memmove(s + prefix_len,
> +                               s + prefix_len + OVL_XATTR_ESCAPE_PREFIX_LEN,
> +                               slen - (prefix_len + OVL_XATTR_ESCAPE_PREFIX_LEN) + len);
> +                       res -= OVL_XATTR_ESCAPE_PREFIX_LEN;
> +                       s += slen - OVL_XATTR_ESCAPE_PREFIX_LEN;
>                 } else {
>                         s += slen;
>                 }
> diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> index 33f88b524627..1dbd01719f63 100644
> --- a/fs/overlayfs/overlayfs.h
> +++ b/fs/overlayfs/overlayfs.h
> @@ -32,6 +32,13 @@ enum ovl_path_type {
>  #define OVL_XATTR_USER_PREFIX XATTR_USER_PREFIX OVL_XATTR_NAMESPACE
>  #define OVL_XATTR_USER_PREFIX_LEN (sizeof(OVL_XATTR_USER_PREFIX) - 1)
>
> +#define OVL_XATTR_ESCAPE_PREFIX OVL_XATTR_NAMESPACE
> +#define OVL_XATTR_ESCAPE_PREFIX_LEN (sizeof(OVL_XATTR_ESCAPE_PREFIX) - 1)
> +#define OVL_XATTR_ESCAPE_TRUSTED_PREFIX OVL_XATTR_TRUSTED_PREFIX OVL_XATTR_ESCAPE_PREFIX
> +#define OVL_XATTR_ESCAPE_TRUSTED_PREFIX_LEN (sizeof(OVL_XATTR_ESCAPE_TRUSTED_PREFIX) - 1)
> +#define OVL_XATTR_ESCAPE_USER_PREFIX OVL_XATTR_USER_PREFIX OVL_XATTR_ESCAPE_PREFIX
> +#define OVL_XATTR_ESCAPE_USER_PREFIX_LEN (sizeof(OVL_XATTR_ESCAPE_USER_PREFIX) - 1)
> +
>  enum ovl_xattr {
>         OVL_XATTR_OPAQUE,
>         OVL_XATTR_REDIRECT,
> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> index def266b5e2a3..97bc94459f7a 100644
> --- a/fs/overlayfs/super.c
> +++ b/fs/overlayfs/super.c
> @@ -434,11 +434,47 @@ static bool ovl_workdir_ok(struct dentry *workdir, struct dentry *upperdir)
>         return ok;
>  }
>
> +static char *ovl_xattr_escape_name(const char *prefix, const char *name)
> +{
> +       size_t prefix_len = strlen(prefix);
> +       size_t name_len = strlen(name);
> +       size_t escaped_len;
> +       char *escaped, *s;
> +
> +       escaped_len = prefix_len + OVL_XATTR_ESCAPE_PREFIX_LEN + name_len;
> +       if (escaped_len > XATTR_NAME_MAX)
> +               return ERR_PTR(-EOPNOTSUPP);
> +
> +       escaped = kmalloc(escaped_len + 1, GFP_KERNEL);
> +       if (escaped == NULL)
> +               return ERR_PTR(-ENOMEM);
> +
> +       s = escaped;
> +       memcpy(s, prefix, prefix_len);
> +       s += prefix_len;
> +       memcpy(s, OVL_XATTR_ESCAPE_PREFIX, OVL_XATTR_ESCAPE_PREFIX_LEN);
> +       s += OVL_XATTR_ESCAPE_PREFIX_LEN;
> +       memcpy(s, name, name_len + 1);
> +
> +       return escaped;
> +}
> +
>  static int ovl_own_xattr_get(const struct xattr_handler *handler,
>                              struct dentry *dentry, struct inode *inode,
>                              const char *name, void *buffer, size_t size)
>  {
> -       return -EOPNOTSUPP;
> +       char *escaped;
> +       int r;
> +
> +       escaped = ovl_xattr_escape_name(handler->prefix, name);
> +       if (IS_ERR(escaped))
> +               return PTR_ERR(escaped);
> +
> +       r = ovl_xattr_get(dentry, inode, escaped, buffer, size);
> +
> +       kfree(escaped);
> +
> +       return r;
>  }
>
>  static int ovl_own_xattr_set(const struct xattr_handler *handler,
> @@ -447,7 +483,18 @@ static int ovl_own_xattr_set(const struct xattr_handler *handler,
>                              const char *name, const void *value,
>                              size_t size, int flags)
>  {
> -       return -EOPNOTSUPP;
> +       char *escaped;
> +       int r;
> +
> +       escaped = ovl_xattr_escape_name(handler->prefix, name);
> +       if (IS_ERR(escaped))
> +               return PTR_ERR(escaped);
> +
> +       r = ovl_xattr_set(dentry, inode, escaped, value, size, flags);
> +
> +       kfree(escaped);
> +
> +       return r;
>  }
>

I thought I posted those comments on github, but I don't see them -
it would be nice to first move the xattr handlers and get/set/listxattr
implementations to xattr.c file before adding escaping support.

We started shrinking down super.c and I'd hate to see it bloat again.
The way I see it, xattr.c only need to export this helper for super.c:

sb->s_xattr = ovl_xattr_handlers(ofs);

Thanks,
Amir.

[PATCH 3/4] ovl: Support creation of whiteout files on overlayfs

[Alexander Larsson]

This is needed to properly stack overlay filesystems, I.E, being able
to create a whiteout file on an overlay mount and then use that as
part of the lowerdir in another overlay mount.

The way this works is that we create a regular whiteout, but set the
`overlay.nowhiteout` xattr on it. Whenever we check if a file is a
whiteout we check this xattr and don't treat it as a whiteout if it is
set. The xattr itself is then stripped and when viewed as part of the
overlayfs mount it looks like a regular whiteout.

Signed-off-by: Alexander Larsson <alexl@redhat.com>
---
 fs/overlayfs/dir.c       | 14 ++++++++------
 fs/overlayfs/namei.c     | 14 +++++++++-----
 fs/overlayfs/overlayfs.h | 13 +++++++++++++
 fs/overlayfs/readdir.c   |  7 ++++++-
 fs/overlayfs/super.c     |  2 +-
 fs/overlayfs/util.c      | 20 ++++++++++++++++++++
 6 files changed, 57 insertions(+), 13 deletions(-)

diff --git a/fs/overlayfs/dir.c b/fs/overlayfs/dir.c
index 033fc0458a3d..4ef3a473700c 100644
--- a/fs/overlayfs/dir.c
+++ b/fs/overlayfs/dir.c
@@ -199,6 +199,12 @@ struct dentry *ovl_create_real(struct ovl_fs *ofs, struct inode *dir,
 		case S_IFSOCK:
 			err = ovl_do_mknod(ofs, dir, newdentry, attr->mode,
 					   attr->rdev);
+
+			if (!err && S_ISCHR(attr->mode) && attr->rdev == WHITEOUT_DEV) {
+				err = ovl_setxattr(ofs, newdentry, OVL_XATTR_NOWHITEOUT,
+						   NULL, 0);
+			}
+
 			break;
 
 		case S_IFLNK:
@@ -477,7 +483,7 @@ static int ovl_create_over_whiteout(struct dentry *dentry, struct inode *inode,
 		goto out_unlock;
 
 	err = -ESTALE;
-	if (d_is_negative(upper) || !IS_WHITEOUT(d_inode(upper)))
+	if (d_is_negative(upper) || !ovl_upper_is_whiteout(ofs, upper))
 		goto out_dput;
 
 	newdentry = ovl_create_temp(ofs, workdir, cattr);
@@ -669,10 +675,6 @@ static int ovl_mkdir(struct mnt_idmap *idmap, struct inode *dir,
 static int ovl_mknod(struct mnt_idmap *idmap, struct inode *dir,
 		     struct dentry *dentry, umode_t mode, dev_t rdev)
 {
-	/* Don't allow creation of "whiteout" on overlay */
-	if (S_ISCHR(mode) && rdev == WHITEOUT_DEV)
-		return -EPERM;
-
 	return ovl_create_object(dentry, mode, rdev, NULL);
 }
 
@@ -1219,7 +1221,7 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir,
 		}
 	} else {
 		if (!d_is_negative(newdentry)) {
-			if (!new_opaque || !ovl_is_whiteout(newdentry))
+			if (!new_opaque || !ovl_upper_is_whiteout(ofs, newdentry))
 				goto out_dput;
 		} else {
 			if (flags & RENAME_EXCHANGE)
diff --git a/fs/overlayfs/namei.c b/fs/overlayfs/namei.c
index 80391c687c2a..e90167789a13 100644
--- a/fs/overlayfs/namei.c
+++ b/fs/overlayfs/namei.c
@@ -251,7 +251,9 @@ static int ovl_lookup_single(struct dentry *base, struct ovl_lookup_data *d,
 		err = -EREMOTE;
 		goto out_err;
 	}
-	if (ovl_is_whiteout(this)) {
+	path.dentry = this;
+	path.mnt = d->mnt;
+	if (ovl_path_is_whiteout(OVL_FS(d->sb), &path)) {
 		d->stop = d->opaque = true;
 		goto put_and_out;
 	}
@@ -264,8 +266,6 @@ static int ovl_lookup_single(struct dentry *base, struct ovl_lookup_data *d,
 		goto put_and_out;
 	}
 
-	path.dentry = this;
-	path.mnt = d->mnt;
 	if (!d_can_lookup(this)) {
 		if (d->is_dir || !last_element) {
 			d->stop = true;
@@ -438,7 +438,7 @@ int ovl_check_origin_fh(struct ovl_fs *ofs, struct ovl_fh *fh, bool connected,
 	else if (IS_ERR(origin))
 		return PTR_ERR(origin);
 
-	if (upperdentry && !ovl_is_whiteout(upperdentry) &&
+	if (upperdentry && !ovl_upper_is_whiteout(ofs, upperdentry) &&
 	    inode_wrong_type(d_inode(upperdentry), d_inode(origin)->i_mode))
 		goto invalid;
 
@@ -1383,7 +1383,11 @@ bool ovl_lower_positive(struct dentry *dentry)
 				break;
 			}
 		} else {
-			positive = !ovl_is_whiteout(this);
+			struct path path = {
+				.dentry = this,
+				.mnt = parentpath->layer->mnt,
+			};
+			positive = !ovl_path_is_whiteout(OVL_FS(dentry->d_sb), &path);
 			done = true;
 			dput(this);
 		}
diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
index 1dbd01719f63..853335ff26f7 100644
--- a/fs/overlayfs/overlayfs.h
+++ b/fs/overlayfs/overlayfs.h
@@ -49,6 +49,7 @@ enum ovl_xattr {
 	OVL_XATTR_UUID,
 	OVL_XATTR_METACOPY,
 	OVL_XATTR_PROTATTR,
+	OVL_XATTR_NOWHITEOUT,
 };
 
 enum ovl_inode_flag {
@@ -469,16 +470,28 @@ void ovl_inode_update(struct inode *inode, struct dentry *upperdentry);
 void ovl_dir_modified(struct dentry *dentry, bool impurity);
 u64 ovl_inode_version_get(struct inode *inode);
 bool ovl_is_whiteout(struct dentry *dentry);
+bool ovl_path_is_whiteout(struct ovl_fs *ofs, const struct path *path);
 struct file *ovl_path_open(const struct path *path, int flags);
 int ovl_copy_up_start(struct dentry *dentry, int flags);
 void ovl_copy_up_end(struct dentry *dentry);
 bool ovl_already_copied_up(struct dentry *dentry, int flags);
 bool ovl_path_check_dir_xattr(struct ovl_fs *ofs, const struct path *path,
 			      enum ovl_xattr ox);
+bool ovl_path_check_nowhiteout_xattr(struct ovl_fs *ofs, const struct path *path);
 bool ovl_path_check_origin_xattr(struct ovl_fs *ofs, const struct path *path);
 bool ovl_init_uuid_xattr(struct super_block *sb, struct ovl_fs *ofs,
 			 const struct path *upperpath);
 
+static inline bool ovl_upper_is_whiteout(struct ovl_fs *ofs,
+					 struct dentry *upperdentry)
+{
+	struct path upperpath = {
+		.dentry = upperdentry,
+		.mnt = ovl_upper_mnt(ofs),
+	};
+	return ovl_path_is_whiteout(ofs, &upperpath);
+}
+
 static inline bool ovl_check_origin_xattr(struct ovl_fs *ofs,
 					  struct dentry *upperdentry)
 {
diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
index de39e067ae65..9cf8e7e2961c 100644
--- a/fs/overlayfs/readdir.c
+++ b/fs/overlayfs/readdir.c
@@ -280,7 +280,12 @@ static int ovl_check_whiteouts(const struct path *path, struct ovl_readdir_data
 			rdd->first_maybe_whiteout = p->next_maybe_whiteout;
 			dentry = lookup_one(mnt_idmap(path->mnt), p->name, dir, p->len);
 			if (!IS_ERR(dentry)) {
-				p->is_whiteout = ovl_is_whiteout(dentry);
+				struct path childpath = {
+					.dentry = dentry,
+					.mnt = path->mnt,
+				};
+				p->is_whiteout = ovl_path_is_whiteout(OVL_FS(rdd->dentry->d_sb),
+								      &childpath);
 				dput(dentry);
 			}
 		}
diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index 97bc94459f7a..71c650ba5a1a 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -688,7 +688,7 @@ static int ovl_check_rename_whiteout(struct ovl_fs *ofs)
 	if (IS_ERR(whiteout))
 		goto cleanup_temp;
 
-	err = ovl_is_whiteout(whiteout);
+	err = ovl_upper_is_whiteout(ofs, whiteout);
 
 	/* Best effort cleanup of whiteout and temp file */
 	if (err)
diff --git a/fs/overlayfs/util.c b/fs/overlayfs/util.c
index 0f387092450e..da6d2abf64dd 100644
--- a/fs/overlayfs/util.c
+++ b/fs/overlayfs/util.c
@@ -575,6 +575,16 @@ bool ovl_is_whiteout(struct dentry *dentry)
 	return inode && IS_WHITEOUT(inode);
 }
 
+/*
+ * Use this over ovl_is_whiteout for upper and lower files, as it also
+ * handles escaped whiteout files.
+ */
+bool ovl_path_is_whiteout(struct ovl_fs *ofs, const struct path *path)
+{
+	return ovl_is_whiteout(path->dentry) &&
+		!ovl_path_check_nowhiteout_xattr(ofs, path);
+}
+
 struct file *ovl_path_open(const struct path *path, int flags)
 {
 	struct inode *inode = d_inode(path->dentry);
@@ -663,6 +673,14 @@ void ovl_copy_up_end(struct dentry *dentry)
 	ovl_inode_unlock(d_inode(dentry));
 }
 
+bool ovl_path_check_nowhiteout_xattr(struct ovl_fs *ofs, const struct path *path)
+{
+	int res;
+
+	res = ovl_path_getxattr(ofs, path, OVL_XATTR_NOWHITEOUT, NULL, 0);
+	return res >= 0;
+}
+
 bool ovl_path_check_origin_xattr(struct ovl_fs *ofs, const struct path *path)
 {
 	int res;
@@ -760,6 +778,7 @@ bool ovl_path_check_dir_xattr(struct ovl_fs *ofs, const struct path *path,
 #define OVL_XATTR_UUID_POSTFIX		"uuid"
 #define OVL_XATTR_METACOPY_POSTFIX	"metacopy"
 #define OVL_XATTR_PROTATTR_POSTFIX	"protattr"
+#define OVL_XATTR_NOWHITEOUT_POSTFIX	"nowhiteout"
 
 #define OVL_XATTR_TAB_ENTRY(x) \
 	[x] = { [false] = OVL_XATTR_TRUSTED_PREFIX x ## _POSTFIX, \
@@ -775,6 +794,7 @@ const char *const ovl_xattr_table[][2] = {
 	OVL_XATTR_TAB_ENTRY(OVL_XATTR_UUID),
 	OVL_XATTR_TAB_ENTRY(OVL_XATTR_METACOPY),
 	OVL_XATTR_TAB_ENTRY(OVL_XATTR_PROTATTR),
+	OVL_XATTR_TAB_ENTRY(OVL_XATTR_NOWHITEOUT),
 };
 
 int ovl_check_setxattr(struct ovl_fs *ofs, struct dentry *upperdentry,
-- 
2.41.0

Re: [PATCH 3/4] ovl: Support creation of whiteout files on overlayfs

[Amir Goldstein]

On Wed, Aug 16, 2023 at 6:29 PM Alexander Larsson <alexl@redhat.com> wrote:
>
> This is needed to properly stack overlay filesystems, I.E, being able
> to create a whiteout file on an overlay mount and then use that as
> part of the lowerdir in another overlay mount.
>
> The way this works is that we create a regular whiteout, but set the
> `overlay.nowhiteout` xattr on it. Whenever we check if a file is a
> whiteout we check this xattr and don't treat it as a whiteout if it is
> set. The xattr itself is then stripped and when viewed as part of the
> overlayfs mount it looks like a regular whiteout.
>
> Signed-off-by: Alexander Larsson <alexl@redhat.com>
> ---
>  fs/overlayfs/dir.c       | 14 ++++++++------
>  fs/overlayfs/namei.c     | 14 +++++++++-----
>  fs/overlayfs/overlayfs.h | 13 +++++++++++++
>  fs/overlayfs/readdir.c   |  7 ++++++-
>  fs/overlayfs/super.c     |  2 +-
>  fs/overlayfs/util.c      | 20 ++++++++++++++++++++
>  6 files changed, 57 insertions(+), 13 deletions(-)
>
> diff --git a/fs/overlayfs/dir.c b/fs/overlayfs/dir.c
> index 033fc0458a3d..4ef3a473700c 100644
> --- a/fs/overlayfs/dir.c
> +++ b/fs/overlayfs/dir.c
> @@ -199,6 +199,12 @@ struct dentry *ovl_create_real(struct ovl_fs *ofs, struct inode *dir,
>                 case S_IFSOCK:
>                         err = ovl_do_mknod(ofs, dir, newdentry, attr->mode,
>                                            attr->rdev);
> +
> +                       if (!err && S_ISCHR(attr->mode) && attr->rdev == WHITEOUT_DEV) {
> +                               err = ovl_setxattr(ofs, newdentry, OVL_XATTR_NOWHITEOUT,
> +                                                  NULL, 0);
> +                       }
> +

That's not an atomic way to create an escaped whiteout.
You'd want to always take the ovl_create_over_whiteout()
branch when creating a whiteout (even if not over a positive lower)

Thanks,
Amir.

>                         break;
>
>                 case S_IFLNK:
> @@ -477,7 +483,7 @@ static int ovl_create_over_whiteout(struct dentry *dentry, struct inode *inode,
>                 goto out_unlock;
>
>         err = -ESTALE;
> -       if (d_is_negative(upper) || !IS_WHITEOUT(d_inode(upper)))
> +       if (d_is_negative(upper) || !ovl_upper_is_whiteout(ofs, upper))
>                 goto out_dput;
>
>         newdentry = ovl_create_temp(ofs, workdir, cattr);
> @@ -669,10 +675,6 @@ static int ovl_mkdir(struct mnt_idmap *idmap, struct inode *dir,
>  static int ovl_mknod(struct mnt_idmap *idmap, struct inode *dir,
>                      struct dentry *dentry, umode_t mode, dev_t rdev)
>  {
> -       /* Don't allow creation of "whiteout" on overlay */
> -       if (S_ISCHR(mode) && rdev == WHITEOUT_DEV)
> -               return -EPERM;
> -
>         return ovl_create_object(dentry, mode, rdev, NULL);
>  }
>
> @@ -1219,7 +1221,7 @@ static int ovl_rename(struct mnt_idmap *idmap, struct inode *olddir,
>                 }
>         } else {
>                 if (!d_is_negative(newdentry)) {
> -                       if (!new_opaque || !ovl_is_whiteout(newdentry))
> +                       if (!new_opaque || !ovl_upper_is_whiteout(ofs, newdentry))
>                                 goto out_dput;
>                 } else {
>                         if (flags & RENAME_EXCHANGE)
> diff --git a/fs/overlayfs/namei.c b/fs/overlayfs/namei.c
> index 80391c687c2a..e90167789a13 100644
> --- a/fs/overlayfs/namei.c
> +++ b/fs/overlayfs/namei.c
> @@ -251,7 +251,9 @@ static int ovl_lookup_single(struct dentry *base, struct ovl_lookup_data *d,
>                 err = -EREMOTE;
>                 goto out_err;
>         }
> -       if (ovl_is_whiteout(this)) {
> +       path.dentry = this;
> +       path.mnt = d->mnt;
> +       if (ovl_path_is_whiteout(OVL_FS(d->sb), &path)) {
>                 d->stop = d->opaque = true;
>                 goto put_and_out;
>         }
> @@ -264,8 +266,6 @@ static int ovl_lookup_single(struct dentry *base, struct ovl_lookup_data *d,
>                 goto put_and_out;
>         }
>
> -       path.dentry = this;
> -       path.mnt = d->mnt;
>         if (!d_can_lookup(this)) {
>                 if (d->is_dir || !last_element) {
>                         d->stop = true;
> @@ -438,7 +438,7 @@ int ovl_check_origin_fh(struct ovl_fs *ofs, struct ovl_fh *fh, bool connected,
>         else if (IS_ERR(origin))
>                 return PTR_ERR(origin);
>
> -       if (upperdentry && !ovl_is_whiteout(upperdentry) &&
> +       if (upperdentry && !ovl_upper_is_whiteout(ofs, upperdentry) &&
>             inode_wrong_type(d_inode(upperdentry), d_inode(origin)->i_mode))
>                 goto invalid;
>
> @@ -1383,7 +1383,11 @@ bool ovl_lower_positive(struct dentry *dentry)
>                                 break;
>                         }
>                 } else {
> -                       positive = !ovl_is_whiteout(this);
> +                       struct path path = {
> +                               .dentry = this,
> +                               .mnt = parentpath->layer->mnt,
> +                       };
> +                       positive = !ovl_path_is_whiteout(OVL_FS(dentry->d_sb), &path);
>                         done = true;
>                         dput(this);
>                 }
> diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
> index 1dbd01719f63..853335ff26f7 100644
> --- a/fs/overlayfs/overlayfs.h
> +++ b/fs/overlayfs/overlayfs.h
> @@ -49,6 +49,7 @@ enum ovl_xattr {
>         OVL_XATTR_UUID,
>         OVL_XATTR_METACOPY,
>         OVL_XATTR_PROTATTR,
> +       OVL_XATTR_NOWHITEOUT,
>  };
>
>  enum ovl_inode_flag {
> @@ -469,16 +470,28 @@ void ovl_inode_update(struct inode *inode, struct dentry *upperdentry);
>  void ovl_dir_modified(struct dentry *dentry, bool impurity);
>  u64 ovl_inode_version_get(struct inode *inode);
>  bool ovl_is_whiteout(struct dentry *dentry);
> +bool ovl_path_is_whiteout(struct ovl_fs *ofs, const struct path *path);
>  struct file *ovl_path_open(const struct path *path, int flags);
>  int ovl_copy_up_start(struct dentry *dentry, int flags);
>  void ovl_copy_up_end(struct dentry *dentry);
>  bool ovl_already_copied_up(struct dentry *dentry, int flags);
>  bool ovl_path_check_dir_xattr(struct ovl_fs *ofs, const struct path *path,
>                               enum ovl_xattr ox);
> +bool ovl_path_check_nowhiteout_xattr(struct ovl_fs *ofs, const struct path *path);
>  bool ovl_path_check_origin_xattr(struct ovl_fs *ofs, const struct path *path);
>  bool ovl_init_uuid_xattr(struct super_block *sb, struct ovl_fs *ofs,
>                          const struct path *upperpath);
>
> +static inline bool ovl_upper_is_whiteout(struct ovl_fs *ofs,
> +                                        struct dentry *upperdentry)
> +{
> +       struct path upperpath = {
> +               .dentry = upperdentry,
> +               .mnt = ovl_upper_mnt(ofs),
> +       };
> +       return ovl_path_is_whiteout(ofs, &upperpath);
> +}
> +
>  static inline bool ovl_check_origin_xattr(struct ovl_fs *ofs,
>                                           struct dentry *upperdentry)
>  {
> diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
> index de39e067ae65..9cf8e7e2961c 100644
> --- a/fs/overlayfs/readdir.c
> +++ b/fs/overlayfs/readdir.c
> @@ -280,7 +280,12 @@ static int ovl_check_whiteouts(const struct path *path, struct ovl_readdir_data
>                         rdd->first_maybe_whiteout = p->next_maybe_whiteout;
>                         dentry = lookup_one(mnt_idmap(path->mnt), p->name, dir, p->len);
>                         if (!IS_ERR(dentry)) {
> -                               p->is_whiteout = ovl_is_whiteout(dentry);
> +                               struct path childpath = {
> +                                       .dentry = dentry,
> +                                       .mnt = path->mnt,
> +                               };
> +                               p->is_whiteout = ovl_path_is_whiteout(OVL_FS(rdd->dentry->d_sb),
> +                                                                     &childpath);
>                                 dput(dentry);
>                         }
>                 }
> diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
> index 97bc94459f7a..71c650ba5a1a 100644
> --- a/fs/overlayfs/super.c
> +++ b/fs/overlayfs/super.c
> @@ -688,7 +688,7 @@ static int ovl_check_rename_whiteout(struct ovl_fs *ofs)
>         if (IS_ERR(whiteout))
>                 goto cleanup_temp;
>
> -       err = ovl_is_whiteout(whiteout);
> +       err = ovl_upper_is_whiteout(ofs, whiteout);
>
>         /* Best effort cleanup of whiteout and temp file */
>         if (err)
> diff --git a/fs/overlayfs/util.c b/fs/overlayfs/util.c
> index 0f387092450e..da6d2abf64dd 100644
> --- a/fs/overlayfs/util.c
> +++ b/fs/overlayfs/util.c
> @@ -575,6 +575,16 @@ bool ovl_is_whiteout(struct dentry *dentry)
>         return inode && IS_WHITEOUT(inode);
>  }
>
> +/*
> + * Use this over ovl_is_whiteout for upper and lower files, as it also
> + * handles escaped whiteout files.
> + */
> +bool ovl_path_is_whiteout(struct ovl_fs *ofs, const struct path *path)
> +{
> +       return ovl_is_whiteout(path->dentry) &&
> +               !ovl_path_check_nowhiteout_xattr(ofs, path);
> +}
> +
>  struct file *ovl_path_open(const struct path *path, int flags)
>  {
>         struct inode *inode = d_inode(path->dentry);
> @@ -663,6 +673,14 @@ void ovl_copy_up_end(struct dentry *dentry)
>         ovl_inode_unlock(d_inode(dentry));
>  }
>
> +bool ovl_path_check_nowhiteout_xattr(struct ovl_fs *ofs, const struct path *path)
> +{
> +       int res;
> +
> +       res = ovl_path_getxattr(ofs, path, OVL_XATTR_NOWHITEOUT, NULL, 0);
> +       return res >= 0;
> +}
> +
>  bool ovl_path_check_origin_xattr(struct ovl_fs *ofs, const struct path *path)
>  {
>         int res;
> @@ -760,6 +778,7 @@ bool ovl_path_check_dir_xattr(struct ovl_fs *ofs, const struct path *path,
>  #define OVL_XATTR_UUID_POSTFIX         "uuid"
>  #define OVL_XATTR_METACOPY_POSTFIX     "metacopy"
>  #define OVL_XATTR_PROTATTR_POSTFIX     "protattr"
> +#define OVL_XATTR_NOWHITEOUT_POSTFIX   "nowhiteout"
>
>  #define OVL_XATTR_TAB_ENTRY(x) \
>         [x] = { [false] = OVL_XATTR_TRUSTED_PREFIX x ## _POSTFIX, \
> @@ -775,6 +794,7 @@ const char *const ovl_xattr_table[][2] = {
>         OVL_XATTR_TAB_ENTRY(OVL_XATTR_UUID),
>         OVL_XATTR_TAB_ENTRY(OVL_XATTR_METACOPY),
>         OVL_XATTR_TAB_ENTRY(OVL_XATTR_PROTATTR),
> +       OVL_XATTR_TAB_ENTRY(OVL_XATTR_NOWHITEOUT),
>  };
>
>  int ovl_check_setxattr(struct ovl_fs *ofs, struct dentry *upperdentry,
> --
> 2.41.0
>

[PATCH 4/4] ovl: Add documentation on nesting of overlayfs mounts

[Alexander Larsson]

Signed-off-by: Alexander Larsson <alexl@redhat.com>
---
 Documentation/filesystems/overlayfs.rst | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/Documentation/filesystems/overlayfs.rst b/Documentation/filesystems/overlayfs.rst
index 35853906accb..e38b2f5fadaf 100644
--- a/Documentation/filesystems/overlayfs.rst
+++ b/Documentation/filesystems/overlayfs.rst
@@ -492,6 +492,28 @@ directory tree on the same or different underlying filesystem, and even
 to a different machine.  With the "inodes index" feature, trying to mount
 the copied layers will fail the verification of the lower root file handle.
 
+Nesting overlayfs mounts
+------------------------
+
+It is possible to use a lower directory that is stored on an overlayfs
+mount. For regular files this does not need any special care. However, files
+that have overlayfs attributes, such as whiteouts or `overlay.*` xattrs will
+be interpreted by the underlying overlayfs mount and stripped out. In order to
+allow the second overlayfs mount to see the attributes they must be escaped.
+
+Overlayfs specific xattrs are escaped by using a special prefix of
+`overlay.overlay.`. So, a file with a `trusted.overlay.overlay.metacopy` xattr
+in the lower dir will be exposed as a regular file with a
+`trusted.overlay.metacopy` xattr in the overlayfs mount. This can be nested
+by repeating the prefix multiple time, as each instance only removes one
+prefix.
+
+Whiteouts files marked with a `overlay.nowhiteout` xattr will cause overlayfs
+not to treat them as a whiteout. Since this xattr is then stripped out, the
+next layer will instead apply the whiteout.
+
+Files created via overlayfs will automatically be created with the right
+escapes in the upper directory.
 
 Non-standard behavior
 ---------------------
-- 
2.41.0