Documenting RENAME_WHITEOUT

Documenting RENAME_WHITEOUT

[Michael Kerrisk (man-pages)]

Hi Miklos,

I just noticed that your RENAME_WHITEOUT flag went into Linux 3.18:
commit 0d7a855526dd672e114aff2ac22b60fc6f155b08
commit 787fb6bc9682ec7c05fb5d9561b57100fbc1cc41

Would you be willing to write some text for the rename(2)/renameat2(2)
man page that described this flag. In that text it would be great to
have an explanation of what a whiteout is and why they are useful.

Cheers,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Documenting RENAME_WHITEOUT

[Michael Kerrisk (man-pages)]

Hi Miklos,

Ping!

Cheers,

Michael


On 29 January 2015 at 11:01, Michael Kerrisk (man-pages)
<mtk.manpages@gmail.com> wrote:
> Hi Miklos,
>
> I just noticed that your RENAME_WHITEOUT flag went into Linux 3.18:
> commit 0d7a855526dd672e114aff2ac22b60fc6f155b08
> commit 787fb6bc9682ec7c05fb5d9561b57100fbc1cc41
>
> Would you be willing to write some text for the rename(2)/renameat2(2)
> man page that described this flag. In that text it would be great to
> have an explanation of what a whiteout is and why they are useful.
>
> Cheers,
>
> Michael
>
> --
> Michael Kerrisk
> Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
> Linux/UNIX System Programming Training: http://man7.org/training/



-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Documenting RENAME_WHITEOUT

[Miklos Szeredi]

On Thu, Jan 29, 2015 at 11:01:08AM +0100, Michael Kerrisk (man-pages) wrote:
> Hi Miklos,
> 
> I just noticed that your RENAME_WHITEOUT flag went into Linux 3.18:
> commit 0d7a855526dd672e114aff2ac22b60fc6f155b08
> commit 787fb6bc9682ec7c05fb5d9561b57100fbc1cc41
> 
> Would you be willing to write some text for the rename(2)/renameat2(2)
> man page that described this flag. In that text it would be great to
> have an explanation of what a whiteout is and why they are useful.

Hi Michael,

Sorry for the delay...

  RENAME_WHITEOUT is a special operation, that only makes sense for
  overlay/union type filesystem implementations.  Currently it is used
  internally by the overlay filesystem.

  Specifying RENAME_WHITEOUT will create a "whiteout" object at the source of
  the rename at the same time as performing the rename.  The whole operation is
  still atomic, so if the rename succeeds then the whiteout will also have been
  created.

  A "whiteout" is an object that has special meaning in union/overlay type file
  system constructs, in these constructs multiple layers exists and only the top
  one is ever modified.  A whiteout on an upper layer will effectively hide the
  matching file on the lower layer, making it appear if the file didn't exist.

  When a file that exists on the lower layer is renamed, the file is first
  copied up (if not already on the upper layer) and then renamed on the upper,
  read-write layer.  At the same time the source file needs to be "whiteouted".
  The whole operation needs to be done atomically.

  When not part of a union/overlay the whiteout appears as a char device with
  0,0 device number.  RENAME_WHITEOUT needs the same privileges as creating a
  device node (CAP_MKNOD) and will fail with EPERM error if that capability is
  missing.

  If RENAME_WHITEOUT is specified together wuth RENAME_EXCHANGE, then the rename
  with fail with EINVAL error.

Thanks,
Miklos
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documenting RENAME_WHITEOUT

[Dave Chinner]

On Fri, Mar 06, 2015 at 05:11:45PM +0100, Miklos Szeredi wrote:
> On Thu, Jan 29, 2015 at 11:01:08AM +0100, Michael Kerrisk (man-pages) wrote:
> > Hi Miklos,
> > 
> > I just noticed that your RENAME_WHITEOUT flag went into Linux 3.18:
> > commit 0d7a855526dd672e114aff2ac22b60fc6f155b08
> > commit 787fb6bc9682ec7c05fb5d9561b57100fbc1cc41
> > 
> > Would you be willing to write some text for the rename(2)/renameat2(2)
> > man page that described this flag. In that text it would be great to
> > have an explanation of what a whiteout is and why they are useful.
> 
> Hi Michael,
> 
> Sorry for the delay...
> 
>   RENAME_WHITEOUT is a special operation, that only makes sense for
>   overlay/union type filesystem implementations.  Currently it is used
>   internally by the overlay filesystem.

However, it is exposed to userspace by renameat2, and filesystem
developers still need to know it's exact semantics documented so
they can implement it.

>   Specifying RENAME_WHITEOUT will create a "whiteout" object at the source of
>   the rename at the same time as performing the rename.  The whole operation is
>   still atomic, so if the rename succeeds then the whiteout will also have been
>   created.
> 
>   A "whiteout" is an object that has special meaning in union/overlay type file
>   system constructs, in these constructs multiple layers exists and only the top
>   one is ever modified.  A whiteout on an upper layer will effectively hide the
>   matching file on the lower layer, making it appear if the file didn't exist.
> 
>   When a file that exists on the lower layer is renamed, the file is first
>   copied up (if not already on the upper layer) and then renamed on the upper,
>   read-write layer.  At the same time the source file needs to be "whiteouted".
>   The whole operation needs to be done atomically.

This doesn't explain exactly what the RENAME_WHITEOUT operation is
supposed to do. It explains how overlayfs uses them, not the
semantics and behaviour of RENAME_WHITEOUT. e.g. source
restrictions, target restrictions, can you RENAME_WHITEOUT over
another whiteout, etc. I've noticed these restrictions are very
different from other rename operations, but I don't know whether
that is ext4 implementation bugs or intentional because there is no
documentation or regression tests in xfstests for it...

Cheers,

Dave.
-- 
Dave Chinner
david-FqsqvQoI3Ljby3iVrkZq2A@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documenting RENAME_WHITEOUT

[Michael Kerrisk (man-pages)]

On 03/06/2015 10:44 PM, Dave Chinner wrote:
> On Fri, Mar 06, 2015 at 05:11:45PM +0100, Miklos Szeredi wrote:
>> On Thu, Jan 29, 2015 at 11:01:08AM +0100, Michael Kerrisk (man-pages) wrote:
>>> Hi Miklos,
>>>
>>> I just noticed that your RENAME_WHITEOUT flag went into Linux 3.18:
>>> commit 0d7a855526dd672e114aff2ac22b60fc6f155b08
>>> commit 787fb6bc9682ec7c05fb5d9561b57100fbc1cc41
>>>
>>> Would you be willing to write some text for the rename(2)/renameat2(2)
>>> man page that described this flag. In that text it would be great to
>>> have an explanation of what a whiteout is and why they are useful.
>>
>> Hi Michael,
>>
>> Sorry for the delay...
>>
>>   RENAME_WHITEOUT is a special operation, that only makes sense for
>>   overlay/union type filesystem implementations.  Currently it is used
>>   internally by the overlay filesystem.
> 
> However, it is exposed to userspace by renameat2, and filesystem
> developers still need to know it's exact semantics documented so
> they can implement it.
> 
>>   Specifying RENAME_WHITEOUT will create a "whiteout" object at the source of
>>   the rename at the same time as performing the rename.  The whole operation is
>>   still atomic, so if the rename succeeds then the whiteout will also have been
>>   created.
>>
>>   A "whiteout" is an object that has special meaning in union/overlay type file
>>   system constructs, in these constructs multiple layers exists and only the top
>>   one is ever modified.  A whiteout on an upper layer will effectively hide the
>>   matching file on the lower layer, making it appear if the file didn't exist.
>>
>>   When a file that exists on the lower layer is renamed, the file is first
>>   copied up (if not already on the upper layer) and then renamed on the upper,
>>   read-write layer.  At the same time the source file needs to be "whiteouted".
>>   The whole operation needs to be done atomically.
> 
> This doesn't explain exactly what the RENAME_WHITEOUT operation is
> supposed to do. It explains how overlayfs uses them, not the
> semantics and behaviour of RENAME_WHITEOUT. e.g. source
> restrictions, target restrictions, can you RENAME_WHITEOUT over
> another whiteout, etc. I've noticed these restrictions are very
> different from other rename operations, but I don't know whether
> that is ext4 implementation bugs or intentional because there is no
> documentation or regression tests in xfstests for it...

What Dave said!

Miklos, AFAICS, RENAME_WHITEOUT is user-space visible. Would you be 
willing to write some piece for the man page to explain things
from a user-space perspective?

Thanks,

Michael


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documenting RENAME_WHITEOUT

[Miklos Szeredi]

On Sun, Mar 08, 2015 at 09:37:39AM +0100, Michael Kerrisk (man-pages) wrote:
> On 03/06/2015 10:44 PM, Dave Chinner wrote:
> > On Fri, Mar 06, 2015 at 05:11:45PM +0100, Miklos Szeredi wrote:
> >> On Thu, Jan 29, 2015 at 11:01:08AM +0100, Michael Kerrisk (man-pages) wrote:
> >>> Hi Miklos,
> >>>
> >>> I just noticed that your RENAME_WHITEOUT flag went into Linux 3.18:
> >>> commit 0d7a855526dd672e114aff2ac22b60fc6f155b08
> >>> commit 787fb6bc9682ec7c05fb5d9561b57100fbc1cc41
> >>>
> >>> Would you be willing to write some text for the rename(2)/renameat2(2)
> >>> man page that described this flag. In that text it would be great to
> >>> have an explanation of what a whiteout is and why they are useful.
> >>
> >> Hi Michael,
> >>
> >> Sorry for the delay...
> >>
> >>   RENAME_WHITEOUT is a special operation, that only makes sense for
> >>   overlay/union type filesystem implementations.  Currently it is used
> >>   internally by the overlay filesystem.
> > 
> > However, it is exposed to userspace by renameat2, and filesystem
> > developers still need to know it's exact semantics documented so
> > they can implement it.
> > 
> >>   Specifying RENAME_WHITEOUT will create a "whiteout" object at the source of
> >>   the rename at the same time as performing the rename.  The whole operation is
> >>   still atomic, so if the rename succeeds then the whiteout will also have been
> >>   created.
> >>
> >>   A "whiteout" is an object that has special meaning in union/overlay type file
> >>   system constructs, in these constructs multiple layers exists and only the top
> >>   one is ever modified.  A whiteout on an upper layer will effectively hide the
> >>   matching file on the lower layer, making it appear if the file didn't exist.
> >>
> >>   When a file that exists on the lower layer is renamed, the file is first
> >>   copied up (if not already on the upper layer) and then renamed on the upper,
> >>   read-write layer.  At the same time the source file needs to be "whiteouted".
> >>   The whole operation needs to be done atomically.
> > 
> > This doesn't explain exactly what the RENAME_WHITEOUT operation is
> > supposed to do. It explains how overlayfs uses them, not the
> > semantics and behaviour of RENAME_WHITEOUT. e.g. source
> > restrictions, target restrictions, can you RENAME_WHITEOUT over
> > another whiteout, etc. I've noticed these restrictions are very
> > different from other rename operations, but I don't know whether
> > that is ext4 implementation bugs or intentional because there is no
> > documentation or regression tests in xfstests for it...
> 
> What Dave said!
> 
> Miklos, AFAICS, RENAME_WHITEOUT is user-space visible. Would you be 
> willing to write some piece for the man page to explain things
> from a user-space perspective?

The text explains why whiteouts are useful.  That's what you asked me, so that's
what I wrote...

If that's not interesting we can just leave that part out:

  Specifying RENAME_WHITEOUT will create a "whiteout" object at the source of
  the rename at the same time as performing the rename.  The whole operation is
  still atomic, so if the rename succeeds then the whiteout will also have been
  created.

  The whiteout is represented as a char device with 0,0 device number.
  RENAME_WHITEOUT needs the same privileges as creating a device node
  (CAP_MKNOD) and will fail with EPERM error if that capability is missing.

  If RENAME_WHITEOUT is specified together wuth RENAME_EXCHANGE, then the rename
  with fail with EINVAL error.

That's it.  No other special behavior relative to normal rename.

Thanks,
Miklos
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documenting RENAME_WHITEOUT

[Michael Kerrisk (man-pages)]

Hello Miklos,


On 03/06/2015 05:11 PM, Miklos Szeredi wrote:
> On Thu, Jan 29, 2015 at 11:01:08AM +0100, Michael Kerrisk (man-pages) wrote:
>> Hi Miklos,
>>
>> I just noticed that your RENAME_WHITEOUT flag went into Linux 3.18:
>> commit 0d7a855526dd672e114aff2ac22b60fc6f155b08
>> commit 787fb6bc9682ec7c05fb5d9561b57100fbc1cc41
>>
>> Would you be willing to write some text for the rename(2)/renameat2(2)
>> man page that described this flag. In that text it would be great to
>> have an explanation of what a whiteout is and why they are useful.
> 
> Hi Michael,
> 
> Sorry for the delay...
> 
>   RENAME_WHITEOUT is a special operation, that only makes sense for
>   overlay/union type filesystem implementations.  Currently it is used
>   internally by the overlay filesystem.
> 
>   Specifying RENAME_WHITEOUT will create a "whiteout" object at the source of
>   the rename at the same time as performing the rename.  The whole operation is
>   still atomic, so if the rename succeeds then the whiteout will also have been
>   created.
> 
>   A "whiteout" is an object that has special meaning in union/overlay type file
>   system constructs, in these constructs multiple layers exists and only the top
>   one is ever modified.  A whiteout on an upper layer will effectively hide the
>   matching file on the lower layer, making it appear if the file didn't exist.
> 
>   When a file that exists on the lower layer is renamed, the file is first
>   copied up (if not already on the upper layer) and then renamed on the upper,
>   read-write layer.  At the same time the source file needs to be "whiteouted".
>   The whole operation needs to be done atomically.
> 
>   When not part of a union/overlay the whiteout appears as a char device with
>   0,0 device number.  RENAME_WHITEOUT needs the same privileges as creating a
>   device node (CAP_MKNOD) and will fail with EPERM error if that capability is
>   missing.
> 
>   If RENAME_WHITEOUT is specified together wuth RENAME_EXCHANGE, then the rename
>   with fail with EINVAL error.

I did some editing of the text and added some details to come up with the
following. Could you please check it over? I also have one question below.
(I have also added some entries under ERRORS, but I've omitted them here.)

       RENAME_WHITEOUT (since Linux 3.18)
              This  operation  makes  sense  only  for overlay/union
              filesystem implementations.

              Specifying RENAME_WHITEOUT creates a "whiteout" object
              at  the  source of the rename at the same time as per‐
              forming the rename.  The whole operation is atomic, so
              that  if  the  rename  succeeds then the whiteout will
              also have been created.

              A "whiteout" is an object that has special meaning  in
              union/overlay  filesystem  constructs.   In these con‐
              structs, multiple layers exist and only the top one is
              ever  modified.   A  whiteout  on  an upper layer will
              effectively hide a matching file in the  lower  layer,
              making it appear as if the file didn't exist.

              When a file that exists on the lower layer is renamed,
              the file is first copied up (if  not  already  on  the
              upper layer) and then renamed on the upper, read-write
              layer.  At the same time, the source file needs to  be
              "whiteouted".   The  whole  operation needs to be done

???
After "whitedout", I am tempted to add: "(so that the version of 
the source file in the lower layer is rendered invisible)".
Is that a correct formulation, and is it helpful to add it?

              atomically.

              When not part of a union/overlay, the whiteout appears
              as a character device with a {0,0} device number.

              RENAME_WHITEOUT requires the same privileges as creat‐
              ing a device node (i.e., the CAP_MKNOD capability).

              RENAME_WHITEOUT  can't  be  employed   together   with
              RENAME_EXCHANGE.

              RENAME_WHITEOUT  requires  support from the underlying
              filesystem.  Among the filesystems that  provide  that
              support  are  shmem  (since  Linux  3.18), ext4 (since
              Linux 3.18), and XFS (since Linux 4.1).

Thanks,

Michael


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-man" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documenting RENAME_WHITEOUT

[Miklos Szeredi]

Hi Michael,

On Wed, May 6, 2015 at 4:17 PM, Michael Kerrisk (man-pages)
<mtk.manpages@gmail.com> wrote:
> I did some editing of the text and added some details to come up with the
> following. Could you please check it over? I also have one question below.
> (I have also added some entries under ERRORS, but I've omitted them here.)
>
>        RENAME_WHITEOUT (since Linux 3.18)
>               This  operation  makes  sense  only  for overlay/union
>               filesystem implementations.
>
>               Specifying RENAME_WHITEOUT creates a "whiteout" object
>               at  the  source of the rename at the same time as per‐
>               forming the rename.  The whole operation is atomic, so
>               that  if  the  rename  succeeds then the whiteout will
>               also have been created.
>
>               A "whiteout" is an object that has special meaning  in
>               union/overlay  filesystem  constructs.   In these con‐
>               structs, multiple layers exist and only the top one is
>               ever  modified.   A  whiteout  on  an upper layer will
>               effectively hide a matching file in the  lower  layer,
>               making it appear as if the file didn't exist.
>
>               When a file that exists on the lower layer is renamed,
>               the file is first copied up (if  not  already  on  the
>               upper layer) and then renamed on the upper, read-write
>               layer.  At the same time, the source file needs to  be
>               "whiteouted".   The  whole  operation needs to be done
>
> ???
> After "whitedout", I am tempted to add: "(so that the version of
> the source file in the lower layer is rendered invisible)".
> Is that a correct formulation, and is it helpful to add it?

Yes, that's correct, and helpful, I think.

>
>               atomically.
>
>               When not part of a union/overlay, the whiteout appears
>               as a character device with a {0,0} device number.
>
>               RENAME_WHITEOUT requires the same privileges as creat‐
>               ing a device node (i.e., the CAP_MKNOD capability).
>
>               RENAME_WHITEOUT  can't  be  employed   together   with
>               RENAME_EXCHANGE.
>
>               RENAME_WHITEOUT  requires  support from the underlying
>               filesystem.  Among the filesystems that  provide  that
>               support  are  shmem  (since  Linux  3.18), ext4 (since
>               Linux 3.18), and XFS (since Linux 4.1).

Looks good to me.

Thanks,
Miklos
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Documenting RENAME_WHITEOUT

[Michael Kerrisk (man-pages)]

Hello Miklos,

On 05/06/2015 04:49 PM, Miklos Szeredi wrote:
> Hi Michael,
> 
> On Wed, May 6, 2015 at 4:17 PM, Michael Kerrisk (man-pages)
> <mtk.manpages@gmail.com> wrote:
>> I did some editing of the text and added some details to come up with the
>> following. Could you please check it over? I also have one question below.
>> (I have also added some entries under ERRORS, but I've omitted them here.)
>>
>>        RENAME_WHITEOUT (since Linux 3.18)
>>               This  operation  makes  sense  only  for overlay/union
>>               filesystem implementations.
>>
>>               Specifying RENAME_WHITEOUT creates a "whiteout" object
>>               at  the  source of the rename at the same time as per‐
>>               forming the rename.  The whole operation is atomic, so
>>               that  if  the  rename  succeeds then the whiteout will
>>               also have been created.
>>
>>               A "whiteout" is an object that has special meaning  in
>>               union/overlay  filesystem  constructs.   In these con‐
>>               structs, multiple layers exist and only the top one is
>>               ever  modified.   A  whiteout  on  an upper layer will
>>               effectively hide a matching file in the  lower  layer,
>>               making it appear as if the file didn't exist.
>>
>>               When a file that exists on the lower layer is renamed,
>>               the file is first copied up (if  not  already  on  the
>>               upper layer) and then renamed on the upper, read-write
>>               layer.  At the same time, the source file needs to  be
>>               "whiteouted".   The  whole  operation needs to be done
>>
>> ???
>> After "whitedout", I am tempted to add: "(so that the version of
>> the source file in the lower layer is rendered invisible)".
>> Is that a correct formulation, and is it helpful to add it?
> 
> Yes, that's correct, and helpful, I think.

Added. Thanks for confirming, and thanks of course for your original text!

Cheers,

Michael

>>               atomically.
>>
>>               When not part of a union/overlay, the whiteout appears
>>               as a character device with a {0,0} device number.
>>
>>               RENAME_WHITEOUT requires the same privileges as creat‐
>>               ing a device node (i.e., the CAP_MKNOD capability).
>>
>>               RENAME_WHITEOUT  can't  be  employed   together   with
>>               RENAME_EXCHANGE.
>>
>>               RENAME_WHITEOUT  requires  support from the underlying
>>               filesystem.  Among the filesystems that  provide  that
>>               support  are  shmem  (since  Linux  3.18), ext4 (since
>>               Linux 3.18), and XFS (since Linux 4.1).
> 
> Looks good to me.
> 
> Thanks,
> Miklos
> 


-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html