[potential issue, question] whiteout shows up in merged directory

[potential issue, question] whiteout shows up in merged directory

[Jingbo Xu]

Hi, all,

I found an issue may be related to overlayfs on the latest master branch
[1] when I'm developing tarfs mode for erofs-utils [2], which converts
and merges tar layers into one merged erofs image with overlayfs-like model.

The issue is that, the whiteout from lowerdir may still shows up in the
merged directory.  Though this issue is initially found with erofs, it
can also be reproduced with ext4.  Following is a simple reproducer with
ext4.

```
mkdir -p /mnt/lower1/dir /mnt/lower2
mknod /mnt/lower1/file1 c 0 0
mknod /mnt/lower1/dir/file2 c 0 0
mount -t overlay none -olowerdir=/mnt/lower1:/mnt/lower2 /mnt2

# ls  -l /mnt2/
total 4
drwxr-xr-x 2 root root 4096 Sep  4 14:40 dir

# ls  -l /mnt2/dir
ls: cannot access /mnt2/dir/file2: No such file or directory
total 0
c????????? ? ? ? ?            ? file2
```

It seems that this issue is relevant to whether the parent directory of
the whiteout is a merged directory or not.  In the above example, file1
is hidden from the merged directory as expected (with its parent
directory '/' a merged directory), while file2 shows up unexpectedly
(with its parent directory '/dir' from lowerdir).


I also noticed that this issue doesn't exist if the whiteout is created
by overlayfs itself rather than handcrafted with mknod like:

```
mkdir -p /mnt/lower/dir /mnt/upper /mnt/work
touch /mnt/lower/file1
touch /mnt/lower/dir/file2
mount -t overlay none
-olowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/work /mnt1
rm /mnt1/file1
rm /mnt1/dir/file2
umount /mnt1
mount -t overlay -olowerdir=/mnt/upper:/mnt/lower none /mnt2

# ls -l /mnt2/
total 8
drwxr-xr-x 1 root root 4096 Sep  4 15:45 dir

# ls -l /mnt2/dir/
total 0
```

I'm not sure if it's a known issue or not, or due to my mishandling.
Appreciate if you could shed a light on this.


[1] git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[2]
https://lore.kernel.org/all/20230901094706.27539-1-jefflexu@linux.alibaba.com/

-- 
Thanks,
Jingbo

Re: [potential issue, question] whiteout shows up in merged directory

[Amir Goldstein]

On Mon, Sep 4, 2023 at 10:47 AM Jingbo Xu <jefflexu@linux.alibaba.com> wrote:
>
> Hi, all,
>
> I found an issue may be related to overlayfs on the latest master branch
> [1] when I'm developing tarfs mode for erofs-utils [2], which converts
> and merges tar layers into one merged erofs image with overlayfs-like model.
>
> The issue is that, the whiteout from lowerdir may still shows up in the
> merged directory.  Though this issue is initially found with erofs, it
> can also be reproduced with ext4.  Following is a simple reproducer with
> ext4.
>
> ```
> mkdir -p /mnt/lower1/dir /mnt/lower2
> mknod /mnt/lower1/file1 c 0 0
> mknod /mnt/lower1/dir/file2 c 0 0
> mount -t overlay none -olowerdir=/mnt/lower1:/mnt/lower2 /mnt2
>
> # ls  -l /mnt2/
> total 4
> drwxr-xr-x 2 root root 4096 Sep  4 14:40 dir
>
> # ls  -l /mnt2/dir
> ls: cannot access /mnt2/dir/file2: No such file or directory
> total 0
> c????????? ? ? ? ?            ? file2
> ```
>
> It seems that this issue is relevant to whether the parent directory of
> the whiteout is a merged directory or not.  In the above example, file1
> is hidden from the merged directory as expected (with its parent
> directory '/' a merged directory), while file2 shows up unexpectedly
> (with its parent directory '/dir' from lowerdir).
>
>
> I also noticed that this issue doesn't exist if the whiteout is created
> by overlayfs itself rather than handcrafted with mknod like:
>
> ```
> mkdir -p /mnt/lower/dir /mnt/upper /mnt/work
> touch /mnt/lower/file1
> touch /mnt/lower/dir/file2
> mount -t overlay none
> -olowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/work /mnt1
> rm /mnt1/file1
> rm /mnt1/dir/file2
> umount /mnt1
> mount -t overlay -olowerdir=/mnt/upper:/mnt/lower none /mnt2
>
> # ls -l /mnt2/
> total 8
> drwxr-xr-x 1 root root 4096 Sep  4 15:45 dir
>
> # ls -l /mnt2/dir/
> total 0
> ```
>
> I'm not sure if it's a known issue or not, or due to my mishandling.
> Appreciate if you could shed a light on this.
>

The case of whiteouts creates by overlayfs itself was reported
by zhangyi and handled by this patchs set:

https://lore.kernel.org/linux-unionfs/1509486350-21362-1-git-send-email-amir73il@gmail.com/

so you could say that the problem is due to the way that you
created those layers.

There is a simple workaround for you though:

mkdir -p /mnt/lower1/dir /mnt/lower2/dir

Making 'dir' a merge dir avoids the problem.
Not sure if that helps.

The alternative way is:
setfattr -n "trusted.overlay.origin" /mnt/lower1/dir

Thanks,
Amir.

Re: [potential issue, question] whiteout shows up in merged directory

[Jingbo Xu]

Hi,

On 9/4/23 4:57 PM, Amir Goldstein wrote:
> On Mon, Sep 4, 2023 at 10:47 AM Jingbo Xu <jefflexu@linux.alibaba.com> wrote:
>>
>> Hi, all,
>>
>> I found an issue may be related to overlayfs on the latest master branch
>> [1] when I'm developing tarfs mode for erofs-utils [2], which converts
>> and merges tar layers into one merged erofs image with overlayfs-like model.
>>
>> The issue is that, the whiteout from lowerdir may still shows up in the
>> merged directory.  Though this issue is initially found with erofs, it
>> can also be reproduced with ext4.  Following is a simple reproducer with
>> ext4.
>>
>> ```
>> mkdir -p /mnt/lower1/dir /mnt/lower2
>> mknod /mnt/lower1/file1 c 0 0
>> mknod /mnt/lower1/dir/file2 c 0 0
>> mount -t overlay none -olowerdir=/mnt/lower1:/mnt/lower2 /mnt2
>>
>> # ls  -l /mnt2/
>> total 4
>> drwxr-xr-x 2 root root 4096 Sep  4 14:40 dir
>>
>> # ls  -l /mnt2/dir
>> ls: cannot access /mnt2/dir/file2: No such file or directory
>> total 0
>> c????????? ? ? ? ?            ? file2
>> ```
>>
>> It seems that this issue is relevant to whether the parent directory of
>> the whiteout is a merged directory or not.  In the above example, file1
>> is hidden from the merged directory as expected (with its parent
>> directory '/' a merged directory), while file2 shows up unexpectedly
>> (with its parent directory '/dir' from lowerdir).
>>
>>
>> I also noticed that this issue doesn't exist if the whiteout is created
>> by overlayfs itself rather than handcrafted with mknod like:
>>
>> ```
>> mkdir -p /mnt/lower/dir /mnt/upper /mnt/work
>> touch /mnt/lower/file1
>> touch /mnt/lower/dir/file2
>> mount -t overlay none
>> -olowerdir=/mnt/lower,upperdir=/mnt/upper,workdir=/mnt/work /mnt1
>> rm /mnt1/file1
>> rm /mnt1/dir/file2
>> umount /mnt1
>> mount -t overlay -olowerdir=/mnt/upper:/mnt/lower none /mnt2
>>
>> # ls -l /mnt2/
>> total 8
>> drwxr-xr-x 1 root root 4096 Sep  4 15:45 dir
>>
>> # ls -l /mnt2/dir/
>> total 0
>> ```
>>
>> I'm not sure if it's a known issue or not, or due to my mishandling.
>> Appreciate if you could shed a light on this.
>>
> 
> The case of whiteouts creates by overlayfs itself was reported
> by zhangyi and handled by this patchs set:
> 
> https://lore.kernel.org/linux-unionfs/1509486350-21362-1-git-send-email-amir73il@gmail.com/

Thanks for the reply and it's really helpful to me.

I can understand in the normal use case, whiteout can not appear in
non-merged directory without origin xattr, except it's hand crafted.

But indeed we suffer from this issue in the tarfs for erofs-utils we are
developing. As described previously, in tarfs mode erofs-utils can
convert each tar layer into one separate erofs image, and then merge
these erofs images into one merged erofs image in a overlayfs-like model.

Suppose:

layer 0 + layer 1   +        layer 2         -->  merged
	  /foo/bar   /foo/bar (whiteout)


To speed the merging process, we may merge the two top-most layers
(layer 1 and layer 2) first, and then make layer0 merged into the final
merged image as:



           layer 1   +        layer 2         -->  merged-intermediate
	  /foo/bar   /foo/bar (whiteout)

layer0 + merged-intermediate		      -->  merged

Then there comes the problem: when merging layer1 and layer2, I need to
keep the whiteout in the intermediate merged image though the target of
the whiteout has showed up in underlying layer (/foo/bar in layer 1),
because I have no idea if "/foo/bar" exits in the following further
underlying layer (layer 0).  Reusing this logic, the whiteout is kept
there in the final merged image after merging layer0 and
merged-intermediate.

Then if "/foo" is not a merged directory, the "/foo/bar" whiteout will
be exposed in the overlayfs unexpectedly.

Currently we work around this in erofs-utils side.  Apart from setting
origin xattr on the parent directory of the whiteout, I'm not sure if
the above use case is reasonable enough to fix this in the kernel side.

-- 
Thanks,
Jingbo

Re: [potential issue, question] whiteout shows up in merged directory

[Gao Xiang]

On 2023/9/4 20:49, Jingbo Xu wrote:

...

> 
> Thanks for the reply and it's really helpful to me.
> 
> I can understand in the normal use case, whiteout can not appear in
> non-merged directory without origin xattr, except it's hand crafted.
> 
> But indeed we suffer from this issue in the tarfs for erofs-utils we are
> developing. As described previously, in tarfs mode erofs-utils can
> convert each tar layer into one separate erofs image, and then merge
> these erofs images into one merged erofs image in a overlayfs-like model.
> 
> Suppose:
> 
> layer 0 + layer 1   +        layer 2         -->  merged
> 	  /foo/bar   /foo/bar (whiteout)
> 
> 
> To speed the merging process, we may merge the two top-most layers
> (layer 1 and layer 2) first, and then make layer0 merged into the final
> merged image as:
> 
> 
> 
>             layer 1   +        layer 2         -->  merged-intermediate
> 	  /foo/bar   /foo/bar (whiteout)
> 
> layer0 + merged-intermediate		      -->  merged


I could add some more background to this, assuming layer 0 is a
baseos layer (e.g. almost all images use this layer); and layer 1 +
layer 2 belongs to some specific workload images;

since layer 1 + layer 2 are always used together, so we could merge
layer 1 + layer 2 as a new merged layer to avoid extra overhead of
too many overlay layer dirs (but to simplify, here we just illustrate
layer 1 and layer 2, there could be layer 3, 4, ...), but layer 1 +
layer 2 has no relationship with layer 0 in principle (in principle,
merge tool doesn't need to know if layer 0 or any underlay layer
exists).

So if we merge layer 1 + layer 2 here first, and use layer0 together
with the merged layer, it could generate such whiteout cases
described before.

Anyway, we could work around this in the merge tool, but I'm not
sure if it's a design constaint of overlayfs.

Thanks,
Gao Xiang

> 
> Then there comes the problem: when merging layer1 and layer2, I need to
> keep the whiteout in the intermediate merged image though the target of
> the whiteout has showed up in underlying layer (/foo/bar in layer 1),
> because I have no idea if "/foo/bar" exits in the following further
> underlying layer (layer 0).  Reusing this logic, the whiteout is kept
> there in the final merged image after merging layer0 and
> merged-intermediate.
> 
> Then if "/foo" is not a merged directory, the "/foo/bar" whiteout will
> be exposed in the overlayfs unexpectedly.
> 
> Currently we work around this in erofs-utils side.  Apart from setting
> origin xattr on the parent directory of the whiteout, I'm not sure if
> the above use case is reasonable enough to fix this in the kernel side.
> 

Re: [potential issue, question] whiteout shows up in merged directory

[Amir Goldstein]

On Mon, Sep 4, 2023 at 4:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>
>
>
> On 2023/9/4 20:49, Jingbo Xu wrote:
>
> ...
>
> >
> > Thanks for the reply and it's really helpful to me.
> >
> > I can understand in the normal use case, whiteout can not appear in
> > non-merged directory without origin xattr, except it's hand crafted.
> >
> > But indeed we suffer from this issue in the tarfs for erofs-utils we are
> > developing. As described previously, in tarfs mode erofs-utils can
> > convert each tar layer into one separate erofs image, and then merge
> > these erofs images into one merged erofs image in a overlayfs-like model.
> >
> > Suppose:
> >
> > layer 0 + layer 1   +        layer 2         -->  merged
> >         /foo/bar   /foo/bar (whiteout)
> >
> >
> > To speed the merging process, we may merge the two top-most layers
> > (layer 1 and layer 2) first, and then make layer0 merged into the final
> > merged image as:
> >
> >
> >
> >             layer 1   +        layer 2         -->  merged-intermediate
> >         /foo/bar   /foo/bar (whiteout)
> >
> > layer0 + merged-intermediate                -->  merged
>
>
> I could add some more background to this, assuming layer 0 is a
> baseos layer (e.g. almost all images use this layer); and layer 1 +
> layer 2 belongs to some specific workload images;
>
> since layer 1 + layer 2 are always used together, so we could merge
> layer 1 + layer 2 as a new merged layer to avoid extra overhead of
> too many overlay layer dirs (but to simplify, here we just illustrate
> layer 1 and layer 2, there could be layer 3, 4, ...), but layer 1 +
> layer 2 has no relationship with layer 0 in principle (in principle,
> merge tool doesn't need to know if layer 0 or any underlay layer
> exists).
>
> So if we merge layer 1 + layer 2 here first, and use layer0 together
> with the merged layer, it could generate such whiteout cases
> described before.
>
...
> >
> > Then there comes the problem: when merging layer1 and layer2, I need to
> > keep the whiteout in the intermediate merged image though the target of
> > the whiteout has showed up in underlying layer (/foo/bar in layer 1),
> > because I have no idea if "/foo/bar" exits in the following further
> > underlying layer (layer 0).  Reusing this logic, the whiteout is kept
> > there in the final merged image after merging layer0 and
> > merged-intermediate.
> >
> > Then if "/foo" is not a merged directory, the "/foo/bar" whiteout will
> > be exposed in the overlayfs unexpectedly.
> >
> > Currently we work around this in erofs-utils side.  Apart from setting
> > origin xattr on the parent directory of the whiteout, I'm not sure if
> > the above use case is reasonable enough to fix this in the kernel side.
> >
> Anyway, we could work around this in the merge tool, but I'm not
> sure if it's a design constaint of overlayfs.
>

Let me put it this way:
If there was an official offline tool to merge overlayfs layers
I would expect that tool to mark the offline merged directories
with an empty "trusted.overlayfs.origin", to be able to distinguish
them from pure non-merge directories.

I do not consider dealing with this in erofs-utils side a workaround
I consider it crafting layers in expected overlayfs format.

You should know that there are potential costs for marking a directory
as merged directory - ovl_iterate() implementation for merged dirs
that needs to filter out whiteouts is quite different than the
ovl_iterate_real() case -
The entire dirs needs to be read into cache before any response
could be returned. For very large dirs this may matter.

So you may want your tool to be able to clear the unneeded whiteouts
and unneeded origin xattr eventually.

OTOH, ovl_dir_read_impure() with xino enabled on layers
not from the same fs, has quite a similar impact.
Not sure if this configuration is relevant for your use case.

Thanks,
Amir.

Re: [potential issue, question] whiteout shows up in merged directory

[Gao Xiang]

Hi Amir,

On 2023/9/4 22:07, Amir Goldstein wrote:
> On Mon, Sep 4, 2023 at 4:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>>
>>
>>
>> On 2023/9/4 20:49, Jingbo Xu wrote:
>>
>> ...
>>
>>>
>>> Thanks for the reply and it's really helpful to me.
>>>
>>> I can understand in the normal use case, whiteout can not appear in
>>> non-merged directory without origin xattr, except it's hand crafted.
>>>
>>> But indeed we suffer from this issue in the tarfs for erofs-utils we are
>>> developing. As described previously, in tarfs mode erofs-utils can
>>> convert each tar layer into one separate erofs image, and then merge
>>> these erofs images into one merged erofs image in a overlayfs-like model.
>>>
>>> Suppose:
>>>
>>> layer 0 + layer 1   +        layer 2         -->  merged
>>>          /foo/bar   /foo/bar (whiteout)
>>>
>>>
>>> To speed the merging process, we may merge the two top-most layers
>>> (layer 1 and layer 2) first, and then make layer0 merged into the final
>>> merged image as:
>>>
>>>
>>>
>>>              layer 1   +        layer 2         -->  merged-intermediate
>>>          /foo/bar   /foo/bar (whiteout)
>>>
>>> layer0 + merged-intermediate                -->  merged
>>
>>
>> I could add some more background to this, assuming layer 0 is a
>> baseos layer (e.g. almost all images use this layer); and layer 1 +
>> layer 2 belongs to some specific workload images;
>>
>> since layer 1 + layer 2 are always used together, so we could merge
>> layer 1 + layer 2 as a new merged layer to avoid extra overhead of
>> too many overlay layer dirs (but to simplify, here we just illustrate
>> layer 1 and layer 2, there could be layer 3, 4, ...), but layer 1 +
>> layer 2 has no relationship with layer 0 in principle (in principle,
>> merge tool doesn't need to know if layer 0 or any underlay layer
>> exists).
>>
>> So if we merge layer 1 + layer 2 here first, and use layer0 together
>> with the merged layer, it could generate such whiteout cases
>> described before.
>>
> ...
>>>
>>> Then there comes the problem: when merging layer1 and layer2, I need to
>>> keep the whiteout in the intermediate merged image though the target of
>>> the whiteout has showed up in underlying layer (/foo/bar in layer 1),
>>> because I have no idea if "/foo/bar" exits in the following further
>>> underlying layer (layer 0).  Reusing this logic, the whiteout is kept
>>> there in the final merged image after merging layer0 and
>>> merged-intermediate.
>>>
>>> Then if "/foo" is not a merged directory, the "/foo/bar" whiteout will
>>> be exposed in the overlayfs unexpectedly.
>>>
>>> Currently we work around this in erofs-utils side.  Apart from setting
>>> origin xattr on the parent directory of the whiteout, I'm not sure if
>>> the above use case is reasonable enough to fix this in the kernel side.
>>>
>> Anyway, we could work around this in the merge tool, but I'm not
>> sure if it's a design constaint of overlayfs.
>>
> 
> Let me put it this way:
> If there was an official offline tool to merge overlayfs layers
> I would expect that tool to mark the offline merged directories
> with an empty "trusted.overlayfs.origin", to be able to distinguish
> them from pure non-merge directories.
> 
> I do not consider dealing with this in erofs-utils side a workaround
> I consider it crafting layers in expected overlayfs format.

Thanks for the hints.

Ok, marking impure makes sense as long as it's properly described.

Just tried to describe the background since the question I think
is not quite erofs-utils specific, btw, if there could be some
reference official offline tool, that would be great!

> 
> You should know that there are potential costs for marking a directory
> as merged directory - ovl_iterate() implementation for merged dirs
> that needs to filter out whiteouts is quite different than the
> ovl_iterate_real() case -
> The entire dirs needs to be read into cache before any response
> could be returned. For very large dirs this may matter.
> 
> So you may want your tool to be able to clear the unneeded whiteouts
> and unneeded origin xattr eventually.

Yes, I know there is some overhead though, so I tend to add
some option to the merge tool called "--keep-whiteout=0" to
formally drop unneeded whiteouts in the end, and I think we
also need to clear unneeded origin xattrs later.  Jingbo once
would like to confirm the best way to describe such situation
to work out the merge tool.

> 
> OTOH, ovl_dir_read_impure() with xino enabled on layers
> not from the same fs, has quite a similar impact.
> Not sure if this configuration is relevant for your use case.

Thanks for the reminder, we will check later (off work now..)

Thanks,
Gao Xiang

> 
> Thanks,
> Amir.

Re: [potential issue, question] whiteout shows up in merged directory

[Amir Goldstein]

On Mon, Sep 4, 2023 at 5:38 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>
> Hi Amir,
>
> On 2023/9/4 22:07, Amir Goldstein wrote:
> > On Mon, Sep 4, 2023 at 4:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
> >>
> >>
> >>
> >> On 2023/9/4 20:49, Jingbo Xu wrote:
> >>
> >> ...
> >>
> >>>
> >>> Thanks for the reply and it's really helpful to me.
> >>>
> >>> I can understand in the normal use case, whiteout can not appear in
> >>> non-merged directory without origin xattr, except it's hand crafted.
> >>>
> >>> But indeed we suffer from this issue in the tarfs for erofs-utils we are
> >>> developing. As described previously, in tarfs mode erofs-utils can
> >>> convert each tar layer into one separate erofs image, and then merge
> >>> these erofs images into one merged erofs image in a overlayfs-like model.
> >>>
> >>> Suppose:
> >>>
> >>> layer 0 + layer 1   +        layer 2         -->  merged
> >>>          /foo/bar   /foo/bar (whiteout)
> >>>
> >>>
> >>> To speed the merging process, we may merge the two top-most layers
> >>> (layer 1 and layer 2) first, and then make layer0 merged into the final
> >>> merged image as:
> >>>
> >>>
> >>>
> >>>              layer 1   +        layer 2         -->  merged-intermediate
> >>>          /foo/bar   /foo/bar (whiteout)
> >>>
> >>> layer0 + merged-intermediate                -->  merged
> >>
> >>
> >> I could add some more background to this, assuming layer 0 is a
> >> baseos layer (e.g. almost all images use this layer); and layer 1 +
> >> layer 2 belongs to some specific workload images;
> >>
> >> since layer 1 + layer 2 are always used together, so we could merge
> >> layer 1 + layer 2 as a new merged layer to avoid extra overhead of
> >> too many overlay layer dirs (but to simplify, here we just illustrate
> >> layer 1 and layer 2, there could be layer 3, 4, ...), but layer 1 +
> >> layer 2 has no relationship with layer 0 in principle (in principle,
> >> merge tool doesn't need to know if layer 0 or any underlay layer
> >> exists).
> >>
> >> So if we merge layer 1 + layer 2 here first, and use layer0 together
> >> with the merged layer, it could generate such whiteout cases
> >> described before.
> >>
> > ...
> >>>
> >>> Then there comes the problem: when merging layer1 and layer2, I need to
> >>> keep the whiteout in the intermediate merged image though the target of
> >>> the whiteout has showed up in underlying layer (/foo/bar in layer 1),
> >>> because I have no idea if "/foo/bar" exits in the following further
> >>> underlying layer (layer 0).  Reusing this logic, the whiteout is kept
> >>> there in the final merged image after merging layer0 and
> >>> merged-intermediate.
> >>>
> >>> Then if "/foo" is not a merged directory, the "/foo/bar" whiteout will
> >>> be exposed in the overlayfs unexpectedly.
> >>>
> >>> Currently we work around this in erofs-utils side.  Apart from setting
> >>> origin xattr on the parent directory of the whiteout, I'm not sure if
> >>> the above use case is reasonable enough to fix this in the kernel side.
> >>>
> >> Anyway, we could work around this in the merge tool, but I'm not
> >> sure if it's a design constaint of overlayfs.
> >>
> >
> > Let me put it this way:
> > If there was an official offline tool to merge overlayfs layers
> > I would expect that tool to mark the offline merged directories
> > with an empty "trusted.overlayfs.origin", to be able to distinguish
> > them from pure non-merge directories.
> >
> > I do not consider dealing with this in erofs-utils side a workaround
> > I consider it crafting layers in expected overlayfs format.
>
> Thanks for the hints.
>
> Ok, marking impure makes sense as long as it's properly described.
>
> Just tried to describe the background since the question I think
> is not quite erofs-utils specific, btw, if there could be some
> reference official offline tool, that would be great!
>

There is this tool from kmxz that supports offline merge:
https://github.com/kmxz/overlayfs-tools
but it is not in any way "official".

I have contributed redirect and metacopy support in 2020
and there hasn't been much traffic since.
This tool does not deal with origin and impure xattrs.

Thanks,
Amir.

Re: [potential issue, question] whiteout shows up in merged directory

[Gao Xiang]

On 2023/9/4 23:03, Amir Goldstein wrote:
> On Mon, Sep 4, 2023 at 5:38 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:

..

>>>>
>>>
>>> Let me put it this way:
>>> If there was an official offline tool to merge overlayfs layers
>>> I would expect that tool to mark the offline merged directories
>>> with an empty "trusted.overlayfs.origin", to be able to distinguish
>>> them from pure non-merge directories.
>>>
>>> I do not consider dealing with this in erofs-utils side a workaround
>>> I consider it crafting layers in expected overlayfs format.
>>
>> Thanks for the hints.
>>
>> Ok, marking impure makes sense as long as it's properly described.
>>
>> Just tried to describe the background since the question I think
>> is not quite erofs-utils specific, btw, if there could be some
>> reference official offline tool, that would be great!
>>
> 
> There is this tool from kmxz that supports offline merge:
> https://github.com/kmxz/overlayfs-tools
> but it is not in any way "official".
> 
> I have contributed redirect and metacopy support in 2020
> and there hasn't been much traffic since.
> This tool does not deal with origin and impure xattrs.

Thanks, very useful link! will also check this later.

> 
> Thanks,
> Amir.

Re: [potential issue, question] whiteout shows up in merged directory

[Jingbo Xu]

On 9/4/23 10:07 PM, Amir Goldstein wrote:
> On Mon, Sep 4, 2023 at 4:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>>
>>
>>
>> On 2023/9/4 20:49, Jingbo Xu wrote:
>>
>> ...
>>
>>>
>>> Thanks for the reply and it's really helpful to me.
>>>
>>> I can understand in the normal use case, whiteout can not appear in
>>> non-merged directory without origin xattr, except it's hand crafted.
>>>
>>> But indeed we suffer from this issue in the tarfs for erofs-utils we are
>>> developing. As described previously, in tarfs mode erofs-utils can
>>> convert each tar layer into one separate erofs image, and then merge
>>> these erofs images into one merged erofs image in a overlayfs-like model.
>>>
>>> Suppose:
>>>
>>> layer 0 + layer 1   +        layer 2         -->  merged
>>>         /foo/bar   /foo/bar (whiteout)
>>>
>>>
>>> To speed the merging process, we may merge the two top-most layers
>>> (layer 1 and layer 2) first, and then make layer0 merged into the final
>>> merged image as:
>>>
>>>
>>>
>>>             layer 1   +        layer 2         -->  merged-intermediate
>>>         /foo/bar   /foo/bar (whiteout)
>>>
>>> layer0 + merged-intermediate                -->  merged
>>
>>
>> I could add some more background to this, assuming layer 0 is a
>> baseos layer (e.g. almost all images use this layer); and layer 1 +
>> layer 2 belongs to some specific workload images;
>>
>> since layer 1 + layer 2 are always used together, so we could merge
>> layer 1 + layer 2 as a new merged layer to avoid extra overhead of
>> too many overlay layer dirs (but to simplify, here we just illustrate
>> layer 1 and layer 2, there could be layer 3, 4, ...), but layer 1 +
>> layer 2 has no relationship with layer 0 in principle (in principle,
>> merge tool doesn't need to know if layer 0 or any underlay layer
>> exists).
>>
>> So if we merge layer 1 + layer 2 here first, and use layer0 together
>> with the merged layer, it could generate such whiteout cases
>> described before.
>>
> ...
>>>
>>> Then there comes the problem: when merging layer1 and layer2, I need to
>>> keep the whiteout in the intermediate merged image though the target of
>>> the whiteout has showed up in underlying layer (/foo/bar in layer 1),
>>> because I have no idea if "/foo/bar" exits in the following further
>>> underlying layer (layer 0).  Reusing this logic, the whiteout is kept
>>> there in the final merged image after merging layer0 and
>>> merged-intermediate.
>>>
>>> Then if "/foo" is not a merged directory, the "/foo/bar" whiteout will
>>> be exposed in the overlayfs unexpectedly.
>>>
>>> Currently we work around this in erofs-utils side.  Apart from setting
>>> origin xattr on the parent directory of the whiteout, I'm not sure if
>>> the above use case is reasonable enough to fix this in the kernel side.
>>>
>> Anyway, we could work around this in the merge tool, but I'm not
>> sure if it's a design constaint of overlayfs.
>>
> 
> Let me put it this way:
> If there was an official offline tool to merge overlayfs layers
> I would expect that tool to mark the offline merged directories
> with an empty "trusted.overlayfs.origin", to be able to distinguish
> them from pure non-merge directories.
> 
> I do not consider dealing with this in erofs-utils side a workaround
> I consider it crafting layers in expected overlayfs format.

Thanks for the suggestion.  I just tested it and marking parent
directory of the whiteout with origin xattr indeed fixes this issue.

> 
> You should know that there are potential costs for marking a directory
> as merged directory - ovl_iterate() implementation for merged dirs
> that needs to filter out whiteouts is quite different than the
> ovl_iterate_real() case -
> The entire dirs needs to be read into cache before any response
> could be returned. For very large dirs this may matter.

Thanks for the reminder.

> 
> So you may want your tool to be able to clear the unneeded whiteouts
> and unneeded origin xattr eventually.
> 
> OTOH, ovl_dir_read_impure() with xino enabled on layers
> not from the same fs, has quite a similar impact.
> Not sure if this configuration is relevant for your use case.

Also will check it later.


-- 
Thanks,
Jingbo