Re: [potential issue, question] whiteout shows up in merged directory

[Jingbo Xu <jefflexu@linux.alibaba.com>]

On 9/4/23 10:07 PM, Amir Goldstein wrote:
> On Mon, Sep 4, 2023 at 4:27 PM Gao Xiang <hsiangkao@linux.alibaba.com> wrote:
>>
>>
>>
>> On 2023/9/4 20:49, Jingbo Xu wrote:
>>
>> ...
>>
>>>
>>> Thanks for the reply and it's really helpful to me.
>>>
>>> I can understand in the normal use case, whiteout can not appear in
>>> non-merged directory without origin xattr, except it's hand crafted.
>>>
>>> But indeed we suffer from this issue in the tarfs for erofs-utils we are
>>> developing. As described previously, in tarfs mode erofs-utils can
>>> convert each tar layer into one separate erofs image, and then merge
>>> these erofs images into one merged erofs image in a overlayfs-like model.
>>>
>>> Suppose:
>>>
>>> layer 0 + layer 1   +        layer 2         -->  merged
>>>         /foo/bar   /foo/bar (whiteout)
>>>
>>>
>>> To speed the merging process, we may merge the two top-most layers
>>> (layer 1 and layer 2) first, and then make layer0 merged into the final
>>> merged image as:
>>>
>>>
>>>
>>>             layer 1   +        layer 2         -->  merged-intermediate
>>>         /foo/bar   /foo/bar (whiteout)
>>>
>>> layer0 + merged-intermediate                -->  merged
>>
>>
>> I could add some more background to this, assuming layer 0 is a
>> baseos layer (e.g. almost all images use this layer); and layer 1 +
>> layer 2 belongs to some specific workload images;
>>
>> since layer 1 + layer 2 are always used together, so we could merge
>> layer 1 + layer 2 as a new merged layer to avoid extra overhead of
>> too many overlay layer dirs (but to simplify, here we just illustrate
>> layer 1 and layer 2, there could be layer 3, 4, ...), but layer 1 +
>> layer 2 has no relationship with layer 0 in principle (in principle,
>> merge tool doesn't need to know if layer 0 or any underlay layer
>> exists).
>>
>> So if we merge layer 1 + layer 2 here first, and use layer0 together
>> with the merged layer, it could generate such whiteout cases
>> described before.
>>
> ...
>>>
>>> Then there comes the problem: when merging layer1 and layer2, I need to
>>> keep the whiteout in the intermediate merged image though the target of
>>> the whiteout has showed up in underlying layer (/foo/bar in layer 1),
>>> because I have no idea if "/foo/bar" exits in the following further
>>> underlying layer (layer 0).  Reusing this logic, the whiteout is kept
>>> there in the final merged image after merging layer0 and
>>> merged-intermediate.
>>>
>>> Then if "/foo" is not a merged directory, the "/foo/bar" whiteout will
>>> be exposed in the overlayfs unexpectedly.
>>>
>>> Currently we work around this in erofs-utils side.  Apart from setting
>>> origin xattr on the parent directory of the whiteout, I'm not sure if
>>> the above use case is reasonable enough to fix this in the kernel side.
>>>
>> Anyway, we could work around this in the merge tool, but I'm not
>> sure if it's a design constaint of overlayfs.
>>
> 
> Let me put it this way:
> If there was an official offline tool to merge overlayfs layers
> I would expect that tool to mark the offline merged directories
> with an empty "trusted.overlayfs.origin", to be able to distinguish
> them from pure non-merge directories.
> 
> I do not consider dealing with this in erofs-utils side a workaround
> I consider it crafting layers in expected overlayfs format.

Thanks for the suggestion.  I just tested it and marking parent
directory of the whiteout with origin xattr indeed fixes this issue.

> 
> You should know that there are potential costs for marking a directory
> as merged directory - ovl_iterate() implementation for merged dirs
> that needs to filter out whiteouts is quite different than the
> ovl_iterate_real() case -
> The entire dirs needs to be read into cache before any response
> could be returned. For very large dirs this may matter.

Thanks for the reminder.

> 
> So you may want your tool to be able to clear the unneeded whiteouts
> and unneeded origin xattr eventually.
> 
> OTOH, ovl_dir_read_impure() with xino enabled on layers
> not from the same fs, has quite a similar impact.
> Not sure if this configuration is relevant for your use case.

Also will check it later.


-- 
Thanks,
Jingbo