Re: Overlay Filesystem Documentation page

[Vyacheslav Yurkov <uvv.mail@gmail.com>]

Hey everyone,
I'm inclined to agree with Amir here. The userspace tools can be 
improved / extended, but it's not the scope of the kernel documentation.

Vyacheslav

On 25.05.2024 09:32, Amir Goldstein wrote:
> On Thu, May 23, 2024 at 11:56 AM Edouard Gaulué <edouard@e-gaulue.com> wrote:
>> Thanks a lot Amir,
>>
>> Here is a proposal, but consider it as a draft:
>>
>> "
>>
>> Changes to the underlying filesystems while part of a mounted overlay filesystem are not supported. Thought Overlayfs will try to handle those changed files in a way it may not result in a crash or deadlock, you shouldn't do it. Due to multiple reasons involving caches, attributes, and others, if the underlying filesystem is changed, the behavior of the overlay gets "undefined", so you can't trust it anymore.
>>
>> Offline changes (i.e. when the overlay is not mounted) are allowed to the upper tree. But beware of remount after offline changes to the lower tree. They are almost supported if the “metacopy”, “index”, “xino” and “redirect_dir” features have not been used. If the lower tree is modified and any of these features has been used before on this overlay, the behavior can also get "undefined".
>>
> Edouard,
>
> I am sorry to be discouraging, but I personally don't see much value
> in this rephrasing
> and I also don't think that the current documentation is lacking in this point.
> This is my personal opinion and review is a community procedure.
> If there are proponents for this rewrite let them speak up.
>
>> "
>>
>> I came to overlayfs, because of chatGPT. It easily proposes to bind mount between upper and lower. Just say: "I want the feature of overlayfs, but for this specific directory, I want it to write on lower". The provided solution writes on the underlying filesystems (through bind), even if the result is quite predictable and almost works. Now I understand better the way overlayfs is working, I think there should be a warning in the documentation (that chatGPT or others may read next time) regarding this:
>>
> Overalyfs is not the only way to merge directories. This is out of scope.
>
>> "
>>
>> Overlayfs will never write on the lower filesystems, so it will never arm them. But mind the interactions you could create outside of overlayfs using tools like bind mounts, "rsync" or even "cp" between upper filesystem (or merged) and lower ones. Those lead to changes to the underlying filesystems and should be avoided as already stated.
>>
> Sorry. This feels out of scope to me.
> I think the introduction sections describe overlayfs and lower and
> upper layers well enough.
>
>> "
>>
>> Finally, I think it would be great to have an option to clean dirs of all previous xattrs set by overlayfs at mount time. Or a command line in the documentation to explain how to get the same. In the meanwhile, I would add:
>>
>> "
>>
>> Note: in those specific cases where data written to the overlay can be recreated without significant effort (like in volatile), you can always recreate an empty upperdir and workdir before remount.
>>
>> "
>>
>> But it doesn't handle the case of those who had bound upper and lower, and decide one day, to use the lower as an upper.
>>
> Sorry, but I am not sure if those details belong in the scope of this document,
> because I don't think we would like to commit to any specific procedure of
> cleaning the upper layer.
>
> I do hear your concerns as a user, but I don't think that better documentation
> alone is going to solve them.
>
> What overlayfs has always been missing is a counterpart library and user tools
> to deal with those things.
>
> There has been an attempt in the past to start overlayfs-progs [1] and later
> overlayfs-tool2 project [2] to work on offline overlayfs layers.
> I even contributed the "overlay deref" command [3] which partly does what
> you are looking for, but it does not look like this project is
> actively developed
> except for a recent merge of the fsck tool from overlayfs-progs.
>
> Thanks,
> Amir.
>
> [1] https://github.com/hisilicon/overlayfs-progs
> [2] https://github.com/kmxz/overlayfs-tools
> [3] https://github.com/kmxz/overlayfs-tools/pull/11