Re: Overlay Filesystem Documentation page

Re: Overlay Filesystem Documentation page

[Amir Goldstein]

On Wed, May 22, 2024 at 4:21 PM Edouard Gaulué <edouard@e-gaulue.com> wrote:
>
> Hi Neil,
>
> Here are 2 remarks regarding: https://docs.kernel.org/filesystems/overlayfs.html
>
>  * 1rst
>
> We have at the begining:
>
> Written by: Neil Brown Please see MAINTAINERS file for where to send questions.
>
> I tried to figure out where that MAINTAINERS file was, looking in overlayfs source code but without luck. I tried to google "overlayfs MAINTAINERS" without success. Hopefully chatGPT leads me to the right place.
>

https://github.com/torvalds/linux/blob/master/MAINTAINERS#L16875

which also lists the mailing list (now CCed)

>  * 2nd
>
> The doc states in Changes to underlying filesystems:
>
> Changes to the underlying filesystems while part of a mounted overlay filesystem are not allowed. If the underlying filesystem is changed, the behavior of the overlay is undefined, though it will not result in a crash or deadlock.
>
> Offline changes, when the overlay is not mounted, are allowed to the upper tree. Offline changes to the lower tree are only allowed if the “metacopy”, “index”, “xino” and “redirect_dir” features have not been used. If the lower tree is modified and any of these features has been used, the behavior of the overlay is undefined, though it will not result in a crash or deadlock.
>

From here below the NFS export story is my own blurb.
I admit is it hard to understand.
I do not consider myself a very good technical writer...

> When the overlay NFS export feature is enabled, overlay filesystems behavior on offline changes of the underlying lower layer is different than the behavior when NFS export is disabled.
>
> On every copy_up, an NFS file handle of the lower inode, along with the UUID of the lower filesystem, are encoded and stored in an extended attribute “trusted.overlay.origin” on the upper inode.
>
> When the NFS export feature is enabled, a lookup of a merged directory, that found a lower directory at the lookup path or at the path pointed to by the “trusted.overlay.redirect” extended attribute, will verify that the found lower directory file handle and lower filesystem UUID match the origin file handle that was stored at copy_up time. If a found lower directory does not match the stored origin, that directory will not be merged with the upper directory.
>
> "Offline changes" expression is not really clear or we lack context. I did share it with colleagues and some understood it as:
>
> changes made while the overlay doesn't exists (is not mounted)
> changes made while the overlay is not NFS exported (as the rest of this paragraph is concerned with NFS)

It is the first one, as written: "Offline changes, *when the overlay
is not mounted*,..."

>
> Most of the people who think the first is the right solution, don't understand why a change in the lower tree could have an impact on the overlay if it is not mount and so doesn't exist. It will need a new mount command anyway.

The meaning is that making changes offline to the lower layers while
the overlay is not mounted, will cause undefined behavior after mounting
the overlay with those lower layers again.

>
> According to me, it's due to xattrs that remains in upper tree if “metacopy”, “index”, “xino” or “redirect_dir” have been in use once.

Yes, those xattrs refer to the information observed in the lower layer
during copy up.
If you, for example, delete and recreate a file/dir of the same name in lower
layer while overlayfs was offline, then after overlayfs is online
again, you may not
be able to access the re-created lower file/dir.

> Would it be possible to clarify?
>

Feel free to suggest a different phrasing based on your understand.

Submitting a patch to Documentation/filesystems/overlayfs.rst is the
preferred format, but if you like to propose a text here that is fine too.

Thanks,
Amir.

Re: Overlay Filesystem Documentation page

[Amir Goldstein]

On Wed, May 22, 2024 at 7:27 PM Edouard Gaulué <edouard@e-gaulue.com> wrote:
>
> I tried to provide a proposal, but I worry there are too much thing I don't fully understand.
>
> Could you look at this "use case" and associated questions (I tried to answer some)? With those answers, I should be able to propose something for the first 2 paragraghs.
>
> /dir1 : production website
> /dir2 : website source code (git repo)
> /dir3 : transitional
> /dir4 : workdir
> /dir5 : dev website
>
> Mount command: sudo mount -t overlay overlay -o lowerdir=/dir2:/dir1,upperdir=/dir3,workdir=/dir4 /dir5
>
> According to me, undefined behaviour is coming from xattrs created in /dir3. So, everytime a file is changed in /dir2: /dir3 is emptied and /dir5 is remounted.
>
>
> Q1: If “metacopy”, “index”, “xino” or “redirect_dir” is not used, do we really need to empty /dir3 ? Should be 'no' due to the fact we won't have those specifics xattrs.

hard to answer what you *should* do.
only thing I can say is that the results would be quite predictable -
dir5/ will be the merge of dir1+dir2+dir3 as one could expect.

Note that there are some other xattr in dir3 files - "origin", "impure"
Specifically, "origin" is always set on copy up and there is no way to
opt out from it. if does have some impact in this scenario related to which
inode numbers files in dir5/ will have after this maneuver
I have no desire to document the expected behavior in this regard.

>
> Q2: If “metacopy”, “index”, “xino” or “redirect_dir” is used. Does emptying /dir3 before remount remove all possible undefined behaviours ? Should be 'yes', we are restarting from scratch.
>

dir3 itself also has xattrs, so it needs to be recreated.
dir4 (the workdir) as well.

> Q3: Knowing that the website will never modify any files in /dir2 (neither production nor dev), do we really need to remount the overlay everytime a file is changed in /dir2 ?
> Files in /dir3 will never overlap with those in /dir2. There shouldn't be any xattrs inconsistency. Is there other risks?
>

Yes.
Overlay has many caches. Changing lower layers while overlay is
mounted will have unpredicted outcome.

> Q4: If production website generate cache files in /dir1 and dev website generate cache files in /dir3 that may opverlap. What are the risks? For /dir1: I suppose none. For /dir3 ? And /dir5 ? In ohter words, what do we mean by "the behavior of the overlay is undefined"? Could we say : "we don't know which layer file will be served"? Or is it worse?
>

For changes while overlay is mounted could be much worse.
Let's just say that the statement "...though it will not result in a
crash or deadlock."
is not really a promise - it is only a statement of intentions.

For changes while overlay is offline you wont know which file will be served
and it is quite possible the -EIO will be served in many cases.

> Q5: Would mount -t overlay overlay -o lowerdir=/dir2:/dir1 /dir5 get to an undefined behavior, if changes to /dir1 or /dir2 are made? According to me, as /dir5 is RO, it should work, always respecting /dir2 priority above /dir1.

For changes while overlay is mounted, same answer as above.
For changes while overlay is offline, same answer as for Q1.
results should be quite predictable - dir5 will be the merge of dir1+dir2.

Please refrain from proposing detailed documentation on what "undefined"
behavior means to kernel documentation, because I do not want to commit
to any specific "undefined" behavior in the future.

Thanks,
Amir.

Re: Overlay Filesystem Documentation page

[Amir Goldstein]

On Thu, May 23, 2024 at 11:56 AM Edouard Gaulué <edouard@e-gaulue.com> wrote:
>
> Thanks a lot Amir,
>
> Here is a proposal, but consider it as a draft:
>
> "
>
> Changes to the underlying filesystems while part of a mounted overlay filesystem are not supported. Thought Overlayfs will try to handle those changed files in a way it may not result in a crash or deadlock, you shouldn't do it. Due to multiple reasons involving caches, attributes, and others, if the underlying filesystem is changed, the behavior of the overlay gets "undefined", so you can't trust it anymore.
>
> Offline changes (i.e. when the overlay is not mounted) are allowed to the upper tree. But beware of remount after offline changes to the lower tree. They are almost supported if the “metacopy”, “index”, “xino” and “redirect_dir” features have not been used. If the lower tree is modified and any of these features has been used before on this overlay, the behavior can also get "undefined".
>

Edouard,

I am sorry to be discouraging, but I personally don't see much value
in this rephrasing
and I also don't think that the current documentation is lacking in this point.
This is my personal opinion and review is a community procedure.
If there are proponents for this rewrite let them speak up.

> "
>
> I came to overlayfs, because of chatGPT. It easily proposes to bind mount between upper and lower. Just say: "I want the feature of overlayfs, but for this specific directory, I want it to write on lower". The provided solution writes on the underlying filesystems (through bind), even if the result is quite predictable and almost works. Now I understand better the way overlayfs is working, I think there should be a warning in the documentation (that chatGPT or others may read next time) regarding this:
>

Overalyfs is not the only way to merge directories. This is out of scope.

> "
>
> Overlayfs will never write on the lower filesystems, so it will never arm them. But mind the interactions you could create outside of overlayfs using tools like bind mounts, "rsync" or even "cp" between upper filesystem (or merged) and lower ones. Those lead to changes to the underlying filesystems and should be avoided as already stated.
>

Sorry. This feels out of scope to me.
I think the introduction sections describe overlayfs and lower and
upper layers well enough.

> "
>
> Finally, I think it would be great to have an option to clean dirs of all previous xattrs set by overlayfs at mount time. Or a command line in the documentation to explain how to get the same. In the meanwhile, I would add:
>
> "
>
> Note: in those specific cases where data written to the overlay can be recreated without significant effort (like in volatile), you can always recreate an empty upperdir and workdir before remount.
>
> "
>
> But it doesn't handle the case of those who had bound upper and lower, and decide one day, to use the lower as an upper.
>

Sorry, but I am not sure if those details belong in the scope of this document,
because I don't think we would like to commit to any specific procedure of
cleaning the upper layer.

I do hear your concerns as a user, but I don't think that better documentation
alone is going to solve them.

What overlayfs has always been missing is a counterpart library and user tools
to deal with those things.

There has been an attempt in the past to start overlayfs-progs [1] and later
overlayfs-tool2 project [2] to work on offline overlayfs layers.
I even contributed the "overlay deref" command [3] which partly does what
you are looking for, but it does not look like this project is
actively developed
except for a recent merge of the fsck tool from overlayfs-progs.

Thanks,
Amir.

[1] https://github.com/hisilicon/overlayfs-progs
[2] https://github.com/kmxz/overlayfs-tools
[3] https://github.com/kmxz/overlayfs-tools/pull/11

Re: Overlay Filesystem Documentation page

[Vyacheslav Yurkov]

Hey everyone,
I'm inclined to agree with Amir here. The userspace tools can be 
improved / extended, but it's not the scope of the kernel documentation.

Vyacheslav

On 25.05.2024 09:32, Amir Goldstein wrote:
> On Thu, May 23, 2024 at 11:56 AM Edouard Gaulué <edouard@e-gaulue.com> wrote:
>> Thanks a lot Amir,
>>
>> Here is a proposal, but consider it as a draft:
>>
>> "
>>
>> Changes to the underlying filesystems while part of a mounted overlay filesystem are not supported. Thought Overlayfs will try to handle those changed files in a way it may not result in a crash or deadlock, you shouldn't do it. Due to multiple reasons involving caches, attributes, and others, if the underlying filesystem is changed, the behavior of the overlay gets "undefined", so you can't trust it anymore.
>>
>> Offline changes (i.e. when the overlay is not mounted) are allowed to the upper tree. But beware of remount after offline changes to the lower tree. They are almost supported if the “metacopy”, “index”, “xino” and “redirect_dir” features have not been used. If the lower tree is modified and any of these features has been used before on this overlay, the behavior can also get "undefined".
>>
> Edouard,
>
> I am sorry to be discouraging, but I personally don't see much value
> in this rephrasing
> and I also don't think that the current documentation is lacking in this point.
> This is my personal opinion and review is a community procedure.
> If there are proponents for this rewrite let them speak up.
>
>> "
>>
>> I came to overlayfs, because of chatGPT. It easily proposes to bind mount between upper and lower. Just say: "I want the feature of overlayfs, but for this specific directory, I want it to write on lower". The provided solution writes on the underlying filesystems (through bind), even if the result is quite predictable and almost works. Now I understand better the way overlayfs is working, I think there should be a warning in the documentation (that chatGPT or others may read next time) regarding this:
>>
> Overalyfs is not the only way to merge directories. This is out of scope.
>
>> "
>>
>> Overlayfs will never write on the lower filesystems, so it will never arm them. But mind the interactions you could create outside of overlayfs using tools like bind mounts, "rsync" or even "cp" between upper filesystem (or merged) and lower ones. Those lead to changes to the underlying filesystems and should be avoided as already stated.
>>
> Sorry. This feels out of scope to me.
> I think the introduction sections describe overlayfs and lower and
> upper layers well enough.
>
>> "
>>
>> Finally, I think it would be great to have an option to clean dirs of all previous xattrs set by overlayfs at mount time. Or a command line in the documentation to explain how to get the same. In the meanwhile, I would add:
>>
>> "
>>
>> Note: in those specific cases where data written to the overlay can be recreated without significant effort (like in volatile), you can always recreate an empty upperdir and workdir before remount.
>>
>> "
>>
>> But it doesn't handle the case of those who had bound upper and lower, and decide one day, to use the lower as an upper.
>>
> Sorry, but I am not sure if those details belong in the scope of this document,
> because I don't think we would like to commit to any specific procedure of
> cleaning the upper layer.
>
> I do hear your concerns as a user, but I don't think that better documentation
> alone is going to solve them.
>
> What overlayfs has always been missing is a counterpart library and user tools
> to deal with those things.
>
> There has been an attempt in the past to start overlayfs-progs [1] and later
> overlayfs-tool2 project [2] to work on offline overlayfs layers.
> I even contributed the "overlay deref" command [3] which partly does what
> you are looking for, but it does not look like this project is
> actively developed
> except for a recent merge of the fsck tool from overlayfs-progs.
>
> Thanks,
> Amir.
>
> [1] https://github.com/hisilicon/overlayfs-progs
> [2] https://github.com/kmxz/overlayfs-tools
> [3] https://github.com/kmxz/overlayfs-tools/pull/11

Re: Overlay Filesystem Documentation page

[Edouard Gaulué]

Le 25/05/2024 à 09:32, Amir Goldstein a écrit :
> On Thu, May 23, 2024 at 11:56 AM Edouard Gaulué <edouard@e-gaulue.com> wrote:
>> Thanks a lot Amir,
>>
>> Here is a proposal, but consider it as a draft:
>>
>> "
>>
>> Changes to the underlying filesystems while part of a mounted overlay filesystem are not supported. Thought Overlayfs will try to handle those changed files in a way it may not result in a crash or deadlock, you shouldn't do it. Due to multiple reasons involving caches, attributes, and others, if the underlying filesystem is changed, the behavior of the overlay gets "undefined", so you can't trust it anymore.
>>
>> Offline changes (i.e. when the overlay is not mounted) are allowed to the upper tree. But beware of remount after offline changes to the lower tree. They are almost supported if the “metacopy”, “index”, “xino” and “redirect_dir” features have not been used. If the lower tree is modified and any of these features has been used before on this overlay, the behavior can also get "undefined".
>>
> Edouard,
>
> I am sorry to be discouraging, but I personally don't see much value
> in this rephrasing
> and I also don't think that the current documentation is lacking in this point.
> This is my personal opinion and review is a community procedure.
> If there are proponents for this rewrite let them speak up.

Amir,

I agree with you, there is no new ideas, it's just rephrasing of the 
current documentation plus the information you brought in this thread. 
It just makes no way of misinterpretation, reason why I initialy opened 
this thread. For us, moving to overlayfs, it was important we undstand 
better those "offline changes" notions. For the kernel user community, I 
can't say if this rephrasing helps. That is just a CC0 proposal.

>
>> "
>>
>> I came to overlayfs, because of chatGPT. It easily proposes to bind mount between upper and lower. Just say: "I want the feature of overlayfs, but for this specific directory, I want it to write on lower". The provided solution writes on the underlying filesystems (through bind), even if the result is quite predictable and almost works. Now I understand better the way overlayfs is working, I think there should be a warning in the documentation (that chatGPT or others may read next time) regarding this:
>>
> Overalyfs is not the only way to merge directories. This is out of scope.
>
>> "
>>
>> Overlayfs will never write on the lower filesystems, so it will never arm them. But mind the interactions you could create outside of overlayfs using tools like bind mounts, "rsync" or even "cp" between upper filesystem (or merged) and lower ones. Those lead to changes to the underlying filesystems and should be avoided as already stated.
>>
> Sorry. This feels out of scope to me.
> I think the introduction sections describe overlayfs and lower and
> upper layers well enough.

Again, I agree with you. The documentation is quite clear, and once 
again, this proposal is more an important reminder for us, as users, 
than something describing the service.

We tried to look for a good place to get strong informations regarding 
overlayfs usages, but the best we found is the kernel documentation (in 
its last version). All the rest (blogs, "unix.stackexchange.com" or 
"stackoverflow.com") sometime gives misinformation regarding usage. 
Unfortunatly those looks to weight more than the kernel documentation in 
the IA algrotihms (or neural networks) and, of course, that is not the 
kernel documentation responsibility.

In a futur, where indexation wil be replace by IA, I just wonder if RFC, 
documentations, man pages and others shouldn't be more verbose and IA 
oriented. Of course, it's out of the scope of this thread.

>
>> "
>>
>> Finally, I think it would be great to have an option to clean dirs of all previous xattrs set by overlayfs at mount time. Or a command line in the documentation to explain how to get the same. In the meanwhile, I would add:
>>
>> "
>>
>> Note: in those specific cases where data written to the overlay can be recreated without significant effort (like in volatile), you can always recreate an empty upperdir and workdir before remount.
>>
>> "
>>
>> But it doesn't handle the case of those who had bound upper and lower, and decide one day, to use the lower as an upper.
>>
> Sorry, but I am not sure if those details belong in the scope of this document,
> because I don't think we would like to commit to any specific procedure of
> cleaning the upper layer.
>
> I do hear your concerns as a user, but I don't think that better documentation
> alone is going to solve them.
Fully agree.
>
> What overlayfs has always been missing is a counterpart library and user tools
> to deal with those things.
>
> There has been an attempt in the past to start overlayfs-progs [1] and later
> overlayfs-tool2 project [2] to work on offline overlayfs layers.
> I even contributed the "overlay deref" command [3] which partly does what
> you are looking for, but it does not look like this project is
> actively developed
> except for a recent merge of the fsck tool from overlayfs-progs.
Really interesting! I will look at it.
>
> Thanks,
> Amir.
>
> [1] https://github.com/hisilicon/overlayfs-progs
> [2] https://github.com/kmxz/overlayfs-tools
> [3] https://github.com/kmxz/overlayfs-tools/pull/11

Thanks a lot, Amir, your answers have always been helpful. The time you 
can give to answer ours questions will ever be better than the 
documentation.

Regards, Edouard