lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Zorro Lang]

Hi,

Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
specific mount options), e.g.

FSTYP         -- overlay
PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt

generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
    --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
    +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
    @@ -1,5 +1,5 @@
     QA output created by 294
    -mknod: SCRATCH_MNT/294.test/testnode: File exists
    -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
    -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
    -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
    +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
    +       dmesg(1) may have more information after failed mount system call.
    ...
    (Run 'diff -u /var/lib/xfstests/tests/generic/294.out /var/lib/xfstests/results//generic/294.out.bad'  to see the entire diff)
Ran: generic/294
Failures: generic/294
Failed 1 of 1 tests

Similar failures happened on g/294, g/306, g/417, g/599, overlay/035,
o/103~109, o/114, etc.

They all looks like related with new mount APIs of util-linux package.

Not sure if this's an issue of overlay, or overlay test cases, or new mount.
So report to get more attention/review, as some new linux distributions might
start to use new util-linux with new mount APIs.

Thanks,
Zorro

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Christian Brauner]

On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
> Hi,
> 
> Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
> specific mount options), e.g.
> 
> FSTYP         -- overlay
> PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
> MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
> 
> generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
>     --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
>     +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
>     @@ -1,5 +1,5 @@
>      QA output created by 294
>     -mknod: SCRATCH_MNT/294.test/testnode: File exists
>     -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
>     -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
>     -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
>     +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
>     +       dmesg(1) may have more information after failed mount system call.

In the new mount api overlayfs has been changed to reject invalid mount
option on remount whereas in the old mount api we just igorned them.
If this a big problem then we need to change overlayfs to continue
ignoring garbage mount options passed to it during remount.

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Zorro Lang]

On Mon, Oct 28, 2024 at 01:22:52PM +0100, Christian Brauner wrote:
> On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
> > Hi,
> > 
> > Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
> > specific mount options), e.g.
> > 
> > FSTYP         -- overlay
> > PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
> > MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
> > MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
> > 
> > generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
> >     --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
> >     +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
> >     @@ -1,5 +1,5 @@
> >      QA output created by 294
> >     -mknod: SCRATCH_MNT/294.test/testnode: File exists
> >     -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
> >     -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
> >     -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
> >     +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
> >     +       dmesg(1) may have more information after failed mount system call.
> 
> In the new mount api overlayfs has been changed to reject invalid mount
> option on remount whereas in the old mount api we just igorned them.

Not only g/294 fails on new mount utils, not sure if all of them are from same issue.
If you need, I can paste all test failures (only from my side) at here.

> If this a big problem then we need to change overlayfs to continue
> ignoring garbage mount options passed to it during remount.

Do you mean this behavior change is only for overlayfs, doesn't affect other fs?

If it's not necessary, I think we'd better to not change the behaviors which we've
used so many years. But if you all agree with this change, then we need to update
related regression test cases and more scripts maybe.

Thanks,
Zorro

> 

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Dave Chinner]

On Tue, Oct 29, 2024 at 03:28:04AM +0800, Zorro Lang wrote:
> On Mon, Oct 28, 2024 at 01:22:52PM +0100, Christian Brauner wrote:
> > On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
> > > Hi,
> > > 
> > > Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
> > > specific mount options), e.g.
> > > 
> > > FSTYP         -- overlay
> > > PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
> > > MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
> > > MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
> > > 
> > > generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
> > >     --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
> > >     +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
> > >     @@ -1,5 +1,5 @@
> > >      QA output created by 294
> > >     -mknod: SCRATCH_MNT/294.test/testnode: File exists
> > >     -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
> > >     -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
> > >     -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
> > >     +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
> > >     +       dmesg(1) may have more information after failed mount system call.
> > 
> > In the new mount api overlayfs has been changed to reject invalid mount
> > option on remount whereas in the old mount api we just igorned them.
> 
> Not only g/294 fails on new mount utils, not sure if all of them are from same issue.
> If you need, I can paste all test failures (only from my side) at here.
> 
> > If this a big problem then we need to change overlayfs to continue
> > ignoring garbage mount options passed to it during remount.
> 
> Do you mean this behavior change is only for overlayfs, doesn't affect other fs?

We tried this with XFS years ago, and reverted back to the old
behaviour of silently ignoring mount options we don't support in
remount. The filesystem code has no idea what mount API
userspace is using for remount - it can't assume that it is ok to
error out on unknown/unsupported options because it uses
the fsreconfigure API internally....

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Ian Kent]

On 29/10/24 07:13, Dave Chinner wrote:
> On Tue, Oct 29, 2024 at 03:28:04AM +0800, Zorro Lang wrote:
>> On Mon, Oct 28, 2024 at 01:22:52PM +0100, Christian Brauner wrote:
>>> On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
>>>> Hi,
>>>>
>>>> Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
>>>> specific mount options), e.g.
>>>>
>>>> FSTYP         -- overlay
>>>> PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
>>>> MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
>>>> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
>>>>
>>>> generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
>>>>      --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
>>>>      +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
>>>>      @@ -1,5 +1,5 @@
>>>>       QA output created by 294
>>>>      -mknod: SCRATCH_MNT/294.test/testnode: File exists
>>>>      -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
>>>>      -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
>>>>      -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
>>>>      +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
>>>>      +       dmesg(1) may have more information after failed mount system call.
>>> In the new mount api overlayfs has been changed to reject invalid mount
>>> option on remount whereas in the old mount api we just igorned them.
>> Not only g/294 fails on new mount utils, not sure if all of them are from same issue.
>> If you need, I can paste all test failures (only from my side) at here.
>>
>>> If this a big problem then we need to change overlayfs to continue
>>> ignoring garbage mount options passed to it during remount.
>> Do you mean this behavior change is only for overlayfs, doesn't affect other fs?
> We tried this with XFS years ago, and reverted back to the old
> behaviour of silently ignoring mount options we don't support in
> remount. The filesystem code has no idea what mount API
> userspace is using for remount - it can't assume that it is ok to
> error out on unknown/unsupported options because it uses
> the fsreconfigure API internally....

I expect that remounting to change options for overlayfs has very 
limited use

cases. Perhaps remounting (the upper layer) read-only is useful ...


The problem here is that xfstests wants to remount the mount read-only 
in this

test which has never actually been done so xfstests reporting a failure 
has no

value!


Ian

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Ian Kent]

On 29/10/24 13:44, Ian Kent wrote:
> On 29/10/24 07:13, Dave Chinner wrote:
>> On Tue, Oct 29, 2024 at 03:28:04AM +0800, Zorro Lang wrote:
>>> On Mon, Oct 28, 2024 at 01:22:52PM +0100, Christian Brauner wrote:
>>>> On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
>>>>> Hi,
>>>>>
>>>>> Recently, I hit lots of fstests cases fail on overlayfs (xfs 
>>>>> underlying, no
>>>>> specific mount options), e.g.
>>>>>
>>>>> FSTYP         -- overlay
>>>>> PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 
>>>>> 25 14:29:18 EDT 2024
>>>>> MKFS_OPTIONS  -- -m 
>>>>> crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 
>>>>> /mnt/fstests/SCRATCH_DIR
>>>>> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 
>>>>> /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
>>>>>
>>>>> generic/294       [failed, exit status 1]- output mismatch (see 
>>>>> /var/lib/xfstests/results//generic/294.out.bad)
>>>>>      --- tests/generic/294.out    2024-10-25 14:38:32.098692473 -0400
>>>>>      +++ /var/lib/xfstests/results//generic/294.out.bad 2024-10-25 
>>>>> 15:02:34.698605062 -0400
>>>>>      @@ -1,5 +1,5 @@
>>>>>       QA output created by 294
>>>>>      -mknod: SCRATCH_MNT/294.test/testnode: File exists
>>>>>      -mkdir: cannot create directory 
>>>>> 'SCRATCH_MNT/294.test/testdir': File exists
>>>>>      -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': 
>>>>> Read-only file system
>>>>>      -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': 
>>>>> File exists
>>>>>      +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system 
>>>>> call failed: overlay: No changes allowed in reconfigure.
>>>>>      +       dmesg(1) may have more information after failed mount 
>>>>> system call.
>>>> In the new mount api overlayfs has been changed to reject invalid 
>>>> mount
>>>> option on remount whereas in the old mount api we just igorned them.
>>> Not only g/294 fails on new mount utils, not sure if all of them are 
>>> from same issue.
>>> If you need, I can paste all test failures (only from my side) at here.
>>>
>>>> If this a big problem then we need to change overlayfs to continue
>>>> ignoring garbage mount options passed to it during remount.
>>> Do you mean this behavior change is only for overlayfs, doesn't 
>>> affect other fs?
>> We tried this with XFS years ago, and reverted back to the old
>> behaviour of silently ignoring mount options we don't support in
>> remount. The filesystem code has no idea what mount API
>> userspace is using for remount - it can't assume that it is ok to
>> error out on unknown/unsupported options because it uses
>> the fsreconfigure API internally....
>
> I expect that remounting to change options for overlayfs has very 
> limited use
>
> cases. Perhaps remounting (the upper layer) read-only is useful ...

Actually, my mistake, it looks like remount read-only might be the only 
case that should

be handled (provided the sb is not already read-only). In this case it 
does a file system

sync and returns the result of that but now fails unconditionally in 
fsconfig().


>
>
> The problem here is that xfstests wants to remount the mount read-only 
> in this
>
> test which has never actually been done so xfstests reporting a 
> failure has no
>
> value!
>
>
> Ian
>

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Ian Kent]

Hey, Amir, Miklos,


On 29/10/24 14:13, Ian Kent wrote:
>
> On 29/10/24 13:44, Ian Kent wrote:
>> On 29/10/24 07:13, Dave Chinner wrote:
>>> On Tue, Oct 29, 2024 at 03:28:04AM +0800, Zorro Lang wrote:
>>>> On Mon, Oct 28, 2024 at 01:22:52PM +0100, Christian Brauner wrote:
>>>>> On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Recently, I hit lots of fstests cases fail on overlayfs (xfs 
>>>>>> underlying, no
>>>>>> specific mount options), e.g.
>>>>>>
>>>>>> FSTYP         -- overlay
>>>>>> PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri 
>>>>>> Oct 25 14:29:18 EDT 2024
>>>>>> MKFS_OPTIONS  -- -m 
>>>>>> crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 
>>>>>> /mnt/fstests/SCRATCH_DIR
>>>>>> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 
>>>>>> /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
>>>>>>
>>>>>> generic/294       [failed, exit status 1]- output mismatch (see 
>>>>>> /var/lib/xfstests/results//generic/294.out.bad)
>>>>>>      --- tests/generic/294.out    2024-10-25 14:38:32.098692473 
>>>>>> -0400
>>>>>>      +++ /var/lib/xfstests/results//generic/294.out.bad 
>>>>>> 2024-10-25 15:02:34.698605062 -0400
>>>>>>      @@ -1,5 +1,5 @@
>>>>>>       QA output created by 294
>>>>>>      -mknod: SCRATCH_MNT/294.test/testnode: File exists
>>>>>>      -mkdir: cannot create directory 
>>>>>> 'SCRATCH_MNT/294.test/testdir': File exists
>>>>>>      -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': 
>>>>>> Read-only file system
>>>>>>      -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': 
>>>>>> File exists
>>>>>>      +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system 
>>>>>> call failed: overlay: No changes allowed in reconfigure.
>>>>>>      +       dmesg(1) may have more information after failed 
>>>>>> mount system call.
>>>>> In the new mount api overlayfs has been changed to reject invalid 
>>>>> mount
>>>>> option on remount whereas in the old mount api we just igorned them.
>>>> Not only g/294 fails on new mount utils, not sure if all of them 
>>>> are from same issue.
>>>> If you need, I can paste all test failures (only from my side) at 
>>>> here.
>>>>
>>>>> If this a big problem then we need to change overlayfs to continue
>>>>> ignoring garbage mount options passed to it during remount.
>>>> Do you mean this behavior change is only for overlayfs, doesn't 
>>>> affect other fs?
>>> We tried this with XFS years ago, and reverted back to the old
>>> behaviour of silently ignoring mount options we don't support in
>>> remount. The filesystem code has no idea what mount API
>>> userspace is using for remount - it can't assume that it is ok to
>>> error out on unknown/unsupported options because it uses
>>> the fsreconfigure API internally....
>>
>> I expect that remounting to change options for overlayfs has very 
>> limited use
>>
>> cases. Perhaps remounting (the upper layer) read-only is useful ...
>
> Actually, my mistake, it looks like remount read-only might be the 
> only case that should
>
> be handled (provided the sb is not already read-only). In this case it 
> does a file system
>
> sync and returns the result of that but now fails unconditionally in 
> fsconfig().

What is the right thing to do here for overlayfs?

Does it even make sense to remount overlayfs to change mount options?


The file system sync on read-only-remount seems worth while so assuming 
it is kept

what should be returned from other remount requests, fail or silently 
fail as before?


Ian

>
>
>>
>>
>> The problem here is that xfstests wants to remount the mount 
>> read-only in this
>>
>> test which has never actually been done so xfstests reporting a 
>> failure has no
>>
>> value!
>>
>>
>> Ian
>>

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Amir Goldstein]

On Mon, Oct 28, 2024 at 8:28 PM Zorro Lang <zlang@redhat.com> wrote:
>
> On Mon, Oct 28, 2024 at 01:22:52PM +0100, Christian Brauner wrote:
> > On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
> > > Hi,
> > >
> > > Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
> > > specific mount options), e.g.
> > >
> > > FSTYP         -- overlay
> > > PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
> > > MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
> > > MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
> > >
> > > generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
> > >     --- tests/generic/294.out       2024-10-25 14:38:32.098692473 -0400
> > >     +++ /var/lib/xfstests/results//generic/294.out.bad      2024-10-25 15:02:34.698605062 -0400
> > >     @@ -1,5 +1,5 @@
> > >      QA output created by 294
> > >     -mknod: SCRATCH_MNT/294.test/testnode: File exists
> > >     -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
> > >     -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
> > >     -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
> > >     +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
> > >     +       dmesg(1) may have more information after failed mount system call.
> >
> > In the new mount api overlayfs has been changed to reject invalid mount
> > option on remount whereas in the old mount api we just igorned them.
>
> Not only g/294 fails on new mount utils, not sure if all of them are from same issue.
> If you need, I can paste all test failures (only from my side) at here.

Please share them.

>
> > If this a big problem then we need to change overlayfs to continue
> > ignoring garbage mount options passed to it during remount.
>
> Do you mean this behavior change is only for overlayfs, doesn't affect other fs?

That is correct.

>
> If it's not necessary, I think we'd better to not change the behaviors which we've
> used so many years. But if you all agree with this change, then we need to update
> related regression test cases and more scripts maybe.

Can you check if this workaround (done for ostree) solves the problem:

--- a/common/overlay
+++ b/common/overlay
@@ -127,7 +127,7 @@ _overlay_base_scratch_mount()
 _overlay_scratch_mount()
 {
        if echo "$*" | grep -q remount; then
-               $MOUNT_PROG $SCRATCH_MNT $*
+               $MOUNT_PROG --options-source=disable $SCRATCH_MNT $*
                return
        fi

I am not sure about the proper fix for general use cases,
but if this is enough to fix fstests, I think we should do this anyway.

Thanks,
Amir.

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Colin Walters]

On Mon, Oct 28, 2024, at 8:22 AM, Christian Brauner wrote:
> On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
>> Hi,
>> 
>> Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
>> specific mount options), e.g.
>> 
>> FSTYP         -- overlay
>> PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
>> MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
>> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
>> 
>> generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
>>     --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
>>     +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
>>     @@ -1,5 +1,5 @@
>>      QA output created by 294
>>     -mknod: SCRATCH_MNT/294.test/testnode: File exists
>>     -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
>>     -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
>>     -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
>>     +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
>>     +       dmesg(1) may have more information after failed mount system call.
>
> In the new mount api overlayfs has been changed to reject invalid mount
> option on remount whereas in the old mount api we just igorned them.
> If this a big problem then we need to change overlayfs to continue
> ignoring garbage mount options passed to it during remount.

https://github.com/ostreedev/ostree/commit/bc62fd519631be6591c5c62302f83b45b9d56328 is related...I probably should have reported it at the time.

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Eric Sandeen]

On 10/28/24 7:22 AM, Christian Brauner wrote:
> On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
>> Hi,
>>
>> Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
>> specific mount options), e.g.
>>
>> FSTYP         -- overlay
>> PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
>> MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
>> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
>>
>> generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
>>     --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
>>     +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
>>     @@ -1,5 +1,5 @@
>>      QA output created by 294
>>     -mknod: SCRATCH_MNT/294.test/testnode: File exists
>>     -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
>>     -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
>>     -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
>>     +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
>>     +       dmesg(1) may have more information after failed mount system call.
> 
> In the new mount api overlayfs has been changed to reject invalid mount
> option on remount whereas in the old mount api we just igorned them.
> If this a big problem then we need to change overlayfs to continue
> ignoring garbage mount options passed to it during remount.
> 

It fails on /any/ overlayfs-specific options during reconfigure, invalid or
not, right?

        if (fc->purpose == FS_CONTEXT_FOR_RECONFIGURE) {
                /*
                 * On remount overlayfs has always ignored all mount
                 * options no matter if malformed or not so for
                 * backwards compatibility we do the same here.
                 */
                if (fc->oldapi)
                        return 0;
                
                /*
                 * Give us the freedom to allow changing mount options
                 * with the new mount api in the future. So instead of
                 * silently ignoring everything we report a proper
                 * error. This is only visible for users of the new
                 * mount api.
                 */
                return invalfc(fc, "No changes allowed in reconfigure");
        }

        opt = fs_parse(fc, ovl_parameter_spec, param, &result);
        if (opt < 0)
                return opt; 

And because today mount(8) will re-specify everything it finds in
/proc/mounts during remount, presumably that's why all these tests are
failing - even a simple remount,ro will fail:

# mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged
# strace -e fsconfig mount -o remount,ro merged
fsconfig(4, FSCONFIG_SET_FLAG, "seclabel", NULL, 0) = 0
fsconfig(4, FSCONFIG_SET_STRING, "lowerdir", "lower", 0) = -1 EINVAL (Invalid argument)
...

Surely mount -o remount,ro should continue to work for overlayfs when the new
API is used.

Maybe there's a third way: accept remount options as long as they match
current options, but fail if they try to modify anything? Not sure how tricky
that would be.

(side note: it's a bit worrisome that there is probably no consistency at
all across filesystems, here.)

-Eric

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Christian Brauner]

On Wed, Oct 30, 2024 at 10:48:15PM -0500, Eric Sandeen wrote:
> On 10/28/24 7:22 AM, Christian Brauner wrote:
> > On Sun, Oct 27, 2024 at 02:07:41AM +0800, Zorro Lang wrote:
> >> Hi,
> >>
> >> Recently, I hit lots of fstests cases fail on overlayfs (xfs underlying, no
> >> specific mount options), e.g.
> >>
> >> FSTYP         -- overlay
> >> PLATFORM      -- Linux/s390x s390x-xxxx 6.12.0-rc4+ #1 SMP Fri Oct 25 14:29:18 EDT 2024
> >> MKFS_OPTIONS  -- -m crc=1,finobt=1,rmapbt=0,reflink=1,inobtcount=1,bigtime=1 /mnt/fstests/SCRATCH_DIR
> >> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /mnt/fstests/SCRATCH_DIR /mnt/fstests/SCRATCH_DIR/ovl-mnt
> >>
> >> generic/294       [failed, exit status 1]- output mismatch (see /var/lib/xfstests/results//generic/294.out.bad)
> >>     --- tests/generic/294.out	2024-10-25 14:38:32.098692473 -0400
> >>     +++ /var/lib/xfstests/results//generic/294.out.bad	2024-10-25 15:02:34.698605062 -0400
> >>     @@ -1,5 +1,5 @@
> >>      QA output created by 294
> >>     -mknod: SCRATCH_MNT/294.test/testnode: File exists
> >>     -mkdir: cannot create directory 'SCRATCH_MNT/294.test/testdir': File exists
> >>     -touch: cannot touch 'SCRATCH_MNT/294.test/testtarget': Read-only file system
> >>     -ln: creating symbolic link 'SCRATCH_MNT/294.test/testlink': File exists
> >>     +mount: /mnt/fstests/SCRATCH_DIR/ovl-mnt: fsconfig system call failed: overlay: No changes allowed in reconfigure.
> >>     +       dmesg(1) may have more information after failed mount system call.
> > 
> > In the new mount api overlayfs has been changed to reject invalid mount
> > option on remount whereas in the old mount api we just igorned them.
> > If this a big problem then we need to change overlayfs to continue
> > ignoring garbage mount options passed to it during remount.
> > 
> 
> It fails on /any/ overlayfs-specific options during reconfigure, invalid or
> not, right?

Yes.

> 
>         if (fc->purpose == FS_CONTEXT_FOR_RECONFIGURE) {
>                 /*
>                  * On remount overlayfs has always ignored all mount
>                  * options no matter if malformed or not so for
>                  * backwards compatibility we do the same here.
>                  */
>                 if (fc->oldapi)
>                         return 0;
>                 
>                 /*
>                  * Give us the freedom to allow changing mount options
>                  * with the new mount api in the future. So instead of
>                  * silently ignoring everything we report a proper
>                  * error. This is only visible for users of the new
>                  * mount api.
>                  */
>                 return invalfc(fc, "No changes allowed in reconfigure");
>         }
> 
>         opt = fs_parse(fc, ovl_parameter_spec, param, &result);
>         if (opt < 0)
>                 return opt; 
> 
> And because today mount(8) will re-specify everything it finds in
> /proc/mounts during remount, presumably that's why all these tests are
> failing - even a simple remount,ro will fail:

The easiest way is to just fallback to the old behavior and continue to
ignore unknown mount options. IOW, ignore my change.

> 
> # mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged
> # strace -e fsconfig mount -o remount,ro merged
> fsconfig(4, FSCONFIG_SET_FLAG, "seclabel", NULL, 0) = 0
> fsconfig(4, FSCONFIG_SET_STRING, "lowerdir", "lower", 0) = -1 EINVAL (Invalid argument)
> ...
> 
> Surely mount -o remount,ro should continue to work for overlayfs when the new
> API is used.
> 
> Maybe there's a third way: accept remount options as long as they match
> current options, but fail if they try to modify anything? Not sure how tricky
> that would be.
> 
> (side note: it's a bit worrisome that there is probably no consistency at
> all across filesystems, here.)

One option would be to add a fsconfig() flag that enforces strict
remount behavior if the filesystem supports it. So it's would become an
opt-in thing.

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Miklos Szeredi]

On Thu, 31 Oct 2024 at 11:30, Christian Brauner <brauner@kernel.org> wrote:

> One option would be to add a fsconfig() flag that enforces strict
> remount behavior if the filesystem supports it. So it's would become an
> opt-in thing.

From what mount(8) does it seems the expected behavior of filesystems
is to reset the configuration state before parsing options in
reconfigure.   But it's not what mount(8) expects on the command line.
I.e. "mount -oremount,ro" will result in all previous options being
added to the list of options (except rw).  There's a big disconnect
between the two interfaces.

I guess your suggestion is to allow filesystem to process only the
options that are changed, right?

I think that makes perfect sense and would allow to slowly get rid of
the above disconnect.

Thanks,
Miklos

Re: lots of fstests cases fail on overlay with util-linux 2.40.2 (new mount APIs)

[Christian Brauner]

On Mon, Nov 04, 2024 at 05:41:18PM +0100, Miklos Szeredi wrote:
> On Thu, 31 Oct 2024 at 11:30, Christian Brauner <brauner@kernel.org> wrote:
> 
> > One option would be to add a fsconfig() flag that enforces strict
> > remount behavior if the filesystem supports it. So it's would become an
> > opt-in thing.
> 
> From what mount(8) does it seems the expected behavior of filesystems
> is to reset the configuration state before parsing options in
> reconfigure.   But it's not what mount(8) expects on the command line.

I'm not sure that's the case but I might misremember what mount(8) does.

As best as I remember it the mount(2) system call has different behavior
for VFS generic options, and filesystem specific mount options.

The difficulty is once again that both are mixed together in the
mount(2) system call which hides the behavioral differences. I'll try to
summarize what I remember below. 

> I.e. "mount -oremount,ro" will result in all previous options being
> added to the list of options (except rw).  There's a big disconnect
> between the two interfaces.

So for VFS generic mount options the behavior of mount(2) is that if one
has a filesystem mounted with nodev,nosuid,ro such as:

	mount(NULL, "/mnt", "tmpfs", 0, "nodev,nosuid");

and one now remounts (proper remount, not MS_BIND | MS_REMOUNT) as "ro":

	mount(NULL, "/mnt", "", MS_REMOUNT, "ro");

then mount(2) will not display additive behavior for the generic VFS
mount options. Instead, it will treat it as a "reset". So "ro" gets
added and "nosuid" and "nodev" get stripped.

That non-additive behavior has actually caused quite some security
issues. So mount(8) works around this, by translating a:

	mount -o remount,ro /mnt

internally into:

	mount(NULL, "/mnt", "", MS_REMOUNT, "ro,nodev,nosuid");

But afair, this reset behavior only applies to generic VFS options (ro,
nosuid, nodev, noexec) but not filesystem specific options during
remount.

In contrast, the problem with filesystem specific mount options during
remount, is that quite a few filesystems ignore unknown mount options or
mount options that cannot be changed on remount.

This is effecitvely what overlayfs does when the remount request comes
from the old mount(2) api. It will just consume anything and ignore even
nonsensical/nonexistent mount options.

This causes other problems where users that want to really ensure that a
mount property gets changed during remount cannot be sure because
anything will succeed.

So initially I had thought we could change that behavior by
differentiating between a request coming from the old or new mount api.
If the request comes from the new mount api we would return errors for
unknown- or mount options that cannot be changed on remount.

However, this seems to break some tools such as mount(8) because it
reassembles all mount options during remount to emulate additive
behavior but then fails because a remount request from the new mount api
rejects mount options that can't be changed during a remount.

But there's definitely use-cases where userspace wants to know whether a
mount option was actually change{d,able} during a remount.

To accomodate the old and new behavior my idea had been to let the
filesystem choose whether to ignore unknown mount options or whether it
will error out if unknown mount options are specified or when mount
options are specified that cannot be changed on remount.

A filesystem that allows for strict mount option parsing could raise a
flag in fs_flags and then a new uapi extension for fsopen() gets added,
e.g., Something like:

        fsopen("overlayfs", FSOPEN_REMOUNT_STRICT);

(Fwiw, mount_setattr() is additive/subtractive, i.e., it does the
right thing and only clears or sets the options that are explicitly
specified, leaving other options alone.)

> 
> I guess your suggestion is to allow filesystem to process only the
> options that are changed, right?
> 
> I think that makes perfect sense and would allow to slowly get rid of
> the above disconnect.