util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Thomas Deutschmann]

Hi,

I didn't find a kernel requirement in latest util-linux 2.39 release 
notes and because a fallback is mentioned I gave it a try on a system 
running latest LTS kernel 5.10.180.

After updating to util=linux 2.39 system immediately failed when trying 
to use mount command.

I.e.

   mount --make-slave /proc

failed with:

> mount: /proc: mount(2) system call failed: Function not implemented.
>        dmesg(1) may have more information after failed mount system call.

Nothing in dmesg about it.


strace shows

> mount_setattr(3, "", AT_EMPTY_PATH, {attr_set=0, attr_clr=0, propagation=MS_SLAVE, userns_fd=0}, 32) = -1 ENOSYS (Function not implemented)


Configure output shows

> checking for syscall mount_setattr... SYS_mount_setattr

But this is coming from Linux headers (I have most recent Linux headers 
from kernel 6.3 installed).

Not sure if this is the problem but using most recent Linux headers with 
older kernel shouldn't be a problem in general according to 
https://sourceware.org/glibc/wiki/FAQ#What_version_of_the_Linux_kernel_headers_should_be_used.3F

Booting a newer kernel (I tried 5.15.112) helped (I guess because the 
system call was added in 5.12).

Thanks!


-- 
Regards,
Thomas

Re: util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Thomas Weißschuh]

Hi Thomas,

On 2023-05-20 02:54:23+0200, Thomas Deutschmann wrote:
> Hi,
> 
> I didn't find a kernel requirement in latest util-linux 2.39 release notes
> and because a fallback is mentioned I gave it a try on a system running
> latest LTS kernel 5.10.180.
> 
> After updating to util=linux 2.39 system immediately failed when trying to
> use mount command.
> 
> I.e.
> 
>   mount --make-slave /proc
> 
> failed with:
> 
> > mount: /proc: mount(2) system call failed: Function not implemented.
> >        dmesg(1) may have more information after failed mount system call.
> 
> Nothing in dmesg about it.
> 
> 
> strace shows
> 
> > mount_setattr(3, "", AT_EMPTY_PATH, {attr_set=0, attr_clr=0, propagation=MS_SLAVE, userns_fd=0}, 32) = -1 ENOSYS (Function not implemented)

Thanks for the report!

There is also a ticket about this here:
https://github.com/util-linux/util-linux/issues/2247 .

> Configure output shows
> 
> > checking for syscall mount_setattr... SYS_mount_setattr
> 
> But this is coming from Linux headers (I have most recent Linux headers from
> kernel 6.3 installed).
> 
> Not sure if this is the problem but using most recent Linux headers with
> older kernel shouldn't be a problem in general according to https://sourceware.org/glibc/wiki/FAQ#What_version_of_the_Linux_kernel_headers_should_be_used.3F

Please note that this is a document about glibc, not util-linux.
Nevertheless the same concept also applies to util-linux.

> Booting a newer kernel (I tried 5.15.112) helped (I guess because the system
> call was added in 5.12).

Could you test the fix proposed in 
https://github.com/util-linux/util-linux/pull/2248 ?

(For testing on a real system you only need the changes to
libmount/src/hook_mount.c, ignore the testing stuff)

Thomas

Re: util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Thomas Deutschmann]

Hi Thomas :)

On 2023-05-20 06:51, Thomas Weißschuh wrote:
> Could you test the fix proposed in
> https://github.com/util-linux/util-linux/pull/2248 ?
> 
> (For testing on a real system you only need the changes to
> libmount/src/hook_mount.c, ignore the testing stuff)

This is addressing the reported

   mount --make-slave /proc

issue. I.e. in strace you will see your added call to 
mount_setattr_is_supported() which will fail because it is not supported 
but afterwards it will try old syscall:

> getuid()                                = 0
> geteuid()                               = 0
> open_tree(AT_FDCWD, "/", OPEN_TREE_CLOEXEC) = 3
> mount_setattr(-1, NULL, 0, NULL, 0)     = -1 ENOSYS (Function not implemented)
> getuid()                                = 0
> geteuid()                               = 0
> getgid()                                = 0
> getegid()                               = 0
> prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
> newfstatat(AT_FDCWD, "/run/mount/utab", 0x7ffdfabad380, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
> mkdir("/run/mount", 0755)               = -1 EEXIST (File exists)
> newfstatat(AT_FDCWD, "/run/mount/utab", 0x7ffdfabad270, 0) = -1 ENOENT (No such file or directory)
> newfstatat(AT_FDCWD, "/run/mount", {st_mode=S_IFDIR|0755, st_size=40, ...}, 0) = 0
> geteuid()                               = 0
> getegid()                               = 0
> getuid()                                = 0
> getgid()                                = 0
> access("/run/mount", R_OK|W_OK)         = 0
> mount("none", "/", NULL, MS_SLAVE, NULL) = 0
> close(3)                                = 0

However, I noticed additional problems. Remounts for example are still 
not working but no error is shown:

> ~ # findmnt /
> TARGET  SOURCE    FSTYPE OPTIONS
> /       /dev/dm-0 ext4   ro,relatime,stripe=32650
> ~ # mount -o remount,rw /
> ~ # findmnt /
> TARGET   SOURCE    FSTYPE OPTIONS
> /        /dev/dm-0 ext4   ro,relatime,stripe=32650

I don't see a fallback in strace:

 > [...]
> open_tree(AT_FDCWD, "/", OPEN_TREE_CLOEXEC) = 3
> mount_setattr(-1, NULL, 0, NULL, 0)     = -1 ENOSYS (Function not implemented)
> getuid()                                = 0
> geteuid()                               = 0
> getgid()                                = 0
> getegid()                               = 0
> prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
> newfstatat(AT_FDCWD, "/run/mount/utab", 0x7ffe4fe7c590, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
> mkdir("/run/mount", 0755)               = -1 EEXIST (File exists)
> newfstatat(AT_FDCWD, "/run/mount/utab", 0x7ffe4fe7c480, 0) = -1 ENOENT (No such file or directory)
> newfstatat(AT_FDCWD, "/run/mount", {st_mode=S_IFDIR|0755, st_size=40, ...}, 0) = 0
> geteuid()                               = 0
> getegid()                               = 0
> getuid()                                = 0
> getgid()                                = 0
> access("/run/mount", R_OK|W_OK)         = 0
> fspick(3, "", FSPICK_NO_AUTOMOUNT|FSPICK_EMPTY_PATH) = 4
> fsconfig(4, FSCONFIG_SET_FLAG, "rw", NULL, 0) = 0
> fsconfig(4, FSCONFIG_CMD_RECONFIGURE, NULL, NULL, 0) = 0
> close(4)                                = 0
> close(3)                                = 0
> dup(1)                                  = 3
> close(3)                                = 0
> dup(2)                                  = 3
> close(3)                                = 0
> exit_group(0)                           = ?
> +++ exited with 0 +++



-- 
Regards,
Thomas

Re: util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Thomas Weißschuh]

On 2023-05-20 17:23:05+0200, Thomas Deutschmann wrote:
> On 2023-05-20 06:51, Thomas Weißschuh wrote:
> > Could you test the fix proposed in
> > https://github.com/util-linux/util-linux/pull/2248 ?
> > 
> > (For testing on a real system you only need the changes to
> > libmount/src/hook_mount.c, ignore the testing stuff)
> 
> This is addressing the reported
> 
>   mount --make-slave /proc

Thanks for confirming!

<snip>

> However, I noticed additional problems. Remounts for example are still not
> working but no error is shown:
> 
> > ~ # findmnt /
> > TARGET  SOURCE    FSTYPE OPTIONS
> > /       /dev/dm-0 ext4   ro,relatime,stripe=32650
> > ~ # mount -o remount,rw /
> > ~ # findmnt /
> > TARGET   SOURCE    FSTYPE OPTIONS
> > /        /dev/dm-0 ext4   ro,relatime,stripe=32650
> 
> I don't see a fallback in strace:
> 
> > [...]
> > open_tree(AT_FDCWD, "/", OPEN_TREE_CLOEXEC) = 3
> > mount_setattr(-1, NULL, 0, NULL, 0)     = -1 ENOSYS (Function not implemented)
> > getuid()                                = 0
> > geteuid()                               = 0
> > getgid()                                = 0
> > getegid()                               = 0
> > prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
> > newfstatat(AT_FDCWD, "/run/mount/utab", 0x7ffe4fe7c590, AT_SYMLINK_NOFOLLOW) = -1 ENOENT (No such file or directory)
> > mkdir("/run/mount", 0755)               = -1 EEXIST (File exists)
> > newfstatat(AT_FDCWD, "/run/mount/utab", 0x7ffe4fe7c480, 0) = -1 ENOENT (No such file or directory)
> > newfstatat(AT_FDCWD, "/run/mount", {st_mode=S_IFDIR|0755, st_size=40, ...}, 0) = 0
> > geteuid()                               = 0
> > getegid()                               = 0
> > getuid()                                = 0
> > getgid()                                = 0
> > access("/run/mount", R_OK|W_OK)         = 0
> > fspick(3, "", FSPICK_NO_AUTOMOUNT|FSPICK_EMPTY_PATH) = 4
> > fsconfig(4, FSCONFIG_SET_FLAG, "rw", NULL, 0) = 0
> > fsconfig(4, FSCONFIG_CMD_RECONFIGURE, NULL, NULL, 0) = 0
> > close(4)                                = 0
> > close(3)                                = 0
> > dup(1)                                  = 3
> > close(3)                                = 0
> > dup(2)                                  = 3
> > close(3)                                = 0
> > exit_group(0)                           = ?
> > +++ exited with 0 +++

I'm not sure that the fallback should be necessary here.
The calls to fsconfig() seem to succeed.

I am having trouble reproducing the issue.

Could you
* provide more details about the used kernel?
* provide reproduction steps that are simple to reproduce, for example
  using tmpfs mounts.

FYI there was a report on GitHub that confirms more or less this exact
usecase to be working with my patch. [0]

[0] https://github.com/util-linux/util-linux/issues/2247#issuecomment-1555967726

Thomas

Re: util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Thomas Deutschmann]

Hi

On 2023-05-21 01:39, Thomas Weißschuh wrote:
> I am having trouble reproducing the issue.
> 
> Could you
> * provide more details about the used kernel?
> * provide reproduction steps that are simple to reproduce, for example
>    using tmpfs mounts.

Used kernel.config: https://0x0.st/HqKc.txt

I cannot reproduce using tmpfs, i.e.

   # mkdir /mnt/test
   # mount -t tmpfs none /mnt/test
   # mount -l | grep test
   none on /mnt/test type tmpfs (rw,relatime)
   # touch /mnt/test/foo.txt
   # mount -o remount,ro /mnt/test
   # date >> /mnt/test/foo.txt
   -bash: /mnt/test/foo.txt: Read-only file system
   # mount -o remount,rw /mnt/test
   # date >> /mnt/test/foo.txt
   # cat /mnt/test/foo.txt
   Sun May 21 11:04:42 PM CEST 2023

works.

But on boot, when the init system remounts root filesystem (using 
command "mount -n -o remount,rw /"), the command seems to be successful 
(mount command exits with zero exit code) but the root filesystem is 
still read-only and all following services expecting writable rootfs 
will fail.

I have

   UUID=e290359f-8335-4fba-9fae-077b3505e0b2 / ext4 defaults 0 1

in /etc/fstab. I saw https://github.com/util-linux/util-linux/pull/2250 
but this doesn't help (and changing "defaults" to "relatime" didn't help 
either).

I added strace to the init script trying to remount root read/write:

Booting linux-5.10.180 using util-linux-2.39 with PR 2248 and PR 2250 
applied: https://0x0.st/HqKL.180.png

=> mount isn't doing anything; root will stay read-only.

When I use same util-linux but boot linux-5.15.112 instead: 
https://0x0.st/HqK9.112.png

=> mount is doing something; root will become read/write.


-- 
Regards,
Thomas

Re: util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Thomas Weißschuh]

On 2023-05-22 00:13:14+0200, Thomas Deutschmann wrote:
> Hi
> 
> On 2023-05-21 01:39, Thomas Weißschuh wrote:
> > I am having trouble reproducing the issue.
> > 
> > Could you
> > * provide more details about the used kernel?
> > * provide reproduction steps that are simple to reproduce, for example
> >    using tmpfs mounts.
> 
> Used kernel.config: https://0x0.st/HqKc.txt
> 
> I cannot reproduce using tmpfs, i.e.
> 
>   # mkdir /mnt/test
>   # mount -t tmpfs none /mnt/test
>   # mount -l | grep test
>   none on /mnt/test type tmpfs (rw,relatime)
>   # touch /mnt/test/foo.txt
>   # mount -o remount,ro /mnt/test
>   # date >> /mnt/test/foo.txt
>   -bash: /mnt/test/foo.txt: Read-only file system
>   # mount -o remount,rw /mnt/test
>   # date >> /mnt/test/foo.txt
>   # cat /mnt/test/foo.txt
>   Sun May 21 11:04:42 PM CEST 2023
> 
> works.
> 
> But on boot, when the init system remounts root filesystem (using command
> "mount -n -o remount,rw /"), the command seems to be successful (mount
> command exits with zero exit code) but the root filesystem is still
> read-only and all following services expecting writable rootfs will fail.
> 
> I have
> 
>   UUID=e290359f-8335-4fba-9fae-077b3505e0b2 / ext4 defaults 0 1
> 
> in /etc/fstab. I saw https://github.com/util-linux/util-linux/pull/2250 but
> this doesn't help (and changing "defaults" to "relatime" didn't help
> either).
> 
> I added strace to the init script trying to remount root read/write:
> 
> Booting linux-5.10.180 using util-linux-2.39 with PR 2248 and PR 2250
> applied: https://0x0.st/HqKL.180.png
> 
> => mount isn't doing anything; root will stay read-only.
> 
> When I use same util-linux but boot linux-5.15.112 instead:
> https://0x0.st/HqK9.112.png
> 
> => mount is doing something; root will become read/write.

I was finally able to reproduce the issue.

Could you test https://github.com/util-linux/util-linux/pull/2248 again?

Thomas

Re: util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Thomas Deutschmann]

Hi,

On 2023-05-22 09:43, Thomas Weißschuh wrote:
> I was finally able to reproduce the issue.
> 
> Could you test https://github.com/util-linux/util-linux/pull/2248 again?

This fixes the issue for me, thank you!

Just one comment regarding the design of the fix:

Due to the introduction of mount_setattr_is_supported(), util-linux will 
now "waste" one syscall on every mount -- even on a modern system -- right?

A more "natural" approach like calling new API first but when this will 
fail with ENOSYS then fallback to old API (so we would only spend an 
additional call on old systems which is fine because we assume that 
their number will decrease over time) cannot be implemented due to the 
way how the hook system works, right?

But maybe this "wasted" call isn't much to think about. I was just 
wondering when I saw a call like

   mount_setattr(-1, NULL, 0, NULL, 0)

even on modern systems.


-- 
Regards,
Thomas

Re: util-linux 2.39 doesn't work with <linux-5.12: mount(2) system call failed: Function not implemented

[Karel Zak]

On Mon, May 22, 2023 at 04:09:28PM +0200, Thomas Deutschmann wrote:
> Hi,
> 
> On 2023-05-22 09:43, Thomas Weißschuh wrote:
> > I was finally able to reproduce the issue.
> > 
> > Could you test https://github.com/util-linux/util-linux/pull/2248 again?
> 
> This fixes the issue for me, thank you!
> 
> Just one comment regarding the design of the fix:
> 
> Due to the introduction of mount_setattr_is_supported(), util-linux will now
> "waste" one syscall on every mount -- even on a modern system -- right?
> 
> A more "natural" approach like calling new API first but when this will fail
> with ENOSYS then fallback to old API (so we would only spend an additional
> call on old systems which is fine because we assume that their number will
> decrease over time) cannot be implemented due to the way how the hook system
> works, right?
> 
> But maybe this "wasted" call isn't much to think about. I was just wondering
> when I saw a call like
> 
>   mount_setattr(-1, NULL, 0, NULL, 0)
> 
> even on modern systems.


The problem is that the new API is not 1:1 with the old API. The new
API is more complex and requires open mount node/fs, preparing mount
options differently, more syscalls, etc. IMHO, it will be a
maintenance nightmare to use "if (ENOSYS) old_way()" for all use
cases.


Maybe it's not optimal for performance, but I have split libmount to
separate modules and legacy mount and new mount API are completely
separated. There is always a "prepare" stage when the module analyzes
the mount options, initializes resources (open_tree(), etc.) and if
the new API is unsuccessful (ENOSYS) in this sage, then it disables
the whole module, and the legacy module can detect it and start to
play its role.

The extra syscall (for remount in this case) is not a big problem (I
think).

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com