UML mount failure with Linux 6.11

UML mount failure with Linux 6.11

[Ritesh Raj Sarraf]

[-- Attachment #1: Type: text/plain, Size: 1265 bytes --]

Hello,


Recently I updated the UML package to Linux 6.11 in Debian. Upon it, it
reports the below regression wherein UML cannot mount file systems of
type hostfs

We have it reported on the Debian bug tracker as well.

* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1086194


The mount failures report as:

```
Oct 30 05:57:41 uml systemd[1]: Mounting usr-lib-modules.mount -
/usr/lib/modules...
Oct 30 05:57:42 uml systemd[1]: usr-lib-modules.mount: Mount process exited, code=exited, status=32/n/a
Oct 30 05:57:42 uml mount[814]: mount: /usr/lib/modules: fsconfig system call failed: hostfs: Unknown parameter '/usr/lib/uml/modules'.
Oct 30 05:57:42 uml mount[814]:        dmesg(1) may have more information after failed mount system call.
Oct 30 05:57:42 uml systemd[1]: usr-lib-modules.mount: Failed with result 'exit-code'.
Oct 30 05:57:42 uml systemd[1]: Failed to mount usr-lib-modules.mount - /usr/lib/modules.

```

As I understand it, the issue was first seen in July, around 6.11 rc1.

*
https://lore.kernel.org/all/CAHk-=wjH5uKPB6xrWoB8WkBMuLEJO2UsidKE1wV8XSXjAUFO8Q@mail.gmail.com/


Thanks,
Ritesh

-- 
Given the large number of mailing lists I follow, I request you to CC
me in replies for quicker response

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: UML mount failure with Linux 6.11

[Benjamin Berg]

Hi,

Newer kernels have become more picky about that with the new mount API.
This is relevant, see the discussion about "Unknown options":
  https://lwn.net/Articles/979166/

We only use hostfs for the root file system and in that case it works
well if you pass the path using "hostfs=/path" on the kernel command
line. Doing that avoids issues when remounting the file system later
on.

I suppose that currently it does not work to mount hostfs later on. No
idea what the right fix is. Maybe the host directory should be an
explicit option like "hostpath=..." or so to make it compatible with
the new mount APIs.

Benjamin

On Wed, 2024-10-30 at 13:43 +0530, Ritesh Raj Sarraf wrote:
> Hello,
> 
> 
> Recently I updated the UML package to Linux 6.11 in Debian. Upon it,
> it
> reports the below regression wherein UML cannot mount file systems of
> type hostfs
> 
> We have it reported on the Debian bug tracker as well.
> 
> * https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1086194
> 
> 
> The mount failures report as:
> 
> ```
> Oct 30 05:57:41 uml systemd[1]: Mounting usr-lib-modules.mount -
> /usr/lib/modules...
> Oct 30 05:57:42 uml systemd[1]: usr-lib-modules.mount: Mount process
> exited, code=exited, status=32/n/a
> Oct 30 05:57:42 uml mount[814]: mount: /usr/lib/modules: fsconfig
> system call failed: hostfs: Unknown parameter '/usr/lib/uml/modules'.
> Oct 30 05:57:42 uml mount[814]:        dmesg(1) may have more
> information after failed mount system call.
> Oct 30 05:57:42 uml systemd[1]: usr-lib-modules.mount: Failed with
> result 'exit-code'.
> Oct 30 05:57:42 uml systemd[1]: Failed to mount usr-lib-modules.mount
> - /usr/lib/modules.
> 
> ```
> 
> As I understand it, the issue was first seen in July, around 6.11
> rc1.
> 
> *
> https://lore.kernel.org/all/CAHk-=wjH5uKPB6xrWoB8WkBMuLEJO2UsidKE1wV8XSXjAUFO8Q@mail.gmail.com/
> 
> 
> Thanks,
> Ritesh
> 

Re: UML mount failure with Linux 6.11

[Ritesh Raj Sarraf]

[-- Attachment #1: Type: text/plain, Size: 1443 bytes --]

Hello Benjamin,

On Thu, 2024-10-31 at 11:07 +0100, Benjamin Berg wrote:
> Hi,
> 
> Newer kernels have become more picky about that with the new mount
> API.
> This is relevant, see the discussion about "Unknown options":
>   https://lwn.net/Articles/979166/
> 
> We only use hostfs for the root file system and in that case it works
> well if you pass the path using "hostfs=/path" on the kernel command
> line. Doing that avoids issues when remounting the file system later
> on.
> 

As upstream developers for UML, what would you conclude it as ?

We've recommended using hostfs for the UML kernel modules as well. What
would be the alternate approach to ensuring a proper boot for a modular
UML kernel ?


> I suppose that currently it does not work to mount hostfs later on.
> No
> idea what the right fix is. Maybe the host directory should be an
> explicit option like "hostpath=..." or so to make it compatible with
> the new mount APIs.

The ability to mount any hostfs mount point was/is a feature provided
by UML. We've used it and integrated with many tools like debos,
fakemachine etc; the Debian bug report has the details.

There'll be more reports following once UML 6.11 hits Debian Testing.

I hadn't expected a working feature to break with a newer Linux
release. :-(

Thanks,
Ritesh

-- 
Ritesh Raj Sarraf | http://people.debian.org/~rrs
Debian - The Universal Operating System

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: UML mount failure with Linux 6.11

[Benjamin Berg]

Hi,

I am probably not the right person to talk to. Maybe Hongbo Li can say
more?

That said, it looks like the filesystem now has the "hostfs" option. So
you can probably just use
  mount -t hostfs -o hostfs=/path none /mount/point
which is nicer anyway. Just a bit annoying as you probably need to pass
it differently for older kernels.

Benjamin

On Wed, 2024-11-06 at 17:22 +0530, Ritesh Raj Sarraf wrote:
> Hello Benjamin,
> 
> On Thu, 2024-10-31 at 11:07 +0100, Benjamin Berg wrote:
> > Hi,
> > 
> > Newer kernels have become more picky about that with the new mount
> > API.
> > This is relevant, see the discussion about "Unknown options":
> >   https://lwn.net/Articles/979166/
> > 
> > We only use hostfs for the root file system and in that case it
> > works
> > well if you pass the path using "hostfs=/path" on the kernel
> > command
> > line. Doing that avoids issues when remounting the file system
> > later
> > on.
> > 
> 
> As upstream developers for UML, what would you conclude it as ?
> 
> We've recommended using hostfs for the UML kernel modules as well.
> What
> would be the alternate approach to ensuring a proper boot for a
> modular
> UML kernel ?
> 
> 
> > I suppose that currently it does not work to mount hostfs later on.
> > No
> > idea what the right fix is. Maybe the host directory should be an
> > explicit option like "hostpath=..." or so to make it compatible
> > with
> > the new mount APIs.
> 
> The ability to mount any hostfs mount point was/is a feature provided
> by UML. We've used it and integrated with many tools like debos,
> fakemachine etc; the Debian bug report has the details.
> 
> There'll be more reports following once UML 6.11 hits Debian Testing.
> 
> I hadn't expected a working feature to break with a newer Linux
> release. :-(
> 
> Thanks,
> Ritesh
> 

Re: UML mount failure with Linux 6.11

[Hongbo Li]

Hi Ritesh and Benjamin,

I have read the context in [1]. It seems your tool has already used new 
mount api to mount the hostfs. It now rejects unknown mount options as 
many other filesystems do regardless of its earlier behavior (which 
treats any option as the root directory in hostfs).

This should be clarified for unknown option. For older hostfs (before 
new mount api), it treats the whole string behind -o as the root 
directory. This will allow any mount options, but also limit its 
extension. It is not the same as other mainline filesystems which will 
split the option string with ','. So for these filesystems, they can 
consider whether reject unknown mount options or not.

I'm not sure it is reasonable in this way. If we accept unknown option 
in the hostfs, it will be treated as root directory. But which one 
should be used (like mount -t hostfs -o unknown,/root/directory none 
/mnt). So in the conversion, we introduce the `hostfs` key to mark the 
root directory. May be we need more discussion about use case.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1086194


Thanks,
Hongbo

On 2024/11/7 3:25, Benjamin Berg wrote:
> Hi,
> 
> I am probably not the right person to talk to. Maybe Hongbo Li can say
> more?
> 
> That said, it looks like the filesystem now has the "hostfs" option. So
> you can probably just use
>    mount -t hostfs -o hostfs=/path none /mount/point
> which is nicer anyway. Just a bit annoying as you probably need to pass
> it differently for older kernels.
> 
> Benjamin
> 
> On Wed, 2024-11-06 at 17:22 +0530, Ritesh Raj Sarraf wrote:
>> Hello Benjamin,
>>
>> On Thu, 2024-10-31 at 11:07 +0100, Benjamin Berg wrote:
>>> Hi,
>>>
>>> Newer kernels have become more picky about that with the new mount
>>> API.
>>> This is relevant, see the discussion about "Unknown options":
>>>    https://lwn.net/Articles/979166/
>>>
>>> We only use hostfs for the root file system and in that case it
>>> works
>>> well if you pass the path using "hostfs=/path" on the kernel
>>> command
>>> line. Doing that avoids issues when remounting the file system
>>> later
>>> on.
>>>
>>
>> As upstream developers for UML, what would you conclude it as ?
>>
>> We've recommended using hostfs for the UML kernel modules as well.
>> What
>> would be the alternate approach to ensuring a proper boot for a
>> modular
>> UML kernel ?
>>
>>
>>> I suppose that currently it does not work to mount hostfs later on.
>>> No
>>> idea what the right fix is. Maybe the host directory should be an
>>> explicit option like "hostpath=..." or so to make it compatible
>>> with
>>> the new mount APIs.
>>
>> The ability to mount any hostfs mount point was/is a feature provided
>> by UML. We've used it and integrated with many tools like debos,
>> fakemachine etc; the Debian bug report has the details.
>>
>> There'll be more reports following once UML 6.11 hits Debian Testing.
>>
>> I hadn't expected a working feature to break with a newer Linux
>> release. :-(
>>
>> Thanks,
>> Ritesh
>>
> 
> 

Re: UML mount failure with Linux 6.11

[Johannes Berg]

Hi,

So took me a while to grok the context, and to understand why it was
working for me, and broken on another machine...


> I have read the context in [1]. It seems your tool has already used new 
> mount api to mount the hostfs.

Yes, however, that's a default that's entirely transparent to the user.
This is why I wasn't seeing the errors, depending on the machine I'm
running this on, because the 'mount' tool either uses the old or new
style and the user can never know.

> It now rejects unknown mount options as 
> many other filesystems do regardless of its earlier behavior (which 
> treats any option as the root directory in hostfs).

And that's clearly the root cause of this regression.

You can't even argue it's not a regression, because before cd140ce9f611
("hostfs: convert hostfs to use the new mount API") it still worked with
the new fsconfig() API, but with the old mount options...

> I'm not sure it is reasonable in this way. If we accept unknown option 
> in the hostfs, it will be treated as root directory. But which one 
> should be used (like mount -t hostfs -o unknown,/root/directory none 
> /mnt). So in the conversion, we introduce the `hostfs` key to mark the 
> root directory. May be we need more discussion about use case.

There's only one option anyway, so I'd think we just need to fix this
and not require the hostfs= key. Perhaps if and only if it starts with
hostfs= we can treat it as a key, otherwise treat it all as a dir? But I
guess the API wouldn't make that easy.

Anyway, I dunno, but it seems like a regression to me and we should try
to find a way to fix it.

johannes

Re: UML mount failure with Linux 6.11

[Hongbo Li]

On 2024/11/7 21:09, Johannes Berg wrote:
> Hi,
> 
> So took me a while to grok the context, and to understand why it was
> working for me, and broken on another machine...
> 
> 
>> I have read the context in [1]. It seems your tool has already used new
>> mount api to mount the hostfs.
> 
> Yes, however, that's a default that's entirely transparent to the user.
> This is why I wasn't seeing the errors, depending on the machine I'm
> running this on, because the 'mount' tool either uses the old or new
> style and the user can never know.
> 
>> It now rejects unknown mount options as
>> many other filesystems do regardless of its earlier behavior (which
>> treats any option as the root directory in hostfs).
> 
> And that's clearly the root cause of this regression.
> 
> You can't even argue it's not a regression, because before cd140ce9f611
> ("hostfs: convert hostfs to use the new mount API") it still worked with
> the new fsconfig() API, but with the old mount options...
> 
>> I'm not sure it is reasonable in this way. If we accept unknown option
>> in the hostfs, it will be treated as root directory. But which one
>> should be used (like mount -t hostfs -o unknown,/root/directory none
>> /mnt). So in the conversion, we introduce the `hostfs` key to mark the
>> root directory. May be we need more discussion about use case.
> 
> There's only one option anyway, so I'd think we just need to fix this
> and not require the hostfs= key. Perhaps if and only if it starts with
> hostfs= we can treat it as a key, otherwise treat it all as a dir? But I

May be we can do that (just record the unknown option in host_root_path 
when fs_parse failed). But this lead us to consider the case in which we 
should handle a long option -o unknown1,hostfs=xxx,unknow2, which one 
should be treated as the root directory? For new mount api, it will call 
fsconfig three times to set the root directory. For older one, if one 
path with that name exactly, may be it can mount successfully.


Thanks,
Hongbo

> guess the API wouldn't make that easy.
> 
> Anyway, I dunno, but it seems like a regression to me and we should try
> to find a way to fix it.
> 
> johannes
> 

Re: UML mount failure with Linux 6.11

[Johannes Berg]

On Thu, 2024-11-07 at 22:17 +0800, Hongbo Li wrote:
> > There's only one option anyway, so I'd think we just need to fix this
> > and not require the hostfs= key. Perhaps if and only if it starts with
> > hostfs= we can treat it as a key, otherwise treat it all as a dir? But I
> 
> May be we can do that (just record the unknown option in host_root_path 
> when fs_parse failed). But this lead us to consider the case in which we 
> should handle a long option -o unknown1,hostfs=xxx,unknow2, which one 
> should be treated as the root directory? For new mount api, it will call 
> fsconfig three times to set the root directory. For older one, if one 
> path with that name exactly, may be it can mount successfully.

Technically, comma _is_ valid in a dir name, as you say ... so perhaps
the new mount API handling would need to be modified to have an escape
for this and not split it automatically, if the underlying FS doesn't
want that? Or we just revert cd140ce9f611 too?

I feel like perhaps we just found a corner case - clearly the new mount
API assumes that mount options are always comma-separated, but, well,
turns out that's simply not true since hostfs has only a single option
and treats the whole thing as a single string.

johannes

Re: UML mount failure with Linux 6.11

[Hongbo Li]

Hi Karel,

We are discussing about the hostfs mount with new mount API in [1]. And 
may need your help.

After finishing the conversion to the new mount API for hostfs, it 
encountered a situation where the old version supported only one mount 
option, and the whole mount option was used as the root path (it is also 
valid for the path to contain commas). But when switching to the new 
mount API, the option part will be split using commas (if I'm not 
mistaken, this step would be done in libmount), which could potentially 
split a complete path into multiple parts, and the call fsconfig syscall 
to set the mount options for underline filesystems. This is different 
from the original intention of hostfs. And this kind of situation is not 
common in other filesystems.

Is it necessary to handle the hostfs situation specially within libmount.

[1] 
https://lore.kernel.org/all/ac1b8ddd62ab22e6311ddba0c07c65b389a1c5df.camel@sipsolutions.net/T/

Thanks,
Hongbo


On 2024/11/7 22:35, Johannes Berg wrote:
> On Thu, 2024-11-07 at 22:17 +0800, Hongbo Li wrote:
>>> There's only one option anyway, so I'd think we just need to fix this
>>> and not require the hostfs= key. Perhaps if and only if it starts with
>>> hostfs= we can treat it as a key, otherwise treat it all as a dir? But I
>>
>> May be we can do that (just record the unknown option in host_root_path
>> when fs_parse failed). But this lead us to consider the case in which we
>> should handle a long option -o unknown1,hostfs=xxx,unknow2, which one
>> should be treated as the root directory? For new mount api, it will call
>> fsconfig three times to set the root directory. For older one, if one
>> path with that name exactly, may be it can mount successfully.
> 
> Technically, comma _is_ valid in a dir name, as you say ... so perhaps
> the new mount API handling would need to be modified to have an escape
> for this and not split it automatically, if the underlying FS doesn't
> want that? Or we just revert cd140ce9f611 too?
> 
> I feel like perhaps we just found a corner case - clearly the new mount
> API assumes that mount options are always comma-separated, but, well,
> turns out that's simply not true since hostfs has only a single option
> and treats the whole thing as a single string.
> 
> johannes
> 

Re: UML mount failure with Linux 6.11

[Karel Zak]

 Hi,

On Mon, Nov 11, 2024 at 09:16:18AM GMT, Hongbo Li wrote:
> We are discussing about the hostfs mount with new mount API in [1]. And may
> need your help.
> 
> After finishing the conversion to the new mount API for hostfs, it
> encountered a situation where the old version supported only one mount
> option, and the whole mount option was used as the root path (it is also
> valid for the path to contain commas). But when switching to the new mount
> API, the option part will be split using commas (if I'm not mistaken, this
> step would be done in libmount), which could potentially split a complete
> path into multiple parts, and the call fsconfig syscall to set the mount
> options for underline filesystems. This is different from the original
> intention of hostfs. And this kind of situation is not common in other
> filesystems.

The options has been always parsed by mount(8) and it's very fragile
to assume that kernel get as in the original order (etc.). 

For decades, commas have been supported in mount options. For example,
SeLinux uses them frequently in context settings. All you need to do
is use quotes, but be careful because the shell will strip them off.
Therefore, double quoting is required.

   mount -o 'rw,bbb="this,is,value",ccc'

It's also supported in fstab, just use name="v,a,l,u,e"

You can try it:

 # strace -e fsconfig mount -t tmpfs -o 'rw,bbb="this,is,value",ccc' tmpfs /dontexist

 fsconfig(3, FSCONFIG_SET_STRING, "source", "tmpfs", 0) = 0
 fsconfig(3, FSCONFIG_SET_FLAG, "rw", NULL, 0) = 0
 fsconfig(3, FSCONFIG_SET_STRING, "bbb", "this,is,value", 0) = -1 EINVAL 

You can see the expected result when using fsconfig().

 Karel


-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: UML mount failure with Linux 6.11

[Hongbo Li]

On 2024/11/13 4:10, Karel Zak wrote:
> 
>   Hi,
> 
> On Mon, Nov 11, 2024 at 09:16:18AM GMT, Hongbo Li wrote:
>> We are discussing about the hostfs mount with new mount API in [1]. And may
>> need your help.
>>
>> After finishing the conversion to the new mount API for hostfs, it
>> encountered a situation where the old version supported only one mount
>> option, and the whole mount option was used as the root path (it is also
>> valid for the path to contain commas). But when switching to the new mount
>> API, the option part will be split using commas (if I'm not mistaken, this
>> step would be done in libmount), which could potentially split a complete
>> path into multiple parts, and the call fsconfig syscall to set the mount
>> options for underline filesystems. This is different from the original
>> intention of hostfs. And this kind of situation is not common in other
>> filesystems.
> 
> The options has been always parsed by mount(8) and it's very fragile
> to assume that kernel get as in the original order (etc.).
> 
> For decades, commas have been supported in mount options. For example,
> SeLinux uses them frequently in context settings. All you need to do
> is use quotes, but be careful because the shell will strip them off.
> Therefore, double quoting is required.
>

Thanks for your reply!

If I'm not mistaken, we should add double quoting explicitly if we need 
commas in mount options. However, it seems different for hostfs. For 
example, with hostfs, if we use "mount -t hostfs none -o 
/home/hostfs,dir /mnt" in the older interface, which can successfully 
mount the host directory `/home/hostfs,dir`, then we should use "mount 
-t hostfs none -o '"/home/hostfs,dir"' /mnt" in the new interface. If 
that is the case, we should change the mount command which is hardcoded 
in the original project.

Thanks,
Hongbo

>     mount -o 'rw,bbb="this,is,value",ccc'
> 
> It's also supported in fstab, just use name="v,a,l,u,e"
> 
> You can try it:
> 
>   # strace -e fsconfig mount -t tmpfs -o 'rw,bbb="this,is,value",ccc' tmpfs /dontexist
> 
>   fsconfig(3, FSCONFIG_SET_STRING, "source", "tmpfs", 0) = 0
>   fsconfig(3, FSCONFIG_SET_FLAG, "rw", NULL, 0) = 0
>   fsconfig(3, FSCONFIG_SET_STRING, "bbb", "this,is,value", 0) = -1 EINVAL
> 
> You can see the expected result when using fsconfig().
> 
>   Karel
> 
> 

Re: UML mount failure with Linux 6.11

[Hongbo Li]

On 2024/11/13 9:23, Hongbo Li wrote:
> 
> 
> On 2024/11/13 4:10, Karel Zak wrote:
>>
>>   Hi,
>>
>> On Mon, Nov 11, 2024 at 09:16:18AM GMT, Hongbo Li wrote:
>>> We are discussing about the hostfs mount with new mount API in [1]. 
>>> And may
>>> need your help.
>>>
>>> After finishing the conversion to the new mount API for hostfs, it
>>> encountered a situation where the old version supported only one mount
>>> option, and the whole mount option was used as the root path (it is also
>>> valid for the path to contain commas). But when switching to the new 
>>> mount
>>> API, the option part will be split using commas (if I'm not mistaken, 
>>> this
>>> step would be done in libmount), which could potentially split a 
>>> complete
>>> path into multiple parts, and the call fsconfig syscall to set the mount
>>> options for underline filesystems. This is different from the original
>>> intention of hostfs. And this kind of situation is not common in other
>>> filesystems.
>>
>> The options has been always parsed by mount(8) and it's very fragile
>> to assume that kernel get as in the original order (etc.).
>>
>> For decades, commas have been supported in mount options. For example,
>> SeLinux uses them frequently in context settings. All you need to do
>> is use quotes, but be careful because the shell will strip them off.
>> Therefore, double quoting is required.
>>
> 
> Thanks for your reply!
> 
> If I'm not mistaken, we should add double quoting explicitly if we need 
> commas in mount options. However, it seems different for hostfs. For 
> example, with hostfs, if we use "mount -t hostfs none -o 
> /home/hostfs,dir /mnt" in the older interface, which can successfully 
> mount the host directory `/home/hostfs,dir`, then we should use "mount 
> -t hostfs none -o '"/home/hostfs,dir"' /mnt" in the new interface. If 
> that is the case, we should change the mount command which is hardcoded 
> in the original project.
Sorry for the disturbance. So am I right?

Thanks,
Hongbo

> 
> Thanks,
> Hongbo
> 
>>     mount -o 'rw,bbb="this,is,value",ccc'
>>
>> It's also supported in fstab, just use name="v,a,l,u,e"
>>
>> You can try it:
>>
>>   # strace -e fsconfig mount -t tmpfs -o 'rw,bbb="this,is,value",ccc' 
>> tmpfs /dontexist
>>
>>   fsconfig(3, FSCONFIG_SET_STRING, "source", "tmpfs", 0) = 0
>>   fsconfig(3, FSCONFIG_SET_FLAG, "rw", NULL, 0) = 0
>>   fsconfig(3, FSCONFIG_SET_STRING, "bbb", "this,is,value", 0) = -1 EINVAL
>>
>> You can see the expected result when using fsconfig().
>>
>>   Karel
>>
>>
> 
> 

Re: UML mount failure with Linux 6.11

[Karel Zak]

On Thu, Nov 21, 2024 at 09:53:06PM GMT, Hongbo Li wrote:
> > > The options has been always parsed by mount(8) and it's very fragile
> > > to assume that kernel get as in the original order (etc.).
> > > 
> > > For decades, commas have been supported in mount options. For example,
> > > SeLinux uses them frequently in context settings. All you need to do
> > > is use quotes, but be careful because the shell will strip them off.
> > > Therefore, double quoting is required.
> > > 
> > 
> > Thanks for your reply!
> > 
> > If I'm not mistaken, we should add double quoting explicitly if we need
> > commas in mount options. However, it seems different for hostfs. For
> > example, with hostfs, if we use "mount -t hostfs none -o
> > /home/hostfs,dir /mnt" in the older interface, which can successfully
> > mount the host directory `/home/hostfs,dir`, then we should use "mount
> > -t hostfs none -o '"/home/hostfs,dir"' /mnt" in the new interface. If
> > that is the case, we should change the mount command which is hardcoded
> > in the original project.
> Sorry for the disturbance. So am I right?

The old interface (mount(2)) sends all options together as one string.
This means that if mount(8)/libmount does not modify the string in an
unwanted way, then you are safe.

The new interface uses one fsconfig() call for each option, so parsing
is required to split the "-o string". This means that you will always
be affected by options separators interpretation. This is the reason
why I have suggested quoting.

I see a different issue (and I apologize for missing it in my previous
reply). mount(8) only supports quoting for named options, which means
using the format NAME="v,a,l,u,e". This is not expected if you do not
use NAME=, for example "/home/hostfs,dir".


I am not sure, but there may be another issue with your use-case. "-o
'/home/hostfs,dir'" means that "/home/hostfs,dir" will be used as the
key (variable name) in the fsconfig() syscall. I have doubts that it
is designed for long strings like paths. At first glance, kernel uses
only 256 bytes for the key (and since it is a variable without a
value, libmount will use it as FSCONFIG_SET_FLAG). 

Maybe Christian will know more details.

For now, my suggestion is to use the LIBMOUNT_FORCE_MOUNT2=always
environment variable to force mount(8) to use the classic mount(2)
syscall.

The long-term solution would be to clean up hostfs and use named
variables, such as "mount -t hostfs none -o 'path="/home/hostfs"'.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: UML mount failure with Linux 6.11

[Johannes Berg]

On Mon, 2024-11-25 at 18:43 +0100, Karel Zak wrote:
> 
> The long-term solution would be to clean up hostfs and use named
> variables, such as "mount -t hostfs none -o 'path="/home/hostfs"'.

That's what Hongbo's commit *did*, afaict, but it is a regression.

Now most of the regression is that with fsconfig() call it was no longer
possible to specify a bare folder, and then we got discussing what
happens if the folder name actually contains a comma...

But this is still a regression, so we need to figure out what to do
short term?

Ignoring the "path with comma" issue, because we can't even fix that in
the kernel given what you describe changed in userspace, we can probably
only

 1) revert the hostfs conversion to the new API, or
 2) somehow not require the hostfs= key?

I don't know if either of those are even possible


Fixing the regression fully (including for paths containing commas)
probably also requires userspace changes. If you don't want to make
those we can only point to your workarounds instead, since we can't do
anything on the kernel side.

I don't know the fsconfig() API, is it possible to have key-less or
value-less calls? What does happen 

johannes

Re: UML mount failure with Linux 6.11

[Hongbo Li]

On 2024/11/26 21:50, Johannes Berg wrote:
> On Mon, 2024-11-25 at 18:43 +0100, Karel Zak wrote:
>>
>> The long-term solution would be to clean up hostfs and use named
>> variables, such as "mount -t hostfs none -o 'path="/home/hostfs"'.
> 
> That's what Hongbo's commit *did*, afaict, but it is a regression.
> 
> Now most of the regression is that with fsconfig() call it was no longer
> possible to specify a bare folder, and then we got discussing what
> happens if the folder name actually contains a comma...
> 
> But this is still a regression, so we need to figure out what to do
> short term?
> 
So for short term, even long term, can we consider handling the hostfs 
situation specially within libmount? Such as treat the whole option as 
one key(also may be with comma, even with equal), in this case, libmount 
will use it as FSCONFIG_SET_FLAG. We can do that because for hostfs, it 
only has one mount option, and no need for extension.

Thanks,
Hongbo

> Ignoring the "path with comma" issue, because we can't even fix that in
> the kernel given what you describe changed in userspace, we can probably
> only
> 
>   1) revert the hostfs conversion to the new API, or
>   2) somehow not require the hostfs= key?
> 
> I don't know if either of those are even possible
> 
> 
> Fixing the regression fully (including for paths containing commas)
> probably also requires userspace changes. If you don't want to make
> those we can only point to your workarounds instead, since we can't do
> anything on the kernel side.
> 
> I don't know the fsconfig() API, is it possible to have key-less or
> value-less calls? What does happen
> 
> johannes
> 

Re: UML mount failure with Linux 6.11

[Karel Zak]

On Wed, Nov 27, 2024 at 09:26:46AM GMT, Hongbo Li wrote:
> 
> 
> On 2024/11/26 21:50, Johannes Berg wrote:
> > On Mon, 2024-11-25 at 18:43 +0100, Karel Zak wrote:
> > > 
> > > The long-term solution would be to clean up hostfs and use named
> > > variables, such as "mount -t hostfs none -o 'path="/home/hostfs"'.
> > 
> > That's what Hongbo's commit *did*, afaict, but it is a regression.
> > 
> > Now most of the regression is that with fsconfig() call it was no longer
> > possible to specify a bare folder, and then we got discussing what
> > happens if the folder name actually contains a comma...
> > 
> > But this is still a regression, so we need to figure out what to do
> > short term?
> > 
> So for short term, even long term, can we consider handling the hostfs
> situation specially within libmount?

Yes (see reply to Johannes ).

> Such as treat the whole option as one
> key(also may be with comma, even with equal)

There could be userspace specific options, VFS flags, etc.

  -o /home/hostfs,ro,noexec

Is it one path or path and two options?

> in this case, libmount will
> use it as FSCONFIG_SET_FLAG. We can do that because for hostfs, it only has
> one mount option, and no need for extension.

We can automatically add a key (e.g. hostfs=) and use FSCONFIG_SET_FLAG.
The goal should be to get the path as a value, because key is limited to
256 bytes.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: UML mount failure with Linux 6.11

[Johannes Berg]

Let me try to unify the threads, and perhaps further my understanding -
you seem to already have much more of that than me :)

> > > But this is still a regression, so we need to figure out what to do
> > > short term?
> > > 
> > So for short term, even long term, can we consider handling the hostfs
> > situation specially within libmount?
> 
> Yes (see reply to Johannes ).

I'd argue though that this doesn't count as fixing the regression, since
the kernel was fine before the changes there (even before porting hostfs
to the new API) with _any_ version of userspace. Except perhaps for when
there's a comma in the path, which I suppose would've broken one way or
the other by mount(8) moving to the new API?

I'd kind of prefer we fixed the immediate regression, at least without
the comma, in the kernel. But I guess you can do whatever you can get
away with ;-) And it's already broken for two kernel releases now ...
but I guess those haven't percolated through the ecosystem *that* much.

[from the other email]
> I can add a temporary workaround to libmount for hostfs, which will
> automatically add the hostfs= key for unnamed paths. This will allow
> you to receive the expected fsconfig() data from userspace.

so I'm not sure this makes too much sense - kernel upgrade broke it, I
guess kernel upgrade should try to fix it?

Assuming no commas, would mount(8) today send the path as the key to a
flag option? We could perhaps find a way to handle that in the kernel,
and then just do the longer-term plan of moving users to hostfs="..."
(by documentation/warning) in the future?


> > Such as treat the whole option as one
> > key(also may be with comma, even with equal)
> 
> There could be userspace specific options, VFS flags, etc.
> 
>   -o /home/hostfs,ro,noexec
> 
> Is it one path or path and two options?

Yeah but there _aren't_ in hostfs, right now. So without the mount API
we'd not even be asking that question since the parsing was in the fs
and hostfs would just never have done it, I guess?

> We can automatically add a key (e.g. hostfs=) and use FSCONFIG_SET_FLAG.
> The goal should be to get the path as a value, because key is limited to
> 256 bytes.

Right, we can do that going forward, but it doesn't really address the
regression?

johannes

Re: UML mount failure with Linux 6.11

[Karel Zak]

On Wed, Nov 27, 2024 at 02:15:54PM GMT, Johannes Berg wrote:
> Let me try to unify the threads, and perhaps further my understanding -
> you seem to already have much more of that than me :)
> 
> > > > But this is still a regression, so we need to figure out what to do
> > > > short term?
> > > > 
> > > So for short term, even long term, can we consider handling the hostfs
> > > situation specially within libmount?
> > 
> > Yes (see reply to Johannes ).
> 
> I'd argue though that this doesn't count as fixing the regression, since
> the kernel was fine before the changes there (even before porting hostfs
> to the new API) with _any_ version of userspace. Except perhaps for when
> there's a comma in the path, which I suppose would've broken one way or
> the other by mount(8) moving to the new API?

Another option is to hardcode a libmount exception that, for hostfs,
the default behavior should be to use the classic mount(2) syscall if
the hostfs= option is not present.
    
 -t hostfs -o /home/hostfs,dir          --> mount(8)
 -t hostfs -o hostfs="/home/hostfs,dir" --> new API

(Alternatively, use the old mount(8) for hostfs regardless of the
options chosen.)

> Assuming no commas, would mount(8) today send the path as the key to a
> flag option? 

 Yes, (I have no hostfs here, so example with ext4):

 # strace -e fsconfig mount -t ext4 -o /home/hostfs none /mnt/test

 fsconfig(3, FSCONFIG_SET_STRING, "source", "none", 0) = 0
 fsconfig(3, FSCONFIG_SET_FLAG, "/home/hostfs", NULL, 0) = -1 EINVAL (Invalid argument)

> We could perhaps find a way to handle that in the kernel,
> and then just do the longer-term plan of moving users to hostfs="..."
> (by documentation/warning) in the future?

The question is whether investing time in using the path-as-key
approach makes sense. Perhaps it would be better to stick with the old
mount(2) and focus on developing a new API that does not have any
legacy issues. Users who choose to use the hostfs= option and the new
kernel will be able to utilize the new API.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: UML mount failure with Linux 6.11

[Johannes Berg]

Hi,

> > I'd argue though that this doesn't count as fixing the regression, since
> > the kernel was fine before the changes there (even before porting hostfs
> > to the new API) with _any_ version of userspace. Except perhaps for when
> > there's a comma in the path, which I suppose would've broken one way or
> > the other by mount(8) moving to the new API?
> 
> Another option is to hardcode a libmount exception that, for hostfs,
> the default behavior should be to use the classic mount(2) syscall if
> the hostfs= option is not present.

I'm not sure using the old mount API would work because the kernel
converted internally to the new one now. Anyway it'd still be a kernel
regression if we have to fix it in userspace, no? :)

> > Assuming no commas, would mount(8) today send the path as the key to a
> > flag option? 
> 
>  Yes, (I have no hostfs here, so example with ext4):
> 
>  # strace -e fsconfig mount -t ext4 -o /home/hostfs none /mnt/test
> 
>  fsconfig(3, FSCONFIG_SET_STRING, "source", "none", 0) = 0
>  fsconfig(3, FSCONFIG_SET_FLAG, "/home/hostfs", NULL, 0) = -1 EINVAL (Invalid argument)

So I guess that means for paths without comma (almost certainly the
overwhelming majority) we could somehow work around it in the kernel.
Hongbo, what do you think?

> > We could perhaps find a way to handle that in the kernel,
> > and then just do the longer-term plan of moving users to hostfs="..."
> > (by documentation/warning) in the future?
> 
> The question is whether investing time in using the path-as-key
> approach makes sense. Perhaps it would be better to stick with the old
> mount(2)

I think it's a matter of not regressing for users, if they just update
the kernel and have old or existing mount tools. And I'm not convinced
using the old API will actually fix the issue, I think maybe the kernel
itself would break that too by parsing it now for the new API?

> and focus on developing a new API that does not have any
> legacy issues. Users who choose to use the hostfs= option and the new
> kernel will be able to utilize the new API.

Right, but we already have that - you can specify hostfs="quoted path"
when everything is new and it'll work just fine?

Question is more around the regression, to me.

johannes

Re: UML mount failure with Linux 6.11

[Hongbo Li]

On 2024/11/27 20:02, Karel Zak wrote:
> On Wed, Nov 27, 2024 at 09:26:46AM GMT, Hongbo Li wrote:
>>
>>
>> On 2024/11/26 21:50, Johannes Berg wrote:
>>> On Mon, 2024-11-25 at 18:43 +0100, Karel Zak wrote:
>>>>
>>>> The long-term solution would be to clean up hostfs and use named
>>>> variables, such as "mount -t hostfs none -o 'path="/home/hostfs"'.
>>>
>>> That's what Hongbo's commit *did*, afaict, but it is a regression.
>>>
>>> Now most of the regression is that with fsconfig() call it was no longer
>>> possible to specify a bare folder, and then we got discussing what
>>> happens if the folder name actually contains a comma...
>>>
>>> But this is still a regression, so we need to figure out what to do
>>> short term?
>>>
>> So for short term, even long term, can we consider handling the hostfs
>> situation specially within libmount?
> 
> Yes (see reply to Johannes ).
> 
>> Such as treat the whole option as one
>> key(also may be with comma, even with equal)
> 
> There could be userspace specific options, VFS flags, etc.
> 
>    -o /home/hostfs,ro,noexec
> 
> Is it one path or path and two options?
> 
Interesting!
Perhaps more documentation is needed. VFS flags is filtered, but for 
hostfs, it is a valid path. The semantics seems not very complete. :)

Thanks,
Hongbo

>> in this case, libmount will
>> use it as FSCONFIG_SET_FLAG. We can do that because for hostfs, it only has
>> one mount option, and no need for extension.
> 
> We can automatically add a key (e.g. hostfs=) and use FSCONFIG_SET_FLAG.
> The goal should be to get the path as a value, because key is limited to
> 256 bytes.
> 
>      Karel
> 

Re: UML mount failure with Linux 6.11

[Karel Zak]

On Tue, Nov 26, 2024 at 02:50:38PM GMT, Johannes Berg wrote:
> On Mon, 2024-11-25 at 18:43 +0100, Karel Zak wrote:
> > 
> > The long-term solution would be to clean up hostfs and use named
> > variables, such as "mount -t hostfs none -o 'path="/home/hostfs"'.
> 
> That's what Hongbo's commit *did*, afaict, but it is a regression.
> 
> Now most of the regression is that with fsconfig() call it was no longer
> possible to specify a bare folder, and then we got discussing what
> happens if the folder name actually contains a comma...
> 
> But this is still a regression, so we need to figure out what to do
> short term?

I will add support for quotes for unnamed options, so that
"/home/hostfs,dir" will be treated as a single option for libmount.

I am unsure how to resolve this issue without using quotes, as we need
a method to distinguish between a path with a comma and other options.

> Ignoring the "path with comma" issue, because we can't even fix that in
> the kernel given what you describe changed in userspace, we can probably
> only
> 
>  1) revert the hostfs conversion to the new API, or
>  2) somehow not require the hostfs= key?

The hostfs= key is likely the most reliable method to use fsconfig().
The goal should be to get from userspace:

    fsconfig(fs, FSCONFIG_SET_STRING, "hostfs", "/home/hostfs,dir");

I can add a temporary workaround to libmount for hostfs, which will
automatically add the hostfs= key for unnamed paths. This will allow
you to receive the expected fsconfig() data from userspace.

If we will go this way, then it would be nice to update hostfs docs to
motivate users to use hostfs="" in configuration, and I will probably
add later a warning to mount(8) to force users to use hostfs="" key.

Does it make sense?

> Fixing the regression fully (including for paths containing commas)
> probably also requires userspace changes. If you don't want to make
> those we can only point to your workarounds instead, since we can't do
> anything on the kernel side.

I am open to temporary FS-specific libmount workarounds, but I would
like to see a long-term solution and plan on how to encourage users to
use it.

> I don't know the fsconfig() API, is it possible to have key-less or
> value-less calls? What does happen 

The key is always required; otherwise, you will receive an -EINVAL error.
https://github.com/torvalds/linux/blob/master/fs/fsopen.c#L369

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: UML mount failure with Linux 6.11

[Geert Uytterhoeven]

Hi Karel,

On Wed, Nov 27, 2024 at 12:56 PM Karel Zak <kzak@redhat.com> wrote:
> On Tue, Nov 26, 2024 at 02:50:38PM GMT, Johannes Berg wrote:
> > On Mon, 2024-11-25 at 18:43 +0100, Karel Zak wrote:
> > > The long-term solution would be to clean up hostfs and use named
> > > variables, such as "mount -t hostfs none -o 'path="/home/hostfs"'.
> >
> > That's what Hongbo's commit *did*, afaict, but it is a regression.
> >
> > Now most of the regression is that with fsconfig() call it was no longer
> > possible to specify a bare folder, and then we got discussing what
> > happens if the folder name actually contains a comma...
> >
> > But this is still a regression, so we need to figure out what to do
> > short term?
>
> I will add support for quotes for unnamed options, so that
> "/home/hostfs,dir" will be treated as a single option for libmount.
>
> I am unsure how to resolve this issue without using quotes, as we need
> a method to distinguish between a path with a comma and other options.

Escape the comma using a backslash?

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds