All of lore.kernel.org
 help / color / mirror / Atom feed
* mmap on 9p broken in 6.4
@ 2023-07-11 20:19 Robert Schwebel
  2023-07-12  9:01 ` Christian Schoenebeck
  2023-07-13 11:22 ` Linux regression tracking #adding (Thorsten Leemhuis)
  0 siblings, 2 replies; 17+ messages in thread
From: Robert Schwebel @ 2023-07-11 20:19 UTC (permalink / raw)
  To: Eric Van Hensbergen
  Cc: Latchesar Ionkov, Dominique Martinet, Christian Schoenebeck, v9fs,
	kernel

Hi Eric,

TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
"rootflags=ignoreqv" fixes it as well, but it feels like a regression.

I'm tracking down an issue which recently turned up in DistroKit [1] (an
embedded Linux distro based on the ptxdist build system). The issue was a bit
uggly, as my CI didn't find it (systems boot up normally after a while, and I
only use 9p for virtual qemu machines, while most of the test farm is real
hardware).

The qemu machine in question is qemu-system-arm, emulating an ARM v7a machine.

When starting the systems interactively, I get a lot of error output from
ldconfig, like this:

[   17.412964] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libgcc_s.so.1.
[   17.418851] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libstdc++.so.
[   17.425009] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libstdc++.so.6.0.30.
[   17.436671] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libstdc++.so.6.
[   17.448451] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libatomic.so.
[   17.456418] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libatomic.so.1.2.0.
...

Running ldconfig with strace shows this, for all libraries::

| statx(AT_FDCWD, "/lib/libnm.so.0", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFLNK|0777, stx_size=14, ...}) = 0
| statx(AT_FDCWD, "/lib/libnm.so.0", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|064 4, stx_size=862228, ...}) = 0
| openat(AT_FDCWD, "/lib/libnm.so.0", O_RDONLY|O_LARGEFILE) = 4
| statx(4, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_s ize=862228, ...}) = 0
| mmap2(NULL, 862228, PROT_READ, MAP_SHARED, 4, 0) = -1 ENODEV (No such device)
| write(2, "ldconfig: ", 10ldconfig: )              = 10
| write(2, "Cannot mmap file /lib/libnm.so.0"..., 34Cannot mmap file /lib/libnm.so.0.) = 34
| write(2, "\n", 1)                       = 1
| close(4)                                = 0

I could track down the breakage to

  1543b4c5071c54d76aad7a7a26a6e43082269b0c

My test setup has, in addition to the patch above, the following patches also
reverted on top of a vanilla 6.4 kernel:

  4eb3117888a923f6b9b1ad2dd093641c49a63ae5
  21e26d5e54ab7cfe6b488fd27d4d70956d07e03b

as 1543b cannot be reverted without those; however, the effect only goes away
when I also revert 1543b. The kernel has no other patches applied, only these
three reverts.

Do you have an idea what happens?

I have the setup easily available, so I could quickly test patches.

Thanks,

rsc

[1] https://git.pengutronix.de/cgit/DistroKit/
-- 
Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-11 20:19 mmap on 9p broken in 6.4 Robert Schwebel
@ 2023-07-12  9:01 ` Christian Schoenebeck
  2023-07-12 14:14   ` Robert Schwebel
  2023-07-13 11:22 ` Linux regression tracking #adding (Thorsten Leemhuis)
  1 sibling, 1 reply; 17+ messages in thread
From: Christian Schoenebeck @ 2023-07-12  9:01 UTC (permalink / raw)
  To: Eric Van Hensbergen, Robert Schwebel
  Cc: Latchesar Ionkov, Dominique Martinet, v9fs, kernel

On Tuesday, July 11, 2023 10:19:41 PM CEST Robert Schwebel wrote:
> Hi Eric,
> 
> TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
> "rootflags=ignoreqv" fixes it as well, but it feels like a regression.

What's the precise set of 9p mount options on guest, which 9p server is used
and what are 9p server's options respectively?

> I'm tracking down an issue which recently turned up in DistroKit [1] (an
> embedded Linux distro based on the ptxdist build system). The issue was a bit
> uggly, as my CI didn't find it (systems boot up normally after a while, and I
> only use 9p for virtual qemu machines, while most of the test farm is real
> hardware).
> 
> The qemu machine in question is qemu-system-arm, emulating an ARM v7a machine.
> 
> When starting the systems interactively, I get a lot of error output from
> ldconfig, like this:
> 
> [   17.412964] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libgcc_s.so.1.
> [   17.418851] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libstdc++.so.
> [   17.425009] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libstdc++.so.6.0.30.
> [   17.436671] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libstdc++.so.6.
> [   17.448451] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libatomic.so.
> [   17.456418] systemd-rc-once[127]: ldconfig: Cannot mmap file /lib/libatomic.so.1.2.0.
> ...
> 
> Running ldconfig with strace shows this, for all libraries::
> 
> | statx(AT_FDCWD, "/lib/libnm.so.0", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFLNK|0777, stx_size=14, ...}) = 0
> | statx(AT_FDCWD, "/lib/libnm.so.0", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|064 4, stx_size=862228, ...}) = 0
> | openat(AT_FDCWD, "/lib/libnm.so.0", O_RDONLY|O_LARGEFILE) = 4
> | statx(4, "", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT|AT_EMPTY_PATH, STATX_BASIC_STATS, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFREG|0644, stx_s ize=862228, ...}) = 0
> | mmap2(NULL, 862228, PROT_READ, MAP_SHARED, 4, 0) = -1 ENODEV (No such device)
> | write(2, "ldconfig: ", 10ldconfig: )              = 10
> | write(2, "Cannot mmap file /lib/libnm.so.0"..., 34Cannot mmap file /lib/libnm.so.0.) = 34
> | write(2, "\n", 1)                       = 1
> | close(4)                                = 0
> 
> I could track down the breakage to
> 
>   1543b4c5071c54d76aad7a7a26a6e43082269b0c

This patch is sensitive about 9p server transmitting a meaningful value for
file's QID version field. That's why more info about deployed 9p server would
help.

> My test setup has, in addition to the patch above, the following patches also
> reverted on top of a vanilla 6.4 kernel:
> 
>   4eb3117888a923f6b9b1ad2dd093641c49a63ae5
>   21e26d5e54ab7cfe6b488fd27d4d70956d07e03b
> 
> as 1543b cannot be reverted without those; however, the effect only goes away
> when I also revert 1543b. The kernel has no other patches applied, only these
> three reverts.
> 
> Do you have an idea what happens?
> 
> I have the setup easily available, so I could quickly test patches.
> 
> Thanks,
> 
> rsc
> 
> [1] https://git.pengutronix.de/cgit/DistroKit/
> 





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-12  9:01 ` Christian Schoenebeck
@ 2023-07-12 14:14   ` Robert Schwebel
  2023-07-12 15:57     ` Christian Schoenebeck
  0 siblings, 1 reply; 17+ messages in thread
From: Robert Schwebel @ 2023-07-12 14:14 UTC (permalink / raw)
  To: Christian Schoenebeck
  Cc: Eric Van Hensbergen, Robert Schwebel, Latchesar Ionkov,
	Dominique Martinet, v9fs, kernel

Christian,

On Wed, Jul 12, 2023 at 11:01:53AM +0200, Christian Schoenebeck wrote:
> > TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
> > "rootflags=ignoreqv" fixes it as well, but it feels like a regression.
> 
> What's the precise set of 9p mount options on guest, which 9p server is used
> and what are 9p server's options respectively?

The server is qemu, started like this:

qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'

This is qemu 8.0.2.

> > I could track down the breakage to
> > 
> >   1543b4c5071c54d76aad7a7a26a6e43082269b0c
> 
> This patch is sensitive about 9p server transmitting a meaningful value for
> file's QID version field. That's why more info about deployed 9p server would
> help.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-12 14:14   ` Robert Schwebel
@ 2023-07-12 15:57     ` Christian Schoenebeck
  2023-07-12 22:15       ` Robert Schwebel
  0 siblings, 1 reply; 17+ messages in thread
From: Christian Schoenebeck @ 2023-07-12 15:57 UTC (permalink / raw)
  To: Robert Schwebel
  Cc: Eric Van Hensbergen, Robert Schwebel, Latchesar Ionkov,
	Dominique Martinet, v9fs, kernel

On Wednesday, July 12, 2023 4:14:02 PM CEST Robert Schwebel wrote:
> Christian,
> 
> On Wed, Jul 12, 2023 at 11:01:53AM +0200, Christian Schoenebeck wrote:
> > > TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
> > > "rootflags=ignoreqv" fixes it as well, but it feels like a regression.
> > 
> > What's the precise set of 9p mount options on guest, which 9p server is used
> > and what are 9p server's options respectively?
> 
> The server is qemu, started like this:
> 
> qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'
> 
> This is qemu 8.0.2.

Does adding QEMU option "multidevs=remap" [1] resolve it?

[1] https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly

> > > I could track down the breakage to
> > > 
> > >   1543b4c5071c54d76aad7a7a26a6e43082269b0c
> > 
> > This patch is sensitive about 9p server transmitting a meaningful value for
> > file's QID version field. That's why more info about deployed 9p server would
> > help.
> 
> rsc
> 





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-12 15:57     ` Christian Schoenebeck
@ 2023-07-12 22:15       ` Robert Schwebel
  2023-07-13  7:50         ` Christian Schoenebeck
  0 siblings, 1 reply; 17+ messages in thread
From: Robert Schwebel @ 2023-07-12 22:15 UTC (permalink / raw)
  To: Christian Schoenebeck
  Cc: Eric Van Hensbergen, Latchesar Ionkov, Dominique Martinet, v9fs,
	kernel

On Wed, Jul 12, 2023 at 05:57:40PM +0200, Christian Schoenebeck wrote:
> > On Wed, Jul 12, 2023 at 11:01:53AM +0200, Christian Schoenebeck wrote:
> > > > TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
> > > > "rootflags=ignoreqv" fixes it as well, but it feels like a regression.
> > > 
> > > What's the precise set of 9p mount options on guest, which 9p server is used
> > > and what are 9p server's options respectively?
> > 
> > The server is qemu, started like this:
> > 
> > qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'
> > 
> > This is qemu 8.0.2.
> 
> Does adding QEMU option "multidevs=remap" [1] resolve it?
> 
> [1] https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly

Unfortunately not:

+ exec ./platform-v7a/sysroot-host/bin/qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device vi
rtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file,multidevs=remap -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/
root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'
[    0.000000] L2C: platform modifies aux control register: 0x02020000 -> 0x02420000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02020000 -> 0x02420000
[    0.004806] smp_twd: clock not found -2
[    1.069767] simple-pm-bus bus@40000000:motherboard-bus@40000000:iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
[    1.085798] simple-pm-bus bus@40000000:motherboard-bus@40000000:iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
[    1.153420] physmap-flash 48000000.psram: map_probe failed
[    1.535013] 9pnet_virtio: no channels available for device root

 ____                        _                   _
|  _ \ ___ _ __   __ _ _   _| |_ _ __ ___  _ __ (_)_  __
| |_) / _ \ '_ \ / _` | | | | __| '__/ _ \| '_ \| \ \/ /
|  __/  __/ | | | (_| | |_| | |_| | | (_) | | | | |>  <
|_|   \___|_| |_|\__, |\__,_|\__|_|  \___/|_| |_|_/_/\_\
                 |___/

 ____  _     _             _  ___ _
|  _ \(_)___| |_ _ __ ___ | |/ (_) |_
| | | | / __| __| '__/ _ \| ' /| | __|
| |_| | \__ \ |_| | | (_) | . \| | |_
|____/|_|___/\__|_|  \___/|_|\_\_|\__|


OSELAS(R)-DistroKit-2019.12.0-00429-g57ffae760eb9 / v7a-2019.12.0-00429-g57ffae760eb9
ptxdist-2023.07.1/2023-07-11T19:56:50+0200

DistroKit login: root
root@DistroKit:~ mount / -o remount,rw
root@DistroKit:~ ldconfig
ldconfig: Cannot mmap file /lib/libgcc_s.so.1.

ldconfig: Cannot mmap file /lib/libstdc++.so.

[...]

rsc
-- 
Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-12 22:15       ` Robert Schwebel
@ 2023-07-13  7:50         ` Christian Schoenebeck
       [not found]           ` <CAFkjPTmrbe9Djs_iYf=vu1xbTLhey0OmePR2Jw=C_dbQVYnAhQ@mail.gmail.com>
  2023-07-14  5:34           ` Robert Schwebel
  0 siblings, 2 replies; 17+ messages in thread
From: Christian Schoenebeck @ 2023-07-13  7:50 UTC (permalink / raw)
  To: Robert Schwebel
  Cc: Eric Van Hensbergen, Latchesar Ionkov, Dominique Martinet, v9fs,
	kernel

On Thursday, July 13, 2023 12:15:29 AM CEST Robert Schwebel wrote:
> On Wed, Jul 12, 2023 at 05:57:40PM +0200, Christian Schoenebeck wrote:
> > > On Wed, Jul 12, 2023 at 11:01:53AM +0200, Christian Schoenebeck wrote:
> > > > > TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
> > > > > "rootflags=ignoreqv" fixes it as well, but it feels like a regression.
> > > > 
> > > > What's the precise set of 9p mount options on guest, which 9p server is used
> > > > and what are 9p server's options respectively?
> > > 
> > > The server is qemu, started like this:
> > > 
> > > qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'
> > > 
> > > This is qemu 8.0.2.
> > 
> > Does adding QEMU option "multidevs=remap" [1] resolve it?
> > 
> > [1] https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly
> 
> Unfortunately not:
> 
> + exec ./platform-v7a/sysroot-host/bin/qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device vi
> rtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file,multidevs=remap -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/
> root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'

I don't see any cache option here. I would expect at least cache=mmap here:

https://wiki.qemu.org/Documentation/9p_root_fs#Boot_the_9p_Root_FS_System

By default it's cache=none, which means no mmap() would be allowed either.

> [    0.000000] L2C: platform modifies aux control register: 0x02020000 -> 0x02420000
> [    0.000000] L2C: DT/platform modifies aux control register: 0x02020000 -> 0x02420000
> [    0.004806] smp_twd: clock not found -2
> [    1.069767] simple-pm-bus bus@40000000:motherboard-bus@40000000:iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
> [    1.085798] simple-pm-bus bus@40000000:motherboard-bus@40000000:iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
> [    1.153420] physmap-flash 48000000.psram: map_probe failed
> [    1.535013] 9pnet_virtio: no channels available for device root
> 
>  ____                        _                   _
> |  _ \ ___ _ __   __ _ _   _| |_ _ __ ___  _ __ (_)_  __
> | |_) / _ \ '_ \ / _` | | | | __| '__/ _ \| '_ \| \ \/ /
> |  __/  __/ | | | (_| | |_| | |_| | | (_) | | | | |>  <
> |_|   \___|_| |_|\__, |\__,_|\__|_|  \___/|_| |_|_/_/\_\
>                  |___/
> 
>  ____  _     _             _  ___ _
> |  _ \(_)___| |_ _ __ ___ | |/ (_) |_
> | | | | / __| __| '__/ _ \| ' /| | __|
> | |_| | \__ \ |_| | | (_) | . \| | |_
> |____/|_|___/\__|_|  \___/|_|\_\_|\__|
> 
> 
> OSELAS(R)-DistroKit-2019.12.0-00429-g57ffae760eb9 / v7a-2019.12.0-00429-g57ffae760eb9
> ptxdist-2023.07.1/2023-07-11T19:56:50+0200
> 
> DistroKit login: root
> root@DistroKit:~ mount / -o remount,rw
> root@DistroKit:~ ldconfig
> ldconfig: Cannot mmap file /lib/libgcc_s.so.1.
> 
> ldconfig: Cannot mmap file /lib/libstdc++.so.

Again, effective 9p mount options are always useful, e.g. `mount | grep 9p`

> 
> [...]
> 
> rsc
> 



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
       [not found]           ` <CAFkjPTmrbe9Djs_iYf=vu1xbTLhey0OmePR2Jw=C_dbQVYnAhQ@mail.gmail.com>
@ 2023-07-13 10:54             ` Christian Schoenebeck
  2023-07-14  5:36               ` Robert Schwebel
  0 siblings, 1 reply; 17+ messages in thread
From: Christian Schoenebeck @ 2023-07-13 10:54 UTC (permalink / raw)
  To: Eric Van Hensbergen
  Cc: Dominique Martinet, Latchesar Ionkov, Robert Schwebel, kernel,
	v9fs, groug

On Thursday, July 13, 2023 11:51:24 AM CEST Eric Van Hensbergen wrote:
> I’m traveling so haven’t been able to dig in, but something funny going
> on.  I would have expected ignoreqv to not have any impact in default cache
> mode (nocache) - so there’s a bug in there somewhere.  in this case its
> read mmap thats failing and that should work without cache=mmap (which is
> only required for writable mmaps) - I did do bunch of reorganization in
> this code path, I’ll try to eyeball the bug when I get back from travel but
> sounds like I also need to
> integrate a test path with a server that doesn't populate qid.vers so this
> case is covered by our regressions.

Or a runtime warning in general if server always just sends qid.version zero.

About QEMU: unless the (actually unrelated) option `multidevs=remap` option is
used, Rreaddir would return all directory entries with qid.version set to
zero. So that's a bug on QEMU side:

https://github.com/qemu/qemu/blob/71d72ececa086114df80fe4cc04d701b59002eb2/hw/9pfs/9p.c#L2534

I try to send a patch for this next week, such that qid.version is always
filled, no matter what the QEMU command line options are. In the meantime
please use `multidevs=remap` as workaround for testing.

- Christian

> On Thu, Jul 13, 2023 at 8:50 AM Christian Schoenebeck <
> linux_oss@crudebyte.com> wrote:
> 
> > On Thursday, July 13, 2023 12:15:29 AM CEST Robert Schwebel wrote:
> > > On Wed, Jul 12, 2023 at 05:57:40PM +0200, Christian Schoenebeck wrote:
> > > > > On Wed, Jul 12, 2023 at 11:01:53AM +0200, Christian Schoenebeck
> > wrote:
> > > > > > > TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
> > > > > > > "rootflags=ignoreqv" fixes it as well, but it feels like a
> > regression.
> > > > > >
> > > > > > What's the precise set of 9p mount options on guest, which 9p
> > server is used
> > > > > > and what are 9p server's options respectively?
> > > > >
> > > > > The server is qemu, started like this:
> > > > >
> > > > > qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M
> > -nographic -no-reboot -net nic,netdev=net1 -netdev
> > user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345
> > -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0
> > -device virtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi
> > -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file
> > -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel
> > ./platform-v7a/images/linuximage -dtb
> > ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/root
> > rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5
> > systemd.show_status=auto'
> > > > >
> > > > > This is qemu 8.0.2.
> > > >
> > > > Does adding QEMU option "multidevs=remap" [1] resolve it?
> > > >
> > > > [1]
> > https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly
> > >
> > > Unfortunately not:
> > >
> > > + exec ./platform-v7a/sysroot-host/bin/qemu-system-arm -machine
> > vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net
> > nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737
> > -:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object
> > rng-random,filename=/dev/random,id=rng0 -device vi
> > > rtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev
> > local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file,multidevs=remap
> > -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel
> > ./platform-v7a/images/linuximage -dtb
> > ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/
> > > root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200
> > loglevel=5 systemd.show_status=auto'
> >
> > I don't see any cache option here. I would expect at least cache=mmap here:
> >
> > https://wiki.qemu.org/Documentation/9p_root_fs#Boot_the_9p_Root_FS_System
> >
> > By default it's cache=none, which means no mmap() would be allowed either.
> >
> > > [    0.000000] L2C: platform modifies aux control register: 0x02020000
> > -> 0x02420000
> > > [    0.000000] L2C: DT/platform modifies aux control register:
> > 0x02020000 -> 0x02420000
> > > [    0.004806] smp_twd: clock not found -2
> > > [    1.069767] simple-pm-bus bus@40000000:motherboard-bus@40000000
> > :iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
> > > [    1.085798] simple-pm-bus bus@40000000:motherboard-bus@40000000
> > :iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
> > > [    1.153420] physmap-flash 48000000.psram: map_probe failed
> > > [    1.535013] 9pnet_virtio: no channels available for device root
> > >
> > >  ____                        _                   _
> > > |  _ \ ___ _ __   __ _ _   _| |_ _ __ ___  _ __ (_)_  __
> > > | |_) / _ \ '_ \ / _` | | | | __| '__/ _ \| '_ \| \ \/ /
> > > |  __/  __/ | | | (_| | |_| | |_| | | (_) | | | | |>  <
> > > |_|   \___|_| |_|\__, |\__,_|\__|_|  \___/|_| |_|_/_/\_\
> > >                  |___/
> > >
> > >  ____  _     _             _  ___ _
> > > |  _ \(_)___| |_ _ __ ___ | |/ (_) |_
> > > | | | | / __| __| '__/ _ \| ' /| | __|
> > > | |_| | \__ \ |_| | | (_) | . \| | |_
> > > |____/|_|___/\__|_|  \___/|_|\_\_|\__|
> > >
> > >
> > > OSELAS(R)-DistroKit-2019.12.0-00429-g57ffae760eb9 /
> > v7a-2019.12.0-00429-g57ffae760eb9
> > > ptxdist-2023.07.1/2023-07-11T19:56:50+0200
> > >
> > > DistroKit login: root
> > > root@DistroKit:~ mount / -o remount,rw
> > > root@DistroKit:~ ldconfig
> > > ldconfig: Cannot mmap file /lib/libgcc_s.so.1.
> > >
> > > ldconfig: Cannot mmap file /lib/libstdc++.so.
> >
> > Again, effective 9p mount options are always useful, e.g. `mount | grep 9p`
> >
> > >
> > > [...]
> > >
> > > rsc
> > >
> >
> >
> >
> 





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-11 20:19 mmap on 9p broken in 6.4 Robert Schwebel
  2023-07-12  9:01 ` Christian Schoenebeck
@ 2023-07-13 11:22 ` Linux regression tracking #adding (Thorsten Leemhuis)
  2023-07-26 14:07   ` Linux regression tracking #update (Thorsten Leemhuis)
  1 sibling, 1 reply; 17+ messages in thread
From: Linux regression tracking #adding (Thorsten Leemhuis) @ 2023-07-13 11:22 UTC (permalink / raw)
  To: Robert Schwebel, Eric Van Hensbergen
  Cc: Latchesar Ionkov, Dominique Martinet, Christian Schoenebeck, v9fs,
	kernel, Linux kernel regressions list

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 11.07.23 22:19, Robert Schwebel wrote:
> 
> TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
> "rootflags=ignoreqv" fixes it as well, but it feels like a regression.
> [...]
> I could track down the breakage to
> 
>   1543b4c5071c54d76aad7a7a26a6e43082269b0c
> [...]

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced 1543b4c5071c54d76aad7a7a26a6e43082269b
#regzbot title 9p: mmap() seems to be broken with qemu as server
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-13  7:50         ` Christian Schoenebeck
       [not found]           ` <CAFkjPTmrbe9Djs_iYf=vu1xbTLhey0OmePR2Jw=C_dbQVYnAhQ@mail.gmail.com>
@ 2023-07-14  5:34           ` Robert Schwebel
  1 sibling, 0 replies; 17+ messages in thread
From: Robert Schwebel @ 2023-07-14  5:34 UTC (permalink / raw)
  To: Christian Schoenebeck
  Cc: Latchesar Ionkov, Dominique Martinet, Eric Van Hensbergen, v9fs,
	kernel

Christian,

On Thu, Jul 13, 2023 at 09:50:42AM +0200, Christian Schoenebeck wrote:
> > > Does adding QEMU option "multidevs=remap" [1] resolve it?
> > >
> > > [1] https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly
> >
> > Unfortunately not:
> >
> > + exec ./platform-v7a/sysroot-host/bin/qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,hostfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device vi
> > rtio-rng-device,rng=rng0 -device virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file,multidevs=remap -device virtio-9p-device,fsdev=rootfs,mount_tag=/dev/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/
> > root rootfstype=9p rootflags=trans=virtio console=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'
>
> I don't see any cache option here. I would expect at least cache=mmap here:
>
> https://wiki.qemu.org/Documentation/9p_root_fs#Boot_the_9p_Root_FS_System
>
> By default it's cache=none, which means no mmap() would be allowed either.

This does help:

----------8<----------
+ exec ./platform-v7a/sysroot-host/bin/qemu-system-arm -machine vexpress-a9 -cpu cortex-a9 -smp 4 -m 1024M -nographic -no-reboot -net nic,netdev=net1 -netdev user,id=net1,ho
stfwd=tcp:127.0.0.1:27737-:22,hostfwd=tcp:127.0.0.1:12345-:12345 -rtc base=2000-01-01 -object rng-random,filename=/dev/random,id=rng0 -device virtio-rng-device,rng=rng0 -dev
ice virtio-scsi-device,id=scsi -fsdev local,id=rootfs,path=./platform-v7a/root,security_model=mapped-file,multidevs=remap -device virtio-9p-device,fsdev=rootfs,mount_tag=/de
v/root -kernel ./platform-v7a/images/linuximage -dtb ./platform-v7a/images/vexpress-v2p-ca9.dtb -append 'root=/dev/root rootfstype=9p rootflags=trans=virtio,cache=mmap conso
le=ttyAMA0,115200 loglevel=5 systemd.show_status=auto'
[    0.000000] L2C: platform modifies aux control register: 0x02020000 -> 0x02420000
[    0.000000] L2C: DT/platform modifies aux control register: 0x02020000 -> 0x02420000
[    0.004881] smp_twd: clock not found -2
[    0.793281] simple-pm-bus bus@40000000:motherboard-bus@40000000:iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
[    0.809253] simple-pm-bus bus@40000000:motherboard-bus@40000000:iofpga@7,00000000: Failed to create device link (0x180) with dcc:tcrefclk
[    0.879343] physmap-flash 48000000.psram: map_probe failed
[    1.267846] 9pnet_virtio: no channels available for device root

 ____                        _                   _
|  _ \ ___ _ __   __ _ _   _| |_ _ __ ___  _ __ (_)_  __
| |_) / _ \ '_ \ / _` | | | | __| '__/ _ \| '_ \| \ \/ /
|  __/  __/ | | | (_| | |_| | |_| | | (_) | | | | |>  <
|_|   \___|_| |_|\__, |\__,_|\__|_|  \___/|_| |_|_/_/\_\
                 |___/

 ____  _     _             _  ___ _
|  _ \(_)___| |_ _ __ ___ | |/ (_) |_
| | | | / __| __| '__/ _ \| ' /| | __|
| |_| | \__ \ |_| | | (_) | . \| | |_
|____/|_|___/\__|_|  \___/|_|\_\_|\__|


OSELAS(R)-DistroKit-2019.12.0-00429-g57ffae760eb9 / v7a-2019.12.0-00429-g57ffae760eb9
ptxdist-2023.07.1/2023-07-11T19:56:50+0200

DistroKit login: root
root@DistroKit:~ mount / -o remount,rw
root@DistroKit:~ ldconfig
root@DistroKit:~
----------8<----------

However, the default behaviour has changed. With 6.3, mmap worked without
cache=mmap.

> Again, effective 9p mount options are always useful, e.g. `mount | grep 9p`

With the change:
/dev/root on / type 9p (rw,relatime,cache=mmap,access=client,trans=virtio)

Without the change:
/dev/root on / type 9p (rw,relatime,access=client,trans=virtio)

rsc
-- 
Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-13 10:54             ` Christian Schoenebeck
@ 2023-07-14  5:36               ` Robert Schwebel
  2023-07-16 14:54                 ` Eric Van Hensbergen
  0 siblings, 1 reply; 17+ messages in thread
From: Robert Schwebel @ 2023-07-14  5:36 UTC (permalink / raw)
  To: Christian Schoenebeck
  Cc: Eric Van Hensbergen, Dominique Martinet, Latchesar Ionkov, kernel,
	v9fs, groug

On Thu, Jul 13, 2023 at 12:54:44PM +0200, Christian Schoenebeck wrote:
> About QEMU: unless the (actually unrelated) option `multidevs=remap` option is
> used, Rreaddir would return all directory entries with qid.version set to
> zero. So that's a bug on QEMU side:
>
> https://github.com/qemu/qemu/blob/71d72ececa086114df80fe4cc04d701b59002eb2/hw/9pfs/9p.c#L2534
>
> I try to send a patch for this next week, such that qid.version is always
> filled, no matter what the QEMU command line options are. In the meantime
> please use `multidevs=remap` as workaround for testing.

As said above in this thread: adding multidevs=remap doesn't help.

rsc
-- 
Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-14  5:36               ` Robert Schwebel
@ 2023-07-16 14:54                 ` Eric Van Hensbergen
  2023-07-16 19:48                   ` Eric Van Hensbergen
  0 siblings, 1 reply; 17+ messages in thread
From: Eric Van Hensbergen @ 2023-07-16 14:54 UTC (permalink / raw)
  To: Robert Schwebel
  Cc: Christian Schoenebeck, Dominique Martinet, Latchesar Ionkov,
	kernel, v9fs, groug

Okay, back after holiday and able to look at this on a proper sized
screen.  Still a bit jet-lagged, and things don't seem quite
consistent in my head with the observed behavior and the code.  Your
trace of the original problem has the error coming through as this:

| mmap2(NULL, 862228, PROT_READ, MAP_SHARED, 4, 0) = -1 ENODEV (No such device)

which I would think comes from v9fs_file_mmap:510 where it returns
ENODEV if it detects a shared mmap without writecaches enabled.  But
this seems overly restrictive - we should only worry about the case
where we have a writable shared mmap and no appropriate caching and
the mmap is clearly only PROT_READ.  Furthermore, I'm a little unsure
of why we don't just fall through to generic_file_readonly_mmap when
sufficient cache levels aren't enabled as this is what 9P2000 (without
dotL) does anyways and what we used to do when we had a seperate
file_operations table for cached versus uncached.  That common code
checks if the mmap is writable shared and provides -EINVAL, so the
check is vestigial anyways and the error code inconsistent.  So, the
first step is to remove v9fs_file:509-510 and that should sort us out.
Seems like the bug was introduced while I removed the writeback_fid
stuff, and the check for VM_WRITE got dropped.  But, like I said, the
whole check was vestigial, so better to just dump it.

The next line (v9fs_file:511) invalidate_inode_pages2 -- also feels
unnecessary since we are in a code block that has already checked if
we have writeback caches - so not sure why we need this.  So maybe it
should go too?  I don't see it in the old code, so I think it got
added when I was adding lots of extra flushes when tracking down cache
inconsistencies.

BUT - this doesn't explain why ignoreqv changes any behavior because
at this level we don't see any checks of fid level modes which this
impacts -- so I'm still a bit confused and will continue to try and
understand why that would impact any behavior and so there must be
another bug in there somewhere.  It may be some inconsistency between
the session flags and the fid modes -- perhaps it would have been more
consistent to use a fid->mode derived from the struct file * versus
using toplevel session flags -- but again I don't see why that would
have changed any behavior here.

However, it does seem like that is probably what we should be doing
because if the underlying file shouldn't be cached (regardless of what
session flags would have wanted) then we shouldn't be allowing
writable mmap and be using direct operations.  What I'm trying to
think through in my head is whether or not there would be a case where
we might have a cache-able file, but then opened it with ODIRECT
before calling mmap() -- not sure why anyone would do that but seems
like a case we might need to guard against?

TLDR; there are two patches I've identified which I think make sense:
1) remove vestigial check for shared and just drop into generic_readonly_mmap
2) switch the check from being a session flag check to being a file mode check

#1 should be all that is required to fix the regression, but #2 seems
like would be good practice.
Meanwhile I'll continue to dig and try and understand why ignoreqv
seemed to change the behavior.

       -eric



On Fri, Jul 14, 2023 at 12:36 AM Robert Schwebel
<r.schwebel@pengutronix.de> wrote:
>
> On Thu, Jul 13, 2023 at 12:54:44PM +0200, Christian Schoenebeck wrote:
> > About QEMU: unless the (actually unrelated) option `multidevs=remap` option is
> > used, Rreaddir would return all directory entries with qid.version set to
> > zero. So that's a bug on QEMU side:
> >
> > https://github.com/qemu/qemu/blob/71d72ececa086114df80fe4cc04d701b59002eb2/hw/9pfs/9p.c#L2534
> >
> > I try to send a patch for this next week, such that qid.version is always
> > filled, no matter what the QEMU command line options are. In the meantime
> > please use `multidevs=remap` as workaround for testing.
>
> As said above in this thread: adding multidevs=remap doesn't help.
>
> rsc
> --
> Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
> Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-16 14:54                 ` Eric Van Hensbergen
@ 2023-07-16 19:48                   ` Eric Van Hensbergen
  2023-07-17 11:26                     ` Robert Schwebel
  0 siblings, 1 reply; 17+ messages in thread
From: Eric Van Hensbergen @ 2023-07-16 19:48 UTC (permalink / raw)
  To: Robert Schwebel
  Cc: Christian Schoenebeck, Dominique Martinet, Latchesar Ionkov,
	kernel, v9fs, groug

A quick update on this, I have a patch and it passes regressions but I
never really reproduced the problem.  I was seeing issues reproducing
on x86_64 and aarch64 and so was immediately suspicious noting that
the original report was from armv7 -- so I started looking at data
types and getting embarrassed by my own sloppiness but then also
realized we never upstreamed
https://lore.kernel.org/all/80bae984fd5ca49b691bb35f2fd8f345f8bb67f1.1682405206.git.christophe.jaillet@wanadoo.fr/
which is still sitting in fixes/next and fixes at least one of these
type issues.

I'm building an armv7 environment and kernel right now to make sure I
can reproduce the problem before I see if cleaning up the broader type
story fixes things.  But it does look like fid_addmode session
arguments are wrong and should both be unsigned int.  I do wonder if I
should be a bit more aggressive and take everything to u8/u32/u64 as
part of the process, but maybe that should be left for a different set
of cleanups so that we can focus code review on fixing the immediate
problem.  Again though it doesn't immediately appear obvious to me
that any of the type problems are root-cause in the mmap code path so
probably something still lurking.

As part of all this I'll also make sure we have a 32-bit arm
regression run in the test suite (maybe I should just base it off of
distrokit) and am looking at incorporating separate regression actions
for qemu roots and builds with all warnings enabled.

       -eric


On Sun, Jul 16, 2023 at 9:54 AM Eric Van Hensbergen <ericvh@kernel.org> wrote:
>
> Okay, back after holiday and able to look at this on a proper sized
> screen.  Still a bit jet-lagged, and things don't seem quite
> consistent in my head with the observed behavior and the code.  Your
> trace of the original problem has the error coming through as this:
>
> | mmap2(NULL, 862228, PROT_READ, MAP_SHARED, 4, 0) = -1 ENODEV (No such device)
>
> which I would think comes from v9fs_file_mmap:510 where it returns
> ENODEV if it detects a shared mmap without writecaches enabled.  But
> this seems overly restrictive - we should only worry about the case
> where we have a writable shared mmap and no appropriate caching and
> the mmap is clearly only PROT_READ.  Furthermore, I'm a little unsure
> of why we don't just fall through to generic_file_readonly_mmap when
> sufficient cache levels aren't enabled as this is what 9P2000 (without
> dotL) does anyways and what we used to do when we had a seperate
> file_operations table for cached versus uncached.  That common code
> checks if the mmap is writable shared and provides -EINVAL, so the
> check is vestigial anyways and the error code inconsistent.  So, the
> first step is to remove v9fs_file:509-510 and that should sort us out.
> Seems like the bug was introduced while I removed the writeback_fid
> stuff, and the check for VM_WRITE got dropped.  But, like I said, the
> whole check was vestigial, so better to just dump it.
>
> The next line (v9fs_file:511) invalidate_inode_pages2 -- also feels
> unnecessary since we are in a code block that has already checked if
> we have writeback caches - so not sure why we need this.  So maybe it
> should go too?  I don't see it in the old code, so I think it got
> added when I was adding lots of extra flushes when tracking down cache
> inconsistencies.
>
> BUT - this doesn't explain why ignoreqv changes any behavior because
> at this level we don't see any checks of fid level modes which this
> impacts -- so I'm still a bit confused and will continue to try and
> understand why that would impact any behavior and so there must be
> another bug in there somewhere.  It may be some inconsistency between
> the session flags and the fid modes -- perhaps it would have been more
> consistent to use a fid->mode derived from the struct file * versus
> using toplevel session flags -- but again I don't see why that would
> have changed any behavior here.
>
> However, it does seem like that is probably what we should be doing
> because if the underlying file shouldn't be cached (regardless of what
> session flags would have wanted) then we shouldn't be allowing
> writable mmap and be using direct operations.  What I'm trying to
> think through in my head is whether or not there would be a case where
> we might have a cache-able file, but then opened it with ODIRECT
> before calling mmap() -- not sure why anyone would do that but seems
> like a case we might need to guard against?
>
> TLDR; there are two patches I've identified which I think make sense:
> 1) remove vestigial check for shared and just drop into generic_readonly_mmap
> 2) switch the check from being a session flag check to being a file mode check
>
> #1 should be all that is required to fix the regression, but #2 seems
> like would be good practice.
> Meanwhile I'll continue to dig and try and understand why ignoreqv
> seemed to change the behavior.
>
>        -eric
>
>
>
> On Fri, Jul 14, 2023 at 12:36 AM Robert Schwebel
> <r.schwebel@pengutronix.de> wrote:
> >
> > On Thu, Jul 13, 2023 at 12:54:44PM +0200, Christian Schoenebeck wrote:
> > > About QEMU: unless the (actually unrelated) option `multidevs=remap` option is
> > > used, Rreaddir would return all directory entries with qid.version set to
> > > zero. So that's a bug on QEMU side:
> > >
> > > https://github.com/qemu/qemu/blob/71d72ececa086114df80fe4cc04d701b59002eb2/hw/9pfs/9p.c#L2534
> > >
> > > I try to send a patch for this next week, such that qid.version is always
> > > filled, no matter what the QEMU command line options are. In the meantime
> > > please use `multidevs=remap` as workaround for testing.
> >
> > As said above in this thread: adding multidevs=remap doesn't help.
> >
> > rsc
> > --
> > Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
> > Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
> > 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> > Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-16 19:48                   ` Eric Van Hensbergen
@ 2023-07-17 11:26                     ` Robert Schwebel
  2023-07-17 15:29                       ` Eric Van Hensbergen
  0 siblings, 1 reply; 17+ messages in thread
From: Robert Schwebel @ 2023-07-17 11:26 UTC (permalink / raw)
  To: Eric Van Hensbergen
  Cc: Christian Schoenebeck, Dominique Martinet, Latchesar Ionkov,
	kernel, v9fs, groug

Hi Eric,

On Sun, Jul 16, 2023 at 02:48:27PM -0500, Eric Van Hensbergen wrote:
> A quick update on this, I have a patch and it passes regressions

Thanks for taking care! If you send the patch(es), I can test them in my
setup.

rsc
-- 
Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-17 11:26                     ` Robert Schwebel
@ 2023-07-17 15:29                       ` Eric Van Hensbergen
  2023-07-17 18:11                         ` Eric Van Hensbergen
  0 siblings, 1 reply; 17+ messages in thread
From: Eric Van Hensbergen @ 2023-07-17 15:29 UTC (permalink / raw)
  To: Robert Schwebel
  Cc: Christian Schoenebeck, Dominique Martinet, Latchesar Ionkov,
	kernel, v9fs, groug

Thanks Robert, I was holding off on sending the patches out until I
could make sure they fixed the problem, but considering I've been
having trouble reproducing in my environment I'll send them out now.
They are also currently available on github on this branch if that
makes it any easier:
https://github.com/v9fs/linux/tree/b4/fixes-overly-restrictive-mmap

        -eric

On Mon, Jul 17, 2023 at 6:26 AM Robert Schwebel
<r.schwebel@pengutronix.de> wrote:
>
> Hi Eric,
>
> On Sun, Jul 16, 2023 at 02:48:27PM -0500, Eric Van Hensbergen wrote:
> > A quick update on this, I have a patch and it passes regressions
>
> Thanks for taking care! If you send the patch(es), I can test them in my
> setup.
>
> rsc
> --
> Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
> Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-17 15:29                       ` Eric Van Hensbergen
@ 2023-07-17 18:11                         ` Eric Van Hensbergen
  2023-07-17 18:59                           ` Eric Van Hensbergen
  0 siblings, 1 reply; 17+ messages in thread
From: Eric Van Hensbergen @ 2023-07-17 18:11 UTC (permalink / raw)
  To: Robert Schwebel
  Cc: Christian Schoenebeck, Dominique Martinet, Latchesar Ionkov,
	kernel, v9fs, groug

nevermind the bit about not being able to reproduce the problem, I
figured out what I was doing wrong there now and can reliably
reproduce the problem on v6.5-rc1 before applying any of my patches.
The patches do appear to fix the problem - but I'm still at a loss as
to why ignoreqv also seemed to solve the problem.

        -eric



On Mon, Jul 17, 2023 at 10:29 AM Eric Van Hensbergen <ericvh@kernel.org> wrote:
>
> Thanks Robert, I was holding off on sending the patches out until I
> could make sure they fixed the problem, but considering I've been
> having trouble reproducing in my environment I'll send them out now.
> They are also currently available on github on this branch if that
> makes it any easier:
> https://github.com/v9fs/linux/tree/b4/fixes-overly-restrictive-mmap
>
>         -eric
>
> On Mon, Jul 17, 2023 at 6:26 AM Robert Schwebel
> <r.schwebel@pengutronix.de> wrote:
> >
> > Hi Eric,
> >
> > On Sun, Jul 16, 2023 at 02:48:27PM -0500, Eric Van Hensbergen wrote:
> > > A quick update on this, I have a patch and it passes regressions
> >
> > Thanks for taking care! If you send the patch(es), I can test them in my
> > setup.
> >
> > rsc
> > --
> > Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
> > Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
> > 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> > Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-17 18:11                         ` Eric Van Hensbergen
@ 2023-07-17 18:59                           ` Eric Van Hensbergen
  0 siblings, 0 replies; 17+ messages in thread
From: Eric Van Hensbergen @ 2023-07-17 18:59 UTC (permalink / raw)
  To: Robert Schwebel
  Cc: Christian Schoenebeck, Dominique Martinet, Latchesar Ionkov,
	kernel, v9fs, groug

forgot to add, while i was able to reproduce the problem I wasn't able
to reproduce ignoreqv solving it, so I'm going to try and stop
worrying about that bit.
I'm cleaning up the new regression test to add to my suite so
something like this doesn't creep in again.  If someone would kindly
review the patches I'll get them moved over into fixes/next so we can
try and get them upstream.

       -eric

On Mon, Jul 17, 2023 at 1:11 PM Eric Van Hensbergen <ericvh@kernel.org> wrote:
>
> nevermind the bit about not being able to reproduce the problem, I
> figured out what I was doing wrong there now and can reliably
> reproduce the problem on v6.5-rc1 before applying any of my patches.
> The patches do appear to fix the problem - but I'm still at a loss as
> to why ignoreqv also seemed to solve the problem.
>
>         -eric
>
>
>
> On Mon, Jul 17, 2023 at 10:29 AM Eric Van Hensbergen <ericvh@kernel.org> wrote:
> >
> > Thanks Robert, I was holding off on sending the patches out until I
> > could make sure they fixed the problem, but considering I've been
> > having trouble reproducing in my environment I'll send them out now.
> > They are also currently available on github on this branch if that
> > makes it any easier:
> > https://github.com/v9fs/linux/tree/b4/fixes-overly-restrictive-mmap
> >
> >         -eric
> >
> > On Mon, Jul 17, 2023 at 6:26 AM Robert Schwebel
> > <r.schwebel@pengutronix.de> wrote:
> > >
> > > Hi Eric,
> > >
> > > On Sun, Jul 16, 2023 at 02:48:27PM -0500, Eric Van Hensbergen wrote:
> > > > A quick update on this, I have a patch and it passes regressions
> > >
> > > Thanks for taking care! If you send the patch(es), I can test them in my
> > > setup.
> > >
> > > rsc
> > > --
> > > Pengutronix e.K.                           | Dipl.-Ing. Robert Schwebel  |
> > > Steuerwalder Str. 21                       | https://www.pengutronix.de/ |
> > > 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> > > Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-9    |

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: mmap on 9p broken in 6.4
  2023-07-13 11:22 ` Linux regression tracking #adding (Thorsten Leemhuis)
@ 2023-07-26 14:07   ` Linux regression tracking #update (Thorsten Leemhuis)
  0 siblings, 0 replies; 17+ messages in thread
From: Linux regression tracking #update (Thorsten Leemhuis) @ 2023-07-26 14:07 UTC (permalink / raw)
  To: Linux kernel regressions list; +Cc: v9fs

[TLDR: This mail in primarily relevant for Linux kernel regression
tracking. See link in footer if these mails annoy you.]

On 13.07.23 13:22, Linux regression tracking #adding (Thorsten Leemhuis)
wrote:

> On 11.07.23 22:19, Robert Schwebel wrote:
>>
>> TL;DR: mmap() seems to be broken on 9pfs on Linux 6.4. setting
>> "rootflags=ignoreqv" fixes it as well, but it feels like a regression.
>> [...]
>> I could track down the breakage to
>>
>>   1543b4c5071c54d76aad7a7a26a6e43082269b0c
>> [...]
> 
> Thanks for the report. To be sure the issue doesn't fall through the
> cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
> tracking bot:
> 
> #regzbot ^introduced 1543b4c5071c54d76aad7a7a26a6e43082269b
> #regzbot title 9p: mmap() seems to be broken with qemu as server
> #regzbot ignore-activity

#regzbot fix: fs/9p: remove unnecessary invalidate_inode_pages2
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2023-07-26 14:07 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-07-11 20:19 mmap on 9p broken in 6.4 Robert Schwebel
2023-07-12  9:01 ` Christian Schoenebeck
2023-07-12 14:14   ` Robert Schwebel
2023-07-12 15:57     ` Christian Schoenebeck
2023-07-12 22:15       ` Robert Schwebel
2023-07-13  7:50         ` Christian Schoenebeck
     [not found]           ` <CAFkjPTmrbe9Djs_iYf=vu1xbTLhey0OmePR2Jw=C_dbQVYnAhQ@mail.gmail.com>
2023-07-13 10:54             ` Christian Schoenebeck
2023-07-14  5:36               ` Robert Schwebel
2023-07-16 14:54                 ` Eric Van Hensbergen
2023-07-16 19:48                   ` Eric Van Hensbergen
2023-07-17 11:26                     ` Robert Schwebel
2023-07-17 15:29                       ` Eric Van Hensbergen
2023-07-17 18:11                         ` Eric Van Hensbergen
2023-07-17 18:59                           ` Eric Van Hensbergen
2023-07-14  5:34           ` Robert Schwebel
2023-07-13 11:22 ` Linux regression tracking #adding (Thorsten Leemhuis)
2023-07-26 14:07   ` Linux regression tracking #update (Thorsten Leemhuis)

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.