Re: QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

Re: QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Richard Yao]

[-- Attachment #1.1: Type: text/plain, Size: 4430 bytes --]

I figured out the problem. There is zerocopy IO is being done via DMA to
a buffer allocated with valloc(). Right now, I am running a hack-fix
locally so I can get some other stuff done first. I will propose a
proper fix to the list in a few days.

On 11/25/2013 10:49 AM, Richard Yao wrote:
> I booted a Gentoo Linux installation in QEMU with a 9P rootfs as follows:
> 
> sudo qemu-kvm -cpu host -m 1024 -kernel
> /mnt/test/usr/src/linux-3.13-rc1/arch/x86/boot/bzImage -append
> 'root=/dev/root rootfstype=9p rootflags=trans=virtio,version=9p2000.L ro
> console=ttyS0' -serial stdio -fsdev
> local,id=root,path=/mnt/test,security_model=none -device
> virtio-9p-pci,fsdev=root,mount_tag=/dev/root
> 
> The system boots fine, but attempting to load any module will fail:
> 
> localhost ~ # modprobe crc32
> qemu-system-x86_64: virtio: trying to map MMIO memory
> 
> The behavior is consistent no matter what combination of things that I
> try. So far, I have tried Linux 3.10.7-gentoo (Gentoo patchset) and
> Linux 3.13-rc1. I have tried QEMU 1.4.2, QEMU 1.6.1 and QEMU HEAD. I
> have also tried booting without KVM, but the behavior is the same:
> 
> sudo qemu-kvm --no-kvm -m 1024 -kernel
> /mnt/test/usr/src/linux-3.13-rc1/arch/x86/boot/bzImage -append
> 'root=/dev/root rootfstype=9p rootflags=trans=virtio,version=9p2000.L ro
> console=ttyS0' -serial stdio -fsdev
> local,id=root,path=/mnt/test,security_model=none -device
> virtio-9p-pci,fsdev=root,mount_tag=/dev/root
> 
> Here is a backtrace of QEMU itself that I obtained using gdb:
> 
> Breakpoint 1, virtqueue_map_sg (sg=0x7f695b797b98, addr=0x7f695b793b98,
> num_sg=3, is_write=1) at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/hw/virtio/virtio.c:434
> 434                 error_report("virtio: trying to map MMIO memory");
> (gdb) bt
> #0  virtqueue_map_sg (sg=0x7f695b797b98, addr=0x7f695b793b98, num_sg=3,
> is_write=1) at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/hw/virtio/virtio.c:434
> #1  0x00000000006eb666 in virtqueue_pop (vq=0x1b23740,
> elem=0x7f695b793b88) at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/hw/virtio/virtio.c:500
> #2  0x00000000004a74ee in handle_9p_output (vdev=0x7f695b1fd910,
> vq=0x1b23740) at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/hw/9pfs/virtio-9p.c:3254
> #3  0x00000000006ec4b5 in virtio_queue_notify_vq (vq=0x1b23740) at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/hw/virtio/virtio.c:720
> #4  0x00000000006edf65 in virtio_queue_host_notifier_read (n=0x1b23790)
> at /usr/src/debug/app-emulation/qemu-9999/qemu-9999/hw/virtio/virtio.c:1116
> #5  0x00000000005d3c9c in qemu_iohandler_poll (pollfds=0x1aae200, ret=1)
> at /usr/src/debug/app-emulation/qemu-9999/qemu-9999/iohandler.c:143
> #6  0x00000000005d4702 in main_loop_wait (nonblocking=0) at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/main-loop.c:484
> #7  0x000000000066c583 in main_loop () at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/vl.c:2014
> #8  0x00000000006730d1 in main (argc=17, argv=0x7fffa93167c8,
> envp=0x7fffa9316858) at
> /usr/src/debug/app-emulation/qemu-9999/qemu-9999/vl.c:4366
> 
> I spoke with Alexander Graf about this in IRC. He suggested that the
> guest is passing QEMU a bad address and gave me a small patch to use to
> get the VM to stop when this happens so that I could attach gdb and poke
> around. Sadly, I was on my way out to dinner when he gave that to me and
> I subsequently lost the patch to the Debian's paste bin's garbage
> collection. I have tried debugging without it by booting qemu with `-s
> -S`, setting break points for the kernel module initialization routines
> and running the system, but the breakpoints are not triggering.
> 
> If I could get another copy of the patch that Alexander gave me, I would
> be able to move forward. Unfortunately, he is not on IRC right now. I
> cannot debug any further with my current knowledge, so I am sending what
> I know so far to the relevant mailing lists. It would be greatly
> appreciated if someone would point me in the right direction (or even
> better, send me a patch to fix what is wrong).
> 
> P.S. As a side note, there appears to be regression between 3.10.7 and
> 3.13-rc1. In specific, the rootfs will fail to mount unless I use
> mount_tag=/dev/root and pass root=/dev/root. With 3.10.7, I could use an
> arbitrary name.
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Christopher Covington]

Hi Richard,

On 11/25/2013 04:50 PM, Richard Yao wrote:
> I figured out the problem. There is zerocopy IO is being done via DMA to
> a buffer allocated with valloc(). Right now, I am running a hack-fix
> locally so I can get some other stuff done first. I will propose a
> proper fix to the list in a few days.

I've also encountered this issue on a non-QEMU simulator and have been
carrying a disable-zero-copy hack for a few months. Let me know if there's
anything I can help with.

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Richard Yao]

[-- Attachment #1.1.1: Type: text/plain, Size: 1607 bytes --]

Christopher,

It sounds like you disabled zero-copy entirely, which is not necessary.
As far as I recall, loading kernel modules is the only case in which
valloc() allocated buffers are used. In the worst case, we only need to
disable zero-copy on such buffers. I have been using a small patch to do
precisely that since yesterday. I have attached it to this email since
it sounds like the first version might be helpful to others while I take
the time to explore a few loose ends.

That being said, I would like to investigate a couple of things before I
send either this patch or some variant of it to the appropriate
subsystem maintainer. First, I need to review the valloc() routines to
ensure that range checking against [VMALLOC_START, VMALLOC_END) is the
correct way to identify valloc() generated buffers. Second, I want to
explore the feasibility of a suggestion by Alexander Graf to instead
rework the zero-copy to properly handle valloc() allocated buffers.

Yours truly,
Richard Yao

On 11/26/2013 10:16 AM, Christopher Covington wrote:
> Hi Richard,
> 
> On 11/25/2013 04:50 PM, Richard Yao wrote:
>> I figured out the problem. There is zerocopy IO is being done via DMA to
>> a buffer allocated with valloc(). Right now, I am running a hack-fix
>> locally so I can get some other stuff done first. I will propose a
>> proper fix to the list in a few days.
> 
> I've also encountered this issue on a non-QEMU simulator and have been
> carrying a disable-zero-copy hack for a few months. Let me know if there's
> anything I can help with.
> 
> Christopher
> 

[-- Attachment #1.1.2: patch --]
[-- Type: text/plain, Size: 556 bytes --]

diff --git a/net/9p/client.c b/net/9p/client.c
index ee8fd6b..0adfcf5 100644
--- a/net/9p/client.c
+++ b/net/9p/client.c
@@ -1557,7 +1557,9 @@ p9_client_read(struct p9_fid *fid, char *data, char __user *udata, u64 offset,
 		rsize = count;
 
 	/* Don't bother zerocopy for small IO (< 1024) */
-	if (clnt->trans_mod->zc_request && rsize > 1024) {
+	if (clnt->trans_mod->zc_request && rsize > 1024 &&
+		!(udata >= (char __user *)VMALLOC_START &&
+			udata < (char __user *)VMALLOC_END)) {
 		char *indata;
 		if (data) {
 			kernel_buf = 1;

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Richard Yao]

[-- Attachment #1.1: Type: text/plain, Size: 2077 bytes --]

I have this bad habit of not reviewing emails until after I send them.
Anyway, Chris, thanks for your offer of help, but I can handle this on
my own. The previous email was mostly to give you an early version of
the patch and let you know what I plan to do to improve upon it before I
propose some version of this patch to the appropriate subsystem
maintainer(s).

On 11/26/2013 10:38 AM, Richard Yao wrote:
> Christopher,
> 
> It sounds like you disabled zero-copy entirely, which is not necessary.
> As far as I recall, loading kernel modules is the only case in which
> valloc() allocated buffers are used. In the worst case, we only need to
> disable zero-copy on such buffers. I have been using a small patch to do
> precisely that since yesterday. I have attached it to this email since
> it sounds like the first version might be helpful to others while I take
> the time to explore a few loose ends.
> 
> That being said, I would like to investigate a couple of things before I
> send either this patch or some variant of it to the appropriate
> subsystem maintainer. First, I need to review the valloc() routines to
> ensure that range checking against [VMALLOC_START, VMALLOC_END) is the
> correct way to identify valloc() generated buffers. Second, I want to
> explore the feasibility of a suggestion by Alexander Graf to instead
> rework the zero-copy to properly handle valloc() allocated buffers.
> Yours truly,
> Richard Yao
> 
> On 11/26/2013 10:16 AM, Christopher Covington wrote:
>> Hi Richard,
>>
>> On 11/25/2013 04:50 PM, Richard Yao wrote:
>>> I figured out the problem. There is zerocopy IO is being done via DMA to
>>> a buffer allocated with valloc(). Right now, I am running a hack-fix
>>> locally so I can get some other stuff done first. I will propose a
>>> proper fix to the list in a few days.
>>
>> I've also encountered this issue on a non-QEMU simulator and have been
>> carrying a disable-zero-copy hack for a few months. Let me know if there's
>> anything I can help with.
>>
>> Christopher
>>



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 901 bytes --]

[-- Attachment #2: Type: text/plain, Size: 183 bytes --]

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [Qemu-devel] QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Christopher Covington]

Hi Richard,

On 11/25/2013 10:49 AM, Richard Yao wrote:
> I booted a Gentoo Linux installation in QEMU with a 9P rootfs as follows:
> 
> sudo qemu-kvm -cpu host -m 1024 -kernel
> /mnt/test/usr/src/linux-3.13-rc1/arch/x86/boot/bzImage -append
> 'root=/dev/root rootfstype=9p rootflags=trans=virtio,version=9p2000.L ro
> console=ttyS0' -serial stdio -fsdev
> local,id=root,path=/mnt/test,security_model=none -device
> virtio-9p-pci,fsdev=root,mount_tag=/dev/root
> 
> The system boots fine, but attempting to load any module will fail:
> 
> localhost ~ # modprobe crc32
> qemu-system-x86_64: virtio: trying to map MMIO memory
> 
> The behavior is consistent no matter what combination of things that I
> try. So far, I have tried Linux 3.10.7-gentoo (Gentoo patchset) and
> Linux 3.13-rc1. I have tried QEMU 1.4.2, QEMU 1.6.1 and QEMU HEAD. I
> have also tried booting without KVM, but the behavior is the same:
> 
> sudo qemu-kvm --no-kvm -m 1024 -kernel
> /mnt/test/usr/src/linux-3.13-rc1/arch/x86/boot/bzImage -append
> 'root=/dev/root rootfstype=9p rootflags=trans=virtio,version=9p2000.L ro
> console=ttyS0' -serial stdio -fsdev
> local,id=root,path=/mnt/test,security_model=none -device
> virtio-9p-pci,fsdev=root,mount_tag=/dev/root

Have you used this setup recently?

With my 3.15.0+ kernel, qemu-system-x86_64 substituted for qemu-kvm, and the
path changed from your arguments I get:

9pnet_virtio: no channels available
VFS: Cannot open root device "root" or unknown-block(0,0): error -2
Please append a correct "root=" boot option; here are the available partitions:
0b00         1048575 sr0  driver: sr
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014

Thanks,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [Qemu-devel] QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Richard Yao]

I use it all the time, but I have not updated to a more recent kernel yet. I am at LinuxCon, so I am unlikely to try it until I get home.

> On Aug 21, 2014, at 2:50 PM, Christopher Covington <cov@codeaurora.org> wrote:
> 
> Hi Richard,
> 
>> On 11/25/2013 10:49 AM, Richard Yao wrote:
>> I booted a Gentoo Linux installation in QEMU with a 9P rootfs as follows:
>> 
>> sudo qemu-kvm -cpu host -m 1024 -kernel
>> /mnt/test/usr/src/linux-3.13-rc1/arch/x86/boot/bzImage -append
>> 'root=/dev/root rootfstype=9p rootflags=trans=virtio,version=9p2000.L ro
>> console=ttyS0' -serial stdio -fsdev
>> local,id=root,path=/mnt/test,security_model=none -device
>> virtio-9p-pci,fsdev=root,mount_tag=/dev/root
>> 
>> The system boots fine, but attempting to load any module will fail:
>> 
>> localhost ~ # modprobe crc32
>> qemu-system-x86_64: virtio: trying to map MMIO memory
>> 
>> The behavior is consistent no matter what combination of things that I
>> try. So far, I have tried Linux 3.10.7-gentoo (Gentoo patchset) and
>> Linux 3.13-rc1. I have tried QEMU 1.4.2, QEMU 1.6.1 and QEMU HEAD. I
>> have also tried booting without KVM, but the behavior is the same:
>> 
>> sudo qemu-kvm --no-kvm -m 1024 -kernel
>> /mnt/test/usr/src/linux-3.13-rc1/arch/x86/boot/bzImage -append
>> 'root=/dev/root rootfstype=9p rootflags=trans=virtio,version=9p2000.L ro
>> console=ttyS0' -serial stdio -fsdev
>> local,id=root,path=/mnt/test,security_model=none -device
>> virtio-9p-pci,fsdev=root,mount_tag=/dev/root
> 
> Have you used this setup recently?
> 
> With my 3.15.0+ kernel, qemu-system-x86_64 substituted for qemu-kvm, and the
> path changed from your arguments I get:
> 
> 9pnet_virtio: no channels available
> VFS: Cannot open root device "root" or unknown-block(0,0): error -2
> Please append a correct "root=" boot option; here are the available partitions:
> 0b00         1048575 sr0  driver: sr
> Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
> rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
> 
> Thanks,
> Christopher
> 
> -- 
> Employee of Qualcomm Innovation Center, Inc.
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> hosted by the Linux Foundation.

Re: [V9fs-developer] [Qemu-devel] QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Christopher Covington]

Hi Dominique,

On 08/22/2014 02:27 AM, Dominique Martinet wrote:
> Hi,
> 
> Christopher Covington wrote on Thu, Aug 21, 2014 at 03:50:58PM -0400:
>> With my 3.15.0+ kernel, qemu-system-x86_64 substituted for qemu-kvm, and the
>> path changed from your arguments I get:
>>
>> 9pnet_virtio: no channels available
>> VFS: Cannot open root device "root" or unknown-block(0,0): error -2
>> Please append a correct "root=" boot option; here are the available partitions:
>> 0b00         1048575 sr0  driver: sr
>> Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
>> rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
> 
> I'm pretty sure that's the same problem that's been reported a few times
> here, could you try again after cherry-picking this commit:
> 
> commit f15844e0777fec936f87a87f97394f83911dacd3
> Author: Dominique Martinet <dominique.martinet@cea.fr>
> Date:   Wed May 21 10:02:12 2014 +0200
> 
>     9P: fix return value in v9fs_fid_xattr_set
>     
>     v9fs_fid_xattr_set is supposed to return 0 on success.
>     
>     This corrects the behaviour introduced in commit
>     bdd5c28dcb8330b9074404cc92a0b83aae5606a
>     "9p: fix return value in case in v9fs_fid_xattr_set()"
>     
>     (The function returns a negative error on error, as expected)
>     
>     Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr>
>     Signed-off-by: Eric Van Hensbergen <ericvh@gmail.com>
> 
> 
> It has gotten in in v3.16-rc1, so your 3.15.0+ proably doesn't have it.

Thanks for the pointer to this patch. I think I started this kernel half way
through the 3.16 merge window. The last non-cherry-picked patch I have is:

commit 6d87c225f5d82d29243dc124f1ffcbb0e14ec358
Merge: 338c09a 22001f6
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date:   Thu Jun 12 23:06:23 2014 -0700

    Merge branch 'for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/sage/ceph-client

So I do have the change you pointed out. Nevertheless, I'll reconfirm with the
latest torvalds/master.

Thanks,
Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.

Re: [V9fs-developer] [Qemu-devel] QEMU dies on any attempt to load a Linux kernel module when using a 9P rootfs

[Christopher Covington]

Hi Dominique,

On 08/22/2014 08:49 AM, Dominique Martinet wrote:
> Hi,
> 
> Christopher Covington wrote on Fri, Aug 22, 2014 at 08:37:52AM -0400:
>> Thanks for the pointer to this patch. I think I started this kernel half way
>> through the 3.16 merge window. The last non-cherry-picked patch I have is:
>>
>> commit 6d87c225f5d82d29243dc124f1ffcbb0e14ec358
>>
>> So I do have the change you pointed out. Nevertheless, I'll reconfirm with the
>> latest torvalds/master.
> 
> Right, so you already do have it. I don't think there's been anything
> else 9P related recently, so it probably will not help much -- sorry for
> blindly assuming it would be this.
> 
> Actually reading the error message now, "no channels available" is
> printed when virtio doesn't find any match in device name so it would
> look like the mount name doesn't match maybe?
> 
> Could we get your exact qemu command line to confirm? (the name that
> needs to match is /dev/root here, used for booting as root= and for qemu
> as mount_tag)

I re-transposed Richard's command line to my environment and it worked. Thanks
for pointing me in that direction. For the record, I used:

$dir/bin/x86_64-linux-gnu/qemu-system-x86_64 \
  -m 1024 \
  -kernel $dir/boot/x86_64-linux-gnu/bzImage-x86_64 \
  -append 'root=/dev/root init=/sbin/x86_64-linux-gnu/init rootfstype=9p
rootflags=trans=virtio,version=9p2000.L ro console=ttyS0' \
  -serial stdio \
  -fsdev local,id=root,path=$dir,security_model=none \
  -device virtio-9p-pci,fsdev=root,mount_tag=/dev/root \
  -monitor none

$dir/bin/x86_64-linux-gnu/qemu-system-arm \
  -M virt \
  -cpu cortex-a15 \
  -m 1024 \
  -kernel $dir/boot/arm-linux-gnueabihf/Image \
  -append 'root=/dev/root init=/sbin/arm-linux-gnueabihf/init rootfstype=9p
rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
  -serial stdio \
  -fsdev local,id=root,path=$dir,security_model=none \
  -device virtio-9p-device,fsdev=root,mount_tag=/dev/root \
  -monitor none

$dir/bin/x86_64-linux-gnu/qemu-system-aarch64 \
  -M virt \
  -cpu cortex-a57 \
  -m 1024 \
  -kernel $dir/boot/aarch64-linux-gnu/Image \
  -append 'root=/dev/root init=/sbin/aarch64-linux-gnu/init rootfstype=9p
rootflags=trans=virtio,version=9p2000.L ro console=ttyAMA0' \
  -serial stdio \
  -fsdev local,id=root,path=$dir,security_model=none \
  -device virtio-9p-device,fsdev=root,mount_tag=/dev/root -monitor none

Christopher

-- 
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
hosted by the Linux Foundation.