* [Virtio-fs] Failure as Operation not permitted on aarch64 machine
@ 2019-07-19 22:50 Masayoshi Mizuma
2019-07-22 16:47 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 4+ messages in thread
From: Masayoshi Mizuma @ 2019-07-19 22:50 UTC (permalink / raw)
To: virtio-fs
Hello,
I would appreciate if you could help me to resolve following failure.
I tried to use virtio-fs on aarch64 [1], however, qemu [2] failed to boot
as Operation not permitted.
---
# ./virtiofsd -o vhost_user_socket=/tmp/vhostqemu -o source=/tmp/share -o cache=none &
# qemu-system-aarch64 -machine virt,accel=kvm,gic-version=3 -cpu host -smp 8 ... [3]
UEFI firmware starting.
kvm_set_phys_mem: error registering slot: Operation not permitted
---
>From the ftrace log, the error happened because kvm_set_user_memory_region()
returned as -1 (EPERM).
---
qemu-system-aar-28381 [003] .... 6683.601097: tracing_mark_write: kvm_set_user_memory Slot#3 flags=0x0 gpa=0x8000000000 size=0x40000000 ua=0xfffe0ba00000 ret=-1
---
kvm_set_user_memory_region() returned -1 because kvm_arch_prepare_memory_region()
in kernel returned as -EPERM.
---
int kvm_arch_prepare_memory_region(struct kvm *kvm,
struct kvm_memory_slot *memslot,
const struct kvm_userspace_memory_region *mem,
enum kvm_mr_change change)
{
...
if (writable && !(vma->vm_flags & VM_WRITE)) {
ret = -EPERM;
break;
}
---
Actually, the /proc/PID/maps showed ua=0xfffe0ba00000 didn't have PROT_WRITE.
---
...
014b0000-014f0000 rw-p 00000000 00:00 0
1c840000-1d470000 rw-p 00000000 00:00 0 [heap]
fffe0ba00000-fffe4ba00000 ---p 00000000 00:00 0 <===
fffe4ba00000-fffe4ba10000 ---p 00000000 00:00 0
fffe4bc00000-fffe4fc00000 rw-p 00000000 00:00 0
fffe4fc00000-fffe4fc10000 ---p 00000000 00:00 0
fffe4fe00000-fffe53e00000 rw-p 00000000 00:00 0
fffe53e00000-fffe53e10000 ---p 00000000 00:00 0
...
---
I'm not sure why ua=0xfffe0ba00000 didn't have PROT_WRITE because the memory
was allocated by qemu_anon_ram_alloc() and it should set PROT_READ | PROT_WRITE.
---
qemu-system-aar-28372 [022] .... 6674.795027: tracing_mark_write: qemu_anon_ram_alloc size 1073741824 ptr 0xfffe0ba00000
---
qemu boots successfully if I remove "-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs",
so I suppose the option is related to the failure...
[1]: host kernel is 5.2.0.
guest kernel is https://github.com/rhvgoyal/linux/, branch: virtio-fs-dev-5.1.
[2]: I got the qemu from:
https://gitlab.com/virtio-fs/qemu.git
[3]: Qemu option is:
$QEMU -machine virt,accel=kvm,gic-version=3 \
-cpu host \
-smp 8 \
-m 4G,maxmem=4G \
-object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on \
-numa node,memdev=mem \
-drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \
-drive file=$VARS,if=pflash,format=raw,unit=1 \
-chardev socket,id=char0,path=/tmp/vhostqemu \
-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \
-nographic \
-serial mon:stdio \
--trace events=/tmp/qemu-trace-events \
-drive if=virtio,file=/var/lib/libvirt/images/guest.qcow2
Thanks,
Masa
^ permalink raw reply [flat|nested] 4+ messages in thread* Re: [Virtio-fs] Failure as Operation not permitted on aarch64 machine 2019-07-19 22:50 [Virtio-fs] Failure as Operation not permitted on aarch64 machine Masayoshi Mizuma @ 2019-07-22 16:47 ` Dr. David Alan Gilbert 2019-07-25 22:40 ` Masayoshi Mizuma 0 siblings, 1 reply; 4+ messages in thread From: Dr. David Alan Gilbert @ 2019-07-22 16:47 UTC (permalink / raw) To: Masayoshi Mizuma; +Cc: virtio-fs * Masayoshi Mizuma (msys.mizuma@gmail.com) wrote: > Hello, Hi Masa, > I would appreciate if you could help me to resolve following failure. > > I tried to use virtio-fs on aarch64 [1], however, qemu [2] failed to boot > as Operation not permitted. > > --- > # ./virtiofsd -o vhost_user_socket=/tmp/vhostqemu -o source=/tmp/share -o cache=none & > > # qemu-system-aarch64 -machine virt,accel=kvm,gic-version=3 -cpu host -smp 8 ... [3] > > UEFI firmware starting. > kvm_set_phys_mem: error registering slot: Operation not permitted > --- > > >From the ftrace log, the error happened because kvm_set_user_memory_region() > returned as -1 (EPERM). > > --- > qemu-system-aar-28381 [003] .... 6683.601097: tracing_mark_write: kvm_set_user_memory Slot#3 flags=0x0 gpa=0x8000000000 size=0x40000000 ua=0xfffe0ba00000 ret=-1 > --- > > kvm_set_user_memory_region() returned -1 because kvm_arch_prepare_memory_region() > in kernel returned as -EPERM. Right, yes we know this is a problem on aarch64; for aarch we disable the DAX mode; if you check out our -dev branches and apply the following hack to the kernel: diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index 3f3c018571ee..60a9724f9c71 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -541,7 +541,7 @@ static int virtio_fs_setup_dax(struct virtio_device *vdev, struct virtio_fs *fs) (u8)VIRTIO_FS_SHMCAP_ID_CACHE); if (!have_cache) { dev_err(&vdev->dev, "%s: No cache capability\n", __func__); - return -ENXIO; + return 0; } else { dev_notice(&vdev->dev, "Cache len: 0x%llx @ 0x%llx\n", cache_reg.len, cache_reg.addr); and start the qemu with: -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=0 it should start up for you. > --- > int kvm_arch_prepare_memory_region(struct kvm *kvm, > struct kvm_memory_slot *memslot, > const struct kvm_userspace_memory_region *mem, > enum kvm_mr_change change) > { > ... > if (writable && !(vma->vm_flags & VM_WRITE)) { > ret = -EPERM; > break; > } > --- > > Actually, the /proc/PID/maps showed ua=0xfffe0ba00000 didn't have PROT_WRITE. > > --- > ... > 014b0000-014f0000 rw-p 00000000 00:00 0 > 1c840000-1d470000 rw-p 00000000 00:00 0 [heap] > fffe0ba00000-fffe4ba00000 ---p 00000000 00:00 0 <=== > fffe4ba00000-fffe4ba10000 ---p 00000000 00:00 0 > fffe4bc00000-fffe4fc00000 rw-p 00000000 00:00 0 > fffe4fc00000-fffe4fc10000 ---p 00000000 00:00 0 > fffe4fe00000-fffe53e00000 rw-p 00000000 00:00 0 > fffe53e00000-fffe53e10000 ---p 00000000 00:00 0 > ... > --- > > I'm not sure why ua=0xfffe0ba00000 didn't have PROT_WRITE because the memory > was allocated by qemu_anon_ram_alloc() and it should set PROT_READ | PROT_WRITE. Because we explicitly mprotect it later (and in current versions we allocate with PROT_NONE) - on x86 this works fine, but aarch doesn't like it; we've not quite figured the rules why yet. Dave > > --- > qemu-system-aar-28372 [022] .... 6674.795027: tracing_mark_write: qemu_anon_ram_alloc size 1073741824 ptr 0xfffe0ba00000 > --- > > qemu boots successfully if I remove "-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs", > so I suppose the option is related to the failure... > > > [1]: host kernel is 5.2.0. > guest kernel is https://github.com/rhvgoyal/linux/, branch: virtio-fs-dev-5.1. > > [2]: I got the qemu from: > https://gitlab.com/virtio-fs/qemu.git > > [3]: Qemu option is: > > $QEMU -machine virt,accel=kvm,gic-version=3 \ > -cpu host \ > -smp 8 \ > -m 4G,maxmem=4G \ > -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on \ > -numa node,memdev=mem \ > -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \ > -drive file=$VARS,if=pflash,format=raw,unit=1 \ > -chardev socket,id=char0,path=/tmp/vhostqemu \ > -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \ > -nographic \ > -serial mon:stdio \ > --trace events=/tmp/qemu-trace-events \ > -drive if=virtio,file=/var/lib/libvirt/images/guest.qcow2 > > Thanks, > Masa > > _______________________________________________ > Virtio-fs mailing list > Virtio-fs@redhat.com > https://www.redhat.com/mailman/listinfo/virtio-fs -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Virtio-fs] Failure as Operation not permitted on aarch64 machine 2019-07-22 16:47 ` Dr. David Alan Gilbert @ 2019-07-25 22:40 ` Masayoshi Mizuma 2019-07-26 8:07 ` Dr. David Alan Gilbert 0 siblings, 1 reply; 4+ messages in thread From: Masayoshi Mizuma @ 2019-07-25 22:40 UTC (permalink / raw) To: dgilbert; +Cc: virtio-fs Hi Dave, On 7/22/19 12:47 PM, Dr. David Alan Gilbert wrote: > * Masayoshi Mizuma (msys.mizuma@gmail.com) wrote: >> Hello, > > Hi Masa, > >> I would appreciate if you could help me to resolve following failure. >> >> I tried to use virtio-fs on aarch64 [1], however, qemu [2] failed to boot >> as Operation not permitted. >> >> --- >> # ./virtiofsd -o vhost_user_socket=/tmp/vhostqemu -o source=/tmp/share -o cache=none & >> >> # qemu-system-aarch64 -machine virt,accel=kvm,gic-version=3 -cpu host -smp 8 ... [3] >> >> UEFI firmware starting. >> kvm_set_phys_mem: error registering slot: Operation not permitted >> --- >> >> >From the ftrace log, the error happened because kvm_set_user_memory_region() >> returned as -1 (EPERM). >> >> --- >> qemu-system-aar-28381 [003] .... 6683.601097: tracing_mark_write: kvm_set_user_memory Slot#3 flags=0x0 gpa=0x8000000000 size=0x40000000 ua=0xfffe0ba00000 ret=-1 >> --- >> >> kvm_set_user_memory_region() returned -1 because kvm_arch_prepare_memory_region() >> in kernel returned as -EPERM. > > Right, yes we know this is a problem on aarch64; for aarch we disable > the DAX mode; if you check out our -dev branches and apply the > following hack to the kernel: > > > diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c > index 3f3c018571ee..60a9724f9c71 100644 > --- a/fs/fuse/virtio_fs.c > +++ b/fs/fuse/virtio_fs.c > @@ -541,7 +541,7 @@ static int virtio_fs_setup_dax(struct virtio_device *vdev, struct virtio_fs *fs) > (u8)VIRTIO_FS_SHMCAP_ID_CACHE); > if (!have_cache) { > dev_err(&vdev->dev, "%s: No cache capability\n", __func__); > - return -ENXIO; > + return 0; > } else { > dev_notice(&vdev->dev, "Cache len: 0x%llx @ 0x%llx\n", > cache_reg.len, cache_reg.addr); > > and start the qemu with: > -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=0 > > it should start up for you. Thanks and the cache-size=0 option works for me! qemu boots successfully and I can mount the myfs. Thanks! Masa > >> --- >> int kvm_arch_prepare_memory_region(struct kvm *kvm, >> struct kvm_memory_slot *memslot, >> const struct kvm_userspace_memory_region *mem, >> enum kvm_mr_change change) >> { >> ... >> if (writable && !(vma->vm_flags & VM_WRITE)) { >> ret = -EPERM; >> break; >> } >> --- >> >> Actually, the /proc/PID/maps showed ua=0xfffe0ba00000 didn't have PROT_WRITE. >> >> --- >> ... >> 014b0000-014f0000 rw-p 00000000 00:00 0 >> 1c840000-1d470000 rw-p 00000000 00:00 0 [heap] >> fffe0ba00000-fffe4ba00000 ---p 00000000 00:00 0 <=== >> fffe4ba00000-fffe4ba10000 ---p 00000000 00:00 0 >> fffe4bc00000-fffe4fc00000 rw-p 00000000 00:00 0 >> fffe4fc00000-fffe4fc10000 ---p 00000000 00:00 0 >> fffe4fe00000-fffe53e00000 rw-p 00000000 00:00 0 >> fffe53e00000-fffe53e10000 ---p 00000000 00:00 0 >> ... >> --- >> >> I'm not sure why ua=0xfffe0ba00000 didn't have PROT_WRITE because the memory >> was allocated by qemu_anon_ram_alloc() and it should set PROT_READ | PROT_WRITE. > > Because we explicitly mprotect it later (and in current versions we > allocate with PROT_NONE) - on x86 this works fine, but aarch doesn't > like it; we've not quite figured the rules why yet. > > Dave > >> >> --- >> qemu-system-aar-28372 [022] .... 6674.795027: tracing_mark_write: qemu_anon_ram_alloc size 1073741824 ptr 0xfffe0ba00000 >> --- >> >> qemu boots successfully if I remove "-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs", >> so I suppose the option is related to the failure... >> >> >> [1]: host kernel is 5.2.0. >> guest kernel is https://github.com/rhvgoyal/linux/, branch: virtio-fs-dev-5.1. >> >> [2]: I got the qemu from: >> https://gitlab.com/virtio-fs/qemu.git >> >> [3]: Qemu option is: >> >> $QEMU -machine virt,accel=kvm,gic-version=3 \ >> -cpu host \ >> -smp 8 \ >> -m 4G,maxmem=4G \ >> -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on \ >> -numa node,memdev=mem \ >> -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \ >> -drive file=$VARS,if=pflash,format=raw,unit=1 \ >> -chardev socket,id=char0,path=/tmp/vhostqemu \ >> -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \ >> -nographic \ >> -serial mon:stdio \ >> --trace events=/tmp/qemu-trace-events \ >> -drive if=virtio,file=/var/lib/libvirt/images/guest.qcow2 >> >> Thanks, >> Masa >> >> _______________________________________________ >> Virtio-fs mailing list >> Virtio-fs@redhat.com >> https://www.redhat.com/mailman/listinfo/virtio-fs > -- > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Virtio-fs] Failure as Operation not permitted on aarch64 machine 2019-07-25 22:40 ` Masayoshi Mizuma @ 2019-07-26 8:07 ` Dr. David Alan Gilbert 0 siblings, 0 replies; 4+ messages in thread From: Dr. David Alan Gilbert @ 2019-07-26 8:07 UTC (permalink / raw) To: Masayoshi Mizuma; +Cc: virtio-fs * Masayoshi Mizuma (msys.mizuma@gmail.com) wrote: > Hi Dave, > > On 7/22/19 12:47 PM, Dr. David Alan Gilbert wrote: > > * Masayoshi Mizuma (msys.mizuma@gmail.com) wrote: > >> Hello, > > > > Hi Masa, > > > >> I would appreciate if you could help me to resolve following failure. > >> > >> I tried to use virtio-fs on aarch64 [1], however, qemu [2] failed to boot > >> as Operation not permitted. > >> > >> --- > >> # ./virtiofsd -o vhost_user_socket=/tmp/vhostqemu -o source=/tmp/share -o cache=none & > >> > >> # qemu-system-aarch64 -machine virt,accel=kvm,gic-version=3 -cpu host -smp 8 ... [3] > >> > >> UEFI firmware starting. > >> kvm_set_phys_mem: error registering slot: Operation not permitted > >> --- > >> > >> >From the ftrace log, the error happened because kvm_set_user_memory_region() > >> returned as -1 (EPERM). > >> > >> --- > >> qemu-system-aar-28381 [003] .... 6683.601097: tracing_mark_write: kvm_set_user_memory Slot#3 flags=0x0 gpa=0x8000000000 size=0x40000000 ua=0xfffe0ba00000 ret=-1 > >> --- > >> > >> kvm_set_user_memory_region() returned -1 because kvm_arch_prepare_memory_region() > >> in kernel returned as -EPERM. > > > > Right, yes we know this is a problem on aarch64; for aarch we disable > > the DAX mode; if you check out our -dev branches and apply the > > following hack to the kernel: > > > > > > diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c > > index 3f3c018571ee..60a9724f9c71 100644 > > --- a/fs/fuse/virtio_fs.c > > +++ b/fs/fuse/virtio_fs.c > > @@ -541,7 +541,7 @@ static int virtio_fs_setup_dax(struct virtio_device *vdev, struct virtio_fs *fs) > > (u8)VIRTIO_FS_SHMCAP_ID_CACHE); > > if (!have_cache) { > > dev_err(&vdev->dev, "%s: No cache capability\n", __func__); > > - return -ENXIO; > > + return 0; > > } else { > > dev_notice(&vdev->dev, "Cache len: 0x%llx @ 0x%llx\n", > > cache_reg.len, cache_reg.addr); > > > > and start the qemu with: > > -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs,cache-size=0 > > > > it should start up for you. > > Thanks and the cache-size=0 option works for me! > qemu boots successfully and I can mount the myfs. That's great; our dev world has that fix in, so it'll be in our next release as well. Dave > Thanks! > Masa > > > > >> --- > >> int kvm_arch_prepare_memory_region(struct kvm *kvm, > >> struct kvm_memory_slot *memslot, > >> const struct kvm_userspace_memory_region *mem, > >> enum kvm_mr_change change) > >> { > >> ... > >> if (writable && !(vma->vm_flags & VM_WRITE)) { > >> ret = -EPERM; > >> break; > >> } > >> --- > >> > >> Actually, the /proc/PID/maps showed ua=0xfffe0ba00000 didn't have PROT_WRITE. > >> > >> --- > >> ... > >> 014b0000-014f0000 rw-p 00000000 00:00 0 > >> 1c840000-1d470000 rw-p 00000000 00:00 0 [heap] > >> fffe0ba00000-fffe4ba00000 ---p 00000000 00:00 0 <=== > >> fffe4ba00000-fffe4ba10000 ---p 00000000 00:00 0 > >> fffe4bc00000-fffe4fc00000 rw-p 00000000 00:00 0 > >> fffe4fc00000-fffe4fc10000 ---p 00000000 00:00 0 > >> fffe4fe00000-fffe53e00000 rw-p 00000000 00:00 0 > >> fffe53e00000-fffe53e10000 ---p 00000000 00:00 0 > >> ... > >> --- > >> > >> I'm not sure why ua=0xfffe0ba00000 didn't have PROT_WRITE because the memory > >> was allocated by qemu_anon_ram_alloc() and it should set PROT_READ | PROT_WRITE. > > > > Because we explicitly mprotect it later (and in current versions we > > allocate with PROT_NONE) - on x86 this works fine, but aarch doesn't > > like it; we've not quite figured the rules why yet. > > > > Dave > > > >> > >> --- > >> qemu-system-aar-28372 [022] .... 6674.795027: tracing_mark_write: qemu_anon_ram_alloc size 1073741824 ptr 0xfffe0ba00000 > >> --- > >> > >> qemu boots successfully if I remove "-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs", > >> so I suppose the option is related to the failure... > >> > >> > >> [1]: host kernel is 5.2.0. > >> guest kernel is https://github.com/rhvgoyal/linux/, branch: virtio-fs-dev-5.1. > >> > >> [2]: I got the qemu from: > >> https://gitlab.com/virtio-fs/qemu.git > >> > >> [3]: Qemu option is: > >> > >> $QEMU -machine virt,accel=kvm,gic-version=3 \ > >> -cpu host \ > >> -smp 8 \ > >> -m 4G,maxmem=4G \ > >> -object memory-backend-file,id=mem,size=4G,mem-path=/dev/shm,share=on \ > >> -numa node,memdev=mem \ > >> -drive file=/usr/share/AAVMF/AAVMF_CODE.fd,if=pflash,format=raw,unit=0,readonly=on \ > >> -drive file=$VARS,if=pflash,format=raw,unit=1 \ > >> -chardev socket,id=char0,path=/tmp/vhostqemu \ > >> -device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \ > >> -nographic \ > >> -serial mon:stdio \ > >> --trace events=/tmp/qemu-trace-events \ > >> -drive if=virtio,file=/var/lib/libvirt/images/guest.qcow2 > >> > >> Thanks, > >> Masa > >> > >> _______________________________________________ > >> Virtio-fs mailing list > >> Virtio-fs@redhat.com > >> https://www.redhat.com/mailman/listinfo/virtio-fs > > -- > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-07-26 8:07 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-07-19 22:50 [Virtio-fs] Failure as Operation not permitted on aarch64 machine Masayoshi Mizuma 2019-07-22 16:47 ` Dr. David Alan Gilbert 2019-07-25 22:40 ` Masayoshi Mizuma 2019-07-26 8:07 ` Dr. David Alan Gilbert
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.