* KVM_COMPAT support on aarch64 @ 2019-11-21 20:16 Daniel Verkamp 2019-11-22 15:29 ` Marc Zyngier 0 siblings, 1 reply; 4+ messages in thread From: Daniel Verkamp @ 2019-11-21 20:16 UTC (permalink / raw) To: Marc Zyngier, kvmarm; +Cc: Dylan Reid Hi Marc, I noticed that in recent kernels (4.19 in our case), KVM no longer works for 32-bit processes running on arm64 machines. We (Crostini/crosvm, Linux VM support on Chromebooks) use this KVM_COMPAT on all arm64-based Chromebooks that support VMs, since our entire userspace (including crosvm) is 32-bit. I found this commit (KVM: arm64: Prevent KVM_COMPAT from being selected): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c Is there any particular reason the compat ioctl is disabled (e.g. actual bugs/breakage), or would it be possible to re-enable KVM_COMPAT for ARM64? In my brief testing with a 4.19-stable-based kernel on a new arm64 board, everything seems to work fine with the commit above reverted. Thanks, -- Daniel _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: KVM_COMPAT support on aarch64 2019-11-21 20:16 KVM_COMPAT support on aarch64 Daniel Verkamp @ 2019-11-22 15:29 ` Marc Zyngier 2019-11-22 17:50 ` Daniel Verkamp 0 siblings, 1 reply; 4+ messages in thread From: Marc Zyngier @ 2019-11-22 15:29 UTC (permalink / raw) To: Daniel Verkamp; +Cc: Dylan Reid, kvmarm Hi Daniel, [please use my kernel.org address, as I'm not employed by ARM anymore] On 2019-11-21 20:16, Daniel Verkamp wrote: > Hi Marc, > > I noticed that in recent kernels (4.19 in our case), KVM no longer > works for 32-bit processes running on arm64 machines. We > (Crostini/crosvm, Linux VM support on Chromebooks) use this > KVM_COMPAT > on all arm64-based Chromebooks that support VMs, since our entire > userspace (including crosvm) is 32-bit. > > I found this commit (KVM: arm64: Prevent KVM_COMPAT from being > selected): > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c > > Is there any particular reason the compat ioctl is disabled (e.g. > actual bugs/breakage), or would it be possible to re-enable > KVM_COMPAT > for ARM64? Well, it never worked the first place! :( The whole API is entirely different, and you should feel extremely lucky that it ever seemed to work! > In my brief testing with a 4.19-stable-based kernel on a new arm64 > board, everything seems to work fine with the commit above reverted. The key word here is *seems*. It may happen that the userspace ABI somehow align for some of the basic stuff, but it will fail for anything moderately serious (I really doubt QEMU would be able to run at all). I suppose to tried this with crosvm? 32 or 64bit guest? Thanks, M. -- Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: KVM_COMPAT support on aarch64 2019-11-22 15:29 ` Marc Zyngier @ 2019-11-22 17:50 ` Daniel Verkamp 2019-11-22 18:31 ` Marc Zyngier 0 siblings, 1 reply; 4+ messages in thread From: Daniel Verkamp @ 2019-11-22 17:50 UTC (permalink / raw) To: Marc Zyngier; +Cc: Dylan Reid, kvmarm On Fri, Nov 22, 2019 at 7:29 AM Marc Zyngier <maz@kernel.org> wrote: > > Hi Daniel, > > [please use my kernel.org address, as I'm not employed by ARM anymore] > > On 2019-11-21 20:16, Daniel Verkamp wrote: > > Hi Marc, > > > > I noticed that in recent kernels (4.19 in our case), KVM no longer > > works for 32-bit processes running on arm64 machines. We > > (Crostini/crosvm, Linux VM support on Chromebooks) use this > > KVM_COMPAT > > on all arm64-based Chromebooks that support VMs, since our entire > > userspace (including crosvm) is 32-bit. > > > > I found this commit (KVM: arm64: Prevent KVM_COMPAT from being > > selected): > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c > > > > Is there any particular reason the compat ioctl is disabled (e.g. > > actual bugs/breakage), or would it be possible to re-enable > > KVM_COMPAT > > for ARM64? > > Well, it never worked the first place! :( The whole API is entirely > different, > and you should feel extremely lucky that it ever seemed to work! I hadn't looked at this before, but it seems like the Rust bindings for KVM we are using are actually generated from the aarch64 C headers (e.g. struct kvm_regs contains sp_el1, etc. rather than svc_regs): https://chromium.googlesource.com/chromiumos/platform/crosvm/+/37bd738b783053c2e94b4c215c32496bae5081ed/kvm_sys/src/aarch64/bindings.rs So we are really passing the native 64-bit structs to the 32-bit compat ioctl, which is probably bogus. I can see how this is unlikely to work in a normal C program compiled against the 32-bit ARM uapi headers. > > In my brief testing with a 4.19-stable-based kernel on a new arm64 > > board, everything seems to work fine with the commit above reverted. > > The key word here is *seems*. It may happen that the userspace ABI > somehow align for some of the basic stuff, but it will fail for > anything moderately serious (I really doubt QEMU would be able to > run at all). > > I suppose to tried this with crosvm? 32 or 64bit guest? Yes, I am able to run the full Crostini environment using 32-bit crosvm on an aarch64 Chromebook kernel based on 4.19.84 stable. The guest kernel and userspace is 64 bit in this case (we don't support 32-bit guest). I also just tried a 32-bit build of kvmtool (lkvm), since we have a convenient ebuild to cross-compile it in the Chrome OS SDK, but I wasn't able to get it to boot (it fails at a call to KVM_GET_ONE_REG). I didn't attempt to debug it, but I'm assuming it's due to the ABI mismatch you mentioned. It sounds like disabling the compat ioctl is probably the right thing to do in this case; I'll have to see if we can build crosvm as a 64-bit executable instead in the future (not sure how easily this will be given the way the Chromium OS build system works). Thanks, -- Daniel _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: KVM_COMPAT support on aarch64 2019-11-22 17:50 ` Daniel Verkamp @ 2019-11-22 18:31 ` Marc Zyngier 0 siblings, 0 replies; 4+ messages in thread From: Marc Zyngier @ 2019-11-22 18:31 UTC (permalink / raw) To: Daniel Verkamp; +Cc: Dylan Reid, kvmarm On Fri, 22 Nov 2019 09:50:50 -0800 Daniel Verkamp <dverkamp@chromium.org> wrote: > On Fri, Nov 22, 2019 at 7:29 AM Marc Zyngier <maz@kernel.org> wrote: > > > > Hi Daniel, > > > > [please use my kernel.org address, as I'm not employed by ARM anymore] > > > > On 2019-11-21 20:16, Daniel Verkamp wrote: > > > Hi Marc, > > > > > > I noticed that in recent kernels (4.19 in our case), KVM no longer > > > works for 32-bit processes running on arm64 machines. We > > > (Crostini/crosvm, Linux VM support on Chromebooks) use this > > > KVM_COMPAT > > > on all arm64-based Chromebooks that support VMs, since our entire > > > userspace (including crosvm) is 32-bit. > > > > > > I found this commit (KVM: arm64: Prevent KVM_COMPAT from being > > > selected): > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c > > > > > > Is there any particular reason the compat ioctl is disabled (e.g. > > > actual bugs/breakage), or would it be possible to re-enable > > > KVM_COMPAT > > > for ARM64? > > > > Well, it never worked the first place! :( The whole API is entirely > > different, > > and you should feel extremely lucky that it ever seemed to work! > > I hadn't looked at this before, but it seems like the Rust bindings > for KVM we are using are actually generated from the aarch64 C headers > (e.g. struct kvm_regs contains sp_el1, etc. rather than svc_regs): > https://chromium.googlesource.com/chromiumos/platform/crosvm/+/37bd738b783053c2e94b4c215c32496bae5081ed/kvm_sys/src/aarch64/bindings.rs > So we are really passing the native 64-bit structs to the 32-bit > compat ioctl, which is probably bogus. Wow. It's amazing it did anything sensible! > I can see how this is unlikely to work in a normal C program compiled > against the 32-bit ARM uapi headers. Indeed. > > > In my brief testing with a 4.19-stable-based kernel on a new arm64 > > > board, everything seems to work fine with the commit above reverted. > > > > The key word here is *seems*. It may happen that the userspace ABI > > somehow align for some of the basic stuff, but it will fail for > > anything moderately serious (I really doubt QEMU would be able to > > run at all). > > > > I suppose to tried this with crosvm? 32 or 64bit guest? > > Yes, I am able to run the full Crostini environment using 32-bit > crosvm on an aarch64 Chromebook kernel based on 4.19.84 stable. The > guest kernel and userspace is 64 bit in this case (we don't support > 32-bit guest). > > I also just tried a 32-bit build of kvmtool (lkvm), since we have a > convenient ebuild to cross-compile it in the Chrome OS SDK, but I > wasn't able to get it to boot (it fails at a call to KVM_GET_ONE_REG). > I didn't attempt to debug it, but I'm assuming it's due to the ABI > mismatch you mentioned. That's what I was expecting indeed. The registers are identified differently to make my life easier (and at some point I considered writing a compat interface before making up my mind and canning the thing altogether). > It sounds like disabling the compat ioctl is probably the right thing > to do in this case; I'll have to see if we can build crosvm as a > 64-bit executable instead in the future (not sure how easily this will > be given the way the Chromium OS build system works). You may have to end-up with a static binary, which is not very appealing... M. -- Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-11-22 18:31 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-11-21 20:16 KVM_COMPAT support on aarch64 Daniel Verkamp 2019-11-22 15:29 ` Marc Zyngier 2019-11-22 17:50 ` Daniel Verkamp 2019-11-22 18:31 ` Marc Zyngier
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox