Re: KVM_COMPAT support on aarch64

[Marc Zyngier <maz@kernel.org>]

On Fri, 22 Nov 2019 09:50:50 -0800
Daniel Verkamp <dverkamp@chromium.org> wrote:

> On Fri, Nov 22, 2019 at 7:29 AM Marc Zyngier <maz@kernel.org> wrote:
> >
> > Hi Daniel,
> >
> > [please use my kernel.org address, as I'm not employed by ARM anymore]
> >
> > On 2019-11-21 20:16, Daniel Verkamp wrote:  
> > > Hi Marc,
> > >
> > > I noticed that in recent kernels (4.19 in our case), KVM no longer
> > > works for 32-bit processes running on arm64 machines. We
> > > (Crostini/crosvm, Linux VM support on Chromebooks) use this
> > > KVM_COMPAT
> > > on all arm64-based Chromebooks that support VMs, since our entire
> > > userspace (including crosvm) is 32-bit.
> > >
> > > I found this commit (KVM: arm64: Prevent KVM_COMPAT from being
> > > selected):
> > >
> > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c
> > >
> > > Is there any particular reason the compat ioctl is disabled (e.g.
> > > actual bugs/breakage), or would it be possible to re-enable
> > > KVM_COMPAT
> > > for ARM64?  
> >
> > Well, it never worked the first place! :( The whole API is entirely
> > different,
> > and you should feel extremely lucky that it ever seemed to work!  
> 
> I hadn't looked at this before, but it seems like the Rust bindings
> for KVM we are using are actually generated from the aarch64 C headers
> (e.g. struct kvm_regs contains sp_el1, etc. rather than svc_regs):
> https://chromium.googlesource.com/chromiumos/platform/crosvm/+/37bd738b783053c2e94b4c215c32496bae5081ed/kvm_sys/src/aarch64/bindings.rs
> So we are really passing the native 64-bit structs to the 32-bit
> compat ioctl, which is probably bogus.

Wow. It's amazing it did anything sensible!

> I can see how this is unlikely to work in a normal C program compiled
> against the 32-bit ARM uapi headers.

Indeed.

> > > In my brief testing with a 4.19-stable-based kernel on a new arm64
> > > board, everything seems to work fine with the commit above reverted.  
> >
> > The key word here is *seems*. It may happen that the userspace ABI
> > somehow align for some of the basic stuff, but it will fail for
> > anything moderately serious (I really doubt QEMU would be able to
> > run at all).
> >
> > I suppose to tried this with crosvm? 32 or 64bit guest?  
> 
> Yes, I am able to run the full Crostini environment using 32-bit
> crosvm on an aarch64 Chromebook kernel based on 4.19.84 stable.  The
> guest kernel and userspace is 64 bit in this case (we don't support
> 32-bit guest).
> 
> I also just tried a 32-bit build of kvmtool (lkvm), since we have a
> convenient ebuild to cross-compile it in the Chrome OS SDK, but I
> wasn't able to get it to boot (it fails at a call to KVM_GET_ONE_REG).
> I didn't attempt to debug it, but I'm assuming it's due to the ABI
> mismatch you mentioned.

That's what I was expecting indeed. The registers are identified
differently to make my life easier (and at some point I considered
writing a compat interface before making up my mind and canning the
thing altogether).

> It sounds like disabling the compat ioctl is probably the right thing
> to do in this case; I'll have to see if we can build crosvm as a
> 64-bit executable instead in the future (not sure how easily this will
> be given the way the Chromium OS build system works).

You may have to end-up with a static binary, which is not very
appealing...


	M.
-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm