KVM on ARM Cortex A53 in 32-bit Mode

KVM on ARM Cortex A53 in 32-bit Mode

[Yasutaka Tanaka]

[-- Attachment #1.1: Type: text/plain, Size: 1305 bytes --]

Hello KVM/ARM community,

I am now trying to execute KVM on Raspberry Pi 3 Model B (RPI3).
Does anyone know whether KVM/ARM can run with 32-bit mode on ARM Cortex A53?

The CPU of RPI3 is ARM Cortex A53 which can be executed in both 32-bit and
64-bit mode. I've managed to display /dev/kvm on 64-bit mode execution
(i.e. with a 64-bit linux kernel) on RPI3, but I failed to do so in 32-bit
mode.

In 32-bit mode, an KVM error messages says CPU not supported. I guess the
reason for the error message is KVM_ARM_TARGET_CORTEX_A53 is only returned
in /arch/arm64/kvm/guest.c not under/arch/arm/kvm/guest.c.

So I'm wondering whether it's possible or not. What I'd like to do is to
compare the execution speed of KVM between 32-bit and 64-bit mode. The
second plan is to compare between KVMs of 32-bit RPI2 and 64-bit RPI3 (I
have both), but since RPI2 spec is a little low and I'm worried that some
numeric comparisons become unfair.

(I was also notified by an KVM/ARM expert that since RPI doesn't have a GIC
and I need to apply some patches)

The 64-bit kernel I used is the official RPI3 one here:
https://github.com/raspberrypi/linux
I compiled after make bcmrpi3_defconfig config:
https://github.com/raspberrypi/linux/blob/rpi-4.9.y/arch/arm64/configs/bcmrpi3_defconfig

Best regards,
Yasutaka Tanaka

[-- Attachment #1.2: Type: text/html, Size: 4296 bytes --]

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM on ARM Cortex A53 in 32-bit Mode

[Christoffer Dall]

Hi Yasutaka,

On Thu, Mar 02, 2017 at 09:57:50AM -0500, Yasutaka Tanaka wrote:
> Hello KVM/ARM community,
> 
> I am now trying to execute KVM on Raspberry Pi 3 Model B (RPI3).
> Does anyone know whether KVM/ARM can run with 32-bit mode on ARM Cortex A53?

Theoretically, it should.

> 
> The CPU of RPI3 is ARM Cortex A53 which can be executed in both 32-bit and
> 64-bit mode. I've managed to display /dev/kvm on 64-bit mode execution
> (i.e. with a 64-bit linux kernel) on RPI3, but I failed to do so in 32-bit
> mode.
> 
> In 32-bit mode, an KVM error messages says CPU not supported. I guess the
> reason for the error message is KVM_ARM_TARGET_CORTEX_A53 is only returned
> in /arch/arm64/kvm/guest.c not under/arch/arm/kvm/guest.c.
> 
> So I'm wondering whether it's possible or not. 

It should be.  You may just have to add the minimal missing definitions
for a 32-bit A53 CPU support.

> What I'd like to do is to
> compare the execution speed of KVM between 32-bit and 64-bit mode. The
> second plan is to compare between KVMs of 32-bit RPI2 and 64-bit RPI3 (I
> have both), but since RPI2 spec is a little low and I'm worried that some
> numeric comparisons become unfair.

I would be careful here, since the RPI3 doesn't have the GIC and you'll
end up going to user space a lot for interrupt handling.  Especially
look out for SMP workloads in the VM which may generate a lot of IPIs
which are really going to suffer from having to go to userspace on the
host.

> 
> (I was also notified by an KVM/ARM expert that since RPI doesn't have a GIC
> and I need to apply some patches)

Yes, you need the "Support userspace irqchip with arch timers" series
and corresponding QEMU patches:

http://www.spinics.net/lists/kvm/msg144685.html

Good luck.
-Christoffer

Re: KVM on ARM Cortex A53 in 32-bit Mode

[Yasutaka, T]

[-- Attachment #1.1: Type: text/plain, Size: 2869 bytes --]

Hi Christoffer,

>> It should be.  You may just have to add the minimal missing definitions for
a 32-bit A53 CPU support.

Okay -- I can spare my time this weekend so I'll try adding the definition
in source files.

>>  GIC
I applied the patch generated by git diff of irqs-to-user-v2 HEAD to the
Linux 4.10-rc5 commit
(where I believe KVM/ARM64 fixes started) to the official RPI OS kernel
(raspberrypi/linux repository).

Since current latest version of the RPI kernel seems to be forked from 4.9
(and I used 4.8.16 now, 1 version behind),
some hunks are failed and manually inserted into code bases.
It compiled successfully and I believe the next step is to find the
corresponding QEMU patches and apply them on your no-kvm-irqchip branch (or
latest QEMU 2.8.0?).

Yasutaka

2017-03-07 4:55 GMT-05:00 Christoffer Dall <christoffer.dall@linaro.org>:

> Hi Yasutaka,
>
> On Thu, Mar 02, 2017 at 09:57:50AM -0500, Yasutaka Tanaka wrote:
> > Hello KVM/ARM community,
> >
> > I am now trying to execute KVM on Raspberry Pi 3 Model B (RPI3).
> > Does anyone know whether KVM/ARM can run with 32-bit mode on ARM Cortex
> A53?
>
> Theoretically, it should.
>
> >
> > The CPU of RPI3 is ARM Cortex A53 which can be executed in both 32-bit
> and
> > 64-bit mode. I've managed to display /dev/kvm on 64-bit mode execution
> > (i.e. with a 64-bit linux kernel) on RPI3, but I failed to do so in
> 32-bit
> > mode.
> >
> > In 32-bit mode, an KVM error messages says CPU not supported. I guess the
> > reason for the error message is KVM_ARM_TARGET_CORTEX_A53 is only
> returned
> > in /arch/arm64/kvm/guest.c not under/arch/arm/kvm/guest.c.
> >
> > So I'm wondering whether it's possible or not.
>
> It should be.  You may just have to add the minimal missing definitions
> for a 32-bit A53 CPU support.
>
> > What I'd like to do is to
> > compare the execution speed of KVM between 32-bit and 64-bit mode. The
> > second plan is to compare between KVMs of 32-bit RPI2 and 64-bit RPI3 (I
> > have both), but since RPI2 spec is a little low and I'm worried that some
> > numeric comparisons become unfair.
>
> I would be careful here, since the RPI3 doesn't have the GIC and you'll
> end up going to user space a lot for interrupt handling.  Especially
> look out for SMP workloads in the VM which may generate a lot of IPIs
> which are really going to suffer from having to go to userspace on the
> host.
>
> >
> > (I was also notified by an KVM/ARM expert that since RPI doesn't have a
> GIC
> > and I need to apply some patches)
>
> Yes, you need the "Support userspace irqchip with arch timers" series
> and corresponding QEMU patches:
>
> http://www.spinics.net/lists/kvm/msg144685.html
>
> Good luck.
> -Christoffer
>



-- 
Yasutaka TANAKA
M.S. Candidate, Computer Science
Fu Foundation School of Engineering and Applied Science
Columbia University

y.tanaka@columbia.edu

[-- Attachment #1.2: Type: text/html, Size: 5062 bytes --]

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM on ARM Cortex A53 in 32-bit Mode

[Yasutaka, T]

[-- Attachment #1.1: Type: text/plain, Size: 4059 bytes --]

Hi Christopher and other KVM/ARM community members,

I tried the following QEMU and Kernel combinations on Raspberry Pi 3
(AArch64 mode) but failed:

Latest commit of Alexander Graf's no-kvm-irqchip QEMU branch:
https://github.com/agraf/qemu/tree/no-kvm-irqchip

raspberrypi/linux's rpi-4.10.y branch (forked from Linux 4.10-rc8?) latest
commit bb0ff9d059c67e1611c7422f7982a6a4876efe67
patched by $(git diff (linux 4.10-rc5 commit
7a308bb3016f57e5be11a677d15b821536419d36) (Christpher's irqs-to-user-v2
latest commit d0a7cc725535df1b9cc64b442d246c20a9edb904) )

and got the following error:

pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $
qemu-system-aarch64 --enable-kvm -M virt
"kvm" accelerator not found.
No accelerator found!

pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $ lkvm run test
  # lkvm run -k test -m 448 -c 4 --name guest-1605
  Error: Unsupported KVM extension detected: KVM_CAP_IRQCHIP
  Fatal: Failed to create virtual GIC


which seems strange for me because dmesg shows kernel is 4.10.2-v8+ and
Boot CPU is AArach64, all CPUs are initialized in HYP mode and /dev/kvm
exists. So I thought GIC emulation is also enabled, or should I need to
enable other config flags or do settings? Or, is this way of applying patch
wrong? Current config.gz is:
https://gist.github.com/caprice-j/d6e144cb8727df813a4fbe716beac3cb

My other concerns are no kvm-ok command is available and /proc/cpuinfo does
not show flags for virtualization support (I believe RPI3's cortex A53
(ARMv8) has builtin virt support and does not show the support flag
explicitly in /proc/cpuinfo ... but I'm unsure that is true)

pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $ cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 38.40
Features        : fp asimd evtstrm crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

I would be delighted if someone gives me suggestions.

Yasutaka

2017-03-07 4:55 GMT-05:00 Christoffer Dall <christoffer.dall@linaro.org>:

> Hi Yasutaka,
>
> On Thu, Mar 02, 2017 at 09:57:50AM -0500, Yasutaka Tanaka wrote:
> > Hello KVM/ARM community,
> >
> > I am now trying to execute KVM on Raspberry Pi 3 Model B (RPI3).
> > Does anyone know whether KVM/ARM can run with 32-bit mode on ARM Cortex
> A53?
>
> Theoretically, it should.
>
> >
> > The CPU of RPI3 is ARM Cortex A53 which can be executed in both 32-bit
> and
> > 64-bit mode. I've managed to display /dev/kvm on 64-bit mode execution
> > (i.e. with a 64-bit linux kernel) on RPI3, but I failed to do so in
> 32-bit
> > mode.
> >
> > In 32-bit mode, an KVM error messages says CPU not supported. I guess the
> > reason for the error message is KVM_ARM_TARGET_CORTEX_A53 is only
> returned
> > in /arch/arm64/kvm/guest.c not under/arch/arm/kvm/guest.c.
> >
> > So I'm wondering whether it's possible or not.
>
> It should be.  You may just have to add the minimal missing definitions
> for a 32-bit A53 CPU support.
>
> > What I'd like to do is to
> > compare the execution speed of KVM between 32-bit and 64-bit mode. The
> > second plan is to compare between KVMs of 32-bit RPI2 and 64-bit RPI3 (I
> > have both), but since RPI2 spec is a little low and I'm worried that some
> > numeric comparisons become unfair.
>
> I would be careful here, since the RPI3 doesn't have the GIC and you'll
> end up going to user space a lot for interrupt handling.  Especially
> look out for SMP workloads in the VM which may generate a lot of IPIs
> which are really going to suffer from having to go to userspace on the
> host.
>
> >
> > (I was also notified by an KVM/ARM expert that since RPI doesn't have a
> GIC
> > and I need to apply some patches)
>
> Yes, you need the "Support userspace irqchip with arch timers" series
> and corresponding QEMU patches:
>
> http://www.spinics.net/lists/kvm/msg144685.html
>
> Good luck.
> -Christoffer
>



-- 
Yasutaka TANAKA
M.S. Candidate, Computer Science
Fu Foundation School of Engineering and Applied Science
Columbia University

y.tanaka@columbia.edu

[-- Attachment #1.2: Type: text/html, Size: 5742 bytes --]

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM on ARM Cortex A53 in 32-bit Mode

[Christoffer Dall]

On Wed, Mar 15, 2017 at 07:50:36PM -0400, Yasutaka, T wrote:
> Hi Christopher and other KVM/ARM community members,
> 
> I tried the following QEMU and Kernel combinations on Raspberry Pi 3
> (AArch64 mode) but failed:
> 
> Latest commit of Alexander Graf's no-kvm-irqchip QEMU branch:
> https://github.com/agraf/qemu/tree/no-kvm-irqchip
> 
> raspberrypi/linux's rpi-4.10.y branch (forked from Linux 4.10-rc8?) latest
> commit bb0ff9d059c67e1611c7422f7982a6a4876efe67
> patched by $(git diff (linux 4.10-rc5 commit
> 7a308bb3016f57e5be11a677d15b821536419d36) (Christpher's irqs-to-user-v2
> latest commit d0a7cc725535df1b9cc64b442d246c20a9edb904) )
> 
> and got the following error:
> 
> pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $
> qemu-system-aarch64 --enable-kvm -M virt
> "kvm" accelerator not found.
> No accelerator found!

You'd need "-machine virt,kernel_irqchip=off" to use the userspace GIC
emulation.  Perhaps QEMU's error message for trying to initialize the
in-kernel GIC is just weird.

> 
> pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $ lkvm run test
>   # lkvm run -k test -m 448 -c 4 --name guest-1605
>   Error: Unsupported KVM extension detected: KVM_CAP_IRQCHIP
>   Fatal: Failed to create virtual GIC

I don't think kvmtool supports running on a system that doesn't have
in-kernel GIC support.

> 
> 
> which seems strange for me because dmesg shows kernel is 4.10.2-v8+ and
> Boot CPU is AArach64, all CPUs are initialized in HYP mode and /dev/kvm
> exists. So I thought GIC emulation is also enabled, or should I need to
> enable other config flags or do settings?

The RPI3 doesn't have a GIC, so you'll never get KVM emulation of the
GIC with that board.  Ever.  That's why you want the irqchip in
userspace patches.

> Or, is this way of applying patch
> wrong? Current config.gz is:
> https://gist.github.com/caprice-j/d6e144cb8727df813a4fbe716beac3cb
> 
> My other concerns are no kvm-ok command is available and /proc/cpuinfo does
> not show flags for virtualization support (I believe RPI3's cortex A53
> (ARMv8) has builtin virt support and does not show the support flag
> explicitly in /proc/cpuinfo ... but I'm unsure that is true)

If you have /dev/kvm then KVM as such is running on your system, no need
to worry further, just fix your setup to emulate the GIC in userspace.

Thanks,
-Christoffer

Re: KVM on ARM Cortex A53 in 32-bit Mode

[Yasutaka, T]

[-- Attachment #1.1: Type: text/plain, Size: 3979 bytes --]

2017-03-16 5:38 GMT-04:00 Christoffer Dall <cdall@linaro.org>:

> On Wed, Mar 15, 2017 at 07:50:36PM -0400, Yasutaka, T wrote:
> > Hi Christopher and other KVM/ARM community members,
> >
> > I tried the following QEMU and Kernel combinations on Raspberry Pi 3
> > (AArch64 mode) but failed:
> >
> > Latest commit of Alexander Graf's no-kvm-irqchip QEMU branch:
> > https://github.com/agraf/qemu/tree/no-kvm-irqchip
> >
> > raspberrypi/linux's rpi-4.10.y branch (forked from Linux 4.10-rc8?)
> latest
> > commit bb0ff9d059c67e1611c7422f7982a6a4876efe67
> > patched by $(git diff (linux 4.10-rc5 commit
> > 7a308bb3016f57e5be11a677d15b821536419d36) (Christpher's irqs-to-user-v2
> > latest commit d0a7cc725535df1b9cc64b442d246c20a9edb904) )
> >
> > and got the following error:
> >
> > pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $
> > qemu-system-aarch64 --enable-kvm -M virt
> > "kvm" accelerator not found.
> > No accelerator found!
>
> You'd need "-machine virt,kernel_irqchip=off" to use the userspace GIC
> emulation.  Perhaps QEMU's error message for trying to initialize the
> in-kernel GIC is just weird.
>

I executed with the flag but still got the same error message (accelerator
not found):

qemu-system-aarch64 --enable-kvm -cpu host -machine
type=virt,kernel_irqchip=off -nographic -smp 1 -m 256 -kernel
aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0"


And I found out that an article saying recent OpenSUSE distro officialy
supports KVM:
https://news.opensuse.org/2016/12/05/opensuse-leap-42-2-gets-64-bit-raspberry-image/

The developer of this distro is the same as the kernel_irqchip=off QEMU
patch author (Alexander Graf), and when I tried the above command on this
[1] distro KVM-QEMU actually worked ("worked" means it reaches to the
buildroot login prompt).

So I guess this distro applied your patches (it proves KVM/QEMU on RPI3 is
actually feasible) and the central reason that my failure of KVM on
Raspibian OS would be just a kernel config problem.

I set my .config file by make menuconfig, but it might need some other
(potentially non-GUI?) configurations. My current config is:
https://gist.github.com/caprice-j/836e3c7627f65c0f3a79e56a8d75368e

I thought I enabled necessary KVM-related flags but I might miss something.

[1]
http://download.opensuse.org/ports/aarch64/distribution/leap/42.2/appliances/openSUSE-Leap42.2-ARM-X11-raspberrypi3.aarch64.raw.xz


> >
> > pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $ lkvm run test
> >   # lkvm run -k test -m 448 -c 4 --name guest-1605
> >   Error: Unsupported KVM extension detected: KVM_CAP_IRQCHIP
> >   Fatal: Failed to create virtual GIC
>
> I don't think kvmtool supports running on a system that doesn't have
> in-kernel GIC support.
>
> >
> >
> > which seems strange for me because dmesg shows kernel is 4.10.2-v8+ and
> > Boot CPU is AArach64, all CPUs are initialized in HYP mode and /dev/kvm
> > exists. So I thought GIC emulation is also enabled, or should I need to
> > enable other config flags or do settings?
>
> The RPI3 doesn't have a GIC, so you'll never get KVM emulation of the
> GIC with that board.  Ever.  That's why you want the irqchip in
> userspace patches.
>
> > Or, is this way of applying patch
> > wrong? Current config.gz is:
> > https://gist.github.com/caprice-j/d6e144cb8727df813a4fbe716beac3cb
> >
> > My other concerns are no kvm-ok command is available and /proc/cpuinfo
> does
> > not show flags for virtualization support (I believe RPI3's cortex A53
> > (ARMv8) has builtin virt support and does not show the support flag
> > explicitly in /proc/cpuinfo ... but I'm unsure that is true)
>
> If you have /dev/kvm then KVM as such is running on your system, no need
> to worry further, just fix your setup to emulate the GIC in userspace.
>
> Thanks,
> -Christoffer
>



-- 
Yasutaka TANAKA
M.S. Candidate, Computer Science
Fu Foundation School of Engineering and Applied Science
Columbia University

y.tanaka@columbia.edu

[-- Attachment #1.2: Type: text/html, Size: 5983 bytes --]

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM on ARM Cortex A53 in 32-bit Mode

[Christoffer Dall]

On Fri, Mar 17, 2017 at 01:09:22AM -0400, Yasutaka, T wrote:
> 2017-03-16 5:38 GMT-04:00 Christoffer Dall <cdall@linaro.org>:
> 
> > On Wed, Mar 15, 2017 at 07:50:36PM -0400, Yasutaka, T wrote:
> > > Hi Christopher and other KVM/ARM community members,
> > >
> > > I tried the following QEMU and Kernel combinations on Raspberry Pi 3
> > > (AArch64 mode) but failed:
> > >
> > > Latest commit of Alexander Graf's no-kvm-irqchip QEMU branch:
> > > https://github.com/agraf/qemu/tree/no-kvm-irqchip
> > >
> > > raspberrypi/linux's rpi-4.10.y branch (forked from Linux 4.10-rc8?)
> > latest
> > > commit bb0ff9d059c67e1611c7422f7982a6a4876efe67
> > > patched by $(git diff (linux 4.10-rc5 commit
> > > 7a308bb3016f57e5be11a677d15b821536419d36) (Christpher's irqs-to-user-v2
> > > latest commit d0a7cc725535df1b9cc64b442d246c20a9edb904) )
> > >
> > > and got the following error:
> > >
> > > pi@raspberrypi:~/os/graf/no-kvm-irqchip/aarch64-softmmu $
> > > qemu-system-aarch64 --enable-kvm -M virt
> > > "kvm" accelerator not found.
> > > No accelerator found!
> >
> > You'd need "-machine virt,kernel_irqchip=off" to use the userspace GIC
> > emulation.  Perhaps QEMU's error message for trying to initialize the
> > in-kernel GIC is just weird.
> >
> 
> I executed with the flag but still got the same error message (accelerator
> not found):
> 
> qemu-system-aarch64 --enable-kvm -cpu host -machine
> type=virt,kernel_irqchip=off -nographic -smp 1 -m 256 -kernel
> aarch64-linux-3.15rc2-buildroot.img --append "console=ttyAMA0"
> 
> 
> And I found out that an article saying recent OpenSUSE distro officialy
> supports KVM:
> https://news.opensuse.org/2016/12/05/opensuse-leap-42-2-gets-64-bit-raspberry-image/
> 
> The developer of this distro is the same as the kernel_irqchip=off QEMU
> patch author (Alexander Graf), and when I tried the above command on this
> [1] distro KVM-QEMU actually worked ("worked" means it reaches to the
> buildroot login prompt).
> 
> So I guess this distro applied your patches (it proves KVM/QEMU on RPI3 is
> actually feasible) and the central reason that my failure of KVM on
> Raspibian OS would be just a kernel config problem.
> 
> I set my .config file by make menuconfig, but it might need some other
> (potentially non-GUI?) configurations. My current config is:
> https://gist.github.com/caprice-j/836e3c7627f65c0f3a79e56a8d75368e

I thought you said you had /dev/kvm ?  Then I'm not sure why it wouldn't
work, or why you're looking at kernel config issues.

It would be worthwhile debugging QEMU and figuring out why you get that
error.

printf in QEMU should work just fine and there should also be mechanisms
to increase the log level and diagnostics in QEMU.

-Christoffer