Re: KVM/ARM64: Booting KVM on FVP/fastmodels

Re: KVM/ARM64: Booting KVM on FVP/fastmodels

[André Przywara]

On 29/03/2021 13:13, yinhua_chen@outlook.com wrote:

Hi,

> I want to boot KVM on simulators.
> 
> I don't want to use QEMU since some features are only supported on
> FVP/fastmodels now.
> 
> However, I cannot find documents/blogs/wiki about booting KVM on
> FVP/fastmodels.
> 
> Does KVM support running on FVP?

The FVP emulates the full architecture, including all exception levels
(so also EL2), and the virtualization support of the GIC and the Generic
Timer.
So yes, the models fully support KVM (or any other hypervisor, for that
matter), and are in fact often used for developing KVM support of new
hardware features.

So what are the problems that you are facing? Just running a kernel like
you would normally do on the model should work out of the box.
If you describe your setup and the error messages, we can probably help
you better.

Cheers,
Andre

> 
> Appreciate for your help!
> 
> Sincerely,
> Yinhua
> 
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
> 

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM/ARM64: Booting KVM on FVP/fastmodels

[Andre Przywara]

On Wed, 14 Apr 2021 02:37:07 +0000
胤桦 陈 <yinhua_chen@outlook.com> wrote:

> Hi, Andre
> 
> Do you mean that if I boot Linux on FVP, then I can run KVM on FVP? Since KVM is just a module in Linux?

Well, on ARM it's not a kernel module, but built-in to the kernel
image, but yes: You can run any normal KVM enabled kernel on
it. So just use the same recipe you would use on any other system to
run KVM guests.

Cheers,
Andre
 
> ________________________________
> From: André Przywara <andre.przywara@arm.com>
> Sent: Tuesday, March 30, 2021 18:22
> To: yinhua_chen@outlook.com <yinhua_chen@outlook.com>; kvmarm@lists.cs.columbia.edu <kvmarm@lists.cs.columbia.edu>
> Subject: Re: KVM/ARM64: Booting KVM on FVP/fastmodels
> 
> On 29/03/2021 13:13, yinhua_chen@outlook.com wrote:
> 
> Hi,
> 
> > I want to boot KVM on simulators.
> >
> > I don't want to use QEMU since some features are only supported on
> > FVP/fastmodels now.
> >
> > However, I cannot find documents/blogs/wiki about booting KVM on
> > FVP/fastmodels.
> >
> > Does KVM support running on FVP?  
> 
> The FVP emulates the full architecture, including all exception levels
> (so also EL2), and the virtualization support of the GIC and the Generic
> Timer.
> So yes, the models fully support KVM (or any other hypervisor, for that
> matter), and are in fact often used for developing KVM support of new
> hardware features.
> 
> So what are the problems that you are facing? Just running a kernel like
> you would normally do on the model should work out of the box.
> If you describe your setup and the error messages, we can probably help
> you better.
> 
> Cheers,
> Andre
> 
> >
> > Appreciate for your help!
> >
> > Sincerely,
> > Yinhua
> >
> > _______________________________________________
> > kvmarm mailing list
> > kvmarm@lists.cs.columbia.edu
> > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
> >  
> 

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM/ARM64: Booting KVM on FVP/fastmodels

[胤桦 陈]

[-- Attachment #1.1: Type: text/plain, Size: 4622 bytes --]

Hi Andre,



I follow the document https://git.linaro.org/landing-teams/working/arm/arm-reference-platforms.git/about/docs/basefvp/user-guide.rst, and I build a Linux-on-FVP environment successfully. However, I still cannot boot KVM, the details are as follows:



1.I make sure that two configuration files enable KVM (I guess these are the right configuration files), they are

layers/meta-arm/meta-arm-bsp/recipes-kernel/linux/files/tc0/defconfig:CONFIG_KVM=y

build-poky/tmp-poky/work-shared/fvp-base/kernel-source/arch/arm64/configs/defconfig:CONFIG_KVM=y



2.I follow the document https://git.linaro.org/landing-teams/working/arm/arm-reference-platforms.git/about/docs/basefvp/user-guide.rst. After building the whole software stack, I try several commands but the results show that KVM does not work. The screenshots are shown as follows[commands]I try three commands, "kvm-ok", "dmesg | grep KVM", and "grep -E -o 'vmx|svm' /proc/cpuinfo". But all of them show negative results.



I guess maybe the reason is that FVP configuration uses a KVM-unsupport CPU, but I do not know the right configuration, and the FVP documents do not mention anything about KVM.


And this is the FVP booting command in the scripts:

===

/FVP_Base_RevC-2xAEMv8A

 -C pctl.startup=0.0.0.0

 -C bp.secure_memory=0

 -C cluster0.NUM_CORES=1

 -C cluster1.NUM_CORES=1

 -C cache_state_modelled=0

 -C bp.pl011_uart0.untimed_fifos=1

 -C bp.pl011_uart0.unbuffered_output=1

 -C bp.pl011_uart0.out_file=uart0-1624328932274439497.log

 -C bp.pl011_uart1.out_file=uart1-1624328932274439497.log

 -C bp.secureflashloader.fname=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/bl1-fvp.bin

 -C bp.flashloader0.fname=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/fip-fvp.bin

 -C bp.hostbridge.interfaceName=tap0

 -C bp.smsc_91c111.enabled=1

 --data cluster0.cpu0=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/Image@0x80080000

 --data cluster0.cpu0=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/fvp-base-gicv3-psci-custom.dtb@0x83000000

 -C bp.ve_sysregs.mmbSiteDefault=0

 -C bp.ve_sysregs.exit_on_shutdown=1

 -C bp.virtioblockdevice.image_path=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/core-image-minimal-fvp-base.disk.img

===



If you can provide any kind of help, I appreciate it very much.

________________________________
From: Andre Przywara <andre.przywara@arm.com>
Sent: Wednesday, April 14, 2021 16:50
To: 胤桦 陈 <yinhua_chen@outlook.com>
Cc: kvmarm@lists.cs.columbia.edu <kvmarm@lists.cs.columbia.edu>
Subject: Re: KVM/ARM64: Booting KVM on FVP/fastmodels

On Wed, 14 Apr 2021 02:37:07 +0000
胤桦 陈 <yinhua_chen@outlook.com> wrote:

> Hi, Andre
>
> Do you mean that if I boot Linux on FVP, then I can run KVM on FVP? Since KVM is just a module in Linux?

Well, on ARM it's not a kernel module, but built-in to the kernel
image, but yes: You can run any normal KVM enabled kernel on
it. So just use the same recipe you would use on any other system to
run KVM guests.

Cheers,
Andre

> ________________________________
> From: André Przywara <andre.przywara@arm.com>
> Sent: Tuesday, March 30, 2021 18:22
> To: yinhua_chen@outlook.com <yinhua_chen@outlook.com>; kvmarm@lists.cs.columbia.edu <kvmarm@lists.cs.columbia.edu>
> Subject: Re: KVM/ARM64: Booting KVM on FVP/fastmodels
>
> On 29/03/2021 13:13, yinhua_chen@outlook.com wrote:
>
> Hi,
>
> > I want to boot KVM on simulators.
> >
> > I don't want to use QEMU since some features are only supported on
> > FVP/fastmodels now.
> >
> > However, I cannot find documents/blogs/wiki about booting KVM on
> > FVP/fastmodels.
> >
> > Does KVM support running on FVP?
>
> The FVP emulates the full architecture, including all exception levels
> (so also EL2), and the virtualization support of the GIC and the Generic
> Timer.
> So yes, the models fully support KVM (or any other hypervisor, for that
> matter), and are in fact often used for developing KVM support of new
> hardware features.
>
> So what are the problems that you are facing? Just running a kernel like
> you would normally do on the model should work out of the box.
> If you describe your setup and the error messages, we can probably help
> you better.
>
> Cheers,
> Andre
>
> >
> > Appreciate for your help!
> >
> > Sincerely,
> > Yinhua
> >
> > _______________________________________________
> > kvmarm mailing list
> > kvmarm@lists.cs.columbia.edu
> > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
> >
>


[-- Attachment #1.2: Type: text/html, Size: 15840 bytes --]

[-- Attachment #2: Type: text/plain, Size: 151 bytes --]

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: KVM/ARM64: Booting KVM on FVP/fastmodels

[Andre Przywara]

On Tue, 22 Jun 2021 02:33:44 +0000
胤桦 陈 <yinhua_chen@outlook.com> wrote:

Hi,

> I follow the document https://git.linaro.org/landing-teams/working/arm/arm-reference-platforms.git/about/docs/basefvp/user-guide.rst,

sorry for that ;-)

> and I build a Linux-on-FVP environment successfully. However, I still cannot boot KVM, the details are as follows:
> 1.I make sure that two configuration files enable KVM (I guess these are the right configuration files), they are
> layers/meta-arm/meta-arm-bsp/recipes-kernel/linux/files/tc0/defconfig:CONFIG_KVM=y
> build-poky/tmp-poky/work-shared/fvp-base/kernel-source/arch/arm64/configs/defconfig:CONFIG_KVM=y

> 2.I follow the document https://git.linaro.org/landing-teams/working/arm/arm-reference-platforms.git/about/docs/basefvp/user-guide.rst. After building the whole software stack, I try several commands but the results show that KVM does not work.

Have you tried actually running a guest? Using kvmtool, for instance?
kvmtool can be statically linked, so you end up with a single
executable file, easy to copy into whatever userland you use.
If you clone it from
git://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git
and do "make lkvm-static" on any arm64 system, you can copy the
resulting binary "lkvm-static" into your rootfs, and then run it on the
model:
# lkvm-static run -k <some_kernel_image>
You might need to create the device file first (see below).

> The screenshots are shown as follows[commands]I try three commands,
> "kvm-ok"

This does not do much on ARM/arm64, it just merely checks for the
existence of /dev/kvm. Depending on your userland, nobody might have
created that - but still KVM might be configured fine.
A simple "mknod /dev/kvm c 10 232" fixes that, but the test passing now
doesn't mean much (as is just checks for the existence of this file,
not actual KVM functionality).

> "dmesg | grep KVM"

Please always use "grep -i" when grep'ing in dmesg, capitalisation of
those names may vary. On my kernel there is indeed not a single hit for
"KVM", but several for "kvm".
Keep in mind that this is not a bullet-proof method: the boot messages
from KVM might have been flushed from the dmesg buffer already - on one
of my systems running for some weeks with some debug messages they
certainly are.
But right after boot there is a high chance of success, though, and
it's the easiest check for KVM.

> "grep -E -o 'vmx|svm' /proc/cpuinfo"

This is x86 specific and does not work at all on ARM/arm64.
There is not much to learn about KVM in cpuinfo on arm64.

> I guess maybe the reason is that FVP configuration uses a KVM-unsupport CPU, but I do not know the right configuration, and the FVP documents do not mention anything about KVM.

I doubt that, the normal models should all come with EL2 enabled. The
RevC model definitely works.

Cheers,
Andre

> And this is the FVP booting command in the scripts:
> 
> ===
> 
> /FVP_Base_RevC-2xAEMv8A
> 
>  -C pctl.startup=0.0.0.0
> 
>  -C bp.secure_memory=0
> 
>  -C cluster0.NUM_CORES=1
> 
>  -C cluster1.NUM_CORES=1
> 
>  -C cache_state_modelled=0
> 
>  -C bp.pl011_uart0.untimed_fifos=1
> 
>  -C bp.pl011_uart0.unbuffered_output=1
> 
>  -C bp.pl011_uart0.out_file=uart0-1624328932274439497.log
> 
>  -C bp.pl011_uart1.out_file=uart1-1624328932274439497.log
> 
>  -C bp.secureflashloader.fname=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/bl1-fvp.bin
> 
>  -C bp.flashloader0.fname=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/fip-fvp.bin
> 
>  -C bp.hostbridge.interfaceName=tap0
> 
>  -C bp.smsc_91c111.enabled=1
> 
>  --data cluster0.cpu0=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/Image@0x80080000
> 
>  --data cluster0.cpu0=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/fvp-base-gicv3-psci-custom.dtb@0x83000000
> 
>  -C bp.ve_sysregs.mmbSiteDefault=0
> 
>  -C bp.ve_sysregs.exit_on_shutdown=1
> 
>  -C bp.virtioblockdevice.image_path=<workspace>/build-poky/tmp-poky/deploy/images/fvp-base/core-image-minimal-fvp-base.disk.img
> 
> ===
> 
> 
> 
> If you can provide any kind of help, I appreciate it very much.
> 
> ________________________________
> From: Andre Przywara <andre.przywara@arm.com>
> Sent: Wednesday, April 14, 2021 16:50
> To: 胤桦 陈 <yinhua_chen@outlook.com>
> Cc: kvmarm@lists.cs.columbia.edu <kvmarm@lists.cs.columbia.edu>
> Subject: Re: KVM/ARM64: Booting KVM on FVP/fastmodels
> 
> On Wed, 14 Apr 2021 02:37:07 +0000
> 胤桦 陈 <yinhua_chen@outlook.com> wrote:
> 
> > Hi, Andre
> >
> > Do you mean that if I boot Linux on FVP, then I can run KVM on FVP? Since KVM is just a module in Linux?  
> 
> Well, on ARM it's not a kernel module, but built-in to the kernel
> image, but yes: You can run any normal KVM enabled kernel on
> it. So just use the same recipe you would use on any other system to
> run KVM guests.
> 
> Cheers,
> Andre
> 
> > ________________________________
> > From: André Przywara <andre.przywara@arm.com>
> > Sent: Tuesday, March 30, 2021 18:22
> > To: yinhua_chen@outlook.com <yinhua_chen@outlook.com>; kvmarm@lists.cs.columbia.edu <kvmarm@lists.cs.columbia.edu>
> > Subject: Re: KVM/ARM64: Booting KVM on FVP/fastmodels
> >
> > On 29/03/2021 13:13, yinhua_chen@outlook.com wrote:
> >
> > Hi,
> >  
> > > I want to boot KVM on simulators.
> > >
> > > I don't want to use QEMU since some features are only supported on
> > > FVP/fastmodels now.
> > >
> > > However, I cannot find documents/blogs/wiki about booting KVM on
> > > FVP/fastmodels.
> > >
> > > Does KVM support running on FVP?  
> >
> > The FVP emulates the full architecture, including all exception levels
> > (so also EL2), and the virtualization support of the GIC and the Generic
> > Timer.
> > So yes, the models fully support KVM (or any other hypervisor, for that
> > matter), and are in fact often used for developing KVM support of new
> > hardware features.
> >
> > So what are the problems that you are facing? Just running a kernel like
> > you would normally do on the model should work out of the box.
> > If you describe your setup and the error messages, we can probably help
> > you better.
> >
> > Cheers,
> > Andre
> >  
> > >
> > > Appreciate for your help!
> > >
> > > Sincerely,
> > > Yinhua
> > >
> > > _______________________________________________
> > > kvmarm mailing list
> > > kvmarm@lists.cs.columbia.edu
> > > https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
> > >  
> >  
> 

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm