[PATCH] qemuarm64: maximise the emulated CPU capabilities

[PATCH] qemuarm64: maximise the emulated CPU capabilities

[ross.burton]

From: Ross Burton <ross.burton@arm.com>

Instead of emulating a Cortex-A57, just ask qemu to emulate everything
it knows how to (apart from Pointer Authentication, where we use qemu's
implementation-defined algorithm which is vastly faster to emulate).

This does not change the tune, so the generated code is identical. The
advantage of doing this is that it lets us verify that features such as
Branch Target Indentification, which requires hardware support, is
actually functional.

Signed-off-by: Ross Burton <ross.burton@arm.com>
---
 meta/conf/machine/qemuarm64.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/machine/qemuarm64.conf b/meta/conf/machine/qemuarm64.conf
index a096d964db5..63af4cd346d 100644
--- a/meta/conf/machine/qemuarm64.conf
+++ b/meta/conf/machine/qemuarm64.conf
@@ -14,7 +14,7 @@ SERIAL_CONSOLES ?= "115200;ttyAMA0 115200;hvc0"
 # For runqemu
 QB_SYSTEM_NAME = "qemu-system-aarch64"
 QB_MACHINE = "-machine virt"
-QB_CPU = "-cpu cortex-a57"
+QB_CPU = "-cpu max,pauth-impdef=on"
 QB_SMP ?= "-smp 4"
 QB_CPU_KVM = "-cpu host -machine gic-version=3"
 # For graphics to work we need to define the VGA device as well as the necessary USB devices
-- 
2.34.1

Re: [OE-core] [PATCH] qemuarm64: maximise the emulated CPU capabilities

[Mikko Rapeli]

Hi,

On Tue, Feb 13, 2024 at 04:42:23PM +0000, Ross Burton wrote:
> From: Ross Burton <ross.burton@arm.com>
> 
> Instead of emulating a Cortex-A57, just ask qemu to emulate everything
> it knows how to (apart from Pointer Authentication, where we use qemu's
> implementation-defined algorithm which is vastly faster to emulate).
> 
> This does not change the tune, so the generated code is identical. The
> advantage of doing this is that it lets us verify that features such as
> Branch Target Indentification, which requires hardware support, is
> actually functional.
> 
> Signed-off-by: Ross Burton <ross.burton@arm.com>
> ---
>  meta/conf/machine/qemuarm64.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/conf/machine/qemuarm64.conf b/meta/conf/machine/qemuarm64.conf
> index a096d964db5..63af4cd346d 100644
> --- a/meta/conf/machine/qemuarm64.conf
> +++ b/meta/conf/machine/qemuarm64.conf
> @@ -14,7 +14,7 @@ SERIAL_CONSOLES ?= "115200;ttyAMA0 115200;hvc0"
>  # For runqemu
>  QB_SYSTEM_NAME = "qemu-system-aarch64"
>  QB_MACHINE = "-machine virt"
> -QB_CPU = "-cpu cortex-a57"
> +QB_CPU = "-cpu max,pauth-impdef=on"

FWIW, this can also be risky and cause instabilities. We've seen boot hangs
at least with old qemu 7.2 and kernel 4.xy where serial console detection
did not correctly work on every boot.

If things work now, then great!

Cheers,

-Mikko

Re: [OE-core] [PATCH] qemuarm64: maximise the emulated CPU capabilities

[Ross Burton]

On 14 Feb 2024, at 11:04, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> FWIW, this can also be risky and cause instabilities. We've seen boot hangs
> at least with old qemu 7.2 and kernel 4.xy where serial console detection
> did not correctly work on every boot.
> 
> If things work now, then great!

It worked on the few tests I did locally. I didn’t run it through the AB yet to stress it in any meaningful way yet.

Ross

Re: [OE-core] [PATCH] qemuarm64: maximise the emulated CPU capabilities

[Richard Purdie]

On Wed, 2024-02-14 at 11:06 +0000, Ross Burton wrote:
> On 14 Feb 2024, at 11:04, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > FWIW, this can also be risky and cause instabilities. We've seen boot hangs
> > at least with old qemu 7.2 and kernel 4.xy where serial console detection
> > did not correctly work on every boot.
> > 
> > If things work now, then great!
> 
> It worked on the few tests I did locally. I didn’t run it through the AB 
> yet to stress it in any meaningful way yet.

I did put it through last night:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6566

and things are still running. The ptest issues so far look unrelated.

Cheers,

Richard

Re: [OE-core] [PATCH] qemuarm64: maximise the emulated CPU capabilities

[Mikko Rapeli]

Hi,

On Wed, Feb 14, 2024 at 11:27:52AM +0000, Richard Purdie wrote:
> On Wed, 2024-02-14 at 11:06 +0000, Ross Burton wrote:
> > On 14 Feb 2024, at 11:04, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> > > FWIW, this can also be risky and cause instabilities. We've seen boot hangs
> > > at least with old qemu 7.2 and kernel 4.xy where serial console detection
> > > did not correctly work on every boot.
> > > 
> > > If things work now, then great!
> > 
> > It worked on the few tests I did locally. I didn’t run it through the AB 
> > yet to stress it in any meaningful way yet.
> 
> I did put it through last night:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6566
> 
> and things are still running. The ptest issues so far look unrelated.

optee xtests are triggering a reproducible RCU stall in our testing with
--cpu max,pauth-impdef=on and qemu-system 8.2.1 from Debian unstable.
This does not happen with --cpu cortex-a57 on this qemu-machine version.

Example test run:

https://ledge.validation.linaro.org/scheduler/job/80815

+ xtest --clear-storage
[  285.552965] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  285.557211] rcu: 	2-...0: (2 GPs behind) idle=a1c4/1/0x4000000000000000 softirq=32180/32180 fqs=2004
[  285.559304] rcu: 	(detected by 0, t=5252 jiffies, g=75961, q=5749 ncpus=4)
[  348.573538] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  348.574771] rcu: 	2-...0: (2 GPs behind) idle=a1c4/1/0x4000000000000000 softirq=32180/32180 fqs=8797
[  348.576032] rcu: 	(detected by 0, t=21007 jiffies, g=75961, q=22677 ncpus=4)
[  411.593692] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  411.594425] rcu: 	2-...0: (2 GPs behind) idle=a1c4/1/0x4000000000000000 softirq=32180/32180 fqs=15573
[  411.595536] rcu: 	(detected by 0, t=36762 jiffies, g=75961, q=40851 ncpus=4)
[  474.614478] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  474.615567] rcu: 	2-...0: (2 GPs behind) idle=a1c4/1/0x4000000000000000 softirq=32180/32180 fqs=22375
[  474.616871] rcu: 	(detected by 0, t=52517 jiffies, g=75961, q=57789 ncpus=4)
[  537.635868] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
[  537.637129] rcu: 	2-...0: (2 GPs behind) idle=a1c4/1/0x4000000000000000 softirq=32180/32180 fqs=29112
[  537.638621] rcu: 	(detected by 0, t=68272 jiffies, g=75961, q=74773 ncpus=4)

The qemu system runs UEFI compatible firmware with u-boot, tf-a, optee etc configured for
secureboot and Linux kernel 6.5.10, initramfs and rootfs with systemd is slightly
behind poky master branch.

This was the kind of issues that we saw also one year ago with max CPU settings on qemu
so the root cause is still not identified or fixed.

Of course the problems could be in our specific configuration with optee and u-boot
based UEFI firmware also on qemu which doesn't need them for basic usage.

I have not yet tried to reproduce this with a local build using native qemu from yocto
build via runqemu/testimage.bbclass.

Cheers,

-Mikko

Re: [OE-core] [PATCH] qemuarm64: maximise the emulated CPU capabilities

[Ross Burton]

On 15 Feb 2024, at 06:40, Mikko Rapeli <mikko.rapeli@linaro.org> wrote:
> optee xtests are triggering a reproducible RCU stall in our testing with
> --cpu max,pauth-impdef=on and qemu-system 8.2.1 from Debian unstable.
> This does not happen with --cpu cortex-a57 on this qemu-machine version.
> 

[sigh]

Can you file a bug against optee about this?

Cheers,
Ross