Re: KVM userspace GICv2 IRQ controller on platform with GICv3

[Marc Zyngier <maz@kernel.org>]

On Mon, 04 Oct 2021 11:30:06 +0100,
Lukas Jünger <lukas.juenger@ice.rwth-aachen.de> wrote:
> 
> [1  <text/plain (en-US); utf-8 (quoted-printable)>]
> On 04.10.21 12:24, Marc Zyngier wrote:
> > Hi Lukas,
> 
> Hi Mark,
> 
> Thanks for your quick reply.
> 
> > On Mon, 04 Oct 2021 11:07:47 +0100,
> > Lukas Jünger <lukas.juenger@ice.rwth-aachen.de> wrote:
> >> Hello,
> >> 
> >> I am trying to run an emulator that uses KVM on arm64 to execute
> >> code. The emulator contains a userspace model of a GICv2 IRQ
> >> controller. The platform that I am running on (n1sdp) has a
> > N1-SDP? My condolences...
> Is there more to this?

How do you like the PCI patches? :D

> >> GICv3. When I boot Linux in the emulator I run into
> >> gic_check_cpu_features()  in drivers/irqchip/irq-gic.c, which taints
> >> the kernel as the host uses system registers to communicate with the
> >> host GICv3. I saw that ICC_SRE_ELx can be used to force MMIO, but
> >> setting this from inside the VM did not work and using KVM_SET_ONE_REG
> >> failed with error.
> > N1-SDP doesn't implement the MMIO interface at all, and our GIC
> > emulation doesn't either. Both are valid implementations.
> > 
> >> Is there a way to use a userspace GICv2 model with KVM on a GICv3 host
> >> without tainting?
> > The tainting happens because you have created a VM with a GICv3
> > irqchip (at some point, your VMM calls into KVM to create a device
> > with the KVM_DEV_TYPE_ARM_VGIC_V3 attribute). The guest then sees that
> > GICv3 is enabled (ICC_SRE_ELx.SRE==1), and yet you somehow expose a
> > GICv2 to the guest (either via DT or ACPI). That's illegal.
> > 
> > If you want a userspace interrupt controller, you need prevent the
> > creation of an in-kernel interrupt controller, which is a change in
> > your VMM or maybe a configuration change.
> I'm not using an in-kernel irq controller, at least I don't set one
> up. This is all custom, so no QEMU etc. The GICv2 is also a custom
> model that lives in user space. The guest gets a DT telling it that
> there is a GICv2 and it should access it via MMIO. This all used to
> work on Raspberry Pi 3

RPI3 doesn't have a GIC at all, so the example is a bit moot.

> and Socionext Synquacer.

This one however is more interesting, as it has a GICv3 + v2 compat.

> The port to N1-SDP is
> giving me trouble. I understand why it is tainting the kernel, I was
> just wondering if I could somehow tell KVM to set this up correctly,
> e.g. by setting the ICC_SRE_ELx.

KVM doesn't *set* ICC_SRE_EL1.SRE. It is RAO/WI on this machine, which
is perfectly legal. However, KVM traps this access and emulates it
(access_gic_sre() returns vcpu->arch.vgic_cpu.vgic_v3.vgic_sre).

So if you see ICC_SRE_EL1.SRE==1 in your guest, that's because
vgic_sre is set to something that is non-zero. The only way for this
bit to be set is in vgic_v3_enable(), which has the following code:

<quote>
	if (vcpu->kvm->arch.vgic.vgic_model == KVM_DEV_TYPE_ARM_VGIC_V3) {
		vgic_v3->vgic_sre = (ICC_SRE_EL1_DIB |
				     ICC_SRE_EL1_DFB |
				     ICC_SRE_EL1_SRE);
		vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE;
	} else {
		vgic_v3->vgic_sre = 0;
	}
</quote>

So short of a terrible bug that would dump random values in this
structure, you are setting vgic_model to a GICv3 implementation. This
can only be done from userspace if you are creating a GICv3 irqchip.

Without seeing what your userspace does, I'm afraid I can't help you
much further. Can you please provide some traces of what it does? A
strace dump would certainly help.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm