[PATCH] KVM: selftests: fix irqfd_test on arm64

[PATCH] KVM: selftests: fix irqfd_test on arm64

[Sebastian Ott]

irqfd_test on arm triggers the following assertion:
==== Test Assertion Failure ====
  include/kvm_util.h:527: !ret
  pid=3643 tid=3643 errno=11 - Resource temporarily unavailable
     1  0x00000000004026d7: kvm_irqfd at kvm_util.h:527
     2  0x0000000000402083: main at irqfd_test.c:100
     3  0x0000ffffa5aab587: ?? ??:0
     4  0x0000ffffa5aab65f: ?? ??:0
     5  0x000000000040236f: _start at ??:?
  KVM_IRQFD failed, rc: -1 errno: 11 (Resource temporarily unavailable)

Fix this by setting up a vgic for the vm.

Signed-off-by: Sebastian Ott <sebott@redhat.com>
---
 tools/testing/selftests/kvm/irqfd_test.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/kvm/irqfd_test.c b/tools/testing/selftests/kvm/irqfd_test.c
index 7c301b4c7005..f7b8766e9d42 100644
--- a/tools/testing/selftests/kvm/irqfd_test.c
+++ b/tools/testing/selftests/kvm/irqfd_test.c
@@ -8,7 +8,11 @@
 #include <stdint.h>
 #include <sys/sysinfo.h>
 
+#include "processor.h"
 #include "kvm_util.h"
+#ifdef __aarch64__
+#include "vgic.h"
+#endif
 
 static struct kvm_vm *vm1;
 static struct kvm_vm *vm2;
@@ -86,14 +90,30 @@ static void juggle_eventfd_primary(struct kvm_vm *vm, int eventfd)
 	kvm_irqfd(vm, GSI_BASE_PRIMARY + 1, eventfd, KVM_IRQFD_FLAG_DEASSIGN);
 }
 
+static struct kvm_vm *test_vm_create(void)
+{
+#ifdef __aarch64__
+	struct kvm_vm *vm;
+	struct kvm_vcpu *vcpu;
+	int gic_fd;
+
+	vm = vm_create_with_one_vcpu(&vcpu, NULL);
+	gic_fd = vgic_v3_setup(vm, 1, 64);
+	__TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3");
+
+	return vm;
+#endif
+	return vm_create(1);
+}
+
 int main(int argc, char *argv[])
 {
 	pthread_t racing_thread;
 	int r, i;
 
 	/* Create "full" VMs, as KVM_IRQFD requires an in-kernel IRQ chip. */
-	vm1 = vm_create(1);
-	vm2 = vm_create(1);
+	vm1 = test_vm_create();
+	vm2 = test_vm_create();
 
 	WRITE_ONCE(__eventfd, kvm_new_eventfd());
 
-- 
2.51.0

Re: [PATCH] KVM: selftests: fix irqfd_test on arm64

[Sean Christopherson]

On Mon, Aug 25, 2025, Sebastian Ott wrote:
> irqfd_test on arm triggers the following assertion:
> ==== Test Assertion Failure ====
>   include/kvm_util.h:527: !ret
>   pid=3643 tid=3643 errno=11 - Resource temporarily unavailable
>      1  0x00000000004026d7: kvm_irqfd at kvm_util.h:527
>      2  0x0000000000402083: main at irqfd_test.c:100
>      3  0x0000ffffa5aab587: ?? ??:0
>      4  0x0000ffffa5aab65f: ?? ??:0
>      5  0x000000000040236f: _start at ??:?
>   KVM_IRQFD failed, rc: -1 errno: 11 (Resource temporarily unavailable)
> 
> Fix this by setting up a vgic for the vm.
> 
> Signed-off-by: Sebastian Ott <sebott@redhat.com>
> ---
> @@ -86,14 +90,30 @@ static void juggle_eventfd_primary(struct kvm_vm *vm, int eventfd)
>  	kvm_irqfd(vm, GSI_BASE_PRIMARY + 1, eventfd, KVM_IRQFD_FLAG_DEASSIGN);
>  }
>  
> +static struct kvm_vm *test_vm_create(void)
> +{
> +#ifdef __aarch64__
> +	struct kvm_vm *vm;
> +	struct kvm_vcpu *vcpu;
> +	int gic_fd;
> +
> +	vm = vm_create_with_one_vcpu(&vcpu, NULL);
> +	gic_fd = vgic_v3_setup(vm, 1, 64);
> +	__TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3");

I don't think this test requires v3+, any GIC will do.

Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()?  E.g.
could we create a v3 GIC when possible, and fall back to v2?  And then provide a
way for tests to express a hard v3 GIC dependency?

Having to worry about things like this in fairly generic code is quite burdensome.

> +
> +	return vm;
> +#endif
> +	return vm_create(1);
> +}

Re: [PATCH] KVM: selftests: fix irqfd_test on arm64

[Marc Zyngier]

On Mon, 25 Aug 2025 20:52:21 +0100,
Sean Christopherson <seanjc@google.com> wrote:
> 
> On Mon, Aug 25, 2025, Sebastian Ott wrote:
> > irqfd_test on arm triggers the following assertion:
> > ==== Test Assertion Failure ====
> >   include/kvm_util.h:527: !ret
> >   pid=3643 tid=3643 errno=11 - Resource temporarily unavailable
> >      1  0x00000000004026d7: kvm_irqfd at kvm_util.h:527
> >      2  0x0000000000402083: main at irqfd_test.c:100
> >      3  0x0000ffffa5aab587: ?? ??:0
> >      4  0x0000ffffa5aab65f: ?? ??:0
> >      5  0x000000000040236f: _start at ??:?
> >   KVM_IRQFD failed, rc: -1 errno: 11 (Resource temporarily unavailable)
> > 
> > Fix this by setting up a vgic for the vm.
> > 
> > Signed-off-by: Sebastian Ott <sebott@redhat.com>
> > ---
> > @@ -86,14 +90,30 @@ static void juggle_eventfd_primary(struct kvm_vm *vm, int eventfd)
> >  	kvm_irqfd(vm, GSI_BASE_PRIMARY + 1, eventfd, KVM_IRQFD_FLAG_DEASSIGN);
> >  }
> >  
> > +static struct kvm_vm *test_vm_create(void)
> > +{
> > +#ifdef __aarch64__
> > +	struct kvm_vm *vm;
> > +	struct kvm_vcpu *vcpu;
> > +	int gic_fd;
> > +
> > +	vm = vm_create_with_one_vcpu(&vcpu, NULL);
> > +	gic_fd = vgic_v3_setup(vm, 1, 64);
> > +	__TEST_REQUIRE(gic_fd >= 0, "Failed to create vgic-v3");
> 
> I don't think this test requires v3+, any GIC will do.

There is no such thing as "any GIC". You need to know what is
available, and ask for something that actually exists. So while the
above is wrong on the ground that this doesn't work on v2 or v5, the
selection has to be explicit.

> Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()?  E.g.
> could we create a v3 GIC when possible, and fall back to v2?  And then provide a
> way for tests to express a hard v3 GIC dependency?

You can ask KVM what's available. Like an actual VMM does. There is no
shortage of examples in the current code base.

And ideally, this should be made an integral part of creating a viable
VM, which the current VM creation hack makes a point in not providing.

	M.

-- 
Jazz isn't dead. It just smells funny.

Re: [PATCH] KVM: selftests: fix irqfd_test on arm64

[Sean Christopherson]

On Mon, Aug 25, 2025, Marc Zyngier wrote:
> On Mon, 25 Aug 2025 20:52:21 +0100,
> Sean Christopherson <seanjc@google.com> wrote:
> > Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()?  E.g.
> > could we create a v3 GIC when possible, and fall back to v2?  And then provide a
> > way for tests to express a hard v3 GIC dependency?
> 
> You can ask KVM what's available. Like an actual VMM does. There is no
> shortage of examples in the current code base.

Right, by "sane" I meant: is there a way to instantiate a supported GIC without
making it hard/painful to write tests, and without having to plumb in arm64
specific requirements to common APIs?

E.g. are there tests that use the common vm_create() APIs and rely on NOT having
a GIC?

> And ideally, this should be made an integral part of creating a viable
> VM, which the current VM creation hack makes a point in not providing.

Re: [PATCH] KVM: selftests: fix irqfd_test on arm64

[Oliver Upton]

On Mon, Aug 25, 2025 at 02:11:30PM -0700, Sean Christopherson wrote:
> On Mon, Aug 25, 2025, Marc Zyngier wrote:
> > On Mon, 25 Aug 2025 20:52:21 +0100,
> > Sean Christopherson <seanjc@google.com> wrote:
> > > Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()?  E.g.
> > > could we create a v3 GIC when possible, and fall back to v2?  And then provide a
> > > way for tests to express a hard v3 GIC dependency?
> > 
> > You can ask KVM what's available. Like an actual VMM does. There is no
> > shortage of examples in the current code base.
> 
> Right, by "sane" I meant: is there a way to instantiate a supported GIC without
> making it hard/painful to write tests, and without having to plumb in arm64
> specific requirements to common APIs?
> 
> E.g. are there tests that use the common vm_create() APIs and rely on NOT having
> a GIC?

Instead of stuffing a GIC in behind vm_create(), I'd rather we have a
specific helper for creating a VM with an irqchip. There's tests in
arm64 that rely on all this generic infrastructure and also need to
select / dimension a GIC appropriately for the test.

The majority of selftests don't even need an irqchip anyway.

Thanks,
Oliver

Re: [PATCH] KVM: selftests: fix irqfd_test on arm64

[Sean Christopherson]

On Mon, Aug 25, 2025, Oliver Upton wrote:
> On Mon, Aug 25, 2025 at 02:11:30PM -0700, Sean Christopherson wrote:
> > On Mon, Aug 25, 2025, Marc Zyngier wrote:
> > > On Mon, 25 Aug 2025 20:52:21 +0100,
> > > Sean Christopherson <seanjc@google.com> wrote:
> > > > Is there a sane way to handle vGIC creation in kvm_arch_vm_post_create()?  E.g.
> > > > could we create a v3 GIC when possible, and fall back to v2?  And then provide a
> > > > way for tests to express a hard v3 GIC dependency?
> > > 
> > > You can ask KVM what's available. Like an actual VMM does. There is no
> > > shortage of examples in the current code base.
> > 
> > Right, by "sane" I meant: is there a way to instantiate a supported GIC without
> > making it hard/painful to write tests, and without having to plumb in arm64
> > specific requirements to common APIs?
> > 
> > E.g. are there tests that use the common vm_create() APIs and rely on NOT having
> > a GIC?
> 
> Instead of stuffing a GIC in behind vm_create(), I'd rather we have a
> specific helper for creating a VM with an irqchip. There's tests in
> arm64 that rely on all this generic infrastructure and also need to
> select / dimension a GIC appropriately for the test.

I've no objection to arm64 providing an API for tests that need a specific GIC
configuration (I think it's a great idea), but I don't think that needs to be
mutually exclusive with having the common APIs instantiate _a_ GIC by default.

What if we use a global flag with thread local storage (because paranoia is
basically free, I think) to communicate that the test wants to create a v3 vGIC?

E.g.

---
static __thread bool vm_wants_custom_vgic;

void kvm_arch_vm_post_create(struct kvm_vm *vm)
{
	if (!vm_wants_custom_vgic) {
		<setup default vgic>
	}
}

struct kvm_vm *vm_create_with_vgic_v3(uint32_t nr_vcpus, void *guest_code,
				      uint32_t nr_irqs, struct kvm_vcpu **vcpus)
{
	TEST_ASSERT(!vm_wants_custom_vgic, "blah blah blah");
	vm_wants_custom_vgic = true;
	vm = vm_create_with_vcpus(nr_vcpus, guest_code, vcpus)
	vm_wants_custom_vgic = false;

	vgic_v3_setup(vm, nr_vcpus, nr_irqs);
}

struct kvm_vm *vm_create_with_one_vcpu_and_vgic_v3(struct kvm_vcpu **vcpu,
						   void *guest_code,
						   uint32_t nr_irqs)
{
	struct kvm_vcpu *vcpus[1];
	struct kvm_vm *vm;

	vm = vm_create_with_vgic_v3(1, guest_code, nr_irqs, vcpus);

	*vcpu = vcpus[0];
	return vm;
}
---

> The majority of selftests don't even need an irqchip anyway.

But it's really, really nice for developers if they can assume a certain level of
configuration is done by the infrastructure, i.e. don't have worry about doing
what is effectively "basic" VM setup.

E.g. x86 selftests creates an IRQCHIP, sets up descriptor tables and exception
handlers, and a handful of other "basic" things, and that has eliminated soooo
much boilerplate code and the associated friction with having to know/discover
that e.g. sending IRQs in a test requires additional setup beyond the obvious
steps like wiring up a handler.

Re: [PATCH] KVM: selftests: fix irqfd_test on arm64

[Oliver Upton]

On Tue, Aug 26, 2025 at 11:51:18AM -0700, Sean Christopherson wrote:
> On Mon, Aug 25, 2025, Oliver Upton wrote:
> > The majority of selftests don't even need an irqchip anyway.
> 
> But it's really, really nice for developers if they can assume a certain level of
> configuration is done by the infrastructure, i.e. don't have worry about doing
> what is effectively "basic" VM setup.

The more we pile behind what a "basic" VM configuration is the less
expressive the tests become. Being able to immediately grok the *intent*
of a test from reading it first pass is a very good thing. Otherwise I
need to go figure out what the definition of "basic" means when I need
to write a test and decide if that is compatible with what I'm trying to
do.

vm_create_with_irqchip() is delightfully unambiguous.

> E.g. x86 selftests creates an IRQCHIP, sets up descriptor tables and exception
> handlers, and a handful of other "basic" things, and that has eliminated soooo
> much boilerplate code and the associated friction with having to know/discover
> that e.g. sending IRQs in a test requires additional setup beyond the obvious
> steps like wiring up a handler.

That simply isn't going to happen on arm64. On top of the fact that the
irqchip configuration depends on the intent of the test (e.g. wired IRQs
v. MSIs), there's a bunch of guest-side initialization that needs to
happen too.

We can add an extremely barebones GIC when asked for (although guest
init isn't addressed) but batteries are not included on this architecture
and I'd rather not attempt to abstract that.

Thanks,
Oliver

Re: [PATCH] KVM: selftests: fix irqfd_test on arm64

[Sean Christopherson]

On Tue, Aug 26, 2025, Oliver Upton wrote:
> On Tue, Aug 26, 2025 at 11:51:18AM -0700, Sean Christopherson wrote:
> > On Mon, Aug 25, 2025, Oliver Upton wrote:
> > > The majority of selftests don't even need an irqchip anyway.
> > 
> > But it's really, really nice for developers if they can assume a certain level of
> > configuration is done by the infrastructure, i.e. don't have worry about doing
> > what is effectively "basic" VM setup.
> 
> The more we pile behind what a "basic" VM configuration is the less
> expressive the tests become. Being able to immediately grok the *intent*
> of a test from reading it first pass is a very good thing. Otherwise I
> need to go figure out what the definition of "basic" means when I need
> to write a test and decide if that is compatible with what I'm trying to
> do.

Eh, I don't buy that argument, not as a blanket statement.

The existence of code doesn't always communicate intent, e.g. the _only_ instance
I can think of where doing more setup by default caused problems was a few crusty
x86 tests that relied on an int3 to cause SHUTDOWN due to lack of an IDT.  OMG was
I increduluous when I figured out what those tests were doing.

And in that case, _not_ doing the "basic" setup hid the intent of the test.  Aside
from the fact that deliberately triggering SHUTDOWN was completely unnecessary in
those tests, IMO forcing such a test to use vm_create_barebones() would better
capture that the test is doing something odd, i.e. has unusual intent.

And explicitly doing something doesn't necessarily communicate the intent of the
test.  E.g. the intent of the irqfd_test is to verify that KVM_IRQFD assign and
deassign behaves as expected.  The test never generates IRQs, i.e. doesn't actually
need an IRQCHIP beyond satisfying KVM's requirements for KVM_IRQFD.

There are undoubtedly other tests that have similar "intent".  E.g. the in-progress
mediated PMU support for x86 requires an in-kernel local APIC, and so tests like
pmu_counters_test.c, pmu_event_filter_test.c, and vmx_pmu_caps_test.c will need
to instantiate an IRQCHIP.  None of those tests actually touch the local APIC in
any way, e.g. don't generate PMU interrupts, so creating an IRQCHIP is once again
nothing more than a means to an end, and not indicative of the test's main intent.

I think the use of vgic_v3_setup() in dirty_log_perf_test.c is also a case where
the existence of code fails to communicate intent.  Without the comment in
arch_setup_vm() to explain that having GICv3 somehow reduces the number of exits,
I would be very confused as to why the test cares about GICv3.

I agree there's a balance to be had in terms of doing too much.  Unfortunately in
this case, it sounds like the fundamental problem is that the balance is simply
different for x86 versus arm64.  Having an in-kernel local APIC is tables stakes
for x86, to the point where I'm looking for any excuse to have KVM create a local
APIC by default.  But for arm64, there's tremendous value in having tests do the
lifting.

> vm_create_with_irqchip() is delightfully unambiguous.
>
> > E.g. x86 selftests creates an IRQCHIP, sets up descriptor tables and exception
> > handlers, and a handful of other "basic" things, and that has eliminated soooo
> > much boilerplate code and the associated friction with having to know/discover
> > that e.g. sending IRQs in a test requires additional setup beyond the obvious
> > steps like wiring up a handler.
> 
> That simply isn't going to happen on arm64. On top of the fact that the
> irqchip configuration depends on the intent of the test (e.g. wired IRQs
> v. MSIs), there's a bunch of guest-side initialization that needs to
> happen too.
> 
> We can add an extremely barebones GIC when asked for (although guest
> init isn't addressed) but batteries are not included on this architecture
> and I'd rather not attempt to abstract that.

What about providing an API to do exactly that, instantiate and initialize a
barebones GIC?  E.g.

	void kvm_arch_init_barebones_irqchip(struct kvm_vm *vm)

Hmm, then we'd also need

	void kvm_arch_vm_free(struct kvm_vm *vm)

to gracefully free the GIC, as done by dirty_log_perf_test.c.  Blech.  Though
maybe we'll end up with that hook sooner or later?

All in all, I have no strong preference at this point.