* [PATCH 0/2] KVM: arm64: gic-v5: Fix irq_desc locks taken under the runqueue lock
@ 2026-11-16 7:13 Jia He
2026-11-16 7:13 ` [PATCH 1/2] KVM: arm64: Give the host timer interrupts a separate lockdep class Jia He
2026-11-16 7:13 ` [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out() Jia He
0 siblings, 2 replies; 5+ messages in thread
From: Jia He @ 2026-11-16 7:13 UTC (permalink / raw)
To: maz, oupton
Cc: fuad.tabba, joey.gouly, seiden, suzuki.poulose, yuzenghui,
bigeasy, linux-arm-kernel, kvmarm, linux-kernel, linux-rt-devel,
Jia He
With a GICv5 guest, kvm_sched_out() takes irq_desc locks with the
runqueue lock held in two places. Lockdep reports both as a circular
dependency with any interrupt that queues affinity notifier work under
its own descriptor lock:
- kvm_timer_vcpu_put() clears the active state of the host timer
interrupts. These are per-CPU and never threaded, so this one is a
false positive. Patch 1 gives them a separate lockdep class, as
commit 7100185bdc11 ("irqchip/gic-v4: Use a separate lockdep class for
vPE doorbell IRQs") in kvmarm/fixes does for the GICv4 vPE doorbells.
- vgic_v5_put() can move the vPE doorbell with irq_set_affinity() when
the vcpu thread is scheduled out in the middle of WFI. This came up in
the review of "KVM: arm64: gic-v5: Request doorbells when VPEs enter
WFI", where the resident check was expected to make it unreachable.
It is reachable, because vgic_v5_load() makes the vPE resident again
when the halting vcpu thread is scheduled back in. As the doorbell can
be force-threaded, this can wake its irq thread under the runqueue
lock, so patch 2 stops moving the doorbell from kvm_sched_out()
rather than hiding it from lockdep.
Tested on the Arm FVP Base RevC AEMvA GICv5 model (11.31.28, 8 CPUs,
VHE) with PROVE_LOCKING, on top of eedaa7d61f1e.
Lockdep reproduction: to provide the other half of the cycle, a test
module registers an affinity notifier on the UART interrupt and changes
its affinity a few times.
- Patch 1: the vgic_v5 selftest pinned to CPU0 next to a busy loop.
Without the patch, lockdep reports the cycle through
kvm_timer_vcpu_put(). With it there is no report, and &rq->__lock
depends on the new class instead of &irq_desc_lock_class.
- Patch 2: a vcpu executing WFI in a loop, woken by an SPI every 5ms,
with halt_poll_ns at 50ms and the vcpu thread moved to another CPU
every 1ms. A kprobe on irq_set_affinity() saw 11 calls from
kvm_sched_out() without the patch, together with the lockdep report
quoted in the commit message, and none with it. The doorbell is still
moved from kvm_vcpu_wfi() itself (51 and 61 calls).
PREEMPT_RT reproduction: PREEMPT_RT force-threads the vPE doorbell, so
patch 2 also fixes a real inversion. Without the patch, and without the
test module, the same WFI test makes lockdep report the cycle closed by
the doorbell's own irq thread: irq_set_affinity() from kvm_vcpu_wfi()
wakes the thread under the descriptor lock, while kvm_sched_out() takes
the descriptor lock under the runqueue lock. The model stops making
progress right after the report. With patch 2 there is no report and the
test completes. (Booting PREEMPT_RT on the GICv5 model also needs
"arm64: smp: Don't force-thread the GICv5 IPIs", posted separately.)
Jia He (2):
KVM: arm64: Give the host timer interrupts a separate lockdep class
KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out()
arch/arm64/kvm/arch_timer.c | 15 +++++++++++++++
arch/arm64/kvm/vgic/vgic-v5.c | 8 +++++++-
2 files changed, 22 insertions(+), 1 deletion(-)
base-commit: eedaa7d61f1ec8ae053a9ece0d553785350ac486
--
2.34.1
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] KVM: arm64: Give the host timer interrupts a separate lockdep class
2026-11-16 7:13 [PATCH 0/2] KVM: arm64: gic-v5: Fix irq_desc locks taken under the runqueue lock Jia He
@ 2026-11-16 7:13 ` Jia He
2026-11-16 7:13 ` [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out() Jia He
1 sibling, 0 replies; 5+ messages in thread
From: Jia He @ 2026-11-16 7:13 UTC (permalink / raw)
To: maz, oupton
Cc: fuad.tabba, joey.gouly, seiden, suzuki.poulose, yuzenghui,
bigeasy, linux-arm-kernel, kvmarm, linux-kernel, linux-rt-devel,
Jia He
With a GICv5 guest, kvm_timer_vcpu_put() clears the active state of the
host timer interrupts with irq_set_irqchip_state(), which takes their
irq_desc lock. On preemption this runs from kvm_sched_out() with the
runqueue lock held:
&rq->__lock --> &irq_desc_lock_class
By default, all irq descriptors share a single lockdep class, so this
closes a cycle with any interrupt that queues affinity notifier work
under its own descriptor lock (mlx5, for example), as already seen with
the GICv4 vPE doorbells:
&irq_desc_lock_class --> &pool->lock --> &p->pi_lock --> &rq->__lock
For the timer interrupts this is a false positive. Nothing that runs
under their descriptor lock leads back to the scheduler:
- They are requested only by KVM, with request_percpu_irq(). Per-CPU
interrupts are never threaded (irq_set_percpu_devid() sets
IRQ_NOTHREAD), so there is no irq thread to wake, unlike the vPE
doorbells with threadirqs.
- Their affinity is fixed and never changed, so no affinity notifier
work or thread affinity update is triggered.
- The irqchip callbacks reached under the lock (the KVM timer_chip,
forwarding to the GIC) only access GIC registers.
Give them a separate lockdep class. This is done for all vGIC types, as
none of the above is specific to GICv5. Both timers share the class, so
dependencies between them are still checked.
Fixes: 9491c63b6cd7 ("KVM: arm64: gic-v5: Enlighten arch timer for GICv5")
Signed-off-by: Jia He <justin.he@arm.com>
---
arch/arm64/kvm/arch_timer.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c
index b2c5ba499aa5..1088563d32a3 100644
--- a/arch/arm64/kvm/arch_timer.c
+++ b/arch/arm64/kvm/arch_timer.c
@@ -1362,6 +1362,17 @@ static void kvm_irq_fixup_flags(unsigned int virq, u32 *flags)
}
}
+/*
+ * With a GICv5 guest, the timer interrupts' irq_desc lock is taken under
+ * the runqueue lock (kvm_sched_out() -> kvm_timer_vcpu_put()). Give them
+ * their own lockdep class, so that they are not mixed up with interrupts
+ * whose affinity notifier queues work under the irq_desc lock. Nothing
+ * under the timers' own irq_desc lock leads back to the scheduler: they
+ * are per-CPU (never threaded) and their affinity never changes.
+ */
+static struct lock_class_key kvm_timer_irq_lock_class;
+static struct lock_class_key kvm_timer_irq_request_class;
+
static int kvm_irq_init(struct arch_timer_kvm_info *info)
{
struct irq_domain *domain = NULL;
@@ -1374,6 +1385,8 @@ static int kvm_irq_init(struct arch_timer_kvm_info *info)
host_vtimer_irq = info->virtual_irq;
kvm_irq_fixup_flags(host_vtimer_irq, &host_vtimer_irq_flags);
+ irq_set_lockdep_class(host_vtimer_irq, &kvm_timer_irq_lock_class,
+ &kvm_timer_irq_request_class);
if (kvm_vgic_global_state.no_hw_deactivation ||
kvm_vgic_global_state.type == VGIC_V5) {
@@ -1401,6 +1414,8 @@ static int kvm_irq_init(struct arch_timer_kvm_info *info)
if (info->physical_irq > 0) {
host_ptimer_irq = info->physical_irq;
kvm_irq_fixup_flags(host_ptimer_irq, &host_ptimer_irq_flags);
+ irq_set_lockdep_class(host_ptimer_irq, &kvm_timer_irq_lock_class,
+ &kvm_timer_irq_request_class);
if (domain)
WARN_ON(irq_domain_push_irq(domain, host_ptimer_irq,
--
2.34.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out()
2026-11-16 7:13 [PATCH 0/2] KVM: arm64: gic-v5: Fix irq_desc locks taken under the runqueue lock Jia He
2026-11-16 7:13 ` [PATCH 1/2] KVM: arm64: Give the host timer interrupts a separate lockdep class Jia He
@ 2026-11-16 7:13 ` Jia He
2026-11-16 8:07 ` sashiko-bot
1 sibling, 1 reply; 5+ messages in thread
From: Jia He @ 2026-11-16 7:13 UTC (permalink / raw)
To: maz, oupton
Cc: fuad.tabba, joey.gouly, seiden, suzuki.poulose, yuzenghui,
bigeasy, linux-arm-kernel, kvmarm, linux-kernel, linux-rt-devel,
Jia He
vgic_v5_put() makes the vPE doorbell affine to the current CPU when the
vcpu enters WFI. This is meant for the explicit put from kvm_vcpu_wfi(),
but the same code also runs from kvm_sched_out() whenever the vPE is
resident with IN_WFI set.
That happens when the vcpu thread is scheduled back in while halting:
unlike vgic_v4_load(), vgic_v5_load() does not bail out on IN_WFI and
makes the vPE resident again, so the next sched-out, from
kvm_vcpu_block() or from a preemption during halt polling, goes through
vgic_v5_put() with IN_WFI still set. If the thread moved to another
CPU in between, irq_set_affinity() is called with the runqueue lock
held:
irq_set_affinity+0x38/0x88
vgic_v5_put+0xf0/0x2b8
kvm_vgic_put+0x64/0xb0
vhe_vcpu_put+0x40/0xa0
kvm_arch_vcpu_put+0x28/0x98
kvm_sched_out+0x4c/0x70
prepare_task_switch+0x1b0/0x388
__schedule+0x234/0x958
schedule+0x58/0x138
kvm_vcpu_block+0x88/0x130
kvm_vcpu_halt+0x120/0x468
kvm_vcpu_wfi+0x80/0x190
irq_set_affinity() takes the doorbell's irq_desc lock and, when the
doorbell is force-threaded (threadirqs, PREEMPT_RT), wakes its irq
thread:
&rq->__lock --> &irq_desc_lock_class --> &p->pi_lock --> &rq->__lock
Moving the doorbell is only an optimisation: the doorbell handler kicks
the vcpu from any CPU, and the next WFI entry moves it again. Skip it
when vcpu->scheduled_out is set, which kvm_vcpu_put_hw_mmu() also uses
to tell a sched-out from an explicit put.
Fixes: 2111261ac1c8 ("KVM: arm64: gic-v5: Request doorbells when VPEs enter WFI")
Signed-off-by: Jia He <justin.he@arm.com>
---
arch/arm64/kvm/vgic/vgic-v5.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/arch/arm64/kvm/vgic/vgic-v5.c b/arch/arm64/kvm/vgic/vgic-v5.c
index 91810fbd99af..5824b00bb50d 100644
--- a/arch/arm64/kvm/vgic/vgic-v5.c
+++ b/arch/arm64/kvm/vgic/vgic-v5.c
@@ -1248,8 +1248,14 @@ void vgic_v5_put(struct kvm_vcpu *vcpu)
* Make the doorbell affine to this CPU, if it isn't
* already. Actively check the cpumask first as it is
* cheaper than changing the affinity every time.
+ *
+ * Don't do it from kvm_sched_out(), which holds the
+ * runqueue lock: irq_set_affinity() takes the
+ * doorbell's irq_desc lock and may wake its irq
+ * thread. The doorbell works from any CPU, and the
+ * next WFI entry moves it.
*/
- if (!cpumask_test_cpu(cpu, aff)) {
+ if (!vcpu->scheduled_out && !cpumask_test_cpu(cpu, aff)) {
int ret;
ret = irq_set_affinity(db_irq, cpumask_of(cpu));
--
2.34.1
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out()
2026-11-16 7:13 ` [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out() Jia He
@ 2026-11-16 8:07 ` sashiko-bot
2026-11-16 9:37 ` Justin He
0 siblings, 1 reply; 5+ messages in thread
From: sashiko-bot @ 2026-11-16 8:07 UTC (permalink / raw)
To: Jia He; +Cc: kvmarm, linux-rt-devel, Oliver Upton, Marc Zyngier, linux-kernel
Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] KVM: arm64: gic-v5: vgic_v5_load() in WFI disarms doorbells
and loses wakeups
--
commit 36844cdbda4df2b177b6c013c2074287d75ff5f3
Author: Jia He <justin.he@arm.com>
KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out()
This commit skips calling irq_set_affinity() in vgic_v5_put() when
vcpu->scheduled_out is set, avoiding acquiring irq_desc and pi_lock under the
runqueue lock during kvm_sched_out().
> diff --git a/arch/arm64/kvm/vgic/vgic-v5.c b/arch/arm64/kvm/vgic/vgic-v5.c
> index 91810fbd99afa..5824b00bb50df 100644
> --- a/arch/arm64/kvm/vgic/vgic-v5.c
> +++ b/arch/arm64/kvm/vgic/vgic-v5.c
> @@ -1248,8 +1248,14 @@ void vgic_v5_put(struct kvm_vcpu *vcpu)
> * Make the doorbell affine to this CPU, if it isn't
> * already. Actively check the cpumask first as it is
> * cheaper than changing the affinity every time.
> + *
> + * Don't do it from kvm_sched_out(), which holds the
> + * runqueue lock: irq_set_affinity() takes the
> + * doorbell's irq_desc lock and may wake its irq
> + * thread. The doorbell works from any CPU, and the
> + * next WFI entry moves it.
> */
> - if (!cpumask_test_cpu(cpu, aff)) {
> + if (!vcpu->scheduled_out && !cpumask_test_cpu(cpu, aff)) {
> int ret;
>
> ret = irq_set_affinity(db_irq, cpumask_of(cpu));
[Severity: Medium]
Should vgic_v5_load() bail out when IN_WFI is set instead of only skipping
irq_set_affinity() here?
Because vgic_v5_load() does not check vcpu_get_flag(vcpu, IN_WFI), any
kvm_sched_in() while the vCPU is halting in kvm_vcpu_halt() calls
__vgic_v5_make_resident(), setting gicv5_vpe.resident = true and clearing
ICH_CONTEXTR_EL2_DB in hardware.
If the vPE doorbell fires while blocked in kvm_vcpu_block(), db_handler()
sets gicv5_vpe.db_fired = true and wakes the vCPU thread. When the thread
schedules in, vgic_v5_load() makes the vPE resident before schedule() returns
to the loop in kvm_vcpu_block(). If the vCPU thread is preempted again before
kvm_vcpu_check_block() consumes db_fired, kvm_sched_out() -> vgic_v5_put()
still calls __vgic_v5_make_non_resident():
arch/arm64/kvm/hyp/vgic-v5-sr.c:__vgic_v5_make_non_resident() {
...
WRITE_ONCE(cpu_if->gicv5_vpe.db_fired, false);
cpu_if->gicv5_vpe.resident = false;
}
When the thread schedules back in, kvm_vcpu_check_block() ->
kvm_vgic_vcpu_pending_irq() sees db_fired == false and goes back to sleep in
schedule(), losing the wakeup.
Making the vPE resident on schedule-in during halt-polling also disarms the
hardware doorbell for the rest of the poll window, so newly pending SPIs and
LPIs do not set db_fired, and a subsequent kvm_sched_out() still runs the EL2
residency hypercalls and vgic_v5_sync_ppi_priorities() under the runqueue
lock. Returning early from vgic_v5_load() when IN_WFI is set keeps
gicv5_vpe.resident false throughout kvm_vcpu_halt(), so kvm_sched_out()
returns immediately in vgic_v5_put().
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261116071304.2037895-1-justin.he@arm.com?part=2
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out()
2026-11-16 8:07 ` sashiko-bot
@ 2026-11-16 9:37 ` Justin He
0 siblings, 0 replies; 5+ messages in thread
From: Justin He @ 2026-11-16 9:37 UTC (permalink / raw)
To: sashiko-reviews@lists.linux.dev
Cc: kvmarm@lists.linux.dev, linux-rt-devel@lists.linux.dev,
Oliver Upton, Marc Zyngier, linux-kernel@vger.kernel.org
> -----Original Message-----
> From: sashiko-bot@kernel.org <sashiko-bot@kernel.org>
> Sent: Monday, November 16, 2026 4:07 PM
> To: Justin He <Justin.He@arm.com>
> Cc: kvmarm@lists.linux.dev; linux-rt-devel@lists.linux.dev; Oliver Upton
> <oupton@kernel.org>; Marc Zyngier <maz@kernel.org>; linux-
> kernel@vger.kernel.org
> Subject: Re: [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell
> from kvm_sched_out()
>
> Thank you for your contribution! Sashiko AI review found 1 potential
> issue to consider:
> - [Medium] KVM: arm64: gic-v5: vgic_v5_load() in WFI disarms doorbells
> and loses wakeups
> --
>
> commit 36844cdbda4df2b177b6c013c2074287d75ff5f3
> Author: Jia He <justin.he@arm.com>
>
> KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out()
>
> This commit skips calling irq_set_affinity() in vgic_v5_put() when
> vcpu->scheduled_out is set, avoiding acquiring irq_desc and pi_lock under
> the
> runqueue lock during kvm_sched_out().
>
> > diff --git a/arch/arm64/kvm/vgic/vgic-v5.c b/arch/arm64/kvm/vgic/vgic-v5.c
> > index 91810fbd99afa..5824b00bb50df 100644
> > --- a/arch/arm64/kvm/vgic/vgic-v5.c
> > +++ b/arch/arm64/kvm/vgic/vgic-v5.c
> > @@ -1248,8 +1248,14 @@ void vgic_v5_put(struct kvm_vcpu *vcpu)
> > * Make the doorbell affine to this CPU, if it isn't
> > * already. Actively check the cpumask first as it is
> > * cheaper than changing the affinity every time.
> > + *
> > + * Don't do it from kvm_sched_out(), which holds the
> > + * runqueue lock: irq_set_affinity() takes the
> > + * doorbell's irq_desc lock and may wake its irq
> > + * thread. The doorbell works from any CPU, and the
> > + * next WFI entry moves it.
> > */
> > - if (!cpumask_test_cpu(cpu, aff)) {
> > + if (!vcpu->scheduled_out && !cpumask_test_cpu(cpu,
> aff)) {
> > int ret;
> >
> > ret = irq_set_affinity(db_irq,
> cpumask_of(cpu));
>
> [Severity: Medium]
> Should vgic_v5_load() bail out when IN_WFI is set instead of only skipping
> irq_set_affinity() here?
>
> Because vgic_v5_load() does not check vcpu_get_flag(vcpu, IN_WFI), any
> kvm_sched_in() while the vCPU is halting in kvm_vcpu_halt() calls
> __vgic_v5_make_resident(), setting gicv5_vpe.resident = true and clearing
> ICH_CONTEXTR_EL2_DB in hardware.
>
> If the vPE doorbell fires while blocked in kvm_vcpu_block(), db_handler()
> sets gicv5_vpe.db_fired = true and wakes the vCPU thread. When the thread
> schedules in, vgic_v5_load() makes the vPE resident before schedule() returns
> to the loop in kvm_vcpu_block(). If the vCPU thread is preempted again
> before
> kvm_vcpu_check_block() consumes db_fired, kvm_sched_out() ->
> vgic_v5_put()
> still calls __vgic_v5_make_non_resident():
>
> arch/arm64/kvm/hyp/vgic-v5-sr.c:__vgic_v5_make_non_resident() {
> ...
> WRITE_ONCE(cpu_if->gicv5_vpe.db_fired, false);
> cpu_if->gicv5_vpe.resident = false;
> }
>
> When the thread schedules back in, kvm_vcpu_check_block() ->
> kvm_vgic_vcpu_pending_irq() sees db_fired == false and goes back to sleep in
> schedule(), losing the wakeup.
>
> Making the vPE resident on schedule-in during halt-polling also disarms the
> hardware doorbell for the rest of the poll window, so newly pending SPIs and
> LPIs do not set db_fired, and a subsequent kvm_sched_out() still runs the EL2
> residency hypercalls and vgic_v5_sync_ppi_priorities() under the runqueue
> lock. Returning early from vgic_v5_load() when IN_WFI is set keeps
> gicv5_vpe.resident false throughout kvm_vcpu_halt(), so kvm_sched_out()
> returns immediately in vgic_v5_put().
But IIUC, the wakeup isn't lost. The second put requests the doorbell
again. The doorbell can be generated for an interrupt that is already
pending when the vPE becomes non-resident, not only for a new pending
interrupt (IKZXCY in IHI 111701). The guest hasn't run in between, so
the interrupt is still pending and the priority mask hasn't changed.
db_fired is cleared before requesting the doorbell again, so the new
doorbell isn't missed. Am I missing sth else?
For the halt-polling case, I agree there is an issue. After a sched-in,
nothing requests the doorbell until the next sched-out. Making
vgic_v5_load() return early on IN_WFI, like v4 does, should fix this.
But since this is an optimisation, I prefer to leave it for a follow-up,
unless the maintainers want it in this patch.
___
Best Regards,
Jia He (Justin)
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-16 9:37 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-16 7:13 [PATCH 0/2] KVM: arm64: gic-v5: Fix irq_desc locks taken under the runqueue lock Jia He
2026-11-16 7:13 ` [PATCH 1/2] KVM: arm64: Give the host timer interrupts a separate lockdep class Jia He
2026-11-16 7:13 ` [PATCH 2/2] KVM: arm64: gic-v5: Don't move the vPE doorbell from kvm_sched_out() Jia He
2026-11-16 8:07 ` sashiko-bot
2026-11-16 9:37 ` Justin He
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox