[PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
with related guest support enabled:

[    1.127941] hv_vmbus: registering driver hyperv_drm

[    1.132518] =============================
[    1.132519] [ BUG: Invalid wait context ]
[    1.132521] 6.19.0-rc8+ #9 Not tainted
[    1.132524] -----------------------------
[    1.132525] swapper/0/0 is trying to lock:
[    1.132526] ffff8b9381bb3c90 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0xc4/0x2b0
[    1.132543] other info that might help us debug this:
[    1.132544] context-{2:2}
[    1.132545] 1 lock held by swapper/0/0:
[    1.132547]  #0: ffffffffa010c4c0 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0x31/0x2b0
[    1.132557] stack backtrace:
[    1.132560] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.19.0-rc8+ #9 PREEMPT_{RT,(lazy)}
[    1.132565] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/25/2025
[    1.132567] Call Trace:
[    1.132570]  <IRQ>
[    1.132573]  dump_stack_lvl+0x6e/0xa0
[    1.132581]  __lock_acquire+0xee0/0x21b0
[    1.132592]  lock_acquire+0xd5/0x2d0
[    1.132598]  ? vmbus_chan_sched+0xc4/0x2b0
[    1.132606]  ? lock_acquire+0xd5/0x2d0
[    1.132613]  ? vmbus_chan_sched+0x31/0x2b0
[    1.132619]  rt_spin_lock+0x3f/0x1f0
[    1.132623]  ? vmbus_chan_sched+0xc4/0x2b0
[    1.132629]  ? vmbus_chan_sched+0x31/0x2b0
[    1.132634]  vmbus_chan_sched+0xc4/0x2b0
[    1.132641]  vmbus_isr+0x2c/0x150
[    1.132648]  __sysvec_hyperv_callback+0x5f/0xa0
[    1.132654]  sysvec_hyperv_callback+0x88/0xb0
[    1.132658]  </IRQ>
[    1.132659]  <TASK>
[    1.132660]  asm_sysvec_hyperv_callback+0x1a/0x20

As code paths that handle vmbus IRQs use sleepy locks under PREEMPT_RT,
the complete vmbus_handler execution needs to be moved into thread
context. Open-coding this allows to skip the IPI that irq_work would
additionally bring and which we do not need, being an IRQ, never an NMI.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

This should resolve what was once brought forward via [1]. If it 
actually resolves all remaining compatibility issues of the hyperv 
support with RT is not yet clear, though. So far, lockdep is happy when 
using this plus [2].

[1] https://lore.kernel.org/all/20230809-b4-rt_preempt-fix-v1-0-7283bbdc8b14@gmail.com/
[2] https://lore.kernel.org/lkml/0c7fb5cd-fb21-4760-8593-e04bade84744@siemens.com/

 arch/x86/kernel/cpu/mshyperv.c | 52 ++++++++++++++++++++++++++++++++--
 1 file changed, 50 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
index 579fb2c64cfd..1194ca452c52 100644
--- a/arch/x86/kernel/cpu/mshyperv.c
+++ b/arch/x86/kernel/cpu/mshyperv.c
@@ -17,6 +17,7 @@
 #include <linux/irq.h>
 #include <linux/kexec.h>
 #include <linux/random.h>
+#include <linux/smpboot.h>
 #include <asm/processor.h>
 #include <asm/hypervisor.h>
 #include <hyperv/hvhdk.h>
@@ -150,6 +151,43 @@ static void (*hv_stimer0_handler)(void);
 static void (*hv_kexec_handler)(void);
 static void (*hv_crash_handler)(struct pt_regs *regs);
 
+static DEFINE_PER_CPU(bool, vmbus_irq_pending);
+static DEFINE_PER_CPU(struct task_struct *, vmbus_irqd);
+
+static void vmbus_irqd_wake(void)
+{
+	struct task_struct *tsk = __this_cpu_read(vmbus_irqd);
+
+	__this_cpu_write(vmbus_irq_pending, true);
+	wake_up_process(tsk);
+}
+
+static void vmbus_irqd_setup(unsigned int cpu)
+{
+	sched_set_fifo(current);
+}
+
+static int vmbus_irqd_should_run(unsigned int cpu)
+{
+	return __this_cpu_read(vmbus_irq_pending);
+}
+
+static void run_vmbus_irqd(unsigned int cpu)
+{
+	vmbus_handler();
+	__this_cpu_write(vmbus_irq_pending, false);
+}
+
+static bool vmbus_irq_initialized;
+
+static struct smp_hotplug_thread vmbus_irq_threads = {
+	.store                  = &vmbus_irqd,
+	.setup			= vmbus_irqd_setup,
+	.thread_should_run      = vmbus_irqd_should_run,
+	.thread_fn              = run_vmbus_irqd,
+	.thread_comm            = "vmbus_irq/%u",
+};
+
 DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
 {
 	struct pt_regs *old_regs = set_irq_regs(regs);
@@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
 	if (mshv_handler)
 		mshv_handler();
 
-	if (vmbus_handler)
-		vmbus_handler();
+	if (vmbus_handler) {
+		if (IS_ENABLED(CONFIG_PREEMPT_RT))
+			vmbus_irqd_wake();
+		else
+			vmbus_handler();
+	}
 
 	if (ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED)
 		apic_eoi();
@@ -174,6 +216,10 @@ void hv_setup_mshv_handler(void (*handler)(void))
 
 void hv_setup_vmbus_handler(void (*handler)(void))
 {
+	if (IS_ENABLED(CONFIG_PREEMPT_RT) && !vmbus_irq_initialized) {
+		BUG_ON(smpboot_register_percpu_thread(&vmbus_irq_threads));
+		vmbus_irq_initialized = true;
+	}
 	vmbus_handler = handler;
 }
 
@@ -181,6 +227,8 @@ void hv_remove_vmbus_handler(void)
 {
 	/* We have no way to deallocate the interrupt gate */
 	vmbus_handler = NULL;
+	smpboot_unregister_percpu_thread(&vmbus_irq_threads);
+	vmbus_irq_initialized = false;
 }
 
 /*
-- 
2.51.0

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Wei Liu]

On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
> with related guest support enabled:

So all it takes to reproduce this is to enabled PREEMPT_RT?

Asking because ...

>  	struct pt_regs *old_regs = set_irq_regs(regs);
> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>  	if (mshv_handler)
>  		mshv_handler();

... to err on the safe side we should probably do the same for
mshv_handler as well.

Note we don't support RT yet, but if issues are found we might as well
just fix them up.

How urgent do you want this patch to get applied?

Thanks,
Wei

>  
> -	if (vmbus_handler)
> -		vmbus_handler();
> +	if (vmbus_handler) {
> +		if (IS_ENABLED(CONFIG_PREEMPT_RT))
> +			vmbus_irqd_wake();
> +		else
> +			vmbus_handler();
> +	}
>  
>  	if (ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED)
>  		apic_eoi();
> @@ -174,6 +216,10 @@ void hv_setup_mshv_handler(void (*handler)(void))
>  
>  void hv_setup_vmbus_handler(void (*handler)(void))
>  {
> +	if (IS_ENABLED(CONFIG_PREEMPT_RT) && !vmbus_irq_initialized) {
> +		BUG_ON(smpboot_register_percpu_thread(&vmbus_irq_threads));
> +		vmbus_irq_initialized = true;
> +	}
>  	vmbus_handler = handler;
>  }
>  
> @@ -181,6 +227,8 @@ void hv_remove_vmbus_handler(void)
>  {
>  	/* We have no way to deallocate the interrupt gate */
>  	vmbus_handler = NULL;
> +	smpboot_unregister_percpu_thread(&vmbus_irq_threads);
> +	vmbus_irq_initialized = false;
>  }
>  
>  /*
> -- 
> 2.51.0

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 04.02.26 08:00, Wei Liu wrote:
> On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
>> with related guest support enabled:
> 
> So all it takes to reproduce this is to enabled PREEMPT_RT?
> 

...and enable CONFIG_PROVE_LOCKING so that you do not have to wait for
your system to actually run into the bug. Lockdep already triggers
during bootup.

> Asking because ...
> 
>>  	struct pt_regs *old_regs = set_irq_regs(regs);
>> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>>  	if (mshv_handler)
>>  		mshv_handler();
> 
> ... to err on the safe side we should probably do the same for
> mshv_handler as well.
> 

Valid question. We so far worked based on lockdep reports, and the
mshv_handler didn't trigger yet. Either it is not run in our setup, or
it is actually already fine. But I have a code review on my agenda
regarding potential remaining issues in mshv.

Is there something needed to trigger the mshv_handler so that we can
test it?

> Note we don't support RT yet, but if issues are found we might as well
> just fix them up.

This is actually not about RT itself but about supporting all
configurable locking semantics of the. And the mshv drivers fail here.

> 
> How urgent do you want this patch to get applied?

If I asked my folks: yesterday (we shipped it...).

We would also need in upstream stable and stable-rt, though it may not
reach the current production kernel anymore (6.1-rt from bookworm)
because it reaching EOL in a few months.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 04.02.26 08:19, Jan Kiszka wrote:
> On 04.02.26 08:00, Wei Liu wrote:
>> On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>
>>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
>>> with related guest support enabled:
>>
>> So all it takes to reproduce this is to enabled PREEMPT_RT?
>>
> 
> ...and enable CONFIG_PROVE_LOCKING so that you do not have to wait for
> your system to actually run into the bug. Lockdep already triggers
> during bootup.
> 
>> Asking because ...
>>
>>>  	struct pt_regs *old_regs = set_irq_regs(regs);
>>> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>>>  	if (mshv_handler)
>>>  		mshv_handler();
>>
>> ... to err on the safe side we should probably do the same for
>> mshv_handler as well.
>>
> 
> Valid question. We so far worked based on lockdep reports, and the
> mshv_handler didn't trigger yet. Either it is not run in our setup, or
> it is actually already fine. But I have a code review on my agenda
> regarding potential remaining issues in mshv.
> 
> Is there something needed to trigger the mshv_handler so that we can
> test it?
> 

Ah, that depends on CONFIG_MSHV_ROOT. Is that related to the accelerator
mode that Magnus presented in [1]? We briefly chatted about it and also
my problems with the drivers after his talk on Saturday.

Jan

[1]
https://fosdem.org/2026/schedule/event/BFQ8XA-introducing-mshv-accelerator-in-qemu/

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Wei Liu]

On Wed, Feb 04, 2026 at 08:26:48AM +0100, Jan Kiszka wrote:
> On 04.02.26 08:19, Jan Kiszka wrote:
> > On 04.02.26 08:00, Wei Liu wrote:
> >> On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
> >>> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>>
> >>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
> >>> with related guest support enabled:
> >>
> >> So all it takes to reproduce this is to enabled PREEMPT_RT?
> >>
> > 
> > ...and enable CONFIG_PROVE_LOCKING so that you do not have to wait for
> > your system to actually run into the bug. Lockdep already triggers
> > during bootup.
> > 
> >> Asking because ...
> >>
> >>>  	struct pt_regs *old_regs = set_irq_regs(regs);
> >>> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
> >>>  	if (mshv_handler)
> >>>  		mshv_handler();
> >>
> >> ... to err on the safe side we should probably do the same for
> >> mshv_handler as well.
> >>
> > 
> > Valid question. We so far worked based on lockdep reports, and the
> > mshv_handler didn't trigger yet. Either it is not run in our setup, or
> > it is actually already fine. But I have a code review on my agenda
> > regarding potential remaining issues in mshv.
> > 
> > Is there something needed to trigger the mshv_handler so that we can
> > test it?
> > 
> 
> Ah, that depends on CONFIG_MSHV_ROOT. Is that related to the accelerator
> mode that Magnus presented in [1]? We briefly chatted about it and also
> my problems with the drivers after his talk on Saturday.

Yes. That is the driver. If PROVE_LOCKING triggers the warning without
running the code, perhaps turning on MSHV_ROOT is enough.

Wei

> 
> Jan
> 
> [1]
> https://fosdem.org/2026/schedule/event/BFQ8XA-introducing-mshv-accelerator-in-qemu/
> 
> -- 
> Siemens AG, Foundational Technologies
> Linux Expert Center
> 

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 04.02.26 08:29, Wei Liu wrote:
> On Wed, Feb 04, 2026 at 08:26:48AM +0100, Jan Kiszka wrote:
>> On 04.02.26 08:19, Jan Kiszka wrote:
>>> On 04.02.26 08:00, Wei Liu wrote:
>>>> On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>
>>>>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
>>>>> with related guest support enabled:
>>>>
>>>> So all it takes to reproduce this is to enabled PREEMPT_RT?
>>>>
>>>
>>> ...and enable CONFIG_PROVE_LOCKING so that you do not have to wait for
>>> your system to actually run into the bug. Lockdep already triggers
>>> during bootup.
>>>
>>>> Asking because ...
>>>>
>>>>>  	struct pt_regs *old_regs = set_irq_regs(regs);
>>>>> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>>>>>  	if (mshv_handler)
>>>>>  		mshv_handler();
>>>>
>>>> ... to err on the safe side we should probably do the same for
>>>> mshv_handler as well.
>>>>
>>>
>>> Valid question. We so far worked based on lockdep reports, and the
>>> mshv_handler didn't trigger yet. Either it is not run in our setup, or
>>> it is actually already fine. But I have a code review on my agenda
>>> regarding potential remaining issues in mshv.
>>>
>>> Is there something needed to trigger the mshv_handler so that we can
>>> test it?
>>>
>>
>> Ah, that depends on CONFIG_MSHV_ROOT. Is that related to the accelerator
>> mode that Magnus presented in [1]? We briefly chatted about it and also
>> my problems with the drivers after his talk on Saturday.
> 
> Yes. That is the driver. If PROVE_LOCKING triggers the warning without
> running the code, perhaps turning on MSHV_ROOT is enough.
> 

But if my VM is not a root partition, I wouldn't use that driver, would I?

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Wei Liu]

On Wed, Feb 04, 2026 at 08:32:04AM +0100, Jan Kiszka wrote:
> On 04.02.26 08:29, Wei Liu wrote:
> > On Wed, Feb 04, 2026 at 08:26:48AM +0100, Jan Kiszka wrote:
> >> On 04.02.26 08:19, Jan Kiszka wrote:
> >>> On 04.02.26 08:00, Wei Liu wrote:
> >>>> On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
> >>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
> >>>>>
> >>>>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
> >>>>> with related guest support enabled:
> >>>>
> >>>> So all it takes to reproduce this is to enabled PREEMPT_RT?
> >>>>
> >>>
> >>> ...and enable CONFIG_PROVE_LOCKING so that you do not have to wait for
> >>> your system to actually run into the bug. Lockdep already triggers
> >>> during bootup.
> >>>
> >>>> Asking because ...
> >>>>
> >>>>>  	struct pt_regs *old_regs = set_irq_regs(regs);
> >>>>> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
> >>>>>  	if (mshv_handler)
> >>>>>  		mshv_handler();
> >>>>
> >>>> ... to err on the safe side we should probably do the same for
> >>>> mshv_handler as well.
> >>>>
> >>>
> >>> Valid question. We so far worked based on lockdep reports, and the
> >>> mshv_handler didn't trigger yet. Either it is not run in our setup, or
> >>> it is actually already fine. But I have a code review on my agenda
> >>> regarding potential remaining issues in mshv.
> >>>
> >>> Is there something needed to trigger the mshv_handler so that we can
> >>> test it?
> >>>
> >>
> >> Ah, that depends on CONFIG_MSHV_ROOT. Is that related to the accelerator
> >> mode that Magnus presented in [1]? We briefly chatted about it and also
> >> my problems with the drivers after his talk on Saturday.
> > 
> > Yes. That is the driver. If PROVE_LOCKING triggers the warning without
> > running the code, perhaps turning on MSHV_ROOT is enough.
> > 
> 
> But if my VM is not a root partition, I wouldn't use that driver, would I?

No, you wouldn't.  You cannot do that until later this year. If you
cannot test that, so be it. I'm fine with applying your patch and then
move the mshv_handler logic later ourselves.

I've CC'ed a few folks from Microsoft.

Saurabh, Long, and Dexuan, can you review and test this patch for VMBus?

Thanks,
Wei

> 
> Jan
> 
> -- 
> Siemens AG, Foundational Technologies
> Linux Expert Center
> 

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 04.02.26 08:36, Wei Liu wrote:
> On Wed, Feb 04, 2026 at 08:32:04AM +0100, Jan Kiszka wrote:
>> On 04.02.26 08:29, Wei Liu wrote:
>>> On Wed, Feb 04, 2026 at 08:26:48AM +0100, Jan Kiszka wrote:
>>>> On 04.02.26 08:19, Jan Kiszka wrote:
>>>>> On 04.02.26 08:00, Wei Liu wrote:
>>>>>> On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>
>>>>>>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
>>>>>>> with related guest support enabled:
>>>>>>
>>>>>> So all it takes to reproduce this is to enabled PREEMPT_RT?
>>>>>>
>>>>>
>>>>> ...and enable CONFIG_PROVE_LOCKING so that you do not have to wait for
>>>>> your system to actually run into the bug. Lockdep already triggers
>>>>> during bootup.
>>>>>
>>>>>> Asking because ...
>>>>>>
>>>>>>>  	struct pt_regs *old_regs = set_irq_regs(regs);
>>>>>>> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>>>>>>>  	if (mshv_handler)
>>>>>>>  		mshv_handler();
>>>>>>
>>>>>> ... to err on the safe side we should probably do the same for
>>>>>> mshv_handler as well.
>>>>>>
>>>>>
>>>>> Valid question. We so far worked based on lockdep reports, and the
>>>>> mshv_handler didn't trigger yet. Either it is not run in our setup, or
>>>>> it is actually already fine. But I have a code review on my agenda
>>>>> regarding potential remaining issues in mshv.
>>>>>
>>>>> Is there something needed to trigger the mshv_handler so that we can
>>>>> test it?
>>>>>
>>>>
>>>> Ah, that depends on CONFIG_MSHV_ROOT. Is that related to the accelerator
>>>> mode that Magnus presented in [1]? We briefly chatted about it and also
>>>> my problems with the drivers after his talk on Saturday.
>>>
>>> Yes. That is the driver. If PROVE_LOCKING triggers the warning without
>>> running the code, perhaps turning on MSHV_ROOT is enough.
>>>
>>
>> But if my VM is not a root partition, I wouldn't use that driver, would I?
> 
> No, you wouldn't.  You cannot do that until later this year. If you
> cannot test that, so be it. I'm fine with applying your patch and then
> move the mshv_handler logic later ourselves.

Based on my review, I bet you have to lift the mshv_handler to a thread
as well, e.g. sysvec_hyperv_callback ... -> kick_vp -> wake_up.

OTOH, the probability that someone tries to use PREEMPT_RT as root Linux
is likely even lower than someone trying it as a normal guest. IMHO, you
could also consider to rule out that combination at Kconfig level.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Naman Jain]

On 2/4/2026 1:06 PM, Wei Liu wrote:
> On Wed, Feb 04, 2026 at 08:32:04AM +0100, Jan Kiszka wrote:
>> On 04.02.26 08:29, Wei Liu wrote:
>>> On Wed, Feb 04, 2026 at 08:26:48AM +0100, Jan Kiszka wrote:
>>>> On 04.02.26 08:19, Jan Kiszka wrote:
>>>>> On 04.02.26 08:00, Wei Liu wrote:
>>>>>> On Tue, Feb 03, 2026 at 05:01:30PM +0100, Jan Kiszka wrote:
>>>>>>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>>>>>>
>>>>>>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
>>>>>>> with related guest support enabled:
>>>>>>
>>>>>> So all it takes to reproduce this is to enabled PREEMPT_RT?
>>>>>>
>>>>>
>>>>> ...and enable CONFIG_PROVE_LOCKING so that you do not have to wait for
>>>>> your system to actually run into the bug. Lockdep already triggers
>>>>> during bootup.
>>>>>
>>>>>> Asking because ...
>>>>>>
>>>>>>>   	struct pt_regs *old_regs = set_irq_regs(regs);
>>>>>>> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>>>>>>>   	if (mshv_handler)
>>>>>>>   		mshv_handler();
>>>>>>
>>>>>> ... to err on the safe side we should probably do the same for
>>>>>> mshv_handler as well.
>>>>>>
>>>>>
>>>>> Valid question. We so far worked based on lockdep reports, and the
>>>>> mshv_handler didn't trigger yet. Either it is not run in our setup, or
>>>>> it is actually already fine. But I have a code review on my agenda
>>>>> regarding potential remaining issues in mshv.
>>>>>
>>>>> Is there something needed to trigger the mshv_handler so that we can
>>>>> test it?
>>>>>
>>>>
>>>> Ah, that depends on CONFIG_MSHV_ROOT. Is that related to the accelerator
>>>> mode that Magnus presented in [1]? We briefly chatted about it and also
>>>> my problems with the drivers after his talk on Saturday.
>>>
>>> Yes. That is the driver. If PROVE_LOCKING triggers the warning without
>>> running the code, perhaps turning on MSHV_ROOT is enough.
>>>
>>
>> But if my VM is not a root partition, I wouldn't use that driver, would I?
> 
> No, you wouldn't.  You cannot do that until later this year. If you
> cannot test that, so be it. I'm fine with applying your patch and then
> move the mshv_handler logic later ourselves.
> 
> I've CC'ed a few folks from Microsoft.
> 
> Saurabh, Long, and Dexuan, can you review and test this patch for VMBus?


I tested this and didn't see any issues with OpenHCL/mshv_vtl.

Regards,
Naman

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Bezdeka, Florian]

On Tue, 2026-02-03 at 17:01 +0100, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
> with related guest support enabled:
> 
> [    1.127941] hv_vmbus: registering driver hyperv_drm
> 
> [    1.132518] =============================
> [    1.132519] [ BUG: Invalid wait context ]
> [    1.132521] 6.19.0-rc8+ #9 Not tainted
> [    1.132524] -----------------------------
> [    1.132525] swapper/0/0 is trying to lock:
> [    1.132526] ffff8b9381bb3c90 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0xc4/0x2b0
> [    1.132543] other info that might help us debug this:
> [    1.132544] context-{2:2}
> [    1.132545] 1 lock held by swapper/0/0:
> [    1.132547]  #0: ffffffffa010c4c0 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0x31/0x2b0
> [    1.132557] stack backtrace:
> [    1.132560] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.19.0-rc8+ #9 PREEMPT_{RT,(lazy)}
> [    1.132565] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/25/2025
> [    1.132567] Call Trace:
> [    1.132570]  <IRQ>
> [    1.132573]  dump_stack_lvl+0x6e/0xa0
> [    1.132581]  __lock_acquire+0xee0/0x21b0
> [    1.132592]  lock_acquire+0xd5/0x2d0
> [    1.132598]  ? vmbus_chan_sched+0xc4/0x2b0
> [    1.132606]  ? lock_acquire+0xd5/0x2d0
> [    1.132613]  ? vmbus_chan_sched+0x31/0x2b0
> [    1.132619]  rt_spin_lock+0x3f/0x1f0
> [    1.132623]  ? vmbus_chan_sched+0xc4/0x2b0
> [    1.132629]  ? vmbus_chan_sched+0x31/0x2b0
> [    1.132634]  vmbus_chan_sched+0xc4/0x2b0
> [    1.132641]  vmbus_isr+0x2c/0x150
> [    1.132648]  __sysvec_hyperv_callback+0x5f/0xa0
> [    1.132654]  sysvec_hyperv_callback+0x88/0xb0
> [    1.132658]  </IRQ>
> [    1.132659]  <TASK>
> [    1.132660]  asm_sysvec_hyperv_callback+0x1a/0x20
> 
> As code paths that handle vmbus IRQs use sleepy locks under PREEMPT_RT,
> the complete vmbus_handler execution needs to be moved into thread
> context. Open-coding this allows to skip the IPI that irq_work would
> additionally bring and which we do not need, being an IRQ, never an NMI.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>

Tested-by: Florian Bezdeka <florian.bezdeka@siemens.com>


This patch survived a 24h stress test with CONFIG_PREEMPT_RT enabled and
heavy load applied to the system.

There was no lockup happening without this patch. The lockdep warning is
gone now.

Best regards,
Florian


-- 
Siemens AG, Foundational Technologies
Linux Expert Center

RE: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Michael Kelley]

From: Jan Kiszka <jan.kiszka@siemens.com> Sent: Tuesday, February 3, 2026 8:02 AM
> 
> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
> with related guest support enabled:
> 
> [    1.127941] hv_vmbus: registering driver hyperv_drm
> 
> [    1.132518] =============================
> [    1.132519] [ BUG: Invalid wait context ]
> [    1.132521] 6.19.0-rc8+ #9 Not tainted
> [    1.132524] -----------------------------
> [    1.132525] swapper/0/0 is trying to lock:
> [    1.132526] ffff8b9381bb3c90 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0xc4/0x2b0
> [    1.132543] other info that might help us debug this:
> [    1.132544] context-{2:2}
> [    1.132545] 1 lock held by swapper/0/0:
> [    1.132547]  #0: ffffffffa010c4c0 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0x31/0x2b0
> [    1.132557] stack backtrace:
> [    1.132560] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.19.0-rc8+ #9 PREEMPT_{RT,(lazy)}
> [    1.132565] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/25/2025
> [    1.132567] Call Trace:
> [    1.132570]  <IRQ>
> [    1.132573]  dump_stack_lvl+0x6e/0xa0
> [    1.132581]  __lock_acquire+0xee0/0x21b0
> [    1.132592]  lock_acquire+0xd5/0x2d0
> [    1.132598]  ? vmbus_chan_sched+0xc4/0x2b0
> [    1.132606]  ? lock_acquire+0xd5/0x2d0
> [    1.132613]  ? vmbus_chan_sched+0x31/0x2b0
> [    1.132619]  rt_spin_lock+0x3f/0x1f0
> [    1.132623]  ? vmbus_chan_sched+0xc4/0x2b0
> [    1.132629]  ? vmbus_chan_sched+0x31/0x2b0
> [    1.132634]  vmbus_chan_sched+0xc4/0x2b0
> [    1.132641]  vmbus_isr+0x2c/0x150
> [    1.132648]  __sysvec_hyperv_callback+0x5f/0xa0
> [    1.132654]  sysvec_hyperv_callback+0x88/0xb0
> [    1.132658]  </IRQ>
> [    1.132659]  <TASK>
> [    1.132660]  asm_sysvec_hyperv_callback+0x1a/0x20
> 
> As code paths that handle vmbus IRQs use sleepy locks under PREEMPT_RT,
> the complete vmbus_handler execution needs to be moved into thread
> context. Open-coding this allows to skip the IPI that irq_work would
> additionally bring and which we do not need, being an IRQ, never an NMI.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> 
> This should resolve what was once brought forward via [1]. If it
> actually resolves all remaining compatibility issues of the hyperv
> support with RT is not yet clear, though. So far, lockdep is happy when
> using this plus [2].
> 
> [1] https://lore.kernel.org/all/20230809-b4-rt_preempt-fix-v1-0-7283bbdc8b14@gmail.com/
> [2] https://lore.kernel.org/lkml/0c7fb5cd-fb21-4760-8593-e04bade84744@siemens.com/
> 
>  arch/x86/kernel/cpu/mshyperv.c | 52 ++++++++++++++++++++++++++++++++--	

You've added this code under arch/x86. But isn't it architecture independent? I
think it should also work on arm64. If that's the case, the code should probably
be added to drivers/hv/vmbus_drv.c instead.

>  1 file changed, 50 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
> index 579fb2c64cfd..1194ca452c52 100644
> --- a/arch/x86/kernel/cpu/mshyperv.c
> +++ b/arch/x86/kernel/cpu/mshyperv.c
> @@ -17,6 +17,7 @@
>  #include <linux/irq.h>
>  #include <linux/kexec.h>
>  #include <linux/random.h>
> +#include <linux/smpboot.h>
>  #include <asm/processor.h>
>  #include <asm/hypervisor.h>
>  #include <hyperv/hvhdk.h>
> @@ -150,6 +151,43 @@ static void (*hv_stimer0_handler)(void);
>  static void (*hv_kexec_handler)(void);
>  static void (*hv_crash_handler)(struct pt_regs *regs);
> 
> +static DEFINE_PER_CPU(bool, vmbus_irq_pending);
> +static DEFINE_PER_CPU(struct task_struct *, vmbus_irqd);
> +
> +static void vmbus_irqd_wake(void)
> +{
> +	struct task_struct *tsk = __this_cpu_read(vmbus_irqd);
> +
> +	__this_cpu_write(vmbus_irq_pending, true);
> +	wake_up_process(tsk);
> +}
> +
> +static void vmbus_irqd_setup(unsigned int cpu)
> +{
> +	sched_set_fifo(current);
> +}
> +
> +static int vmbus_irqd_should_run(unsigned int cpu)
> +{
> +	return __this_cpu_read(vmbus_irq_pending);
> +}
> +
> +static void run_vmbus_irqd(unsigned int cpu)
> +{
> +	vmbus_handler();
> +	__this_cpu_write(vmbus_irq_pending, false);
> +}

The two statements in this function should be swapped. This function
runs with pre-emption enabled and interrupts enabled. If a VMBus
interrupt comes in as vmbus_handler() is finishing, vmbus_irqd_wake()
will run and set vmbus_irq_pending to "true". This function will then set
vmbus_irq_pending to 'false", wiping out the "true" setting. The hotplug
thread will decide it doesn't need to run again, and whatever generated
the new interrupt doesn't get processed (at least until another interrupt
comes in).

This scenario could specifically happen because of the way VMBus messages
are processed. The vmbus_handler function calls vmbus_message_sched(),
which always processes a single message. When that message is handled,
Hyper-V sends the next message that may have been queued up, and
generates another interrupt to the guest VM. There's no looping in the Linux
code to process all messages, so Linux depends on getting a new interrupt for
each subsequent message in order to run vmbus_message_sched() again.

There might be a similar situation with vmbus_chan_sched() and channel
interrupts. There are three interrupt handling modes across multiple VMBus
devices, and it would take some additional sleuthing to see if any of them
depend on a similar scheme of needing a new interrupt for each channel
event.

Please double-check my thinking. The likelihood of the problem occurring
is very low, because VMBus messages generally are used only when VMBus
devices are being added (or removed), which is usually during boot, and
the timing window must be hit just right. But the fix is simple, so it should
be done.

Michael

> +
> +static bool vmbus_irq_initialized;
> +
> +static struct smp_hotplug_thread vmbus_irq_threads = {
> +	.store                  = &vmbus_irqd,
> +	.setup			= vmbus_irqd_setup,
> +	.thread_should_run      = vmbus_irqd_should_run,
> +	.thread_fn              = run_vmbus_irqd,
> +	.thread_comm            = "vmbus_irq/%u",
> +};
> +
>  DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>  {
>  	struct pt_regs *old_regs = set_irq_regs(regs);
> @@ -158,8 +196,12 @@ DEFINE_IDTENTRY_SYSVEC(sysvec_hyperv_callback)
>  	if (mshv_handler)
>  		mshv_handler();
> 
> -	if (vmbus_handler)
> -		vmbus_handler();
> +	if (vmbus_handler) {
> +		if (IS_ENABLED(CONFIG_PREEMPT_RT))
> +			vmbus_irqd_wake();
> +		else
> +			vmbus_handler();
> +	}
> 
>  	if (ms_hyperv.hints & HV_DEPRECATING_AEOI_RECOMMENDED)
>  		apic_eoi();
> @@ -174,6 +216,10 @@ void hv_setup_mshv_handler(void (*handler)(void))
> 
>  void hv_setup_vmbus_handler(void (*handler)(void))
>  {
> +	if (IS_ENABLED(CONFIG_PREEMPT_RT) && !vmbus_irq_initialized) {
> +		BUG_ON(smpboot_register_percpu_thread(&vmbus_irq_threads));
> +		vmbus_irq_initialized = true;
> +	}
>  	vmbus_handler = handler;
>  }
> 
> @@ -181,6 +227,8 @@ void hv_remove_vmbus_handler(void)
>  {
>  	/* We have no way to deallocate the interrupt gate */
>  	vmbus_handler = NULL;
> +	smpboot_unregister_percpu_thread(&vmbus_irq_threads);
> +	vmbus_irq_initialized = false;
>  }
> 
>  /*
> --
> 2.51.0

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 05.02.26 19:55, Michael Kelley wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com> Sent: Tuesday, February 3, 2026 8:02 AM
>>
>> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
>> with related guest support enabled:
>>
>> [    1.127941] hv_vmbus: registering driver hyperv_drm
>>
>> [    1.132518] =============================
>> [    1.132519] [ BUG: Invalid wait context ]
>> [    1.132521] 6.19.0-rc8+ #9 Not tainted
>> [    1.132524] -----------------------------
>> [    1.132525] swapper/0/0 is trying to lock:
>> [    1.132526] ffff8b9381bb3c90 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0xc4/0x2b0
>> [    1.132543] other info that might help us debug this:
>> [    1.132544] context-{2:2}
>> [    1.132545] 1 lock held by swapper/0/0:
>> [    1.132547]  #0: ffffffffa010c4c0 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0x31/0x2b0
>> [    1.132557] stack backtrace:
>> [    1.132560] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.19.0-rc8+ #9 PREEMPT_{RT,(lazy)}
>> [    1.132565] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/25/2025
>> [    1.132567] Call Trace:
>> [    1.132570]  <IRQ>
>> [    1.132573]  dump_stack_lvl+0x6e/0xa0
>> [    1.132581]  __lock_acquire+0xee0/0x21b0
>> [    1.132592]  lock_acquire+0xd5/0x2d0
>> [    1.132598]  ? vmbus_chan_sched+0xc4/0x2b0
>> [    1.132606]  ? lock_acquire+0xd5/0x2d0
>> [    1.132613]  ? vmbus_chan_sched+0x31/0x2b0
>> [    1.132619]  rt_spin_lock+0x3f/0x1f0
>> [    1.132623]  ? vmbus_chan_sched+0xc4/0x2b0
>> [    1.132629]  ? vmbus_chan_sched+0x31/0x2b0
>> [    1.132634]  vmbus_chan_sched+0xc4/0x2b0
>> [    1.132641]  vmbus_isr+0x2c/0x150
>> [    1.132648]  __sysvec_hyperv_callback+0x5f/0xa0
>> [    1.132654]  sysvec_hyperv_callback+0x88/0xb0
>> [    1.132658]  </IRQ>
>> [    1.132659]  <TASK>
>> [    1.132660]  asm_sysvec_hyperv_callback+0x1a/0x20
>>
>> As code paths that handle vmbus IRQs use sleepy locks under PREEMPT_RT,
>> the complete vmbus_handler execution needs to be moved into thread
>> context. Open-coding this allows to skip the IPI that irq_work would
>> additionally bring and which we do not need, being an IRQ, never an NMI.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> This should resolve what was once brought forward via [1]. If it
>> actually resolves all remaining compatibility issues of the hyperv
>> support with RT is not yet clear, though. So far, lockdep is happy when
>> using this plus [2].
>>
>> [1] https://lore.kernel.org/all/20230809-b4-rt_preempt-fix-v1-0-7283bbdc8b14@gmail.com/
>> [2] https://lore.kernel.org/lkml/0c7fb5cd-fb21-4760-8593-e04bade84744@siemens.com/
>>
>>  arch/x86/kernel/cpu/mshyperv.c | 52 ++++++++++++++++++++++++++++++++--	
> 
> You've added this code under arch/x86. But isn't it architecture independent? I
> think it should also work on arm64. If that's the case, the code should probably
> be added to drivers/hv/vmbus_drv.c instead.
> 

I checked that before: arm64 uses normal IRQs, not over-optimized APIC
vectors. And those IRQs are auto-threaded.

That said, someone with an arm64 Hyper-V deployment should still try to
run things there once (PREEMPT_RT + PROVE_LOCKING). I don't have such a
setup.

>>  1 file changed, 50 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/x86/kernel/cpu/mshyperv.c b/arch/x86/kernel/cpu/mshyperv.c
>> index 579fb2c64cfd..1194ca452c52 100644
>> --- a/arch/x86/kernel/cpu/mshyperv.c
>> +++ b/arch/x86/kernel/cpu/mshyperv.c
>> @@ -17,6 +17,7 @@
>>  #include <linux/irq.h>
>>  #include <linux/kexec.h>
>>  #include <linux/random.h>
>> +#include <linux/smpboot.h>
>>  #include <asm/processor.h>
>>  #include <asm/hypervisor.h>
>>  #include <hyperv/hvhdk.h>
>> @@ -150,6 +151,43 @@ static void (*hv_stimer0_handler)(void);
>>  static void (*hv_kexec_handler)(void);
>>  static void (*hv_crash_handler)(struct pt_regs *regs);
>>
>> +static DEFINE_PER_CPU(bool, vmbus_irq_pending);
>> +static DEFINE_PER_CPU(struct task_struct *, vmbus_irqd);
>> +
>> +static void vmbus_irqd_wake(void)
>> +{
>> +	struct task_struct *tsk = __this_cpu_read(vmbus_irqd);
>> +
>> +	__this_cpu_write(vmbus_irq_pending, true);
>> +	wake_up_process(tsk);
>> +}
>> +
>> +static void vmbus_irqd_setup(unsigned int cpu)
>> +{
>> +	sched_set_fifo(current);
>> +}
>> +
>> +static int vmbus_irqd_should_run(unsigned int cpu)
>> +{
>> +	return __this_cpu_read(vmbus_irq_pending);
>> +}
>> +
>> +static void run_vmbus_irqd(unsigned int cpu)
>> +{
>> +	vmbus_handler();
>> +	__this_cpu_write(vmbus_irq_pending, false);
>> +}
> 
> The two statements in this function should be swapped. This function
> runs with pre-emption enabled and interrupts enabled. If a VMBus
> interrupt comes in as vmbus_handler() is finishing, vmbus_irqd_wake()
> will run and set vmbus_irq_pending to "true". This function will then set
> vmbus_irq_pending to 'false", wiping out the "true" setting. The hotplug
> thread will decide it doesn't need to run again, and whatever generated
> the new interrupt doesn't get processed (at least until another interrupt
> comes in).

You are absolutely right. The reordered pattern is the same as in
irq_work - for the very same reason. I'll send v2.

Thanks,
Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

RE: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Michael Kelley]

From: Jan Kiszka <jan.kiszka@siemens.com> Sent: Thursday, February 5, 2026 10:41 PM
> 
> On 05.02.26 19:55, Michael Kelley wrote:
> > From: Jan Kiszka <jan.kiszka@siemens.com> Sent: Tuesday, February 3, 2026 8:02 AM
> >>
> >> Resolves the following lockdep report when booting PREEMPT_RT on Hyper-V
> >> with related guest support enabled:
> >>
> >> [    1.127941] hv_vmbus: registering driver hyperv_drm
> >>
> >> [    1.132518] =============================
> >> [    1.132519] [ BUG: Invalid wait context ]
> >> [    1.132521] 6.19.0-rc8+ #9 Not tainted
> >> [    1.132524] -----------------------------
> >> [    1.132525] swapper/0/0 is trying to lock:
> >> [    1.132526] ffff8b9381bb3c90 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0xc4/0x2b0
> >> [    1.132543] other info that might help us debug this:
> >> [    1.132544] context-{2:2}
> >> [    1.132545] 1 lock held by swapper/0/0:
> >> [    1.132547]  #0: ffffffffa010c4c0 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0x31/0x2b0
> >> [    1.132557] stack backtrace:
> >> [    1.132560] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.19.0-rc8+ #9 PREEMPT_{RT,(lazy)}
> >> [    1.132565] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 09/25/2025
> >> [    1.132567] Call Trace:
> >> [    1.132570]  <IRQ>
> >> [    1.132573]  dump_stack_lvl+0x6e/0xa0
> >> [    1.132581]  __lock_acquire+0xee0/0x21b0
> >> [    1.132592]  lock_acquire+0xd5/0x2d0
> >> [    1.132598]  ? vmbus_chan_sched+0xc4/0x2b0
> >> [    1.132606]  ? lock_acquire+0xd5/0x2d0
> >> [    1.132613]  ? vmbus_chan_sched+0x31/0x2b0
> >> [    1.132619]  rt_spin_lock+0x3f/0x1f0
> >> [    1.132623]  ? vmbus_chan_sched+0xc4/0x2b0
> >> [    1.132629]  ? vmbus_chan_sched+0x31/0x2b0
> >> [    1.132634]  vmbus_chan_sched+0xc4/0x2b0
> >> [    1.132641]  vmbus_isr+0x2c/0x150
> >> [    1.132648]  __sysvec_hyperv_callback+0x5f/0xa0
> >> [    1.132654]  sysvec_hyperv_callback+0x88/0xb0
> >> [    1.132658]  </IRQ>
> >> [    1.132659]  <TASK>
> >> [    1.132660]  asm_sysvec_hyperv_callback+0x1a/0x20
> >>
> >> As code paths that handle vmbus IRQs use sleepy locks under PREEMPT_RT,
> >> the complete vmbus_handler execution needs to be moved into thread
> >> context. Open-coding this allows to skip the IPI that irq_work would
> >> additionally bring and which we do not need, being an IRQ, never an NMI.
> >>
> >> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> >> ---

[snip]

> >>
> >>  arch/x86/kernel/cpu/mshyperv.c | 52 ++++++++++++++++++++++++++++++++--
> >
> > You've added this code under arch/x86. But isn't it architecture independent? I
> > think it should also work on arm64. If that's the case, the code should probably
> > be added to drivers/hv/vmbus_drv.c instead.
> >
> 
> I checked that before: arm64 uses normal IRQs, not over-optimized APIC
> vectors. And those IRQs are auto-threaded.

Just to clarify, with CONFIG_PREEMPT_RT=y, you expect the normal Linux
IRQ handling mechanism to offload a per-CPU interrupt handler from interrupt
level onto a thread?

> 
> That said, someone with an arm64 Hyper-V deployment should still try to
> run things there once (PREEMPT_RT + PROVE_LOCKING). I don't have such a
> setup.
> 

I've run your suggested experiment on an arm64 VM in the Azure cloud. My
kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
(neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
The VM comes up and runs, but with this warning during boot:

[    3.075604] hv_utils: Registering HyperV Utility Driver
[    3.075636] hv_vmbus: registering driver hv_utils
[    3.085920] =============================
[    3.088128] hv_vmbus: registering driver hv_netvsc
[    3.091180] [ BUG: Invalid wait context ]
[    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
[    3.097582] -----------------------------
[    3.099899] systemd-udevd/284 is trying to lock:
[    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
[    3.108208] other info that might help us debug this:
[    3.111454] context-{2:2}
[    3.112987] 1 lock held by systemd-udevd/284:
[    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
[    3.121224] stack backtrace:
[    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E       6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
[    3.129631] Tainted: [E]=UNSIGNED_MODULE
[    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
[    3.138553] Call trace:
[    3.140015]  show_stack+0x20/0x38 (C)
[    3.142137]  dump_stack_lvl+0x9c/0x158
[    3.144340]  dump_stack+0x18/0x28
[    3.146290]  __lock_acquire+0x488/0x1e20
[    3.148569]  lock_acquire+0x11c/0x388
[    3.150703]  rt_spin_lock+0x54/0x230
[    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
[    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
[    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
[    3.160848]  handle_percpu_devid_irq+0xdc/0x348
[    3.163495]  handle_irq_desc+0x48/0x68
[    3.165851]  generic_handle_domain_irq+0x20/0x38
[    3.168664]  gic_handle_irq+0x1dc/0x430
[    3.170868]  call_on_irq_stack+0x30/0x70
[    3.173161]  do_interrupt_handler+0x88/0xa0
[    3.175724]  el1_interrupt+0x4c/0xb0
[    3.177855]  el1h_64_irq_handler+0x18/0x28
[    3.180332]  el1h_64_irq+0x84/0x88
[    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
[    3.185493]  rt_mutex_slowunlock+0x404/0x440
[    3.187951]  rt_spin_unlock+0xb8/0x178
[    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
[    3.193100]  alloc_empty_file+0x64/0x148
[    3.195461]  path_openat+0x58/0xaa0
[    3.197658]  do_file_open+0xa0/0x140
[    3.199752]  do_sys_openat2+0x190/0x278
[    3.202124]  do_sys_open+0x60/0xb8
[    3.204047]  __arm64_sys_openat+0x2c/0x48
[    3.206433]  invoke_syscall+0x6c/0xf8
[    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
[    3.211050]  do_el0_svc+0x24/0x38
[    3.212990]  el0_svc+0x164/0x3c8
[    3.214842]  el0t_64_sync_handler+0xd0/0xe8
[    3.217251]  el0t_64_sync+0x1b0/0x1b8
[    3.219450] hv_utils: Heartbeat IC version 3.0
[    3.219471] hv_utils: Shutdown IC version 3.2
[    3.219844] hv_utils: TimeSync IC version 4.0 

I don't see an indication that vmbus_isr() has been offloaded from
interrupt level onto a thread.  The stack starting with el1h_64_irq()
and going forward is the stack for normal per-cpu interrupt handling.
Maybe arm64 with PREEMPT_RT does the offload to a thread only
for SPIs and LPIs, but not for PPIs? I haven't looked at the source code
for how PREEMPT_RT affects arm64 interrupt handling.

Also, I had expected to see a problem with storvsc because I did
not apply your storvsc patch. But there was no such problem, even
with some disk I/O load (read only). arm64 VMs in Azure use exactly
the same virtual SCSI devices that are used with x86 VMs in Azure or
on local Hyper-V. I don't have an explanation. Will think about it.

Michael

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Florian Bezdeka]

On Sat, 2026-02-07 at 01:30 +0000, Michael Kelley wrote:

[snip]
> 
> I've run your suggested experiment on an arm64 VM in the Azure cloud. My
> kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
> CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
> (neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
> The VM comes up and runs, but with this warning during boot:
> 
> [    3.075604] hv_utils: Registering HyperV Utility Driver
> [    3.075636] hv_vmbus: registering driver hv_utils
> [    3.085920] =============================
> [    3.088128] hv_vmbus: registering driver hv_netvsc
> [    3.091180] [ BUG: Invalid wait context ]
> [    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
> [    3.097582] -----------------------------
> [    3.099899] systemd-udevd/284 is trying to lock:
> [    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> [    3.108208] other info that might help us debug this:
> [    3.111454] context-{2:2}
> [    3.112987] 1 lock held by systemd-udevd/284:
> [    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
> [    3.121224] stack backtrace:
> [    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E       6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
> [    3.129631] Tainted: [E]=UNSIGNED_MODULE
> [    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
> [    3.138553] Call trace:
> [    3.140015]  show_stack+0x20/0x38 (C)
> [    3.142137]  dump_stack_lvl+0x9c/0x158
> [    3.144340]  dump_stack+0x18/0x28
> [    3.146290]  __lock_acquire+0x488/0x1e20
> [    3.148569]  lock_acquire+0x11c/0x388
> [    3.150703]  rt_spin_lock+0x54/0x230
> [    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> [    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
> [    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
> [    3.160848]  handle_percpu_devid_irq+0xdc/0x348
> [    3.163495]  handle_irq_desc+0x48/0x68
> [    3.165851]  generic_handle_domain_irq+0x20/0x38
> [    3.168664]  gic_handle_irq+0x1dc/0x430
> [    3.170868]  call_on_irq_stack+0x30/0x70
> [    3.173161]  do_interrupt_handler+0x88/0xa0
> [    3.175724]  el1_interrupt+0x4c/0xb0
> [    3.177855]  el1h_64_irq_handler+0x18/0x28
> [    3.180332]  el1h_64_irq+0x84/0x88
> [    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
> [    3.185493]  rt_mutex_slowunlock+0x404/0x440
> [    3.187951]  rt_spin_unlock+0xb8/0x178
> [    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
> [    3.193100]  alloc_empty_file+0x64/0x148
> [    3.195461]  path_openat+0x58/0xaa0
> [    3.197658]  do_file_open+0xa0/0x140
> [    3.199752]  do_sys_openat2+0x190/0x278
> [    3.202124]  do_sys_open+0x60/0xb8
> [    3.204047]  __arm64_sys_openat+0x2c/0x48
> [    3.206433]  invoke_syscall+0x6c/0xf8
> [    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
> [    3.211050]  do_el0_svc+0x24/0x38
> [    3.212990]  el0_svc+0x164/0x3c8
> [    3.214842]  el0t_64_sync_handler+0xd0/0xe8
> [    3.217251]  el0t_64_sync+0x1b0/0x1b8
> [    3.219450] hv_utils: Heartbeat IC version 3.0
> [    3.219471] hv_utils: Shutdown IC version 3.2
> [    3.219844] hv_utils: TimeSync IC version 4.0 

That matches with my expectation that the same problem exists on arm64.
The patch from Jan addresses that issue for x86 (only, so far) as we do
not have a working test environment for arm64 yet.

> 
> I don't see an indication that vmbus_isr() has been offloaded from
> interrupt level onto a thread.  The stack starting with el1h_64_irq()
> and going forward is the stack for normal per-cpu interrupt handling.
> Maybe arm64 with PREEMPT_RT does the offload to a thread only
> for SPIs and LPIs, but not for PPIs? I haven't looked at the source code
> for how PREEMPT_RT affects arm64 interrupt handling.
> 
> Also, I had expected to see a problem with storvsc because I did
> not apply your storvsc patch. But there was no such problem, even
> with some disk I/O load (read only). arm64 VMs in Azure use exactly
> the same virtual SCSI devices that are used with x86 VMs in Azure or
> on local Hyper-V. I don't have an explanation. Will think about it.
> 

Running the --iomix stressor provided by stress-ng was able to trigger
the SCSI problem within 2 minutes. The result was a completely frozen
system. For completeness the complete stress-ng command line:

# stress-ng --cpu 2 --iomix 8 --vm 2 --vm-bytes 128M --fork 4

Best regards,
Florian

RE: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Michael Kelley]

From: Florian Bezdeka <florian.bezdeka@siemens.com> Sent: Monday, February 9, 2026 2:35 AM
> 
> On Sat, 2026-02-07 at 01:30 +0000, Michael Kelley wrote:
> 
> [snip]
> >
> > I've run your suggested experiment on an arm64 VM in the Azure cloud. My
> > kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
> > CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
> > (neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
> > The VM comes up and runs, but with this warning during boot:
> >
> > [    3.075604] hv_utils: Registering HyperV Utility Driver
> > [    3.075636] hv_vmbus: registering driver hv_utils
> > [    3.085920] =============================
> > [    3.088128] hv_vmbus: registering driver hv_netvsc
> > [    3.091180] [ BUG: Invalid wait context ]
> > [    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
> > [    3.097582] -----------------------------
> > [    3.099899] systemd-udevd/284 is trying to lock:
> > [    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> > [    3.108208] other info that might help us debug this:
> > [    3.111454] context-{2:2}
> > [    3.112987] 1 lock held by systemd-udevd/284:
> > [    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
> > [    3.121224] stack backtrace:
> > [    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E  6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
> > [    3.129631] Tainted: [E]=UNSIGNED_MODULE
> > [    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
> > [    3.138553] Call trace:
> > [    3.140015]  show_stack+0x20/0x38 (C)
> > [    3.142137]  dump_stack_lvl+0x9c/0x158
> > [    3.144340]  dump_stack+0x18/0x28
> > [    3.146290]  __lock_acquire+0x488/0x1e20
> > [    3.148569]  lock_acquire+0x11c/0x388
> > [    3.150703]  rt_spin_lock+0x54/0x230
> > [    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> > [    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
> > [    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
> > [    3.160848]  handle_percpu_devid_irq+0xdc/0x348
> > [    3.163495]  handle_irq_desc+0x48/0x68
> > [    3.165851]  generic_handle_domain_irq+0x20/0x38
> > [    3.168664]  gic_handle_irq+0x1dc/0x430
> > [    3.170868]  call_on_irq_stack+0x30/0x70
> > [    3.173161]  do_interrupt_handler+0x88/0xa0
> > [    3.175724]  el1_interrupt+0x4c/0xb0
> > [    3.177855]  el1h_64_irq_handler+0x18/0x28
> > [    3.180332]  el1h_64_irq+0x84/0x88
> > [    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
> > [    3.185493]  rt_mutex_slowunlock+0x404/0x440
> > [    3.187951]  rt_spin_unlock+0xb8/0x178
> > [    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
> > [    3.193100]  alloc_empty_file+0x64/0x148
> > [    3.195461]  path_openat+0x58/0xaa0
> > [    3.197658]  do_file_open+0xa0/0x140
> > [    3.199752]  do_sys_openat2+0x190/0x278
> > [    3.202124]  do_sys_open+0x60/0xb8
> > [    3.204047]  __arm64_sys_openat+0x2c/0x48
> > [    3.206433]  invoke_syscall+0x6c/0xf8
> > [    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
> > [    3.211050]  do_el0_svc+0x24/0x38
> > [    3.212990]  el0_svc+0x164/0x3c8
> > [    3.214842]  el0t_64_sync_handler+0xd0/0xe8
> > [    3.217251]  el0t_64_sync+0x1b0/0x1b8
> > [    3.219450] hv_utils: Heartbeat IC version 3.0
> > [    3.219471] hv_utils: Shutdown IC version 3.2
> > [    3.219844] hv_utils: TimeSync IC version 4.0
> 
> That matches with my expectation that the same problem exists on arm64.
> The patch from Jan addresses that issue for x86 (only, so far) as we do
> not have a working test environment for arm64 yet.

OK. I had understood Jan's earlier comments to mean that the VMBus
interrupt problem was implicitly solved on arm64 because of VMBus using
a standard Linux IRQ on arm64. But evidently that's not the case. So my
earlier comment stands: The code changes should go into the architecture
independent portion of the VMBus driver, and not under arch/x86. I
can probably work with you to test on arm64 if need be.

> 
> >
> > I don't see an indication that vmbus_isr() has been offloaded from
> > interrupt level onto a thread.  The stack starting with el1h_64_irq()
> > and going forward is the stack for normal per-cpu interrupt handling.
> > Maybe arm64 with PREEMPT_RT does the offload to a thread only
> > for SPIs and LPIs, but not for PPIs? I haven't looked at the source code
> > for how PREEMPT_RT affects arm64 interrupt handling.
> >
> > Also, I had expected to see a problem with storvsc because I did
> > not apply your storvsc patch. But there was no such problem, even
> > with some disk I/O load (read only). arm64 VMs in Azure use exactly
> > the same virtual SCSI devices that are used with x86 VMs in Azure or
> > on local Hyper-V. I don't have an explanation. Will think about it.
> >
> 
> Running the --iomix stressor provided by stress-ng was able to trigger
> the SCSI problem within 2 minutes. The result was a completely frozen
> system. For completeness the complete stress-ng command line:
> 
> # stress-ng --cpu 2 --iomix 8 --vm 2 --vm-bytes 128M --fork 4
> 

Thanks!

Yes, that command line reproduced the storvsc problem on arm64. And
then applying the storvsc patch made the problem go away. FWIW, on
arm64 Linux recovered and kept running after hitting the storvsc
problem.

Michael

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 09.02.26 19:25, Michael Kelley wrote:
> From: Florian Bezdeka <florian.bezdeka@siemens.com> Sent: Monday, February 9, 2026 2:35 AM
>>
>> On Sat, 2026-02-07 at 01:30 +0000, Michael Kelley wrote:
>>
>> [snip]
>>>
>>> I've run your suggested experiment on an arm64 VM in the Azure cloud. My
>>> kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
>>> CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
>>> (neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
>>> The VM comes up and runs, but with this warning during boot:
>>>
>>> [    3.075604] hv_utils: Registering HyperV Utility Driver
>>> [    3.075636] hv_vmbus: registering driver hv_utils
>>> [    3.085920] =============================
>>> [    3.088128] hv_vmbus: registering driver hv_netvsc
>>> [    3.091180] [ BUG: Invalid wait context ]
>>> [    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
>>> [    3.097582] -----------------------------
>>> [    3.099899] systemd-udevd/284 is trying to lock:
>>> [    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
>>> [    3.108208] other info that might help us debug this:
>>> [    3.111454] context-{2:2}
>>> [    3.112987] 1 lock held by systemd-udevd/284:
>>> [    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
>>> [    3.121224] stack backtrace:
>>> [    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E  6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
>>> [    3.129631] Tainted: [E]=UNSIGNED_MODULE
>>> [    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
>>> [    3.138553] Call trace:
>>> [    3.140015]  show_stack+0x20/0x38 (C)
>>> [    3.142137]  dump_stack_lvl+0x9c/0x158
>>> [    3.144340]  dump_stack+0x18/0x28
>>> [    3.146290]  __lock_acquire+0x488/0x1e20
>>> [    3.148569]  lock_acquire+0x11c/0x388
>>> [    3.150703]  rt_spin_lock+0x54/0x230
>>> [    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
>>> [    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
>>> [    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
>>> [    3.160848]  handle_percpu_devid_irq+0xdc/0x348
>>> [    3.163495]  handle_irq_desc+0x48/0x68
>>> [    3.165851]  generic_handle_domain_irq+0x20/0x38
>>> [    3.168664]  gic_handle_irq+0x1dc/0x430
>>> [    3.170868]  call_on_irq_stack+0x30/0x70
>>> [    3.173161]  do_interrupt_handler+0x88/0xa0
>>> [    3.175724]  el1_interrupt+0x4c/0xb0
>>> [    3.177855]  el1h_64_irq_handler+0x18/0x28
>>> [    3.180332]  el1h_64_irq+0x84/0x88
>>> [    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
>>> [    3.185493]  rt_mutex_slowunlock+0x404/0x440
>>> [    3.187951]  rt_spin_unlock+0xb8/0x178
>>> [    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
>>> [    3.193100]  alloc_empty_file+0x64/0x148
>>> [    3.195461]  path_openat+0x58/0xaa0
>>> [    3.197658]  do_file_open+0xa0/0x140
>>> [    3.199752]  do_sys_openat2+0x190/0x278
>>> [    3.202124]  do_sys_open+0x60/0xb8
>>> [    3.204047]  __arm64_sys_openat+0x2c/0x48
>>> [    3.206433]  invoke_syscall+0x6c/0xf8
>>> [    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
>>> [    3.211050]  do_el0_svc+0x24/0x38
>>> [    3.212990]  el0_svc+0x164/0x3c8
>>> [    3.214842]  el0t_64_sync_handler+0xd0/0xe8
>>> [    3.217251]  el0t_64_sync+0x1b0/0x1b8
>>> [    3.219450] hv_utils: Heartbeat IC version 3.0
>>> [    3.219471] hv_utils: Shutdown IC version 3.2
>>> [    3.219844] hv_utils: TimeSync IC version 4.0
>>
>> That matches with my expectation that the same problem exists on arm64.
>> The patch from Jan addresses that issue for x86 (only, so far) as we do
>> not have a working test environment for arm64 yet.
> 
> OK. I had understood Jan's earlier comments to mean that the VMBus
> interrupt problem was implicitly solved on arm64 because of VMBus using
> a standard Linux IRQ on arm64. But evidently that's not the case. So my
> earlier comment stands: The code changes should go into the architecture
> independent portion of the VMBus driver, and not under arch/x86. I
> can probably work with you to test on arm64 if need be.
> 

I can move the code, sure, but I still haven't understood what
invalidates my assumptions (beside what you observed). vmbus_drv calls
request_percpu_irq, and that is - as far as I can see - not injecting
IRQF_NO_THREAD. Any explanations welcome.

Reproduction is still not possible for me. I was playing a bit with qemu
in the hope to make it provide its minimal vmbus support (for
ballooning), but that was not yet successful on arm64.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

RE: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Michael Kelley]

From: Jan Kiszka <jan.kiszka@siemens.com> Sent: Thursday, February 12, 2026 8:06 AM
> 
> On 09.02.26 19:25, Michael Kelley wrote:
> > From: Florian Bezdeka <florian.bezdeka@siemens.com> Sent: Monday, February 9, 2026 2:35 AM
> >>
> >> On Sat, 2026-02-07 at 01:30 +0000, Michael Kelley wrote:
> >>
> >> [snip]
> >>>
> >>> I've run your suggested experiment on an arm64 VM in the Azure cloud. My
> >>> kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
> >>> CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
> >>> (neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
> >>> The VM comes up and runs, but with this warning during boot:
> >>>
> >>> [    3.075604] hv_utils: Registering HyperV Utility Driver
> >>> [    3.075636] hv_vmbus: registering driver hv_utils
> >>> [    3.085920] =============================
> >>> [    3.088128] hv_vmbus: registering driver hv_netvsc
> >>> [    3.091180] [ BUG: Invalid wait context ]
> >>> [    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
> >>> [    3.097582] -----------------------------
> >>> [    3.099899] systemd-udevd/284 is trying to lock:
> >>> [    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> >>> [    3.108208] other info that might help us debug this:
> >>> [    3.111454] context-{2:2}
> >>> [    3.112987] 1 lock held by systemd-udevd/284:
> >>> [    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
> >>> [    3.121224] stack backtrace:
> >>> [    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E 6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
> >>> [    3.129631] Tainted: [E]=UNSIGNED_MODULE
> >>> [    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
> >>> [    3.138553] Call trace:
> >>> [    3.140015]  show_stack+0x20/0x38 (C)
> >>> [    3.142137]  dump_stack_lvl+0x9c/0x158
> >>> [    3.144340]  dump_stack+0x18/0x28
> >>> [    3.146290]  __lock_acquire+0x488/0x1e20
> >>> [    3.148569]  lock_acquire+0x11c/0x388
> >>> [    3.150703]  rt_spin_lock+0x54/0x230
> >>> [    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> >>> [    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
> >>> [    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
> >>> [    3.160848]  handle_percpu_devid_irq+0xdc/0x348
> >>> [    3.163495]  handle_irq_desc+0x48/0x68
> >>> [    3.165851]  generic_handle_domain_irq+0x20/0x38
> >>> [    3.168664]  gic_handle_irq+0x1dc/0x430
> >>> [    3.170868]  call_on_irq_stack+0x30/0x70
> >>> [    3.173161]  do_interrupt_handler+0x88/0xa0
> >>> [    3.175724]  el1_interrupt+0x4c/0xb0
> >>> [    3.177855]  el1h_64_irq_handler+0x18/0x28
> >>> [    3.180332]  el1h_64_irq+0x84/0x88
> >>> [    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
> >>> [    3.185493]  rt_mutex_slowunlock+0x404/0x440
> >>> [    3.187951]  rt_spin_unlock+0xb8/0x178
> >>> [    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
> >>> [    3.193100]  alloc_empty_file+0x64/0x148
> >>> [    3.195461]  path_openat+0x58/0xaa0
> >>> [    3.197658]  do_file_open+0xa0/0x140
> >>> [    3.199752]  do_sys_openat2+0x190/0x278
> >>> [    3.202124]  do_sys_open+0x60/0xb8
> >>> [    3.204047]  __arm64_sys_openat+0x2c/0x48
> >>> [    3.206433]  invoke_syscall+0x6c/0xf8
> >>> [    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
> >>> [    3.211050]  do_el0_svc+0x24/0x38
> >>> [    3.212990]  el0_svc+0x164/0x3c8
> >>> [    3.214842]  el0t_64_sync_handler+0xd0/0xe8
> >>> [    3.217251]  el0t_64_sync+0x1b0/0x1b8
> >>> [    3.219450] hv_utils: Heartbeat IC version 3.0
> >>> [    3.219471] hv_utils: Shutdown IC version 3.2
> >>> [    3.219844] hv_utils: TimeSync IC version 4.0
> >>
> >> That matches with my expectation that the same problem exists on arm64.
> >> The patch from Jan addresses that issue for x86 (only, so far) as we do
> >> not have a working test environment for arm64 yet.
> >
> > OK. I had understood Jan's earlier comments to mean that the VMBus
> > interrupt problem was implicitly solved on arm64 because of VMBus using
> > a standard Linux IRQ on arm64. But evidently that's not the case. So my
> > earlier comment stands: The code changes should go into the architecture
> > independent portion of the VMBus driver, and not under arch/x86. I
> > can probably work with you to test on arm64 if need be.
> >
> 
> I can move the code, sure, but I still haven't understood what
> invalidates my assumptions (beside what you observed). vmbus_drv calls
> request_percpu_irq, and that is - as far as I can see - not injecting
> IRQF_NO_THREAD. Any explanations welcome.
> 
> Reproduction is still not possible for me. I was playing a bit with qemu
> in the hope to make it provide its minimal vmbus support (for
> ballooning), but that was not yet successful on arm64.
> 

Let me try to debug my experiment on arm64 and see why it isn't
handing off the VMBus interrupt to a thread. Maybe there's something
missing in my .config. But it will be sometime next week before
I can do it.

Michael 

RE: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[mhklkml]

From: Jan Kiszka <jan.kiszka@siemens.com> Sent: Thursday, February 12, 2026 8:06 AM
> 
> On 09.02.26 19:25, Michael Kelley wrote:
> > From: Florian Bezdeka <florian.bezdeka@siemens.com> Sent: Monday, February 9, 2026 2:35 AM
> >>
> >> On Sat, 2026-02-07 at 01:30 +0000, Michael Kelley wrote:
> >>
> >> [snip]
> >>>
> >>> I've run your suggested experiment on an arm64 VM in the Azure cloud. My
> >>> kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
> >>> CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
> >>> (neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
> >>> The VM comes up and runs, but with this warning during boot:
> >>>
> >>> [    3.075604] hv_utils: Registering HyperV Utility Driver
> >>> [    3.075636] hv_vmbus: registering driver hv_utils
> >>> [    3.085920] =============================
> >>> [    3.088128] hv_vmbus: registering driver hv_netvsc
> >>> [    3.091180] [ BUG: Invalid wait context ]
> >>> [    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
> >>> [    3.097582] -----------------------------
> >>> [    3.099899] systemd-udevd/284 is trying to lock:
> >>> [    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> >>> [    3.108208] other info that might help us debug this:
> >>> [    3.111454] context-{2:2}
> >>> [    3.112987] 1 lock held by systemd-udevd/284:
> >>> [    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
> >>> [    3.121224] stack backtrace:
> >>> [    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E 6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
> >>> [    3.129631] Tainted: [E]=UNSIGNED_MODULE
> >>> [    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
> >>> [    3.138553] Call trace:
> >>> [    3.140015]  show_stack+0x20/0x38 (C)
> >>> [    3.142137]  dump_stack_lvl+0x9c/0x158
> >>> [    3.144340]  dump_stack+0x18/0x28
> >>> [    3.146290]  __lock_acquire+0x488/0x1e20
> >>> [    3.148569]  lock_acquire+0x11c/0x388
> >>> [    3.150703]  rt_spin_lock+0x54/0x230
> >>> [    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
> >>> [    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
> >>> [    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
> >>> [    3.160848]  handle_percpu_devid_irq+0xdc/0x348
> >>> [    3.163495]  handle_irq_desc+0x48/0x68
> >>> [    3.165851]  generic_handle_domain_irq+0x20/0x38
> >>> [    3.168664]  gic_handle_irq+0x1dc/0x430
> >>> [    3.170868]  call_on_irq_stack+0x30/0x70
> >>> [    3.173161]  do_interrupt_handler+0x88/0xa0
> >>> [    3.175724]  el1_interrupt+0x4c/0xb0
> >>> [    3.177855]  el1h_64_irq_handler+0x18/0x28
> >>> [    3.180332]  el1h_64_irq+0x84/0x88
> >>> [    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
> >>> [    3.185493]  rt_mutex_slowunlock+0x404/0x440
> >>> [    3.187951]  rt_spin_unlock+0xb8/0x178
> >>> [    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
> >>> [    3.193100]  alloc_empty_file+0x64/0x148
> >>> [    3.195461]  path_openat+0x58/0xaa0
> >>> [    3.197658]  do_file_open+0xa0/0x140
> >>> [    3.199752]  do_sys_openat2+0x190/0x278
> >>> [    3.202124]  do_sys_open+0x60/0xb8
> >>> [    3.204047]  __arm64_sys_openat+0x2c/0x48
> >>> [    3.206433]  invoke_syscall+0x6c/0xf8
> >>> [    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
> >>> [    3.211050]  do_el0_svc+0x24/0x38
> >>> [    3.212990]  el0_svc+0x164/0x3c8
> >>> [    3.214842]  el0t_64_sync_handler+0xd0/0xe8
> >>> [    3.217251]  el0t_64_sync+0x1b0/0x1b8
> >>> [    3.219450] hv_utils: Heartbeat IC version 3.0
> >>> [    3.219471] hv_utils: Shutdown IC version 3.2
> >>> [    3.219844] hv_utils: TimeSync IC version 4.0
> >>
> >> That matches with my expectation that the same problem exists on arm64.
> >> The patch from Jan addresses that issue for x86 (only, so far) as we do
> >> not have a working test environment for arm64 yet.
> >
> > OK. I had understood Jan's earlier comments to mean that the VMBus
> > interrupt problem was implicitly solved on arm64 because of VMBus using
> > a standard Linux IRQ on arm64. But evidently that's not the case. So my
> > earlier comment stands: The code changes should go into the architecture
> > independent portion of the VMBus driver, and not under arch/x86. I
> > can probably work with you to test on arm64 if need be.
> >
> 
> I can move the code, sure, but I still haven't understood what
> invalidates my assumptions (beside what you observed). vmbus_drv calls
> request_percpu_irq, and that is - as far as I can see - not injecting
> IRQF_NO_THREAD. Any explanations welcome.

I haven't setup detailed debugging on arm64 yet, but in prep for that
I went looking at the places in the kernel IRQ handling where
IRQF_NO_THREAD influences behavior. The key function appears to be
irq_setup_forced_threading(). This function first checks force_irqthreads(),
which will be "true" when PREEMPT_RT is set. The function then checks
the IRQF_NO_THREAD flag and the IRQF_PERCPU flag. From what I can
see, the IRQF_PERCPU flag is treated like the IRQF_NO_THREAD flag, and
causes forced threading to *not* be done. So the behavior ends up being
the same as when PREEMPT_RT is not set.

Since the VMBus interrupt is a per-cpu interrupt, forced threading is not
done. In that case, the stack trace I reported makes sense. Take a look at
the code and see if you agree.

Michael

> 
> Reproduction is still not possible for me. I was playing a bit with qemu
> in the hope to make it provide its minimal vmbus support (for
> ballooning), but that was not yet successful on arm64.
> 

Re: [PATCH] x86: mshyperv: Use kthread for vmbus interrupts on PREEMPT_RT

[Jan Kiszka]

On 13.02.26 22:35, mhklkml@zohomail.com wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com> Sent: Thursday, February 12, 2026 8:06 AM
>>
>> On 09.02.26 19:25, Michael Kelley wrote:
>>> From: Florian Bezdeka <florian.bezdeka@siemens.com> Sent: Monday, February 9, 2026 2:35 AM
>>>>
>>>> On Sat, 2026-02-07 at 01:30 +0000, Michael Kelley wrote:
>>>>
>>>> [snip]
>>>>>
>>>>> I've run your suggested experiment on an arm64 VM in the Azure cloud. My
>>>>> kernel was linux-next 20260128. I set CONFIG_PREEMPT_RT=y and
>>>>> CONFIG_PROVE_LOCKING=y, but did not add either of your two patches
>>>>> (neither the storvsc driver patch nor the x86 VMBus interrupt handling patch).
>>>>> The VM comes up and runs, but with this warning during boot:
>>>>>
>>>>> [    3.075604] hv_utils: Registering HyperV Utility Driver
>>>>> [    3.075636] hv_vmbus: registering driver hv_utils
>>>>> [    3.085920] =============================
>>>>> [    3.088128] hv_vmbus: registering driver hv_netvsc
>>>>> [    3.091180] [ BUG: Invalid wait context ]
>>>>> [    3.093544] 6.19.0-rc7-next-20260128+ #3 Tainted: G            E
>>>>> [    3.097582] -----------------------------
>>>>> [    3.099899] systemd-udevd/284 is trying to lock:
>>>>> [    3.102568] ffff000100e24490 (&channel->sched_lock){....}-{3:3}, at: vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
>>>>> [    3.108208] other info that might help us debug this:
>>>>> [    3.111454] context-{2:2}
>>>>> [    3.112987] 1 lock held by systemd-udevd/284:
>>>>> [    3.115626]  #0: ffffd5cfc20bcc80 (rcu_read_lock){....}-{1:3}, at: vmbus_chan_sched+0xcc/0x3b8 [hv_vmbus]
>>>>> [    3.121224] stack backtrace:
>>>>> [    3.122897] CPU: 0 UID: 0 PID: 284 Comm: systemd-udevd Tainted: G            E 6.19.0-rc7-next-20260128+ #3 PREEMPT_RT
>>>>> [    3.129631] Tainted: [E]=UNSIGNED_MODULE
>>>>> [    3.131946] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 06/10/2025
>>>>> [    3.138553] Call trace:
>>>>> [    3.140015]  show_stack+0x20/0x38 (C)
>>>>> [    3.142137]  dump_stack_lvl+0x9c/0x158
>>>>> [    3.144340]  dump_stack+0x18/0x28
>>>>> [    3.146290]  __lock_acquire+0x488/0x1e20
>>>>> [    3.148569]  lock_acquire+0x11c/0x388
>>>>> [    3.150703]  rt_spin_lock+0x54/0x230
>>>>> [    3.152785]  vmbus_chan_sched+0x128/0x3b8 [hv_vmbus]
>>>>> [    3.155611]  vmbus_isr+0x34/0x80 [hv_vmbus]
>>>>> [    3.158093]  vmbus_percpu_isr+0x18/0x30 [hv_vmbus]
>>>>> [    3.160848]  handle_percpu_devid_irq+0xdc/0x348
>>>>> [    3.163495]  handle_irq_desc+0x48/0x68
>>>>> [    3.165851]  generic_handle_domain_irq+0x20/0x38
>>>>> [    3.168664]  gic_handle_irq+0x1dc/0x430
>>>>> [    3.170868]  call_on_irq_stack+0x30/0x70
>>>>> [    3.173161]  do_interrupt_handler+0x88/0xa0
>>>>> [    3.175724]  el1_interrupt+0x4c/0xb0
>>>>> [    3.177855]  el1h_64_irq_handler+0x18/0x28
>>>>> [    3.180332]  el1h_64_irq+0x84/0x88
>>>>> [    3.182378]  _raw_spin_unlock_irqrestore+0x4c/0xb0 (P)
>>>>> [    3.185493]  rt_mutex_slowunlock+0x404/0x440
>>>>> [    3.187951]  rt_spin_unlock+0xb8/0x178
>>>>> [    3.190394]  kmem_cache_alloc_noprof+0xf0/0x4f8
>>>>> [    3.193100]  alloc_empty_file+0x64/0x148
>>>>> [    3.195461]  path_openat+0x58/0xaa0
>>>>> [    3.197658]  do_file_open+0xa0/0x140
>>>>> [    3.199752]  do_sys_openat2+0x190/0x278
>>>>> [    3.202124]  do_sys_open+0x60/0xb8
>>>>> [    3.204047]  __arm64_sys_openat+0x2c/0x48
>>>>> [    3.206433]  invoke_syscall+0x6c/0xf8
>>>>> [    3.208519]  el0_svc_common.constprop.0+0x48/0xf0
>>>>> [    3.211050]  do_el0_svc+0x24/0x38
>>>>> [    3.212990]  el0_svc+0x164/0x3c8
>>>>> [    3.214842]  el0t_64_sync_handler+0xd0/0xe8
>>>>> [    3.217251]  el0t_64_sync+0x1b0/0x1b8
>>>>> [    3.219450] hv_utils: Heartbeat IC version 3.0
>>>>> [    3.219471] hv_utils: Shutdown IC version 3.2
>>>>> [    3.219844] hv_utils: TimeSync IC version 4.0
>>>>
>>>> That matches with my expectation that the same problem exists on arm64.
>>>> The patch from Jan addresses that issue for x86 (only, so far) as we do
>>>> not have a working test environment for arm64 yet.
>>>
>>> OK. I had understood Jan's earlier comments to mean that the VMBus
>>> interrupt problem was implicitly solved on arm64 because of VMBus using
>>> a standard Linux IRQ on arm64. But evidently that's not the case. So my
>>> earlier comment stands: The code changes should go into the architecture
>>> independent portion of the VMBus driver, and not under arch/x86. I
>>> can probably work with you to test on arm64 if need be.
>>>
>>
>> I can move the code, sure, but I still haven't understood what
>> invalidates my assumptions (beside what you observed). vmbus_drv calls
>> request_percpu_irq, and that is - as far as I can see - not injecting
>> IRQF_NO_THREAD. Any explanations welcome.
> 
> I haven't setup detailed debugging on arm64 yet, but in prep for that
> I went looking at the places in the kernel IRQ handling where
> IRQF_NO_THREAD influences behavior. The key function appears to be
> irq_setup_forced_threading(). This function first checks force_irqthreads(),
> which will be "true" when PREEMPT_RT is set. The function then checks
> the IRQF_NO_THREAD flag and the IRQF_PERCPU flag. From what I can
> see, the IRQF_PERCPU flag is treated like the IRQF_NO_THREAD flag, and
> causes forced threading to *not* be done. So the behavior ends up being
> the same as when PREEMPT_RT is not set.
> 
> Since the VMBus interrupt is a per-cpu interrupt, forced threading is not
> done. In that case, the stack trace I reported makes sense. Take a look at
> the code and see if you agree.

Indeed, missed the IRQF_PERCPU impact on auto-threading. I'll rework my
patch to perform the transition arch-independently.

Thanks,
Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center