[PATCH 0/2] Re-raise irq_work lost during Xen suspend

[PATCH 0/2] Re-raise irq_work lost during Xen suspend

[Julian Vetter]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]

We hit this issue with an Ubuntu 26.04 cloudimage guests (7.0 based
kernel with the 8250 as an nbcon console) on XCP-ng 8.3 / Xen 4.17. With
a kernel commandline adding two consoles, i.e., 'console=ttyS1
console=ttyS0'. During a live migration on that setup the printk()s from
gnttab_resume() and xen_irq_resume() queue nbcon_irq_work while the APIC
is disabled. Thus the list transitions from empty to non-empty, but
doesn't receive the IPI to empty the list again. Afterwards the list
stays non-empty, so the IPI never fires and all irq_work piles up.

The Commit 560f4deda327 ("printk/nbcon: Change nbcon_irq_work to
IRQ_WORK_LAZY") removed this particular issue because declaring the IRQ
work to be lazy, causes the timer interrupt to trigger the irq work
instead. But any other non-lazy irq_work queued in that window is still
lost the same way and will cause the list to fill up without being
handled.

Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
---
Julian Vetter (2):
  irq_work: Add irq_work_raise_pending()
  xen/manage: Re-raise pending irq_work after suspend

 drivers/xen/manage.c     |  3 +++
 include/linux/irq_work.h |  2 ++
 kernel/irq_work.c        | 10 ++++++++++
 3 files changed, 15 insertions(+)

-- 
2.53.0



--
Julian Vetter | Vates Hypervisor & Kernel Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[PATCH 1/2] irq_work: Add irq_work_raise_pending()

[Julian Vetter]

[-- Attachment #1: Type: text/plain, Size: 2535 bytes --]

irq_work_queue() only raises the irq_work interrupt when the per-CPU
list transitions from empty to non-empty. Later queuers rely on that
interrupt to also run their work.

If that interrupt gets lost, nothing runs the raised list on that CPU
anymore. On architectures with an irq_work interrupt irq_work_tick()
skips the raised list, and further irq_work_queue() calls find the list
non-empty and don't raise the interrupt again. All irq_work on that CPU
is stuck.

This happens when irq_work is queued while the local interrupt
controller is disabled, e.g. a self-IPI is dropped when sent while the
x86 local APIC is software-disabled.

Add irq_work_raise_pending(), which re-raises the irq_work interrupt if
work is pending on the current CPU. Code which disables the local
interrupt controller can call it once interrupts are accepted again.

Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
---
 include/linux/irq_work.h |  2 ++
 kernel/irq_work.c        | 10 ++++++++++
 2 files changed, 12 insertions(+)

diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h
index 7184e1273965..712a92ead4a5 100644
--- a/include/linux/irq_work.h
+++ b/include/linux/irq_work.h
@@ -62,12 +62,14 @@ void irq_work_run(void);
 void irq_work_run_cpu(unsigned int cpu);
 bool irq_work_needs_cpu(void);
 void irq_work_single(void *arg);
+void irq_work_raise_pending(void);
 
 void arch_irq_work_raise(void);
 
 #else
 static inline bool irq_work_needs_cpu(void) { return false; }
 static inline void irq_work_run(void) { }
+static inline void irq_work_raise_pending(void) { }
 static inline void irq_work_run_cpu(unsigned int cpu) { }
 static inline void irq_work_single(void *arg) { }
 #endif
diff --git a/kernel/irq_work.c b/kernel/irq_work.c
index 73eabcbdcd50..1720a07f12bf 100644
--- a/kernel/irq_work.c
+++ b/kernel/irq_work.c
@@ -112,6 +112,16 @@ static void __irq_work_queue_local(struct irq_work *work)
 		irq_work_raise(work);
 }
 
+/* Re-raise the irq_work interrupt on this CPU */
+void irq_work_raise_pending(void)
+{
+	lockdep_assert_irqs_disabled();
+
+	if (!llist_empty(this_cpu_ptr(&raised_list)) ||
+	    (tick_nohz_tick_stopped() && !llist_empty(this_cpu_ptr(&lazy_list))))
+		arch_irq_work_raise();
+}
+
 /* Enqueue the irq work @work on the current CPU */
 bool irq_work_queue(struct irq_work *work)
 {
-- 
2.53.0



--
Julian Vetter | Vates Hypervisor & Kernel Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

[PATCH 2/2] xen/manage: Re-raise pending irq_work after suspend

[Julian Vetter]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

xen_suspend() calls syscore_suspend(), which on x86 disables the APIC,
and only re-enables it in syscore_resume(). But, in between,
gnttab_resume() and xen_irq_resume() call printk().

With an nbcon console, e.g. the 8250, printk() queues irq_work, which
sends a self-IPI. As the APIC is disabled, the IPI is dropped. The
irq_work stays on the raised list, and no later irq_work_queue() on that
CPU raises the IPI again. All irq_work on that CPU is dead after resume.

Commit 560f4deda327 ("printk/nbcon: Change nbcon_irq_work to
IRQ_WORK_LAZY") removed this particular issue, but any other non-lazy
irq_work queued in this window would cause the irq_work to stall in the
same way.

To fix the issue, re-raise pending irq_work once syscore_resume() has
re-enabled the APIC.

Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
---
 drivers/xen/manage.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/xen/manage.c b/drivers/xen/manage.c
index 05d7de128e71..5054fcb25337 100644
--- a/drivers/xen/manage.c
+++ b/drivers/xen/manage.c
@@ -13,6 +13,7 @@
 #include <linux/stop_machine.h>
 #include <linux/suspend.h>
 #include <linux/freezer.h>
+#include <linux/irq_work.h>
 #include <linux/syscore_ops.h>
 #include <linux/export.h>
 
@@ -86,6 +87,8 @@ static int xen_suspend(void *data)
 
 	syscore_resume();
 
+	irq_work_raise_pending();
+
 	return 0;
 }
 
-- 
2.53.0



--
Julian Vetter | Vates Hypervisor & Kernel Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech

Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend

[Sebastian Andrzej Siewior]

On 2026-11-06 15:57:40 [+0100], Julian Vetter wrote:
> We hit this issue with an Ubuntu 26.04 cloudimage guests (7.0 based
> kernel with the 8250 as an nbcon console) on XCP-ng 8.3 / Xen 4.17. With
> a kernel commandline adding two consoles, i.e., 'console=ttyS1
> console=ttyS0'. During a live migration on that setup the printk()s from
> gnttab_resume() and xen_irq_resume() queue nbcon_irq_work while the APIC
> is disabled. Thus the list transitions from empty to non-empty, but
> doesn't receive the IPI to empty the list again. Afterwards the list
> stays non-empty, so the IPI never fires and all irq_work piles up.
> 
> The Commit 560f4deda327 ("printk/nbcon: Change nbcon_irq_work to
> IRQ_WORK_LAZY") removed this particular issue because declaring the IRQ
> work to be lazy, causes the timer interrupt to trigger the irq work
> instead. But any other non-lazy irq_work queued in that window is still
> lost the same way and will cause the list to fill up without being
> handled.

Adrian has this:
   https://lore.kernel.org/all/20261102074039.26983-2-adrian.hunter@intel.com/

I sort of asked for flush on suspend but Adrian pointed out the NMI
problem.

Looking at #1, you could probably limit it to a nop on
!arch_irq_work_has_interrupt(). The tick should not be disabled if
irq_work_needs_cpu() returns true (you don't add anything).

And lazy items get handled via irq_work_tick() anyway. This make me
think if it gets any easier if we handle both in irq_work_tick(). On x86
raised_list is evaluated anyway so by dropping
arch_irq_work_has_interrupt() we would avoid the APIC check and handle
list in this rare case. If the IRQ-work is working then the list remains
empty.

> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>

Sebastian

Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend

[Julian Vetter]

On 11/6/26 4:39 PM, Sebastian Andrzej Siewior wrote:
> On 2026-11-06 15:57:40 [+0100], Julian Vetter wrote:
>> We hit this issue with an Ubuntu 26.04 cloudimage guests (7.0 based
>> kernel with the 8250 as an nbcon console) on XCP-ng 8.3 / Xen 4.17. With
>> a kernel commandline adding two consoles, i.e., 'console=ttyS1
>> console=ttyS0'. During a live migration on that setup the printk()s from
>> gnttab_resume() and xen_irq_resume() queue nbcon_irq_work while the APIC
>> is disabled. Thus the list transitions from empty to non-empty, but
>> doesn't receive the IPI to empty the list again. Afterwards the list
>> stays non-empty, so the IPI never fires and all irq_work piles up.
>>
>> The Commit 560f4deda327 ("printk/nbcon: Change nbcon_irq_work to
>> IRQ_WORK_LAZY") removed this particular issue because declaring the IRQ
>> work to be lazy, causes the timer interrupt to trigger the irq work
>> instead. But any other non-lazy irq_work queued in that window is still
>> lost the same way and will cause the list to fill up without being
>> handled.
> 
> Adrian has this:
>     https://lore.kernel.org/all/20261102074039.26983-2-adrian.hunter@intel.com/
> 
> I sort of asked for flush on suspend but Adrian pointed out the NMI
> problem.

Thanks, I missed that. It is the same issue, just a different trigger. 
Adrian's example even ends in the same hung task we see 
(fsnotify_mark_destroy_workfn waiting on SRCU). Also, Adrian's 
syscore_resume() placement would cover the Xen case as well, as 
xen_suspend() calls syscore_resume() after those printk()s.

> 
> Looking at #1, you could probably limit it to a nop on
> !arch_irq_work_has_interrupt(). The tick should not be disabled if
> irq_work_needs_cpu() returns true (you don't add anything).
> 

Yes, right, the lazy part is pointless.

> And lazy items get handled via irq_work_tick() anyway. This make me
> think if it gets any easier if we handle both in irq_work_tick(). On x86
> raised_list is evaluated anyway so by dropping
> arch_irq_work_has_interrupt() we would avoid the APIC check and handle
> list in this rare case. If the IRQ-work is working then the list remains
> empty.
> 

This is maybe a cleaner solution, it needs no APIC access and no call 
site in the suspend paths. But irq_work_needs_cpu() would need the same 
change, otherwise in a CONFIG_NOHZ=y setting an idle CPU would still sit 
on the work until it leaves idle and re-enables the timer tick, no? I 
will test this.

>> Signed-off-by: Julian Vetter <julian.vetter@vates.tech>
> 
> Sebastian
> 

(Sorry, sending from my personal address, my work mail server currently 
mangles mails to the list.)

Julian

Re: [PATCH 0/2] Re-raise irq_work lost during Xen suspend

[Adrian Hunter]

On 09/11/2026 12:16, Julian Vetter wrote:
> On 11/6/26 4:39 PM, Sebastian Andrzej Siewior wrote:
>>
>> Looking at #1, you could probably limit it to a nop on
>> !arch_irq_work_has_interrupt(). The tick should not be disabled if
>> irq_work_needs_cpu() returns true (you don't add anything).
>>
> 
> Yes, right, the lazy part is pointless.
> 
>> And lazy items get handled via irq_work_tick() anyway. This make me
>> think if it gets any easier if we handle both in irq_work_tick(). On x86
>> raised_list is evaluated anyway so by dropping
>> arch_irq_work_has_interrupt() we would avoid the APIC check and handle
>> list in this rare case. If the IRQ-work is working then the list remains
>> empty.
>>
> 
> This is maybe a cleaner solution, it needs no APIC access and no call site in the suspend paths. But irq_work_needs_cpu() would need the same change, otherwise in a CONFIG_NOHZ=y setting an idle CPU would still sit on the work until it leaves idle and re-enables the timer tick, no? I will test this.

A fix for stable is needed, so need to check that there is no
irq_work that would have a problem running in the tick context.