* Dovetail 6.15: x86: Invalid wait context
@ 2025-06-03 13:55 Florian Bezdeka
2025-06-05 8:00 ` Philippe Gerum
0 siblings, 1 reply; 13+ messages in thread
From: Florian Bezdeka @ 2025-06-03 13:55 UTC (permalink / raw)
To: xenomai, Philippe Gerum; +Cc: Jan Kiszka
Hi Philippe,
the following is taken from our CI, testing Dovetail 6.15.
On x86 we have an invalid wait context reported:
[ 151.574032]
[ 151.574039] =============================
[ 151.574043] [ BUG: Invalid wait context ]
[ 151.574046] 6.15.0 #1 Not tainted
[ 151.574048] -----------------------------
[ 151.574048] swapper/0/0 is trying to lock:
[ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60
[ 151.574063] other info that might help us debug this:
[ 151.574064] context-{2:2}
[ 151.574065] no locks held by swapper/0/0.
[ 151.574066] stack backtrace:
[ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full)
[ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
[ 151.574079] IRQ stage: Linux
[ 151.574083] Call Trace:
[ 151.574086] <IRQ>
[ 151.574088] dump_stack_lvl+0x79/0xe0
[ 151.574095] __lock_acquire+0x942/0xbf0
[ 151.574104] lock_acquire+0xe2/0x2f0
[ 151.574107] ? __wake_up+0x21/0x60
[ 151.574111] ? find_held_lock+0x2b/0x80
[ 151.574115] _raw_spin_lock_irqsave+0x49/0x60
[ 151.574120] ? __wake_up+0x21/0x60
[ 151.574122] __wake_up+0x21/0x60
[ 151.574125] xnpipe_wakeup_proc+0x152/0x590
[ 151.574132] handle_synthetic_irq+0xc2/0x250
[ 151.574137] arch_do_IRQ_pipelined+0xca/0x180
[ 151.574141] </IRQ>
[ 151.574142] <TASK>
[ 151.574144] sync_current_irq_stage+0xaa/0x110
[ 151.574147] inband_irq_enable+0x42/0x60
[ 151.574151] cpuidle_idle_call+0x17d/0x200
[ 151.574155] do_idle+0x89/0xd0
[ 151.574158] cpu_startup_entry+0x29/0x30
[ 151.574160] rest_init+0xf0/0x190
[ 151.574164] start_kernel+0x632/0x700
[ 151.574179] x86_64_start_reservations+0x18/0x30
[ 151.574185] x86_64_start_kernel+0x78/0x80
[ 151.574188] common_startup_64+0x13e/0x148
[ 151.574198] </TASK>
That seems to be triggered by the Xenomai 3 smokey testsuite.
Any ideas?
armhf/arm64 seem to be fine.
Best regards,
Florian
^ permalink raw reply [flat|nested] 13+ messages in thread* Re: Dovetail 6.15: x86: Invalid wait context 2025-06-03 13:55 Dovetail 6.15: x86: Invalid wait context Florian Bezdeka @ 2025-06-05 8:00 ` Philippe Gerum 2025-06-05 8:14 ` Florian Bezdeka 0 siblings, 1 reply; 13+ messages in thread From: Philippe Gerum @ 2025-06-05 8:00 UTC (permalink / raw) To: Florian Bezdeka; +Cc: xenomai, Jan Kiszka Florian Bezdeka <florian.bezdeka@siemens.com> writes: > Hi Philippe, > > the following is taken from our CI, testing Dovetail 6.15. > On x86 we have an invalid wait context reported: > > [ 151.574032] > [ 151.574039] ============================= > [ 151.574043] [ BUG: Invalid wait context ] > [ 151.574046] 6.15.0 #1 Not tainted > [ 151.574048] ----------------------------- > [ 151.574048] swapper/0/0 is trying to lock: > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 > [ 151.574063] other info that might help us debug this: > [ 151.574064] context-{2:2} > [ 151.574065] no locks held by swapper/0/0. > [ 151.574066] stack backtrace: > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > [ 151.574079] IRQ stage: Linux > [ 151.574083] Call Trace: > [ 151.574086] <IRQ> > [ 151.574088] dump_stack_lvl+0x79/0xe0 > [ 151.574095] __lock_acquire+0x942/0xbf0 > [ 151.574104] lock_acquire+0xe2/0x2f0 > [ 151.574107] ? __wake_up+0x21/0x60 > [ 151.574111] ? find_held_lock+0x2b/0x80 > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 > [ 151.574120] ? __wake_up+0x21/0x60 > [ 151.574122] __wake_up+0x21/0x60 > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 > [ 151.574132] handle_synthetic_irq+0xc2/0x250 > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 > [ 151.574141] </IRQ> > [ 151.574142] <TASK> > [ 151.574144] sync_current_irq_stage+0xaa/0x110 > [ 151.574147] inband_irq_enable+0x42/0x60 > [ 151.574151] cpuidle_idle_call+0x17d/0x200 > [ 151.574155] do_idle+0x89/0xd0 > [ 151.574158] cpu_startup_entry+0x29/0x30 > [ 151.574160] rest_init+0xf0/0x190 > [ 151.574164] start_kernel+0x632/0x700 > [ 151.574179] x86_64_start_reservations+0x18/0x30 > [ 151.574185] x86_64_start_kernel+0x78/0x80 > [ 151.574188] common_startup_64+0x13e/0x148 > [ 151.574198] </TASK> > > That seems to be triggered by the Xenomai 3 smokey testsuite. > > Any ideas? Does this happen when full preemption is disabled on x86? -- Philippe. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-06-05 8:00 ` Philippe Gerum @ 2025-06-05 8:14 ` Florian Bezdeka 2025-06-05 16:21 ` Bezdeka, Florian 2025-07-11 8:40 ` Philippe Gerum 0 siblings, 2 replies; 13+ messages in thread From: Florian Bezdeka @ 2025-06-05 8:14 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai, Jan Kiszka On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > Hi Philippe, > > > > the following is taken from our CI, testing Dovetail 6.15. > > On x86 we have an invalid wait context reported: > > > > [ 151.574032] > > [ 151.574039] ============================= > > [ 151.574043] [ BUG: Invalid wait context ] > > [ 151.574046] 6.15.0 #1 Not tainted > > [ 151.574048] ----------------------------- > > [ 151.574048] swapper/0/0 is trying to lock: > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 > > [ 151.574063] other info that might help us debug this: > > [ 151.574064] context-{2:2} > > [ 151.574065] no locks held by swapper/0/0. > > [ 151.574066] stack backtrace: > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > > [ 151.574079] IRQ stage: Linux > > [ 151.574083] Call Trace: > > [ 151.574086] <IRQ> > > [ 151.574088] dump_stack_lvl+0x79/0xe0 > > [ 151.574095] __lock_acquire+0x942/0xbf0 > > [ 151.574104] lock_acquire+0xe2/0x2f0 > > [ 151.574107] ? __wake_up+0x21/0x60 > > [ 151.574111] ? find_held_lock+0x2b/0x80 > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 > > [ 151.574120] ? __wake_up+0x21/0x60 > > [ 151.574122] __wake_up+0x21/0x60 > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 > > [ 151.574141] </IRQ> > > [ 151.574142] <TASK> > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 > > [ 151.574147] inband_irq_enable+0x42/0x60 > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 > > [ 151.574155] do_idle+0x89/0xd0 > > [ 151.574158] cpu_startup_entry+0x29/0x30 > > [ 151.574160] rest_init+0xf0/0x190 > > [ 151.574164] start_kernel+0x632/0x700 > > [ 151.574179] x86_64_start_reservations+0x18/0x30 > > [ 151.574185] x86_64_start_kernel+0x78/0x80 > > [ 151.574188] common_startup_64+0x13e/0x148 > > [ 151.574198] </TASK> > > > > That seems to be triggered by the Xenomai 3 smokey testsuite. > > > > Any ideas? > > Does this happen when full preemption is disabled on x86? Test-Log: https://lava.xenomai.org/scheduler/job/21679 Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-06-05 8:14 ` Florian Bezdeka @ 2025-06-05 16:21 ` Bezdeka, Florian 2025-06-05 17:56 ` Philippe Gerum 2025-07-11 8:40 ` Philippe Gerum 1 sibling, 1 reply; 13+ messages in thread From: Bezdeka, Florian @ 2025-06-05 16:21 UTC (permalink / raw) To: rpm@xenomai.org; +Cc: Kiszka, Jan, xenomai@lists.linux.dev On Thu, 2025-06-05 at 10:14 +0200, Florian Bezdeka wrote: > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > Hi Philippe, > > > > > > the following is taken from our CI, testing Dovetail 6.15. > > > On x86 we have an invalid wait context reported: > > > > > > [ 151.574032] > > > [ 151.574039] ============================= > > > [ 151.574043] [ BUG: Invalid wait context ] > > > [ 151.574046] 6.15.0 #1 Not tainted > > > [ 151.574048] ----------------------------- > > > [ 151.574048] swapper/0/0 is trying to lock: > > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 > > > [ 151.574063] other info that might help us debug this: > > > [ 151.574064] context-{2:2} > > > [ 151.574065] no locks held by swapper/0/0. > > > [ 151.574066] stack backtrace: > > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) > > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > > > [ 151.574079] IRQ stage: Linux > > > [ 151.574083] Call Trace: > > > [ 151.574086] <IRQ> > > > [ 151.574088] dump_stack_lvl+0x79/0xe0 > > > [ 151.574095] __lock_acquire+0x942/0xbf0 > > > [ 151.574104] lock_acquire+0xe2/0x2f0 > > > [ 151.574107] ? __wake_up+0x21/0x60 > > > [ 151.574111] ? find_held_lock+0x2b/0x80 > > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 > > > [ 151.574120] ? __wake_up+0x21/0x60 > > > [ 151.574122] __wake_up+0x21/0x60 > > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 > > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 > > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 > > > [ 151.574141] </IRQ> > > > [ 151.574142] <TASK> > > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 > > > [ 151.574147] inband_irq_enable+0x42/0x60 > > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 > > > [ 151.574155] do_idle+0x89/0xd0 > > > [ 151.574158] cpu_startup_entry+0x29/0x30 > > > [ 151.574160] rest_init+0xf0/0x190 > > > [ 151.574164] start_kernel+0x632/0x700 > > > [ 151.574179] x86_64_start_reservations+0x18/0x30 > > > [ 151.574185] x86_64_start_kernel+0x78/0x80 > > > [ 151.574188] common_startup_64+0x13e/0x148 > > > [ 151.574198] </TASK> > > > > > > That seems to be triggered by the Xenomai 3 smokey testsuite. > > > > > > Any ideas? > > > > Does this happen when full preemption is disabled on x86? > > Test-Log: https://lava.xenomai.org/scheduler/job/21679 > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads > Btw: Enabling CONFIG_DEBUG_ENTRY on 6.15 delivers: vmlinux.o: warning: objtool: do_int80_emulation+0x204: return with instrumentation enabled vmlinux.o: warning: objtool: irqentry_exit+0xdc: call to synchronize_pipeline() leaves .noinstr.text section vmlinux.o: warning: objtool: ct_idle_exit+0x1: call to inband_irq_save() leaves .noinstr.text section vmlinux.o: warning: objtool: cpuidle_enter_state+0x1a4: return with instrumentation enabled vmlinux.o: warning: objtool: default_idle+0x7: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: mwait_idle+0x44: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0x9a: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x60: call to inband_irq_enable() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0x54: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_safe_halt+0x1a: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: poll_idle+0x1d: call to inband_irq_enable() leaves .noinstr.text section Is that something we should look into? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-06-05 16:21 ` Bezdeka, Florian @ 2025-06-05 17:56 ` Philippe Gerum 2025-06-22 13:06 ` Florian Bezdeka 0 siblings, 1 reply; 13+ messages in thread From: Philippe Gerum @ 2025-06-05 17:56 UTC (permalink / raw) To: Bezdeka, Florian; +Cc: Kiszka, Jan, xenomai@lists.linux.dev "Bezdeka, Florian" <florian.bezdeka@siemens.com> writes: > On Thu, 2025-06-05 at 10:14 +0200, Florian Bezdeka wrote: >> On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: >> > Florian Bezdeka <florian.bezdeka@siemens.com> writes: >> > >> > > Hi Philippe, >> > > >> > > the following is taken from our CI, testing Dovetail 6.15. >> > > On x86 we have an invalid wait context reported: >> > > >> > > [ 151.574032] >> > > [ 151.574039] ============================= >> > > [ 151.574043] [ BUG: Invalid wait context ] >> > > [ 151.574046] 6.15.0 #1 Not tainted >> > > [ 151.574048] ----------------------------- >> > > [ 151.574048] swapper/0/0 is trying to lock: >> > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 >> > > [ 151.574063] other info that might help us debug this: >> > > [ 151.574064] context-{2:2} >> > > [ 151.574065] no locks held by swapper/0/0. >> > > [ 151.574066] stack backtrace: >> > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) >> > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 >> > > [ 151.574079] IRQ stage: Linux >> > > [ 151.574083] Call Trace: >> > > [ 151.574086] <IRQ> >> > > [ 151.574088] dump_stack_lvl+0x79/0xe0 >> > > [ 151.574095] __lock_acquire+0x942/0xbf0 >> > > [ 151.574104] lock_acquire+0xe2/0x2f0 >> > > [ 151.574107] ? __wake_up+0x21/0x60 >> > > [ 151.574111] ? find_held_lock+0x2b/0x80 >> > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 >> > > [ 151.574120] ? __wake_up+0x21/0x60 >> > > [ 151.574122] __wake_up+0x21/0x60 >> > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 >> > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 >> > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 >> > > [ 151.574141] </IRQ> >> > > [ 151.574142] <TASK> >> > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 >> > > [ 151.574147] inband_irq_enable+0x42/0x60 >> > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 >> > > [ 151.574155] do_idle+0x89/0xd0 >> > > [ 151.574158] cpu_startup_entry+0x29/0x30 >> > > [ 151.574160] rest_init+0xf0/0x190 >> > > [ 151.574164] start_kernel+0x632/0x700 >> > > [ 151.574179] x86_64_start_reservations+0x18/0x30 >> > > [ 151.574185] x86_64_start_kernel+0x78/0x80 >> > > [ 151.574188] common_startup_64+0x13e/0x148 >> > > [ 151.574198] </TASK> >> > > >> > > That seems to be triggered by the Xenomai 3 smokey testsuite. >> > > >> > > Any ideas? >> > >> > Does this happen when full preemption is disabled on x86? >> >> Test-Log: https://lava.xenomai.org/scheduler/job/21679 >> Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads >> > > Btw: Enabling CONFIG_DEBUG_ENTRY on 6.15 delivers: > > vmlinux.o: warning: objtool: do_int80_emulation+0x204: return with instrumentation enabled > vmlinux.o: warning: objtool: irqentry_exit+0xdc: call to synchronize_pipeline() leaves .noinstr.text section > vmlinux.o: warning: objtool: ct_idle_exit+0x1: call to inband_irq_save() leaves .noinstr.text section > vmlinux.o: warning: objtool: cpuidle_enter_state+0x1a4: return with instrumentation enabled > vmlinux.o: warning: objtool: default_idle+0x7: call to inband_irq_disable() leaves .noinstr.text section > vmlinux.o: warning: objtool: mwait_idle+0x44: call to inband_irq_disable() leaves .noinstr.text section > vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0x9a: call to inband_irq_disable() leaves .noinstr.text section > vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x60: call to inband_irq_enable() leaves .noinstr.text section > vmlinux.o: warning: objtool: intel_idle_irq+0x54: call to inband_irq_disable() leaves .noinstr.text section > vmlinux.o: warning: objtool: acpi_safe_halt+0x1a: call to inband_irq_disable() leaves .noinstr.text section > vmlinux.o: warning: objtool: poll_idle+0x1d: call to inband_irq_enable() leaves .noinstr.text section > > Is that something we should look into? Yes, we need to keep those calls within the instrumentation_begin/end blocks, or annotate them. There are imbalanced calls to fix as well, e.g. diff --git a/arch/x86/entry/syscall_32.c b/arch/x86/entry/syscall_32.c index 54749f4736633..cb1b4ad38568c 100644 --- a/arch/x86/entry/syscall_32.c +++ b/arch/x86/entry/syscall_32.c @@ -173,6 +173,7 @@ __visible noinstr void do_int80_emulation(struct pt_regs *regs) if (dovetailing()) { if (nr == EXIT_SYSCALL_OOB) { hard_local_irq_disable(); + instrumentation_end(); return; } if (nr == EXIT_SYSCALL_TAIL) -- Philippe. ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-06-05 17:56 ` Philippe Gerum @ 2025-06-22 13:06 ` Florian Bezdeka 0 siblings, 0 replies; 13+ messages in thread From: Florian Bezdeka @ 2025-06-22 13:06 UTC (permalink / raw) To: Philippe Gerum; +Cc: Kiszka, Jan, xenomai@lists.linux.dev On Thu, 2025-06-05 at 19:56 +0200, Philippe Gerum wrote: > > > > Btw: Enabling CONFIG_DEBUG_ENTRY on 6.15 delivers: > > > > vmlinux.o: warning: objtool: do_int80_emulation+0x204: return with instrumentation enabled > > vmlinux.o: warning: objtool: irqentry_exit+0xdc: call to synchronize_pipeline() leaves .noinstr.text section > > vmlinux.o: warning: objtool: ct_idle_exit+0x1: call to inband_irq_save() leaves .noinstr.text section > > vmlinux.o: warning: objtool: cpuidle_enter_state+0x1a4: return with instrumentation enabled > > vmlinux.o: warning: objtool: default_idle+0x7: call to inband_irq_disable() leaves .noinstr.text section > > vmlinux.o: warning: objtool: mwait_idle+0x44: call to inband_irq_disable() leaves .noinstr.text section > > vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0x9a: call to inband_irq_disable() leaves .noinstr.text section > > vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x60: call to inband_irq_enable() leaves .noinstr.text section > > vmlinux.o: warning: objtool: intel_idle_irq+0x54: call to inband_irq_disable() leaves .noinstr.text section > > vmlinux.o: warning: objtool: acpi_safe_halt+0x1a: call to inband_irq_disable() leaves .noinstr.text section > > vmlinux.o: warning: objtool: poll_idle+0x1d: call to inband_irq_enable() leaves .noinstr.text section > > > > Is that something we should look into? > > Yes, we need to keep those calls within the instrumentation_begin/end > blocks, or annotate them. There are imbalanced calls to fix as well, > e.g. > > diff --git a/arch/x86/entry/syscall_32.c b/arch/x86/entry/syscall_32.c > index 54749f4736633..cb1b4ad38568c 100644 > --- a/arch/x86/entry/syscall_32.c > +++ b/arch/x86/entry/syscall_32.c > @@ -173,6 +173,7 @@ __visible noinstr void do_int80_emulation(struct pt_regs *regs) > if (dovetailing()) { > if (nr == EXIT_SYSCALL_OOB) { > hard_local_irq_disable(); > + instrumentation_end(); > return; > } > if (nr == EXIT_SYSCALL_TAIL) After addressing the low hanging fruits (results parked at [1] the following remains: vmlinux.o: warning: objtool: ct_idle_exit+0x1: call to inband_irq_save() leaves .noinstr.text section vmlinux.o: warning: objtool: default_idle+0x13: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: mwait_idle+0x51: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_processor_ffh_cstate_enter+0x9a: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: cpu_idle_poll.isra.0+0x60: call to inband_irq_enable() leaves .noinstr.text section vmlinux.o: warning: objtool: intel_idle_irq+0x54: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: acpi_safe_halt+0x1a: call to inband_irq_disable() leaves .noinstr.text section vmlinux.o: warning: objtool: poll_idle+0x2a: call to inband_irq_enable() leaves .noinstr.text section The problem / pattern is always the same: Idle implementations - located inside the __cpuidle section - using the raw_local_irq_*() interfaces. Linux has no real "call" in the affected code paths. Everything is inlined, so objtool is happy. inbaind_irq_*() are functions located outside the noinstr / cpuidle section. All functions in the affected call paths have to reside inside the noinstr section. That's why moving inbaind_irq_*() to this section seems too invasive. Any ideas? Best regards, Florian [1] https://gitlab.com/Xenomai/dovetail-hacker-space/-/commits/flo/add-hyperv-support-for-6-15?ref_type=heads ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-06-05 8:14 ` Florian Bezdeka 2025-06-05 16:21 ` Bezdeka, Florian @ 2025-07-11 8:40 ` Philippe Gerum 2025-07-14 14:57 ` Florian Bezdeka 1 sibling, 1 reply; 13+ messages in thread From: Philippe Gerum @ 2025-07-11 8:40 UTC (permalink / raw) To: Florian Bezdeka; +Cc: xenomai, Jan Kiszka Florian Bezdeka <florian.bezdeka@siemens.com> writes: > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: >> Florian Bezdeka <florian.bezdeka@siemens.com> writes: >> >> > Hi Philippe, >> > >> > the following is taken from our CI, testing Dovetail 6.15. >> > On x86 we have an invalid wait context reported: >> > >> > [ 151.574032] >> > [ 151.574039] ============================= >> > [ 151.574043] [ BUG: Invalid wait context ] >> > [ 151.574046] 6.15.0 #1 Not tainted >> > [ 151.574048] ----------------------------- >> > [ 151.574048] swapper/0/0 is trying to lock: >> > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 >> > [ 151.574063] other info that might help us debug this: >> > [ 151.574064] context-{2:2} >> > [ 151.574065] no locks held by swapper/0/0. >> > [ 151.574066] stack backtrace: >> > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) >> > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 >> > [ 151.574079] IRQ stage: Linux >> > [ 151.574083] Call Trace: >> > [ 151.574086] <IRQ> >> > [ 151.574088] dump_stack_lvl+0x79/0xe0 >> > [ 151.574095] __lock_acquire+0x942/0xbf0 >> > [ 151.574104] lock_acquire+0xe2/0x2f0 >> > [ 151.574107] ? __wake_up+0x21/0x60 >> > [ 151.574111] ? find_held_lock+0x2b/0x80 >> > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 >> > [ 151.574120] ? __wake_up+0x21/0x60 >> > [ 151.574122] __wake_up+0x21/0x60 >> > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 >> > [ 151.574132] handle_synthetic_irq+0xc2/0x250 >> > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 >> > [ 151.574141] </IRQ> >> > [ 151.574142] <TASK> >> > [ 151.574144] sync_current_irq_stage+0xaa/0x110 >> > [ 151.574147] inband_irq_enable+0x42/0x60 >> > [ 151.574151] cpuidle_idle_call+0x17d/0x200 >> > [ 151.574155] do_idle+0x89/0xd0 >> > [ 151.574158] cpu_startup_entry+0x29/0x30 >> > [ 151.574160] rest_init+0xf0/0x190 >> > [ 151.574164] start_kernel+0x632/0x700 >> > [ 151.574179] x86_64_start_reservations+0x18/0x30 >> > [ 151.574185] x86_64_start_kernel+0x78/0x80 >> > [ 151.574188] common_startup_64+0x13e/0x148 >> > [ 151.574198] </TASK> >> > >> > That seems to be triggered by the Xenomai 3 smokey testsuite. >> > >> > Any ideas? >> >> Does this happen when full preemption is disabled on x86? > > Test-Log: https://lava.xenomai.org/scheduler/job/21679 > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads I can reproduce a similar issue with EVL right now, but this requires full preemption to be enabled for the bug to show up, starting from 6.15. I can see the reason for lockdep to complain in this case, since synthetic irqs are not relayed via softirqs yet. However, this can't explain why an oldish 4.14 kernel would complain. root@corei7-64-evl:~# evl test basic-xbuf [ 17.771991] [ 17.771994] ================================ [ 17.771995] WARNING: inconsistent lock state [ 17.771998] 6.15.0-00925-g63f9fd7a1279 #2 Tainted: G W [ 17.772000] -------------------------------- [ 17.772002] inconsistent {HARDIRQ-ON-W} -> {IN-HARDIRQ-W} usage. [ 17.772004] swapper/1/0 [HC1[1]:SC0[0]:HE0:SE1] takes: [ 17.772007] ffff888103bf1960 (&xbuf->ibnd.i_event){?.+.}-{3:3}, at: __wake_up+0x1f/0x50 [ 17.772017] {HARDIRQ-ON-W} state was registered at: [ 17.772019] __lock_acquire+0x3af/0x12c0 [ 17.772024] lock_acquire+0xc9/0x2b0 [ 17.772028] rt_spin_lock+0x30/0x160 [ 17.772031] prepare_to_wait_event+0x24/0x1c0 [ 17.772037] inbound_wait_input+0xb7/0xe0 [ 17.772041] do_xbuf_read+0x98/0x240 [ 17.772044] xbuf_read+0x4b/0x60 [ 17.772048] vfs_read+0xcf/0x320 [ 17.772053] ksys_read+0xae/0xd0 [ 17.772055] do_syscall_64+0xbc/0x220 [ 17.772060] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 17.772063] irq event stamp: 28592 [ 17.772064] hardirqs last enabled at (28591): [<ffffffff813f6ffc>] finish_task_switch+0xdc/0x300 [ 17.772070] hardirqs last disabled at (28592): [<ffffffff8145f60b>] sync_current_irq_stage+0x12b/0x160 [ 17.772076] softirqs last enabled at (0): [<ffffffff813a7a86>] copy_process+0x836/0x19b0 [ 17.772079] softirqs last disabled at (0): [<0000000000000000>] 0x0 [ 17.772082] [ 17.772082] other info that might help us debug this: [ 17.772083] Possible unsafe locking scenario: [ 17.772083] [ 17.772084] CPU0 [ 17.772085] ---- [ 17.772085] lock(&xbuf->ibnd.i_event); [ 17.772087] <Interrupt> [ 17.772088] lock(&xbuf->ibnd.i_event); [ 17.772090] [ 17.772090] *** DEADLOCK *** [ 17.772090] [ 17.772090] no locks held by swapper/1/0. [ 17.772092] [ 17.772092] stack backtrace: [ 17.772094] CPU: 1 UID: 0 PID: 0 Comm: swapper/1 Tainted: G W 6.15.0-00925-g63f9fd7a1279 #2 PREEMPT_{RT,(full)} [ 17.772100] Tainted: [W]=WARN [ 17.772101] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-4.fc41 04/01/2014 [ 17.772102] IRQ stage: Linux [ 17.772104] Call Trace: [ 17.772105] <IRQ> [ 17.772108] dump_stack_lvl+0x80/0xc0 [ 17.772113] print_usage_bug.part.0+0x22f/0x2c0 [ 17.772119] mark_lock_irq+0x563/0x8a0 [ 17.772124] ? stack_trace_save+0x3e/0x50 [ 17.772130] ? save_trace+0x53/0x240 [ 17.772135] mark_lock+0x1bd/0x600 [ 17.772140] mark_usage+0x109/0x120 [ 17.772144] __lock_acquire+0x3af/0x12c0 [ 17.772149] ? __resched_curr+0x64/0x260 [ 17.772153] lock_acquire+0xc9/0x2b0 [ 17.772156] ? __wake_up+0x1f/0x50 [ 17.772162] rt_spin_lock+0x30/0x160 [ 17.772165] ? __wake_up+0x1f/0x50 [ 17.772167] ? rcuwait_wake_up+0x83/0x170 [ 17.772170] ? __lock_release.isra.0+0x54/0x150 [ 17.772175] __wake_up+0x1f/0x50 [ 17.772178] irq_work_single+0x79/0xa0 [ 17.772183] irq_work_run+0x40/0x50 [ 17.772187] inband_work_interrupt+0xe/0x20 [ 17.772192] handle_synthetic_irq+0xc2/0x250 [ 17.772197] arch_do_IRQ_pipelined+0x15d/0x1b0 [ 17.772201] </IRQ> [ 17.772202] <TASK> [ 17.772204] sync_current_irq_stage+0xc1/0x160 [ 17.772209] __inband_irq_enable+0x48/0x60 [ 17.772212] finish_task_switch+0xe1/0x300 [ 17.772216] ? finish_task_switch+0xae/0x300 [ 17.772221] ? resume_oob_task+0x84/0x90 [ 17.772226] __schedule+0x342/0x920 [ 17.772233] schedule_idle+0x23/0x40 [ 17.772237] cpu_startup_entry+0x29/0x30 [ 17.772240] start_secondary+0xfc/0x100 [ 17.772243] common_startup_64+0x12c/0x138 [ 17.772251] </TASK> -- Philippe. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-07-11 8:40 ` Philippe Gerum @ 2025-07-14 14:57 ` Florian Bezdeka 2025-07-19 13:24 ` Philippe Gerum 0 siblings, 1 reply; 13+ messages in thread From: Florian Bezdeka @ 2025-07-14 14:57 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai, Jan Kiszka On Fri, 2025-07-11 at 10:40 +0200, Philippe Gerum wrote: > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > > > Hi Philippe, > > > > > > > > the following is taken from our CI, testing Dovetail 6.15. > > > > On x86 we have an invalid wait context reported: > > > > > > > > [ 151.574032] > > > > [ 151.574039] ============================= > > > > [ 151.574043] [ BUG: Invalid wait context ] > > > > [ 151.574046] 6.15.0 #1 Not tainted > > > > [ 151.574048] ----------------------------- > > > > [ 151.574048] swapper/0/0 is trying to lock: > > > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 > > > > [ 151.574063] other info that might help us debug this: > > > > [ 151.574064] context-{2:2} > > > > [ 151.574065] no locks held by swapper/0/0. > > > > [ 151.574066] stack backtrace: > > > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) > > > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > > > > [ 151.574079] IRQ stage: Linux > > > > [ 151.574083] Call Trace: > > > > [ 151.574086] <IRQ> > > > > [ 151.574088] dump_stack_lvl+0x79/0xe0 > > > > [ 151.574095] __lock_acquire+0x942/0xbf0 > > > > [ 151.574104] lock_acquire+0xe2/0x2f0 > > > > [ 151.574107] ? __wake_up+0x21/0x60 > > > > [ 151.574111] ? find_held_lock+0x2b/0x80 > > > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 > > > > [ 151.574120] ? __wake_up+0x21/0x60 > > > > [ 151.574122] __wake_up+0x21/0x60 > > > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 > > > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 > > > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 > > > > [ 151.574141] </IRQ> > > > > [ 151.574142] <TASK> > > > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 > > > > [ 151.574147] inband_irq_enable+0x42/0x60 > > > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 > > > > [ 151.574155] do_idle+0x89/0xd0 > > > > [ 151.574158] cpu_startup_entry+0x29/0x30 > > > > [ 151.574160] rest_init+0xf0/0x190 > > > > [ 151.574164] start_kernel+0x632/0x700 > > > > [ 151.574179] x86_64_start_reservations+0x18/0x30 > > > > [ 151.574185] x86_64_start_kernel+0x78/0x80 > > > > [ 151.574188] common_startup_64+0x13e/0x148 > > > > [ 151.574198] </TASK> > > > > > > > > That seems to be triggered by the Xenomai 3 smokey testsuite. > > > > > > > > Any ideas? > > > > > > Does this happen when full preemption is disabled on x86? > > > > Test-Log: https://lava.xenomai.org/scheduler/job/21679 > > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads > > I can reproduce a similar issue with EVL right now, but this requires > full preemption to be enabled for the bug to show up, starting from > 6.15. I can see the reason for lockdep to complain in this case, since > synthetic irqs are not relayed via softirqs yet. However, this can't > explain why an oldish 4.14 kernel would complain. Thanks for letting us know. But: Where is 4.14 coming into the game? Our reports (from CI testing) were 6.15 related, nothing older. Is there a specific change in 6.15 that broke the synthetic IRQ handling? Are you already working an a fix? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-07-14 14:57 ` Florian Bezdeka @ 2025-07-19 13:24 ` Philippe Gerum 2025-07-19 15:25 ` Philippe Gerum 0 siblings, 1 reply; 13+ messages in thread From: Philippe Gerum @ 2025-07-19 13:24 UTC (permalink / raw) To: Florian Bezdeka; +Cc: xenomai, Jan Kiszka Florian Bezdeka <florian.bezdeka@siemens.com> writes: > On Fri, 2025-07-11 at 10:40 +0200, Philippe Gerum wrote: >> Florian Bezdeka <florian.bezdeka@siemens.com> writes: >> >> > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: >> > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: >> > > >> > > > Hi Philippe, >> > > > >> > > > the following is taken from our CI, testing Dovetail 6.15. >> > > > On x86 we have an invalid wait context reported: >> > > > >> > > > [ 151.574032] >> > > > [ 151.574039] ============================= >> > > > [ 151.574043] [ BUG: Invalid wait context ] >> > > > [ 151.574046] 6.15.0 #1 Not tainted >> > > > [ 151.574048] ----------------------------- >> > > > [ 151.574048] swapper/0/0 is trying to lock: >> > > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 >> > > > [ 151.574063] other info that might help us debug this: >> > > > [ 151.574064] context-{2:2} >> > > > [ 151.574065] no locks held by swapper/0/0. >> > > > [ 151.574066] stack backtrace: >> > > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) >> > > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 >> > > > [ 151.574079] IRQ stage: Linux >> > > > [ 151.574083] Call Trace: >> > > > [ 151.574086] <IRQ> >> > > > [ 151.574088] dump_stack_lvl+0x79/0xe0 >> > > > [ 151.574095] __lock_acquire+0x942/0xbf0 >> > > > [ 151.574104] lock_acquire+0xe2/0x2f0 >> > > > [ 151.574107] ? __wake_up+0x21/0x60 >> > > > [ 151.574111] ? find_held_lock+0x2b/0x80 >> > > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 >> > > > [ 151.574120] ? __wake_up+0x21/0x60 >> > > > [ 151.574122] __wake_up+0x21/0x60 >> > > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 >> > > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 >> > > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 >> > > > [ 151.574141] </IRQ> >> > > > [ 151.574142] <TASK> >> > > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 >> > > > [ 151.574147] inband_irq_enable+0x42/0x60 >> > > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 >> > > > [ 151.574155] do_idle+0x89/0xd0 >> > > > [ 151.574158] cpu_startup_entry+0x29/0x30 >> > > > [ 151.574160] rest_init+0xf0/0x190 >> > > > [ 151.574164] start_kernel+0x632/0x700 >> > > > [ 151.574179] x86_64_start_reservations+0x18/0x30 >> > > > [ 151.574185] x86_64_start_kernel+0x78/0x80 >> > > > [ 151.574188] common_startup_64+0x13e/0x148 >> > > > [ 151.574198] </TASK> >> > > > >> > > > That seems to be triggered by the Xenomai 3 smokey testsuite. >> > > > >> > > > Any ideas? >> > > >> > > Does this happen when full preemption is disabled on x86? >> > >> > Test-Log: https://lava.xenomai.org/scheduler/job/21679 >> > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads >> >> I can reproduce a similar issue with EVL right now, but this requires >> full preemption to be enabled for the bug to show up, starting from >> 6.15. I can see the reason for lockdep to complain in this case, since >> synthetic irqs are not relayed via softirqs yet. However, this can't >> explain why an oldish 4.14 kernel would complain. > > Thanks for letting us know. But: Where is 4.14 coming into the game? Because the link you sent me referred to a kernel config for 4.14.71. I assumed the yocto recipe which contains it did produce the qemu image showing the issue. > Our reports (from CI testing) were 6.15 related, nothing older. > > Is there a specific change in 6.15 that broke the synthetic IRQ > handling? 814b824b4466b irq_work: dovetail: never defer local work posted from oob > Are you already working an a fix? Yep. -- Philippe. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-07-19 13:24 ` Philippe Gerum @ 2025-07-19 15:25 ` Philippe Gerum 2025-07-28 14:37 ` Florian Bezdeka 0 siblings, 1 reply; 13+ messages in thread From: Philippe Gerum @ 2025-07-19 15:25 UTC (permalink / raw) To: Florian Bezdeka; +Cc: xenomai, Jan Kiszka Philippe Gerum <rpm@xenomai.org> writes: > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > >> On Fri, 2025-07-11 at 10:40 +0200, Philippe Gerum wrote: >>> Florian Bezdeka <florian.bezdeka@siemens.com> writes: >>> >>> > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: >>> > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: >>> > > >>> > > > Hi Philippe, >>> > > > >>> > > > the following is taken from our CI, testing Dovetail 6.15. >>> > > > On x86 we have an invalid wait context reported: >>> > > > >>> > > > [ 151.574032] >>> > > > [ 151.574039] ============================= >>> > > > [ 151.574043] [ BUG: Invalid wait context ] >>> > > > [ 151.574046] 6.15.0 #1 Not tainted >>> > > > [ 151.574048] ----------------------------- >>> > > > [ 151.574048] swapper/0/0 is trying to lock: >>> > > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 >>> > > > [ 151.574063] other info that might help us debug this: >>> > > > [ 151.574064] context-{2:2} >>> > > > [ 151.574065] no locks held by swapper/0/0. >>> > > > [ 151.574066] stack backtrace: >>> > > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) >>> > > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 >>> > > > [ 151.574079] IRQ stage: Linux >>> > > > [ 151.574083] Call Trace: >>> > > > [ 151.574086] <IRQ> >>> > > > [ 151.574088] dump_stack_lvl+0x79/0xe0 >>> > > > [ 151.574095] __lock_acquire+0x942/0xbf0 >>> > > > [ 151.574104] lock_acquire+0xe2/0x2f0 >>> > > > [ 151.574107] ? __wake_up+0x21/0x60 >>> > > > [ 151.574111] ? find_held_lock+0x2b/0x80 >>> > > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 >>> > > > [ 151.574120] ? __wake_up+0x21/0x60 >>> > > > [ 151.574122] __wake_up+0x21/0x60 >>> > > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 >>> > > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 >>> > > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 >>> > > > [ 151.574141] </IRQ> >>> > > > [ 151.574142] <TASK> >>> > > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 >>> > > > [ 151.574147] inband_irq_enable+0x42/0x60 >>> > > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 >>> > > > [ 151.574155] do_idle+0x89/0xd0 >>> > > > [ 151.574158] cpu_startup_entry+0x29/0x30 >>> > > > [ 151.574160] rest_init+0xf0/0x190 >>> > > > [ 151.574164] start_kernel+0x632/0x700 >>> > > > [ 151.574179] x86_64_start_reservations+0x18/0x30 >>> > > > [ 151.574185] x86_64_start_kernel+0x78/0x80 >>> > > > [ 151.574188] common_startup_64+0x13e/0x148 >>> > > > [ 151.574198] </TASK> >>> > > > >>> > > > That seems to be triggered by the Xenomai 3 smokey testsuite. >>> > > > >>> > > > Any ideas? >>> > > >>> > > Does this happen when full preemption is disabled on x86? >>> > >>> > Test-Log: https://lava.xenomai.org/scheduler/job/21679 >>> > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads >>> >>> I can reproduce a similar issue with EVL right now, but this requires >>> full preemption to be enabled for the bug to show up, starting from >>> 6.15. I can see the reason for lockdep to complain in this case, since >>> synthetic irqs are not relayed via softirqs yet. However, this can't >>> explain why an oldish 4.14 kernel would complain. >> >> Thanks for letting us know. But: Where is 4.14 coming into the game? > > Because the link you sent me referred to a kernel config for 4.14.71. I > assumed the yocto recipe which contains it did produce the qemu image > showing the issue. > >> Our reports (from CI testing) were 6.15 related, nothing older. >> >> Is there a specific change in 6.15 that broke the synthetic IRQ >> handling? > > 814b824b4466b irq_work: dovetail: never defer local work posted from oob > >> Are you already working an a fix? > > Yep. The broken commit was reverted from v6.1.y-cip-rebase, v6.12.y-rebase and v6.15, things are back to normal for x4 now. I believe this should fix x3 as well. -- Philippe. ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-07-19 15:25 ` Philippe Gerum @ 2025-07-28 14:37 ` Florian Bezdeka 2025-08-20 11:15 ` Bezdeka, Florian 0 siblings, 1 reply; 13+ messages in thread From: Florian Bezdeka @ 2025-07-28 14:37 UTC (permalink / raw) To: Philippe Gerum; +Cc: xenomai, Jan Kiszka On Sat, 2025-07-19 at 17:25 +0200, Philippe Gerum wrote: > Philippe Gerum <rpm@xenomai.org> writes: > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > On Fri, 2025-07-11 at 10:40 +0200, Philippe Gerum wrote: > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > > > > > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: > > > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > > > > > > > > > Hi Philippe, > > > > > > > > > > > > > > the following is taken from our CI, testing Dovetail 6.15. > > > > > > > On x86 we have an invalid wait context reported: > > > > > > > > > > > > > > [ 151.574032] > > > > > > > [ 151.574039] ============================= > > > > > > > [ 151.574043] [ BUG: Invalid wait context ] > > > > > > > [ 151.574046] 6.15.0 #1 Not tainted > > > > > > > [ 151.574048] ----------------------------- > > > > > > > [ 151.574048] swapper/0/0 is trying to lock: > > > > > > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 > > > > > > > [ 151.574063] other info that might help us debug this: > > > > > > > [ 151.574064] context-{2:2} > > > > > > > [ 151.574065] no locks held by swapper/0/0. > > > > > > > [ 151.574066] stack backtrace: > > > > > > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) > > > > > > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > > > > > > > [ 151.574079] IRQ stage: Linux > > > > > > > [ 151.574083] Call Trace: > > > > > > > [ 151.574086] <IRQ> > > > > > > > [ 151.574088] dump_stack_lvl+0x79/0xe0 > > > > > > > [ 151.574095] __lock_acquire+0x942/0xbf0 > > > > > > > [ 151.574104] lock_acquire+0xe2/0x2f0 > > > > > > > [ 151.574107] ? __wake_up+0x21/0x60 > > > > > > > [ 151.574111] ? find_held_lock+0x2b/0x80 > > > > > > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 > > > > > > > [ 151.574120] ? __wake_up+0x21/0x60 > > > > > > > [ 151.574122] __wake_up+0x21/0x60 > > > > > > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 > > > > > > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 > > > > > > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 > > > > > > > [ 151.574141] </IRQ> > > > > > > > [ 151.574142] <TASK> > > > > > > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 > > > > > > > [ 151.574147] inband_irq_enable+0x42/0x60 > > > > > > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 > > > > > > > [ 151.574155] do_idle+0x89/0xd0 > > > > > > > [ 151.574158] cpu_startup_entry+0x29/0x30 > > > > > > > [ 151.574160] rest_init+0xf0/0x190 > > > > > > > [ 151.574164] start_kernel+0x632/0x700 > > > > > > > [ 151.574179] x86_64_start_reservations+0x18/0x30 > > > > > > > [ 151.574185] x86_64_start_kernel+0x78/0x80 > > > > > > > [ 151.574188] common_startup_64+0x13e/0x148 > > > > > > > [ 151.574198] </TASK> > > > > > > > > > > > > > > That seems to be triggered by the Xenomai 3 smokey testsuite. > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > Does this happen when full preemption is disabled on x86? > > > > > > > > > > Test-Log: https://lava.xenomai.org/scheduler/job/21679 > > > > > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads > > > > > > > > I can reproduce a similar issue with EVL right now, but this requires > > > > full preemption to be enabled for the bug to show up, starting from > > > > 6.15. I can see the reason for lockdep to complain in this case, since > > > > synthetic irqs are not relayed via softirqs yet. However, this can't > > > > explain why an oldish 4.14 kernel would complain. > > > > > > Thanks for letting us know. But: Where is 4.14 coming into the game? > > > > Because the link you sent me referred to a kernel config for 4.14.71. I > > assumed the yocto recipe which contains it did produce the qemu image > > showing the issue. > > > > > Our reports (from CI testing) were 6.15 related, nothing older. > > > > > > Is there a specific change in 6.15 that broke the synthetic IRQ > > > handling? > > > > 814b824b4466b irq_work: dovetail: never defer local work posted from oob > > > > > Are you already working an a fix? > > > > Yep. > > The broken commit was reverted from v6.1.y-cip-rebase, v6.12.y-rebase > and v6.15, things are back to normal for x4 now. I believe this should > fix x3 as well. At least our virtual x86 target is still reporting the same problem when running the x3 testsuite. Other archs are still running, let's see. We do not have full preemption enabled. Test run: https://lava.xenomai.org/scheduler/job/22325 Report : https://lava.xenomai.org/scheduler/job/22325#L864 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-07-28 14:37 ` Florian Bezdeka @ 2025-08-20 11:15 ` Bezdeka, Florian 2025-08-20 14:03 ` Philippe Gerum 0 siblings, 1 reply; 13+ messages in thread From: Bezdeka, Florian @ 2025-08-20 11:15 UTC (permalink / raw) To: rpm@xenomai.org; +Cc: Kiszka, Jan, xenomai@lists.linux.dev [-- Attachment #1: Type: text/plain, Size: 5954 bytes --] On Mon, 2025-07-28 at 16:37 +0200, Florian Bezdeka wrote: > On Sat, 2025-07-19 at 17:25 +0200, Philippe Gerum wrote: > > Philippe Gerum <rpm@xenomai.org> writes: > > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > > > On Fri, 2025-07-11 at 10:40 +0200, Philippe Gerum wrote: > > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > > > > > > > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: > > > > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: > > > > > > > > > > > > > > > Hi Philippe, > > > > > > > > > > > > > > > > the following is taken from our CI, testing Dovetail 6.15. > > > > > > > > On x86 we have an invalid wait context reported: > > > > > > > > > > > > > > > > [ 151.574032] > > > > > > > > [ 151.574039] ============================= > > > > > > > > [ 151.574043] [ BUG: Invalid wait context ] > > > > > > > > [ 151.574046] 6.15.0 #1 Not tainted > > > > > > > > [ 151.574048] ----------------------------- > > > > > > > > [ 151.574048] swapper/0/0 is trying to lock: > > > > > > > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 > > > > > > > > [ 151.574063] other info that might help us debug this: > > > > > > > > [ 151.574064] context-{2:2} > > > > > > > > [ 151.574065] no locks held by swapper/0/0. > > > > > > > > [ 151.574066] stack backtrace: > > > > > > > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) > > > > > > > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 > > > > > > > > [ 151.574079] IRQ stage: Linux > > > > > > > > [ 151.574083] Call Trace: > > > > > > > > [ 151.574086] <IRQ> > > > > > > > > [ 151.574088] dump_stack_lvl+0x79/0xe0 > > > > > > > > [ 151.574095] __lock_acquire+0x942/0xbf0 > > > > > > > > [ 151.574104] lock_acquire+0xe2/0x2f0 > > > > > > > > [ 151.574107] ? __wake_up+0x21/0x60 > > > > > > > > [ 151.574111] ? find_held_lock+0x2b/0x80 > > > > > > > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 > > > > > > > > [ 151.574120] ? __wake_up+0x21/0x60 > > > > > > > > [ 151.574122] __wake_up+0x21/0x60 > > > > > > > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 > > > > > > > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 > > > > > > > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 > > > > > > > > [ 151.574141] </IRQ> > > > > > > > > [ 151.574142] <TASK> > > > > > > > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 > > > > > > > > [ 151.574147] inband_irq_enable+0x42/0x60 > > > > > > > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 > > > > > > > > [ 151.574155] do_idle+0x89/0xd0 > > > > > > > > [ 151.574158] cpu_startup_entry+0x29/0x30 > > > > > > > > [ 151.574160] rest_init+0xf0/0x190 > > > > > > > > [ 151.574164] start_kernel+0x632/0x700 > > > > > > > > [ 151.574179] x86_64_start_reservations+0x18/0x30 > > > > > > > > [ 151.574185] x86_64_start_kernel+0x78/0x80 > > > > > > > > [ 151.574188] common_startup_64+0x13e/0x148 > > > > > > > > [ 151.574198] </TASK> > > > > > > > > > > > > > > > > That seems to be triggered by the Xenomai 3 smokey testsuite. > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > Does this happen when full preemption is disabled on x86? > > > > > > > > > > > > Test-Log: https://lava.xenomai.org/scheduler/job/21679 > > > > > > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads > > > > > > > > > > I can reproduce a similar issue with EVL right now, but this requires > > > > > full preemption to be enabled for the bug to show up, starting from > > > > > 6.15. I can see the reason for lockdep to complain in this case, since > > > > > synthetic irqs are not relayed via softirqs yet. However, this can't > > > > > explain why an oldish 4.14 kernel would complain. > > > > > > > > Thanks for letting us know. But: Where is 4.14 coming into the game? > > > > > > Because the link you sent me referred to a kernel config for 4.14.71. I > > > assumed the yocto recipe which contains it did produce the qemu image > > > showing the issue. > > > > > > > Our reports (from CI testing) were 6.15 related, nothing older. > > > > > > > > Is there a specific change in 6.15 that broke the synthetic IRQ > > > > handling? > > > > > > 814b824b4466b irq_work: dovetail: never defer local work posted from oob > > > > > > > Are you already working an a fix? > > > > > > Yep. > > > > The broken commit was reverted from v6.1.y-cip-rebase, v6.12.y-rebase > > and v6.15, things are back to normal for x4 now. I believe this should > > fix x3 as well. > > At least our virtual x86 target is still reporting the same problem > when running the x3 testsuite. Other archs are still running, let's > see. > > We do not have full preemption enabled. > > Test run: https://lava.xenomai.org/scheduler/job/22325 > Report : https://lava.xenomai.org/scheduler/job/22325#L864 > I'm now able to reproduce the same issue with 6.16. The condensed defconfig is attached. xnpipe_wakeup_proc() tries to wake up a wait queue from inband IRQ context ending up in __wake_up_common_lock() where we have spin_lock_irqsave(&wq_head->lock, flags); remaining = __wake_up_common(...); spin_unlock_irqrestore(&wq_head->lock, flags); Is the interrupt state now mixed up, so that lockdep complains? On the other hand having a sleeping mutex in case of PREEMPT_RT enabled in the inband IRQ context is also a problem, especially as this synthetic IRQ would not be threaded. See pipeline_create_inband_sirq() using the IRQF_NO_THREAD flag. That looks like a Xenomai 3 bug to me. I can't tell why 6.15 onward is now complaining. Any other/additional thoughts? Florian [-- Attachment #2: defconfig --] [-- Type: text/plain, Size: 12863 bytes --] CONFIG_LOCALVERSION="-hyprv-xenomai-0" CONFIG_KERNEL_XZ=y CONFIG_SYSVIPC=y CONFIG_POSIX_MQUEUE=y CONFIG_NO_HZ_FULL=y CONFIG_HIGH_RES_TIMERS=y CONFIG_CLOCKSOURCE_WATCHDOG_MAX_SKEW_US=100 CONFIG_BPF_SYSCALL=y CONFIG_BPF_JIT=y CONFIG_BPF_LSM=y CONFIG_PREEMPT_VOLUNTARY=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_BSD_PROCESS_ACCT_V3=y CONFIG_TASKSTATS=y CONFIG_TASK_DELAY_ACCT=y CONFIG_TASK_XACCT=y CONFIG_TASK_IO_ACCOUNTING=y CONFIG_PSI=y CONFIG_IKCONFIG=m CONFIG_MEMCG=y CONFIG_BLK_CGROUP=y CONFIG_CFS_BANDWIDTH=y CONFIG_CGROUP_PIDS=y CONFIG_CGROUP_RDMA=y CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_HUGETLB=y CONFIG_CPUSETS=y CONFIG_CGROUP_DEVICE=y CONFIG_CGROUP_CPUACCT=y CONFIG_CGROUP_PERF=y CONFIG_CGROUP_BPF=y CONFIG_CGROUP_MISC=y CONFIG_NAMESPACES=y CONFIG_USER_NS=y CONFIG_CHECKPOINT_RESTORE=y CONFIG_SCHED_AUTOGROUP=y CONFIG_BLK_DEV_INITRD=y CONFIG_SYSFS_SYSCALL=y CONFIG_EXPERT=y CONFIG_PROFILING=y CONFIG_KEXEC=y CONFIG_KEXEC_FILE=y CONFIG_KEXEC_SIG=y CONFIG_KEXEC_BZIMAGE_VERIFY_SIG=y CONFIG_SMP=y CONFIG_X86_CPU_RESCTRL=y # CONFIG_X86_EXTENDED_PLATFORM is not set CONFIG_X86_INTEL_LPSS=y CONFIG_X86_AMD_PLATFORM_DEVICE=y CONFIG_HYPERVISOR_GUEST=y CONFIG_PARAVIRT=y CONFIG_PARAVIRT_SPINLOCKS=y CONFIG_PVH=y CONFIG_GART_IOMMU=y CONFIG_MAXSMP=y CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y # CONFIG_PERF_EVENTS_INTEL_UNCORE is not set CONFIG_PERF_EVENTS_INTEL_RAPL=m # CONFIG_PERF_EVENTS_INTEL_CSTATE is not set CONFIG_NUMA=y # CONFIG_X86_KERNEL_IBT is not set CONFIG_X86_SGX=y CONFIG_EFI=y CONFIG_EFI_STUB=y CONFIG_EFI_MIXED=y CONFIG_LEGACY_VSYSCALL_NONE=y CONFIG_LIVEPATCH=y CONFIG_HIBERNATION=y CONFIG_PM_DEBUG=y CONFIG_PM_ADVANCED_DEBUG=y CONFIG_ENERGY_MODEL=y # CONFIG_ACPI_AC is not set # CONFIG_ACPI_BATTERY is not set CONFIG_ACPI_BUTTON=m # CONFIG_ACPI_FAN is not set CONFIG_ACPI_DOCK=y CONFIG_ACPI_PCI_SLOT=y CONFIG_ACPI_HOTPLUG_MEMORY=y CONFIG_ACPI_BGRT=y CONFIG_ACPI_HMAT=y CONFIG_ACPI_APEI=y CONFIG_ACPI_APEI_GHES=y CONFIG_ACPI_APEI_PCIEAER=y CONFIG_ACPI_APEI_MEMORY_FAILURE=y CONFIG_ACPI_EXTLOG=y CONFIG_PMIC_OPREGION=y CONFIG_BYTCRC_PMIC_OPREGION=y CONFIG_CHTCRC_PMIC_OPREGION=y CONFIG_CHT_WC_PMIC_OPREGION=y CONFIG_CPU_FREQ_STAT=y CONFIG_CPU_IDLE_GOV_MENU=y CONFIG_INTEL_IDLE=y CONFIG_IA32_EMULATION=y CONFIG_X86_X32_ABI=y CONFIG_KPROBES=y CONFIG_JUMP_LABEL=y CONFIG_RANDOMIZE_KSTACK_OFFSET_DEFAULT=y CONFIG_MODULES=y CONFIG_MODULE_FORCE_LOAD=y CONFIG_MODULE_UNLOAD=y CONFIG_MODULE_FORCE_UNLOAD=y CONFIG_MODVERSIONS=y CONFIG_MODULE_SIG_SHA256=y CONFIG_BLK_DEV_INTEGRITY=y CONFIG_BLK_DEV_ZONED=y CONFIG_BLK_DEV_THROTTLING=y CONFIG_BLK_WBT=y CONFIG_BLK_CGROUP_IOCOST=y CONFIG_BLK_SED_OPAL=y CONFIG_PARTITION_ADVANCED=y CONFIG_ACORN_PARTITION=y CONFIG_ACORN_PARTITION_ICS=y CONFIG_ACORN_PARTITION_RISCIX=y CONFIG_OSF_PARTITION=y CONFIG_AMIGA_PARTITION=y CONFIG_ATARI_PARTITION=y CONFIG_MAC_PARTITION=y CONFIG_BSD_DISKLABEL=y CONFIG_MINIX_SUBPARTITION=y CONFIG_SOLARIS_X86_PARTITION=y CONFIG_UNIXWARE_DISKLABEL=y CONFIG_LDM_PARTITION=y CONFIG_SGI_PARTITION=y CONFIG_ULTRIX_PARTITION=y CONFIG_SUN_PARTITION=y CONFIG_KARMA_PARTITION=y # CONFIG_MQ_IOSCHED_KYBER is not set CONFIG_ZSWAP=y CONFIG_SLAB_FREELIST_RANDOM=y CONFIG_SLAB_FREELIST_HARDENED=y # CONFIG_COMPAT_BRK is not set CONFIG_MEMORY_HOTPLUG=y # CONFIG_COMPACTION is not set CONFIG_PAGE_REPORTING=y # CONFIG_MIGRATION is not set CONFIG_KSM=y CONFIG_DEFAULT_MMAP_MIN_ADDR=65536 CONFIG_MEMORY_FAILURE=y CONFIG_MEM_SOFT_DIRTY=y CONFIG_DEFERRED_STRUCT_PAGE_INIT=y CONFIG_USERFAULTFD=y CONFIG_LRU_GEN=y CONFIG_NUMA_EMU=y CONFIG_NET=y CONFIG_PACKET=y CONFIG_XFRM_SUB_POLICY=y CONFIG_XFRM_MIGRATE=y CONFIG_XFRM_STATISTICS=y CONFIG_XDP_SOCKETS=y CONFIG_IP_MULTICAST=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_FIB_TRIE_STATS=y CONFIG_IP_MULTIPLE_TABLES=y CONFIG_IP_ROUTE_MULTIPATH=y CONFIG_IP_ROUTE_VERBOSE=y CONFIG_IP_MROUTE=y CONFIG_IP_MROUTE_MULTIPLE_TABLES=y CONFIG_IP_PIMSM_V1=y CONFIG_IP_PIMSM_V2=y CONFIG_SYN_COOKIES=y # CONFIG_INET_DIAG is not set CONFIG_TCP_CONG_ADVANCED=y # CONFIG_TCP_CONG_BIC is not set # CONFIG_TCP_CONG_WESTWOOD is not set # CONFIG_TCP_CONG_HTCP is not set CONFIG_TCP_MD5SIG=y CONFIG_IPV6_ROUTER_PREF=y CONFIG_IPV6_ROUTE_INFO=y CONFIG_IPV6_OPTIMISTIC_DAD=y CONFIG_IPV6_MIP6=y # CONFIG_IPV6_SIT is not set CONFIG_IPV6_SUBTREES=y CONFIG_IPV6_MROUTE=y CONFIG_IPV6_MROUTE_MULTIPLE_TABLES=y CONFIG_IPV6_PIMSM_V2=y CONFIG_IPV6_SEG6_LWTUNNEL=y CONFIG_IPV6_SEG6_HMAC=y CONFIG_NETLABEL=y CONFIG_MPTCP=y CONFIG_NETWORK_PHY_TIMESTAMPING=y CONFIG_NETFILTER=y CONFIG_NETFILTER_XTABLES_COMPAT=y CONFIG_IP_NF_IPTABLES_LEGACY=m CONFIG_NET_SCHED=y CONFIG_NET_SCH_FQ_CODEL=y CONFIG_NET_SCH_DEFAULT=y CONFIG_DEFAULT_FQ_CODEL=y CONFIG_NET_EMATCH=y CONFIG_NET_CLS_ACT=y CONFIG_DCB=y CONFIG_MPLS=y CONFIG_NET_MPLS_GSO=y CONFIG_NET_SWITCHDEV=y CONFIG_NET_L3_MASTER_DEV=y CONFIG_CGROUP_NET_PRIO=y CONFIG_CGROUP_NET_CLASSID=y CONFIG_BPF_STREAM_PARSER=y CONFIG_HAMRADIO=y CONFIG_NET_9P=y CONFIG_NET_9P_VIRTIO=y CONFIG_PAGE_POOL_STATS=y CONFIG_PCI=y CONFIG_PCIEPORTBUS=y CONFIG_HOTPLUG_PCI_PCIE=y CONFIG_PCIEAER=y CONFIG_PCIE_DPC=y CONFIG_PCIE_PTM=y CONFIG_PCI_REALLOC_ENABLE_AUTO=y CONFIG_PCI_IOV=y CONFIG_HOTPLUG_PCI=y CONFIG_HOTPLUG_PCI_ACPI=y CONFIG_HOTPLUG_PCI_CPCI=y CONFIG_HOTPLUG_PCI_SHPC=y CONFIG_CXL_BUS=y # CONFIG_CXL_PCI is not set # CONFIG_CXL_ACPI is not set CONFIG_DEVTMPFS=y CONFIG_FW_LOADER_USER_HELPER=y CONFIG_FW_LOADER_COMPRESS=y CONFIG_CONNECTOR=y CONFIG_DMI_SYSFS=y CONFIG_GOOGLE_FIRMWARE=y CONFIG_GOOGLE_COREBOOT_TABLE=y CONFIG_GOOGLE_FRAMEBUFFER_COREBOOT=y CONFIG_EFI_VARS_PSTORE=m CONFIG_APPLE_PROPERTIES=y CONFIG_RESET_ATTACK_MITIGATION=y # CONFIG_PNP_DEBUG_MESSAGES is not set CONFIG_BLK_DEV_FD=m CONFIG_BLK_DEV_LOOP=m # CONFIG_INTEL_MEI is not set CONFIG_PVPANIC=y # CONFIG_SCSI_PROC_FS is not set CONFIG_BLK_DEV_SD=y CONFIG_BLK_DEV_SR=m CONFIG_CHR_DEV_SG=y CONFIG_SCSI_CONSTANTS=y CONFIG_SCSI_LOGGING=y CONFIG_SCSI_SCAN_ASYNC=y CONFIG_SCSI_FC_ATTRS=m CONFIG_MEGARAID_NEWGEN=y CONFIG_SCSI_DH=y CONFIG_ATA=y CONFIG_SATA_ZPODD=y CONFIG_SATA_AHCI=y CONFIG_SATA_AHCI_PLATFORM=y CONFIG_ATA_PIIX=m CONFIG_ATA_GENERIC=m CONFIG_MD=y CONFIG_BLK_DEV_DM=m CONFIG_DM_UEVENT=y CONFIG_DM_AUDIT=y CONFIG_FUSION=y CONFIG_MACINTOSH_DRIVERS=y CONFIG_MAC_EMUMOUSEBTN=y CONFIG_NETDEVICES=y CONFIG_NET_FC=y # CONFIG_NET_VENDOR_ARC is not set CONFIG_NET_TULIP=y # CONFIG_NET_VENDOR_SEEQ is not set CONFIG_FDDI=y CONFIG_HIPPI=y CONFIG_ATH5K_PCI=y # CONFIG_WLAN_VENDOR_TI is not set CONFIG_WAN=y CONFIG_ISDN=y CONFIG_INPUT_MOUSEDEV=y CONFIG_INPUT_MOUSEDEV_PSAUX=y CONFIG_INPUT_JOYDEV=m CONFIG_INPUT_EVDEV=m CONFIG_MOUSE_PS2=m CONFIG_MOUSE_PS2_ELANTECH=y CONFIG_MOUSE_PS2_SENTELIC=y CONFIG_MOUSE_PS2_VMMOUSE=y CONFIG_INPUT_JOYSTICK=y CONFIG_INPUT_TABLET=y CONFIG_INPUT_TOUCHSCREEN=y CONFIG_INPUT_MISC=y CONFIG_INPUT_PCSPKR=m # CONFIG_SERIO_SERPORT is not set CONFIG_SERIO_RAW=m # CONFIG_LEGACY_PTYS is not set CONFIG_SERIAL_8250=y # CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set CONFIG_SERIAL_8250_FINTEK=y CONFIG_SERIAL_8250_CONSOLE=y # CONFIG_SERIAL_8250_EXAR is not set CONFIG_SERIAL_8250_NR_UARTS=32 CONFIG_SERIAL_8250_EXTENDED=y CONFIG_SERIAL_8250_MANY_PORTS=y CONFIG_SERIAL_8250_SHARE_IRQ=y CONFIG_SERIAL_8250_RSA=y CONFIG_SERIAL_8250_DW=y # CONFIG_SERIAL_8250_LPSS is not set CONFIG_SERIAL_NONSTANDARD=y CONFIG_SERIAL_DEV_BUS=y # CONFIG_HW_RANDOM is not set CONFIG_HPET=y CONFIG_I2C=y CONFIG_I2C_PIIX4=m CONFIG_I2C_DESIGNWARE_CORE=y CONFIG_I2C_DESIGNWARE_BAYTRAIL=y CONFIG_SPI=y CONFIG_SPI_MEM=y CONFIG_SPI_SPIDEV=y # CONFIG_PTP_1588_CLOCK_KVM is not set CONFIG_PINCTRL_AMD=y CONFIG_PINCTRL_BAYTRAIL=y CONFIG_PINCTRL_CHERRYVIEW=y CONFIG_PINCTRL_BROXTON=y CONFIG_PINCTRL_CANNONLAKE=y CONFIG_PINCTRL_CEDARFORK=y CONFIG_PINCTRL_DENVERTON=y CONFIG_PINCTRL_GEMINILAKE=y CONFIG_PINCTRL_ICELAKE=y CONFIG_PINCTRL_LEWISBURG=y CONFIG_PINCTRL_SUNRISEPOINT=y CONFIG_PINCTRL_TIGERLAKE=y CONFIG_GPIO_SYSFS=y CONFIG_THERMAL_STATISTICS=y CONFIG_THERMAL_GOV_FAIR_SHARE=y CONFIG_THERMAL_GOV_BANG_BANG=y CONFIG_THERMAL_GOV_USER_SPACE=y CONFIG_THERMAL_GOV_POWER_ALLOCATOR=y CONFIG_DEVFREQ_THERMAL=y # CONFIG_X86_PKG_TEMP_THERMAL is not set CONFIG_INTEL_HFI_THERMAL=y CONFIG_WATCHDOG=y CONFIG_WATCHDOG_SYSFS=y CONFIG_WATCHDOG_HRTIMER_PRETIMEOUT=y CONFIG_INTEL_SOC_PMIC=y CONFIG_INTEL_SOC_PMIC_CHTWC=y CONFIG_MFD_SYSCON=y CONFIG_REGULATOR=y CONFIG_MEDIA_CEC_SUPPORT=y CONFIG_AGP=y CONFIG_AGP_AMD64=y CONFIG_AGP_INTEL=y CONFIG_AGP_SIS=y CONFIG_AGP_VIA=y CONFIG_VGA_SWITCHEROO=y CONFIG_DRM=m # CONFIG_DRM_DEBUG_MODESET_LOCK is not set CONFIG_DRM_LOAD_EDID_FIRMWARE=y CONFIG_FB=y CONFIG_FB_VESA=y CONFIG_FB_EFI=y CONFIG_FB_SIMPLE=y CONFIG_FIRMWARE_EDID=y CONFIG_FB_MODE_HELPERS=y CONFIG_FB_TILEBLITTING=y CONFIG_BACKLIGHT_CLASS_DEVICE=y CONFIG_FRAMEBUFFER_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y CONFIG_SOUND=m # CONFIG_SOUND_OSS_CORE_PRECLAIM is not set CONFIG_SND=m CONFIG_SND_OSSEMUL=y CONFIG_SND_DYNAMIC_MINORS=y CONFIG_HID=m CONFIG_HID_BATTERY_STRENGTH=y CONFIG_HIDRAW=y # CONFIG_I2C_HID is not set CONFIG_NEW_LEDS=y CONFIG_LEDS_CLASS=y CONFIG_LEDS_BRIGHTNESS_HW_CHANGED=y CONFIG_LEDS_TRIGGERS=y CONFIG_LEDS_TRIGGER_DISK=y CONFIG_LEDS_TRIGGER_CPU=y CONFIG_LEDS_TRIGGER_PANIC=y CONFIG_ACCESSIBILITY=y CONFIG_A11Y_BRAILLE_CONSOLE=y CONFIG_EDAC=y # CONFIG_EDAC_DECODE_MCE is not set CONFIG_RTC_CLASS=y CONFIG_DMADEVICES=y CONFIG_ASYNC_TX_DMA=y CONFIG_VIRT_DRIVERS=y CONFIG_VIRTIO_PCI=y CONFIG_STAGING=y CONFIG_STAGING_MEDIA=y CONFIG_CHROME_PLATFORMS=y CONFIG_X86_PLATFORM_DRIVERS_DELL=y # CONFIG_DCDBAS is not set # CONFIG_DELL_RBU is not set # CONFIG_DELL_SMBIOS is not set # CONFIG_DELL_SMO8800 is not set CONFIG_INTEL_PMC_CORE=m CONFIG_INTEL_PMT_TELEMETRY=m CONFIG_INTEL_TURBO_MAX_3=y CONFIG_INTEL_VSEC=m CONFIG_AMD_IOMMU=y CONFIG_INTEL_IOMMU=y CONFIG_INTEL_IOMMU_SVM=y # CONFIG_INTEL_IOMMU_DEFAULT_ON is not set CONFIG_IRQ_REMAP=y CONFIG_PM_DEVFREQ=y CONFIG_MEMORY=y CONFIG_PWM=y CONFIG_PWM_CRC=y CONFIG_RESET_CONTROLLER=y CONFIG_GENERIC_PHY=y CONFIG_POWERCAP=y CONFIG_INTEL_RAPL=m CONFIG_IDLE_INJECT=y CONFIG_DAX=m CONFIG_EXT4_FS=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y CONFIG_EXPORTFS_BLOCK_OPS=y CONFIG_FS_ENCRYPTION=y CONFIG_FS_VERITY=y CONFIG_FS_VERITY_BUILTIN_SIGNATURES=y CONFIG_FANOTIFY=y CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y CONFIG_QUOTA=y CONFIG_QUOTA_NETLINK_INTERFACE=y CONFIG_AUTOFS_FS=y CONFIG_FUSE_FS=m CONFIG_VIRTIO_FS=m CONFIG_VFAT_FS=y CONFIG_EXFAT_FS=y CONFIG_PROC_KCORE=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_TMPFS_INODE64=y CONFIG_HUGETLBFS=y CONFIG_CONFIGFS_FS=m CONFIG_EFIVAR_FS=y CONFIG_9P_FS=y CONFIG_NLS_DEFAULT="utf8" CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_ISO8859_1=y CONFIG_UNICODE=y CONFIG_PERSISTENT_KEYRINGS=y CONFIG_KEY_DH_OPERATIONS=y CONFIG_SECURITY_DMESG_RESTRICT=y CONFIG_SECURITY=y CONFIG_SECURITY_NETWORK_XFRM=y CONFIG_INTEL_TXT=y CONFIG_SECURITY_SELINUX=y CONFIG_SECURITY_TOMOYO=y CONFIG_SECURITY_APPARMOR=y CONFIG_SECURITY_YAMA=y CONFIG_SECURITY_LOCKDOWN_LSM=y CONFIG_SECURITY_LOCKDOWN_LSM_EARLY=y CONFIG_SECURITY_LANDLOCK=y CONFIG_INTEGRITY_SIGNATURE=y CONFIG_INTEGRITY_ASYMMETRIC_KEYS=y CONFIG_INTEGRITY_PLATFORM_KEYRING=y CONFIG_INTEGRITY_MACHINE_KEYRING=y CONFIG_IMA=y CONFIG_IMA_SIG_TEMPLATE=y CONFIG_IMA_DEFAULT_HASH_SHA256=y CONFIG_IMA_APPRAISE=y CONFIG_IMA_ARCH_POLICY=y CONFIG_EVM=y CONFIG_DEFAULT_SECURITY_APPARMOR=y CONFIG_LSM="landlock,lockdown,yama,loadpin,safesetid,integrity,apparmor,selinux,smack,tomoyo,bpf" CONFIG_INIT_ON_ALLOC_DEFAULT_ON=y CONFIG_FORTIFY_SOURCE=y CONFIG_HARDENED_USERCOPY=y CONFIG_BUG_ON_DATA_CORRUPTION=y CONFIG_CRYPTO_CRC32C=m CONFIG_CRYPTO_DEFLATE=y CONFIG_CRYPTO_DRBG_HASH=y CONFIG_CRYPTO_DRBG_CTR=y CONFIG_CRYPTO_AES_NI_INTEL=m CONFIG_CRYPTO_SHA1_SSSE3=m CONFIG_CRYPTO_SHA512_SSSE3=m CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL=m CONFIG_CRYPTO_DEV_CCP=y # CONFIG_CRYPTO_DEV_CCP_DD is not set CONFIG_SIGNED_PE_FILE_VERIFICATION=y CONFIG_SECONDARY_TRUSTED_KEYRING=y CONFIG_SYSTEM_BLACKLIST_KEYRING=y CONFIG_PACKING=y CONFIG_CRYPTO_BLAKE2S_X86=y # CONFIG_XZ_DEC_POWERPC is not set # CONFIG_XZ_DEC_ARM is not set # CONFIG_XZ_DEC_ARMTHUMB is not set # CONFIG_XZ_DEC_SPARC is not set CONFIG_IRQ_POLL=y CONFIG_FONTS=y CONFIG_FONT_8x8=y CONFIG_FONT_8x16=y CONFIG_FONT_TER16x32=y CONFIG_PRINTK_TIME=y CONFIG_BOOT_PRINTK_DELAY=y CONFIG_DYNAMIC_DEBUG=y CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y CONFIG_DEBUG_INFO_BTF=y CONFIG_MODULE_ALLOW_BTF_MISMATCH=y CONFIG_STRIP_ASM_SYMS=y CONFIG_MAGIC_SYSRQ=y CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x01b6 CONFIG_PAGE_EXTENSION=y CONFIG_PAGE_POISONING=y CONFIG_DEBUG_WX=y CONFIG_SCHED_STACK_END_CHECK=y CONFIG_DEBUG_MEMORY_INIT=y CONFIG_DEBUG_DOVETAIL=y CONFIG_HARDLOCKUP_DETECTOR=y CONFIG_SCHEDSTATS=y CONFIG_PROVE_LOCKING=y CONFIG_DEBUG_HARD_LOCKS=y CONFIG_DEBUG_LIST=y # CONFIG_RCU_TRACE is not set CONFIG_STACK_TRACER=y CONFIG_MMIOTRACE=y CONFIG_FTRACE_SYSCALLS=y CONFIG_TRACER_SNAPSHOT=y CONFIG_BLK_DEV_IO_TRACE=y CONFIG_HIST_TRIGGERS=y CONFIG_IO_STRICT_DEVMEM=y # CONFIG_X86_VERBOSE_BOOTUP is not set CONFIG_DEBUG_ENTRY=y CONFIG_MEMTEST=y ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Dovetail 6.15: x86: Invalid wait context 2025-08-20 11:15 ` Bezdeka, Florian @ 2025-08-20 14:03 ` Philippe Gerum 0 siblings, 0 replies; 13+ messages in thread From: Philippe Gerum @ 2025-08-20 14:03 UTC (permalink / raw) To: Bezdeka, Florian; +Cc: Kiszka, Jan, xenomai@lists.linux.dev "Bezdeka, Florian" <florian.bezdeka@siemens.com> writes: > On Mon, 2025-07-28 at 16:37 +0200, Florian Bezdeka wrote: >> On Sat, 2025-07-19 at 17:25 +0200, Philippe Gerum wrote: >> > Philippe Gerum <rpm@xenomai.org> writes: >> > >> > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: >> > > >> > > > On Fri, 2025-07-11 at 10:40 +0200, Philippe Gerum wrote: >> > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: >> > > > > >> > > > > > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote: >> > > > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes: >> > > > > > > >> > > > > > > > Hi Philippe, >> > > > > > > > >> > > > > > > > the following is taken from our CI, testing Dovetail 6.15. >> > > > > > > > On x86 we have an invalid wait context reported: >> > > > > > > > >> > > > > > > > [ 151.574032] >> > > > > > > > [ 151.574039] ============================= >> > > > > > > > [ 151.574043] [ BUG: Invalid wait context ] >> > > > > > > > [ 151.574046] 6.15.0 #1 Not tainted >> > > > > > > > [ 151.574048] ----------------------------- >> > > > > > > > [ 151.574048] swapper/0/0 is trying to lock: >> > > > > > > > [ 151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60 >> > > > > > > > [ 151.574063] other info that might help us debug this: >> > > > > > > > [ 151.574064] context-{2:2} >> > > > > > > > [ 151.574065] no locks held by swapper/0/0. >> > > > > > > > [ 151.574066] stack backtrace: >> > > > > > > > [ 151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) >> > > > > > > > [ 151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014 >> > > > > > > > [ 151.574079] IRQ stage: Linux >> > > > > > > > [ 151.574083] Call Trace: >> > > > > > > > [ 151.574086] <IRQ> >> > > > > > > > [ 151.574088] dump_stack_lvl+0x79/0xe0 >> > > > > > > > [ 151.574095] __lock_acquire+0x942/0xbf0 >> > > > > > > > [ 151.574104] lock_acquire+0xe2/0x2f0 >> > > > > > > > [ 151.574107] ? __wake_up+0x21/0x60 >> > > > > > > > [ 151.574111] ? find_held_lock+0x2b/0x80 >> > > > > > > > [ 151.574115] _raw_spin_lock_irqsave+0x49/0x60 >> > > > > > > > [ 151.574120] ? __wake_up+0x21/0x60 >> > > > > > > > [ 151.574122] __wake_up+0x21/0x60 >> > > > > > > > [ 151.574125] xnpipe_wakeup_proc+0x152/0x590 >> > > > > > > > [ 151.574132] handle_synthetic_irq+0xc2/0x250 >> > > > > > > > [ 151.574137] arch_do_IRQ_pipelined+0xca/0x180 >> > > > > > > > [ 151.574141] </IRQ> >> > > > > > > > [ 151.574142] <TASK> >> > > > > > > > [ 151.574144] sync_current_irq_stage+0xaa/0x110 >> > > > > > > > [ 151.574147] inband_irq_enable+0x42/0x60 >> > > > > > > > [ 151.574151] cpuidle_idle_call+0x17d/0x200 >> > > > > > > > [ 151.574155] do_idle+0x89/0xd0 >> > > > > > > > [ 151.574158] cpu_startup_entry+0x29/0x30 >> > > > > > > > [ 151.574160] rest_init+0xf0/0x190 >> > > > > > > > [ 151.574164] start_kernel+0x632/0x700 >> > > > > > > > [ 151.574179] x86_64_start_reservations+0x18/0x30 >> > > > > > > > [ 151.574185] x86_64_start_kernel+0x78/0x80 >> > > > > > > > [ 151.574188] common_startup_64+0x13e/0x148 >> > > > > > > > [ 151.574198] </TASK> >> > > > > > > > >> > > > > > > > That seems to be triggered by the Xenomai 3 smokey testsuite. >> > > > > > > > >> > > > > > > > Any ideas? >> > > > > > > >> > > > > > > Does this happen when full preemption is disabled on x86? >> > > > > > >> > > > > > Test-Log: https://lava.xenomai.org/scheduler/job/21679 >> > > > > > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads >> > > > > >> > > > > I can reproduce a similar issue with EVL right now, but this requires >> > > > > full preemption to be enabled for the bug to show up, starting from >> > > > > 6.15. I can see the reason for lockdep to complain in this case, since >> > > > > synthetic irqs are not relayed via softirqs yet. However, this can't >> > > > > explain why an oldish 4.14 kernel would complain. >> > > > >> > > > Thanks for letting us know. But: Where is 4.14 coming into the game? >> > > >> > > Because the link you sent me referred to a kernel config for 4.14.71. I >> > > assumed the yocto recipe which contains it did produce the qemu image >> > > showing the issue. >> > > >> > > > Our reports (from CI testing) were 6.15 related, nothing older. >> > > > >> > > > Is there a specific change in 6.15 that broke the synthetic IRQ >> > > > handling? >> > > >> > > 814b824b4466b irq_work: dovetail: never defer local work posted from oob >> > > >> > > > Are you already working an a fix? >> > > >> > > Yep. >> > >> > The broken commit was reverted from v6.1.y-cip-rebase, v6.12.y-rebase >> > and v6.15, things are back to normal for x4 now. I believe this should >> > fix x3 as well. >> >> At least our virtual x86 target is still reporting the same problem >> when running the x3 testsuite. Other archs are still running, let's >> see. >> >> We do not have full preemption enabled. >> >> Test run: https://lava.xenomai.org/scheduler/job/22325 >> Report : https://lava.xenomai.org/scheduler/job/22325#L864 >> > > I'm now able to reproduce the same issue with 6.16. The condensed > defconfig is attached. > > xnpipe_wakeup_proc() tries to wake up a wait queue from inband IRQ > context ending up in __wake_up_common_lock() where we have > > spin_lock_irqsave(&wq_head->lock, flags); > remaining = __wake_up_common(...); > spin_unlock_irqrestore(&wq_head->lock, flags); > > Is the interrupt state now mixed up, so that lockdep complains? On the > other hand having a sleeping mutex in case of PREEMPT_RT enabled in the > inband IRQ context is also a problem, especially as this synthetic IRQ > would not be threaded. See pipeline_create_inband_sirq() using the > IRQF_NO_THREAD flag. > > That looks like a Xenomai 3 bug to me. I can't tell why 6.15 onward is > now complaining. Any other/additional thoughts? > Correct, this is an issue fixed recently while working on the network stack for x4. As a rule of thumb, sirqs in x3 should be replaced by irq_work whenever possible, with IRQ_WORK_HARD_IRQ set for the work struct when the handler is really expected to run in interrupt context. Otherwise, PREEMPT_RT mode would offload the call to a kernel worker thread. -- Philippe. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-08-20 14:03 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-06-03 13:55 Dovetail 6.15: x86: Invalid wait context Florian Bezdeka 2025-06-05 8:00 ` Philippe Gerum 2025-06-05 8:14 ` Florian Bezdeka 2025-06-05 16:21 ` Bezdeka, Florian 2025-06-05 17:56 ` Philippe Gerum 2025-06-22 13:06 ` Florian Bezdeka 2025-07-11 8:40 ` Philippe Gerum 2025-07-14 14:57 ` Florian Bezdeka 2025-07-19 13:24 ` Philippe Gerum 2025-07-19 15:25 ` Philippe Gerum 2025-07-28 14:37 ` Florian Bezdeka 2025-08-20 11:15 ` Bezdeka, Florian 2025-08-20 14:03 ` Philippe Gerum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.