Re: Dovetail 6.15: x86: Invalid wait context

[Philippe Gerum <rpm@xenomai.org>]

"Bezdeka, Florian" <florian.bezdeka@siemens.com> writes:

> On Mon, 2025-07-28 at 16:37 +0200, Florian Bezdeka wrote:
>> On Sat, 2025-07-19 at 17:25 +0200, Philippe Gerum wrote:
>> > Philippe Gerum <rpm@xenomai.org> writes:
>> > 
>> > > Florian Bezdeka <florian.bezdeka@siemens.com> writes:
>> > > 
>> > > > On Fri, 2025-07-11 at 10:40 +0200, Philippe Gerum wrote:
>> > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes:
>> > > > > 
>> > > > > > On Thu, 2025-06-05 at 10:00 +0200, Philippe Gerum wrote:
>> > > > > > > Florian Bezdeka <florian.bezdeka@siemens.com> writes:
>> > > > > > > 
>> > > > > > > > Hi Philippe,
>> > > > > > > > 
>> > > > > > > > the following is taken from our CI, testing Dovetail 6.15.
>> > > > > > > > On x86 we have an invalid wait context reported:
>> > > > > > > > 
>> > > > > > > > [  151.574032] 
>> > > > > > > > [  151.574039] =============================
>> > > > > > > > [  151.574043] [ BUG: Invalid wait context ]
>> > > > > > > > [  151.574046] 6.15.0 #1 Not tainted
>> > > > > > > > [  151.574048] -----------------------------
>> > > > > > > > [  151.574048] swapper/0/0 is trying to lock:
>> > > > > > > > [  151.574050] ffffffff841174c0 (&state->readq){....}-{3:3}, at: __wake_up+0x21/0x60
>> > > > > > > > [  151.574063] other info that might help us debug this:
>> > > > > > > > [  151.574064] context-{2:2}
>> > > > > > > > [  151.574065] no locks held by swapper/0/0.
>> > > > > > > > [  151.574066] stack backtrace:
>> > > > > > > > [  151.574073] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0 #1 PREEMPT(full) 
>> > > > > > > > [  151.574078] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2~bpo12+1 04/01/2014
>> > > > > > > > [  151.574079] IRQ stage: Linux
>> > > > > > > > [  151.574083] Call Trace:
>> > > > > > > > [  151.574086]  <IRQ>
>> > > > > > > > [  151.574088]  dump_stack_lvl+0x79/0xe0
>> > > > > > > > [  151.574095]  __lock_acquire+0x942/0xbf0
>> > > > > > > > [  151.574104]  lock_acquire+0xe2/0x2f0
>> > > > > > > > [  151.574107]  ? __wake_up+0x21/0x60
>> > > > > > > > [  151.574111]  ? find_held_lock+0x2b/0x80
>> > > > > > > > [  151.574115]  _raw_spin_lock_irqsave+0x49/0x60
>> > > > > > > > [  151.574120]  ? __wake_up+0x21/0x60
>> > > > > > > > [  151.574122]  __wake_up+0x21/0x60
>> > > > > > > > [  151.574125]  xnpipe_wakeup_proc+0x152/0x590
>> > > > > > > > [  151.574132]  handle_synthetic_irq+0xc2/0x250
>> > > > > > > > [  151.574137]  arch_do_IRQ_pipelined+0xca/0x180
>> > > > > > > > [  151.574141]  </IRQ>
>> > > > > > > > [  151.574142]  <TASK>
>> > > > > > > > [  151.574144]  sync_current_irq_stage+0xaa/0x110
>> > > > > > > > [  151.574147]  inband_irq_enable+0x42/0x60
>> > > > > > > > [  151.574151]  cpuidle_idle_call+0x17d/0x200
>> > > > > > > > [  151.574155]  do_idle+0x89/0xd0
>> > > > > > > > [  151.574158]  cpu_startup_entry+0x29/0x30
>> > > > > > > > [  151.574160]  rest_init+0xf0/0x190
>> > > > > > > > [  151.574164]  start_kernel+0x632/0x700
>> > > > > > > > [  151.574179]  x86_64_start_reservations+0x18/0x30
>> > > > > > > > [  151.574185]  x86_64_start_kernel+0x78/0x80
>> > > > > > > > [  151.574188]  common_startup_64+0x13e/0x148
>> > > > > > > > [  151.574198]  </TASK>
>> > > > > > > > 
>> > > > > > > > That seems to be triggered by the Xenomai 3 smokey testsuite.
>> > > > > > > > 
>> > > > > > > > Any ideas?
>> > > > > > > 
>> > > > > > > Does this happen when full preemption is disabled on x86?
>> > > > > > 
>> > > > > > Test-Log: https://lava.xenomai.org/scheduler/job/21679
>> > > > > > Kernel-Config: https://source.denx.de/Xenomai/xenomai-images/-/blob/next/recipes-kernel/linux/files/amd64_defconfig?ref_type=heads
>> > > > > 
>> > > > > I can reproduce a similar issue with EVL right now, but this requires
>> > > > > full preemption to be enabled for the bug to show up, starting from
>> > > > > 6.15. I can see the reason for lockdep to complain in this case, since
>> > > > > synthetic irqs are not relayed via softirqs yet. However, this can't
>> > > > > explain why an oldish 4.14 kernel would complain.
>> > > > 
>> > > > Thanks for letting us know. But: Where is 4.14 coming into the game?
>> > > 
>> > > Because the link you sent me referred to a kernel config for 4.14.71. I
>> > > assumed the yocto recipe which contains it did produce the qemu image
>> > > showing the issue.
>> > > 
>> > > > Our reports (from CI testing) were 6.15 related, nothing older.
>> > > > 
>> > > > Is there a specific change in 6.15 that broke the synthetic IRQ
>> > > > handling?
>> > > 
>> > > 814b824b4466b irq_work: dovetail: never defer local work posted from oob
>> > > 
>> > > > Are you already working an a fix?
>> > > 
>> > > Yep.
>> > 
>> > The broken commit was reverted from v6.1.y-cip-rebase, v6.12.y-rebase
>> > and v6.15, things are back to normal for x4 now. I believe this should
>> > fix x3 as well.
>> 
>> At least our virtual x86 target is still reporting the same problem
>> when running the x3 testsuite. Other archs are still running, let's
>> see.
>> 
>> We do not have full preemption enabled.
>> 
>> Test run: https://lava.xenomai.org/scheduler/job/22325
>> Report  : https://lava.xenomai.org/scheduler/job/22325#L864
>> 
>
> I'm now able to reproduce the same issue with 6.16. The condensed
> defconfig is attached.
>
> xnpipe_wakeup_proc() tries to wake up a wait queue from inband IRQ
> context ending up in __wake_up_common_lock() where we have	
>
> 	spin_lock_irqsave(&wq_head->lock, flags);
> 	remaining = __wake_up_common(...);
> 	spin_unlock_irqrestore(&wq_head->lock, flags);
>
> Is the interrupt state now mixed up, so that lockdep complains? On the
> other hand having a sleeping mutex in case of PREEMPT_RT enabled in the
> inband IRQ context is also a problem, especially as this synthetic IRQ
> would not be threaded. See pipeline_create_inband_sirq() using the
> IRQF_NO_THREAD flag.
>
> That looks like a Xenomai 3 bug to me. I can't tell why 6.15 onward is
> now complaining. Any other/additional thoughts?
>

Correct, this is an issue fixed recently while working on the network stack
for x4. As a rule of thumb, sirqs in x3 should be replaced by irq_work
whenever possible, with IRQ_WORK_HARD_IRQ set for the work struct when
the handler is really expected to run in interrupt context. Otherwise,
PREEMPT_RT mode would offload the call to a kernel worker thread.

-- 
Philippe.