Re: [PATCH Dovetail 0/2] Obsolete marking tick IRQs with IRQF_TIMER

[Philippe Gerum <rpm@xenomai.org>]

"Bezdeka, Florian" <florian.bezdeka@siemens.com> writes:

> On Fri, 2026-09-11 at 16:01 +0200, Florian Bezdeka wrote:
>> On Fri, 2026-09-11 at 14:53 +0200, Andrew MacPherson wrote:
>> > On Fri, 11 Sept 2026 at 10:48, Florian Bezdeka
>> > <florian.bezdeka@siemens.com> wrote:
>> > > 
>> > > On Thu, 2026-09-10 at 16:38 +0200, Andrew MacPherson wrote:
>> > > > 
>> > > > > kernel BUG at kernel/irq_work.c:245!
>> > > > > Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
>> > > > > CPU: 0 PID: 520 Comm: XXXXXX Tainted: G           O       6.6.48 #4
>> > > > > PC is at irq_work_run_list+0x14/0x64
>> > > > > LR is at irq_work_run_list+0xc/0x64
>> > > > > > 
>> > > > > >  irq_work_run_list from irq_work_run+0x28/0x3c
>> > > > > >  irq_work_run from armv7pmu_handle_irq+0x148/0x150
>> > > > > >  armv7pmu_handle_irq from armpmu_dispatch_irq+0x28/0x80
>> > > > > >  armpmu_dispatch_irq from __handle_irq_event_percpu+0x68/0x224
>> > > > > >  __handle_irq_event_percpu from handle_irq_event+0x58/0xf8
>> > > > > >  handle_irq_event from handle_fasteoi_irq+0x154/0x2f4
>> > > > > >  handle_fasteoi_irq from handle_irq_desc+0x20/0x30
>> > > > > >  handle_irq_desc from arch_do_IRQ_pipelined+0x58/0x90
>> > > > > >  arch_do_IRQ_pipelined from sync_current_irq_stage+0xac/0xe0
>> > > > > >  sync_current_irq_stage from handle_irq_pipelined_finish+0xa0/0x198
>> > > > > >  handle_irq_pipelined_finish from __irq_usr+0x60/0x80
>> > > > > > 
>> > > > > > Kernel panic - not syncing: Fatal exception in interrupt
>> > > > > > 
>> > > > > > 
>> > > 
>> > > I can't reproduce that one here. Not on 7.2 nor on 6.6.49.
>> > > Might be that this is depending on
>> > >   - your workload (irq work)
>> > >   - kernel configuration
>> > >   - xenomai version (which one do you use?)
>> > >   - qemu vs. real hw
>> > > 
>> > > I'm quite sure that this is a different issue as we hit a BUG() in
>> > > irq_work_run_list():
>> > > 
>> > >         BUG_ON(!irqs_disabled() && !IS_ENABLED(CONFIG_PREEMPT_RT));
>> > > 
>> > > On first glance that looks like a corruption of the virtual interrupt
>> > > state. An irq_work event was waiting in the IRQ log and got applied with
>> > > a wrong inband IRQ state.
>> > > 
>> > > Last time we found issues on arm was probably [1]. Seems that this
>> > > series was not backported (yet). [1] got merged into 7.1. Maybe you can
>> > > give it a try. I don't have futher ideas at the moment.
>> > > 
>> > > Best regards,
>> > > Florian
>> > > 
>> > > [1] https://lore.kernel.org/xenomai/87ik6qvqck.fsf@xenomai.org/T/#m31701ab94206574c96cc2d2b24d77d9793359892
>> > 
>> > I tried backporting the patches from [1] and they do in fact resolve
>> > the crash! Narrowing it down somewhat it seems that specifically
>> > patches 5+6 together are enough to do it, if I only apply those and
>> > skip the rest the crash is still resolved. With these patches plus the
>> > earlier one I now get clean results from a perf callgraph run.
>> > 
>> > An LLM's static analysis of the issue is: do_page_fault() checked the
>> > raw hardware IRQ flag instead of the correct in-band-stall state
>> > before re-enabling interrupts - and page faults are frequent enough
>> > (routine memory access) that this wrong check fired constantly. Since
>> > real hardware IRQs are deliberately left on during Dovetail's in-band
>> > IRQ replay, and the stall bit that guards against reentrancy is a
>> > single un-counted flag, that wrong early local_irq_enable() call
>> > opened a window for a genuinely nested interrupt to corrupt the
>> > stall/hardirq bookkeeping that irq_work_run_list()'s assertion depends
>> > on.
>> > 
>> > Thanks again for the help tracking this down!
>> 
>> Thanks for testing and reporting back. Highly appreciated!
>> 
>> So let me inform "stable" maintainers that [1] fixes a real problem. I
>> was just reviewing parts of the arm pipeline implementation back then
>> and realized that there are potential problems. Now they got real ;-)
>> 
>> @Jan, Philippe:
>> Could you please take care of [1] being applied into older, but still
>> maintained branches. Thanks! 
>> 
>> This series would also be needed to fix some perf problems on arm/arm64.
>> But let's wait for some more feedback first.
>
> Forgot one end: Additionally we need
>
> 58e8e0f74c2b ("ARM: irq_pipeline: save registers used for walking tick frames")
>
> to fix the perf issues. Couldn't find it on the list. Seems it sneaked
> in silently ;-)
>

https://lore.kernel.org/xenomai/CAKndYJHo--bOwSh4Sg7wyuPuWnFcHkPrg_bBRLNYWBm3VQ-VnQ@mail.gmail.com/T/#mf36158864c1a3225ef5305f1557052c955260ae2

-- 
Philippe.