[dovetail][PATCH 0/4] ARM trap fixes and generic handling of trap nesting

[dovetail][PATCH 0/4] ARM trap fixes and generic handling of trap nesting

[Jan Kiszka]

This is both an expression of personal laziness as well as an experiment
update about AI support for Xenomai core development - and it seems it
was fairly successful:

After noticing [1], I simply prompted in Opencode using Fable 5.1 to
review the ARM trap entry paths for inconsistencies:

"in arch/arm, analyse the annotation of trap entries with
dovetail_fault_entry and mark_trap_entry regarding completness and also
corresponding dovetail_fault_exit / mark_trap_exit"

For that, I just presented the wip/flo/v7.3-dovetail-rebase branch. And
the model came back with finding on [2] as well was these patches here.
Next, I asked for a review of arm64 and x86 as well, but there were no
findings.

Finally, I asked for the extension of smokey with some trap test cases
that demonstrate the ARM issues and test similar paths on arm64 and x86.
That turned out to be quite wasteful - I should have filtered out
irrelevant traps first, namely:

- ARM VFP exceptions - no hardware exit to stress them. But I'm leaving
  the related kernel patch in here because the code was already
  instrumented, just apparently incorrectly now.

- x86 Alignment Checks - can be enabled by userspace, but everything
  left and right in common libs will then explode. IOW, they have no use
  case in modern software. At the same time, some patching of libcobalt
  would be needed to handle SIGDEBUG for an #AC migration.

I will share the lengthy test case as well once I'm done with stripping
it down further (if possible). What the tests already demonstrated is
that unpatched kernels crash.

These patches here are almost completely generated. One of my few
contributions was to encode _TLF_OOBTRAP into oob_trap_regs. The idea
of having oob_trap_regs as a nesting detector came from the AI, though.

As usual, AI is still rather verbose in descriptions and comments. Some
of them I already shortened, but most is still left. We can cut down
further, but I would first of all like to have another review round.

While doing that, keep in mind that AI tends to ignore issues left and
right when being told to do a specific task. It may have simply missed
further preexisting issues or even created new ones now. Watch out!

That said, it is definitely impressing to see that a frontier model is
able to build up decent understanding of the Dovetail architecture
without further hand-holding. I would not blindly trust them yet, but
they seem to be ready now to act as debugging agent or at least
companion, even in complex scenarios. Giving some LLM access to a CI
pipeline may even result in automatic regression tracking, probably
already today. I definitely want to explore further what is helpful and
reasonably needed to support stable kernel maintenance.

Jan

PS: If someone is interested in the token costs: Due to my inefficient
model usage, the counter stopped at about 170 EUR. Most tokes went into
test case development and its debugging (#AC issues, e.g.). These
patches here likely costed about 40 EUR with Fable 5.1, checking the
other archs maybe another 30 EUR.

[1] https://lore.kernel.org/xenomai/8ecd3d1a-86d1-4f39-b175-b712c0be6eec@siemens.com/
[2] https://lore.kernel.org/xenomai/2e3b8eec-7911-4abe-8082-97296000e88d@siemens.com/

Jan Kiszka (4):
  dovetail: Track the owner of a trap notification by exception frame
  ARM: dovetail: Fix trap unwind condition in __und_svc
  ARM: dovetail: Notify the core about undefined instruction traps from
    user mode
  ARM: dovetail: Fix VFP exception trap notification

 arch/arm/include/asm/thread_info.h   |  1 -
 arch/arm/include/asm/traps.h         |  2 ++
 arch/arm/kernel/asm-offsets.c        |  2 --
 arch/arm/kernel/entry-armv.S         | 20 ++++----------------
 arch/arm/kernel/swp_emulate.c        | 22 ++++++++++++++++++----
 arch/arm/kernel/traps.c              | 27 +++++++++++++++++++++++++++
 arch/arm/probes/uprobes/core.c       | 10 ++++++++++
 arch/arm/vfp/vfpmodule.c             | 27 +++++++++++++++++++--------
 arch/arm64/include/asm/thread_info.h |  1 -
 arch/riscv/include/asm/thread_info.h |  1 -
 arch/x86/include/asm/thread_info.h   |  2 --
 include/linux/dovetail.h             | 12 ++++++++++--
 include/linux/sched.h                |  5 +++++
 kernel/dovetail.c                    | 10 ++++++++--
 14 files changed, 103 insertions(+), 39 deletions(-)

-- 
2.47.3

[dovetail][PATCH 2/4] ARM: dovetail: Fix trap unwind condition in __und_svc

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

The kernel mode undefined instruction entry notifies the companion
core from assembly, testing the current stage both before the notify
and before the unwind call. The second test is wrong: if the core
demoted the thread to in-band upon notification - the common case -
the thread is no longer running oob when __und_svc_finish is reached,
so the unwind is skipped altogether. As a result,

- handle_oob_trap_exit() is never called for that trap,

- oob_trap_regs stays set, suppressing every later trap notification
  for this thread until some unrelated trap exit eventually clears it
  with a stale unwind.

The C helpers get this right, keying the unwind on the owning exception
frame, not on the stage, also acting as nesting guard.

Replace the open-coded tests with calls to small asmlinkage wrappers
around oob_trap_notify()/oob_trap_unwind(), so that the assembly entry
follows the exact same rules as the C trap handlers. Overhead is
negligible in the exception path. The wrappers are blacklisted from
kprobes like do_undefinstr(), since a probe there would recurse through
__und_svc.

The TI_PIPELINE_STATE/TI_OOB_MASK offsets and the asm/dovetail.h
inclusion in entry-armv.S have no users left, drop them.

Assisted-by: opencode:Fable-5.1
Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch/arm/include/asm/traps.h  |  2 ++
 arch/arm/kernel/asm-offsets.c |  2 --
 arch/arm/kernel/entry-armv.S  | 20 ++++----------------
 arch/arm/kernel/traps.c       | 15 +++++++++++++++
 4 files changed, 21 insertions(+), 18 deletions(-)

diff --git a/arch/arm/include/asm/traps.h b/arch/arm/include/asm/traps.h
index 2621b9fb9b19b..8f7d526c36ecc 100644
--- a/arch/arm/include/asm/traps.h
+++ b/arch/arm/include/asm/traps.h
@@ -44,5 +44,7 @@ asmlinkage int arm_syscall(int no, struct pt_regs *regs);
 asmlinkage void baddataabort(int code, unsigned long instr, struct pt_regs *regs);
 asmlinkage void __div0(void);
 asmlinkage void handle_bad_stack(struct pt_regs *regs);
+asmlinkage void und_svc_trap_notify(struct pt_regs *regs);
+asmlinkage void und_svc_trap_unwind(struct pt_regs *regs);
  #endif
diff --git a/arch/arm/kernel/asm-offsets.c b/arch/arm/kernel/asm-offsets.c
index a5e5a666930ae..6135e9352335b 100644
--- a/arch/arm/kernel/asm-offsets.c
+++ b/arch/arm/kernel/asm-offsets.c
@@ -55,8 +55,6 @@ int main(void)
   DEFINE(TI_ABI_SYSCALL,	offsetof(struct thread_info, abi_syscall));
   DEFINE(TI_TP_VALUE,		offsetof(struct thread_info, tp_value));
   DEFINE(TI_FPSTATE,		offsetof(struct thread_info, fpstate));
-  DEFINE(TI_PIPELINE_STATE,	offsetof(struct thread_info, irq_pipeline_state));
-  DEFINE(TI_OOB_MASK,		STAGE_MASK);
 #ifdef CONFIG_VFP
   DEFINE(TI_VFPSTATE,		offsetof(struct thread_info, vfpstate));
 #ifdef CONFIG_SMP
diff --git a/arch/arm/kernel/entry-armv.S b/arch/arm/kernel/entry-armv.S
index a314c3152072b..1f44c9fad0186 100644
--- a/arch/arm/kernel/entry-armv.S
+++ b/arch/arm/kernel/entry-armv.S
@@ -30,7 +30,6 @@
  #include "entry-header.S"
 #include <asm/probes.h>
-#include <asm/dovetail.h>
  #ifdef CONFIG_HAVE_LD_DEAD_CODE_DATA_ELIMINATION
 #define RELOC_TEXT_NONE .reloc  .text, R_ARM_NONE, .
@@ -298,14 +297,8 @@ __und_svc:
 	svc_entry
 #endif
 #ifdef CONFIG_DOVETAIL
-	get_thread_info tsk
-	ldr	r0, [tsk, #TI_PIPELINE_STATE]	@ get pipeline state
-	tst	r0, #TI_OOB_MASK		@ oob stage?
-	beq	1f
-	mov	r0, #ARM_TRAP_UNDEFINSTR
-	mov	r1, sp				@ r1 = &regs
-	bl	__oob_trap_notify
-1:
+	mov	r0, sp				@ struct pt_regs *regs
+	bl	und_svc_trap_notify
 #endif
  	mov	r1, #4				@ PC correction to apply
@@ -317,13 +310,8 @@ __und_svc:
 __und_svc_finish:
 	get_thread_info tsk
 #ifdef CONFIG_DOVETAIL
-	ldr	r0, [tsk, #TI_PIPELINE_STATE]	@ get pipeline state
-	tst	r0, #TI_OOB_MASK		@ oob stage?
-	beq	1f
-	mov	r0, #ARM_TRAP_UNDEFINSTR
-	mov	r1, sp				@ r1 = &regs
-	bl	__oob_trap_unwind
-1:
+	mov	r0, sp				@ struct pt_regs *regs
+	bl	und_svc_trap_unwind
 #endif
 	ldr	r5, [sp, #S_PSR]		@ Get SVC cpsr
 	svc_exit r5				@ return from exception
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index 60c4725fa8530..8ca5ba26d2598 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -27,6 +27,7 @@
 #include <linux/sched/task_stack.h>
 #include <linux/irq.h>
 #include <linux/vmalloc.h>
+#include <linux/dovetail.h>
  #include <linux/atomic.h>
 #include <asm/cacheflush.h>
@@ -495,6 +496,20 @@ asmlinkage void do_undefinstr(struct pt_regs *regs)
 }
 NOKPROBE_SYMBOL(do_undefinstr)
 +#ifdef CONFIG_DOVETAIL
+asmlinkage void und_svc_trap_notify(struct pt_regs *regs)
+{
+	oob_trap_notify(ARM_TRAP_UNDEFINSTR, regs);
+}
+NOKPROBE_SYMBOL(und_svc_trap_notify)
+
+asmlinkage void und_svc_trap_unwind(struct pt_regs *regs)
+{
+	oob_trap_unwind(ARM_TRAP_UNDEFINSTR, regs);
+}
+NOKPROBE_SYMBOL(und_svc_trap_unwind)
+#endif /* CONFIG_DOVETAIL */
+
 /*
  * Handle FIQ similarly to NMI on x86 systems.
  *
-- 
2.47.3

Re: [dovetail][PATCH 0/4] ARM trap fixes and generic handling of trap nesting

[Philippe Gerum]

Jan Kiszka <jan.kiszka@siemens.com> writes:

> Jan Kiszka (4):
>   dovetail: Track the owner of a trap notification by exception frame
>   ARM: dovetail: Fix trap unwind condition in __und_svc
>   ARM: dovetail: Notify the core about undefined instruction traps from
>     user mode
>   ARM: dovetail: Fix VFP exception trap notification
>
>  arch/arm/include/asm/thread_info.h   |  1 -
>  arch/arm/include/asm/traps.h         |  2 ++
>  arch/arm/kernel/asm-offsets.c        |  2 --
>  arch/arm/kernel/entry-armv.S         | 20 ++++----------------
>  arch/arm/kernel/swp_emulate.c        | 22 ++++++++++++++++++----
>  arch/arm/kernel/traps.c              | 27 +++++++++++++++++++++++++++
>  arch/arm/probes/uprobes/core.c       | 10 ++++++++++
>  arch/arm/vfp/vfpmodule.c             | 27 +++++++++++++++++++--------
>  arch/arm64/include/asm/thread_info.h |  1 -
>  arch/riscv/include/asm/thread_info.h |  1 -
>  arch/x86/include/asm/thread_info.h   |  2 --
>  include/linux/dovetail.h             | 12 ++++++++++--
>  include/linux/sched.h                |  5 +++++
>  kernel/dovetail.c                    | 10 ++++++++--
>  14 files changed, 103 insertions(+), 39 deletions(-)

Please resend the whole series, some patches arrived garbled.

-- 
Philippe.

Re: [dovetail][PATCH 0/4] ARM trap fixes and generic handling of trap nesting

[Jan Kiszka]

On 11.11.26 09:31, Philippe Gerum wrote:
> Jan Kiszka <jan.kiszka@siemens.com> writes:
> 
>> Jan Kiszka (4):
>>   dovetail: Track the owner of a trap notification by exception frame
>>   ARM: dovetail: Fix trap unwind condition in __und_svc
>>   ARM: dovetail: Notify the core about undefined instruction traps from
>>     user mode
>>   ARM: dovetail: Fix VFP exception trap notification
>>
>>  arch/arm/include/asm/thread_info.h   |  1 -
>>  arch/arm/include/asm/traps.h         |  2 ++
>>  arch/arm/kernel/asm-offsets.c        |  2 --
>>  arch/arm/kernel/entry-armv.S         | 20 ++++----------------
>>  arch/arm/kernel/swp_emulate.c        | 22 ++++++++++++++++++----
>>  arch/arm/kernel/traps.c              | 27 +++++++++++++++++++++++++++
>>  arch/arm/probes/uprobes/core.c       | 10 ++++++++++
>>  arch/arm/vfp/vfpmodule.c             | 27 +++++++++++++++++++--------
>>  arch/arm64/include/asm/thread_info.h |  1 -
>>  arch/riscv/include/asm/thread_info.h |  1 -
>>  arch/x86/include/asm/thread_info.h   |  2 --
>>  include/linux/dovetail.h             | 12 ++++++++++--
>>  include/linux/sched.h                |  5 +++++
>>  kernel/dovetail.c                    | 10 ++++++++--
>>  14 files changed, 103 insertions(+), 39 deletions(-)
> 
> Please resend the whole series, some patches arrived garbled.
> 

Done, hope this one was better.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [dovetail][PATCH 0/4] ARM trap fixes and generic handling of trap nesting

[Florian Bezdeka]

On Wed, 2026-11-11 at 13:15 +0100, Jan Kiszka wrote:
> On 11.11.26 09:31, Philippe Gerum wrote:
> > Jan Kiszka <jan.kiszka@siemens.com> writes:
> > 
> > > Jan Kiszka (4):
> > >   dovetail: Track the owner of a trap notification by exception frame
> > >   ARM: dovetail: Fix trap unwind condition in __und_svc
> > >   ARM: dovetail: Notify the core about undefined instruction traps from
> > >     user mode
> > >   ARM: dovetail: Fix VFP exception trap notification
> > > 
> > >  arch/arm/include/asm/thread_info.h   |  1 -
> > >  arch/arm/include/asm/traps.h         |  2 ++
> > >  arch/arm/kernel/asm-offsets.c        |  2 --
> > >  arch/arm/kernel/entry-armv.S         | 20 ++++----------------
> > >  arch/arm/kernel/swp_emulate.c        | 22 ++++++++++++++++++----
> > >  arch/arm/kernel/traps.c              | 27 +++++++++++++++++++++++++++
> > >  arch/arm/probes/uprobes/core.c       | 10 ++++++++++
> > >  arch/arm/vfp/vfpmodule.c             | 27 +++++++++++++++++++--------
> > >  arch/arm64/include/asm/thread_info.h |  1 -
> > >  arch/riscv/include/asm/thread_info.h |  1 -
> > >  arch/x86/include/asm/thread_info.h   |  2 --
> > >  include/linux/dovetail.h             | 12 ++++++++++--
> > >  include/linux/sched.h                |  5 +++++
> > >  kernel/dovetail.c                    | 10 ++++++++--
> > >  14 files changed, 103 insertions(+), 39 deletions(-)
> > 
> > Please resend the whole series, some patches arrived garbled.
> > 
> 
> Done, hope this one was better.
> 

Thanks. I have the same problem here. Only half of the patches made it
to my inbox. Most likely one of the internal infrastructure problems
that we face from time to time.

Florian