Crash in __do_IRQ with gcc 15

Crash in __do_IRQ with gcc 15

[Paul Mackerras]

Running Linux on Microwatt with a kernel compiled on an x86-64 system
running Fedora 42 (using the packaged cross-compiler, i.e. the
gcc-powerpcle64-linux-gnu package), I'm seeing a crash like this:

[    0.141591] smp: Bringing up secondary CPUs ...
[    0.167628] BUG: Unable to handle kernel data access on write at 0xc00a0000be8d6004
[    0.175409] Faulting instruction address: 0xc00000000000fcb4
cpu 0x0: Vector: 300 (Data Access) at [c0000000012f78d0]
    pc: c00000000000fcb4: __do_IRQ+0x64/0x84
    lr: c00000000000fccc: __do_IRQ+0x7c/0x84
    sp: c0000000012f7b70
   msr: 9000000000001033
   dar: c00a0000be8d6004
 dsisr: 42000000
  current = 0xc0000000012de000
  paca    = 0xc00000000135d000   irqmask: 0x03   irq_happened: 0x01
    pid   = 0, comm = swapper/0
Linux version 6.15.0-rc1-00001-g72b73737d483-dirty (paulus@thinks) (powerpc64le-linux-gnu-gcc (GCC) 15.0.1 20250329 (Red Hat Cross 15.0.1-0), GNU ld version 2.44-1.fc42) #5 SMP Thu May  8 22:20:34 AEST 2025
enter ? for help
[c0000000012f7b70] c00000000000fd50 do_IRQ+0x7c/0x90 (unreliable)
[c0000000012f7ba0] c000000000007db4 hardware_interrupt_common_virt+0x1c4/0x1d0
--- Exception: 500 (Hardware Interrupt) at c00000000001c2ec arch_local_irq_restore+0x60/0xc4
[c0000000012f7ea0] c000000000083c68 do_idle+0xd4/0xf4
[c0000000012f7ee0] c000000000083e08 cpu_startup_entry+0x34/0x38
[c0000000012f7f10] c00000000000cc7c kernel_init+0x0/0x144
[c0000000012f7f40] c000000001000ecc do_one_initcall+0x0/0x160
[c0000000012f7fe0] c00000000000ba6c start_here_common+0x1c/0x20
0:mon>

What's happening is that gcc 15 seems to be using r2 as an ordinary
register, and r2 has a live value in it at the point where __do_IRQ()
calls call_do_irq().  Since r2 is not in the clobber list for the
inline asm in call_do_irq(), it doesn't get saved and restored around
the call to __do_irq(), and when we come back to __do_IRQ(), it has
been modified.  Then when __do_IRQ() subsequently does a store using
r2, it blows up like the above.

Adding r2 to the clobber list in call_do_irq() fixes it.  Does this
seem like the right fix?  Does it need to be conditional on the gcc
version?  Or is there a better way to fix the problem?

Thanks,
Paul.

Re: Crash in __do_IRQ with gcc 15

[Madhavan Srinivasan]

On 5/10/25 7:05 AM, Paul Mackerras wrote:
> Running Linux on Microwatt with a kernel compiled on an x86-64 system
> running Fedora 42 (using the packaged cross-compiler, i.e. the
> gcc-powerpcle64-linux-gnu package), I'm seeing a crash like this:
> 
> [    0.141591] smp: Bringing up secondary CPUs ...
> [    0.167628] BUG: Unable to handle kernel data access on write at 0xc00a0000be8d6004
> [    0.175409] Faulting instruction address: 0xc00000000000fcb4
> cpu 0x0: Vector: 300 (Data Access) at [c0000000012f78d0]
>     pc: c00000000000fcb4: __do_IRQ+0x64/0x84
>     lr: c00000000000fccc: __do_IRQ+0x7c/0x84
>     sp: c0000000012f7b70
>    msr: 9000000000001033
>    dar: c00a0000be8d6004
>  dsisr: 42000000
>   current = 0xc0000000012de000
>   paca    = 0xc00000000135d000   irqmask: 0x03   irq_happened: 0x01
>     pid   = 0, comm = swapper/0
> Linux version 6.15.0-rc1-00001-g72b73737d483-dirty (paulus@thinks) (powerpc64le-linux-gnu-gcc (GCC) 15.0.1 20250329 (Red Hat Cross 15.0.1-0), GNU ld version 2.44-1.fc42) #5 SMP Thu May  8 22:20:34 AEST 2025
> enter ? for help
> [c0000000012f7b70] c00000000000fd50 do_IRQ+0x7c/0x90 (unreliable)
> [c0000000012f7ba0] c000000000007db4 hardware_interrupt_common_virt+0x1c4/0x1d0
> --- Exception: 500 (Hardware Interrupt) at c00000000001c2ec arch_local_irq_restore+0x60/0xc4
> [c0000000012f7ea0] c000000000083c68 do_idle+0xd4/0xf4
> [c0000000012f7ee0] c000000000083e08 cpu_startup_entry+0x34/0x38
> [c0000000012f7f10] c00000000000cc7c kernel_init+0x0/0x144
> [c0000000012f7f40] c000000001000ecc do_one_initcall+0x0/0x160
> [c0000000012f7fe0] c00000000000ba6c start_here_common+0x1c/0x20
> 0:mon>
> 

Thanks for the debug.

> What's happening is that gcc 15 seems to be using r2 as an ordinary
> register, and r2 has a live value in it at the point where __do_IRQ()
> calls call_do_irq().  Since r2 is not in the clobber list for the
> inline asm in call_do_irq(), it doesn't get saved and restored around
> the call to __do_irq(), and when we come back to __do_IRQ(), it has
> been modified.  Then when __do_IRQ() subsequently does a store using
> r2, it blows up like the above.
> 
> Adding r2 to the clobber list in call_do_irq() fixes it.  Does this
> seem like the right fix?  Does it need to be conditional on the gcc
> version?  Or is there a better way to fix the problem?

I guess if we plan to go with adding r2 to clobber, it has to be gcc 
specific, since my FC41 (gcc 14.1) complains of PIC registers usage
in the clobber list.

inlined from ‘__do_IRQ’ at arch/powerpc/kernel/irq.c:297:3:                              
arch/powerpc/kernel/irq.c:262:9: error: PIC register clobbered by ‘r2’ in ‘asm’              
 262 |         asm volatile (                                                               
     |         ^~~                      

Can you try with this patch, I am testing this in my setup.

Maddy


diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index a0e8b998c9b5..2ce7a4f2b2fb 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -276,7 +276,11 @@ static __always_inline void call_do_irq(struct pt_regs *regs, void *sp)
                   [callee] "i" (__do_irq)
                 : // Clobbers
                   "lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6",
+#if __GNUC__ >= 15
+                  "cr7", "r0", "r2", "r4", "r5", "r6", "r7", "r8", "r9", "r10",
+#else
                   "cr7", "r0", "r4", "r5", "r6", "r7", "r8", "r9", "r10",
+#endif
                   "r11", "r12"
        );
 }



> 
> Thanks,
> Paul.
> 

Re: Crash in __do_IRQ with gcc 15

[Michael Ellerman]

Paul Mackerras <paulus@ozlabs.org> writes:
> Running Linux on Microwatt with a kernel compiled on an x86-64 system
> running Fedora 42 (using the packaged cross-compiler, i.e. the
> gcc-powerpcle64-linux-gnu package), I'm seeing a crash like this:
>
> [    0.141591] smp: Bringing up secondary CPUs ...
> [    0.167628] BUG: Unable to handle kernel data access on write at 0xc00a0000be8d6004
> [    0.175409] Faulting instruction address: 0xc00000000000fcb4
> cpu 0x0: Vector: 300 (Data Access) at [c0000000012f78d0]
>     pc: c00000000000fcb4: __do_IRQ+0x64/0x84
>     lr: c00000000000fccc: __do_IRQ+0x7c/0x84
>     sp: c0000000012f7b70
>    msr: 9000000000001033
>    dar: c00a0000be8d6004
>  dsisr: 42000000
>   current = 0xc0000000012de000
>   paca    = 0xc00000000135d000   irqmask: 0x03   irq_happened: 0x01
>     pid   = 0, comm = swapper/0
> Linux version 6.15.0-rc1-00001-g72b73737d483-dirty (paulus@thinks) (powerpc64le-linux-gnu-gcc (GCC) 15.0.1 20250329 (Red Hat Cross 15.0.1-0), GNU ld version 2.44-1.fc42) #5 SMP Thu May  8 22:20:34 AEST 2025
> enter ? for help
> [c0000000012f7b70] c00000000000fd50 do_IRQ+0x7c/0x90 (unreliable)
> [c0000000012f7ba0] c000000000007db4 hardware_interrupt_common_virt+0x1c4/0x1d0
> --- Exception: 500 (Hardware Interrupt) at c00000000001c2ec arch_local_irq_restore+0x60/0xc4
> [c0000000012f7ea0] c000000000083c68 do_idle+0xd4/0xf4
> [c0000000012f7ee0] c000000000083e08 cpu_startup_entry+0x34/0x38
> [c0000000012f7f10] c00000000000cc7c kernel_init+0x0/0x144
> [c0000000012f7f40] c000000001000ecc do_one_initcall+0x0/0x160
> [c0000000012f7fe0] c00000000000ba6c start_here_common+0x1c/0x20
> 0:mon>
>
> What's happening is that gcc 15 seems to be using r2 as an ordinary
> register, and r2 has a live value in it at the point where __do_IRQ()
> calls call_do_irq().  Since r2 is not in the clobber list for the
> inline asm in call_do_irq(), it doesn't get saved and restored around
> the call to __do_irq(), and when we come back to __do_IRQ(), it has
> been modified.  Then when __do_IRQ() subsequently does a store using
> r2, it blows up like the above.

Are you building with pcrel? Otherwise r2 shouldn't be getting used as
an ordinary register.

Can you show the disassembly of where it's getting used?

There was a change to r2 handling in GCC 15, but AFAICS it was meant to
only affect pcrel code.

Still it's likely our bug because we are being weird and calling a
function inside an inline asm block.

cheers

Re: Crash in __do_IRQ with gcc 15

[LEROY Christophe]

Hi Arnd,

Do you know when you will be able to add GCC 15 to 
https://mirrors.edge.kernel.org/pub/tools/crosstool/ ?

Thanks
Christophe

Le 10/05/2025 à 03:35, Paul Mackerras a écrit :
> Running Linux on Microwatt with a kernel compiled on an x86-64 system
> running Fedora 42 (using the packaged cross-compiler, i.e. the
> gcc-powerpcle64-linux-gnu package), I'm seeing a crash like this:
> 
> [    0.141591] smp: Bringing up secondary CPUs ...
> [    0.167628] BUG: Unable to handle kernel data access on write at 0xc00a0000be8d6004
> [    0.175409] Faulting instruction address: 0xc00000000000fcb4
> cpu 0x0: Vector: 300 (Data Access) at [c0000000012f78d0]
>      pc: c00000000000fcb4: __do_IRQ+0x64/0x84
>      lr: c00000000000fccc: __do_IRQ+0x7c/0x84
>      sp: c0000000012f7b70
>     msr: 9000000000001033
>     dar: c00a0000be8d6004
>   dsisr: 42000000
>    current = 0xc0000000012de000
>    paca    = 0xc00000000135d000   irqmask: 0x03   irq_happened: 0x01
>      pid   = 0, comm = swapper/0
> Linux version 6.15.0-rc1-00001-g72b73737d483-dirty (paulus@thinks) (powerpc64le-linux-gnu-gcc (GCC) 15.0.1 20250329 (Red Hat Cross 15.0.1-0), GNU ld version 2.44-1.fc42) #5 SMP Thu May  8 22:20:34 AEST 2025
> enter ? for help
> [c0000000012f7b70] c00000000000fd50 do_IRQ+0x7c/0x90 (unreliable)
> [c0000000012f7ba0] c000000000007db4 hardware_interrupt_common_virt+0x1c4/0x1d0
> --- Exception: 500 (Hardware Interrupt) at c00000000001c2ec arch_local_irq_restore+0x60/0xc4
> [c0000000012f7ea0] c000000000083c68 do_idle+0xd4/0xf4
> [c0000000012f7ee0] c000000000083e08 cpu_startup_entry+0x34/0x38
> [c0000000012f7f10] c00000000000cc7c kernel_init+0x0/0x144
> [c0000000012f7f40] c000000001000ecc do_one_initcall+0x0/0x160
> [c0000000012f7fe0] c00000000000ba6c start_here_common+0x1c/0x20
> 0:mon>
> 
> What's happening is that gcc 15 seems to be using r2 as an ordinary
> register, and r2 has a live value in it at the point where __do_IRQ()
> calls call_do_irq().  Since r2 is not in the clobber list for the
> inline asm in call_do_irq(), it doesn't get saved and restored around
> the call to __do_irq(), and when we come back to __do_IRQ(), it has
> been modified.  Then when __do_IRQ() subsequently does a store using
> r2, it blows up like the above.
> 
> Adding r2 to the clobber list in call_do_irq() fixes it.  Does this
> seem like the right fix?  Does it need to be conditional on the gcc
> version?  Or is there a better way to fix the problem?
> 
> Thanks,
> Paul.
> 

Re: Crash in __do_IRQ with gcc 15

[Arnd Bergmann]

On Mon, May 12, 2025, at 20:13, LEROY Christophe wrote:
> Hi Arnd,
>
> Do you know when you will be able to add GCC 15 to 
> https://mirrors.edge.kernel.org/pub/tools/crosstool/ ?

I have to create a new build environment for gcc-15 after I lost
access to the machine I used for the earlier builds. I should
be able to do that within a few weeks, but unfortunately it's
a little more work than just rerunning my scripts (which I still
have).

    Arnd

Re: Crash in __do_IRQ with gcc 15

[Paul Mackerras]

On Mon, May 12, 2025 at 01:31:34PM +1000, Michael Ellerman wrote:

> Are you building with pcrel? Otherwise r2 shouldn't be getting used as
> an ordinary register.

Yes, CONFIG_PPC_KERNEL_PCREL=y.  I'll try with n and see what happens.

> Can you show the disassembly of where it's getting used?

c00000000000fc50 <__do_IRQ>:
c00000000000fc50:	a6 02 08 7c 	mflr    r0
c00000000000fc54:	f0 ff c1 fb 	std     r30,-16(r1)
c00000000000fc58:	f8 ff e1 fb 	std     r31,-8(r1)
c00000000000fc5c:	26 01 10 06 	pla     r2,19317860	# c00000000127c0c0 <__irq_regs>
c00000000000fc60:	64 c4 40 38 
c00000000000fc64:	2e 01 10 06 	pla     r10,19857060	# c0000000012ffb08 <hardirq_ctx>
c00000000000fc68:	a4 fe 40 39 
c00000000000fc6c:	10 00 01 f8 	std     r0,16(r1)
c00000000000fc70:	d1 ff 21 f8 	stdu    r1,-48(r1)
c00000000000fc74:	20 00 0d e9 	ld      r8,32(r13)
c00000000000fc78:	78 13 49 7c 	mr      r9,r2
c00000000000fc7c:	2a 40 c9 7f 	ldx     r30,r9,r8
c00000000000fc80:	2a 49 68 7c 	stdx    r3,r8,r9
c00000000000fc84:	00 00 2d a1 	lhz     r9,0(r13)
c00000000000fc88:	24 1f 29 79 	sldi    r9,r9,3
c00000000000fc8c:	2a 48 ea 7f 	ldx     r31,r10,r9
c00000000000fc90:	64 04 29 78 	clrrdi  r9,r1,14
c00000000000fc94:	40 48 3f 7c 	cmpld   r31,r9
c00000000000fc98:	24 00 82 40 	bne     c00000000000fcbc <__do_IRQ+0x6c>
c00000000000fc9c:	78 0b 24 7c 	mr      r4,r1
c00000000000fca0:	e5 f9 ff 4b 	bl      c00000000000f684 <__do_irq>
c00000000000fca4:	26 01 10 06 	pla     r2,19317788	# c00000000127c0c0 <__irq_regs>
c00000000000fca8:	1c c4 40 38 
c00000000000fcac:	20 00 2d e9 	ld      r9,32(r13)
c00000000000fcb0:	30 00 21 38 	addi    r1,r1,48
c00000000000fcb4:	2a 11 c9 7f 	stdx    r30,r9,r2
c00000000000fcb8:	4c c0 50 48 	b       c00000000051bd04 <_restgpr0_30>
c00000000000fcbc:	e1 3f 3f f8 	stdu    r1,16352(r31)
c00000000000fcc0:	78 0b 24 7c 	mr      r4,r1
c00000000000fcc4:	78 fb e1 7f 	mr      r1,r31
c00000000000fcc8:	bd f9 ff 4b 	bl      c00000000000f684 <__do_irq>
c00000000000fccc:	00 00 21 e8 	ld      r1,0(r1)
c00000000000fcd0:	dc ff ff 4b 	b       c00000000000fcac <__do_IRQ+0x5c>

That's the whole of __do_IRQ().  It explodes at c00000000000fcb4 after
the return from the call to __do_irq() at c00000000000fcc8, which is
the one in the inline asm.  The compiler does reload r2 after the
ordinary call to __do_irq() at c00000000000fca0.

> There was a change to r2 handling in GCC 15, but AFAICS it was meant to
> only affect pcrel code.
> 
> Still it's likely our bug because we are being weird and calling a
> function inside an inline asm block.

Yep.

Thanks,
Paul.

Re: Crash in __do_IRQ with gcc 15

[Segher Boessenkool]

Hi!

On Mon, May 12, 2025 at 01:31:34PM +1000, Michael Ellerman wrote:
> Paul Mackerras <paulus@ozlabs.org> writes:
> Are you building with pcrel? Otherwise r2 shouldn't be getting used as
> an ordinary register.

With any ELFv2 in fact, which is implied by the target triple.  PCrel
has nothing to do with it (well, indirectly, functions that use PCrel
less often need a TOC register after all).

GPR2 usually *is* a normal volatile register in ELFv2.  (You detect
ELFv2 by checking if _CALL_ELF is 2).  If you want to treat it
differently (just like on other ABIs) you'll need to arrange for that,
maybe using some inline asm or such?  Ideally this will work everywhere
of course, and you do no longer depend on it accidentally working
elsewhere :-)


Segher

Re: Crash in __do_IRQ with gcc 15

[Paul Mackerras]

On Sun, May 11, 2025 at 06:32:25PM +0530, Madhavan Srinivasan wrote:
> 
> Can you try with this patch, I am testing this in my setup.
> 
> Maddy
> 
> 
> diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
> index a0e8b998c9b5..2ce7a4f2b2fb 100644
> --- a/arch/powerpc/kernel/irq.c
> +++ b/arch/powerpc/kernel/irq.c
> @@ -276,7 +276,11 @@ static __always_inline void call_do_irq(struct pt_regs *regs, void *sp)
>                    [callee] "i" (__do_irq)
>                  : // Clobbers
>                    "lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6",
> +#if __GNUC__ >= 15
> +                  "cr7", "r0", "r2", "r4", "r5", "r6", "r7", "r8", "r9", "r10",
> +#else
>                    "cr7", "r0", "r4", "r5", "r6", "r7", "r8", "r9", "r10",
> +#endif
>                    "r11", "r12"
>         );
>  }

That works as long as I have CONFIG_PPC_KERNEL_PCREL enabled, but with
it disabled, it fails to compile.  With PCREL disabled, the kernel
runs just fine without the r2 clobber.  I guess it needs to be:

#if __GNUC__ >= 15 && IS_ENABLED(CONFIG_PPC_KERNEL_PCREL)

or something like that.

Thanks,
Paul.

Re: Crash in __do_IRQ with gcc 15

[Arnd Bergmann]

On Tue, May 13, 2025, at 12:08, Arnd Bergmann wrote:
> On Mon, May 12, 2025, at 20:13, LEROY Christophe wrote:
>> Hi Arnd,
>>
>> Do you know when you will be able to add GCC 15 to 
>> https://mirrors.edge.kernel.org/pub/tools/crosstool/ ?
>
> I have to create a new build environment for gcc-15 after I lost
> access to the machine I used for the earlier builds. I should
> be able to do that within a few weeks, but unfortunately it's
> a little more work than just rerunning my scripts (which I still
> have).

I've uploaded gcc-15.1 binaries now. I've only tested the arm64,
as usual, so please let me know if the powerpc and x86 hosted
ones cause any problems.

     Arnd

Re: Crash in __do_IRQ with gcc 15

[LEROY Christophe]

Le 14/05/2025 à 04:06, Paul Mackerras a écrit :
> On Sun, May 11, 2025 at 06:32:25PM +0530, Madhavan Srinivasan wrote:
>>
>> Can you try with this patch, I am testing this in my setup.
>>
>> Maddy
>>
>>
>> diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
>> index a0e8b998c9b5..2ce7a4f2b2fb 100644
>> --- a/arch/powerpc/kernel/irq.c
>> +++ b/arch/powerpc/kernel/irq.c
>> @@ -276,7 +276,11 @@ static __always_inline void call_do_irq(struct pt_regs *regs, void *sp)
>>                     [callee] "i" (__do_irq)
>>                   : // Clobbers
>>                     "lr", "xer", "ctr", "memory", "cr0", "cr1", "cr5", "cr6",
>> +#if __GNUC__ >= 15
>> +                  "cr7", "r0", "r2", "r4", "r5", "r6", "r7", "r8", "r9", "r10",
>> +#else
>>                     "cr7", "r0", "r4", "r5", "r6", "r7", "r8", "r9", "r10",
>> +#endif
>>                     "r11", "r12"
>>          );
>>   }
> 
> That works as long as I have CONFIG_PPC_KERNEL_PCREL enabled, but with
> it disabled, it fails to compile.  With PCREL disabled, the kernel
> runs just fine without the r2 clobber.  I guess it needs to be:
> 
> #if __GNUC__ >= 15 && IS_ENABLED(CONFIG_PPC_KERNEL_PCREL)
> 
> or something like that.
> 

I have now done some tests with both gcc 14 and gcc 15 and they behave 
the same, so at the end I think all we need is:

#ifdef CONFIG_PPC_KERNEL_PCREL

without any mention to GCC version.

Re: Crash in __do_IRQ with gcc 15

[Segher Boessenkool]

On Mon, May 19, 2025 at 04:05:05PM +0200, Arnd Bergmann wrote:
> On Tue, May 13, 2025, at 12:08, Arnd Bergmann wrote:
> > On Mon, May 12, 2025, at 20:13, LEROY Christophe wrote:
> >> Hi Arnd,
> >>
> >> Do you know when you will be able to add GCC 15 to 
> >> https://mirrors.edge.kernel.org/pub/tools/crosstool/ ?
> >
> > I have to create a new build environment for gcc-15 after I lost
> > access to the machine I used for the earlier builds. I should
> > be able to do that within a few weeks, but unfortunately it's
> > a little more work than just rerunning my scripts (which I still
> > have).
> 
> I've uploaded gcc-15.1 binaries now. I've only tested the arm64,
> as usual, so please let me know if the powerpc and x86 hosted
> ones cause any problems.

As always: thank you!


Segher