Re: Kernel Oops on alpha with kernel version >=6.9.x

[John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>]

Hi Magnus,

On Wed, 2024-12-04 at 23:22 +0100, Magnus Lindholm wrote:
> I've been looking a bit closer at the RCU problem on Alpha, in the
> case with the bug
> related to interface-renaming after the changes in the networking code
> the code fails
> with an invalid pointer reference. From the stack trace one can
> conclude that this
> happens when using synchronize_rcu_expedited() in stead of
> synchronize_rcu_normal().
> The use of rcu_normal can be enforced by setting kernel parameter
> rcupdate.rcu_normal=1
> at boot. This makes recent kernels boot again on my Alphas, a simple enough
> workaround for now.
> 
> The code fails inside work-queue handler wait_rcu_exp_gp() when its
> trying to call
> rcu_exp_sel_wait_wake(). looking at the code generated from the
> compiler the call
> to rcu_exp_sel_wait_wake() appears to be inline-optimized, so no
> actual call to this
> function. If I add some bogus-code (i.e a print call that references
> the address of a
> local variable, something that the compiler can't optimize away)
> before the call to
> rcu_exp_sel_wait_wake(), the code works! The same effect is achieved
> by declaring
> the local variable as volatile.
> 
> I've also noted a similar behavior in the scsi driver code, where
> unloading of a scsi
> driver kernel module (in my case qla1280) will trigger a kernel Oops. As in the
> example above, this can be mitigated by adding a reference to local variables.
> When doing "rmmod qla1280" scsi_host_dev_release() calls rcu_barrier(). In this
> function call I noticed that the stack was somehow corrupted and the
> return address
> to scsi_host_dev_release() was overwritten. The stack corruption occurs in the
> "for_each_possible_cpu(cpu)" loop inside rcu_barrier(). Below are stack dumps
> from before/after the for_each_possible_cpu loop. The call to
> scsi_host_dev_release
> disappears in stack trace since its return address (fffffc0000b6a3ec)
> is replaced
> by a '1' and at the of the call to rcu_barrier(). We get a kernel Oops
> since the $ra=1 is used as return address.
> 
> In both RCU cases above, stack corruption occurs and the sections that cause
> problems involve the use of kernel threads so concurrency might be an
> issue here.
> Since the RCU code works on other platforms and can be "fixed" on Alpha as well
> just by declaring certain variables as volatile (or by other means
> making sure that
> they are not optimized away from the code) can this be a compiler issue on
> alpha or is it the result of not taking proper measures, in the code,
> to account for the
> weak memory model on Alpha? Or a combination of the two?
> 
> 
> /Magnus Lindholm
> 
> 
> Stack traces showing the corrupted stack frames:
> ----------------------------------------------------------------
> 
> rcu: inside rcu_barrier 5
> CPU: 1 UID: 0 PID: 1430 Comm: rmmod Not tainted 6.12.1-gentoo #43
>         fffffc000987fc88 fffffc0000e66440 fffffc00003a8bc8 0000000000000000
>         fffffc0000e667b0 fffffc000480b5d8 fffffc0000b6a3ec fffffc0004a2a000
>         fffffc0004a2a240 fffffc000480b5d8 0000000000000000 fffffffc00502068
>         0000020001043480 00000200010422a0 0000000000000000 0000000000000000
>         fffffc0000b68efc fffffc0004a2a240 fffffc0006319300 0000000000000000
>         fffffc0000b2ed80 fffffc0004a2a240 fffffc0000b9d278 0000000000000000
>  Trace:
>  [<fffffc00003a8bc8>] rcu_barrier+0x1f8/0x580
>  [<fffffc0000b6a3ec>] scsi_host_dev_release+0xac/0x1cc
>  [<fffffc0000b68efc>] device_release+0x148/0x218
>  [<fffffc0000b2ed80>] kobject_put+0x1d0/0x270
>  [<fffffc00007cac3c>] put_device+0x1c/0x30
>  [<fffffc00007f47cc>] scsi_host_put+0x1c/0x30
>  [<fffffc00007554a4>] pci_device_remove+0x34/0x90
>  [<fffffc00007d5c04>] device_remove+0x64/0xb0
>  [<fffffc00007d7694>] device_release_driver_internal+0x294/0x380
>  [<fffffc00007d783c>] driver_detach+0x7c/0x110
>  [<fffffc00007d5240>] bus_remove_driver+0xa0/0x150
>  [<fffffc00007d80c4>] driver_unregister+0x44/0xa0
>  [<fffffc00007552f8>] pci_unregister_driver+0x38/0xd0
>  [<fffffc00003bbb7c>] sys_delete_module+0x19c/0x320
>  [<fffffc0000310d34>] entSys+0xa4/0xc0
> 
> 
> rcu: inside rcu_barrier 6
> CPU: 1 UID: 0 PID: 1430 Comm: rmmod Not tainted 6.12.1-gentoo #43
>         fffffc000987fc88 fffffc0000e66440 fffffc00003a8c44 0000000000000002
>         fffffc0000e667b0 fffffc0000e44240 0000000000000001 fffffc0004a2a000
>         fffffc0004a2a240 fffffc000480b5d8 0000000000000000 fffffffc00502068
>         0000020001043480 00000200010422a0 0000000000000000 0000000000000000
>         fffffc0000b68efc fffffc0004a2a240 fffffc0006319300 0000000000000000
>         fffffc0000b2ed80 fffffc0004a2a240 fffffc0000b9d278 0000000000000000
>  Trace:
>  [<fffffc00003a8c44>] rcu_barrier+0x274/0x580
>  [<fffffc0000b68efc>] device_release+0x148/0x218
>  [<fffffc0000b2ed80>] kobject_put+0x1d0/0x270
>  [<fffffc00007cac3c>] put_device+0x1c/0x30
>  [<fffffc00007f47cc>] scsi_host_put+0x1c/0x30
>  [<fffffc00007554a4>] pci_device_remove+0x34/0x90
>  [<fffffc00007d5c04>] device_remove+0x64/0xb0
>  [<fffffc00007d7694>] device_release_driver_internal+0x294/0x380
>  [<fffffc00007d783c>] driver_detach+0x7c/0x110
>  [<fffffc00007d5240>] bus_remove_driver+0xa0/0x150
>  [<fffffc00007d80c4>] driver_unregister+0x44/0xa0
>  [<fffffc00007552f8>] pci_unregister_driver+0x38/0xd0
>  [<fffffc00003bbb7c>] sys_delete_module+0x19c/0x320
>  [<fffffc0000310d34>] entSys+0xa4/0xc0
> 
> 
> Unable to handle kernel paging request at virtual address 0000000000000000
> CPU 1
> rmmod(1430): Oops -1
>  pc = [<0000000000000000>]  ra = [<0000000000000001>]  ps = 0000    Not tainted
>  pc is at 0x0
>  ra is at 0x1
>  v0 = 0000000000000007  t0 = fffffc0000ec7aa8  t1 = ffffffffffffffff
>  t2 = fffffc0000e65df0  t3 = 00000000000026f0  t4 = 00000000000028f1
>  t5 = 00000000000c2e20  t6 = 00000000000c2e68  t7 = fffffc000987c000
>  s0 = fffffc0004a2a000  s1 = fffffc0004a2a240  s2 = fffffc000480b5d8
>  s3 = 0000000000000000  s4 = fffffffc00502068  s5 = 0000020001043480
>  s6 = 00000200010422a0
>  a0 = 0000000000000000  a1 = 0000000000000001  a2 = 00000000000028f0
>  a3 = fffffc000987fa38  a4 = 0000000000000000  a5 = 0000000000000000
>  t8 = 00000000000c2e20  t9 = ffffffffffffffec  t10= 0000000000000001
>  t11= 00000001000024f0  pv = fffffc000038a1f0  at = 0000000000000000
>  gp = fffffc0000eb7aa8  sp = 00000000183e6a07
>  Disabling lock debugging due to kernel taint
>  Trace:
>  [<fffffc0000b68efc>] device_release+0x148/0x218
>  [<fffffc0000b2ed80>] kobject_put+0x1d0/0x270
>  [<fffffc00007cac3c>] put_device+0x1c/0x30
>  [<fffffc00007f47cc>] scsi_host_put+0x1c/0x30
>  [<fffffc00007554a4>] pci_device_remove+0x34/0x90
>  [<fffffc00007d5c04>] device_remove+0x64/0xb0
>  [<fffffc00007d7694>] device_release_driver_internal+0x294/0x380
>  [<fffffc00007d783c>] driver_detach+0x7c/0x110
>  [<fffffc00007d5240>] bus_remove_driver+0xa0/0x150
>  [<fffffc00007d80c4>] driver_unregister+0x44/0xa0
>  [<fffffc00007552f8>] pci_unregister_driver+0x38/0xd0
>  [<fffffc00003bbb7c>] sys_delete_module+0x19c/0x320
>  [<fffffc0000310d34>] entSys+0xa4/0xc0
> 
> 
> Below are the changes I made to the kernel source in order mitigate
> the stack corruption problem
> this is not really a fix but it can be of use to gain further
> knowledge on whats really going on:
> ------------------------------------------------------------------------------------
> 
> 
> diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> index ff98233d4aa5..8241313404f7 100644
> --- a/kernel/rcu/tree.c
> +++ b/kernel/rcu/tree.c
> @@ -4553,7 +4553,7 @@ static void rcu_barrier_handler(void *cpu_in)
>   */
>  void rcu_barrier(void)
>  {
> -       uintptr_t cpu;
> +       volatile uintptr_t cpu;
>         unsigned long flags;
>         unsigned long gseq;
>         struct rcu_data *rdp;
> diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> index fb664d3a01c9..afba0ebc80e4 100644
> --- a/kernel/rcu/tree_exp.h
> +++ b/kernel/rcu/tree_exp.h
> @@ -477,7 +477,7 @@ static inline void
> sync_rcu_exp_select_cpus_flush_work(struct rcu_node *rnp)
>   */
>  static void wait_rcu_exp_gp(struct kthread_work *wp)
>  {
> -       struct rcu_exp_work *rewp;
> +       volatile struct rcu_exp_work *rewp;
> 
>         rewp = container_of(wp, struct rcu_exp_work, rew_work);
>         rcu_exp_sel_wait_wake(rewp->rew_s);
> @@ -705,6 +705,7 @@ static void rcu_exp_wait_wake(unsigned long s)
>   */
>  static void rcu_exp_sel_wait_wake(unsigned long s)
>  {
> +       pr_warn("inside rcu_exp_sel_wait_wake, %llx\n",(void*)s);
>         /* Initialize the rcu_node tree in preparation for the wait. */
>         sync_rcu_exp_select_cpus();

I wonder whether these RCU bugs are related to the SMP issue on alpha that's
been introduced/uncovered by f2f84b05e02b7710a201f0017b3272ad7ef703d1 [1].

In particular, one suggestion was to run the RCU stress test and see if the
results will tell anything about issues that might be related to the SMP
problem.

Adrian

> [1] https://bugzilla.kernel.org/show_bug.cgi?id=213143

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer
`. `'   Physicist
  `-    GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913