[PATCH] risc-v: Fix order of IPI enablement vs RCU startup

[PATCH] risc-v: Fix order of IPI enablement vs RCU startup

[Marc Zyngier]

Conor reports that risc-v tries to enable IPIs before telling the
core code to enable RCU. With the introduction of the mapple tree
as a backing store for the irq descriptors, this results in
a very shouty boot sequence, as RCU is legitimately upset.

Restore some sanity by moving the risc_ipi_enable() call after
notify_cpu_starting(), which explicitly enables RCU on the calling
CPU.

Fixes: 832f15f42646 ("RISC-V: Treat IPIs as normal Linux IRQs")
Reported-by: Conor Dooley <conor@kernel.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20230703-dupe-frying-79ae2ccf94eb@spud
Cc: Anup Patel <apatel@ventanamicro.com>
Cc: Palmer Dabbelt <palmer@rivosinc.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
---
 arch/riscv/kernel/smpboot.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
index bb0b76e1a6d4..f4d6acb38dd0 100644
--- a/arch/riscv/kernel/smpboot.c
+++ b/arch/riscv/kernel/smpboot.c
@@ -238,10 +238,11 @@ asmlinkage __visible void smp_callin(void)
 	mmgrab(mm);
 	current->active_mm = mm;
 
-	riscv_ipi_enable();
-
 	store_cpu_topology(curr_cpuid);
 	notify_cpu_starting(curr_cpuid);
+
+	riscv_ipi_enable();
+
 	numa_add_cpu(curr_cpuid);
 	set_cpu_online(curr_cpuid, 1);
 	probe_vendor_features(curr_cpuid);
-- 
2.34.1

Re: [PATCH] risc-v: Fix order of IPI enablement vs RCU startup

[Palmer Dabbelt]

On Mon, 03 Jul 2023 11:31:26 PDT (-0700), Marc Zyngier wrote:
> Conor reports that risc-v tries to enable IPIs before telling the
> core code to enable RCU. With the introduction of the mapple tree
> as a backing store for the irq descriptors, this results in
> a very shouty boot sequence, as RCU is legitimately upset.
>
> Restore some sanity by moving the risc_ipi_enable() call after
> notify_cpu_starting(), which explicitly enables RCU on the calling
> CPU.
>
> Fixes: 832f15f42646 ("RISC-V: Treat IPIs as normal Linux IRQs")
> Reported-by: Conor Dooley <conor@kernel.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Link: https://lore.kernel.org/r/20230703-dupe-frying-79ae2ccf94eb@spud
> Cc: Anup Patel <apatel@ventanamicro.com>
> Cc: Palmer Dabbelt <palmer@rivosinc.com>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> ---
>  arch/riscv/kernel/smpboot.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
> index bb0b76e1a6d4..f4d6acb38dd0 100644
> --- a/arch/riscv/kernel/smpboot.c
> +++ b/arch/riscv/kernel/smpboot.c
> @@ -238,10 +238,11 @@ asmlinkage __visible void smp_callin(void)
>  	mmgrab(mm);
>  	current->active_mm = mm;
>
> -	riscv_ipi_enable();
> -
>  	store_cpu_topology(curr_cpuid);
>  	notify_cpu_starting(curr_cpuid);
> +
> +	riscv_ipi_enable();
> +
>  	numa_add_cpu(curr_cpuid);
>  	set_cpu_online(curr_cpuid, 1);
>  	probe_vendor_features(curr_cpuid);

Thanks.  I was going to send another PR this week anyway, I can just 
pick this up if you want?  Either way I'll look, I've still got a few 
hiccups from trying to clean up my staging bits post-merge but hopefully 
it's not too bad...

Re: [PATCH] risc-v: Fix order of IPI enablement vs RCU startup

[Marc Zyngier]

On Mon, 03 Jul 2023 19:45:43 +0100,
Palmer Dabbelt <palmer@rivosinc.com> wrote:
> 
> On Mon, 03 Jul 2023 11:31:26 PDT (-0700), Marc Zyngier wrote:
> > Conor reports that risc-v tries to enable IPIs before telling the
> > core code to enable RCU. With the introduction of the mapple tree
> > as a backing store for the irq descriptors, this results in
> > a very shouty boot sequence, as RCU is legitimately upset.
> > 
> > Restore some sanity by moving the risc_ipi_enable() call after
> > notify_cpu_starting(), which explicitly enables RCU on the calling
> > CPU.
> > 
> > Fixes: 832f15f42646 ("RISC-V: Treat IPIs as normal Linux IRQs")
> > Reported-by: Conor Dooley <conor@kernel.org>
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > Link: https://lore.kernel.org/r/20230703-dupe-frying-79ae2ccf94eb@spud
> > Cc: Anup Patel <apatel@ventanamicro.com>
> > Cc: Palmer Dabbelt <palmer@rivosinc.com>
> > Cc: Linus Torvalds <torvalds@linux-foundation.org>
> > ---
> >  arch/riscv/kernel/smpboot.c | 5 +++--
> >  1 file changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
> > index bb0b76e1a6d4..f4d6acb38dd0 100644
> > --- a/arch/riscv/kernel/smpboot.c
> > +++ b/arch/riscv/kernel/smpboot.c
> > @@ -238,10 +238,11 @@ asmlinkage __visible void smp_callin(void)
> >  	mmgrab(mm);
> >  	current->active_mm = mm;
> > 
> > -	riscv_ipi_enable();
> > -
> >  	store_cpu_topology(curr_cpuid);
> >  	notify_cpu_starting(curr_cpuid);
> > +
> > +	riscv_ipi_enable();
> > +
> >  	numa_add_cpu(curr_cpuid);
> >  	set_cpu_online(curr_cpuid, 1);
> >  	probe_vendor_features(curr_cpuid);
> 
> Thanks.  I was going to send another PR this week anyway, I can just
> pick this up if you want?

Yes please!

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH] risc-v: Fix order of IPI enablement vs RCU startup

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 1729 bytes --]

On Mon, Jul 03, 2023 at 07:31:26PM +0100, Marc Zyngier wrote:
> Conor reports that risc-v tries to enable IPIs before telling the
> core code to enable RCU. With the introduction of the mapple tree
> as a backing store for the irq descriptors, this results in
> a very shouty boot sequence, as RCU is legitimately upset.
> 
> Restore some sanity by moving the risc_ipi_enable() call after
> notify_cpu_starting(), which explicitly enables RCU on the calling
> CPU.
> 
> Fixes: 832f15f42646 ("RISC-V: Treat IPIs as normal Linux IRQs")
> Reported-by: Conor Dooley <conor@kernel.org>

RCU no longer chucks the toys at least, and none of the wheels seem to
have fallen off the pram in the process.
Tested-by: Conor Dooley <conor.dooley@microchip.com>

Thanks guys!

> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Link: https://lore.kernel.org/r/20230703-dupe-frying-79ae2ccf94eb@spud
> Cc: Anup Patel <apatel@ventanamicro.com>
> Cc: Palmer Dabbelt <palmer@rivosinc.com>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> ---
>  arch/riscv/kernel/smpboot.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
> index bb0b76e1a6d4..f4d6acb38dd0 100644
> --- a/arch/riscv/kernel/smpboot.c
> +++ b/arch/riscv/kernel/smpboot.c
> @@ -238,10 +238,11 @@ asmlinkage __visible void smp_callin(void)
>  	mmgrab(mm);
>  	current->active_mm = mm;
>  
> -	riscv_ipi_enable();
> -
>  	store_cpu_topology(curr_cpuid);
>  	notify_cpu_starting(curr_cpuid);
> +
> +	riscv_ipi_enable();
> +
>  	numa_add_cpu(curr_cpuid);
>  	set_cpu_online(curr_cpuid, 1);
>  	probe_vendor_features(curr_cpuid);
> -- 
> 2.34.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH] risc-v: Fix order of IPI enablement vs RCU startup

[Palmer Dabbelt]

On Mon, 03 Jul 2023 19:31:26 +0100, Marc Zyngier wrote:
> Conor reports that risc-v tries to enable IPIs before telling the
> core code to enable RCU. With the introduction of the mapple tree
> as a backing store for the irq descriptors, this results in
> a very shouty boot sequence, as RCU is legitimately upset.
> 
> Restore some sanity by moving the risc_ipi_enable() call after
> notify_cpu_starting(), which explicitly enables RCU on the calling
> CPU.
> 
> [...]

Applied, thanks!

[1/1] risc-v: Fix order of IPI enablement vs RCU startup
      https://git.kernel.org/palmer/c/6259f3443c6a

Best regards,
-- 
Palmer Dabbelt <palmer@rivosinc.com>