[PATCH 3/3] MIPS: SMP: Properly stop secondary CPUs for restart

[PATCH 3/3] MIPS: SMP: Properly stop secondary CPUs for restart

[Rany Hany]

Some MT7621 devices deadlock in the platform restart callback unless
the other CPUs are stopped by clocking them off.

This fixes restart deadlocks on those MT7621, although the exact
reason why this is required is still unclear.

Signed-off-by: Rany Hany <rany_hany@riseup.net>
---
 arch/mips/kernel/smp.c | 58 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 8d2932e81..e50208c4a 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -32,6 +32,7 @@
 #include <asm/processor.h>
 #include <asm/idle.h>
 #include <asm/r4k-timer.h>
+#include <asm/r4kcache.h>
 #include <asm/mips-cps.h>
 #include <asm/mmu_context.h>
 #include <asm/time.h>
@@ -413,6 +414,8 @@ asmlinkage void start_secondary(void)
 	cpu_startup_entry(CPUHP_AP_ONLINE_IDLE);
 }
 
+static atomic_t core_stop_count[NR_CPUS];
+
 static void stop_this_cpu(void *dummy)
 {
 	/*
@@ -422,13 +425,66 @@ static void stop_this_cpu(void *dummy)
 	set_cpu_online(smp_processor_id(), false);
 	calculate_cpu_foreign_map();
 	local_irq_disable();
-	while (1);
+
+	if (mips_cm_present() && r4k_blast_dcache) {
+		unsigned int core = cpu_core(&current_cpu_data);
+
+		if (atomic_dec_and_test(&core_stop_count[core])) {
+			/* Flush data cache */
+			r4k_blast_dcache();
+			__sync();
+
+			if (mips_cm_revision() < CM_REV_CM3) {
+				/* Restrict coherence to own core first */
+				write_gcr_cl_coherence(1 << core);
+				read_gcr_cl_coherence();
+				__sync();
+			}
+
+			/* Disable coherence */
+			write_gcr_cl_coherence(0);
+			read_gcr_cl_coherence();
+
+			/* Gate the core clock */
+			if (mips_cpc_present())
+				write_cpc_cl_cmd(CPC_Cx_CMD_CLOCKOFF);
+		}
+	}
+
+	if (cpu_has_mipsmt) {
+		/* The last active VPE on the core will gate the core clock
+		 * and all other remaining VPEs will halt this TC instead.
+		 *
+		 * Note that on systems without CPC, this will be the
+		 * only way to shutdown the CPU.
+		 */
+		write_c0_tchalt(TCHALT_H);
+		instruction_hazard();
+	}
+
+	while (1)
+		cpu_relax();
 }
 
 void smp_send_stop(void)
 {
+	static unsigned long stop_in_progress;
 	unsigned long timeout;
 
+	if (test_and_set_bit(0, &stop_in_progress))
+		return;
+
+	if (mips_cm_present()) {
+		unsigned int cpu;
+
+		for_each_online_cpu(cpu) {
+			unsigned int core;
+
+			core = cpu_core(&cpu_data[cpu]);
+			atomic_inc(&core_stop_count[core]);
+		}
+	}
+
 	smp_call_function(stop_this_cpu, NULL, 0);
 
 	/* Wait up to 1s for other CPUs to stop */
-- 
2.53.0

[PATCH 0/3] Fix MT7621 restart deadlock

[Rany Hany]

These patches fix an issue on some MT7621 devices where the reboot
hangs. They were submitted to OpenWrt and have been merged.

The first two patches aren't particularly remarkable as they just
align MIPS with ARM behavior and are required for the full fix.

The last patch is the most interesting as it uses CPC to transition
the CPU to clock off state. It is not clear to me why this is required
but without it it ends up hanging. Initially only flushing d-cache
and using TCHalt was attempted but ClkOff appears to be necessary. 

This fix was only tested on ASUS RT-AX53U as I don't have other
MT7621 devices with this problem.

Unrelated note:

Sorry I had to resend this as my Thunderbird client was not properly
setup for this. I used imap-send which just caused issues. I am not
sure how I previously setup Thunderbird so that it groups the patches
in one thread.

In any case, I ended up just using send-email as I didn't want to mess
with Thunderbird again.

Rany Hany (3):
  MIPS: Stop secondary CPUs before platform restart/halt/poweroff
  MIPS: SMP: Wait for secondary CPUs to stop in smp_send_stop()
  MIPS: SMP: Properly stop secondary CPUs for restart

 arch/mips/kernel/reset.c | 19 ++++++-----
 arch/mips/kernel/smp.c   | 68 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 78 insertions(+), 9 deletions(-)

-- 
2.53.0

[PATCH 1/3] MIPS: Stop secondary CPUs before platform restart/halt/poweroff

[Rany Hany]

smp_send_stop() was placed after the platform restart, halt, and
power-off callbacks. These callbacks never return, so smp_send_stop()
was dead code and secondary CPUs were never stopped before the system
reset.

Move smp_send_stop() before the callbacks.

Signed-off-by: Rany Hany <rany_hany@riseup.net>
---
 arch/mips/kernel/reset.c | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/arch/mips/kernel/reset.c b/arch/mips/kernel/reset.c
index e7ce07b3e..12c54d1b7 100644
--- a/arch/mips/kernel/reset.c
+++ b/arch/mips/kernel/reset.c
@@ -87,13 +87,14 @@ static void machine_hang(void)
 
 void machine_restart(char *command)
 {
-	if (_machine_restart)
-		_machine_restart(command);
-
 #ifdef CONFIG_SMP
 	preempt_disable();
 	smp_send_stop();
 #endif
+
+	if (_machine_restart)
+		_machine_restart(command);
+
 	do_kernel_restart(command);
 	mdelay(1000);
 	pr_emerg("Reboot failed -- System halted\n");
@@ -102,23 +103,25 @@ void machine_restart(char *command)
 
 void machine_halt(void)
 {
-	if (_machine_halt)
-		_machine_halt();
-
 #ifdef CONFIG_SMP
 	preempt_disable();
 	smp_send_stop();
 #endif
+
+	if (_machine_halt)
+		_machine_halt();
+
 	machine_hang();
 }
 
 void machine_power_off(void)
 {
-	do_kernel_power_off();
-
 #ifdef CONFIG_SMP
 	preempt_disable();
 	smp_send_stop();
 #endif
+
+	do_kernel_power_off();
+
 	machine_hang();
 }
-- 
2.53.0

[PATCH 2/3] MIPS: SMP: Wait for secondary CPUs to stop in smp_send_stop()

[Rany Hany]

smp_send_stop() returns immediately without waiting.

Wait up to one second for secondary CPUs to mark themselves offline
before returning, similar to what ARM and ARM64 already do.

Signed-off-by: Rany Hany <rany_hany@riseup.net>
---
 arch/mips/kernel/smp.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 4868e79f3..8d2932e81 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -427,7 +427,17 @@ static void stop_this_cpu(void *dummy)
 
 void smp_send_stop(void)
 {
+	unsigned long timeout;
+
 	smp_call_function(stop_this_cpu, NULL, 0);
+
+	/* Wait up to 1s for other CPUs to stop */
+	timeout = USEC_PER_SEC;
+	while (num_online_cpus() > 1 && timeout--)
+		udelay(1);
+
+	if (num_online_cpus() > 1)
+		pr_warn("SMP: failed to stop secondary CPUs\n");
 }
 
 void __init smp_cpus_done(unsigned int max_cpus)
-- 
2.53.0

[PATCH 3/3] MIPS: SMP: Properly stop secondary CPUs for restart

[Rany Hany]

Some MT7621 devices deadlock in the platform restart callback unless
the other CPUs are stopped by clocking them off. Initially only
flushing d-cache and using TCHalt was attempted but ClkOff appears
to be necessary to fix this issue.

This fixes restart deadlocks on those MT7621, although the exact
reason why this is required is still unclear.

Signed-off-by: Rany Hany <rany_hany@riseup.net>
---
 arch/mips/kernel/smp.c | 58 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/smp.c b/arch/mips/kernel/smp.c
index 8d2932e81..e50208c4a 100644
--- a/arch/mips/kernel/smp.c
+++ b/arch/mips/kernel/smp.c
@@ -32,6 +32,7 @@
 #include <asm/processor.h>
 #include <asm/idle.h>
 #include <asm/r4k-timer.h>
+#include <asm/r4kcache.h>
 #include <asm/mips-cps.h>
 #include <asm/mmu_context.h>
 #include <asm/time.h>
@@ -413,6 +414,8 @@ asmlinkage void start_secondary(void)
 	cpu_startup_entry(CPUHP_AP_ONLINE_IDLE);
 }
 
+static atomic_t core_stop_count[NR_CPUS];
+
 static void stop_this_cpu(void *dummy)
 {
 	/*
@@ -422,13 +425,66 @@ static void stop_this_cpu(void *dummy)
 	set_cpu_online(smp_processor_id(), false);
 	calculate_cpu_foreign_map();
 	local_irq_disable();
-	while (1);
+
+	if (mips_cm_present() && r4k_blast_dcache) {
+		unsigned int core = cpu_core(&current_cpu_data);
+
+		if (atomic_dec_and_test(&core_stop_count[core])) {
+			/* Flush data cache */
+			r4k_blast_dcache();
+			__sync();
+
+			if (mips_cm_revision() < CM_REV_CM3) {
+				/* Restrict coherence to own core first */
+				write_gcr_cl_coherence(1 << core);
+				read_gcr_cl_coherence();
+				__sync();
+			}
+
+			/* Disable coherence */
+			write_gcr_cl_coherence(0);
+			read_gcr_cl_coherence();
+
+			/* Gate the core clock */
+			if (mips_cpc_present())
+				write_cpc_cl_cmd(CPC_Cx_CMD_CLOCKOFF);
+		}
+	}
+
+	if (cpu_has_mipsmt) {
+		/* The last active VPE on the core will gate the core clock
+		 * and all other remaining VPEs will halt this TC instead.
+		 *
+		 * Note that on systems without CPC, this will be the
+		 * only way to shutdown the CPU.
+		 */
+		write_c0_tchalt(TCHALT_H);
+		instruction_hazard();
+	}
+
+	while (1)
+		cpu_relax();
 }
 
 void smp_send_stop(void)
 {
+	static unsigned long stop_in_progress;
 	unsigned long timeout;
 
+	if (test_and_set_bit(0, &stop_in_progress))
+		return;
+
+	if (mips_cm_present()) {
+		unsigned int cpu;
+
+		for_each_online_cpu(cpu) {
+			unsigned int core;
+
+			core = cpu_core(&cpu_data[cpu]);
+			atomic_inc(&core_stop_count[core]);
+		}
+	}
+
 	smp_call_function(stop_this_cpu, NULL, 0);
 
 	/* Wait up to 1s for other CPUs to stop */
-- 
2.53.0

Re: [PATCH 3/3] MIPS: SMP: Properly stop secondary CPUs for restart

[Thomas Bogendoerfer]

On Mon, Apr 06, 2026 at 02:59:55AM +0300, Rany Hany wrote:
> @@ -422,13 +425,66 @@ static void stop_this_cpu(void *dummy)
>  	set_cpu_online(smp_processor_id(), false);
>  	calculate_cpu_foreign_map();
>  	local_irq_disable();
> -	while (1);
> +
> +	if (mips_cm_present() && r4k_blast_dcache) {
> +		unsigned int core = cpu_core(&current_cpu_data);
> +
> +		if (atomic_dec_and_test(&core_stop_count[core])) {
> +			/* Flush data cache */
> +			r4k_blast_dcache();
> +			__sync();
> +
> +			if (mips_cm_revision() < CM_REV_CM3) {
> +				/* Restrict coherence to own core first */
> +				write_gcr_cl_coherence(1 << core);
> +				read_gcr_cl_coherence();
> +				__sync();
> +			}
> +
> +			/* Disable coherence */
> +			write_gcr_cl_coherence(0);
> +			read_gcr_cl_coherence();
> +
> +			/* Gate the core clock */
> +			if (mips_cpc_present())
> +				write_cpc_cl_cmd(CPC_Cx_CMD_CLOCKOFF);
> +		}
> +	}
> +
> +	if (cpu_has_mipsmt) {
> +		/* The last active VPE on the core will gate the core clock
> +		 * and all other remaining VPEs will halt this TC instead.
> +		 *
> +		 * Note that on systems without CPC, this will be the
> +		 * only way to shutdown the CPU.
> +		 */
> +		write_c0_tchalt(TCHALT_H);
> +		instruction_hazard();
> +	}
> +
> +	while (1)
> +		cpu_relax();
>  }

is this whole dance safe for _all_ SOCs with CM and MIPSMT ? if not,
this should be moved to a not so generic place...

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessarily a
good idea.                                                [ RFC1925, 2.3 ]