[PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Huacai Chen]

Kexec-tools use mem=X@Y to pass usable memories to crash kernel, but in
commit a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map") all
BIOS passed memories are removed by early_parse_mem(). I think this is
reasonable for a normal kernel but not for a crash kernel, because a
crash kernel should be able to see all old memories, even though it is
not supposed to use them.

Fixes: a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map")
Signed-off-by: Huacai Chen <chenhc@lemote.com>
---
 arch/mips/kernel/setup.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index 4c04a86..e2804a2 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -392,8 +392,10 @@ static int __init early_parse_mem(char *p)
 	 */
 	if (usermem == 0) {
 		usermem = 1;
+#ifndef CONFIG_CRASH_DUMP
 		memblock_remove(memblock_start_of_DRAM(),
 			memblock_end_of_DRAM() - memblock_start_of_DRAM());
+#endif
 	}
 	start = 0;
 	size = memparse(p, &p);
-- 
2.7.0

[PATCH 2/3] MIPS: Reserve extra memory for crash dump

[Huacai Chen]

Traditionally, MIPS's contiguous low memory can be as less as 256M, so
crashkernel=X@Y may be unable to large enough in some cases. Moreover,
for the "multi numa node + sparse memory model" case, it is attempt to
allocate section_mem_maps on every node. Thus, if the total memory of a
node is more than 1GB, we reserve the top 128MB for the crash kernel.

Signed-off-by: Huacai Chen <chenhc@lemote.com>
---
 arch/mips/kernel/setup.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index e2804a2..90d4a2e 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -29,6 +29,7 @@
 #include <linux/of_fdt.h>
 #include <linux/of_reserved_mem.h>
 #include <linux/dmi.h>
+#include <linux/crash_dump.h>
 
 #include <asm/addrspace.h>
 #include <asm/bootinfo.h>
@@ -55,6 +56,10 @@ EXPORT_SYMBOL(cpu_data);
 struct screen_info screen_info;
 #endif
 
+#ifdef CONFIG_CRASH_DUMP
+static phys_addr_t crashmem_start, crashmem_size;
+#endif
+
 /*
  * Setup information
  *
@@ -404,6 +409,13 @@ static int __init early_parse_mem(char *p)
 
 	add_memory_region(start, size, BOOT_MEM_RAM);
 
+#ifdef CONFIG_CRASH_DUMP
+	if (start && size) {
+		crashmem_start = start;
+		crashmem_size = size;
+	}
+#endif
+
 	return 0;
 }
 early_param("mem", early_parse_mem);
@@ -642,6 +654,48 @@ static void __init bootcmdline_init(void)
 		bootcmdline_append(builtin_cmdline, COMMAND_LINE_SIZE);
 }
 
+/* Traditionally, MIPS's contiguous low memory is 256M, so crashkernel=X@Y is
+ * unable to be large enough in some cases. Thus, if the total memory of a node
+ * is more than 1GB, we reserve the top 128MB for the crash kernel */
+static void reserve_crashm_region(int node, unsigned long s0, unsigned long e0)
+{
+#ifdef CONFIG_KEXEC
+	if (crashk_res.start == crashk_res.end)
+		return;
+
+	if ((e0 - s0) <= (SZ_1G >> PAGE_SHIFT))
+		return;
+
+	s0 = e0 - (SZ_128M >> PAGE_SHIFT);
+
+	memblock_reserve(PFN_PHYS(s0), (e0 - s0) << PAGE_SHIFT);
+#endif
+}
+
+static void reserve_oldmem_region(int node, unsigned long s0, unsigned long e0)
+{
+#ifdef CONFIG_CRASH_DUMP
+	unsigned long s1, e1;
+
+	if (!is_kdump_kernel())
+		return;
+
+	if ((e0 - s0) > (SZ_1G >> PAGE_SHIFT))
+		e0 = e0 - (SZ_128M >> PAGE_SHIFT);
+
+	/* crashmem_start is crashk_res reserved by primary kernel */
+	s1 = PFN_UP(crashmem_start);
+	e1 = PFN_DOWN(crashmem_start + crashmem_size);
+
+	if (node == 0) {
+		memblock_reserve(PFN_PHYS(s0), (s1 - s0) << PAGE_SHIFT);
+		memblock_reserve(PFN_PHYS(e1), (e0 - e1) << PAGE_SHIFT);
+	} else {
+		memblock_reserve(PFN_PHYS(s0), (e0 - s0) << PAGE_SHIFT);
+	}
+#endif
+}
+
 /*
  * arch_mem_init - initialize memory management subsystem
  *
@@ -666,6 +720,9 @@ static void __init bootcmdline_init(void)
  */
 static void __init arch_mem_init(char **cmdline_p)
 {
+	unsigned int node;
+	unsigned long start_pfn, end_pfn;
+
 	/* call board setup routine */
 	plat_mem_setup();
 	memblock_set_bottom_up(true);
@@ -711,6 +768,12 @@ static void __init arch_mem_init(char **cmdline_p)
 	if (crashk_res.start != crashk_res.end)
 		memblock_reserve(crashk_res.start, resource_size(&crashk_res));
 #endif
+	for_each_online_node(node) {
+		get_pfn_range_for_nid(node, &start_pfn, &end_pfn);
+		reserve_crashm_region(node, start_pfn, end_pfn);
+		reserve_oldmem_region(node, start_pfn, end_pfn);
+	}
+
 	device_tree_init();
 
 	/*
-- 
2.7.0

[PATCH 3/3] MIPS: Loongson64: Add kexec/kdump support

[Huacai Chen]

Add kexec/kdump support for Loongson64 by:
1, Provide Loongson-specific kexec functions: loongson_kexec_prepare,
   loongson_kexec_shutdown and loongson_crash_shutdown;
2, Provide Loongson-specific assembly code in kexec_smp_wait;
3, Clear mailbox in loongson3_smp_setup() since KEXEC bypass BIOS;
4, KEXEC always run at boot CPU, but KDUMP may triggered at non-boot
   CPU. Loongson64 assume boot CPU is the first possible cpu, so fix
   boot_cpu_id in prom_init_env();

To start Loongson64, The boot CPU needs 3 parameters:
fw_arg0: the number of arguments in cmdline (i.e., argc).
fw_arg1: structure holds cmdline such as "root=/dev/sda1 console=tty"
         (i.e., argv).
fw_arg2: environment (i.e., envp, additional boot parameters from LEFI).

Non-boot CPUs do not need parameters at once. They query their own IPI
mailbox to get PC, SP and GP in a loop until boot CPU brings them up.

loongson_kexec_prepare(): Setup cmdline for kexec/kdump. The kexec/kdump
cmdline comes from kexec's "append" option string. This structure will
be parsed in fw_init_cmdline() of arch/mips/fw/lib/cmdline.c. Both image
->control_code_page and the cmdline need to be in a safe memory region
(memory allocated by the old kernel may be corrupted by the new kernel).
In order to maintain compatibility for the old firmware, the low 2MB is
reserverd and safe for Loongson. So let KEXEC_CTRL_CODE and KEXEC_ARGV_
ADDR be here. LEFI parameters may be corrupted at runtime, so backup it
at mips_reboot_setup(), and then restore it at loongson_kexec_shutdown()
/loongson_crash_shutdown().

loongson_kexec_shutdown(): Wake up all present CPUs and let them go to
reboot_code_buffer. Pass the kexec parameters to kexec_args.

loongson_crash_shutdown(): Pass the kdump parameters to kexec_args.

The assembly part in kexec_smp_wait provide a routine as BIOS does, in
order to keep secondary CPUs in a querying loop.

Cc: Eric Biederman <ebiederm@xmission.com>
Signed-off-by: Huacai Chen <chenhc@lemote.com>
Signed-off-by: Jinyang He <hejinyang@loongson.cn>
Signed-off-by: Youling Tang <tangyouling@loongson.cn>
---
 arch/mips/kernel/relocate_kernel.S |  27 +++++++++
 arch/mips/loongson64/env.c         |   7 +++
 arch/mips/loongson64/reset.c       | 111 +++++++++++++++++++++++++++++++++++++
 arch/mips/loongson64/smp.c         |   5 ++
 4 files changed, 150 insertions(+)

diff --git a/arch/mips/kernel/relocate_kernel.S b/arch/mips/kernel/relocate_kernel.S
index ac87089..91b2932 100644
--- a/arch/mips/kernel/relocate_kernel.S
+++ b/arch/mips/kernel/relocate_kernel.S
@@ -133,6 +133,33 @@ LEAF(kexec_smp_wait)
 #else
 	sync
 #endif
+
+#ifdef CONFIG_CPU_LOONGSON64
+#define MAILBOX_BASE 0x900000003ff01000
+	/* s0:prid s1:initfn */
+	/* t0:base t1:cpuid t2:node t9:count */
+	mfc0  t1, CP0_EBASE
+	andi  t1, MIPS_EBASE_CPUNUM
+	dli   t0, MAILBOX_BASE    /* mailbox base */
+	dins  t0, t1, 8, 2        /* insert core id*/
+	dext  t2, t1, 2, 2
+	dins  t0, t2, 44, 2       /* insert node id */
+	mfc0  s0, CP0_PRID
+	andi  s0, s0, 0xf
+	blt   s0, 0x6, 1f         /* Loongson-3A1000 */
+	bgt   s0, 0x7, 1f         /* Loongson-3A2000/3A3000 */
+	dins  t0, t2, 14, 2       /* Loongson-3B1000/3B1500 need bit 15~14 */
+1:	li    t9, 0x100           /* wait for init loop */
+2:	addiu t9, -1              /* limit mailbox access */
+	bnez  t9, 2b
+	ld    s1, 0x20(t0)        /* get PC via mailbox reg0 */
+	beqz  s1, 1b
+	ld    sp, 0x28(t0)        /* get SP via mailbox reg1 */
+	ld    gp, 0x30(t0)        /* get GP via mailbox reg2 */
+	ld    a1, 0x38(t0)
+	jr    s1                  /* jump to initial PC */
+#endif
+
 	j		s1
 	END(kexec_smp_wait)
 #endif
diff --git a/arch/mips/loongson64/env.c b/arch/mips/loongson64/env.c
index 134cb8e..e937f31 100644
--- a/arch/mips/loongson64/env.c
+++ b/arch/mips/loongson64/env.c
@@ -120,6 +120,13 @@ void __init prom_init_env(void)
 	loongson_sysconf.nr_cpus = ecpu->nr_cpus;
 	loongson_sysconf.boot_cpu_id = ecpu->cpu_startup_core_id;
 	loongson_sysconf.reserved_cpus_mask = ecpu->reserved_cores_mask;
+#ifdef CONFIG_KEXEC
+	loongson_sysconf.boot_cpu_id = get_ebase_cpunum();
+	loongson_sysconf.reserved_cpus_mask |=
+		(1 << loongson_sysconf.boot_cpu_id) - 1;
+	pr_info("Boot CPU ID is being fixed from %d to %d\n",
+		ecpu->cpu_startup_core_id, loongson_sysconf.boot_cpu_id);
+#endif
 	if (ecpu->nr_cpus > NR_CPUS || ecpu->nr_cpus == 0)
 		loongson_sysconf.nr_cpus = NR_CPUS;
 	loongson_sysconf.nr_nodes = (loongson_sysconf.nr_cpus +
diff --git a/arch/mips/loongson64/reset.c b/arch/mips/loongson64/reset.c
index 3bb8a1e..b1e71f37 100644
--- a/arch/mips/loongson64/reset.c
+++ b/arch/mips/loongson64/reset.c
@@ -6,9 +6,14 @@
  * Copyright (C) 2009 Lemote, Inc.
  * Author: Zhangjin Wu, wuzhangjin@gmail.com
  */
+#include <linux/cpu.h>
+#include <linux/delay.h>
 #include <linux/init.h>
+#include <linux/kexec.h>
 #include <linux/pm.h>
+#include <linux/slab.h>
 
+#include <asm/bootinfo.h>
 #include <asm/idle.h>
 #include <asm/reboot.h>
 
@@ -47,12 +52,118 @@ static void loongson_halt(void)
 	}
 }
 
+#ifdef CONFIG_KEXEC
+
+/* 0X80000000~0X80200000 is safe */
+#define MAX_ARGS	64
+#define KEXEC_CTRL_CODE	0xFFFFFFFF80100000UL
+#define KEXEC_ARGV_ADDR	0xFFFFFFFF80108000UL
+#define KEXEC_ARGV_SIZE	COMMAND_LINE_SIZE
+#define KEXEC_ENVP_SIZE	4800
+
+static int kexec_argc;
+static int kdump_argc;
+static void *kexec_argv;
+static void *kdump_argv;
+static void *kexec_envp;
+
+static int loongson_kexec_prepare(struct kimage *image)
+{
+	int i, argc = 0;
+	unsigned int *argv;
+	char *str, *ptr, *bootloader = "kexec";
+
+	/* argv at offset 0, argv[] at offset KEXEC_ARGV_SIZE/2 */
+	if (image->type == KEXEC_TYPE_DEFAULT)
+		argv = (unsigned int *)kexec_argv;
+	else
+		argv = (unsigned int *)kdump_argv;
+
+	argv[argc++] = (unsigned int)(KEXEC_ARGV_ADDR + KEXEC_ARGV_SIZE/2);
+
+	for (i = 0; i < image->nr_segments; i++) {
+		if (!strncmp(bootloader, (char *)image->segment[i].buf,
+				strlen(bootloader))) {
+			/*
+			 * convert command line string to array
+			 * of parameters (as bootloader does).
+			 */
+			int offt;
+			str = (char *)argv + KEXEC_ARGV_SIZE/2;
+			memcpy(str, image->segment[i].buf, KEXEC_ARGV_SIZE/2);
+			ptr = strchr(str, ' ');
+
+			while (ptr && (argc < MAX_ARGS)) {
+				*ptr = '\0';
+				if (ptr[1] != ' ') {
+					offt = (int)(ptr - str + 1);
+					argv[argc] = KEXEC_ARGV_ADDR + KEXEC_ARGV_SIZE/2 + offt;
+					argc++;
+				}
+				ptr = strchr(ptr + 1, ' ');
+			}
+			break;
+		}
+	}
+
+	if (image->type == KEXEC_TYPE_DEFAULT)
+		kexec_argc = argc;
+	else
+		kdump_argc = argc;
+
+	/* kexec/kdump need a safe page to save reboot_code_buffer */
+	image->control_code_page = virt_to_page((void *)KEXEC_CTRL_CODE);
+
+	return 0;
+}
+
+static void loongson_kexec_shutdown(void)
+{
+#ifdef CONFIG_SMP
+	int cpu;
+
+	/* All CPUs go to reboot_code_buffer */
+	for_each_possible_cpu(cpu)
+		if (!cpu_online(cpu))
+			cpu_device_up(get_cpu_device(cpu));
+#endif
+	kexec_args[0] = kexec_argc;
+	kexec_args[1] = fw_arg1;
+	kexec_args[2] = fw_arg2;
+	memcpy((void *)fw_arg1, kexec_argv, KEXEC_ARGV_SIZE);
+	memcpy((void *)fw_arg2, kexec_envp, KEXEC_ENVP_SIZE);
+}
+
+static void loongson_crash_shutdown(struct pt_regs *regs)
+{
+	default_machine_crash_shutdown(regs);
+	kexec_args[0] = kdump_argc;
+	kexec_args[1] = fw_arg1;
+	kexec_args[2] = fw_arg2;
+	memcpy((void *)fw_arg1, kdump_argv, KEXEC_ARGV_SIZE);
+	memcpy((void *)fw_arg2, kexec_envp, KEXEC_ENVP_SIZE);
+}
+
+#endif
+
 static int __init mips_reboot_setup(void)
 {
 	_machine_restart = loongson_restart;
 	_machine_halt = loongson_halt;
 	pm_power_off = loongson_poweroff;
 
+#ifdef CONFIG_KEXEC
+	kexec_argv = kmalloc(KEXEC_ARGV_SIZE, GFP_KERNEL);
+	kdump_argv = kmalloc(KEXEC_ARGV_SIZE, GFP_KERNEL);
+	kexec_envp = kmalloc(KEXEC_ENVP_SIZE, GFP_KERNEL);
+	fw_arg1 = KEXEC_ARGV_ADDR;
+	memcpy(kexec_envp, (void *)fw_arg2, KEXEC_ENVP_SIZE);
+
+	_machine_kexec_prepare = loongson_kexec_prepare;
+	_machine_kexec_shutdown = loongson_kexec_shutdown;
+	_machine_crash_shutdown = loongson_crash_shutdown;
+#endif
+
 	return 0;
 }
 
diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
index e744e1b..55697ac 100644
--- a/arch/mips/loongson64/smp.c
+++ b/arch/mips/loongson64/smp.c
@@ -420,6 +420,11 @@ static void __init loongson3_smp_setup(void)
 	ipi_status0_regs_init();
 	ipi_en0_regs_init();
 	ipi_mailbox_buf_init();
+
+	/* BIOS clear the mailbox, but KEXEC bypass BIOS so clear here */
+	for (i = 0; i < loongson_sysconf.nr_cpus; i++)
+		loongson3_ipi_write64(0, (void *)(ipi_mailbox_buf[i]+0x0));
+
 	cpu_set_core(&cpu_data[0],
 		     cpu_logical_map(0) % loongson_sysconf.cores_per_package);
 	cpu_data[0].package = cpu_logical_map(0) / loongson_sysconf.cores_per_package;
-- 
2.7.0

Re: [PATCH 3/3] MIPS: Loongson64: Add kexec/kdump support

[Jinyang He]

Hi, Huacai,

On 09/23/2020 10:30 AM, Huacai Chen wrote:
> Add kexec/kdump support for Loongson64 by:
> 1, Provide Loongson-specific kexec functions: loongson_kexec_prepare,
>     loongson_kexec_shutdown and loongson_crash_shutdown;
> 2, Provide Loongson-specific assembly code in kexec_smp_wait;
> 3, Clear mailbox in loongson3_smp_setup() since KEXEC bypass BIOS;
> 4, KEXEC always run at boot CPU, but KDUMP may triggered at non-boot
>     CPU. Loongson64 assume boot CPU is the first possible cpu, so fix
>     boot_cpu_id in prom_init_env();
>
> To start Loongson64, The boot CPU needs 3 parameters:
> fw_arg0: the number of arguments in cmdline (i.e., argc).
> fw_arg1: structure holds cmdline such as "root=/dev/sda1 console=tty"
>           (i.e., argv).
> fw_arg2: environment (i.e., envp, additional boot parameters from LEFI).
>
> Non-boot CPUs do not need parameters at once. They query their own IPI
> mailbox to get PC, SP and GP in a loop until boot CPU brings them up.
>
> loongson_kexec_prepare(): Setup cmdline for kexec/kdump. The kexec/kdump
> cmdline comes from kexec's "append" option string. This structure will
> be parsed in fw_init_cmdline() of arch/mips/fw/lib/cmdline.c. Both image
> ->control_code_page and the cmdline need to be in a safe memory region
> (memory allocated by the old kernel may be corrupted by the new kernel).
> In order to maintain compatibility for the old firmware, the low 2MB is
> reserverd and safe for Loongson. So let KEXEC_CTRL_CODE and KEXEC_ARGV_
> ADDR be here. LEFI parameters may be corrupted at runtime, so backup it
> at mips_reboot_setup(), and then restore it at loongson_kexec_shutdown()
> /loongson_crash_shutdown().
>
> loongson_kexec_shutdown(): Wake up all present CPUs and let them go to
> reboot_code_buffer. Pass the kexec parameters to kexec_args.
>
> loongson_crash_shutdown(): Pass the kdump parameters to kexec_args.
>
> The assembly part in kexec_smp_wait provide a routine as BIOS does, in
> order to keep secondary CPUs in a querying loop.
>
> Cc: Eric Biederman <ebiederm@xmission.com>
> Signed-off-by: Huacai Chen <chenhc@lemote.com>
> Signed-off-by: Jinyang He <hejinyang@loongson.cn>
> Signed-off-by: Youling Tang <tangyouling@loongson.cn>
> ---
>   arch/mips/kernel/relocate_kernel.S |  27 +++++++++
>   arch/mips/loongson64/env.c         |   7 +++
>   arch/mips/loongson64/reset.c       | 111 +++++++++++++++++++++++++++++++++++++
>   arch/mips/loongson64/smp.c         |   5 ++
>   4 files changed, 150 insertions(+)
>
> diff --git a/arch/mips/kernel/relocate_kernel.S b/arch/mips/kernel/relocate_kernel.S
> index ac87089..91b2932 100644
> --- a/arch/mips/kernel/relocate_kernel.S
> +++ b/arch/mips/kernel/relocate_kernel.S
> @@ -133,6 +133,33 @@ LEAF(kexec_smp_wait)
>   #else
>   	sync
>   #endif
> +
> +#ifdef CONFIG_CPU_LOONGSON64
> +#define MAILBOX_BASE 0x900000003ff01000
> +	/* s0:prid s1:initfn */
> +	/* t0:base t1:cpuid t2:node t9:count */
> +	mfc0  t1, CP0_EBASE
> +	andi  t1, MIPS_EBASE_CPUNUM
> +	dli   t0, MAILBOX_BASE    /* mailbox base */
> +	dins  t0, t1, 8, 2        /* insert core id*/
> +	dext  t2, t1, 2, 2
> +	dins  t0, t2, 44, 2       /* insert node id */
> +	mfc0  s0, CP0_PRID
> +	andi  s0, s0, 0xf
> +	blt   s0, 0x6, 1f         /* Loongson-3A1000 */
> +	bgt   s0, 0x7, 1f         /* Loongson-3A2000/3A3000 */
> +	dins  t0, t2, 14, 2       /* Loongson-3B1000/3B1500 need bit 15~14 */

Here is right, and I have a trivial idea that machine type
can be passed by secondary_kexec_args.

> +1:	li    t9, 0x100           /* wait for init loop */
> +2:	addiu t9, -1              /* limit mailbox access */
> +	bnez  t9, 2b
> +	ld    s1, 0x20(t0)        /* get PC via mailbox reg0 */
> +	beqz  s1, 1b
> +	ld    sp, 0x28(t0)        /* get SP via mailbox reg1 */
> +	ld    gp, 0x30(t0)        /* get GP via mailbox reg2 */
> +	ld    a1, 0x38(t0)
> +	jr    s1                  /* jump to initial PC */
> +#endif
> +
>   	j		s1
>   	END(kexec_smp_wait)
>   #endif
> diff --git a/arch/mips/loongson64/env.c b/arch/mips/loongson64/env.c
> index 134cb8e..e937f31 100644
> --- a/arch/mips/loongson64/env.c
> +++ b/arch/mips/loongson64/env.c
> @@ -120,6 +120,13 @@ void __init prom_init_env(void)
>   	loongson_sysconf.nr_cpus = ecpu->nr_cpus;
>   	loongson_sysconf.boot_cpu_id = ecpu->cpu_startup_core_id;
>   	loongson_sysconf.reserved_cpus_mask = ecpu->reserved_cores_mask;
> +#ifdef CONFIG_KEXEC
> +	loongson_sysconf.boot_cpu_id = get_ebase_cpunum();
> +	loongson_sysconf.reserved_cpus_mask |=
> +		(1 << loongson_sysconf.boot_cpu_id) - 1;
> +	pr_info("Boot CPU ID is being fixed from %d to %d\n",
> +		ecpu->cpu_startup_core_id, loongson_sysconf.boot_cpu_id);
> +#endif

I don't think secondary CPU will go here.
At arch/mips/kernel/machine_kexec.c: kexec_reboot(), CPU0 goes
relocate_new_kernel() and secondary CPUs go to smp_wait. Why
kdump may triggered at non-boot CPU?

>   	if (ecpu->nr_cpus > NR_CPUS || ecpu->nr_cpus == 0)
>   		loongson_sysconf.nr_cpus = NR_CPUS;
>   	loongson_sysconf.nr_nodes = (loongson_sysconf.nr_cpus +
> diff --git a/arch/mips/loongson64/reset.c b/arch/mips/loongson64/reset.c
> index 3bb8a1e..b1e71f37 100644
> --- a/arch/mips/loongson64/reset.c
> +++ b/arch/mips/loongson64/reset.c
> @@ -6,9 +6,14 @@
>    * Copyright (C) 2009 Lemote, Inc.
>    * Author: Zhangjin Wu, wuzhangjin@gmail.com
>    */
> +#include <linux/cpu.h>
> +#include <linux/delay.h>
>   #include <linux/init.h>
> +#include <linux/kexec.h>
>   #include <linux/pm.h>
> +#include <linux/slab.h>
>   
> +#include <asm/bootinfo.h>
>   #include <asm/idle.h>
>   #include <asm/reboot.h>
>   
> @@ -47,12 +52,118 @@ static void loongson_halt(void)
>   	}
>   }
>   
> +#ifdef CONFIG_KEXEC
> +
> +/* 0X80000000~0X80200000 is safe */
> +#define MAX_ARGS	64
> +#define KEXEC_CTRL_CODE	0xFFFFFFFF80100000UL
> +#define KEXEC_ARGV_ADDR	0xFFFFFFFF80108000UL
> +#define KEXEC_ARGV_SIZE	COMMAND_LINE_SIZE
> +#define KEXEC_ENVP_SIZE	4800
> +
> +static int kexec_argc;
> +static int kdump_argc;
> +static void *kexec_argv;
> +static void *kdump_argv;
> +static void *kexec_envp;
> +
> +static int loongson_kexec_prepare(struct kimage *image)
> +{
> +	int i, argc = 0;
> +	unsigned int *argv;
> +	char *str, *ptr, *bootloader = "kexec";
> +
> +	/* argv at offset 0, argv[] at offset KEXEC_ARGV_SIZE/2 */
> +	if (image->type == KEXEC_TYPE_DEFAULT)
> +		argv = (unsigned int *)kexec_argv;
> +	else
> +		argv = (unsigned int *)kdump_argv;
> +
> +	argv[argc++] = (unsigned int)(KEXEC_ARGV_ADDR + KEXEC_ARGV_SIZE/2);
> +
> +	for (i = 0; i < image->nr_segments; i++) {
> +		if (!strncmp(bootloader, (char *)image->segment[i].buf,
> +				strlen(bootloader))) {
> +			/*
> +			 * convert command line string to array
> +			 * of parameters (as bootloader does).
> +			 */
> +			int offt;
> +			str = (char *)argv + KEXEC_ARGV_SIZE/2;
> +			memcpy(str, image->segment[i].buf, KEXEC_ARGV_SIZE/2);
> +			ptr = strchr(str, ' ');
> +
> +			while (ptr && (argc < MAX_ARGS)) {
> +				*ptr = '\0';
> +				if (ptr[1] != ' ') {
> +					offt = (int)(ptr - str + 1);
> +					argv[argc] = KEXEC_ARGV_ADDR + KEXEC_ARGV_SIZE/2 + offt;
> +					argc++;
> +				}
> +				ptr = strchr(ptr + 1, ' ');
> +			}
> +			break;
> +		}
> +	}
> +
> +	if (image->type == KEXEC_TYPE_DEFAULT)
> +		kexec_argc = argc;
> +	else
> +		kdump_argc = argc;
> +
> +	/* kexec/kdump need a safe page to save reboot_code_buffer */
> +	image->control_code_page = virt_to_page((void *)KEXEC_CTRL_CODE);

I thought find a hole in Crash Kernel may better than a static position.
It just a trivial idea, too.

> +
> +	return 0;
> +}
> +
> +static void loongson_kexec_shutdown(void)
> +{
> +#ifdef CONFIG_SMP
> +	int cpu;
> +
> +	/* All CPUs go to reboot_code_buffer */
> +	for_each_possible_cpu(cpu)
> +		if (!cpu_online(cpu))
> +			cpu_device_up(get_cpu_device(cpu));
> +#endif
> +	kexec_args[0] = kexec_argc;
> +	kexec_args[1] = fw_arg1;
> +	kexec_args[2] = fw_arg2;
> +	memcpy((void *)fw_arg1, kexec_argv, KEXEC_ARGV_SIZE);
> +	memcpy((void *)fw_arg2, kexec_envp, KEXEC_ENVP_SIZE);
> +}
> +
> +static void loongson_crash_shutdown(struct pt_regs *regs)
> +{
> +	default_machine_crash_shutdown(regs);
> +	kexec_args[0] = kdump_argc;
> +	kexec_args[1] = fw_arg1;
> +	kexec_args[2] = fw_arg2;
> +	memcpy((void *)fw_arg1, kdump_argv, KEXEC_ARGV_SIZE);
> +	memcpy((void *)fw_arg2, kexec_envp, KEXEC_ENVP_SIZE);
> +}
> +
> +#endif
> +
>   static int __init mips_reboot_setup(void)
>   {
>   	_machine_restart = loongson_restart;
>   	_machine_halt = loongson_halt;
>   	pm_power_off = loongson_poweroff;
>   
> +#ifdef CONFIG_KEXEC
> +	kexec_argv = kmalloc(KEXEC_ARGV_SIZE, GFP_KERNEL);
> +	kdump_argv = kmalloc(KEXEC_ARGV_SIZE, GFP_KERNEL);
> +	kexec_envp = kmalloc(KEXEC_ENVP_SIZE, GFP_KERNEL);
> +	fw_arg1 = KEXEC_ARGV_ADDR;
> +	memcpy(kexec_envp, (void *)fw_arg2, KEXEC_ENVP_SIZE);
> +
> +	_machine_kexec_prepare = loongson_kexec_prepare;
> +	_machine_kexec_shutdown = loongson_kexec_shutdown;
> +	_machine_crash_shutdown = loongson_crash_shutdown;
> +#endif
> +
>   	return 0;
>   }
>   
> diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
> index e744e1b..55697ac 100644
> --- a/arch/mips/loongson64/smp.c
> +++ b/arch/mips/loongson64/smp.c
> @@ -420,6 +420,11 @@ static void __init loongson3_smp_setup(void)
>   	ipi_status0_regs_init();
>   	ipi_en0_regs_init();
>   	ipi_mailbox_buf_init();
> +
> +	/* BIOS clear the mailbox, but KEXEC bypass BIOS so clear here */
> +	for (i = 0; i < loongson_sysconf.nr_cpus; i++)
> +		loongson3_ipi_write64(0, (void *)(ipi_mailbox_buf[i]+0x0));
> +

Does it need? It take a long time after secondary CPUs geting PC in a loop.
Or you can clear it early.

Thanks,
- Jinyang.

>   	cpu_set_core(&cpu_data[0],
>   		     cpu_logical_map(0) % loongson_sysconf.cores_per_package);
>   	cpu_data[0].package = cpu_logical_map(0) / loongson_sysconf.cores_per_package;

Re: [PATCH 3/3] MIPS: Loongson64: Add kexec/kdump support

[Huacai Chen]

Hi, Jinyang,

On Wed, Sep 23, 2020 at 11:35 AM Jinyang He <hejinyang@loongson.cn> wrote:
>
> Hi, Huacai,
>
> On 09/23/2020 10:30 AM, Huacai Chen wrote:
> > Add kexec/kdump support for Loongson64 by:
> > 1, Provide Loongson-specific kexec functions: loongson_kexec_prepare,
> >     loongson_kexec_shutdown and loongson_crash_shutdown;
> > 2, Provide Loongson-specific assembly code in kexec_smp_wait;
> > 3, Clear mailbox in loongson3_smp_setup() since KEXEC bypass BIOS;
> > 4, KEXEC always run at boot CPU, but KDUMP may triggered at non-boot
> >     CPU. Loongson64 assume boot CPU is the first possible cpu, so fix
> >     boot_cpu_id in prom_init_env();
> >
> > To start Loongson64, The boot CPU needs 3 parameters:
> > fw_arg0: the number of arguments in cmdline (i.e., argc).
> > fw_arg1: structure holds cmdline such as "root=/dev/sda1 console=tty"
> >           (i.e., argv).
> > fw_arg2: environment (i.e., envp, additional boot parameters from LEFI).
> >
> > Non-boot CPUs do not need parameters at once. They query their own IPI
> > mailbox to get PC, SP and GP in a loop until boot CPU brings them up.
> >
> > loongson_kexec_prepare(): Setup cmdline for kexec/kdump. The kexec/kdump
> > cmdline comes from kexec's "append" option string. This structure will
> > be parsed in fw_init_cmdline() of arch/mips/fw/lib/cmdline.c. Both image
> > ->control_code_page and the cmdline need to be in a safe memory region
> > (memory allocated by the old kernel may be corrupted by the new kernel).
> > In order to maintain compatibility for the old firmware, the low 2MB is
> > reserverd and safe for Loongson. So let KEXEC_CTRL_CODE and KEXEC_ARGV_
> > ADDR be here. LEFI parameters may be corrupted at runtime, so backup it
> > at mips_reboot_setup(), and then restore it at loongson_kexec_shutdown()
> > /loongson_crash_shutdown().
> >
> > loongson_kexec_shutdown(): Wake up all present CPUs and let them go to
> > reboot_code_buffer. Pass the kexec parameters to kexec_args.
> >
> > loongson_crash_shutdown(): Pass the kdump parameters to kexec_args.
> >
> > The assembly part in kexec_smp_wait provide a routine as BIOS does, in
> > order to keep secondary CPUs in a querying loop.
> >
> > Cc: Eric Biederman <ebiederm@xmission.com>
> > Signed-off-by: Huacai Chen <chenhc@lemote.com>
> > Signed-off-by: Jinyang He <hejinyang@loongson.cn>
> > Signed-off-by: Youling Tang <tangyouling@loongson.cn>
> > ---
> >   arch/mips/kernel/relocate_kernel.S |  27 +++++++++
> >   arch/mips/loongson64/env.c         |   7 +++
> >   arch/mips/loongson64/reset.c       | 111 +++++++++++++++++++++++++++++++++++++
> >   arch/mips/loongson64/smp.c         |   5 ++
> >   4 files changed, 150 insertions(+)
> >
> > diff --git a/arch/mips/kernel/relocate_kernel.S b/arch/mips/kernel/relocate_kernel.S
> > index ac87089..91b2932 100644
> > --- a/arch/mips/kernel/relocate_kernel.S
> > +++ b/arch/mips/kernel/relocate_kernel.S
> > @@ -133,6 +133,33 @@ LEAF(kexec_smp_wait)
> >   #else
> >       sync
> >   #endif
> > +
> > +#ifdef CONFIG_CPU_LOONGSON64
> > +#define MAILBOX_BASE 0x900000003ff01000
> > +     /* s0:prid s1:initfn */
> > +     /* t0:base t1:cpuid t2:node t9:count */
> > +     mfc0  t1, CP0_EBASE
> > +     andi  t1, MIPS_EBASE_CPUNUM
> > +     dli   t0, MAILBOX_BASE    /* mailbox base */
> > +     dins  t0, t1, 8, 2        /* insert core id*/
> > +     dext  t2, t1, 2, 2
> > +     dins  t0, t2, 44, 2       /* insert node id */
> > +     mfc0  s0, CP0_PRID
> > +     andi  s0, s0, 0xf
> > +     blt   s0, 0x6, 1f         /* Loongson-3A1000 */
> > +     bgt   s0, 0x7, 1f         /* Loongson-3A2000/3A3000 */
> > +     dins  t0, t2, 14, 2       /* Loongson-3B1000/3B1500 need bit 15~14 */
>
> Here is right, and I have a trivial idea that machine type
> can be passed by secondary_kexec_args.
It seems not a good idea, if something is detectable, don't use an argument.

>
> > +1:   li    t9, 0x100           /* wait for init loop */
> > +2:   addiu t9, -1              /* limit mailbox access */
> > +     bnez  t9, 2b
> > +     ld    s1, 0x20(t0)        /* get PC via mailbox reg0 */
> > +     beqz  s1, 1b
> > +     ld    sp, 0x28(t0)        /* get SP via mailbox reg1 */
> > +     ld    gp, 0x30(t0)        /* get GP via mailbox reg2 */
> > +     ld    a1, 0x38(t0)
> > +     jr    s1                  /* jump to initial PC */
> > +#endif
> > +
> >       j               s1
> >       END(kexec_smp_wait)
> >   #endif
> > diff --git a/arch/mips/loongson64/env.c b/arch/mips/loongson64/env.c
> > index 134cb8e..e937f31 100644
> > --- a/arch/mips/loongson64/env.c
> > +++ b/arch/mips/loongson64/env.c
> > @@ -120,6 +120,13 @@ void __init prom_init_env(void)
> >       loongson_sysconf.nr_cpus = ecpu->nr_cpus;
> >       loongson_sysconf.boot_cpu_id = ecpu->cpu_startup_core_id;
> >       loongson_sysconf.reserved_cpus_mask = ecpu->reserved_cores_mask;
> > +#ifdef CONFIG_KEXEC
> > +     loongson_sysconf.boot_cpu_id = get_ebase_cpunum();
> > +     loongson_sysconf.reserved_cpus_mask |=
> > +             (1 << loongson_sysconf.boot_cpu_id) - 1;
> > +     pr_info("Boot CPU ID is being fixed from %d to %d\n",
> > +             ecpu->cpu_startup_core_id, loongson_sysconf.boot_cpu_id);
> > +#endif
>
> I don't think secondary CPU will go here.
> At arch/mips/kernel/machine_kexec.c: kexec_reboot(), CPU0 goes
> relocate_new_kernel() and secondary CPUs go to smp_wait. Why
> kdump may triggered at non-boot CPU?
In the past, kexec_reboot() may not be executed on CPU0. For the kexec
case, kernel_kexec() calls migrate_to_reboot_cpu(), which ensures
kexec_reboot() executed on CPU0. But for the kdump case, crash_kexec()
doesn't call migrate_to_reboot_cpu(), so kexec_reboot() may execute on
any CPU because panic can happen on any CPU. Moreover, crash_kexec()
shouldn't call migrate_to_reboot_cpu(), because it is in a "dangerous"
status, which is different from kexec.

However, after commit 62cac480f33f8f9413d609cb1601b0ee5 ("MIPS: kexec:
Make a framework for both jumping and halting on nonboot CPUs"),
kexec_reboot() works as you say. So you are right here and the "fixup"
will be removed in the next version. Since the "fixup" doesn't break
anything, I have never removed it.

>
> >       if (ecpu->nr_cpus > NR_CPUS || ecpu->nr_cpus == 0)
> >               loongson_sysconf.nr_cpus = NR_CPUS;
> >       loongson_sysconf.nr_nodes = (loongson_sysconf.nr_cpus +
> > diff --git a/arch/mips/loongson64/reset.c b/arch/mips/loongson64/reset.c
> > index 3bb8a1e..b1e71f37 100644
> > --- a/arch/mips/loongson64/reset.c
> > +++ b/arch/mips/loongson64/reset.c
> > @@ -6,9 +6,14 @@
> >    * Copyright (C) 2009 Lemote, Inc.
> >    * Author: Zhangjin Wu, wuzhangjin@gmail.com
> >    */
> > +#include <linux/cpu.h>
> > +#include <linux/delay.h>
> >   #include <linux/init.h>
> > +#include <linux/kexec.h>
> >   #include <linux/pm.h>
> > +#include <linux/slab.h>
> >
> > +#include <asm/bootinfo.h>
> >   #include <asm/idle.h>
> >   #include <asm/reboot.h>
> >
> > @@ -47,12 +52,118 @@ static void loongson_halt(void)
> >       }
> >   }
> >
> > +#ifdef CONFIG_KEXEC
> > +
> > +/* 0X80000000~0X80200000 is safe */
> > +#define MAX_ARGS     64
> > +#define KEXEC_CTRL_CODE      0xFFFFFFFF80100000UL
> > +#define KEXEC_ARGV_ADDR      0xFFFFFFFF80108000UL
> > +#define KEXEC_ARGV_SIZE      COMMAND_LINE_SIZE
> > +#define KEXEC_ENVP_SIZE      4800
> > +
> > +static int kexec_argc;
> > +static int kdump_argc;
> > +static void *kexec_argv;
> > +static void *kdump_argv;
> > +static void *kexec_envp;
> > +
> > +static int loongson_kexec_prepare(struct kimage *image)
> > +{
> > +     int i, argc = 0;
> > +     unsigned int *argv;
> > +     char *str, *ptr, *bootloader = "kexec";
> > +
> > +     /* argv at offset 0, argv[] at offset KEXEC_ARGV_SIZE/2 */
> > +     if (image->type == KEXEC_TYPE_DEFAULT)
> > +             argv = (unsigned int *)kexec_argv;
> > +     else
> > +             argv = (unsigned int *)kdump_argv;
> > +
> > +     argv[argc++] = (unsigned int)(KEXEC_ARGV_ADDR + KEXEC_ARGV_SIZE/2);
> > +
> > +     for (i = 0; i < image->nr_segments; i++) {
> > +             if (!strncmp(bootloader, (char *)image->segment[i].buf,
> > +                             strlen(bootloader))) {
> > +                     /*
> > +                      * convert command line string to array
> > +                      * of parameters (as bootloader does).
> > +                      */
> > +                     int offt;
> > +                     str = (char *)argv + KEXEC_ARGV_SIZE/2;
> > +                     memcpy(str, image->segment[i].buf, KEXEC_ARGV_SIZE/2);
> > +                     ptr = strchr(str, ' ');
> > +
> > +                     while (ptr && (argc < MAX_ARGS)) {
> > +                             *ptr = '\0';
> > +                             if (ptr[1] != ' ') {
> > +                                     offt = (int)(ptr - str + 1);
> > +                                     argv[argc] = KEXEC_ARGV_ADDR + KEXEC_ARGV_SIZE/2 + offt;
> > +                                     argc++;
> > +                             }
> > +                             ptr = strchr(ptr + 1, ' ');
> > +                     }
> > +                     break;
> > +             }
> > +     }
> > +
> > +     if (image->type == KEXEC_TYPE_DEFAULT)
> > +             kexec_argc = argc;
> > +     else
> > +             kdump_argc = argc;
> > +
> > +     /* kexec/kdump need a safe page to save reboot_code_buffer */
> > +     image->control_code_page = virt_to_page((void *)KEXEC_CTRL_CODE);
>
> I thought find a hole in Crash Kernel may better than a static position.
> It just a trivial idea, too.
It is very difficult, the "safe region" should not be corrupted by
both the old and new kernel, so the low 2MB is a good place.


>
> > +
> > +     return 0;
> > +}
> > +
> > +static void loongson_kexec_shutdown(void)
> > +{
> > +#ifdef CONFIG_SMP
> > +     int cpu;
> > +
> > +     /* All CPUs go to reboot_code_buffer */
> > +     for_each_possible_cpu(cpu)
> > +             if (!cpu_online(cpu))
> > +                     cpu_device_up(get_cpu_device(cpu));
> > +#endif
> > +     kexec_args[0] = kexec_argc;
> > +     kexec_args[1] = fw_arg1;
> > +     kexec_args[2] = fw_arg2;
> > +     memcpy((void *)fw_arg1, kexec_argv, KEXEC_ARGV_SIZE);
> > +     memcpy((void *)fw_arg2, kexec_envp, KEXEC_ENVP_SIZE);
> > +}
> > +
> > +static void loongson_crash_shutdown(struct pt_regs *regs)
> > +{
> > +     default_machine_crash_shutdown(regs);
> > +     kexec_args[0] = kdump_argc;
> > +     kexec_args[1] = fw_arg1;
> > +     kexec_args[2] = fw_arg2;
> > +     memcpy((void *)fw_arg1, kdump_argv, KEXEC_ARGV_SIZE);
> > +     memcpy((void *)fw_arg2, kexec_envp, KEXEC_ENVP_SIZE);
> > +}
> > +
> > +#endif
> > +
> >   static int __init mips_reboot_setup(void)
> >   {
> >       _machine_restart = loongson_restart;
> >       _machine_halt = loongson_halt;
> >       pm_power_off = loongson_poweroff;
> >
> > +#ifdef CONFIG_KEXEC
> > +     kexec_argv = kmalloc(KEXEC_ARGV_SIZE, GFP_KERNEL);
> > +     kdump_argv = kmalloc(KEXEC_ARGV_SIZE, GFP_KERNEL);
> > +     kexec_envp = kmalloc(KEXEC_ENVP_SIZE, GFP_KERNEL);
> > +     fw_arg1 = KEXEC_ARGV_ADDR;
> > +     memcpy(kexec_envp, (void *)fw_arg2, KEXEC_ENVP_SIZE);
> > +
> > +     _machine_kexec_prepare = loongson_kexec_prepare;
> > +     _machine_kexec_shutdown = loongson_kexec_shutdown;
> > +     _machine_crash_shutdown = loongson_crash_shutdown;
> > +#endif
> > +
> >       return 0;
> >   }
> >
> > diff --git a/arch/mips/loongson64/smp.c b/arch/mips/loongson64/smp.c
> > index e744e1b..55697ac 100644
> > --- a/arch/mips/loongson64/smp.c
> > +++ b/arch/mips/loongson64/smp.c
> > @@ -420,6 +420,11 @@ static void __init loongson3_smp_setup(void)
> >       ipi_status0_regs_init();
> >       ipi_en0_regs_init();
> >       ipi_mailbox_buf_init();
> > +
> > +     /* BIOS clear the mailbox, but KEXEC bypass BIOS so clear here */
> > +     for (i = 0; i < loongson_sysconf.nr_cpus; i++)
> > +             loongson3_ipi_write64(0, (void *)(ipi_mailbox_buf[i]+0x0));
> > +
>
> Does it need? It take a long time after secondary CPUs geting PC in a loop.
> Or you can clear it early.
This is nearly the same story as the above "fixup". In the past, if
kdump triggered on a non-boot CPU, the boot CPU would go to the query
loop. So, this will also be removed. Thank you.

>
> Thanks,
> - Jinyang.
>
> >       cpu_set_core(&cpu_data[0],
> >                    cpu_logical_map(0) % loongson_sysconf.cores_per_package);
> >       cpu_data[0].package = cpu_logical_map(0) / loongson_sysconf.cores_per_package;
>

Re: [PATCH 3/3] MIPS: Loongson64: Add kexec/kdump support

[Jiaxun Yang]

在 2020/9/23 10:30, Huacai Chen 写道:
> Add kexec/kdump support for Loongson64 by:
> 1, Provide Loongson-specific kexec functions: loongson_kexec_prepare,
>     loongson_kexec_shutdown and loongson_crash_shutdown;
> 2, Provide Loongson-specific assembly code in kexec_smp_wait;
> 3, Clear mailbox in loongson3_smp_setup() since KEXEC bypass BIOS;
> 4, KEXEC always run at boot CPU, but KDUMP may triggered at non-boot
>     CPU. Loongson64 assume boot CPU is the first possible cpu, so fix
>     boot_cpu_id in prom_init_env();
>
> To start Loongson64, The boot CPU needs 3 parameters:
> fw_arg0: the number of arguments in cmdline (i.e., argc).
> fw_arg1: structure holds cmdline such as "root=/dev/sda1 console=tty"
>           (i.e., argv).
> fw_arg2: environment (i.e., envp, additional boot parameters from LEFI).
>
> Non-boot CPUs do not need parameters at once. They query their own IPI
> mailbox to get PC, SP and GP in a loop until boot CPU brings them up.
>
> loongson_kexec_prepare(): Setup cmdline for kexec/kdump. The kexec/kdump
> cmdline comes from kexec's "append" option string. This structure will
> be parsed in fw_init_cmdline() of arch/mips/fw/lib/cmdline.c. Both image
> ->control_code_page and the cmdline need to be in a safe memory region
> (memory allocated by the old kernel may be corrupted by the new kernel).
> In order to maintain compatibility for the old firmware, the low 2MB is
> reserverd and safe for Loongson. So let KEXEC_CTRL_CODE and KEXEC_ARGV_
> ADDR be here. LEFI parameters may be corrupted at runtime, so backup it
> at mips_reboot_setup(), and then restore it at loongson_kexec_shutdown()
> /loongson_crash_shutdown().
>
> loongson_kexec_shutdown(): Wake up all present CPUs and let them go to
> reboot_code_buffer. Pass the kexec parameters to kexec_args.
>
> loongson_crash_shutdown(): Pass the kdump parameters to kexec_args.
>
> The assembly part in kexec_smp_wait provide a routine as BIOS does, in
> order to keep secondary CPUs in a querying loop.
>
> Cc: Eric Biederman <ebiederm@xmission.com>
> Signed-off-by: Huacai Chen <chenhc@lemote.com>
> Signed-off-by: Jinyang He <hejinyang@loongson.cn>
> Signed-off-by: Youling Tang <tangyouling@loongson.cn>
> ---
>   arch/mips/kernel/relocate_kernel.S |  27 +++++++++
>   arch/mips/loongson64/env.c         |   7 +++
>   arch/mips/loongson64/reset.c       | 111 +++++++++++++++++++++++++++++++++++++
>   arch/mips/loongson64/smp.c         |   5 ++
>   4 files changed, 150 insertions(+)
>
> diff --git a/arch/mips/kernel/relocate_kernel.S b/arch/mips/kernel/relocate_kernel.S
> index ac87089..91b2932 100644
> --- a/arch/mips/kernel/relocate_kernel.S
> +++ b/arch/mips/kernel/relocate_kernel.S
> @@ -133,6 +133,33 @@ LEAF(kexec_smp_wait)
>   #else
>   	sync
>   #endif

Hi Jingyang, Huacai,

Thanks a lot for what you have done on Loongson64 support.

Just add my cents here.

> +
> +#ifdef CONFIG_CPU_LOONGSON64
> +#define MAILBOX_BASE 0x900000003ff01000
> +	/* s0:prid s1:initfn */
> +	/* t0:base t1:cpuid t2:node t9:count */
> +	mfc0  t1, CP0_EBASE
> +	andi  t1, MIPS_EBASE_CPUNUM
> +	dli   t0, MAILBOX_BASE    /* mailbox base */
> +	dins  t0, t1, 8, 2        /* insert core id*/
> +	dext  t2, t1, 2, 2
> +	dins  t0, t2, 44, 2       /* insert node id */
> +	mfc0  s0, CP0_PRID
> +	andi  s0, s0, 0xf
> +	blt   s0, 0x6, 1f         /* Loongson-3A1000 */
> +	bgt   s0, 0x7, 1f         /* Loongson-3A2000/3A3000 */
> +	dins  t0, t2, 14, 2       /* Loongson-3B1000/3B1500 need bit 15~14 */
> +1:	li    t9, 0x100           /* wait for init loop */
> +2:	addiu t9, -1              /* limit mailbox access */
> +	bnez  t9, 2b
> +	ld    s1, 0x20(t0)        /* get PC via mailbox reg0 */
> +	beqz  s1, 1b
> +	ld    sp, 0x28(t0)        /* get SP via mailbox reg1 */
> +	ld    gp, 0x30(t0)        /* get GP via mailbox reg2 */
> +	ld    a1, 0x38(t0)
> +	jr    s1                  /* jump to initial PC */
> +#endif
> +

I'm just a little bit uncomfortable with this kind of hardcoded address.
Is it possible to generate kexec_smp_wait with uasm, or pass the SMP
base as a parameter of this function?

Also I do think we can handle kexec_flag in Loongson's platform SMP,
or even generic MIPS SMP code just like what cavium did so this kind
of platform quirk can be avoided.

>   	j		s1
>   	END(kexec_smp_wait)
>   #endif
> diff --git a/arch/mips/loongson64/env.c b/arch/mips/loongson64/env.c
> index 134cb8e..e937f31 100644
> --- a/arch/mips/loongson64/env.c
> +++ b/arch/mips/loongson64/env.c
> @@ -120,6 +120,13 @@ void __init prom_init_env(void)
>   	loongson_sysconf.nr_cpus = ecpu->nr_cpus;
>   	loongson_sysconf.boot_cpu_id = ecpu->cpu_startup_core_id;
>   	loongson_sysconf.reserved_cpus_mask = ecpu->reserved_cores_mask;
> +#ifdef CONFIG_KEXEC
> +	loongson_sysconf.boot_cpu_id = get_ebase_cpunum();
> +	loongson_sysconf.reserved_cpus_mask |=
> +		(1 << loongson_sysconf.boot_cpu_id) - 1;
> +	pr_info("Boot CPU ID is being fixed from %d to %d\n",
> +		ecpu->cpu_startup_core_id, loongson_sysconf.boot_cpu_id);
> +#endif

This could be done unconditionally and split to another patch,
"MIPS: Loongson64: Quirk inaccurate bootcore from the firmware"

>   	if (ecpu->nr_cpus > NR_CPUS || ecpu->nr_cpus == 0)
>   		loongson_sysconf.nr_cpus = NR_CPUS;
>   	loongson_sysconf.nr_nodes = (loongson_sysconf.nr_cpus +
> diff --git a/arch/mips/loongson64/reset.c b/arch/mips/loongson64/reset.c
> index 3bb8a1e..b1e71f37 100644
> --- a/arch/mips/loongson64/reset.c
> +++ b/arch/mips/loongson64/reset.c
> @@ -6,9 +6,14 @@
>    * Copyright (C) 2009 Lemote, Inc.
>    * Author: Zhangjin Wu, wuzhangjin@gmail.com
>    */
> +#include <linux/cpu.h>
> +#include <linux/delay.h>
>   #include <linux/init.h>
> +#include <linux/kexec.h>
>   #include <linux/pm.h>
> +#include <linux/slab.h>
>   
> +#include <asm/bootinfo.h>
>   #include <asm/idle.h>
>   #include <asm/reboot.h>
>   
> @@ -47,12 +52,118 @@ static void loongson_halt(void)
>   	}
>   }
>   
> +#ifdef CONFIG_KEXEC
> +
> +/* 0X80000000~0X80200000 is safe */
> +#define MAX_ARGS	64
> +#define KEXEC_CTRL_CODE	0xFFFFFFFF80100000UL
> +#define KEXEC_ARGV_ADDR	0xFFFFFFFF80108000UL
> +#define KEXEC_ARGV_SIZE	COMMAND_LINE_SIZE
> +#define KEXEC_ENVP_SIZE	4800
I won't say it's safe.
Loongson-2K's PMON may place reboot vector here, also some
UEFI firmware may place their suspend stack here.
What if we allocate these buffer at runtime?

Thanks.

- Jiaxun

Re: [PATCH 3/3] MIPS: Loongson64: Add kexec/kdump support

[Huacai Chen]

Hi, Jiaxun,

On Wed, Sep 23, 2020 at 9:37 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:
>
>
>
> 在 2020/9/23 10:30, Huacai Chen 写道:
> > Add kexec/kdump support for Loongson64 by:
> > 1, Provide Loongson-specific kexec functions: loongson_kexec_prepare,
> >     loongson_kexec_shutdown and loongson_crash_shutdown;
> > 2, Provide Loongson-specific assembly code in kexec_smp_wait;
> > 3, Clear mailbox in loongson3_smp_setup() since KEXEC bypass BIOS;
> > 4, KEXEC always run at boot CPU, but KDUMP may triggered at non-boot
> >     CPU. Loongson64 assume boot CPU is the first possible cpu, so fix
> >     boot_cpu_id in prom_init_env();
> >
> > To start Loongson64, The boot CPU needs 3 parameters:
> > fw_arg0: the number of arguments in cmdline (i.e., argc).
> > fw_arg1: structure holds cmdline such as "root=/dev/sda1 console=tty"
> >           (i.e., argv).
> > fw_arg2: environment (i.e., envp, additional boot parameters from LEFI).
> >
> > Non-boot CPUs do not need parameters at once. They query their own IPI
> > mailbox to get PC, SP and GP in a loop until boot CPU brings them up.
> >
> > loongson_kexec_prepare(): Setup cmdline for kexec/kdump. The kexec/kdump
> > cmdline comes from kexec's "append" option string. This structure will
> > be parsed in fw_init_cmdline() of arch/mips/fw/lib/cmdline.c. Both image
> > ->control_code_page and the cmdline need to be in a safe memory region
> > (memory allocated by the old kernel may be corrupted by the new kernel).
> > In order to maintain compatibility for the old firmware, the low 2MB is
> > reserverd and safe for Loongson. So let KEXEC_CTRL_CODE and KEXEC_ARGV_
> > ADDR be here. LEFI parameters may be corrupted at runtime, so backup it
> > at mips_reboot_setup(), and then restore it at loongson_kexec_shutdown()
> > /loongson_crash_shutdown().
> >
> > loongson_kexec_shutdown(): Wake up all present CPUs and let them go to
> > reboot_code_buffer. Pass the kexec parameters to kexec_args.
> >
> > loongson_crash_shutdown(): Pass the kdump parameters to kexec_args.
> >
> > The assembly part in kexec_smp_wait provide a routine as BIOS does, in
> > order to keep secondary CPUs in a querying loop.
> >
> > Cc: Eric Biederman <ebiederm@xmission.com>
> > Signed-off-by: Huacai Chen <chenhc@lemote.com>
> > Signed-off-by: Jinyang He <hejinyang@loongson.cn>
> > Signed-off-by: Youling Tang <tangyouling@loongson.cn>
> > ---
> >   arch/mips/kernel/relocate_kernel.S |  27 +++++++++
> >   arch/mips/loongson64/env.c         |   7 +++
> >   arch/mips/loongson64/reset.c       | 111 +++++++++++++++++++++++++++++++++++++
> >   arch/mips/loongson64/smp.c         |   5 ++
> >   4 files changed, 150 insertions(+)
> >
> > diff --git a/arch/mips/kernel/relocate_kernel.S b/arch/mips/kernel/relocate_kernel.S
> > index ac87089..91b2932 100644
> > --- a/arch/mips/kernel/relocate_kernel.S
> > +++ b/arch/mips/kernel/relocate_kernel.S
> > @@ -133,6 +133,33 @@ LEAF(kexec_smp_wait)
> >   #else
> >       sync
> >   #endif
>
> Hi Jingyang, Huacai,
>
> Thanks a lot for what you have done on Loongson64 support.
>
> Just add my cents here.
>
> > +
> > +#ifdef CONFIG_CPU_LOONGSON64
> > +#define MAILBOX_BASE 0x900000003ff01000
> > +     /* s0:prid s1:initfn */
> > +     /* t0:base t1:cpuid t2:node t9:count */
> > +     mfc0  t1, CP0_EBASE
> > +     andi  t1, MIPS_EBASE_CPUNUM
> > +     dli   t0, MAILBOX_BASE    /* mailbox base */
> > +     dins  t0, t1, 8, 2        /* insert core id*/
> > +     dext  t2, t1, 2, 2
> > +     dins  t0, t2, 44, 2       /* insert node id */
> > +     mfc0  s0, CP0_PRID
> > +     andi  s0, s0, 0xf
> > +     blt   s0, 0x6, 1f         /* Loongson-3A1000 */
> > +     bgt   s0, 0x7, 1f         /* Loongson-3A2000/3A3000 */
> > +     dins  t0, t2, 14, 2       /* Loongson-3B1000/3B1500 need bit 15~14 */
> > +1:   li    t9, 0x100           /* wait for init loop */
> > +2:   addiu t9, -1              /* limit mailbox access */
> > +     bnez  t9, 2b
> > +     ld    s1, 0x20(t0)        /* get PC via mailbox reg0 */
> > +     beqz  s1, 1b
> > +     ld    sp, 0x28(t0)        /* get SP via mailbox reg1 */
> > +     ld    gp, 0x30(t0)        /* get GP via mailbox reg2 */
> > +     ld    a1, 0x38(t0)
> > +     jr    s1                  /* jump to initial PC */
> > +#endif
> > +
>
> I'm just a little bit uncomfortable with this kind of hardcoded address.
> Is it possible to generate kexec_smp_wait with uasm, or pass the SMP
> base as a parameter of this function?
This is very difficult, and moreover, uasm wrap the assembly in C
functions, can it be more beautiful? In my opinion, uasm is only used
for performance-critical routines, not for beautiful code. But anyway,
0x900000003ff01000 should be improved.

>
> Also I do think we can handle kexec_flag in Loongson's platform SMP,
> or even generic MIPS SMP code just like what cavium did so this kind
> of platform quirk can be avoided.
I doubt this can be done, every CPU has its own method of SMP bringup.

>
> >       j               s1
> >       END(kexec_smp_wait)
> >   #endif
> > diff --git a/arch/mips/loongson64/env.c b/arch/mips/loongson64/env.c
> > index 134cb8e..e937f31 100644
> > --- a/arch/mips/loongson64/env.c
> > +++ b/arch/mips/loongson64/env.c
> > @@ -120,6 +120,13 @@ void __init prom_init_env(void)
> >       loongson_sysconf.nr_cpus = ecpu->nr_cpus;
> >       loongson_sysconf.boot_cpu_id = ecpu->cpu_startup_core_id;
> >       loongson_sysconf.reserved_cpus_mask = ecpu->reserved_cores_mask;
> > +#ifdef CONFIG_KEXEC
> > +     loongson_sysconf.boot_cpu_id = get_ebase_cpunum();
> > +     loongson_sysconf.reserved_cpus_mask |=
> > +             (1 << loongson_sysconf.boot_cpu_id) - 1;
> > +     pr_info("Boot CPU ID is being fixed from %d to %d\n",
> > +             ecpu->cpu_startup_core_id, loongson_sysconf.boot_cpu_id);
> > +#endif
>
> This could be done unconditionally and split to another patch,
> "MIPS: Loongson64: Quirk inaccurate bootcore from the firmware"
This will be wholly removed.

>
> >       if (ecpu->nr_cpus > NR_CPUS || ecpu->nr_cpus == 0)
> >               loongson_sysconf.nr_cpus = NR_CPUS;
> >       loongson_sysconf.nr_nodes = (loongson_sysconf.nr_cpus +
> > diff --git a/arch/mips/loongson64/reset.c b/arch/mips/loongson64/reset.c
> > index 3bb8a1e..b1e71f37 100644
> > --- a/arch/mips/loongson64/reset.c
> > +++ b/arch/mips/loongson64/reset.c
> > @@ -6,9 +6,14 @@
> >    * Copyright (C) 2009 Lemote, Inc.
> >    * Author: Zhangjin Wu, wuzhangjin@gmail.com
> >    */
> > +#include <linux/cpu.h>
> > +#include <linux/delay.h>
> >   #include <linux/init.h>
> > +#include <linux/kexec.h>
> >   #include <linux/pm.h>
> > +#include <linux/slab.h>
> >
> > +#include <asm/bootinfo.h>
> >   #include <asm/idle.h>
> >   #include <asm/reboot.h>
> >
> > @@ -47,12 +52,118 @@ static void loongson_halt(void)
> >       }
> >   }
> >
> > +#ifdef CONFIG_KEXEC
> > +
> > +/* 0X80000000~0X80200000 is safe */
> > +#define MAX_ARGS     64
> > +#define KEXEC_CTRL_CODE      0xFFFFFFFF80100000UL
> > +#define KEXEC_ARGV_ADDR      0xFFFFFFFF80108000UL
> > +#define KEXEC_ARGV_SIZE      COMMAND_LINE_SIZE
> > +#define KEXEC_ENVP_SIZE      4800
> I won't say it's safe.
> Loongson-2K's PMON may place reboot vector here, also some
> UEFI firmware may place their suspend stack here.
> What if we allocate these buffer at runtime?
The layout of low 2MB in our design:
0x80000000 the first MB, the first 64K：Exception vector
0x80010000 the first MB, the second 64K：STR(suspend) data
0x80020000 the first MB, the third and fourth 64K：UEFI HOB
0x80040000 the first MB, the fifth 64K：RT-Thread for SMC
0x80100000 the second MB, the first 64K：KEXEC code
0x80108000 the second MB, the second 64K：KEXEC data

All allocated buffer from the old kernel is not safe, because the new
kernel may be larger than the old kernel. Even if the low 2MB is not a
perfect place, it is the best place we can choose.

Huacai

>
> Thanks.
>
> - Jiaxun
>

Re: [PATCH 3/3] MIPS: Loongson64: Add kexec/kdump support

[Jiaxun Yang]

在 2020/9/24 9:19, Huacai Chen 写道:
> Hi, Jiaxun,
[...]
>
>
> I'm just a little bit uncomfortable with this kind of hardcoded address.
> Is it possible to generate kexec_smp_wait with uasm, or pass the SMP
> base as a parameter of this function?
> This is very difficult, and moreover, uasm wrap the assembly in C
> functions, can it be more beautiful? In my opinion, uasm is only used
> for performance-critical routines, not for beautiful code. But anyway,
> 0x900000003ff01000 should be improved.
>
>> Also I do think we can handle kexec_flag in Loongson's platform SMP,
>> or even generic MIPS SMP code just like what cavium did so this kind
>> of platform quirk can be avoided.
> I doubt this can be done, every CPU has its own method of SMP bringup.
Hi Huacai,

I don't know kexec very well.. So please ignore my ideas if they're not 
applicatable.

Hmm, as we've got control of all secondnary processors at the point, I 
suspect
we can skip platform bring-up code.

Cavium makes use of secondary_kexec_args, can we do it as well?

I think kexec_flag was designed for this reason.
Current code is sending all secondnary processors to the entry point
of the new kernel, probably we can do something here.

>>
>> I won't say it's safe.
>> Loongson-2K's PMON may place reboot vector here, also some
>> UEFI firmware may place their suspend stack here.
>> What if we allocate these buffer at runtime?
> The layout of low 2MB in our design:
> 0x80000000 the first MB, the first 64K：Exception vector
> 0x80010000 the first MB, the second 64K：STR(suspend) data
> 0x80020000 the first MB, the third and fourth 64K：UEFI HOB
> 0x80040000 the first MB, the fifth 64K：RT-Thread for SMC
> 0x80100000 the second MB, the first 64K：KEXEC code
> 0x80108000 the second MB, the second 64K：KEXEC data

Well, there are some other vendors violating this design.
Especially for embedded systems like Loongson-2K.

> All allocated buffer from the old kernel is not safe, because the new
> kernel may be larger than the old kernel. Even if the low 2MB is not a
> perfect place, it is the best place we can choose.
Oops I ignored overlapping..
Then it looks fine for me..

Thanks.

- Jiaxun


>
> Huacai
>
>> Thanks.
>>
>> - Jiaxun
>>

Re: [PATCH 3/3] MIPS: Loongson64: Add kexec/kdump support

[Jinyang He]

On 09/24/2020 09:19 AM, Huacai Chen wrote:
> Hi, Jiaxun,
>
> On Wed, Sep 23, 2020 at 9:37 PM Jiaxun Yang <jiaxun.yang@flygoat.com> wrote:
>>
>>
>> 在 2020/9/23 10:30, Huacai Chen 写道:
>>> Add kexec/kdump support for Loongson64 by:
>>> 1, Provide Loongson-specific kexec functions: loongson_kexec_prepare,
>>>      loongson_kexec_shutdown and loongson_crash_shutdown;
>>> 2, Provide Loongson-specific assembly code in kexec_smp_wait;
>>> 3, Clear mailbox in loongson3_smp_setup() since KEXEC bypass BIOS;
>>> 4, KEXEC always run at boot CPU, but KDUMP may triggered at non-boot
>>>      CPU. Loongson64 assume boot CPU is the first possible cpu, so fix
>>>      boot_cpu_id in prom_init_env();
>>>
>>> To start Loongson64, The boot CPU needs 3 parameters:
>>> fw_arg0: the number of arguments in cmdline (i.e., argc).
>>> fw_arg1: structure holds cmdline such as "root=/dev/sda1 console=tty"
>>>            (i.e., argv).
>>> fw_arg2: environment (i.e., envp, additional boot parameters from LEFI).
>>>
>>> Non-boot CPUs do not need parameters at once. They query their own IPI
>>> mailbox to get PC, SP and GP in a loop until boot CPU brings them up.
>>>
>>> loongson_kexec_prepare(): Setup cmdline for kexec/kdump. The kexec/kdump
>>> cmdline comes from kexec's "append" option string. This structure will
>>> be parsed in fw_init_cmdline() of arch/mips/fw/lib/cmdline.c. Both image
>>> ->control_code_page and the cmdline need to be in a safe memory region
>>> (memory allocated by the old kernel may be corrupted by the new kernel).
>>> In order to maintain compatibility for the old firmware, the low 2MB is
>>> reserverd and safe for Loongson. So let KEXEC_CTRL_CODE and KEXEC_ARGV_
>>> ADDR be here. LEFI parameters may be corrupted at runtime, so backup it
>>> at mips_reboot_setup(), and then restore it at loongson_kexec_shutdown()
>>> /loongson_crash_shutdown().
>>>
>>> loongson_kexec_shutdown(): Wake up all present CPUs and let them go to
>>> reboot_code_buffer. Pass the kexec parameters to kexec_args.
>>>
>>> loongson_crash_shutdown(): Pass the kdump parameters to kexec_args.
>>>
>>> The assembly part in kexec_smp_wait provide a routine as BIOS does, in
>>> order to keep secondary CPUs in a querying loop.
>>>
>>> Cc: Eric Biederman <ebiederm@xmission.com>
>>> Signed-off-by: Huacai Chen <chenhc@lemote.com>
>>> Signed-off-by: Jinyang He <hejinyang@loongson.cn>
>>> Signed-off-by: Youling Tang <tangyouling@loongson.cn>
>>> ---
>>>    arch/mips/kernel/relocate_kernel.S |  27 +++++++++
>>>    arch/mips/loongson64/env.c         |   7 +++
>>>    arch/mips/loongson64/reset.c       | 111 +++++++++++++++++++++++++++++++++++++
>>>    arch/mips/loongson64/smp.c         |   5 ++
>>>    4 files changed, 150 insertions(+)
>>>
>>> diff --git a/arch/mips/kernel/relocate_kernel.S b/arch/mips/kernel/relocate_kernel.S
>>> index ac87089..91b2932 100644
>>> --- a/arch/mips/kernel/relocate_kernel.S
>>> +++ b/arch/mips/kernel/relocate_kernel.S
>>> @@ -133,6 +133,33 @@ LEAF(kexec_smp_wait)
>>>    #else
>>>        sync
>>>    #endif
>> Hi Jingyang, Huacai,
>>
>> Thanks a lot for what you have done on Loongson64 support.
>>
>> Just add my cents here.
>>
>>> +
>>> +#ifdef CONFIG_CPU_LOONGSON64
>>> +#define MAILBOX_BASE 0x900000003ff01000
>>> +     /* s0:prid s1:initfn */
>>> +     /* t0:base t1:cpuid t2:node t9:count */
>>> +     mfc0  t1, CP0_EBASE
>>> +     andi  t1, MIPS_EBASE_CPUNUM
>>> +     dli   t0, MAILBOX_BASE    /* mailbox base */
>>> +     dins  t0, t1, 8, 2        /* insert core id*/
>>> +     dext  t2, t1, 2, 2
>>> +     dins  t0, t2, 44, 2       /* insert node id */
>>> +     mfc0  s0, CP0_PRID
>>> +     andi  s0, s0, 0xf
>>> +     blt   s0, 0x6, 1f         /* Loongson-3A1000 */
>>> +     bgt   s0, 0x7, 1f         /* Loongson-3A2000/3A3000 */
>>> +     dins  t0, t2, 14, 2       /* Loongson-3B1000/3B1500 need bit 15~14 */
>>> +1:   li    t9, 0x100           /* wait for init loop */
>>> +2:   addiu t9, -1              /* limit mailbox access */
>>> +     bnez  t9, 2b
>>> +     ld    s1, 0x20(t0)        /* get PC via mailbox reg0 */
>>> +     beqz  s1, 1b
>>> +     ld    sp, 0x28(t0)        /* get SP via mailbox reg1 */
>>> +     ld    gp, 0x30(t0)        /* get GP via mailbox reg2 */
>>> +     ld    a1, 0x38(t0)
>>> +     jr    s1                  /* jump to initial PC */
>>> +#endif
>>> +
>> I'm just a little bit uncomfortable with this kind of hardcoded address.
>> Is it possible to generate kexec_smp_wait with uasm, or pass the SMP
>> base as a parameter of this function?
> This is very difficult, and moreover, uasm wrap the assembly in C
> functions, can it be more beautiful? In my opinion, uasm is only used
> for performance-critical routines, not for beautiful code. But anyway,
> 0x900000003ff01000 should be improved.
>
>> Also I do think we can handle kexec_flag in Loongson's platform SMP,
>> or even generic MIPS SMP code just like what cavium did so this kind
>> of platform quirk can be avoided.
> I doubt this can be done, every CPU has its own method of SMP bringup.
>
>>>        j               s1
>>>        END(kexec_smp_wait)
>>>    #endif
>>> diff --git a/arch/mips/loongson64/env.c b/arch/mips/loongson64/env.c
>>> index 134cb8e..e937f31 100644
>>> --- a/arch/mips/loongson64/env.c
>>> +++ b/arch/mips/loongson64/env.c
>>> @@ -120,6 +120,13 @@ void __init prom_init_env(void)
>>>        loongson_sysconf.nr_cpus = ecpu->nr_cpus;
>>>        loongson_sysconf.boot_cpu_id = ecpu->cpu_startup_core_id;
>>>        loongson_sysconf.reserved_cpus_mask = ecpu->reserved_cores_mask;
>>> +#ifdef CONFIG_KEXEC
>>> +     loongson_sysconf.boot_cpu_id = get_ebase_cpunum();
>>> +     loongson_sysconf.reserved_cpus_mask |=
>>> +             (1 << loongson_sysconf.boot_cpu_id) - 1;
>>> +     pr_info("Boot CPU ID is being fixed from %d to %d\n",
>>> +             ecpu->cpu_startup_core_id, loongson_sysconf.boot_cpu_id);
>>> +#endif
>> This could be done unconditionally and split to another patch,
>> "MIPS: Loongson64: Quirk inaccurate bootcore from the firmware"
> This will be wholly removed.
>
>>>        if (ecpu->nr_cpus > NR_CPUS || ecpu->nr_cpus == 0)
>>>                loongson_sysconf.nr_cpus = NR_CPUS;
>>>        loongson_sysconf.nr_nodes = (loongson_sysconf.nr_cpus +
>>> diff --git a/arch/mips/loongson64/reset.c b/arch/mips/loongson64/reset.c
>>> index 3bb8a1e..b1e71f37 100644
>>> --- a/arch/mips/loongson64/reset.c
>>> +++ b/arch/mips/loongson64/reset.c
>>> @@ -6,9 +6,14 @@
>>>     * Copyright (C) 2009 Lemote, Inc.
>>>     * Author: Zhangjin Wu, wuzhangjin@gmail.com
>>>     */
>>> +#include <linux/cpu.h>
>>> +#include <linux/delay.h>
>>>    #include <linux/init.h>
>>> +#include <linux/kexec.h>
>>>    #include <linux/pm.h>
>>> +#include <linux/slab.h>
>>>
>>> +#include <asm/bootinfo.h>
>>>    #include <asm/idle.h>
>>>    #include <asm/reboot.h>
>>>
>>> @@ -47,12 +52,118 @@ static void loongson_halt(void)
>>>        }
>>>    }
>>>
>>> +#ifdef CONFIG_KEXEC
>>> +
>>> +/* 0X80000000~0X80200000 is safe */
>>> +#define MAX_ARGS     64
>>> +#define KEXEC_CTRL_CODE      0xFFFFFFFF80100000UL
>>> +#define KEXEC_ARGV_ADDR      0xFFFFFFFF80108000UL
>>> +#define KEXEC_ARGV_SIZE      COMMAND_LINE_SIZE
>>> +#define KEXEC_ENVP_SIZE      4800
>> I won't say it's safe.
>> Loongson-2K's PMON may place reboot vector here, also some
>> UEFI firmware may place their suspend stack here.
>> What if we allocate these buffer at runtime?
> The layout of low 2MB in our design:
> 0x80000000 the first MB, the first 64K：Exception vector
> 0x80010000 the first MB, the second 64K：STR(suspend) data
> 0x80020000 the first MB, the third and fourth 64K：UEFI HOB
> 0x80040000 the first MB, the fifth 64K：RT-Thread for SMC
> 0x80100000 the second MB, the first 64K：KEXEC code
> 0x80108000 the second MB, the second 64K：KEXEC data

Hi, Huacai,

Thank you for your explanation. It's really a safe region.
What I want to do is similar to other architecture which find a hole
in Crash kernel. It's also safe if "mem=" limit Capture kernel RAM. "mem="
is useful and sorry for the wrong comment yesterday. "mem=" excluded
cmdline segment and elfcorehdr segment. I have an idea that add a segment
for Loongson64 platform control_code_page. Find the hole by top-down that
different from common bottom-up way.
As you said, it's really difficult. More work should be do not only kernel
but also kexec-tools. At this stage, I am in favor of you that
control_code_page should point to the safe ragion in low 2MB.

> All allocated buffer from the old kernel is not safe, because the new
> kernel may be larger than the old kernel. Even if the low 2MB is not a
> perfect place, it is the best place we can choose.
>
> Huacai
>
>> Thanks.
>>
>> - Jiaxun
>>

Re: [PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Baoquan He]

On 09/23/20 at 10:30am, Huacai Chen wrote:
> Kexec-tools use mem=X@Y to pass usable memories to crash kernel, but in
> commit a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map") all
> BIOS passed memories are removed by early_parse_mem(). I think this is
> reasonable for a normal kernel but not for a crash kernel, because a
> crash kernel should be able to see all old memories, even though it is
> not supposed to use them.

I am not familiar with MIPS code, but we analyze and fill memmap= to
pass usable memory to crashkenrel in kexec-tools, do you mean you are
specifying memmap= or mem= by hand?

And we don't have mem=X@Y, only mem=nn[KMG].

> 
> Fixes: a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map")
> Signed-off-by: Huacai Chen <chenhc@lemote.com>
> ---
>  arch/mips/kernel/setup.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> index 4c04a86..e2804a2 100644
> --- a/arch/mips/kernel/setup.c
> +++ b/arch/mips/kernel/setup.c
> @@ -392,8 +392,10 @@ static int __init early_parse_mem(char *p)
>  	 */
>  	if (usermem == 0) {
>  		usermem = 1;
> +#ifndef CONFIG_CRASH_DUMP
>  		memblock_remove(memblock_start_of_DRAM(),
>  			memblock_end_of_DRAM() - memblock_start_of_DRAM());
> +#endif
>  	}
>  	start = 0;
>  	size = memparse(p, &p);
> -- 
> 2.7.0
> 

Re: [PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Huacai Chen]

Hi, Baoquan,

On Wed, Sep 23, 2020 at 10:46 AM Baoquan He <bhe@redhat.com> wrote:
>
> On 09/23/20 at 10:30am, Huacai Chen wrote:
> > Kexec-tools use mem=X@Y to pass usable memories to crash kernel, but in
> > commit a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map") all
> > BIOS passed memories are removed by early_parse_mem(). I think this is
> > reasonable for a normal kernel but not for a crash kernel, because a
> > crash kernel should be able to see all old memories, even though it is
> > not supposed to use them.
>
> I am not familiar with MIPS code, but we analyze and fill memmap= to
> pass usable memory to crashkenrel in kexec-tools, do you mean you are
> specifying memmap= or mem= by hand?
Not by hand, but by code of kexec-tools via the "mem=" parameter.

As I know, kexec-tools of MIPS only use "mem=" to pass "usable"
memory, but not "visible" memory. "Visible" memory of the crash kernel
is still passed by BIOS (strictly, by the old kernel who duplicates
information from BIOS). If memblock_remove() executed here, it would
remove all "visible" memory and make "visible" memory the same as
"usable" memory, and I think this is not correct.

>
> And we don't have mem=X@Y, only mem=nn[KMG].
The relocatable kernel of MIPS is still unusable now, so MIPS should
use mem=X@Y, and the crash kernel is always different from normal
kernel.

Huacai

>
> >
> > Fixes: a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map")
> > Signed-off-by: Huacai Chen <chenhc@lemote.com>
> > ---
> >  arch/mips/kernel/setup.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> > index 4c04a86..e2804a2 100644
> > --- a/arch/mips/kernel/setup.c
> > +++ b/arch/mips/kernel/setup.c
> > @@ -392,8 +392,10 @@ static int __init early_parse_mem(char *p)
> >        */
> >       if (usermem == 0) {
> >               usermem = 1;
> > +#ifndef CONFIG_CRASH_DUMP
> >               memblock_remove(memblock_start_of_DRAM(),
> >                       memblock_end_of_DRAM() - memblock_start_of_DRAM());
> > +#endif
> >       }
> >       start = 0;
> >       size = memparse(p, &p);
> > --
> > 2.7.0
> >
>

Re: [PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Baoquan He]

On 09/23/20 at 10:58am, Huacai Chen wrote:
> Hi, Baoquan,
> 
> On Wed, Sep 23, 2020 at 10:46 AM Baoquan He <bhe@redhat.com> wrote:
> >
> > On 09/23/20 at 10:30am, Huacai Chen wrote:
> > > Kexec-tools use mem=X@Y to pass usable memories to crash kernel, but in
> > > commit a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map") all
> > > BIOS passed memories are removed by early_parse_mem(). I think this is
> > > reasonable for a normal kernel but not for a crash kernel, because a
> > > crash kernel should be able to see all old memories, even though it is
> > > not supposed to use them.
> >
> > I am not familiar with MIPS code, but we analyze and fill memmap= to
> > pass usable memory to crashkenrel in kexec-tools, do you mean you are
> > specifying memmap= or mem= by hand?
> Not by hand, but by code of kexec-tools via the "mem=" parameter.

OK, please ignore my comments.

> 
> As I know, kexec-tools of MIPS only use "mem=" to pass "usable"
> memory, but not "visible" memory. "Visible" memory of the crash kernel
> is still passed by BIOS (strictly, by the old kernel who duplicates
> information from BIOS). If memblock_remove() executed here, it would
> remove all "visible" memory and make "visible" memory the same as
> "usable" memory, and I think this is not correct.
> 
> >
> > And we don't have mem=X@Y, only mem=nn[KMG].
> The relocatable kernel of MIPS is still unusable now, so MIPS should
> use mem=X@Y, and the crash kernel is always different from normal
> kernel.

Interesting. Seems MIPS does support mem=X@Y, even though the document
of 'mem=' says it's used to specify amount of memory, but not memory
region. Anyway, leave this to mips reviewers, thanks for replying.

~~~~~~~~~~~~~~~~~~~
        mem=nn[KMG]     [KNL,BOOT] Force usage of a specific amount of memory
                        Amount of memory to be used in cases as follows:

> 
> >
> > >
> > > Fixes: a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map")
> > > Signed-off-by: Huacai Chen <chenhc@lemote.com>
> > > ---
> > >  arch/mips/kernel/setup.c | 2 ++
> > >  1 file changed, 2 insertions(+)
> > >
> > > diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> > > index 4c04a86..e2804a2 100644
> > > --- a/arch/mips/kernel/setup.c
> > > +++ b/arch/mips/kernel/setup.c
> > > @@ -392,8 +392,10 @@ static int __init early_parse_mem(char *p)
> > >        */
> > >       if (usermem == 0) {
> > >               usermem = 1;
> > > +#ifndef CONFIG_CRASH_DUMP
> > >               memblock_remove(memblock_start_of_DRAM(),
> > >                       memblock_end_of_DRAM() - memblock_start_of_DRAM());
> > > +#endif
> > >       }
> > >       start = 0;
> > >       size = memparse(p, &p);
> > > --
> > > 2.7.0
> > >
> >
> 

Re: [PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Maciej W. Rozycki]

On Wed, 23 Sep 2020, Baoquan He wrote:

> > > And we don't have mem=X@Y, only mem=nn[KMG].
> > The relocatable kernel of MIPS is still unusable now, so MIPS should
> > use mem=X@Y, and the crash kernel is always different from normal
> > kernel.
> 
> Interesting. Seems MIPS does support mem=X@Y, even though the document
> of 'mem=' says it's used to specify amount of memory, but not memory
> region. Anyway, leave this to mips reviewers, thanks for replying.
> 
> ~~~~~~~~~~~~~~~~~~~
>         mem=nn[KMG]     [KNL,BOOT] Force usage of a specific amount of memory
>                         Amount of memory to be used in cases as follows:

 Yep, I implemented it for the DECstation back in 2000:

commit 97b7ae4257ef7ba8ed9b7944a4f56a49af3e8abb
Author: Ralf Baechle <ralf@linux-mips.org>
Date:   Mon Dec 11 16:41:05 2000 +0000

    Memmap fixes from Maciej.

(from the LMO repo) in line with the x86 syntax.  I don't know why it 
hasn't ever been documented in "Kernel Parameters" (for any port), but I'm 
fairly sure it has been somewhere.

 FWIW,

  Maciej

Re: [PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Maciej W. Rozycki]

On Sat, 3 Oct 2020, Maciej W. Rozycki wrote:

> > Interesting. Seems MIPS does support mem=X@Y, even though the document
> > of 'mem=' says it's used to specify amount of memory, but not memory
> > region. Anyway, leave this to mips reviewers, thanks for replying.
> > 
> > ~~~~~~~~~~~~~~~~~~~
> >         mem=nn[KMG]     [KNL,BOOT] Force usage of a specific amount of memory
> >                         Amount of memory to be used in cases as follows:
> 
>  Yep, I implemented it for the DECstation back in 2000:
> 
> commit 97b7ae4257ef7ba8ed9b7944a4f56a49af3e8abb
> Author: Ralf Baechle <ralf@linux-mips.org>
> Date:   Mon Dec 11 16:41:05 2000 +0000
> 
>     Memmap fixes from Maciej.
> 
> (from the LMO repo) in line with the x86 syntax.  I don't know why it 
> hasn't ever been documented in "Kernel Parameters" (for any port), but I'm 
> fairly sure it has been somewhere.

 Self-correction: documentation used to be there, but was removed around 
Linux 2.5.65:

commit 041a679cb20e4fcb841665b09cf7c6d24ab4ad39
Author: Ralf Baechle <ralf@linux-mips.org>
Date:   Thu Jun 5 00:04:28 2003 +0000

    Merge with Linux 2.5.65.

(same repo) as the parameter was renamed to `memmap=' in the x86 port.  
Obviously whoever did that did not bother to check other ports, even 
though the parameter was marked (and `memmap=' still is) generic.

  Maciej

Re: [PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Jinyang He]

On 09/23/2020 10:30 AM, Huacai Chen wrote:
> Kexec-tools use mem=X@Y to pass usable memories to crash kernel, but in
> commit a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map") all
> BIOS passed memories are removed by early_parse_mem(). I think this is
> reasonable for a normal kernel but not for a crash kernel, because a
> crash kernel should be able to see all old memories, even though it is
> not supposed to use them.
>
> Fixes: a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map")
> Signed-off-by: Huacai Chen <chenhc@lemote.com>
> ---
>   arch/mips/kernel/setup.c | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> index 4c04a86..e2804a2 100644
> --- a/arch/mips/kernel/setup.c
> +++ b/arch/mips/kernel/setup.c
> @@ -392,8 +392,10 @@ static int __init early_parse_mem(char *p)
>   	 */
>   	if (usermem == 0) {
>   		usermem = 1;
> +#ifndef CONFIG_CRASH_DUMP
>   		memblock_remove(memblock_start_of_DRAM(),
>   			memblock_end_of_DRAM() - memblock_start_of_DRAM());
> +#endif

Hi, Huacai,

For this patch, I knew something what had happened. "mem=" parsing
works wrong for Loongson64. You can referenced the follow patch:
https://patchwork.kernel.org/patch/11789555/

memblock_add() calls memblock_add_range(,,, MAX_NUMNODES,)
For Loongson64 enabling NUMA, we need memblock_add_node(). (Or
using memblock_set_node() after memblock_add())

Besides, "mem=" may be useless for kdump. Youling had submitted a patch
about removing "mem="  serveral days ago.

For Loongson64 platform, you can try crashkernel=SIZE@38M after fixed 
"mem=".
38M means 40M - 2M, 2M is needed because old firmware compatibility.

Thanks,
- Jinyang.

>   	}
>   	start = 0;
>   	size = memparse(p, &p);

Re: [PATCH 1/3] MIPS: Crash kernel should be able to see old memories

[Huacai Chen]

Hi, Jinyang,

On Wed, Sep 23, 2020 at 11:02 AM Jinyang He <hejinyang@loongson.cn> wrote:
>
> On 09/23/2020 10:30 AM, Huacai Chen wrote:
> > Kexec-tools use mem=X@Y to pass usable memories to crash kernel, but in
> > commit a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map") all
> > BIOS passed memories are removed by early_parse_mem(). I think this is
> > reasonable for a normal kernel but not for a crash kernel, because a
> > crash kernel should be able to see all old memories, even though it is
> > not supposed to use them.
> >
> > Fixes: a94e4f24ec836c8984f83959 ("MIPS: init: Drop boot_mem_map")
> > Signed-off-by: Huacai Chen <chenhc@lemote.com>
> > ---
> >   arch/mips/kernel/setup.c | 2 ++
> >   1 file changed, 2 insertions(+)
> >
> > diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> > index 4c04a86..e2804a2 100644
> > --- a/arch/mips/kernel/setup.c
> > +++ b/arch/mips/kernel/setup.c
> > @@ -392,8 +392,10 @@ static int __init early_parse_mem(char *p)
> >        */
> >       if (usermem == 0) {
> >               usermem = 1;
> > +#ifndef CONFIG_CRASH_DUMP
> >               memblock_remove(memblock_start_of_DRAM(),
> >                       memblock_end_of_DRAM() - memblock_start_of_DRAM());
> > +#endif
>
> Hi, Huacai,
>
> For this patch, I knew something what had happened. "mem=" parsing
> works wrong for Loongson64. You can referenced the follow patch:
> https://patchwork.kernel.org/patch/11789555/
>
> memblock_add() calls memblock_add_range(,,, MAX_NUMNODES,)
> For Loongson64 enabling NUMA, we need memblock_add_node(). (Or
> using memblock_set_node() after memblock_add())
This seems like another story.

>
> Besides, "mem=" may be useless for kdump. Youling had submitted a patch
> about removing "mem="  serveral days ago.
kexec-tools use "mem=" to pass usable memory for the crash kernel,
Youling removes the base address of "mem=", but "mem=" is still here.

>
> For Loongson64 platform, you can try crashkernel=SIZE@38M after fixed
> "mem=".
> 38M means 40M - 2M, 2M is needed because old firmware compatibility.
"crashkernel=" is for the normal kernel specified by BIOS, and "mem="
is for the crash kernel specified by kexec-tools.

Huacai
>
> Thanks,
> - Jinyang.
>
> >       }
> >       start = 0;
> >       size = memparse(p, &p);
>