[PATCH 0/6] Implement hart hotplug using HSM extension for

[PATCH 0/6] Implement hart hotplug using HSM extension for

[Yu Chien Peter Lin]

The folloing patches include implementing deep sleep for the offline
hart and some fixes, base on OpenSBI v1.2 and test with the CPU listed
below:

RV64:
* AX45MP quad-core
* AX45MP octa-core
* AX25MP quad-core

RV32:
* A45MP quad-core
* A25MP quad-core

Yu Chien Peter Lin (6):
  docs: generic.md: fix typo of andes-ae350
  lib: sbi_hsm: handle failure when hart_stop returns SBI_ENOTSUPP
  include: types: add always inline compiler attribute
  platform: andes/ae350: Implement hart hotplug using HSM extension
  lib: reset/fdt_reset_atcwdt200: use defined macros in atcsmu.h
  lib: sbi_hsm: Introduce hart_secondary_boot() callback

 docs/platform/generic.md                 |   2 +-
 include/sbi/sbi_hsm.h                    |  15 ++-
 include/sbi/sbi_types.h                  |   1 +
 lib/sbi/sbi_hsm.c                        |  29 ++---
 lib/utils/reset/fdt_reset_atcwdt200.c    |  12 +-
 platform/generic/andes/ae350.c           | 138 +++++++++++++++++++++++
 platform/generic/andes/objects.mk        |   2 +-
 platform/generic/andes/sleep.S           |  61 ++++++++++
 platform/generic/include/andes/andes45.h |  10 ++
 platform/generic/include/andes/atcsmu.h  |  44 ++++++++
 10 files changed, 289 insertions(+), 25 deletions(-)
 create mode 100644 platform/generic/andes/sleep.S
 create mode 100644 platform/generic/include/andes/andes45.h
 create mode 100644 platform/generic/include/andes/atcsmu.h

-- 
2.34.1

[PATCH 1/6] docs: generic.md: fix typo of andes-ae350

[Yu Chien Peter Lin]

Fix hyperlink due to the typo.

Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 docs/platform/generic.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/platform/generic.md b/docs/platform/generic.md
index 8e4cf2d..c29eb04 100644
--- a/docs/platform/generic.md
+++ b/docs/platform/generic.md
@@ -53,7 +53,7 @@ RISC-V Platforms Using Generic Platform
 * **Spike** (*[spike.md]*)
 * **T-HEAD C9xx series Processors** (*[thead-c9xx.md]*)
 
-[andes-ae350.md]: andse-ae350.md
+[andes-ae350.md]: andes-ae350.md
 [qemu_virt.md]: qemu_virt.md
 [renesas-rzfive.md]: renesas-rzfive.md
 [shakti_cclass.md]: shakti_cclass.md
-- 
2.34.1

[PATCH 2/6] lib: sbi_hsm: handle failure when hart_stop returns SBI_ENOTSUPP

[Yu Chien Peter Lin]

Make use of generic warm-boot path when platform hart_stop callback
returns SBI_ENOTSUPP, in case certain hart can not turn off its
power domain, or it detects some error occured in power management
unit, it can fall through warmboot flow and wait for interrupt in
sbi_hsm_hart_wait().

Also improves comment in sbi_hsm_hart_wait().

Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
On ae350-ax25mp, hart0 shares power domain with L2-cache, thus turning
it off would break working system.
---
 include/sbi/sbi_hsm.h | 8 ++++++--
 lib/sbi/sbi_hsm.c     | 7 +++----
 2 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/include/sbi/sbi_hsm.h b/include/sbi/sbi_hsm.h
index d6cc468..1e23884 100644
--- a/include/sbi/sbi_hsm.h
+++ b/include/sbi/sbi_hsm.h
@@ -21,8 +21,12 @@ struct sbi_hsm_device {
 	int (*hart_start)(u32 hartid, ulong saddr);
 
 	/**
-	 * Stop (or power-down) the current hart from running. This call
-	 * doesn't expect to return if success.
+	 * Stop (or power-down) the current hart from running.
+	 *
+	 * Return SBI_ENOTSUPP if the hart does not support platform-specific
+	 * stop actions.
+	 *
+	 * For successful stop, the call won't return.
 	 */
 	int (*hart_stop)(void);
 
diff --git a/lib/sbi/sbi_hsm.c b/lib/sbi/sbi_hsm.c
index 836008f..b89253b 100644
--- a/lib/sbi/sbi_hsm.c
+++ b/lib/sbi/sbi_hsm.c
@@ -116,7 +116,7 @@ static void sbi_hsm_hart_wait(struct sbi_scratch *scratch, u32 hartid)
 	/* Set MSIE and MEIE bits to receive IPI */
 	csr_set(CSR_MIE, MIP_MSIP | MIP_MEIP);
 
-	/* Wait for hart_add call*/
+	/* Wait for state transition requested by sbi_hsm_hart_start() */
 	while (atomic_read(&hdata->state) != SBI_HSM_STATE_START_PENDING) {
 		wfi();
 	};
@@ -228,9 +228,8 @@ void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch)
 		goto fail_exit;
 
 	if (hsm_device_has_hart_hotplug()) {
-		hsm_device_hart_stop();
-		/* It should never reach here */
-		goto fail_exit;
+		if (hsm_device_hart_stop() != SBI_ENOTSUPP)
+			goto fail_exit;
 	}
 
 	/**
-- 
2.34.1

[PATCH 3/6] include: types: add always inline compiler attribute

[Yu Chien Peter Lin]

Provide __always_inline to sbi_types header.

Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 include/sbi/sbi_types.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/sbi/sbi_types.h b/include/sbi/sbi_types.h
index 7fb1af7..c48d4d1 100644
--- a/include/sbi/sbi_types.h
+++ b/include/sbi/sbi_types.h
@@ -64,6 +64,7 @@ typedef unsigned long		physical_size_t;
 #define __packed		__attribute__((packed))
 #define __noreturn		__attribute__((noreturn))
 #define __aligned(x)		__attribute__((aligned(x)))
+#define __always_inline	inline __attribute__((always_inline))
 
 #define likely(x) __builtin_expect((x), 1)
 #define unlikely(x) __builtin_expect((x), 0)
-- 
2.34.1

[PATCH 4/6] platform: andes/ae350: Implement hart hotplug using HSM extension

[Yu Chien Peter Lin]

Add hart_start() and hart_stop() callbacks for the multi-core ae350
platform, it utilizes the ATCSMU to put the harts into power-gated
deep sleep mode. The programming sequence is stated as below:

1. Set the wakeup events to PCSm_WE
2. Set the sleep command to PCSm_CTL
3. Set the reset vector to HARTm_RESET_VECTOR_{LO|HI}
4. Write back and invalidate D-cache by executing the CCTL command L1D_WBINVAL_ALL
5. Disable I/D-cache by clearing mcache_ctl.{I|D}C_EN
6. Disable D-cache coherency by clearing mcache_ctl_.DC_COHEN
7. Wait for mcache_ctl.DC_COHSTA to be cleared to ensure the previous step is completed
8. Execute WFI

Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 platform/generic/andes/ae350.c           | 138 +++++++++++++++++++++++
 platform/generic/andes/objects.mk        |   2 +-
 platform/generic/andes/sleep.S           |  61 ++++++++++
 platform/generic/include/andes/andes45.h |  10 ++
 platform/generic/include/andes/atcsmu.h  |  44 ++++++++
 5 files changed, 254 insertions(+), 1 deletion(-)
 create mode 100644 platform/generic/andes/sleep.S
 create mode 100644 platform/generic/include/andes/andes45.h
 create mode 100644 platform/generic/include/andes/atcsmu.h

diff --git a/platform/generic/andes/ae350.c b/platform/generic/andes/ae350.c
index cf7f6f2..62aba50 100644
--- a/platform/generic/andes/ae350.c
+++ b/platform/generic/andes/ae350.c
@@ -10,6 +10,143 @@
 #include <platform_override.h>
 #include <sbi_utils/fdt/fdt_helper.h>
 #include <sbi_utils/fdt/fdt_fixup.h>
+#include <sbi/riscv_io.h>
+#include <sbi/sbi_bitops.h>
+#include <sbi/sbi_console.h>
+#include <sbi/sbi_error.h>
+#include <sbi/sbi_hsm.h>
+#include <sbi/sbi_ipi.h>
+
+#include <andes/atcsmu.h>
+#include <andes/andes45.h>
+
+struct smu_data smu;
+extern void __ae350_enable_coherency_warmboot(void);
+extern void __ae350_disable_coherency(void);
+
+static __always_inline bool is_andes25(void)
+{
+	ulong marchid = csr_read(CSR_MARCHID);
+	return EXTRACT_FIELD(marchid, CSR_MARCHID_MICROID) == 0xa25;
+}
+
+static void smu_set_wakeup_events(u32 events, u32 hartid)
+{
+	writel(events, (void *)(smu.addr + PCSm_WE_OFFSET(hartid)));
+}
+
+static bool smu_support_sleep_mode(u32 sleep_mode, u32 hartid)
+{
+	u32 pcs_cfg;
+
+	pcs_cfg = readl((void *)(smu.addr + PCSm_CFG_OFFSET(hartid)));
+
+	switch (sleep_mode) {
+	case LIGHTSLEEP_MODE:
+		if (EXTRACT_FIELD(pcs_cfg, PCS_CFG_LIGHT_SLEEP) == 0) {
+			sbi_printf(
+				"SMU: hart%d (PCS%d) does not support light sleep mode\n",
+				hartid, hartid + 3);
+			return false;
+		}
+	case DEEPSLEEP_MODE:
+		if (EXTRACT_FIELD(pcs_cfg, PCS_CFG_DEEP_SLEEP) == 0) {
+			sbi_printf(
+				"SMU: hart%d (PCS%d) does not support deep sleep mode\n",
+				hartid, hartid + 3);
+			return false;
+		}
+	};
+
+	return true;
+}
+
+static void smu_set_command(u32 pcs_ctl, u32 hartid)
+{
+	writel(pcs_ctl, (void *)(smu.addr + PCSm_CTL_OFFSET(hartid)));
+}
+
+static void smu_set_reset_vector(ulong wakeup_addr, u32 hartid)
+{
+	writel(wakeup_addr,
+	       (void *)(smu.addr + HARTn_RESET_VEC_LO(hartid)));
+	writel((u64)wakeup_addr >> 32,
+	       (void *)(smu.addr + HARTn_RESET_VEC_HI(hartid)));
+}
+
+static int ae350_hart_start(u32 hartid, ulong saddr)
+{
+	if (is_andes25() && hartid == 0)
+		return sbi_ipi_raw_send(hartid);
+
+	/* Write wakeup command to the sleep hart */
+	smu_set_command(WAKEUP_CMD, hartid);
+
+	return 0;
+}
+
+static int ae350_hart_stop(void)
+{
+	u32 hartid = current_hartid();
+
+	/**
+	 * The hart0 shares power domain with L2-cache,
+	 * instead of turning it off, it falls through
+	 * and jump to warmboot_addr.
+	 */
+	if (is_andes25() && hartid == 0)
+		return SBI_ENOTSUPP;
+
+	if (!smu_support_sleep_mode(DEEPSLEEP_MODE, hartid))
+		return SBI_ENOTSUPP;
+
+	/**
+	 * disable all events, the current hart will be
+	 * woken up from reset vector when other hart
+	 * writes its PCS (power control slot) control
+	 * register
+	 */
+	smu_set_wakeup_events(0x0, hartid);
+	smu_set_command(DEEP_SLEEP_CMD, hartid);
+	smu_set_reset_vector((ulong)__ae350_enable_coherency_warmboot,
+			       hartid);
+	__ae350_disable_coherency();
+
+	wfi();
+
+	/* It should never reach here */
+	sbi_hart_hang();
+	return 0;
+}
+
+static const struct sbi_hsm_device andes_smu = {
+	.name	      = "andes_smu",
+	.hart_start   = ae350_hart_start,
+	.hart_stop    = ae350_hart_stop,
+};
+
+static void ae350_hsm_device_init(void)
+{
+	int rc;
+	void *fdt;
+
+	fdt = fdt_get_address();
+
+	rc = fdt_parse_compat_addr(fdt, (uint64_t *)&smu.addr,
+				   "andestech,atcsmu");
+
+	if (!rc) {
+		sbi_hsm_set_device(&andes_smu);
+	}
+}
+
+static int ae350_final_init(bool cold_boot, const struct fdt_match *match)
+{
+	if (cold_boot)
+		ae350_hsm_device_init();
+
+	return 0;
+}
 
 static const struct fdt_match andes_ae350_match[] = {
 	{ .compatible = "andestech,ae350" },
@@ -18,4 +155,5 @@ static const struct fdt_match andes_ae350_match[] = {
 
 const struct platform_override andes_ae350 = {
 	.match_table = andes_ae350_match,
+	.final_init  = ae350_final_init,
 };
diff --git a/platform/generic/andes/objects.mk b/platform/generic/andes/objects.mk
index dd6408d..28275ef 100644
--- a/platform/generic/andes/objects.mk
+++ b/platform/generic/andes/objects.mk
@@ -3,4 +3,4 @@
 #
 
 carray-platform_override_modules-$(CONFIG_PLATFORM_ANDES_AE350) += andes_ae350
-platform-objs-$(CONFIG_PLATFORM_ANDES_AE350) += andes/ae350.o
+platform-objs-$(CONFIG_PLATFORM_ANDES_AE350) += andes/ae350.o andes/sleep.o
diff --git a/platform/generic/andes/sleep.S b/platform/generic/andes/sleep.S
new file mode 100644
index 0000000..2171f0d
--- /dev/null
+++ b/platform/generic/andes/sleep.S
@@ -0,0 +1,61 @@
+/*
+ * SPDX-License-Identifier: BSD-2-Clause
+ */
+
+#include <sbi/riscv_encoding.h>
+#include <sbi/riscv_asm.h>
+#include <andes/andes45.h>
+
+	.section .text, "ax", %progbits
+	.align 3
+	.global __ae350_disable_coherency
+__ae350_disable_coherency:
+	/* flush D-cache */
+	csrw	CSR_MCCTLCOMMAND, 0x6
+	/* disable I/D-cache */
+	csrc	CSR_MCACHECTL, 0x3
+	/* disable D-cache coherency */
+	lui	t1, 0x80
+	csrc	CSR_MCACHECTL, t1
+	/* 45-series: wait for mcache_ctl.DC_COHSTA to be cleared */
+check_cm_disabled:
+	csrr	t1, CSR_MCACHECTL
+	srli	t1, t1, 20
+	andi	t1, t1, 0x1
+	bnez	t1, check_cm_disabled
+
+	ret
+
+	.section .text, "ax", %progbits
+	.align 3
+	.global __ae350_enable_coherency
+__ae350_enable_coherency:
+	/* enable D-cache coherency */
+	lui		t1, 0x80
+	csrs	CSR_MCACHECTL, t1
+	/*
+	 * check CM support
+	 * 25-series: mcache_ctl.DC_COHEN is hard-wired to 0
+	 */
+	csrr	t1, CSR_MCACHECTL
+	srli	t1, t1, 20
+	andi	t1, t1, 0x1
+	beqz	t1, enable_L1_cache
+	/* 45-series: wait for mcache_ctl.DC_COHSTA to be set */
+check_cm_enabled:
+	csrr	t1, CSR_MCACHECTL
+	srli	t1, t1, 20
+	andi	t1, t1, 0x1
+	beqz	t1, check_cm_enabled
+enable_L1_cache:
+	/* enable I/D-cache */
+	csrs	CSR_MCACHECTL, 0x3
+
+	ret
+
+	.section .text, "ax", %progbits
+	.align 3
+	.global __ae350_enable_coherency_warmboot
+__ae350_enable_coherency_warmboot:
+	call ra, __ae350_enable_coherency
+	j _start_warm
diff --git a/platform/generic/include/andes/andes45.h b/platform/generic/include/andes/andes45.h
new file mode 100644
index 0000000..aea2368
--- /dev/null
+++ b/platform/generic/include/andes/andes45.h
@@ -0,0 +1,10 @@
+#ifndef _RISCV_ANDES45_H
+#define _RISCV_ANDES45_H
+
+#define CSR_MARCHID_MICROID 0xfff
+
+/* Memory and Miscellaneous Registers */
+#define CSR_MCACHECTL 0x7ca
+#define CSR_MCCTLCOMMAND 0x7cc
+
+#endif /* _RISCV_ANDES45_H */
diff --git a/platform/generic/include/andes/atcsmu.h b/platform/generic/include/andes/atcsmu.h
new file mode 100644
index 0000000..c72600c
--- /dev/null
+++ b/platform/generic/include/andes/atcsmu.h
@@ -0,0 +1,44 @@
+#ifndef _RISCV_ATCSMU_H
+#define _RISCV_ATCSMU_H
+
+#define PCS0_WE_OFFSET 0x90
+#define PCSm_WE_OFFSET(i) ((i + 3) * 0x20 + PCS0_WE_OFFSET)
+
+#define PCS0_CTL_OFFSET 0x94
+#define PCSm_CTL_OFFSET(i) ((i + 3) * 0x20 + PCS0_CTL_OFFSET)
+#define PCS_CTL_CMD_SHIFT 0
+#define PCS_CTL_PARAM_SHIFT 3
+#define SLEEP_CMD 0x3
+#define WAKEUP_CMD (0x0 | (1 << PCS_CTL_PARAM_SHIFT))
+#define LIGHTSLEEP_MODE 0
+#define DEEPSLEEP_MODE 1
+#define LIGHT_SLEEP_CMD (SLEEP_CMD | (LIGHTSLEEP_MODE << PCS_CTL_PARAM_SHIFT))
+#define DEEP_SLEEP_CMD (SLEEP_CMD | (DEEPSLEEP_MODE << PCS_CTL_PARAM_SHIFT))
+
+#define PCS0_CFG_OFFSET 0x80
+#define PCSm_CFG_OFFSET(i) ((i + 3) * 0x20 + PCS0_CFG_OFFSET)
+#define PCS_CFG_LIGHT_SLEEP_SHIFT 2
+#define PCS_CFG_LIGHT_SLEEP (1 << PCS_CFG_LIGHT_SLEEP_SHIFT)
+#define PCS_CFG_DEEP_SLEEP_SHIFT 3
+#define PCS_CFG_DEEP_SLEEP (1 << PCS_CFG_DEEP_SLEEP_SHIFT)
+#define RESET_VEC_LO_OFFSET 0x50
+#define RESET_VEC_HI_OFFSET 0x60
+#define RESET_VEC_8CORE_OFFSET 0x1a0
+#define HARTn_RESET_VEC_LO(n)  \
+	(RESET_VEC_LO_OFFSET + \
+	 ((n) < 4 ? 0 : RESET_VEC_8CORE_OFFSET) + ((n) * 0x4))
+#define HARTn_RESET_VEC_HI(n)  \
+	(RESET_VEC_HI_OFFSET + \
+	 ((n) < 4 ? 0 : RESET_VEC_8CORE_OFFSET) + ((n) * 0x4))
+#define PCS_MAX_NR 8
+#define FLASH_BASE 0x80000000ULL
+
+#ifndef __ASSEMBLER__
+
+struct smu_data {
+	unsigned long addr;
+};
+
+#endif /* __ASSEMBLER__ */
+
+#endif /* _RISCV_ATCSMU_H */
-- 
2.34.1

[PATCH 5/6] lib: reset/fdt_reset_atcwdt200: use defined macros in atcsmu.h

[Yu Chien Peter Lin]

Reuse the smu related macros defined in atcsmu.h.

Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 lib/utils/reset/fdt_reset_atcwdt200.c | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/lib/utils/reset/fdt_reset_atcwdt200.c b/lib/utils/reset/fdt_reset_atcwdt200.c
index 91acc9f..a6e6834 100644
--- a/lib/utils/reset/fdt_reset_atcwdt200.c
+++ b/lib/utils/reset/fdt_reset_atcwdt200.c
@@ -17,6 +17,8 @@
 #include <sbi_utils/fdt/fdt_helper.h>
 #include <sbi_utils/reset/fdt_reset.h>
 
+#include <andes/atcsmu.h>
+
 #define ATCWDT200_WP_NUM 0x5aa5
 #define WREN_REG 0x18
 #define CTRL_REG 0x10
@@ -41,12 +43,6 @@
 #define CLK_PCLK (1 << 1)
 #define WDT_EN (1 << 0)
 
-#define FLASH_BASE 0x80000000ULL
-#define SMU_RESET_VEC_LO_OFF 0x50
-#define SMU_RESET_VEC_HI_OFF 0x60
-#define SMU_HARTn_RESET_VEC_LO(n) (SMU_RESET_VEC_LO_OFF + (n * 0x4))
-#define SMU_HARTn_RESET_VEC_HI(n) (SMU_RESET_VEC_HI_OFF + (n * 0x4))
-
 static volatile char *wdt_addr;
 static volatile char *smu_addr;
 
@@ -67,8 +63,8 @@ static void ae350_system_reset(u32 type, u32 reason)
 	const struct sbi_platform *plat = sbi_platform_thishart_ptr();
 
 	for (int i = 0; i < sbi_platform_hart_count(plat); i++) {
-		writel(FLASH_BASE, smu_addr + SMU_HARTn_RESET_VEC_LO(i));
-		writel(FLASH_BASE >> 32, smu_addr + SMU_HARTn_RESET_VEC_HI(i));
+		writel(FLASH_BASE, smu_addr + HARTn_RESET_VEC_LO(i));
+		writel(FLASH_BASE >> 32, smu_addr + HARTn_RESET_VEC_HI(i));
 	}
 
 	/* Program WDT control register  */
-- 
2.34.1

[PATCH 6/6] lib: sbi_hsm: Introduce hart_secondary_boot() callback

[Yu Chien Peter Lin]

When platform supports hotplug, i.e. both hart_start() and hart_stop()
callbacks are provided, the former doesn't need to be performed at
the boot-time. Thus, add a callback for the case of secondary boot.

Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 include/sbi/sbi_hsm.h |  7 +++++++
 lib/sbi/sbi_hsm.c     | 22 +++++++++++++---------
 2 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/include/sbi/sbi_hsm.h b/include/sbi/sbi_hsm.h
index 1e23884..94b3bbb 100644
--- a/include/sbi/sbi_hsm.h
+++ b/include/sbi/sbi_hsm.h
@@ -49,6 +49,13 @@ struct sbi_hsm_device {
 	 * non-retentive suspend.
 	 */
 	void (*hart_resume)(void);
+
+	/**
+	 * Perform platform-specific actions on non-boot harts at boot-time
+	 *
+	 * For successful secondary boot, the call will return 0.
+	 */
+	int (*hart_secondary_boot)(u32 hartid, ulong saddr);
 };
 
 struct sbi_domain;
diff --git a/lib/sbi/sbi_hsm.c b/lib/sbi/sbi_hsm.c
index b89253b..c9ab6a3 100644
--- a/lib/sbi/sbi_hsm.c
+++ b/lib/sbi/sbi_hsm.c
@@ -152,11 +152,18 @@ static bool hsm_device_has_hart_hotplug(void)
 
 static bool hsm_device_has_hart_secondary_boot(void)
 {
-	if (hsm_dev && hsm_dev->hart_start && !hsm_dev->hart_stop)
+	if (hsm_dev && hsm_dev->hart_secondary_boot)
 		return true;
 	return false;
 }
 
+static int hsm_device_hart_secondary_boot(u32 hartid, ulong saddr)
+{
+	if (hsm_dev && hsm_dev->hart_secondary_boot)
+		return hsm_dev->hart_secondary_boot(hartid, saddr);
+	return SBI_ENOTSUPP;
+}
+
 static int hsm_device_hart_start(u32 hartid, ulong saddr)
 {
 	if (hsm_dev && hsm_dev->hart_start)
@@ -284,16 +291,13 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch,
 	rscratch->next_addr = saddr;
 	rscratch->next_mode = smode;
 
-	if (hsm_device_has_hart_hotplug() ||
-	   (hsm_device_has_hart_secondary_boot() && !init_count)) {
+	if (hsm_device_has_hart_secondary_boot() && !init_count)
+		return hsm_device_hart_secondary_boot(hartid, scratch->warmboot_addr);
+
+	if (hsm_device_has_hart_hotplug() && init_count)
 		return hsm_device_hart_start(hartid, scratch->warmboot_addr);
-	} else {
-		int rc = sbi_ipi_raw_send(hartid);
-		if (rc)
-		    return rc;
-	}
 
-	return 0;
+	return sbi_ipi_raw_send(hartid);
 }
 
 int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow)
-- 
2.34.1

[PATCH 2/6] lib: sbi_hsm: handle failure when hart_stop returns SBI_ENOTSUPP

[Andrew Jones]

On Wed, Jan 04, 2023 at 02:29:23PM +0800, Yu Chien Peter Lin wrote:
> Make use of generic warm-boot path when platform hart_stop callback
> returns SBI_ENOTSUPP, in case certain hart can not turn off its
> power domain, or it detects some error occured in power management
> unit, it can fall through warmboot flow and wait for interrupt in
> sbi_hsm_hart_wait().
> 
> Also improves comment in sbi_hsm_hart_wait().
> 
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> ---
> On ae350-ax25mp, hart0 shares power domain with L2-cache, thus turning
> it off would break working system.
> ---
>  include/sbi/sbi_hsm.h | 8 ++++++--
>  lib/sbi/sbi_hsm.c     | 7 +++----
>  2 files changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/include/sbi/sbi_hsm.h b/include/sbi/sbi_hsm.h
> index d6cc468..1e23884 100644
> --- a/include/sbi/sbi_hsm.h
> +++ b/include/sbi/sbi_hsm.h
> @@ -21,8 +21,12 @@ struct sbi_hsm_device {
>  	int (*hart_start)(u32 hartid, ulong saddr);
>  
>  	/**
> -	 * Stop (or power-down) the current hart from running. This call
> -	 * doesn't expect to return if success.
> +	 * Stop (or power-down) the current hart from running.
> +	 *
> +	 * Return SBI_ENOTSUPP if the hart does not support platform-specific
> +	 * stop actions.
> +	 *
> +	 * For successful stop, the call won't return.
>  	 */
>  	int (*hart_stop)(void);
>  
> diff --git a/lib/sbi/sbi_hsm.c b/lib/sbi/sbi_hsm.c
> index 836008f..b89253b 100644
> --- a/lib/sbi/sbi_hsm.c
> +++ b/lib/sbi/sbi_hsm.c
> @@ -116,7 +116,7 @@ static void sbi_hsm_hart_wait(struct sbi_scratch *scratch, u32 hartid)
>  	/* Set MSIE and MEIE bits to receive IPI */
>  	csr_set(CSR_MIE, MIP_MSIP | MIP_MEIP);
>  
> -	/* Wait for hart_add call*/
> +	/* Wait for state transition requested by sbi_hsm_hart_start() */
>  	while (atomic_read(&hdata->state) != SBI_HSM_STATE_START_PENDING) {
>  		wfi();
>  	};
> @@ -228,9 +228,8 @@ void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch)
>  		goto fail_exit;
>  
>  	if (hsm_device_has_hart_hotplug()) {
> -		hsm_device_hart_stop();
> -		/* It should never reach here */
> -		goto fail_exit;
> +		if (hsm_device_hart_stop() != SBI_ENOTSUPP)
> +			goto fail_exit;
>  	}
>  
>  	/**
> -- 
> 2.34.1
>

Reviewed-by: Andrew Jones <ajones@ventanamicro.com>

[PATCH 6/6] lib: sbi_hsm: Introduce hart_secondary_boot() callback

[Andrew Jones]

On Wed, Jan 04, 2023 at 02:29:27PM +0800, Yu Chien Peter Lin wrote:
> When platform supports hotplug, i.e. both hart_start() and hart_stop()
> callbacks are provided, the former doesn't need to be performed at
> the boot-time. Thus, add a callback for the case of secondary boot.
> 
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> ---
>  include/sbi/sbi_hsm.h |  7 +++++++
>  lib/sbi/sbi_hsm.c     | 22 +++++++++++++---------
>  2 files changed, 20 insertions(+), 9 deletions(-)

I'm not sure why this patch is in the series as the change isn't used.

Thanks,
drew

> 
> diff --git a/include/sbi/sbi_hsm.h b/include/sbi/sbi_hsm.h
> index 1e23884..94b3bbb 100644
> --- a/include/sbi/sbi_hsm.h
> +++ b/include/sbi/sbi_hsm.h
> @@ -49,6 +49,13 @@ struct sbi_hsm_device {
>  	 * non-retentive suspend.
>  	 */
>  	void (*hart_resume)(void);
> +
> +	/**
> +	 * Perform platform-specific actions on non-boot harts at boot-time
> +	 *
> +	 * For successful secondary boot, the call will return 0.
> +	 */
> +	int (*hart_secondary_boot)(u32 hartid, ulong saddr);
>  };
>  
>  struct sbi_domain;
> diff --git a/lib/sbi/sbi_hsm.c b/lib/sbi/sbi_hsm.c
> index b89253b..c9ab6a3 100644
> --- a/lib/sbi/sbi_hsm.c
> +++ b/lib/sbi/sbi_hsm.c
> @@ -152,11 +152,18 @@ static bool hsm_device_has_hart_hotplug(void)
>  
>  static bool hsm_device_has_hart_secondary_boot(void)
>  {
> -	if (hsm_dev && hsm_dev->hart_start && !hsm_dev->hart_stop)
> +	if (hsm_dev && hsm_dev->hart_secondary_boot)
>  		return true;
>  	return false;
>  }
>  
> +static int hsm_device_hart_secondary_boot(u32 hartid, ulong saddr)
> +{
> +	if (hsm_dev && hsm_dev->hart_secondary_boot)
> +		return hsm_dev->hart_secondary_boot(hartid, saddr);
> +	return SBI_ENOTSUPP;
> +}
> +
>  static int hsm_device_hart_start(u32 hartid, ulong saddr)
>  {
>  	if (hsm_dev && hsm_dev->hart_start)
> @@ -284,16 +291,13 @@ int sbi_hsm_hart_start(struct sbi_scratch *scratch,
>  	rscratch->next_addr = saddr;
>  	rscratch->next_mode = smode;
>  
> -	if (hsm_device_has_hart_hotplug() ||
> -	   (hsm_device_has_hart_secondary_boot() && !init_count)) {
> +	if (hsm_device_has_hart_secondary_boot() && !init_count)
> +		return hsm_device_hart_secondary_boot(hartid, scratch->warmboot_addr);
> +
> +	if (hsm_device_has_hart_hotplug() && init_count)
>  		return hsm_device_hart_start(hartid, scratch->warmboot_addr);
> -	} else {
> -		int rc = sbi_ipi_raw_send(hartid);
> -		if (rc)
> -		    return rc;
> -	}
>  
> -	return 0;
> +	return sbi_ipi_raw_send(hartid);
>  }
>  
>  int sbi_hsm_hart_stop(struct sbi_scratch *scratch, bool exitnow)
> -- 
> 2.34.1
> 
> 
> -- 
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH 6/6] lib: sbi_hsm: Introduce hart_secondary_boot() callback

[Yu-Chien Peter Lin]

Hi Andrew,

On Fri, Jan 06, 2023 at 03:29:46PM +0100, Andrew Jones wrote:
> On Wed, Jan 04, 2023 at 02:29:27PM +0800, Yu Chien Peter Lin wrote:
> > When platform supports hotplug, i.e. both hart_start() and hart_stop()
> > callbacks are provided, the former doesn't need to be performed at
> > the boot-time. Thus, add a callback for the case of secondary boot.
> > 
> > Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> > ---
> >  include/sbi/sbi_hsm.h |  7 +++++++
> >  lib/sbi/sbi_hsm.c     | 22 +++++++++++++---------
> >  2 files changed, 20 insertions(+), 9 deletions(-)
> 
> I'm not sure why this patch is in the series as the change isn't used.
> 
> Thanks,
> drew

Thanks for the review.

I'm just found it's not necessary to run platform-specific hart_start()
at boot-time since it's for CPU hotplug, I'm not sure hart_start() is
supposed to do at boot-time so we can ignore this patch if it's not worth
adding a callback.

Best regards,
Peter Lin

[PATCH 2/6] lib: sbi_hsm: handle failure when hart_stop returns SBI_ENOTSUPP

[Atish Patra]

On Tue, Jan 3, 2023 at 10:30 PM Yu Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Make use of generic warm-boot path when platform hart_stop callback
> returns SBI_ENOTSUPP, in case certain hart can not turn off its
> power domain, or it detects some error occured in power management
> unit, it can fall through warmboot flow and wait for interrupt in
> sbi_hsm_hart_wait().
>
> Also improves comment in sbi_hsm_hart_wait().
>
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> ---
> On ae350-ax25mp, hart0 shares power domain with L2-cache, thus turning
> it off would break working system.
> ---
>  include/sbi/sbi_hsm.h | 8 ++++++--
>  lib/sbi/sbi_hsm.c     | 7 +++----
>  2 files changed, 9 insertions(+), 6 deletions(-)
>
> diff --git a/include/sbi/sbi_hsm.h b/include/sbi/sbi_hsm.h
> index d6cc468..1e23884 100644
> --- a/include/sbi/sbi_hsm.h
> +++ b/include/sbi/sbi_hsm.h
> @@ -21,8 +21,12 @@ struct sbi_hsm_device {
>         int (*hart_start)(u32 hartid, ulong saddr);
>
>         /**
> -        * Stop (or power-down) the current hart from running. This call
> -        * doesn't expect to return if success.
> +        * Stop (or power-down) the current hart from running.
> +        *
> +        * Return SBI_ENOTSUPP if the hart does not support platform-specific
> +        * stop actions.
> +        *
> +        * For successful stop, the call won't return.
>          */
>         int (*hart_stop)(void);
>
> diff --git a/lib/sbi/sbi_hsm.c b/lib/sbi/sbi_hsm.c
> index 836008f..b89253b 100644
> --- a/lib/sbi/sbi_hsm.c
> +++ b/lib/sbi/sbi_hsm.c
> @@ -116,7 +116,7 @@ static void sbi_hsm_hart_wait(struct sbi_scratch *scratch, u32 hartid)
>         /* Set MSIE and MEIE bits to receive IPI */
>         csr_set(CSR_MIE, MIP_MSIP | MIP_MEIP);
>
> -       /* Wait for hart_add call*/
> +       /* Wait for state transition requested by sbi_hsm_hart_start() */
>         while (atomic_read(&hdata->state) != SBI_HSM_STATE_START_PENDING) {
>                 wfi();
>         };
> @@ -228,9 +228,8 @@ void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch)
>                 goto fail_exit;
>
>         if (hsm_device_has_hart_hotplug()) {
> -               hsm_device_hart_stop();
> -               /* It should never reach here */
> -               goto fail_exit;
> +               if (hsm_device_hart_stop() != SBI_ENOTSUPP)
> +                       goto fail_exit;
>         }
>
>         /**
> --
> 2.34.1
>
>
> --
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi


Reviewed-by: Atish Patra <atishp@rivosinc.com>

-- 
Regards,
Atish

[PATCH 1/6] docs: generic.md: fix typo of andes-ae350

[Anup Patel]

On Wed, Jan 4, 2023 at 12:00 PM Yu Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Fix hyperlink due to the typo.
>
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

> ---
>  docs/platform/generic.md | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/docs/platform/generic.md b/docs/platform/generic.md
> index 8e4cf2d..c29eb04 100644
> --- a/docs/platform/generic.md
> +++ b/docs/platform/generic.md
> @@ -53,7 +53,7 @@ RISC-V Platforms Using Generic Platform
>  * **Spike** (*[spike.md]*)
>  * **T-HEAD C9xx series Processors** (*[thead-c9xx.md]*)
>
> -[andes-ae350.md]: andse-ae350.md
> +[andes-ae350.md]: andes-ae350.md
>  [qemu_virt.md]: qemu_virt.md
>  [renesas-rzfive.md]: renesas-rzfive.md
>  [shakti_cclass.md]: shakti_cclass.md
> --
> 2.34.1
>
>
> --
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH 2/6] lib: sbi_hsm: handle failure when hart_stop returns SBI_ENOTSUPP

[Anup Patel]

On Wed, Jan 4, 2023 at 12:00 PM Yu Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Make use of generic warm-boot path when platform hart_stop callback
> returns SBI_ENOTSUPP, in case certain hart can not turn off its
> power domain, or it detects some error occured in power management
> unit, it can fall through warmboot flow and wait for interrupt in
> sbi_hsm_hart_wait().
>
> Also improves comment in sbi_hsm_hart_wait().
>
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

> ---
> On ae350-ax25mp, hart0 shares power domain with L2-cache, thus turning
> it off would break working system.
> ---
>  include/sbi/sbi_hsm.h | 8 ++++++--
>  lib/sbi/sbi_hsm.c     | 7 +++----
>  2 files changed, 9 insertions(+), 6 deletions(-)
>
> diff --git a/include/sbi/sbi_hsm.h b/include/sbi/sbi_hsm.h
> index d6cc468..1e23884 100644
> --- a/include/sbi/sbi_hsm.h
> +++ b/include/sbi/sbi_hsm.h
> @@ -21,8 +21,12 @@ struct sbi_hsm_device {
>         int (*hart_start)(u32 hartid, ulong saddr);
>
>         /**
> -        * Stop (or power-down) the current hart from running. This call
> -        * doesn't expect to return if success.
> +        * Stop (or power-down) the current hart from running.
> +        *
> +        * Return SBI_ENOTSUPP if the hart does not support platform-specific
> +        * stop actions.
> +        *
> +        * For successful stop, the call won't return.
>          */
>         int (*hart_stop)(void);
>
> diff --git a/lib/sbi/sbi_hsm.c b/lib/sbi/sbi_hsm.c
> index 836008f..b89253b 100644
> --- a/lib/sbi/sbi_hsm.c
> +++ b/lib/sbi/sbi_hsm.c
> @@ -116,7 +116,7 @@ static void sbi_hsm_hart_wait(struct sbi_scratch *scratch, u32 hartid)
>         /* Set MSIE and MEIE bits to receive IPI */
>         csr_set(CSR_MIE, MIP_MSIP | MIP_MEIP);
>
> -       /* Wait for hart_add call*/
> +       /* Wait for state transition requested by sbi_hsm_hart_start() */
>         while (atomic_read(&hdata->state) != SBI_HSM_STATE_START_PENDING) {
>                 wfi();
>         };
> @@ -228,9 +228,8 @@ void __noreturn sbi_hsm_exit(struct sbi_scratch *scratch)
>                 goto fail_exit;
>
>         if (hsm_device_has_hart_hotplug()) {
> -               hsm_device_hart_stop();
> -               /* It should never reach here */
> -               goto fail_exit;
> +               if (hsm_device_hart_stop() != SBI_ENOTSUPP)
> +                       goto fail_exit;
>         }
>
>         /**
> --
> 2.34.1
>
>
> --
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH 3/6] include: types: add always inline compiler attribute

[Anup Patel]

On Wed, Jan 4, 2023 at 12:02 PM Yu Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Provide __always_inline to sbi_types header.
>
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

> ---
>  include/sbi/sbi_types.h | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/include/sbi/sbi_types.h b/include/sbi/sbi_types.h
> index 7fb1af7..c48d4d1 100644
> --- a/include/sbi/sbi_types.h
> +++ b/include/sbi/sbi_types.h
> @@ -64,6 +64,7 @@ typedef unsigned long         physical_size_t;
>  #define __packed               __attribute__((packed))
>  #define __noreturn             __attribute__((noreturn))
>  #define __aligned(x)           __attribute__((aligned(x)))
> +#define __always_inline        inline __attribute__((always_inline))
>
>  #define likely(x) __builtin_expect((x), 1)
>  #define unlikely(x) __builtin_expect((x), 0)
> --
> 2.34.1
>
>
> --
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH 4/6] platform: andes/ae350: Implement hart hotplug using HSM extension

[Anup Patel]

On Wed, Jan 4, 2023 at 12:02 PM Yu Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Add hart_start() and hart_stop() callbacks for the multi-core ae350
> platform, it utilizes the ATCSMU to put the harts into power-gated
> deep sleep mode. The programming sequence is stated as below:
>
> 1. Set the wakeup events to PCSm_WE
> 2. Set the sleep command to PCSm_CTL
> 3. Set the reset vector to HARTm_RESET_VECTOR_{LO|HI}
> 4. Write back and invalidate D-cache by executing the CCTL command L1D_WBINVAL_ALL
> 5. Disable I/D-cache by clearing mcache_ctl.{I|D}C_EN
> 6. Disable D-cache coherency by clearing mcache_ctl_.DC_COHEN
> 7. Wait for mcache_ctl.DC_COHSTA to be cleared to ensure the previous step is completed
> 8. Execute WFI
>
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

> ---
>  platform/generic/andes/ae350.c           | 138 +++++++++++++++++++++++
>  platform/generic/andes/objects.mk        |   2 +-
>  platform/generic/andes/sleep.S           |  61 ++++++++++
>  platform/generic/include/andes/andes45.h |  10 ++
>  platform/generic/include/andes/atcsmu.h  |  44 ++++++++
>  5 files changed, 254 insertions(+), 1 deletion(-)
>  create mode 100644 platform/generic/andes/sleep.S
>  create mode 100644 platform/generic/include/andes/andes45.h
>  create mode 100644 platform/generic/include/andes/atcsmu.h
>
> diff --git a/platform/generic/andes/ae350.c b/platform/generic/andes/ae350.c
> index cf7f6f2..62aba50 100644
> --- a/platform/generic/andes/ae350.c
> +++ b/platform/generic/andes/ae350.c
> @@ -10,6 +10,143 @@
>  #include <platform_override.h>
>  #include <sbi_utils/fdt/fdt_helper.h>
>  #include <sbi_utils/fdt/fdt_fixup.h>
> +#include <sbi/riscv_io.h>
> +#include <sbi/sbi_bitops.h>
> +#include <sbi/sbi_console.h>
> +#include <sbi/sbi_error.h>
> +#include <sbi/sbi_hsm.h>
> +#include <sbi/sbi_ipi.h>
> +
> +#include <andes/atcsmu.h>
> +#include <andes/andes45.h>
> +
> +struct smu_data smu;
> +extern void __ae350_enable_coherency_warmboot(void);
> +extern void __ae350_disable_coherency(void);
> +
> +static __always_inline bool is_andes25(void)
> +{
> +       ulong marchid = csr_read(CSR_MARCHID);
> +       return EXTRACT_FIELD(marchid, CSR_MARCHID_MICROID) == 0xa25;
> +}
> +
> +static void smu_set_wakeup_events(u32 events, u32 hartid)
> +{
> +       writel(events, (void *)(smu.addr + PCSm_WE_OFFSET(hartid)));
> +}
> +
> +static bool smu_support_sleep_mode(u32 sleep_mode, u32 hartid)
> +{
> +       u32 pcs_cfg;
> +
> +       pcs_cfg = readl((void *)(smu.addr + PCSm_CFG_OFFSET(hartid)));
> +
> +       switch (sleep_mode) {
> +       case LIGHTSLEEP_MODE:
> +               if (EXTRACT_FIELD(pcs_cfg, PCS_CFG_LIGHT_SLEEP) == 0) {
> +                       sbi_printf(
> +                               "SMU: hart%d (PCS%d) does not support light sleep mode\n",
> +                               hartid, hartid + 3);
> +                       return false;
> +               }
> +       case DEEPSLEEP_MODE:
> +               if (EXTRACT_FIELD(pcs_cfg, PCS_CFG_DEEP_SLEEP) == 0) {
> +                       sbi_printf(
> +                               "SMU: hart%d (PCS%d) does not support deep sleep mode\n",
> +                               hartid, hartid + 3);
> +                       return false;
> +               }
> +       };
> +
> +       return true;
> +}
> +
> +static void smu_set_command(u32 pcs_ctl, u32 hartid)
> +{
> +       writel(pcs_ctl, (void *)(smu.addr + PCSm_CTL_OFFSET(hartid)));
> +}
> +
> +static void smu_set_reset_vector(ulong wakeup_addr, u32 hartid)
> +{
> +       writel(wakeup_addr,
> +              (void *)(smu.addr + HARTn_RESET_VEC_LO(hartid)));
> +       writel((u64)wakeup_addr >> 32,
> +              (void *)(smu.addr + HARTn_RESET_VEC_HI(hartid)));
> +}
> +
> +static int ae350_hart_start(u32 hartid, ulong saddr)
> +{
> +       if (is_andes25() && hartid == 0)
> +               return sbi_ipi_raw_send(hartid);
> +
> +       /* Write wakeup command to the sleep hart */
> +       smu_set_command(WAKEUP_CMD, hartid);
> +
> +       return 0;
> +}
> +
> +static int ae350_hart_stop(void)
> +{
> +       u32 hartid = current_hartid();
> +
> +       /**
> +        * The hart0 shares power domain with L2-cache,
> +        * instead of turning it off, it falls through
> +        * and jump to warmboot_addr.
> +        */
> +       if (is_andes25() && hartid == 0)
> +               return SBI_ENOTSUPP;
> +
> +       if (!smu_support_sleep_mode(DEEPSLEEP_MODE, hartid))
> +               return SBI_ENOTSUPP;
> +
> +       /**
> +        * disable all events, the current hart will be
> +        * woken up from reset vector when other hart
> +        * writes its PCS (power control slot) control
> +        * register
> +        */
> +       smu_set_wakeup_events(0x0, hartid);
> +       smu_set_command(DEEP_SLEEP_CMD, hartid);
> +       smu_set_reset_vector((ulong)__ae350_enable_coherency_warmboot,
> +                              hartid);
> +       __ae350_disable_coherency();
> +
> +       wfi();
> +
> +       /* It should never reach here */
> +       sbi_hart_hang();
> +       return 0;
> +}
> +
> +static const struct sbi_hsm_device andes_smu = {
> +       .name         = "andes_smu",
> +       .hart_start   = ae350_hart_start,
> +       .hart_stop    = ae350_hart_stop,
> +};
> +
> +static void ae350_hsm_device_init(void)
> +{
> +       int rc;
> +       void *fdt;
> +
> +       fdt = fdt_get_address();
> +
> +       rc = fdt_parse_compat_addr(fdt, (uint64_t *)&smu.addr,
> +                                  "andestech,atcsmu");
> +
> +       if (!rc) {
> +               sbi_hsm_set_device(&andes_smu);
> +       }
> +}
> +
> +static int ae350_final_init(bool cold_boot, const struct fdt_match *match)
> +{
> +       if (cold_boot)
> +               ae350_hsm_device_init();
> +
> +       return 0;
> +}
>
>  static const struct fdt_match andes_ae350_match[] = {
>         { .compatible = "andestech,ae350" },
> @@ -18,4 +155,5 @@ static const struct fdt_match andes_ae350_match[] = {
>
>  const struct platform_override andes_ae350 = {
>         .match_table = andes_ae350_match,
> +       .final_init  = ae350_final_init,
>  };
> diff --git a/platform/generic/andes/objects.mk b/platform/generic/andes/objects.mk
> index dd6408d..28275ef 100644
> --- a/platform/generic/andes/objects.mk
> +++ b/platform/generic/andes/objects.mk
> @@ -3,4 +3,4 @@
>  #
>
>  carray-platform_override_modules-$(CONFIG_PLATFORM_ANDES_AE350) += andes_ae350
> -platform-objs-$(CONFIG_PLATFORM_ANDES_AE350) += andes/ae350.o
> +platform-objs-$(CONFIG_PLATFORM_ANDES_AE350) += andes/ae350.o andes/sleep.o
> diff --git a/platform/generic/andes/sleep.S b/platform/generic/andes/sleep.S
> new file mode 100644
> index 0000000..2171f0d
> --- /dev/null
> +++ b/platform/generic/andes/sleep.S
> @@ -0,0 +1,61 @@
> +/*
> + * SPDX-License-Identifier: BSD-2-Clause
> + */
> +
> +#include <sbi/riscv_encoding.h>
> +#include <sbi/riscv_asm.h>
> +#include <andes/andes45.h>
> +
> +       .section .text, "ax", %progbits
> +       .align 3
> +       .global __ae350_disable_coherency
> +__ae350_disable_coherency:
> +       /* flush D-cache */
> +       csrw    CSR_MCCTLCOMMAND, 0x6
> +       /* disable I/D-cache */
> +       csrc    CSR_MCACHECTL, 0x3
> +       /* disable D-cache coherency */
> +       lui     t1, 0x80
> +       csrc    CSR_MCACHECTL, t1
> +       /* 45-series: wait for mcache_ctl.DC_COHSTA to be cleared */
> +check_cm_disabled:
> +       csrr    t1, CSR_MCACHECTL
> +       srli    t1, t1, 20
> +       andi    t1, t1, 0x1
> +       bnez    t1, check_cm_disabled
> +
> +       ret
> +
> +       .section .text, "ax", %progbits
> +       .align 3
> +       .global __ae350_enable_coherency
> +__ae350_enable_coherency:
> +       /* enable D-cache coherency */
> +       lui             t1, 0x80
> +       csrs    CSR_MCACHECTL, t1
> +       /*
> +        * check CM support
> +        * 25-series: mcache_ctl.DC_COHEN is hard-wired to 0
> +        */
> +       csrr    t1, CSR_MCACHECTL
> +       srli    t1, t1, 20
> +       andi    t1, t1, 0x1
> +       beqz    t1, enable_L1_cache
> +       /* 45-series: wait for mcache_ctl.DC_COHSTA to be set */
> +check_cm_enabled:
> +       csrr    t1, CSR_MCACHECTL
> +       srli    t1, t1, 20
> +       andi    t1, t1, 0x1
> +       beqz    t1, check_cm_enabled
> +enable_L1_cache:
> +       /* enable I/D-cache */
> +       csrs    CSR_MCACHECTL, 0x3
> +
> +       ret
> +
> +       .section .text, "ax", %progbits
> +       .align 3
> +       .global __ae350_enable_coherency_warmboot
> +__ae350_enable_coherency_warmboot:
> +       call ra, __ae350_enable_coherency
> +       j _start_warm
> diff --git a/platform/generic/include/andes/andes45.h b/platform/generic/include/andes/andes45.h
> new file mode 100644
> index 0000000..aea2368
> --- /dev/null
> +++ b/platform/generic/include/andes/andes45.h
> @@ -0,0 +1,10 @@
> +#ifndef _RISCV_ANDES45_H
> +#define _RISCV_ANDES45_H
> +
> +#define CSR_MARCHID_MICROID 0xfff
> +
> +/* Memory and Miscellaneous Registers */
> +#define CSR_MCACHECTL 0x7ca
> +#define CSR_MCCTLCOMMAND 0x7cc
> +
> +#endif /* _RISCV_ANDES45_H */
> diff --git a/platform/generic/include/andes/atcsmu.h b/platform/generic/include/andes/atcsmu.h
> new file mode 100644
> index 0000000..c72600c
> --- /dev/null
> +++ b/platform/generic/include/andes/atcsmu.h
> @@ -0,0 +1,44 @@
> +#ifndef _RISCV_ATCSMU_H
> +#define _RISCV_ATCSMU_H
> +
> +#define PCS0_WE_OFFSET 0x90
> +#define PCSm_WE_OFFSET(i) ((i + 3) * 0x20 + PCS0_WE_OFFSET)
> +
> +#define PCS0_CTL_OFFSET 0x94
> +#define PCSm_CTL_OFFSET(i) ((i + 3) * 0x20 + PCS0_CTL_OFFSET)
> +#define PCS_CTL_CMD_SHIFT 0
> +#define PCS_CTL_PARAM_SHIFT 3
> +#define SLEEP_CMD 0x3
> +#define WAKEUP_CMD (0x0 | (1 << PCS_CTL_PARAM_SHIFT))
> +#define LIGHTSLEEP_MODE 0
> +#define DEEPSLEEP_MODE 1
> +#define LIGHT_SLEEP_CMD (SLEEP_CMD | (LIGHTSLEEP_MODE << PCS_CTL_PARAM_SHIFT))
> +#define DEEP_SLEEP_CMD (SLEEP_CMD | (DEEPSLEEP_MODE << PCS_CTL_PARAM_SHIFT))
> +
> +#define PCS0_CFG_OFFSET 0x80
> +#define PCSm_CFG_OFFSET(i) ((i + 3) * 0x20 + PCS0_CFG_OFFSET)
> +#define PCS_CFG_LIGHT_SLEEP_SHIFT 2
> +#define PCS_CFG_LIGHT_SLEEP (1 << PCS_CFG_LIGHT_SLEEP_SHIFT)
> +#define PCS_CFG_DEEP_SLEEP_SHIFT 3
> +#define PCS_CFG_DEEP_SLEEP (1 << PCS_CFG_DEEP_SLEEP_SHIFT)
> +#define RESET_VEC_LO_OFFSET 0x50
> +#define RESET_VEC_HI_OFFSET 0x60
> +#define RESET_VEC_8CORE_OFFSET 0x1a0
> +#define HARTn_RESET_VEC_LO(n)  \
> +       (RESET_VEC_LO_OFFSET + \
> +        ((n) < 4 ? 0 : RESET_VEC_8CORE_OFFSET) + ((n) * 0x4))
> +#define HARTn_RESET_VEC_HI(n)  \
> +       (RESET_VEC_HI_OFFSET + \
> +        ((n) < 4 ? 0 : RESET_VEC_8CORE_OFFSET) + ((n) * 0x4))
> +#define PCS_MAX_NR 8
> +#define FLASH_BASE 0x80000000ULL
> +
> +#ifndef __ASSEMBLER__
> +
> +struct smu_data {
> +       unsigned long addr;
> +};
> +
> +#endif /* __ASSEMBLER__ */
> +
> +#endif /* _RISCV_ATCSMU_H */
> --
> 2.34.1
>
>
> --
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH 5/6] lib: reset/fdt_reset_atcwdt200: use defined macros in atcsmu.h

[Anup Patel]

On Wed, Jan 4, 2023 at 12:02 PM Yu Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Reuse the smu related macros defined in atcsmu.h.
>
> Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> ---
>  lib/utils/reset/fdt_reset_atcwdt200.c | 12 ++++--------
>  1 file changed, 4 insertions(+), 8 deletions(-)
>
> diff --git a/lib/utils/reset/fdt_reset_atcwdt200.c b/lib/utils/reset/fdt_reset_atcwdt200.c
> index 91acc9f..a6e6834 100644
> --- a/lib/utils/reset/fdt_reset_atcwdt200.c
> +++ b/lib/utils/reset/fdt_reset_atcwdt200.c
> @@ -17,6 +17,8 @@
>  #include <sbi_utils/fdt/fdt_helper.h>
>  #include <sbi_utils/reset/fdt_reset.h>
>
> +#include <andes/atcsmu.h>
> +

This can result in compile error because andes/atcsmu.h is
part of platform code.

I suggest that andes/atcsmu.h should be in a common location
include/sbi_utils/sys/ which is meant for common system devices.

>  #define ATCWDT200_WP_NUM 0x5aa5
>  #define WREN_REG 0x18
>  #define CTRL_REG 0x10
> @@ -41,12 +43,6 @@
>  #define CLK_PCLK (1 << 1)
>  #define WDT_EN (1 << 0)
>
> -#define FLASH_BASE 0x80000000ULL
> -#define SMU_RESET_VEC_LO_OFF 0x50
> -#define SMU_RESET_VEC_HI_OFF 0x60
> -#define SMU_HARTn_RESET_VEC_LO(n) (SMU_RESET_VEC_LO_OFF + (n * 0x4))
> -#define SMU_HARTn_RESET_VEC_HI(n) (SMU_RESET_VEC_HI_OFF + (n * 0x4))
> -
>  static volatile char *wdt_addr;
>  static volatile char *smu_addr;
>
> @@ -67,8 +63,8 @@ static void ae350_system_reset(u32 type, u32 reason)
>         const struct sbi_platform *plat = sbi_platform_thishart_ptr();
>
>         for (int i = 0; i < sbi_platform_hart_count(plat); i++) {
> -               writel(FLASH_BASE, smu_addr + SMU_HARTn_RESET_VEC_LO(i));
> -               writel(FLASH_BASE >> 32, smu_addr + SMU_HARTn_RESET_VEC_HI(i));
> +               writel(FLASH_BASE, smu_addr + HARTn_RESET_VEC_LO(i));
> +               writel(FLASH_BASE >> 32, smu_addr + HARTn_RESET_VEC_HI(i));
>         }
>
>         /* Program WDT control register  */
> --
> 2.34.1
>
>
> --
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

Regards,
Anup

[PATCH 6/6] lib: sbi_hsm: Introduce hart_secondary_boot() callback

[Anup Patel]

On Sun, Jan 8, 2023 at 6:40 PM Yu-Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Hi Andrew,
>
> On Fri, Jan 06, 2023 at 03:29:46PM +0100, Andrew Jones wrote:
> > On Wed, Jan 04, 2023 at 02:29:27PM +0800, Yu Chien Peter Lin wrote:
> > > When platform supports hotplug, i.e. both hart_start() and hart_stop()
> > > callbacks are provided, the former doesn't need to be performed at
> > > the boot-time. Thus, add a callback for the case of secondary boot.
> > >
> > > Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> > > ---
> > >  include/sbi/sbi_hsm.h |  7 +++++++
> > >  lib/sbi/sbi_hsm.c     | 22 +++++++++++++---------
> > >  2 files changed, 20 insertions(+), 9 deletions(-)
> >
> > I'm not sure why this patch is in the series as the change isn't used.
> >
> > Thanks,
> > drew
>
> Thanks for the review.
>
> I'm just found it's not necessary to run platform-specific hart_start()
> at boot-time since it's for CPU hotplug, I'm not sure hart_start() is
> supposed to do at boot-time so we can ignore this patch if it's not worth
> adding a callback.

For now it looks like we don't need this callback but we can revisit
in the future.

Regards,
Anup

>
> Best regards,
> Peter Lin
>
>
>
> --
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH 6/6] lib: sbi_hsm: Introduce hart_secondary_boot() callback

[Yu-Chien Peter Lin]

Hi Anup,

On Mon, Jan 16, 2023 at 02:48:46PM +0530, Anup Patel wrote:
> On Sun, Jan 8, 2023 at 6:40 PM Yu-Chien Peter Lin
> <peterlin@andestech.com> wrote:
> >
> > Hi Andrew,
> >
> > On Fri, Jan 06, 2023 at 03:29:46PM +0100, Andrew Jones wrote:
> > > On Wed, Jan 04, 2023 at 02:29:27PM +0800, Yu Chien Peter Lin wrote:
> > > > When platform supports hotplug, i.e. both hart_start() and hart_stop()
> > > > callbacks are provided, the former doesn't need to be performed at
> > > > the boot-time. Thus, add a callback for the case of secondary boot.
> > > >
> > > > Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> > > > ---
> > > >  include/sbi/sbi_hsm.h |  7 +++++++
> > > >  lib/sbi/sbi_hsm.c     | 22 +++++++++++++---------
> > > >  2 files changed, 20 insertions(+), 9 deletions(-)
> > >
> > > I'm not sure why this patch is in the series as the change isn't used.
> > >
> > > Thanks,
> > > drew
> >
> > Thanks for the review.
> >
> > I'm just found it's not necessary to run platform-specific hart_start()
> > at boot-time since it's for CPU hotplug, I'm not sure hart_start() is
> > supposed to do at boot-time so we can ignore this patch if it's not worth
> > adding a callback.
> 
> For now it looks like we don't need this callback but we can revisit
> in the future.
> 
> Regards,
> Anup
> 

Thanks for the review, I will drop this callback.

Best regards,
Peter Lin

[PATCH 5/6] lib: reset/fdt_reset_atcwdt200: use defined macros in atcsmu.h

[Yu-Chien Peter Lin]

Hi Anup,

On Mon, Jan 16, 2023 at 02:45:51PM +0530, Anup Patel wrote:
> On Wed, Jan 4, 2023 at 12:02 PM Yu Chien Peter Lin
> <peterlin@andestech.com> wrote:
> >
> > Reuse the smu related macros defined in atcsmu.h.
> >
> > Signed-off-by: Yu Chien Peter Lin <peterlin@andestech.com>
> > ---
> >  lib/utils/reset/fdt_reset_atcwdt200.c | 12 ++++--------
> >  1 file changed, 4 insertions(+), 8 deletions(-)
> >
> > diff --git a/lib/utils/reset/fdt_reset_atcwdt200.c b/lib/utils/reset/fdt_reset_atcwdt200.c
> > index 91acc9f..a6e6834 100644
> > --- a/lib/utils/reset/fdt_reset_atcwdt200.c
> > +++ b/lib/utils/reset/fdt_reset_atcwdt200.c
> > @@ -17,6 +17,8 @@
> >  #include <sbi_utils/fdt/fdt_helper.h>
> >  #include <sbi_utils/reset/fdt_reset.h>
> >
> > +#include <andes/atcsmu.h>
> > +
> 
> This can result in compile error because andes/atcsmu.h is
> part of platform code.
> 
> I suggest that andes/atcsmu.h should be in a common location
> include/sbi_utils/sys/ which is meant for common system devices.
>
> Regards,
> Anup

OK, will fix.

Thanks,
Peter Lin