[PATCH v5 0/3] Renesas RZ/Five add support to configure PMA and add vendor specific SBI extension

[PATCH v5 0/3] Renesas RZ/Five add support to configure PMA and add vendor specific SBI extension

[Lad Prabhakar]

Hi All,

This patch series aims to add support to configure the PMA regions present
on the Andes AX45MP core (andes45-pma.c which can be shared with other
Andes vendors) and configures the required PMA region for RZ/Five SoC.

This patch series also adds the SBI vendor extension
RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to check if IOCP is present on
the core and if un-available it also checks if SW workaround for cache
management needs to be applied.

v4 -> v5
-> Included RB tags from Yu Chien Peter Lin
-> Moved PMA configuration code into andes folder
-> Rebased the changes on top of [0]
[0] https://patchwork.ozlabs.org/project/opensbi/list/?series=341535

RFC v3 -> v4
-> Used bit field for flags in struct andes45_pma_region
-> Renamed andes45-pma.h -> andes45_pma.h
-> Included RB tag from Lin san for patch 2/3
-> Moved CSR to platform/generic/include/andes/andes45.h
-> Included sbi/sbi_error.h in andes45-pma.c to fix build issue

RFC v2 -> RFC v3
-> Fixed review comments pointed by Lin-san
   * Used switch case macro
   * Added a check to verify the PMA size is power-of-2
   * Dropped un-needed calculation done in PMA code for NAPOT.
   * Dropped MISA macro
-> Fixed comments pointed by Himanshu
-> Fixed check patch issue

RFC v2 -> RFC v1
-> Fixed review comments pointed by Lin-san
-> Implemented PMA as library so that it can be shared

RFC v1:
https://patchwork.ozlabs.org/project/opensbi/patch/20221212094421.14556-1-prabhakar.mahadev-lad.rj at bp.renesas.com/

Cheers,
Prabhakar

Lad Prabhakar (3):
  platform: generic: renesas: rzfive: Add support to configure the PMA
  platform: generic: renesas: rzfive: Configure the PMA region
  platform: generic: renesas: rzfive: Add SBI EXT to check for enabling
    IOCP errata

 platform/generic/Kconfig                     |   3 +
 platform/generic/andes/Kconfig               |   5 +
 platform/generic/andes/andes45-pma.c         | 350 +++++++++++++++++++
 platform/generic/andes/objects.mk            |   2 +
 platform/generic/include/andes/andes45.h     |  25 +-
 platform/generic/include/andes/andes45_pma.h |  48 +++
 platform/generic/renesas/rzfive/rzfive.c     |  62 ++++
 7 files changed, 493 insertions(+), 2 deletions(-)
 create mode 100644 platform/generic/andes/Kconfig
 create mode 100644 platform/generic/andes/andes45-pma.c
 create mode 100644 platform/generic/include/andes/andes45_pma.h

-- 
2.17.1

[PATCH v5 1/3] platform: generic: renesas: rzfive: Add support to configure the PMA

[Lad Prabhakar]

I/O Coherence Port (IOCP) provides an AXI interface for connecting
external non-caching masters, such as DMA controllers. The accesses
from IOCP are coherent with D-Caches and L2 Cache.

IOCP is a specification option and is disabled on the Renesas RZ/Five
SoC due to this reason IP blocks using DMA will fail.

The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
block that allows dynamic adjustment of memory attributes in the runtime.
It contains a configurable amount of PMA entries implemented as CSR
registers to control the attributes of memory locations in interest.
Below are the memory attributes supported:
* Device, Non-bufferable
* Device, bufferable
* Memory, Non-cacheable, Non-bufferable
* Memory, Non-cacheable, Bufferable
* Memory, Write-back, No-allocate
* Memory, Write-back, Read-allocate
* Memory, Write-back, Write-allocate
* Memory, Write-back, Read and Write-allocate

More info about PMA (section 10.3):
Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf

As a workaround for SoCs with IOCP disabled CMO needs to be handled by
software. Firstly OpenSBI configures the memory region as
"Memory, Non-cacheable, Bufferable" and passes this region as a global
shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
allocations happen from this region and synchronization callbacks are
implemented to synchronize when doing DMA transactions.

Example PMA region passed as a DT node from OpenSBI:
    reserved-memory {
        #address-cells = <2>;
        #size-cells = <2>;
        ranges;

        pma_resv0 at 58000000 {
            compatible = "shared-dma-pool";
            reg = <0x0 0x58000000 0x0 0x08000000>;
            no-map;
            linux,dma-default;
        };
    };

Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 platform/generic/Kconfig                     |   2 +
 platform/generic/andes/Kconfig               |   5 +
 platform/generic/andes/andes45-pma.c         | 350 +++++++++++++++++++
 platform/generic/andes/objects.mk            |   2 +
 platform/generic/include/andes/andes45_pma.h |  48 +++
 5 files changed, 407 insertions(+)
 create mode 100644 platform/generic/andes/Kconfig
 create mode 100644 platform/generic/andes/andes45-pma.c
 create mode 100644 platform/generic/include/andes/andes45_pma.h

diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
index c7f198a..39fb4e9 100644
--- a/platform/generic/Kconfig
+++ b/platform/generic/Kconfig
@@ -50,4 +50,6 @@ config PLATFORM_STARFIVE_JH7110
 	bool "StarFive JH7110 support"
 	default n
 
+source "$(OPENSBI_SRC_DIR)/platform/generic/andes/Kconfig"
+
 endif
diff --git a/platform/generic/andes/Kconfig b/platform/generic/andes/Kconfig
new file mode 100644
index 0000000..3ad4e4c
--- /dev/null
+++ b/platform/generic/andes/Kconfig
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: BSD-2-Clause
+
+config ANDES45_PMA
+	bool "Andes PMA support"
+	default n
diff --git a/platform/generic/andes/andes45-pma.c b/platform/generic/andes/andes45-pma.c
new file mode 100644
index 0000000..2745bc3
--- /dev/null
+++ b/platform/generic/andes/andes45-pma.c
@@ -0,0 +1,350 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2023 Renesas Electronics Corp.
+ *
+ * Copyright (c) 2020 Andes Technology Corporation
+ *
+ * Authors:
+ *      Nick Hu <nickhu@andestech.com>
+ *      Nylon Chen <nylon7@andestech.com>
+ *      Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
+ */
+
+#include <andes/andes45_pma.h>
+#include <libfdt.h>
+#include <sbi/riscv_asm.h>
+#include <sbi/riscv_io.h>
+#include <sbi/sbi_console.h>
+#include <sbi/sbi_error.h>
+#include <sbi_utils/fdt/fdt_helper.h>
+
+/* Configuration Registers */
+#define ANDES45_CSR_MMSC_CFG		0xFC2
+#define ANDES45_CSR_MMSC_PPMA_OFFSET	(1 << 30)
+
+#define ANDES45_PMAADDR_0		0xBD0
+
+#define ANDES45_PMACFG_0		0xBC0
+
+static inline unsigned long andes45_pma_read_cfg(unsigned int pma_cfg_off)
+{
+#define switchcase_pma_cfg_read(__pma_cfg_off, __val)		\
+	case __pma_cfg_off:					\
+		__val = csr_read(__pma_cfg_off);		\
+		break;
+#define switchcase_pma_cfg_read_2(__pma_cfg_off, __val)		\
+	switchcase_pma_cfg_read(__pma_cfg_off + 0, __val)	\
+	switchcase_pma_cfg_read(__pma_cfg_off + 2, __val)
+
+	unsigned long ret = 0;
+
+	switch (pma_cfg_off) {
+	switchcase_pma_cfg_read_2(ANDES45_PMACFG_0, ret)
+
+	default:
+		sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
+		break;
+	}
+
+	return ret;
+
+#undef switchcase_pma_cfg_read_2
+#undef switchcase_pma_cfg_read
+}
+
+static inline void andes45_pma_write_cfg(unsigned int pma_cfg_off, unsigned long val)
+{
+#define switchcase_pma_cfg_write(__pma_cfg_off, __val)		\
+	case __pma_cfg_off:					\
+		csr_write(__pma_cfg_off, __val);		\
+		break;
+#define switchcase_pma_cfg_write_2(__pma_cfg_off, __val)	\
+	switchcase_pma_cfg_write(__pma_cfg_off + 0, __val)	\
+	switchcase_pma_cfg_write(__pma_cfg_off + 2, __val)
+
+	switch (pma_cfg_off) {
+	switchcase_pma_cfg_write_2(ANDES45_PMACFG_0, val)
+
+	default:
+		sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
+		break;
+	}
+
+#undef switchcase_pma_cfg_write_2
+#undef switchcase_pma_cfg_write
+}
+
+static inline void andes45_pma_write_addr(unsigned int pma_addr_off, unsigned long val)
+{
+#define switchcase_pma_write(__pma_addr_off, __val)		\
+	case __pma_addr_off:					\
+		csr_write(__pma_addr_off, __val);		\
+		break;
+#define switchcase_pma_write_2(__pma_addr_off, __val)		\
+	switchcase_pma_write(__pma_addr_off + 0, __val)		\
+	switchcase_pma_write(__pma_addr_off + 1, __val)
+#define switchcase_pma_write_4(__pma_addr_off, __val)		\
+	switchcase_pma_write_2(__pma_addr_off + 0, __val)	\
+	switchcase_pma_write_2(__pma_addr_off + 2, __val)
+#define switchcase_pma_write_8(__pma_addr_off, __val)		\
+	switchcase_pma_write_4(__pma_addr_off + 0, __val)	\
+	switchcase_pma_write_4(__pma_addr_off + 4, __val)
+#define switchcase_pma_write_16(__pma_addr_off, __val)		\
+	switchcase_pma_write_8(__pma_addr_off + 0, __val)	\
+	switchcase_pma_write_8(__pma_addr_off + 8, __val)
+
+	switch (pma_addr_off) {
+	switchcase_pma_write_16(ANDES45_PMAADDR_0, val)
+
+	default:
+		sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
+		break;
+	}
+
+#undef switchcase_pma_write_16
+#undef switchcase_pma_write_8
+#undef switchcase_pma_write_4
+#undef switchcase_pma_write_2
+#undef switchcase_pma_write
+}
+
+static inline unsigned long andes45_pma_read_addr(unsigned int pma_addr_off)
+{
+#define switchcase_pma_read(__pma_addr_off, __val)		\
+	case __pma_addr_off:					\
+		__val = csr_read(__pma_addr_off);		\
+		break;
+#define switchcase_pma_read_2(__pma_addr_off, __val)		\
+	switchcase_pma_read(__pma_addr_off + 0, __val)		\
+	switchcase_pma_read(__pma_addr_off + 1, __val)
+#define switchcase_pma_read_4(__pma_addr_off, __val)		\
+	switchcase_pma_read_2(__pma_addr_off + 0, __val)	\
+	switchcase_pma_read_2(__pma_addr_off + 2, __val)
+#define switchcase_pma_read_8(__pma_addr_off, __val)		\
+	switchcase_pma_read_4(__pma_addr_off + 0, __val)	\
+	switchcase_pma_read_4(__pma_addr_off + 4, __val)
+#define switchcase_pma_read_16(__pma_addr_off, __val)		\
+	switchcase_pma_read_8(__pma_addr_off + 0, __val)	\
+	switchcase_pma_read_8(__pma_addr_off + 8, __val)
+
+	unsigned long ret = 0;
+
+	switch (pma_addr_off) {
+	switchcase_pma_read_16(ANDES45_PMAADDR_0, ret)
+
+	default:
+		sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
+		break;
+	}
+
+	return ret;
+
+#undef switchcase_pma_read_16
+#undef switchcase_pma_read_8
+#undef switchcase_pma_read_4
+#undef switchcase_pma_read_2
+#undef switchcase_pma_read
+}
+
+static unsigned long
+andes45_pma_setup(const struct andes45_pma_region *pma_region,
+		  unsigned int entry_id)
+{
+	unsigned long size = pma_region->size;
+	unsigned long addr = pma_region->pa;
+	unsigned int pma_cfg_addr;
+	unsigned long pmacfg_val;
+	unsigned long pmaaddr;
+	char *pmaxcfg;
+
+	/* Check for 4KiB granularity */
+	if (size < (1 << 12))
+		return SBI_EINVAL;
+
+	/* Check size is power of 2 */
+	if (size & (size - 1))
+		return SBI_EINVAL;
+
+	if (entry_id > 15)
+		return SBI_EINVAL;
+
+	if (!(pma_region->flags & ANDES45_PMACFG_ETYP_NAPOT))
+		return SBI_EINVAL;
+
+	if ((addr & (size - 1)) != 0)
+		return SBI_EINVAL;
+
+	pma_cfg_addr = entry_id / 8 ? ANDES45_PMACFG_0 + 2 : ANDES45_PMACFG_0;
+	pmacfg_val = andes45_pma_read_cfg(pma_cfg_addr);
+	pmaxcfg = (char *)&pmacfg_val + (entry_id % 8);
+	*pmaxcfg = 0;
+	*pmaxcfg = pma_region->flags;
+
+	andes45_pma_write_cfg(pma_cfg_addr, pmacfg_val);
+
+	pmaaddr = (addr >> 2) + (size >> 3) - 1;
+
+	andes45_pma_write_addr(ANDES45_PMAADDR_0 + entry_id, pmaaddr);
+
+	return andes45_pma_read_addr(ANDES45_PMAADDR_0 + entry_id) == pmaaddr ?
+			pmaaddr : SBI_EINVAL;
+}
+
+static int andes45_fdt_pma_resv(void *fdt, const struct andes45_pma_region *pma,
+				unsigned int index, int parent)
+{
+	int na = fdt_address_cells(fdt, 0);
+	int ns = fdt_size_cells(fdt, 0);
+	static bool dma_default = false;
+	fdt32_t addr_high, addr_low;
+	fdt32_t size_high, size_low;
+	int subnode, err;
+	fdt32_t reg[4];
+	fdt32_t *val;
+	char name[32];
+
+	addr_high = (u64)pma->pa >> 32;
+	addr_low = pma->pa;
+	size_high = (u64)pma->size >> 32;
+	size_low = pma->size;
+
+	if (na > 1 && addr_high)
+		sbi_snprintf(name, sizeof(name),
+			     "pma_resv%d@%x,%x", index,
+			     addr_high, addr_low);
+	else
+		sbi_snprintf(name, sizeof(name),
+			     "pma_resv%d@%x", index,
+			     addr_low);
+
+	subnode = fdt_add_subnode(fdt, parent, name);
+	if (subnode < 0)
+		return subnode;
+
+	if (pma->shared_dma) {
+		err = fdt_setprop_string(fdt, subnode, "compatible", "shared-dma-pool");
+		if (err < 0)
+			return err;
+	}
+
+	if (pma->no_map) {
+		err = fdt_setprop_empty(fdt, subnode, "no-map");
+		if (err < 0)
+			return err;
+	}
+
+	/* Linux allows single linux,dma-default region. */
+	if (pma->dma_default) {
+		if (dma_default)
+			return SBI_EINVAL;
+
+		err = fdt_setprop_empty(fdt, subnode, "linux,dma-default");
+		if (err < 0)
+			return err;
+		dma_default = true;
+	}
+
+	/* encode the <reg> property value */
+	val = reg;
+	if (na > 1)
+		*val++ = cpu_to_fdt32(addr_high);
+	*val++ = cpu_to_fdt32(addr_low);
+	if (ns > 1)
+		*val++ = cpu_to_fdt32(size_high);
+	*val++ = cpu_to_fdt32(size_low);
+
+	err = fdt_setprop(fdt, subnode, "reg", reg,
+			  (na + ns) * sizeof(fdt32_t));
+	if (err < 0)
+		return err;
+
+	return 0;
+}
+
+static int andes45_fdt_reserved_memory_fixup(void *fdt,
+					     const struct andes45_pma_region *pma,
+					     unsigned int entry)
+{
+	int parent;
+
+	/* try to locate the reserved memory node */
+	parent = fdt_path_offset(fdt, "/reserved-memory");
+	if (parent < 0) {
+		int na = fdt_address_cells(fdt, 0);
+		int ns = fdt_size_cells(fdt, 0);
+		int err;
+
+		/* if such node does not exist, create one */
+		parent = fdt_add_subnode(fdt, 0, "reserved-memory");
+		if (parent < 0)
+			return parent;
+
+		err = fdt_setprop_empty(fdt, parent, "ranges");
+		if (err < 0)
+			return err;
+
+		err = fdt_setprop_u32(fdt, parent, "#size-cells", ns);
+		if (err < 0)
+			return err;
+
+		err = fdt_setprop_u32(fdt, parent, "#address-cells", na);
+		if (err < 0)
+			return err;
+	}
+
+	return andes45_fdt_pma_resv(fdt, pma, entry, parent);
+}
+
+int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
+			      unsigned int pma_regions_count)
+{
+	unsigned long mmsc = csr_read(ANDES45_CSR_MMSC_CFG);
+	unsigned int dt_populate_cnt;
+	unsigned int i, j;
+	unsigned long pa;
+	void *fdt;
+	int ret;
+
+	if (!pma_regions || !pma_regions_count)
+		return 0;
+
+	if (pma_regions_count > ANDES45_MAX_PMA_REGIONS)
+		return SBI_EINVAL;
+
+	if ((mmsc & ANDES45_CSR_MMSC_PPMA_OFFSET) == 0)
+		return SBI_ENOTSUPP;
+
+	/* Configure the PMA regions */
+	for (i = 0; i < pma_regions_count; i++) {
+		pa = andes45_pma_setup(&pma_regions[i], i);
+		if (pa == SBI_EINVAL)
+			return SBI_EINVAL;
+	}
+
+	dt_populate_cnt = 0;
+	for (i = 0; i < pma_regions_count; i++) {
+		if (!pma_regions[i].dt_populate)
+			continue;
+		dt_populate_cnt++;
+	}
+
+	if (!dt_populate_cnt)
+		return 0;
+
+	fdt = fdt_get_address();
+
+	ret = fdt_open_into(fdt, fdt, fdt_totalsize(fdt) + (64 * dt_populate_cnt));
+	if (ret < 0)
+		return ret;
+
+	for (i = 0, j = 0; i < pma_regions_count; i++) {
+		if (!pma_regions[i].dt_populate)
+			continue;
+
+		ret = andes45_fdt_reserved_memory_fixup(fdt, &pma_regions[i], j++);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
diff --git a/platform/generic/andes/objects.mk b/platform/generic/andes/objects.mk
index 28275ef..ea6b561 100644
--- a/platform/generic/andes/objects.mk
+++ b/platform/generic/andes/objects.mk
@@ -4,3 +4,5 @@
 
 carray-platform_override_modules-$(CONFIG_PLATFORM_ANDES_AE350) += andes_ae350
 platform-objs-$(CONFIG_PLATFORM_ANDES_AE350) += andes/ae350.o andes/sleep.o
+
+platform-objs-$(CONFIG_ANDES45_PMA) += andes/andes45-pma.o
diff --git a/platform/generic/include/andes/andes45_pma.h b/platform/generic/include/andes/andes45_pma.h
new file mode 100644
index 0000000..37ec77c
--- /dev/null
+++ b/platform/generic/include/andes/andes45_pma.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2023 Renesas Electronics Corp.
+ */
+
+#ifndef _ANDES45_PMA_H_
+#define _ANDES45_PMA_H_
+
+#include <sbi/sbi_types.h>
+
+#define ANDES45_MAX_PMA_REGIONS			16
+
+/* Naturally aligned power of 2 region */
+#define ANDES45_PMACFG_ETYP_NAPOT		3
+
+/* Memory, Non-cacheable, Bufferable */
+#define ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF	(3 << 2)
+
+/**
+ * struct andes45_pma_region - Describes PMA regions
+ *
+ * @pa: Address to be configured in the PMA
+ * @size: Size of the region
+ * @flags: Flags to be set for the PMA region
+ * @dt_populate: Boolean flag indicating if the DT entry should be
+ *               populated for the given PMA region
+ * @shared_dma: Boolean flag if set "shared-dma-pool" property will
+ *              be set in the DT node
+ * @no_map: Boolean flag if set "no-map" property will be set in the
+ *          DT node
+ * @dma_default: Boolean flag if set "linux,dma-default" property will
+ *              be set in the DT node. Note Linux expects single node
+ *              with this property set.
+ */
+struct andes45_pma_region {
+	unsigned long pa;
+	unsigned long size;
+	u8 flags:7;
+	bool dt_populate;
+	bool shared_dma;
+	bool no_map;
+	bool dma_default;
+};
+
+int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
+			      unsigned int pma_regions_count);
+
+#endif /* _ANDES45_PMA_H_ */
-- 
2.17.1

[PATCH v5 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Lad Prabhakar]

On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
and populate this region as PMA reserve DT node with shared DMA pool and
no-map flags set so that Linux drivers requesting any DMA'able memory go
through this region.

PMA node passed to the above stack:

        reserved-memory {
            #address-cells = <2>;
            #size-cells = <2>;
            ranges;

            pma_resv0 at 58000000 {
                compatible = "shared-dma-pool";
                reg = <0x0 0x58000000 0x0 0x08000000>;
                no-map;
                linux,dma-default;
            };
        };

Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 platform/generic/Kconfig                 |  1 +
 platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
index 39fb4e9..1f4f8e1 100644
--- a/platform/generic/Kconfig
+++ b/platform/generic/Kconfig
@@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
 
 config PLATFORM_RENESAS_RZFIVE
 	bool "Renesas RZ/Five support"
+	select ANDES45_PMA
 	default n
 
 config PLATFORM_SIFIVE_FU540
diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
index ee9c9c4..4d71d0d 100644
--- a/platform/generic/renesas/rzfive/rzfive.c
+++ b/platform/generic/renesas/rzfive/rzfive.c
@@ -4,10 +4,30 @@
  *
  */
 
+#include <andes/andes45_pma.h>
 #include <platform_override.h>
 #include <sbi/sbi_domain.h>
 #include <sbi_utils/fdt/fdt_helper.h>
 
+static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
+	{
+		.pa = 0x58000000,
+		.size = 0x8000000,
+		.flags = ANDES45_PMACFG_ETYP_NAPOT |
+			 ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF,
+		.dt_populate = true,
+		.shared_dma = true,
+		.no_map = true,
+		.dma_default = true,
+	},
+};
+
+static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *match)
+{
+	return andes45_pma_setup_regions(renesas_rzfive_pma_regions,
+					 array_size(renesas_rzfive_pma_regions));
+}
+
 int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
 {
 	/*
@@ -34,4 +54,5 @@ static const struct fdt_match renesas_rzfive_match[] = {
 const struct platform_override renesas_rzfive = {
 	.match_table = renesas_rzfive_match,
 	.early_init = renesas_rzfive_early_init,
+	.final_init = renesas_rzfive_final_init,
 };
-- 
2.17.1

[PATCH v5 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Lad Prabhakar]

I/O Coherence Port (IOCP) provides an AXI interface for connecting
external non-caching masters, such as DMA controllers. The accesses
from IOCP are coherent with D-Caches and L2 Cache.

IOCP is a specification option and is disabled on the Renesas RZ/Five
SoC due to this reason IP blocks using DMA will fail.

As a workaround for SoCs with IOCP disabled CMO needs to be handled by
software. Firstly OpenSBI configures the memory region as
"Memory, Non-cacheable, Bufferable" and passes this region as a global
shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
allocations happen from this region and synchronization callbacks are
implemented to synchronize when doing DMA transactions.

RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
errata should be applied to handle cache management.

Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
---
 platform/generic/include/andes/andes45.h | 25 +++++++++++++--
 platform/generic/renesas/rzfive/rzfive.c | 41 ++++++++++++++++++++++++
 2 files changed, 64 insertions(+), 2 deletions(-)

diff --git a/platform/generic/include/andes/andes45.h b/platform/generic/include/andes/andes45.h
index 08b3d18..d5d265f 100644
--- a/platform/generic/include/andes/andes45.h
+++ b/platform/generic/include/andes/andes45.h
@@ -4,7 +4,28 @@
 #define CSR_MARCHID_MICROID 0xfff
 
 /* Memory and Miscellaneous Registers */
-#define CSR_MCACHE_CTL 0x7ca
-#define CSR_MCCTLCOMMAND 0x7cc
+#define CSR_MCACHE_CTL		0x7ca
+#define CSR_MCCTLCOMMAND	0x7cc
+#define CSR_MICM_CFG		0xfc0
+#define CSR_MDCM_CFG		0xfc1
+#define CSR_MMSC_CFG		0xfc2
+
+#define MISA_20_OFFSET			20
+#define MISA_20_MASK			(0x1 << MISA_20_OFFSET)
+
+#define MICM_CFG_ISZ_OFFSET		6
+#define MICM_CFG_ISZ_MASK		(0x7  << MICM_CFG_ISZ_OFFSET)
+
+#define MDCM_CFG_DSZ_OFFSET		6
+#define MDCM_CFG_DSZ_MASK		(0x7  << MDCM_CFG_DSZ_OFFSET)
+
+#define MMSC_CFG_CCTLCSR_OFFSET		16
+#define MMSC_CFG_CCTLCSR_MASK		(0x1 << MMSC_CFG_CCTLCSR_OFFSET)
+#define MMSC_IOCP_OFFSET			47
+#define MMSC_IOCP_MASK			(0x1ULL << MMSC_IOCP_OFFSET)
+
+#define MCACHE_CTL_CCTL_SUEN_OFFSET	8
+#define MCACHE_CTL_CCTL_SUEN_MASK	(0x1 << MCACHE_CTL_CCTL_SUEN_OFFSET)
+
 
 #endif /* _RISCV_ANDES45_H */
diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
index 4d71d0d..9530d28 100644
--- a/platform/generic/renesas/rzfive/rzfive.c
+++ b/platform/generic/renesas/rzfive/rzfive.c
@@ -4,11 +4,14 @@
  *
  */
 
+#include <andes/andes45.h>
 #include <andes/andes45_pma.h>
 #include <platform_override.h>
 #include <sbi/sbi_domain.h>
 #include <sbi_utils/fdt/fdt_helper.h>
 
+#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND	0
+
 static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
 	{
 		.pa = 0x58000000,
@@ -28,6 +31,43 @@ static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *mat
 					 array_size(renesas_rzfive_pma_regions));
 }
 
+static bool renesas_rzfive_cpu_cache_controlable(void)
+{
+	return (((csr_read(CSR_MICM_CFG) & MICM_CFG_ISZ_MASK) ||
+		 (csr_read(CSR_MDCM_CFG) & MDCM_CFG_DSZ_MASK)) &&
+		(csr_read(CSR_MISA) & MISA_20_MASK) &&
+		(csr_read(CSR_MMSC_CFG) & MMSC_CFG_CCTLCSR_MASK) &&
+		(csr_read(CSR_MCACHE_CTL) & MCACHE_CTL_CCTL_SUEN_MASK));
+}
+
+static bool renesas_rzfive_cpu_iocp_disabled(void)
+{
+	return (csr_read(CSR_MMSC_CFG) & MMSC_IOCP_MASK) ? false : true;
+}
+
+static bool renesas_rzfive_apply_iocp_sw_workaround(void)
+{
+	return renesas_rzfive_cpu_cache_controlable() & renesas_rzfive_cpu_iocp_disabled();
+}
+
+static int renesas_rzfive_vendor_ext_provider(long funcid,
+					      const struct sbi_trap_regs *regs,
+					      unsigned long *out_value,
+					      struct sbi_trap_info *out_trap,
+					      const struct fdt_match *match)
+{
+	switch (funcid) {
+	case RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND:
+		*out_value = renesas_rzfive_apply_iocp_sw_workaround();
+		break;
+
+	default:
+		break;
+	}
+
+	return 0;
+}
+
 int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
 {
 	/*
@@ -55,4 +95,5 @@ const struct platform_override renesas_rzfive = {
 	.match_table = renesas_rzfive_match,
 	.early_init = renesas_rzfive_early_init,
 	.final_init = renesas_rzfive_final_init,
+	.vendor_ext_provider = renesas_rzfive_vendor_ext_provider,
 };
-- 
2.17.1

[PATCH v5 1/3] platform: generic: renesas: rzfive: Add support to configure the PMA

[Anup Patel]

On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
<prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>
> I/O Coherence Port (IOCP) provides an AXI interface for connecting
> external non-caching masters, such as DMA controllers. The accesses
> from IOCP are coherent with D-Caches and L2 Cache.
>
> IOCP is a specification option and is disabled on the Renesas RZ/Five
> SoC due to this reason IP blocks using DMA will fail.
>
> The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
> block that allows dynamic adjustment of memory attributes in the runtime.
> It contains a configurable amount of PMA entries implemented as CSR
> registers to control the attributes of memory locations in interest.
> Below are the memory attributes supported:
> * Device, Non-bufferable
> * Device, bufferable
> * Memory, Non-cacheable, Non-bufferable
> * Memory, Non-cacheable, Bufferable
> * Memory, Write-back, No-allocate
> * Memory, Write-back, Read-allocate
> * Memory, Write-back, Write-allocate
> * Memory, Write-back, Read and Write-allocate
>
> More info about PMA (section 10.3):
> Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf
>
> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> software. Firstly OpenSBI configures the memory region as
> "Memory, Non-cacheable, Bufferable" and passes this region as a global
> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> allocations happen from this region and synchronization callbacks are
> implemented to synchronize when doing DMA transactions.
>
> Example PMA region passed as a DT node from OpenSBI:
>     reserved-memory {
>         #address-cells = <2>;
>         #size-cells = <2>;
>         ranges;
>
>         pma_resv0 at 58000000 {
>             compatible = "shared-dma-pool";
>             reg = <0x0 0x58000000 0x0 0x08000000>;
>             no-map;
>             linux,dma-default;
>         };
>     };
>
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

> ---
>  platform/generic/Kconfig                     |   2 +
>  platform/generic/andes/Kconfig               |   5 +
>  platform/generic/andes/andes45-pma.c         | 350 +++++++++++++++++++
>  platform/generic/andes/objects.mk            |   2 +
>  platform/generic/include/andes/andes45_pma.h |  48 +++
>  5 files changed, 407 insertions(+)
>  create mode 100644 platform/generic/andes/Kconfig
>  create mode 100644 platform/generic/andes/andes45-pma.c
>  create mode 100644 platform/generic/include/andes/andes45_pma.h
>
> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> index c7f198a..39fb4e9 100644
> --- a/platform/generic/Kconfig
> +++ b/platform/generic/Kconfig
> @@ -50,4 +50,6 @@ config PLATFORM_STARFIVE_JH7110
>         bool "StarFive JH7110 support"
>         default n
>
> +source "$(OPENSBI_SRC_DIR)/platform/generic/andes/Kconfig"
> +
>  endif
> diff --git a/platform/generic/andes/Kconfig b/platform/generic/andes/Kconfig
> new file mode 100644
> index 0000000..3ad4e4c
> --- /dev/null
> +++ b/platform/generic/andes/Kconfig
> @@ -0,0 +1,5 @@
> +# SPDX-License-Identifier: BSD-2-Clause
> +
> +config ANDES45_PMA
> +       bool "Andes PMA support"
> +       default n
> diff --git a/platform/generic/andes/andes45-pma.c b/platform/generic/andes/andes45-pma.c
> new file mode 100644
> index 0000000..2745bc3
> --- /dev/null
> +++ b/platform/generic/andes/andes45-pma.c
> @@ -0,0 +1,350 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (C) 2023 Renesas Electronics Corp.
> + *
> + * Copyright (c) 2020 Andes Technology Corporation
> + *
> + * Authors:
> + *      Nick Hu <nickhu@andestech.com>
> + *      Nylon Chen <nylon7@andestech.com>
> + *      Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> + */
> +
> +#include <andes/andes45_pma.h>
> +#include <libfdt.h>
> +#include <sbi/riscv_asm.h>
> +#include <sbi/riscv_io.h>
> +#include <sbi/sbi_console.h>
> +#include <sbi/sbi_error.h>
> +#include <sbi_utils/fdt/fdt_helper.h>
> +
> +/* Configuration Registers */
> +#define ANDES45_CSR_MMSC_CFG           0xFC2
> +#define ANDES45_CSR_MMSC_PPMA_OFFSET   (1 << 30)
> +
> +#define ANDES45_PMAADDR_0              0xBD0
> +
> +#define ANDES45_PMACFG_0               0xBC0
> +
> +static inline unsigned long andes45_pma_read_cfg(unsigned int pma_cfg_off)
> +{
> +#define switchcase_pma_cfg_read(__pma_cfg_off, __val)          \
> +       case __pma_cfg_off:                                     \
> +               __val = csr_read(__pma_cfg_off);                \
> +               break;
> +#define switchcase_pma_cfg_read_2(__pma_cfg_off, __val)                \
> +       switchcase_pma_cfg_read(__pma_cfg_off + 0, __val)       \
> +       switchcase_pma_cfg_read(__pma_cfg_off + 2, __val)
> +
> +       unsigned long ret = 0;
> +
> +       switch (pma_cfg_off) {
> +       switchcase_pma_cfg_read_2(ANDES45_PMACFG_0, ret)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
> +               break;
> +       }
> +
> +       return ret;
> +
> +#undef switchcase_pma_cfg_read_2
> +#undef switchcase_pma_cfg_read
> +}
> +
> +static inline void andes45_pma_write_cfg(unsigned int pma_cfg_off, unsigned long val)
> +{
> +#define switchcase_pma_cfg_write(__pma_cfg_off, __val)         \
> +       case __pma_cfg_off:                                     \
> +               csr_write(__pma_cfg_off, __val);                \
> +               break;
> +#define switchcase_pma_cfg_write_2(__pma_cfg_off, __val)       \
> +       switchcase_pma_cfg_write(__pma_cfg_off + 0, __val)      \
> +       switchcase_pma_cfg_write(__pma_cfg_off + 2, __val)
> +
> +       switch (pma_cfg_off) {
> +       switchcase_pma_cfg_write_2(ANDES45_PMACFG_0, val)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
> +               break;
> +       }
> +
> +#undef switchcase_pma_cfg_write_2
> +#undef switchcase_pma_cfg_write
> +}
> +
> +static inline void andes45_pma_write_addr(unsigned int pma_addr_off, unsigned long val)
> +{
> +#define switchcase_pma_write(__pma_addr_off, __val)            \
> +       case __pma_addr_off:                                    \
> +               csr_write(__pma_addr_off, __val);               \
> +               break;
> +#define switchcase_pma_write_2(__pma_addr_off, __val)          \
> +       switchcase_pma_write(__pma_addr_off + 0, __val)         \
> +       switchcase_pma_write(__pma_addr_off + 1, __val)
> +#define switchcase_pma_write_4(__pma_addr_off, __val)          \
> +       switchcase_pma_write_2(__pma_addr_off + 0, __val)       \
> +       switchcase_pma_write_2(__pma_addr_off + 2, __val)
> +#define switchcase_pma_write_8(__pma_addr_off, __val)          \
> +       switchcase_pma_write_4(__pma_addr_off + 0, __val)       \
> +       switchcase_pma_write_4(__pma_addr_off + 4, __val)
> +#define switchcase_pma_write_16(__pma_addr_off, __val)         \
> +       switchcase_pma_write_8(__pma_addr_off + 0, __val)       \
> +       switchcase_pma_write_8(__pma_addr_off + 8, __val)
> +
> +       switch (pma_addr_off) {
> +       switchcase_pma_write_16(ANDES45_PMAADDR_0, val)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
> +               break;
> +       }
> +
> +#undef switchcase_pma_write_16
> +#undef switchcase_pma_write_8
> +#undef switchcase_pma_write_4
> +#undef switchcase_pma_write_2
> +#undef switchcase_pma_write
> +}
> +
> +static inline unsigned long andes45_pma_read_addr(unsigned int pma_addr_off)
> +{
> +#define switchcase_pma_read(__pma_addr_off, __val)             \
> +       case __pma_addr_off:                                    \
> +               __val = csr_read(__pma_addr_off);               \
> +               break;
> +#define switchcase_pma_read_2(__pma_addr_off, __val)           \
> +       switchcase_pma_read(__pma_addr_off + 0, __val)          \
> +       switchcase_pma_read(__pma_addr_off + 1, __val)
> +#define switchcase_pma_read_4(__pma_addr_off, __val)           \
> +       switchcase_pma_read_2(__pma_addr_off + 0, __val)        \
> +       switchcase_pma_read_2(__pma_addr_off + 2, __val)
> +#define switchcase_pma_read_8(__pma_addr_off, __val)           \
> +       switchcase_pma_read_4(__pma_addr_off + 0, __val)        \
> +       switchcase_pma_read_4(__pma_addr_off + 4, __val)
> +#define switchcase_pma_read_16(__pma_addr_off, __val)          \
> +       switchcase_pma_read_8(__pma_addr_off + 0, __val)        \
> +       switchcase_pma_read_8(__pma_addr_off + 8, __val)
> +
> +       unsigned long ret = 0;
> +
> +       switch (pma_addr_off) {
> +       switchcase_pma_read_16(ANDES45_PMAADDR_0, ret)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
> +               break;
> +       }
> +
> +       return ret;
> +
> +#undef switchcase_pma_read_16
> +#undef switchcase_pma_read_8
> +#undef switchcase_pma_read_4
> +#undef switchcase_pma_read_2
> +#undef switchcase_pma_read
> +}
> +
> +static unsigned long
> +andes45_pma_setup(const struct andes45_pma_region *pma_region,
> +                 unsigned int entry_id)
> +{
> +       unsigned long size = pma_region->size;
> +       unsigned long addr = pma_region->pa;
> +       unsigned int pma_cfg_addr;
> +       unsigned long pmacfg_val;
> +       unsigned long pmaaddr;
> +       char *pmaxcfg;
> +
> +       /* Check for 4KiB granularity */
> +       if (size < (1 << 12))
> +               return SBI_EINVAL;
> +
> +       /* Check size is power of 2 */
> +       if (size & (size - 1))
> +               return SBI_EINVAL;
> +
> +       if (entry_id > 15)
> +               return SBI_EINVAL;
> +
> +       if (!(pma_region->flags & ANDES45_PMACFG_ETYP_NAPOT))
> +               return SBI_EINVAL;
> +
> +       if ((addr & (size - 1)) != 0)
> +               return SBI_EINVAL;
> +
> +       pma_cfg_addr = entry_id / 8 ? ANDES45_PMACFG_0 + 2 : ANDES45_PMACFG_0;
> +       pmacfg_val = andes45_pma_read_cfg(pma_cfg_addr);
> +       pmaxcfg = (char *)&pmacfg_val + (entry_id % 8);
> +       *pmaxcfg = 0;
> +       *pmaxcfg = pma_region->flags;
> +
> +       andes45_pma_write_cfg(pma_cfg_addr, pmacfg_val);
> +
> +       pmaaddr = (addr >> 2) + (size >> 3) - 1;
> +
> +       andes45_pma_write_addr(ANDES45_PMAADDR_0 + entry_id, pmaaddr);
> +
> +       return andes45_pma_read_addr(ANDES45_PMAADDR_0 + entry_id) == pmaaddr ?
> +                       pmaaddr : SBI_EINVAL;
> +}
> +
> +static int andes45_fdt_pma_resv(void *fdt, const struct andes45_pma_region *pma,
> +                               unsigned int index, int parent)
> +{
> +       int na = fdt_address_cells(fdt, 0);
> +       int ns = fdt_size_cells(fdt, 0);
> +       static bool dma_default = false;
> +       fdt32_t addr_high, addr_low;
> +       fdt32_t size_high, size_low;
> +       int subnode, err;
> +       fdt32_t reg[4];
> +       fdt32_t *val;
> +       char name[32];
> +
> +       addr_high = (u64)pma->pa >> 32;
> +       addr_low = pma->pa;
> +       size_high = (u64)pma->size >> 32;
> +       size_low = pma->size;
> +
> +       if (na > 1 && addr_high)
> +               sbi_snprintf(name, sizeof(name),
> +                            "pma_resv%d@%x,%x", index,
> +                            addr_high, addr_low);
> +       else
> +               sbi_snprintf(name, sizeof(name),
> +                            "pma_resv%d@%x", index,
> +                            addr_low);
> +
> +       subnode = fdt_add_subnode(fdt, parent, name);
> +       if (subnode < 0)
> +               return subnode;
> +
> +       if (pma->shared_dma) {
> +               err = fdt_setprop_string(fdt, subnode, "compatible", "shared-dma-pool");
> +               if (err < 0)
> +                       return err;
> +       }
> +
> +       if (pma->no_map) {
> +               err = fdt_setprop_empty(fdt, subnode, "no-map");
> +               if (err < 0)
> +                       return err;
> +       }
> +
> +       /* Linux allows single linux,dma-default region. */
> +       if (pma->dma_default) {
> +               if (dma_default)
> +                       return SBI_EINVAL;
> +
> +               err = fdt_setprop_empty(fdt, subnode, "linux,dma-default");
> +               if (err < 0)
> +                       return err;
> +               dma_default = true;
> +       }
> +
> +       /* encode the <reg> property value */
> +       val = reg;
> +       if (na > 1)
> +               *val++ = cpu_to_fdt32(addr_high);
> +       *val++ = cpu_to_fdt32(addr_low);
> +       if (ns > 1)
> +               *val++ = cpu_to_fdt32(size_high);
> +       *val++ = cpu_to_fdt32(size_low);
> +
> +       err = fdt_setprop(fdt, subnode, "reg", reg,
> +                         (na + ns) * sizeof(fdt32_t));
> +       if (err < 0)
> +               return err;
> +
> +       return 0;
> +}
> +
> +static int andes45_fdt_reserved_memory_fixup(void *fdt,
> +                                            const struct andes45_pma_region *pma,
> +                                            unsigned int entry)
> +{
> +       int parent;
> +
> +       /* try to locate the reserved memory node */
> +       parent = fdt_path_offset(fdt, "/reserved-memory");
> +       if (parent < 0) {
> +               int na = fdt_address_cells(fdt, 0);
> +               int ns = fdt_size_cells(fdt, 0);
> +               int err;
> +
> +               /* if such node does not exist, create one */
> +               parent = fdt_add_subnode(fdt, 0, "reserved-memory");
> +               if (parent < 0)
> +                       return parent;
> +
> +               err = fdt_setprop_empty(fdt, parent, "ranges");
> +               if (err < 0)
> +                       return err;
> +
> +               err = fdt_setprop_u32(fdt, parent, "#size-cells", ns);
> +               if (err < 0)
> +                       return err;
> +
> +               err = fdt_setprop_u32(fdt, parent, "#address-cells", na);
> +               if (err < 0)
> +                       return err;
> +       }
> +
> +       return andes45_fdt_pma_resv(fdt, pma, entry, parent);
> +}
> +
> +int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
> +                             unsigned int pma_regions_count)
> +{
> +       unsigned long mmsc = csr_read(ANDES45_CSR_MMSC_CFG);
> +       unsigned int dt_populate_cnt;
> +       unsigned int i, j;
> +       unsigned long pa;
> +       void *fdt;
> +       int ret;
> +
> +       if (!pma_regions || !pma_regions_count)
> +               return 0;
> +
> +       if (pma_regions_count > ANDES45_MAX_PMA_REGIONS)
> +               return SBI_EINVAL;
> +
> +       if ((mmsc & ANDES45_CSR_MMSC_PPMA_OFFSET) == 0)
> +               return SBI_ENOTSUPP;
> +
> +       /* Configure the PMA regions */
> +       for (i = 0; i < pma_regions_count; i++) {
> +               pa = andes45_pma_setup(&pma_regions[i], i);
> +               if (pa == SBI_EINVAL)
> +                       return SBI_EINVAL;
> +       }
> +
> +       dt_populate_cnt = 0;
> +       for (i = 0; i < pma_regions_count; i++) {
> +               if (!pma_regions[i].dt_populate)
> +                       continue;
> +               dt_populate_cnt++;
> +       }
> +
> +       if (!dt_populate_cnt)
> +               return 0;
> +
> +       fdt = fdt_get_address();
> +
> +       ret = fdt_open_into(fdt, fdt, fdt_totalsize(fdt) + (64 * dt_populate_cnt));
> +       if (ret < 0)
> +               return ret;
> +
> +       for (i = 0, j = 0; i < pma_regions_count; i++) {
> +               if (!pma_regions[i].dt_populate)
> +                       continue;
> +
> +               ret = andes45_fdt_reserved_memory_fixup(fdt, &pma_regions[i], j++);
> +               if (ret)
> +                       return ret;
> +       }
> +
> +       return 0;
> +}
> diff --git a/platform/generic/andes/objects.mk b/platform/generic/andes/objects.mk
> index 28275ef..ea6b561 100644
> --- a/platform/generic/andes/objects.mk
> +++ b/platform/generic/andes/objects.mk
> @@ -4,3 +4,5 @@
>
>  carray-platform_override_modules-$(CONFIG_PLATFORM_ANDES_AE350) += andes_ae350
>  platform-objs-$(CONFIG_PLATFORM_ANDES_AE350) += andes/ae350.o andes/sleep.o
> +
> +platform-objs-$(CONFIG_ANDES45_PMA) += andes/andes45-pma.o
> diff --git a/platform/generic/include/andes/andes45_pma.h b/platform/generic/include/andes/andes45_pma.h
> new file mode 100644
> index 0000000..37ec77c
> --- /dev/null
> +++ b/platform/generic/include/andes/andes45_pma.h
> @@ -0,0 +1,48 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2023 Renesas Electronics Corp.
> + */
> +
> +#ifndef _ANDES45_PMA_H_
> +#define _ANDES45_PMA_H_
> +
> +#include <sbi/sbi_types.h>
> +
> +#define ANDES45_MAX_PMA_REGIONS                        16
> +
> +/* Naturally aligned power of 2 region */
> +#define ANDES45_PMACFG_ETYP_NAPOT              3
> +
> +/* Memory, Non-cacheable, Bufferable */
> +#define ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF  (3 << 2)
> +
> +/**
> + * struct andes45_pma_region - Describes PMA regions
> + *
> + * @pa: Address to be configured in the PMA
> + * @size: Size of the region
> + * @flags: Flags to be set for the PMA region
> + * @dt_populate: Boolean flag indicating if the DT entry should be
> + *               populated for the given PMA region
> + * @shared_dma: Boolean flag if set "shared-dma-pool" property will
> + *              be set in the DT node
> + * @no_map: Boolean flag if set "no-map" property will be set in the
> + *          DT node
> + * @dma_default: Boolean flag if set "linux,dma-default" property will
> + *              be set in the DT node. Note Linux expects single node
> + *              with this property set.
> + */
> +struct andes45_pma_region {
> +       unsigned long pa;
> +       unsigned long size;
> +       u8 flags:7;
> +       bool dt_populate;
> +       bool shared_dma;
> +       bool no_map;
> +       bool dma_default;
> +};
> +
> +int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
> +                             unsigned int pma_regions_count);
> +
> +#endif /* _ANDES45_PMA_H_ */
> --
> 2.17.1
>

[PATCH v5 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Anup Patel]

On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
<prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>
> On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
> ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
> and populate this region as PMA reserve DT node with shared DMA pool and
> no-map flags set so that Linux drivers requesting any DMA'able memory go
> through this region.
>
> PMA node passed to the above stack:
>
>         reserved-memory {
>             #address-cells = <2>;
>             #size-cells = <2>;
>             ranges;
>
>             pma_resv0 at 58000000 {
>                 compatible = "shared-dma-pool";
>                 reg = <0x0 0x58000000 0x0 0x08000000>;
>                 no-map;
>                 linux,dma-default;
>             };
>         };
>
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

> ---
>  platform/generic/Kconfig                 |  1 +
>  platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
>  2 files changed, 22 insertions(+)
>
> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> index 39fb4e9..1f4f8e1 100644
> --- a/platform/generic/Kconfig
> +++ b/platform/generic/Kconfig
> @@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
>
>  config PLATFORM_RENESAS_RZFIVE
>         bool "Renesas RZ/Five support"
> +       select ANDES45_PMA
>         default n
>
>  config PLATFORM_SIFIVE_FU540
> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> index ee9c9c4..4d71d0d 100644
> --- a/platform/generic/renesas/rzfive/rzfive.c
> +++ b/platform/generic/renesas/rzfive/rzfive.c
> @@ -4,10 +4,30 @@
>   *
>   */
>
> +#include <andes/andes45_pma.h>
>  #include <platform_override.h>
>  #include <sbi/sbi_domain.h>
>  #include <sbi_utils/fdt/fdt_helper.h>
>
> +static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
> +       {
> +               .pa = 0x58000000,
> +               .size = 0x8000000,
> +               .flags = ANDES45_PMACFG_ETYP_NAPOT |
> +                        ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF,
> +               .dt_populate = true,
> +               .shared_dma = true,
> +               .no_map = true,
> +               .dma_default = true,
> +       },
> +};
> +
> +static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *match)
> +{
> +       return andes45_pma_setup_regions(renesas_rzfive_pma_regions,
> +                                        array_size(renesas_rzfive_pma_regions));
> +}
> +
>  int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
>  {
>         /*
> @@ -34,4 +54,5 @@ static const struct fdt_match renesas_rzfive_match[] = {
>  const struct platform_override renesas_rzfive = {
>         .match_table = renesas_rzfive_match,
>         .early_init = renesas_rzfive_early_init,
> +       .final_init = renesas_rzfive_final_init,
>  };
> --
> 2.17.1
>

[PATCH v5 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Anup Patel]

On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
<prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>
> I/O Coherence Port (IOCP) provides an AXI interface for connecting
> external non-caching masters, such as DMA controllers. The accesses
> from IOCP are coherent with D-Caches and L2 Cache.
>
> IOCP is a specification option and is disabled on the Renesas RZ/Five
> SoC due to this reason IP blocks using DMA will fail.
>
> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> software. Firstly OpenSBI configures the memory region as
> "Memory, Non-cacheable, Bufferable" and passes this region as a global
> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> allocations happen from this region and synchronization callbacks are
> implemented to synchronize when doing DMA transactions.
>
> RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
> errata should be applied to handle cache management.
>
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
> ---
>  platform/generic/include/andes/andes45.h | 25 +++++++++++++--
>  platform/generic/renesas/rzfive/rzfive.c | 41 ++++++++++++++++++++++++
>  2 files changed, 64 insertions(+), 2 deletions(-)
>
> diff --git a/platform/generic/include/andes/andes45.h b/platform/generic/include/andes/andes45.h
> index 08b3d18..d5d265f 100644
> --- a/platform/generic/include/andes/andes45.h
> +++ b/platform/generic/include/andes/andes45.h
> @@ -4,7 +4,28 @@
>  #define CSR_MARCHID_MICROID 0xfff
>
>  /* Memory and Miscellaneous Registers */
> -#define CSR_MCACHE_CTL 0x7ca
> -#define CSR_MCCTLCOMMAND 0x7cc
> +#define CSR_MCACHE_CTL         0x7ca
> +#define CSR_MCCTLCOMMAND       0x7cc
> +#define CSR_MICM_CFG           0xfc0
> +#define CSR_MDCM_CFG           0xfc1
> +#define CSR_MMSC_CFG           0xfc2
> +
> +#define MISA_20_OFFSET                 20
> +#define MISA_20_MASK                   (0x1 << MISA_20_OFFSET)
> +
> +#define MICM_CFG_ISZ_OFFSET            6
> +#define MICM_CFG_ISZ_MASK              (0x7  << MICM_CFG_ISZ_OFFSET)
> +
> +#define MDCM_CFG_DSZ_OFFSET            6
> +#define MDCM_CFG_DSZ_MASK              (0x7  << MDCM_CFG_DSZ_OFFSET)
> +
> +#define MMSC_CFG_CCTLCSR_OFFSET                16
> +#define MMSC_CFG_CCTLCSR_MASK          (0x1 << MMSC_CFG_CCTLCSR_OFFSET)
> +#define MMSC_IOCP_OFFSET                       47
> +#define MMSC_IOCP_MASK                 (0x1ULL << MMSC_IOCP_OFFSET)
> +
> +#define MCACHE_CTL_CCTL_SUEN_OFFSET    8
> +#define MCACHE_CTL_CCTL_SUEN_MASK      (0x1 << MCACHE_CTL_CCTL_SUEN_OFFSET)
> +
>
>  #endif /* _RISCV_ANDES45_H */
> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> index 4d71d0d..9530d28 100644
> --- a/platform/generic/renesas/rzfive/rzfive.c
> +++ b/platform/generic/renesas/rzfive/rzfive.c
> @@ -4,11 +4,14 @@
>   *
>   */
>
> +#include <andes/andes45.h>
>  #include <andes/andes45_pma.h>
>  #include <platform_override.h>
>  #include <sbi/sbi_domain.h>
>  #include <sbi_utils/fdt/fdt_helper.h>
>
> +#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND      0

The SBI vendor extension_id will be the same for all SoCs based on Andes
CPUs so the function_id space will be shared across all such SoCs.

How does Andes plan to manage the function_id space of SBI vendor extension ?

I suggest the following:
1) The function_id == 0 can be used to return an SoC identifier. For example,
     Renesas RzFive can have an SoC identifier "0".
2) Move RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to a
    separate header at platform/generic/include/andes/andes_sbi.h and
    change it's value to "1"

The above suggestion will help multiple SoC with Andes CPU share same
SBI function_id space for SBI vendor extension.

Regards,
Anup


> +
>  static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
>         {
>                 .pa = 0x58000000,
> @@ -28,6 +31,43 @@ static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *mat
>                                          array_size(renesas_rzfive_pma_regions));
>  }
>
> +static bool renesas_rzfive_cpu_cache_controlable(void)
> +{
> +       return (((csr_read(CSR_MICM_CFG) & MICM_CFG_ISZ_MASK) ||
> +                (csr_read(CSR_MDCM_CFG) & MDCM_CFG_DSZ_MASK)) &&
> +               (csr_read(CSR_MISA) & MISA_20_MASK) &&
> +               (csr_read(CSR_MMSC_CFG) & MMSC_CFG_CCTLCSR_MASK) &&
> +               (csr_read(CSR_MCACHE_CTL) & MCACHE_CTL_CCTL_SUEN_MASK));
> +}
> +
> +static bool renesas_rzfive_cpu_iocp_disabled(void)
> +{
> +       return (csr_read(CSR_MMSC_CFG) & MMSC_IOCP_MASK) ? false : true;
> +}
> +
> +static bool renesas_rzfive_apply_iocp_sw_workaround(void)
> +{
> +       return renesas_rzfive_cpu_cache_controlable() & renesas_rzfive_cpu_iocp_disabled();
> +}
> +
> +static int renesas_rzfive_vendor_ext_provider(long funcid,
> +                                             const struct sbi_trap_regs *regs,
> +                                             unsigned long *out_value,
> +                                             struct sbi_trap_info *out_trap,
> +                                             const struct fdt_match *match)
> +{
> +       switch (funcid) {
> +       case RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND:
> +               *out_value = renesas_rzfive_apply_iocp_sw_workaround();
> +               break;
> +
> +       default:
> +               break;
> +       }
> +
> +       return 0;
> +}
> +
>  int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
>  {
>         /*
> @@ -55,4 +95,5 @@ const struct platform_override renesas_rzfive = {
>         .match_table = renesas_rzfive_match,
>         .early_init = renesas_rzfive_early_init,
>         .final_init = renesas_rzfive_final_init,
> +       .vendor_ext_provider = renesas_rzfive_vendor_ext_provider,
>  };
> --
> 2.17.1
>

Re: [PATCH v5 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Conor Dooley]

On 20 February 2023 15:31:32 GMT, Anup Patel <anup@brainfault.org> wrote:
>On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
><prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>>
>> I/O Coherence Port (IOCP) provides an AXI interface for connecting
>> external non-caching masters, such as DMA controllers. The accesses
>> from IOCP are coherent with D-Caches and L2 Cache.
>>
>> IOCP is a specification option and is disabled on the Renesas RZ/Five
>> SoC due to this reason IP blocks using DMA will fail.
>>
>> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
>> software. Firstly OpenSBI configures the memory region as
>> "Memory, Non-cacheable, Bufferable" and passes this region as a global
>> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
>> allocations happen from this region and synchronization callbacks are
>> implemented to synchronize when doing DMA transactions.
>>
>> RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
>> errata should be applied to handle cache management.
>>
>> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
>> Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
>> ---
>>  platform/generic/include/andes/andes45.h | 25 +++++++++++++--
>>  platform/generic/renesas/rzfive/rzfive.c | 41 ++++++++++++++++++++++++
>>  2 files changed, 64 insertions(+), 2 deletions(-)
>>
>> diff --git a/platform/generic/include/andes/andes45.h b/platform/generic/include/andes/andes45.h
>> index 08b3d18..d5d265f 100644
>> --- a/platform/generic/include/andes/andes45.h
>> +++ b/platform/generic/include/andes/andes45.h
>> @@ -4,7 +4,28 @@
>>  #define CSR_MARCHID_MICROID 0xfff
>>
>>  /* Memory and Miscellaneous Registers */
>> -#define CSR_MCACHE_CTL 0x7ca
>> -#define CSR_MCCTLCOMMAND 0x7cc
>> +#define CSR_MCACHE_CTL         0x7ca
>> +#define CSR_MCCTLCOMMAND       0x7cc
>> +#define CSR_MICM_CFG           0xfc0
>> +#define CSR_MDCM_CFG           0xfc1
>> +#define CSR_MMSC_CFG           0xfc2
>> +
>> +#define MISA_20_OFFSET                 20
>> +#define MISA_20_MASK                   (0x1 << MISA_20_OFFSET)
>> +
>> +#define MICM_CFG_ISZ_OFFSET            6
>> +#define MICM_CFG_ISZ_MASK              (0x7  << MICM_CFG_ISZ_OFFSET)
>> +
>> +#define MDCM_CFG_DSZ_OFFSET            6
>> +#define MDCM_CFG_DSZ_MASK              (0x7  << MDCM_CFG_DSZ_OFFSET)
>> +
>> +#define MMSC_CFG_CCTLCSR_OFFSET                16
>> +#define MMSC_CFG_CCTLCSR_MASK          (0x1 << MMSC_CFG_CCTLCSR_OFFSET)
>> +#define MMSC_IOCP_OFFSET                       47
>> +#define MMSC_IOCP_MASK                 (0x1ULL << MMSC_IOCP_OFFSET)
>> +
>> +#define MCACHE_CTL_CCTL_SUEN_OFFSET    8
>> +#define MCACHE_CTL_CCTL_SUEN_MASK      (0x1 << MCACHE_CTL_CCTL_SUEN_OFFSET)
>> +
>>
>>  #endif /* _RISCV_ANDES45_H */
>> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
>> index 4d71d0d..9530d28 100644
>> --- a/platform/generic/renesas/rzfive/rzfive.c
>> +++ b/platform/generic/renesas/rzfive/rzfive.c
>> @@ -4,11 +4,14 @@
>>   *
>>   */
>>
>> +#include <andes/andes45.h>
>>  #include <andes/andes45_pma.h>
>>  #include <platform_override.h>
>>  #include <sbi/sbi_domain.h>
>>  #include <sbi_utils/fdt/fdt_helper.h>
>>
>> +#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND      0
>
>The SBI vendor extension_id will be the same for all SoCs based on Andes
>CPUs so the function_id space will be shared across all such SoCs.

Apologies for sticking my nose in here, but I'm a bit confused as to why is being added as an rz/five specific thing in the first place?
My uninformed view of the world sees other purchasers of the IP needing to apply this workaround too, no?

>How does Andes plan to manage the function_id space of SBI vendor extension ?
>
>I suggest the following:
>1) The function_id == 0 can be used to return an SoC identifier. For example,
>     Renesas RzFive can have an SoC identifier "0".
>2) Move RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to a
>    separate header at platform/generic/include/andes/andes_sbi.h and
>    change it's value to "1"
>
>The above suggestion will help multiple SoC with Andes CPU share same
>SBI function_id space for SBI vendor extension.

Sharing here makes complete sense to me, so that you don't end up duplicating this the next time someone buys the IP without the iocp.

Cheers,
Conor.

>
>
>> +
>>  static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
>>         {
>>                 .pa = 0x58000000,
>> @@ -28,6 +31,43 @@ static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *mat
>>                                          array_size(renesas_rzfive_pma_regions));
>>  }
>>
>> +static bool renesas_rzfive_cpu_cache_controlable(void)
>> +{
>> +       return (((csr_read(CSR_MICM_CFG) & MICM_CFG_ISZ_MASK) ||
>> +                (csr_read(CSR_MDCM_CFG) & MDCM_CFG_DSZ_MASK)) &&
>> +               (csr_read(CSR_MISA) & MISA_20_MASK) &&
>> +               (csr_read(CSR_MMSC_CFG) & MMSC_CFG_CCTLCSR_MASK) &&
>> +               (csr_read(CSR_MCACHE_CTL) & MCACHE_CTL_CCTL_SUEN_MASK));
>> +}
>> +
>> +static bool renesas_rzfive_cpu_iocp_disabled(void)
>> +{
>> +       return (csr_read(CSR_MMSC_CFG) & MMSC_IOCP_MASK) ? false : true;
>> +}
>> +
>> +static bool renesas_rzfive_apply_iocp_sw_workaround(void)
>> +{
>> +       return renesas_rzfive_cpu_cache_controlable() & renesas_rzfive_cpu_iocp_disabled();
>> +}
>> +
>> +static int renesas_rzfive_vendor_ext_provider(long funcid,
>> +                                             const struct sbi_trap_regs *regs,
>> +                                             unsigned long *out_value,
>> +                                             struct sbi_trap_info *out_trap,
>> +                                             const struct fdt_match *match)
>> +{
>> +       switch (funcid) {
>> +       case RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND:
>> +               *out_value = renesas_rzfive_apply_iocp_sw_workaround();
>> +               break;
>> +
>> +       default:
>> +               break;
>> +       }
>> +
>> +       return 0;
>> +}
>> +
>>  int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
>>  {
>>         /*
>> @@ -55,4 +95,5 @@ const struct platform_override renesas_rzfive = {
>>         .match_table = renesas_rzfive_match,
>>         .early_init = renesas_rzfive_early_init,
>>         .final_init = renesas_rzfive_final_init,
>> +       .vendor_ext_provider = renesas_rzfive_vendor_ext_provider,
>>  };
>> --
>> 2.17.1
>>
>

[PATCH v5 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Yu-Chien Peter Lin]

On Mon, Feb 20, 2023 at 09:01:32PM +0530, Anup Patel wrote:
> On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
> <prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
> >
> > I/O Coherence Port (IOCP) provides an AXI interface for connecting
> > external non-caching masters, such as DMA controllers. The accesses
> > from IOCP are coherent with D-Caches and L2 Cache.
> >
> > IOCP is a specification option and is disabled on the Renesas RZ/Five
> > SoC due to this reason IP blocks using DMA will fail.
> >
> > As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> > software. Firstly OpenSBI configures the memory region as
> > "Memory, Non-cacheable, Bufferable" and passes this region as a global
> > shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> > allocations happen from this region and synchronization callbacks are
> > implemented to synchronize when doing DMA transactions.
> >
> > RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
> > errata should be applied to handle cache management.
> >
> > Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> > Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
> > ---
> >  platform/generic/include/andes/andes45.h | 25 +++++++++++++--
> >  platform/generic/renesas/rzfive/rzfive.c | 41 ++++++++++++++++++++++++
> >  2 files changed, 64 insertions(+), 2 deletions(-)
> >
> > diff --git a/platform/generic/include/andes/andes45.h b/platform/generic/include/andes/andes45.h
> > index 08b3d18..d5d265f 100644
> > --- a/platform/generic/include/andes/andes45.h
> > +++ b/platform/generic/include/andes/andes45.h
> > @@ -4,7 +4,28 @@
> >  #define CSR_MARCHID_MICROID 0xfff
> >
> >  /* Memory and Miscellaneous Registers */
> > -#define CSR_MCACHE_CTL 0x7ca
> > -#define CSR_MCCTLCOMMAND 0x7cc
> > +#define CSR_MCACHE_CTL         0x7ca
> > +#define CSR_MCCTLCOMMAND       0x7cc
> > +#define CSR_MICM_CFG           0xfc0
> > +#define CSR_MDCM_CFG           0xfc1
> > +#define CSR_MMSC_CFG           0xfc2
> > +
> > +#define MISA_20_OFFSET                 20
> > +#define MISA_20_MASK                   (0x1 << MISA_20_OFFSET)
> > +
> > +#define MICM_CFG_ISZ_OFFSET            6
> > +#define MICM_CFG_ISZ_MASK              (0x7  << MICM_CFG_ISZ_OFFSET)
> > +
> > +#define MDCM_CFG_DSZ_OFFSET            6
> > +#define MDCM_CFG_DSZ_MASK              (0x7  << MDCM_CFG_DSZ_OFFSET)
> > +
> > +#define MMSC_CFG_CCTLCSR_OFFSET                16
> > +#define MMSC_CFG_CCTLCSR_MASK          (0x1 << MMSC_CFG_CCTLCSR_OFFSET)
> > +#define MMSC_IOCP_OFFSET                       47
> > +#define MMSC_IOCP_MASK                 (0x1ULL << MMSC_IOCP_OFFSET)
> > +
> > +#define MCACHE_CTL_CCTL_SUEN_OFFSET    8
> > +#define MCACHE_CTL_CCTL_SUEN_MASK      (0x1 << MCACHE_CTL_CCTL_SUEN_OFFSET)
> > +
> >
> >  #endif /* _RISCV_ANDES45_H */
> > diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> > index 4d71d0d..9530d28 100644
> > --- a/platform/generic/renesas/rzfive/rzfive.c
> > +++ b/platform/generic/renesas/rzfive/rzfive.c
> > @@ -4,11 +4,14 @@
> >   *
> >   */
> >
> > +#include <andes/andes45.h>
> >  #include <andes/andes45_pma.h>
> >  #include <platform_override.h>
> >  #include <sbi/sbi_domain.h>
> >  #include <sbi_utils/fdt/fdt_helper.h>
> >
> > +#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND      0
> 
> The SBI vendor extension_id will be the same for all SoCs based on Andes
> CPUs so the function_id space will be shared across all such SoCs.
> 
> How does Andes plan to manage the function_id space of SBI vendor extension ?
> 
> I suggest the following:
> 1) The function_id == 0 can be used to return an SoC identifier. For example,
>      Renesas RzFive can have an SoC identifier "0".
> 2) Move RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to a
>     separate header at platform/generic/include/andes/andes_sbi.h and
>     change it's value to "1"
> 
> The above suggestion will help multiple SoC with Andes CPU share same
> SBI function_id space for SBI vendor extension.
> 
> Regards,
> Anup

Hi Anup,

Does this mean that all the SoC using Andes CPUs must have a unique
fid for each vendor SBI call function? If so, how do we make the new
vendors aware of this rule? As this won't be defined in the SBI spec.

The fid#0 - fid#24 have been used in our production code, as long as
the renesas_rzfive_vendor_ext_provider hook only gets called when the
root node compatible string is "renesas,r9a07g043f01", the functions
will be invoked correctly even though the fid conflicts with other
Andes SoCs?

Best regards,
Peter Lin

[PATCH v5 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Yu-Chien Peter Lin]

On Tue, Feb 21, 2023 at 10:06:31AM +0000, Conor Dooley wrote:
> 
> 
> On 20 February 2023 15:31:32 GMT, Anup Patel <anup@brainfault.org> wrote:
> >On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
> ><prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
> >>
> >> I/O Coherence Port (IOCP) provides an AXI interface for connecting
> >> external non-caching masters, such as DMA controllers. The accesses
> >> from IOCP are coherent with D-Caches and L2 Cache.
> >>
> >> IOCP is a specification option and is disabled on the Renesas RZ/Five
> >> SoC due to this reason IP blocks using DMA will fail.
> >>
> >> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> >> software. Firstly OpenSBI configures the memory region as
> >> "Memory, Non-cacheable, Bufferable" and passes this region as a global
> >> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> >> allocations happen from this region and synchronization callbacks are
> >> implemented to synchronize when doing DMA transactions.
> >>
> >> RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
> >> errata should be applied to handle cache management.
> >>
> >> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> >> Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>
> >> ---
> >>  platform/generic/include/andes/andes45.h | 25 +++++++++++++--
> >>  platform/generic/renesas/rzfive/rzfive.c | 41 ++++++++++++++++++++++++
> >>  2 files changed, 64 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/platform/generic/include/andes/andes45.h b/platform/generic/include/andes/andes45.h
> >> index 08b3d18..d5d265f 100644
> >> --- a/platform/generic/include/andes/andes45.h
> >> +++ b/platform/generic/include/andes/andes45.h
> >> @@ -4,7 +4,28 @@
> >>  #define CSR_MARCHID_MICROID 0xfff
> >>
> >>  /* Memory and Miscellaneous Registers */
> >> -#define CSR_MCACHE_CTL 0x7ca
> >> -#define CSR_MCCTLCOMMAND 0x7cc
> >> +#define CSR_MCACHE_CTL         0x7ca
> >> +#define CSR_MCCTLCOMMAND       0x7cc
> >> +#define CSR_MICM_CFG           0xfc0
> >> +#define CSR_MDCM_CFG           0xfc1
> >> +#define CSR_MMSC_CFG           0xfc2
> >> +
> >> +#define MISA_20_OFFSET                 20
> >> +#define MISA_20_MASK                   (0x1 << MISA_20_OFFSET)
> >> +
> >> +#define MICM_CFG_ISZ_OFFSET            6
> >> +#define MICM_CFG_ISZ_MASK              (0x7  << MICM_CFG_ISZ_OFFSET)
> >> +
> >> +#define MDCM_CFG_DSZ_OFFSET            6
> >> +#define MDCM_CFG_DSZ_MASK              (0x7  << MDCM_CFG_DSZ_OFFSET)
> >> +
> >> +#define MMSC_CFG_CCTLCSR_OFFSET                16
> >> +#define MMSC_CFG_CCTLCSR_MASK          (0x1 << MMSC_CFG_CCTLCSR_OFFSET)
> >> +#define MMSC_IOCP_OFFSET                       47
> >> +#define MMSC_IOCP_MASK                 (0x1ULL << MMSC_IOCP_OFFSET)
> >> +
> >> +#define MCACHE_CTL_CCTL_SUEN_OFFSET    8
> >> +#define MCACHE_CTL_CCTL_SUEN_MASK      (0x1 << MCACHE_CTL_CCTL_SUEN_OFFSET)
> >> +
> >>
> >>  #endif /* _RISCV_ANDES45_H */
> >> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> >> index 4d71d0d..9530d28 100644
> >> --- a/platform/generic/renesas/rzfive/rzfive.c
> >> +++ b/platform/generic/renesas/rzfive/rzfive.c
> >> @@ -4,11 +4,14 @@
> >>   *
> >>   */
> >>
> >> +#include <andes/andes45.h>
> >>  #include <andes/andes45_pma.h>
> >>  #include <platform_override.h>
> >>  #include <sbi/sbi_domain.h>
> >>  #include <sbi_utils/fdt/fdt_helper.h>
> >>
> >> +#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND      0
> >
> >The SBI vendor extension_id will be the same for all SoCs based on Andes
> >CPUs so the function_id space will be shared across all such SoCs.
> 
> Apologies for sticking my nose in here, but I'm a bit confused as to why is being added as an rz/five specific thing in the first place?
> My uninformed view of the world sees other purchasers of the IP needing to apply this workaround too, no?
> 
> >How does Andes plan to manage the function_id space of SBI vendor extension ?
> >
> >I suggest the following:
> >1) The function_id == 0 can be used to return an SoC identifier. For example,
> >     Renesas RzFive can have an SoC identifier "0".
> >2) Move RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to a
> >    separate header at platform/generic/include/andes/andes_sbi.h and
> >    change it's value to "1"
> >
> >The above suggestion will help multiple SoC with Andes CPU share same
> >SBI function_id space for SBI vendor extension.
> 
> Sharing here makes complete sense to me, so that you don't end up duplicating this the next time someone buys the IP without the iocp.
> 
> Cheers,
> Conor.

Hi Anup, Conor,

Apologies for any confusion caused, please disregard my previous email.
Sure we can go ahead with your suggestion of sharing the FID space with
all SoC based on Andes CPU.

Thanks,
Peter Lin

[PATCH v5 1/3] platform: generic: renesas: rzfive: Add support to configure the PMA

[Anup Patel]

On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
<prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>
> I/O Coherence Port (IOCP) provides an AXI interface for connecting
> external non-caching masters, such as DMA controllers. The accesses
> from IOCP are coherent with D-Caches and L2 Cache.
>
> IOCP is a specification option and is disabled on the Renesas RZ/Five
> SoC due to this reason IP blocks using DMA will fail.
>
> The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
> block that allows dynamic adjustment of memory attributes in the runtime.
> It contains a configurable amount of PMA entries implemented as CSR
> registers to control the attributes of memory locations in interest.
> Below are the memory attributes supported:
> * Device, Non-bufferable
> * Device, bufferable
> * Memory, Non-cacheable, Non-bufferable
> * Memory, Non-cacheable, Bufferable
> * Memory, Write-back, No-allocate
> * Memory, Write-back, Read-allocate
> * Memory, Write-back, Write-allocate
> * Memory, Write-back, Read and Write-allocate
>
> More info about PMA (section 10.3):
> Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf
>
> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> software. Firstly OpenSBI configures the memory region as
> "Memory, Non-cacheable, Bufferable" and passes this region as a global
> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> allocations happen from this region and synchronization callbacks are
> implemented to synchronize when doing DMA transactions.
>
> Example PMA region passed as a DT node from OpenSBI:
>     reserved-memory {
>         #address-cells = <2>;
>         #size-cells = <2>;
>         ranges;
>
>         pma_resv0 at 58000000 {
>             compatible = "shared-dma-pool";
>             reg = <0x0 0x58000000 0x0 0x08000000>;
>             no-map;
>             linux,dma-default;
>         };
>     };
>
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

Applied this patch to the riscv/opensbi repo.

Thanks,
Anup

> ---
>  platform/generic/Kconfig                     |   2 +
>  platform/generic/andes/Kconfig               |   5 +
>  platform/generic/andes/andes45-pma.c         | 350 +++++++++++++++++++
>  platform/generic/andes/objects.mk            |   2 +
>  platform/generic/include/andes/andes45_pma.h |  48 +++
>  5 files changed, 407 insertions(+)
>  create mode 100644 platform/generic/andes/Kconfig
>  create mode 100644 platform/generic/andes/andes45-pma.c
>  create mode 100644 platform/generic/include/andes/andes45_pma.h
>
> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> index c7f198a..39fb4e9 100644
> --- a/platform/generic/Kconfig
> +++ b/platform/generic/Kconfig
> @@ -50,4 +50,6 @@ config PLATFORM_STARFIVE_JH7110
>         bool "StarFive JH7110 support"
>         default n
>
> +source "$(OPENSBI_SRC_DIR)/platform/generic/andes/Kconfig"
> +
>  endif
> diff --git a/platform/generic/andes/Kconfig b/platform/generic/andes/Kconfig
> new file mode 100644
> index 0000000..3ad4e4c
> --- /dev/null
> +++ b/platform/generic/andes/Kconfig
> @@ -0,0 +1,5 @@
> +# SPDX-License-Identifier: BSD-2-Clause
> +
> +config ANDES45_PMA
> +       bool "Andes PMA support"
> +       default n
> diff --git a/platform/generic/andes/andes45-pma.c b/platform/generic/andes/andes45-pma.c
> new file mode 100644
> index 0000000..2745bc3
> --- /dev/null
> +++ b/platform/generic/andes/andes45-pma.c
> @@ -0,0 +1,350 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (C) 2023 Renesas Electronics Corp.
> + *
> + * Copyright (c) 2020 Andes Technology Corporation
> + *
> + * Authors:
> + *      Nick Hu <nickhu@andestech.com>
> + *      Nylon Chen <nylon7@andestech.com>
> + *      Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> + */
> +
> +#include <andes/andes45_pma.h>
> +#include <libfdt.h>
> +#include <sbi/riscv_asm.h>
> +#include <sbi/riscv_io.h>
> +#include <sbi/sbi_console.h>
> +#include <sbi/sbi_error.h>
> +#include <sbi_utils/fdt/fdt_helper.h>
> +
> +/* Configuration Registers */
> +#define ANDES45_CSR_MMSC_CFG           0xFC2
> +#define ANDES45_CSR_MMSC_PPMA_OFFSET   (1 << 30)
> +
> +#define ANDES45_PMAADDR_0              0xBD0
> +
> +#define ANDES45_PMACFG_0               0xBC0
> +
> +static inline unsigned long andes45_pma_read_cfg(unsigned int pma_cfg_off)
> +{
> +#define switchcase_pma_cfg_read(__pma_cfg_off, __val)          \
> +       case __pma_cfg_off:                                     \
> +               __val = csr_read(__pma_cfg_off);                \
> +               break;
> +#define switchcase_pma_cfg_read_2(__pma_cfg_off, __val)                \
> +       switchcase_pma_cfg_read(__pma_cfg_off + 0, __val)       \
> +       switchcase_pma_cfg_read(__pma_cfg_off + 2, __val)
> +
> +       unsigned long ret = 0;
> +
> +       switch (pma_cfg_off) {
> +       switchcase_pma_cfg_read_2(ANDES45_PMACFG_0, ret)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
> +               break;
> +       }
> +
> +       return ret;
> +
> +#undef switchcase_pma_cfg_read_2
> +#undef switchcase_pma_cfg_read
> +}
> +
> +static inline void andes45_pma_write_cfg(unsigned int pma_cfg_off, unsigned long val)
> +{
> +#define switchcase_pma_cfg_write(__pma_cfg_off, __val)         \
> +       case __pma_cfg_off:                                     \
> +               csr_write(__pma_cfg_off, __val);                \
> +               break;
> +#define switchcase_pma_cfg_write_2(__pma_cfg_off, __val)       \
> +       switchcase_pma_cfg_write(__pma_cfg_off + 0, __val)      \
> +       switchcase_pma_cfg_write(__pma_cfg_off + 2, __val)
> +
> +       switch (pma_cfg_off) {
> +       switchcase_pma_cfg_write_2(ANDES45_PMACFG_0, val)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
> +               break;
> +       }
> +
> +#undef switchcase_pma_cfg_write_2
> +#undef switchcase_pma_cfg_write
> +}
> +
> +static inline void andes45_pma_write_addr(unsigned int pma_addr_off, unsigned long val)
> +{
> +#define switchcase_pma_write(__pma_addr_off, __val)            \
> +       case __pma_addr_off:                                    \
> +               csr_write(__pma_addr_off, __val);               \
> +               break;
> +#define switchcase_pma_write_2(__pma_addr_off, __val)          \
> +       switchcase_pma_write(__pma_addr_off + 0, __val)         \
> +       switchcase_pma_write(__pma_addr_off + 1, __val)
> +#define switchcase_pma_write_4(__pma_addr_off, __val)          \
> +       switchcase_pma_write_2(__pma_addr_off + 0, __val)       \
> +       switchcase_pma_write_2(__pma_addr_off + 2, __val)
> +#define switchcase_pma_write_8(__pma_addr_off, __val)          \
> +       switchcase_pma_write_4(__pma_addr_off + 0, __val)       \
> +       switchcase_pma_write_4(__pma_addr_off + 4, __val)
> +#define switchcase_pma_write_16(__pma_addr_off, __val)         \
> +       switchcase_pma_write_8(__pma_addr_off + 0, __val)       \
> +       switchcase_pma_write_8(__pma_addr_off + 8, __val)
> +
> +       switch (pma_addr_off) {
> +       switchcase_pma_write_16(ANDES45_PMAADDR_0, val)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
> +               break;
> +       }
> +
> +#undef switchcase_pma_write_16
> +#undef switchcase_pma_write_8
> +#undef switchcase_pma_write_4
> +#undef switchcase_pma_write_2
> +#undef switchcase_pma_write
> +}
> +
> +static inline unsigned long andes45_pma_read_addr(unsigned int pma_addr_off)
> +{
> +#define switchcase_pma_read(__pma_addr_off, __val)             \
> +       case __pma_addr_off:                                    \
> +               __val = csr_read(__pma_addr_off);               \
> +               break;
> +#define switchcase_pma_read_2(__pma_addr_off, __val)           \
> +       switchcase_pma_read(__pma_addr_off + 0, __val)          \
> +       switchcase_pma_read(__pma_addr_off + 1, __val)
> +#define switchcase_pma_read_4(__pma_addr_off, __val)           \
> +       switchcase_pma_read_2(__pma_addr_off + 0, __val)        \
> +       switchcase_pma_read_2(__pma_addr_off + 2, __val)
> +#define switchcase_pma_read_8(__pma_addr_off, __val)           \
> +       switchcase_pma_read_4(__pma_addr_off + 0, __val)        \
> +       switchcase_pma_read_4(__pma_addr_off + 4, __val)
> +#define switchcase_pma_read_16(__pma_addr_off, __val)          \
> +       switchcase_pma_read_8(__pma_addr_off + 0, __val)        \
> +       switchcase_pma_read_8(__pma_addr_off + 8, __val)
> +
> +       unsigned long ret = 0;
> +
> +       switch (pma_addr_off) {
> +       switchcase_pma_read_16(ANDES45_PMAADDR_0, ret)
> +
> +       default:
> +               sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
> +               break;
> +       }
> +
> +       return ret;
> +
> +#undef switchcase_pma_read_16
> +#undef switchcase_pma_read_8
> +#undef switchcase_pma_read_4
> +#undef switchcase_pma_read_2
> +#undef switchcase_pma_read
> +}
> +
> +static unsigned long
> +andes45_pma_setup(const struct andes45_pma_region *pma_region,
> +                 unsigned int entry_id)
> +{
> +       unsigned long size = pma_region->size;
> +       unsigned long addr = pma_region->pa;
> +       unsigned int pma_cfg_addr;
> +       unsigned long pmacfg_val;
> +       unsigned long pmaaddr;
> +       char *pmaxcfg;
> +
> +       /* Check for 4KiB granularity */
> +       if (size < (1 << 12))
> +               return SBI_EINVAL;
> +
> +       /* Check size is power of 2 */
> +       if (size & (size - 1))
> +               return SBI_EINVAL;
> +
> +       if (entry_id > 15)
> +               return SBI_EINVAL;
> +
> +       if (!(pma_region->flags & ANDES45_PMACFG_ETYP_NAPOT))
> +               return SBI_EINVAL;
> +
> +       if ((addr & (size - 1)) != 0)
> +               return SBI_EINVAL;
> +
> +       pma_cfg_addr = entry_id / 8 ? ANDES45_PMACFG_0 + 2 : ANDES45_PMACFG_0;
> +       pmacfg_val = andes45_pma_read_cfg(pma_cfg_addr);
> +       pmaxcfg = (char *)&pmacfg_val + (entry_id % 8);
> +       *pmaxcfg = 0;
> +       *pmaxcfg = pma_region->flags;
> +
> +       andes45_pma_write_cfg(pma_cfg_addr, pmacfg_val);
> +
> +       pmaaddr = (addr >> 2) + (size >> 3) - 1;
> +
> +       andes45_pma_write_addr(ANDES45_PMAADDR_0 + entry_id, pmaaddr);
> +
> +       return andes45_pma_read_addr(ANDES45_PMAADDR_0 + entry_id) == pmaaddr ?
> +                       pmaaddr : SBI_EINVAL;
> +}
> +
> +static int andes45_fdt_pma_resv(void *fdt, const struct andes45_pma_region *pma,
> +                               unsigned int index, int parent)
> +{
> +       int na = fdt_address_cells(fdt, 0);
> +       int ns = fdt_size_cells(fdt, 0);
> +       static bool dma_default = false;
> +       fdt32_t addr_high, addr_low;
> +       fdt32_t size_high, size_low;
> +       int subnode, err;
> +       fdt32_t reg[4];
> +       fdt32_t *val;
> +       char name[32];
> +
> +       addr_high = (u64)pma->pa >> 32;
> +       addr_low = pma->pa;
> +       size_high = (u64)pma->size >> 32;
> +       size_low = pma->size;
> +
> +       if (na > 1 && addr_high)
> +               sbi_snprintf(name, sizeof(name),
> +                            "pma_resv%d@%x,%x", index,
> +                            addr_high, addr_low);
> +       else
> +               sbi_snprintf(name, sizeof(name),
> +                            "pma_resv%d@%x", index,
> +                            addr_low);
> +
> +       subnode = fdt_add_subnode(fdt, parent, name);
> +       if (subnode < 0)
> +               return subnode;
> +
> +       if (pma->shared_dma) {
> +               err = fdt_setprop_string(fdt, subnode, "compatible", "shared-dma-pool");
> +               if (err < 0)
> +                       return err;
> +       }
> +
> +       if (pma->no_map) {
> +               err = fdt_setprop_empty(fdt, subnode, "no-map");
> +               if (err < 0)
> +                       return err;
> +       }
> +
> +       /* Linux allows single linux,dma-default region. */
> +       if (pma->dma_default) {
> +               if (dma_default)
> +                       return SBI_EINVAL;
> +
> +               err = fdt_setprop_empty(fdt, subnode, "linux,dma-default");
> +               if (err < 0)
> +                       return err;
> +               dma_default = true;
> +       }
> +
> +       /* encode the <reg> property value */
> +       val = reg;
> +       if (na > 1)
> +               *val++ = cpu_to_fdt32(addr_high);
> +       *val++ = cpu_to_fdt32(addr_low);
> +       if (ns > 1)
> +               *val++ = cpu_to_fdt32(size_high);
> +       *val++ = cpu_to_fdt32(size_low);
> +
> +       err = fdt_setprop(fdt, subnode, "reg", reg,
> +                         (na + ns) * sizeof(fdt32_t));
> +       if (err < 0)
> +               return err;
> +
> +       return 0;
> +}
> +
> +static int andes45_fdt_reserved_memory_fixup(void *fdt,
> +                                            const struct andes45_pma_region *pma,
> +                                            unsigned int entry)
> +{
> +       int parent;
> +
> +       /* try to locate the reserved memory node */
> +       parent = fdt_path_offset(fdt, "/reserved-memory");
> +       if (parent < 0) {
> +               int na = fdt_address_cells(fdt, 0);
> +               int ns = fdt_size_cells(fdt, 0);
> +               int err;
> +
> +               /* if such node does not exist, create one */
> +               parent = fdt_add_subnode(fdt, 0, "reserved-memory");
> +               if (parent < 0)
> +                       return parent;
> +
> +               err = fdt_setprop_empty(fdt, parent, "ranges");
> +               if (err < 0)
> +                       return err;
> +
> +               err = fdt_setprop_u32(fdt, parent, "#size-cells", ns);
> +               if (err < 0)
> +                       return err;
> +
> +               err = fdt_setprop_u32(fdt, parent, "#address-cells", na);
> +               if (err < 0)
> +                       return err;
> +       }
> +
> +       return andes45_fdt_pma_resv(fdt, pma, entry, parent);
> +}
> +
> +int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
> +                             unsigned int pma_regions_count)
> +{
> +       unsigned long mmsc = csr_read(ANDES45_CSR_MMSC_CFG);
> +       unsigned int dt_populate_cnt;
> +       unsigned int i, j;
> +       unsigned long pa;
> +       void *fdt;
> +       int ret;
> +
> +       if (!pma_regions || !pma_regions_count)
> +               return 0;
> +
> +       if (pma_regions_count > ANDES45_MAX_PMA_REGIONS)
> +               return SBI_EINVAL;
> +
> +       if ((mmsc & ANDES45_CSR_MMSC_PPMA_OFFSET) == 0)
> +               return SBI_ENOTSUPP;
> +
> +       /* Configure the PMA regions */
> +       for (i = 0; i < pma_regions_count; i++) {
> +               pa = andes45_pma_setup(&pma_regions[i], i);
> +               if (pa == SBI_EINVAL)
> +                       return SBI_EINVAL;
> +       }
> +
> +       dt_populate_cnt = 0;
> +       for (i = 0; i < pma_regions_count; i++) {
> +               if (!pma_regions[i].dt_populate)
> +                       continue;
> +               dt_populate_cnt++;
> +       }
> +
> +       if (!dt_populate_cnt)
> +               return 0;
> +
> +       fdt = fdt_get_address();
> +
> +       ret = fdt_open_into(fdt, fdt, fdt_totalsize(fdt) + (64 * dt_populate_cnt));
> +       if (ret < 0)
> +               return ret;
> +
> +       for (i = 0, j = 0; i < pma_regions_count; i++) {
> +               if (!pma_regions[i].dt_populate)
> +                       continue;
> +
> +               ret = andes45_fdt_reserved_memory_fixup(fdt, &pma_regions[i], j++);
> +               if (ret)
> +                       return ret;
> +       }
> +
> +       return 0;
> +}
> diff --git a/platform/generic/andes/objects.mk b/platform/generic/andes/objects.mk
> index 28275ef..ea6b561 100644
> --- a/platform/generic/andes/objects.mk
> +++ b/platform/generic/andes/objects.mk
> @@ -4,3 +4,5 @@
>
>  carray-platform_override_modules-$(CONFIG_PLATFORM_ANDES_AE350) += andes_ae350
>  platform-objs-$(CONFIG_PLATFORM_ANDES_AE350) += andes/ae350.o andes/sleep.o
> +
> +platform-objs-$(CONFIG_ANDES45_PMA) += andes/andes45-pma.o
> diff --git a/platform/generic/include/andes/andes45_pma.h b/platform/generic/include/andes/andes45_pma.h
> new file mode 100644
> index 0000000..37ec77c
> --- /dev/null
> +++ b/platform/generic/include/andes/andes45_pma.h
> @@ -0,0 +1,48 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2023 Renesas Electronics Corp.
> + */
> +
> +#ifndef _ANDES45_PMA_H_
> +#define _ANDES45_PMA_H_
> +
> +#include <sbi/sbi_types.h>
> +
> +#define ANDES45_MAX_PMA_REGIONS                        16
> +
> +/* Naturally aligned power of 2 region */
> +#define ANDES45_PMACFG_ETYP_NAPOT              3
> +
> +/* Memory, Non-cacheable, Bufferable */
> +#define ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF  (3 << 2)
> +
> +/**
> + * struct andes45_pma_region - Describes PMA regions
> + *
> + * @pa: Address to be configured in the PMA
> + * @size: Size of the region
> + * @flags: Flags to be set for the PMA region
> + * @dt_populate: Boolean flag indicating if the DT entry should be
> + *               populated for the given PMA region
> + * @shared_dma: Boolean flag if set "shared-dma-pool" property will
> + *              be set in the DT node
> + * @no_map: Boolean flag if set "no-map" property will be set in the
> + *          DT node
> + * @dma_default: Boolean flag if set "linux,dma-default" property will
> + *              be set in the DT node. Note Linux expects single node
> + *              with this property set.
> + */
> +struct andes45_pma_region {
> +       unsigned long pa;
> +       unsigned long size;
> +       u8 flags:7;
> +       bool dt_populate;
> +       bool shared_dma;
> +       bool no_map;
> +       bool dma_default;
> +};
> +
> +int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
> +                             unsigned int pma_regions_count);
> +
> +#endif /* _ANDES45_PMA_H_ */
> --
> 2.17.1
>

[PATCH v5 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Anup Patel]

On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
<prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>
> On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
> ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
> and populate this region as PMA reserve DT node with shared DMA pool and
> no-map flags set so that Linux drivers requesting any DMA'able memory go
> through this region.
>
> PMA node passed to the above stack:
>
>         reserved-memory {
>             #address-cells = <2>;
>             #size-cells = <2>;
>             ranges;
>
>             pma_resv0 at 58000000 {
>                 compatible = "shared-dma-pool";
>                 reg = <0x0 0x58000000 0x0 0x08000000>;
>                 no-map;
>                 linux,dma-default;
>             };
>         };
>
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

Applied this patch to the riscv/opensbi repo.

Thanks,
Anup

> ---
>  platform/generic/Kconfig                 |  1 +
>  platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
>  2 files changed, 22 insertions(+)
>
> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> index 39fb4e9..1f4f8e1 100644
> --- a/platform/generic/Kconfig
> +++ b/platform/generic/Kconfig
> @@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
>
>  config PLATFORM_RENESAS_RZFIVE
>         bool "Renesas RZ/Five support"
> +       select ANDES45_PMA
>         default n
>
>  config PLATFORM_SIFIVE_FU540
> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> index ee9c9c4..4d71d0d 100644
> --- a/platform/generic/renesas/rzfive/rzfive.c
> +++ b/platform/generic/renesas/rzfive/rzfive.c
> @@ -4,10 +4,30 @@
>   *
>   */
>
> +#include <andes/andes45_pma.h>
>  #include <platform_override.h>
>  #include <sbi/sbi_domain.h>
>  #include <sbi_utils/fdt/fdt_helper.h>
>
> +static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
> +       {
> +               .pa = 0x58000000,
> +               .size = 0x8000000,
> +               .flags = ANDES45_PMACFG_ETYP_NAPOT |
> +                        ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF,
> +               .dt_populate = true,
> +               .shared_dma = true,
> +               .no_map = true,
> +               .dma_default = true,
> +       },
> +};
> +
> +static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *match)
> +{
> +       return andes45_pma_setup_regions(renesas_rzfive_pma_regions,
> +                                        array_size(renesas_rzfive_pma_regions));
> +}
> +
>  int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
>  {
>         /*
> @@ -34,4 +54,5 @@ static const struct fdt_match renesas_rzfive_match[] = {
>  const struct platform_override renesas_rzfive = {
>         .match_table = renesas_rzfive_match,
>         .early_init = renesas_rzfive_early_init,
> +       .final_init = renesas_rzfive_final_init,
>  };
> --
> 2.17.1
>

[PATCH v5 0/3] Renesas RZ/Five add support to configure PMA and add vendor specific SBI extension

[Anup Patel]

On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
<prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>
> Hi All,
>
> This patch series aims to add support to configure the PMA regions present
> on the Andes AX45MP core (andes45-pma.c which can be shared with other
> Andes vendors) and configures the required PMA region for RZ/Five SoC.
>
> This patch series also adds the SBI vendor extension
> RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to check if IOCP is present on
> the core and if un-available it also checks if SW workaround for cache
> management needs to be applied.
>
> v4 -> v5
> -> Included RB tags from Yu Chien Peter Lin
> -> Moved PMA configuration code into andes folder
> -> Rebased the changes on top of [0]
> [0] https://patchwork.ozlabs.org/project/opensbi/list/?series=341535
>
> RFC v3 -> v4
> -> Used bit field for flags in struct andes45_pma_region
> -> Renamed andes45-pma.h -> andes45_pma.h
> -> Included RB tag from Lin san for patch 2/3
> -> Moved CSR to platform/generic/include/andes/andes45.h
> -> Included sbi/sbi_error.h in andes45-pma.c to fix build issue
>
> RFC v2 -> RFC v3
> -> Fixed review comments pointed by Lin-san
>    * Used switch case macro
>    * Added a check to verify the PMA size is power-of-2
>    * Dropped un-needed calculation done in PMA code for NAPOT.
>    * Dropped MISA macro
> -> Fixed comments pointed by Himanshu
> -> Fixed check patch issue
>
> RFC v2 -> RFC v1
> -> Fixed review comments pointed by Lin-san
> -> Implemented PMA as library so that it can be shared
>
> RFC v1:
> https://patchwork.ozlabs.org/project/opensbi/patch/20221212094421.14556-1-prabhakar.mahadev-lad.rj at bp.renesas.com/
>
> Cheers,
> Prabhakar
>
> Lad Prabhakar (3):
>   platform: generic: renesas: rzfive: Add support to configure the PMA
>   platform: generic: renesas: rzfive: Configure the PMA region
>   platform: generic: renesas: rzfive: Add SBI EXT to check for enabling
>     IOCP errata

Applied first two patches of this series. Please work with Andes folks
and send v6 of PATCH3.

Thanks,
Anup

>
>  platform/generic/Kconfig                     |   3 +
>  platform/generic/andes/Kconfig               |   5 +
>  platform/generic/andes/andes45-pma.c         | 350 +++++++++++++++++++
>  platform/generic/andes/objects.mk            |   2 +
>  platform/generic/include/andes/andes45.h     |  25 +-
>  platform/generic/include/andes/andes45_pma.h |  48 +++
>  platform/generic/renesas/rzfive/rzfive.c     |  62 ++++
>  7 files changed, 493 insertions(+), 2 deletions(-)
>  create mode 100644 platform/generic/andes/Kconfig
>  create mode 100644 platform/generic/andes/andes45-pma.c
>  create mode 100644 platform/generic/include/andes/andes45_pma.h
>
> --
> 2.17.1
>

[PATCH v5 0/3] Renesas RZ/Five add support to configure PMA and add vendor specific SBI extension

[Lad, Prabhakar]

Hi Anup,

On Mon, Feb 27, 2023 at 6:13?AM Anup Patel <anup@brainfault.org> wrote:
>
> On Tue, Feb 14, 2023 at 3:21 AM Lad Prabhakar
> <prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
> >
> > Hi All,
> >
> > This patch series aims to add support to configure the PMA regions present
> > on the Andes AX45MP core (andes45-pma.c which can be shared with other
> > Andes vendors) and configures the required PMA region for RZ/Five SoC.
> >
> > This patch series also adds the SBI vendor extension
> > RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to check if IOCP is present on
> > the core and if un-available it also checks if SW workaround for cache
> > management needs to be applied.
> >
> > v4 -> v5
> > -> Included RB tags from Yu Chien Peter Lin
> > -> Moved PMA configuration code into andes folder
> > -> Rebased the changes on top of [0]
> > [0] https://patchwork.ozlabs.org/project/opensbi/list/?series=341535
> >
> > RFC v3 -> v4
> > -> Used bit field for flags in struct andes45_pma_region
> > -> Renamed andes45-pma.h -> andes45_pma.h
> > -> Included RB tag from Lin san for patch 2/3
> > -> Moved CSR to platform/generic/include/andes/andes45.h
> > -> Included sbi/sbi_error.h in andes45-pma.c to fix build issue
> >
> > RFC v2 -> RFC v3
> > -> Fixed review comments pointed by Lin-san
> >    * Used switch case macro
> >    * Added a check to verify the PMA size is power-of-2
> >    * Dropped un-needed calculation done in PMA code for NAPOT.
> >    * Dropped MISA macro
> > -> Fixed comments pointed by Himanshu
> > -> Fixed check patch issue
> >
> > RFC v2 -> RFC v1
> > -> Fixed review comments pointed by Lin-san
> > -> Implemented PMA as library so that it can be shared
> >
> > RFC v1:
> > https://patchwork.ozlabs.org/project/opensbi/patch/20221212094421.14556-1-prabhakar.mahadev-lad.rj at bp.renesas.com/
> >
> > Cheers,
> > Prabhakar
> >
> > Lad Prabhakar (3):
> >   platform: generic: renesas: rzfive: Add support to configure the PMA
> >   platform: generic: renesas: rzfive: Configure the PMA region
> >   platform: generic: renesas: rzfive: Add SBI EXT to check for enabling
> >     IOCP errata
>
> Applied first two patches of this series. Please work with Andes folks
> and send v6 of PATCH3.
>
Sure will do, thanks.

Cheers,
Prabhakar