[RFC PATCH v3 0/3] Renesas RZ/Five add support to configure PMA and add vendor specific SBI extension

[RFC PATCH v3 0/3] Renesas RZ/Five add support to configure PMA and add vendor specific SBI extension

[Lad Prabhakar]

Hi All,

This patch series aims to add support to configure the PMA regions present
on the Andes AX45MP core (andes45-pma.c which can be shared with other
Andes vendors) and configures the required PMA region for RZ/Five SoC.

This patch series also adds the SBI vendor extension
RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND to check if IOCP is present on
the core and if un-available it also checks if SW workaround for cache
management needs to be applied.

RFC v2 -> RFC v3
-> Fixed review comments pointed by Lin-san
   * Used switch case macro
   * Added a check to verify the PMA size is power-of-2
   * Dropped un-needed calculation done in PMA code for NAPOT.
   * Dropped MISA macro
-> Fixed comments pointed by Himanshu
-> Fixed check patch issue

RFC v2 -> RFC v1
-> Fixed review comments pointed by Lin-san
-> Implemented PMA as library so that it can be shared

RFC v1:
https://patchwork.ozlabs.org/project/opensbi/patch/20221212094421.14556-1-prabhakar.mahadev-lad.rj at bp.renesas.com/

Cheers,
Prabhakar

Lad Prabhakar (3):
  platform: generic: renesas: rzfive: Add support to configure the PMA
  platform: generic: renesas: rzfive: Configure the PMA region
  platform: generic: renesas: rzfive: Add SBI EXT to check for enabling
    IOCP errata

 platform/generic/Kconfig                      |   9 +
 platform/generic/include/andes45-pma.h        |  48 +++
 platform/generic/renesas/rzfive/andes45-pma.c | 350 ++++++++++++++++++
 platform/generic/renesas/rzfive/objects.mk    |   1 +
 platform/generic/renesas/rzfive/rzfive.c      |  90 +++++
 5 files changed, 498 insertions(+)
 create mode 100644 platform/generic/include/andes45-pma.h
 create mode 100644 platform/generic/renesas/rzfive/andes45-pma.c

-- 
2.17.1

[RFC PATCH v3 1/3] platform: generic: renesas: rzfive: Add support to configure the PMA

[Lad Prabhakar]

I/O Coherence Port (IOCP) provides an AXI interface for connecting
external non-caching masters, such as DMA controllers. The accesses
from IOCP are coherent with D-Caches and L2 Cache.

IOCP is a specification option and is disabled on the Renesas RZ/Five
SoC due to this reason IP blocks using DMA will fail.

The Andes AX45MP core has a Programmable Physical Memory Attributes (PMA)
block that allows dynamic adjustment of memory attributes in the runtime.
It contains a configurable amount of PMA entries implemented as CSR
registers to control the attributes of memory locations in interest.
Below are the memory attributes supported:
* Device, Non-bufferable
* Device, bufferable
* Memory, Non-cacheable, Non-bufferable
* Memory, Non-cacheable, Bufferable
* Memory, Write-back, No-allocate
* Memory, Write-back, Read-allocate
* Memory, Write-back, Write-allocate
* Memory, Write-back, Read and Write-allocate

More info about PMA (section 10.3):
Link: http://www.andestech.com/wp-content/uploads/AX45MP-1C-Rev.-5.0.0-Datasheet.pdf

As a workaround for SoCs with IOCP disabled CMO needs to be handled by
software. Firstly OpenSBI configures the memory region as
"Memory, Non-cacheable, Bufferable" and passes this region as a global
shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
allocations happen from this region and synchronization callbacks are
implemented to synchronize when doing DMA transactions.

Example PMA region passed as a DT node from OpenSBI:
    reserved-memory {
        #address-cells = <2>;
        #size-cells = <2>;
        ranges;

        pma_resv0 at 58000000 {
            compatible = "shared-dma-pool";
            reg = <0x0 0x58000000 0x0 0x08000000>;
            no-map;
            linux,dma-default;
        };
    };

Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
---
 platform/generic/Kconfig                      |   8 +
 platform/generic/include/andes45-pma.h        |  48 +++
 platform/generic/renesas/rzfive/andes45-pma.c | 350 ++++++++++++++++++
 platform/generic/renesas/rzfive/objects.mk    |   1 +
 4 files changed, 407 insertions(+)
 create mode 100644 platform/generic/include/andes45-pma.h
 create mode 100644 platform/generic/renesas/rzfive/andes45-pma.c

diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
index c7f198a..9ee9fe3 100644
--- a/platform/generic/Kconfig
+++ b/platform/generic/Kconfig
@@ -51,3 +51,11 @@ config PLATFORM_STARFIVE_JH7110
 	default n
 
 endif
+
+if PLATFORM_RENESAS_RZFIVE
+
+config ANDES45_PMA
+	bool "Andes PMA support"
+	default n
+
+endif
diff --git a/platform/generic/include/andes45-pma.h b/platform/generic/include/andes45-pma.h
new file mode 100644
index 0000000..06ed72b
--- /dev/null
+++ b/platform/generic/include/andes45-pma.h
@@ -0,0 +1,48 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2023 Renesas Electronics Corp.
+ */
+
+#ifndef _ANDES45_PMA_H_
+#define _ANDES45_PMA_H_
+
+#include <sbi/sbi_types.h>
+
+#define ANDES45_MAX_PMA_REGIONS			16
+
+/* Naturally aligned power of 2 region */
+#define ANDES45_PMACFG_ETYP_NAPOT		3
+
+/* Memory, Non-cacheable, Bufferable */
+#define ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF	(3 << 2)
+
+/**
+ * struct andes45_pma_region - Describes PMA regions
+ *
+ * @pa: Address to be configured in the PMA
+ * @size: Size of the region
+ * @flags: Flags to be set for the PMA region
+ * @dt_populate: Boolean flag indicating if the DT entry should be
+ *               populated for the given PMA region
+ * @shared_dma: Boolean flag if set "shared-dma-pool" property will
+ *              be set in the DT node
+ * @no_map: Boolean flag if set "no-map" property will be set in the
+ *          DT node
+ * @dma_default: Boolean flag if set "linux,dma-default" property will
+ *              be set in the DT node. Note Linux expects single node
+ *              with this property set.
+ */
+struct andes45_pma_region {
+	unsigned long pa;
+	unsigned long size;
+	u8 flags;
+	bool dt_populate;
+	bool shared_dma;
+	bool no_map;
+	bool dma_default;
+};
+
+int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
+			      unsigned int pma_regions_count);
+
+#endif /* _ANDES45_PMA_H_ */
diff --git a/platform/generic/renesas/rzfive/andes45-pma.c b/platform/generic/renesas/rzfive/andes45-pma.c
new file mode 100644
index 0000000..d9f1bf1
--- /dev/null
+++ b/platform/generic/renesas/rzfive/andes45-pma.c
@@ -0,0 +1,350 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2023 Renesas Electronics Corp.
+ *
+ * Copyright (c) 2020 Andes Technology Corporation
+ *
+ * Authors:
+ *      Nick Hu <nickhu@andestech.com>
+ *      Nylon Chen <nylon7@andestech.com>
+ *      Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
+ */
+
+#include <andes45-pma.h>
+#include <libfdt.h>
+#include <sbi/riscv_asm.h>
+#include <sbi/riscv_io.h>
+#include <sbi/sbi_console.h>
+#include <sbi/sbi_error.h>
+#include <sbi_utils/fdt/fdt_helper.h>
+
+/* Configuration Registers */
+#define ANDES45_CSR_MMSC_CFG		0xFC2
+#define ANDES45_CSR_MMSC_PPMA_OFFSET	(1 << 30)
+
+#define ANDES45_PMAADDR_0		0xBD0
+
+#define ANDES45_PMACFG_0		0xBC0
+
+static inline unsigned long andes45_pma_read_cfg(unsigned int pma_cfg_off)
+{
+#define switchcase_pma_cfg_read(__pma_cfg_off, __val)		\
+	case __pma_cfg_off:					\
+		__val = csr_read(__pma_cfg_off);		\
+		break;
+#define switchcase_pma_cfg_read_2(__pma_cfg_off, __val)		\
+	switchcase_pma_cfg_read(__pma_cfg_off + 0, __val)	\
+	switchcase_pma_cfg_read(__pma_cfg_off + 2, __val)
+
+	unsigned long ret = 0;
+
+	switch (pma_cfg_off) {
+	switchcase_pma_cfg_read_2(ANDES45_PMACFG_0, ret)
+
+	default:
+		sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
+		break;
+	}
+
+	return ret;
+
+#undef switchcase_pma_cfg_read_2
+#undef switchcase_pma_cfg_read
+}
+
+static inline void andes45_pma_write_cfg(unsigned int pma_cfg_off, unsigned long val)
+{
+#define switchcase_pma_cfg_write(__pma_cfg_off, __val)		\
+	case __pma_cfg_off:					\
+		csr_write(__pma_cfg_off, __val);		\
+		break;
+#define switchcase_pma_cfg_write_2(__pma_cfg_off, __val)	\
+	switchcase_pma_cfg_write(__pma_cfg_off + 0, __val)	\
+	switchcase_pma_cfg_write(__pma_cfg_off + 2, __val)
+
+	switch (pma_cfg_off) {
+	switchcase_pma_cfg_write_2(ANDES45_PMACFG_0, val)
+
+	default:
+		sbi_panic("%s: Unknown PMA CFG offset %#x", __func__, pma_cfg_off);
+		break;
+	}
+
+#undef switchcase_pma_cfg_write_2
+#undef switchcase_pma_cfg_write
+}
+
+static inline void andes45_pma_write_addr(unsigned int pma_addr_off, unsigned long val)
+{
+#define switchcase_pma_write(__pma_addr_off, __val)		\
+	case __pma_addr_off:					\
+		csr_write(__pma_addr_off, __val);		\
+		break;
+#define switchcase_pma_write_2(__pma_addr_off, __val)		\
+	switchcase_pma_write(__pma_addr_off + 0, __val)		\
+	switchcase_pma_write(__pma_addr_off + 1, __val)
+#define switchcase_pma_write_4(__pma_addr_off, __val)		\
+	switchcase_pma_write_2(__pma_addr_off + 0, __val)	\
+	switchcase_pma_write_2(__pma_addr_off + 2, __val)
+#define switchcase_pma_write_8(__pma_addr_off, __val)		\
+	switchcase_pma_write_4(__pma_addr_off + 0, __val)	\
+	switchcase_pma_write_4(__pma_addr_off + 4, __val)
+#define switchcase_pma_write_16(__pma_addr_off, __val)		\
+	switchcase_pma_write_8(__pma_addr_off + 0, __val)	\
+	switchcase_pma_write_8(__pma_addr_off + 8, __val)
+
+	switch (pma_addr_off) {
+	switchcase_pma_write_16(ANDES45_PMAADDR_0, val)
+
+	default:
+		sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
+		break;
+	}
+
+#undef switchcase_pma_write_16
+#undef switchcase_pma_write_8
+#undef switchcase_pma_write_4
+#undef switchcase_pma_write_2
+#undef switchcase_pma_write
+}
+
+static inline unsigned long andes45_pma_read_addr(unsigned int pma_addr_off)
+{
+#define switchcase_pma_read(__pma_addr_off, __val)		\
+	case __pma_addr_off:					\
+		__val = csr_read(__pma_addr_off);		\
+		break;
+#define switchcase_pma_read_2(__pma_addr_off, __val)		\
+	switchcase_pma_read(__pma_addr_off + 0, __val)		\
+	switchcase_pma_read(__pma_addr_off + 1, __val)
+#define switchcase_pma_read_4(__pma_addr_off, __val)		\
+	switchcase_pma_read_2(__pma_addr_off + 0, __val)	\
+	switchcase_pma_read_2(__pma_addr_off + 2, __val)
+#define switchcase_pma_read_8(__pma_addr_off, __val)		\
+	switchcase_pma_read_4(__pma_addr_off + 0, __val)	\
+	switchcase_pma_read_4(__pma_addr_off + 4, __val)
+#define switchcase_pma_read_16(__pma_addr_off, __val)		\
+	switchcase_pma_read_8(__pma_addr_off + 0, __val)	\
+	switchcase_pma_read_8(__pma_addr_off + 8, __val)
+
+	unsigned long ret = 0;
+
+	switch (pma_addr_off) {
+	switchcase_pma_read_16(ANDES45_PMAADDR_0, ret)
+
+	default:
+		sbi_panic("%s: Unknown PMA ADDR offset %#x", __func__, pma_addr_off);
+		break;
+	}
+
+	return ret;
+
+#undef switchcase_pma_read_16
+#undef switchcase_pma_read_8
+#undef switchcase_pma_read_4
+#undef switchcase_pma_read_2
+#undef switchcase_pma_read
+}
+
+static unsigned long
+andes45_pma_setup(const struct andes45_pma_region *pma_region,
+		  unsigned int entry_id)
+{
+	unsigned long size = pma_region->size;
+	unsigned long addr = pma_region->pa;
+	unsigned int pma_cfg_addr;
+	unsigned long pmacfg_val;
+	unsigned long pmaaddr;
+	char *pmaxcfg;
+
+	/* Check for 4KiB granularity */
+	if (size < (1 << 12))
+		return SBI_EINVAL;
+
+	/* Check size is power of 2 */
+	if (size & (size - 1))
+		return SBI_EINVAL;
+
+	if (pma_region->flags > 0xff || entry_id > 15)
+		return SBI_EINVAL;
+
+	if (!(pma_region->flags & ANDES45_PMACFG_ETYP_NAPOT))
+		return SBI_EINVAL;
+
+	if ((addr & (size - 1)) != 0)
+		return SBI_EINVAL;
+
+	pma_cfg_addr = entry_id / 8 ? ANDES45_PMACFG_0 + 2 : ANDES45_PMACFG_0;
+	pmacfg_val = andes45_pma_read_cfg(pma_cfg_addr);
+	pmaxcfg = (char *)&pmacfg_val + (entry_id % 8);
+	*pmaxcfg = 0;
+	*pmaxcfg = pma_region->flags;
+
+	andes45_pma_write_cfg(pma_cfg_addr, pmacfg_val);
+
+	pmaaddr = (addr >> 2) + (size >> 3) - 1;
+
+	andes45_pma_write_addr(ANDES45_PMAADDR_0 + entry_id, pmaaddr);
+
+	return andes45_pma_read_addr(ANDES45_PMAADDR_0 + entry_id) == pmaaddr ?
+			pmaaddr : SBI_EINVAL;
+}
+
+static int andes45_fdt_pma_resv(void *fdt, const struct andes45_pma_region *pma,
+				unsigned int index, int parent)
+{
+	int na = fdt_address_cells(fdt, 0);
+	int ns = fdt_size_cells(fdt, 0);
+	static bool dma_default = false;
+	fdt32_t addr_high, addr_low;
+	fdt32_t size_high, size_low;
+	int subnode, err;
+	fdt32_t reg[4];
+	fdt32_t *val;
+	char name[32];
+
+	addr_high = (u64)pma->pa >> 32;
+	addr_low = pma->pa;
+	size_high = (u64)pma->size >> 32;
+	size_low = pma->size;
+
+	if (na > 1 && addr_high)
+		sbi_snprintf(name, sizeof(name),
+			     "pma_resv%d@%x,%x", index,
+			     addr_high, addr_low);
+	else
+		sbi_snprintf(name, sizeof(name),
+			     "pma_resv%d@%x", index,
+			     addr_low);
+
+	subnode = fdt_add_subnode(fdt, parent, name);
+	if (subnode < 0)
+		return subnode;
+
+	if (pma->shared_dma) {
+		err = fdt_setprop_string(fdt, subnode, "compatible", "shared-dma-pool");
+		if (err < 0)
+			return err;
+	}
+
+	if (pma->no_map) {
+		err = fdt_setprop_empty(fdt, subnode, "no-map");
+		if (err < 0)
+			return err;
+	}
+
+	/* Linux allows single linux,dma-default region. */
+	if (pma->dma_default) {
+		if (dma_default)
+			return SBI_EINVAL;
+
+		err = fdt_setprop_empty(fdt, subnode, "linux,dma-default");
+		if (err < 0)
+			return err;
+		dma_default = true;
+	}
+
+	/* encode the <reg> property value */
+	val = reg;
+	if (na > 1)
+		*val++ = cpu_to_fdt32(addr_high);
+	*val++ = cpu_to_fdt32(addr_low);
+	if (ns > 1)
+		*val++ = cpu_to_fdt32(size_high);
+	*val++ = cpu_to_fdt32(size_low);
+
+	err = fdt_setprop(fdt, subnode, "reg", reg,
+			  (na + ns) * sizeof(fdt32_t));
+	if (err < 0)
+		return err;
+
+	return 0;
+}
+
+static int andes45_fdt_reserved_memory_fixup(void *fdt,
+					     const struct andes45_pma_region *pma,
+					     unsigned int entry)
+{
+	int parent;
+
+	/* try to locate the reserved memory node */
+	parent = fdt_path_offset(fdt, "/reserved-memory");
+	if (parent < 0) {
+		int na = fdt_address_cells(fdt, 0);
+		int ns = fdt_size_cells(fdt, 0);
+		int err;
+
+		/* if such node does not exist, create one */
+		parent = fdt_add_subnode(fdt, 0, "reserved-memory");
+		if (parent < 0)
+			return parent;
+
+		err = fdt_setprop_empty(fdt, parent, "ranges");
+		if (err < 0)
+			return err;
+
+		err = fdt_setprop_u32(fdt, parent, "#size-cells", ns);
+		if (err < 0)
+			return err;
+
+		err = fdt_setprop_u32(fdt, parent, "#address-cells", na);
+		if (err < 0)
+			return err;
+	}
+
+	return andes45_fdt_pma_resv(fdt, pma, entry, parent);
+}
+
+int andes45_pma_setup_regions(const struct andes45_pma_region *pma_regions,
+			      unsigned int pma_regions_count)
+{
+	unsigned long mmsc = csr_read(ANDES45_CSR_MMSC_CFG);
+	unsigned int dt_populate_cnt;
+	unsigned int i, j;
+	unsigned long pa;
+	void *fdt;
+	int ret;
+
+	if (!pma_regions || !pma_regions_count)
+		return 0;
+
+	if (pma_regions_count > ANDES45_MAX_PMA_REGIONS)
+		return SBI_EINVAL;
+
+	if ((mmsc & ANDES45_CSR_MMSC_PPMA_OFFSET) == 0)
+		return SBI_ENOTSUPP;
+
+	/* Configure the PMA regions */
+	for (i = 0; i < pma_regions_count; i++) {
+		pa = andes45_pma_setup(&pma_regions[i], i);
+		if (pa == SBI_EINVAL)
+			return SBI_EINVAL;
+	}
+
+	dt_populate_cnt = 0;
+	for (i = 0; i < pma_regions_count; i++) {
+		if (!pma_regions[i].dt_populate)
+			continue;
+		dt_populate_cnt++;
+	}
+
+	if (!dt_populate_cnt)
+		return 0;
+
+	fdt = fdt_get_address();
+
+	ret = fdt_open_into(fdt, fdt, fdt_totalsize(fdt) + (64 * dt_populate_cnt));
+	if (ret < 0)
+		return ret;
+
+	for (i = 0, j = 0; i < pma_regions_count; i++) {
+		if (!pma_regions[i].dt_populate)
+			continue;
+
+		ret = andes45_fdt_reserved_memory_fixup(fdt, &pma_regions[i], j++);
+		if (ret)
+			return ret;
+	}
+
+	return 0;
+}
diff --git a/platform/generic/renesas/rzfive/objects.mk b/platform/generic/renesas/rzfive/objects.mk
index 2e7e37f..fd9e7c4 100644
--- a/platform/generic/renesas/rzfive/objects.mk
+++ b/platform/generic/renesas/rzfive/objects.mk
@@ -6,3 +6,4 @@
 
 carray-platform_override_modules-$(CONFIG_PLATFORM_RENESAS_RZFIVE) += renesas_rzfive
 platform-objs-$(CONFIG_PLATFORM_RENESAS_RZFIVE) += renesas/rzfive/rzfive.o
+platform-objs-$(CONFIG_ANDES45_PMA) += renesas/rzfive/andes45-pma.o
-- 
2.17.1

[RFC PATCH v3 1/3] platform: generic: renesas: rzfive: Add support to configure the PMA

[Yu-Chien Peter Lin]

Hi Prabhakar,

> +static unsigned long
> +andes45_pma_setup(const struct andes45_pma_region *pma_region,
> +		  unsigned int entry_id)
> +{
> +	unsigned long size = pma_region->size;
> +	unsigned long addr = pma_region->pa;
> +	unsigned int pma_cfg_addr;
> +	unsigned long pmacfg_val;
> +	unsigned long pmaaddr;
> +	char *pmaxcfg;
> +
> +	/* Check for 4KiB granularity */
> +	if (size < (1 << 12))
> +		return SBI_EINVAL;
> +
> +	/* Check size is power of 2 */
> +	if (size & (size - 1))
> +		return SBI_EINVAL;
> +
> +	if (pma_region->flags > 0xff || entry_id > 15)

(pma_region->flags > 0xff) is always false as its type is unsigned char?
I guess we can drop the condition and specify the bit-field of flags as
7-bit wide (the MSB is reserved), i.e.

struct andes45_pma_region {
    unsigned long pa;
    unsigned long size;
    u8 flags:7;
[...]

So if someone sets the flags to a value greater than 0x7f, it will cause
a compilation error.

Regards,
Peter Lin

[RFC PATCH v3 1/3] platform: generic: renesas: rzfive: Add support to configure the PMA

[Lad, Prabhakar]

Hi Yu-Chien,

Thank you for the review.

On Sat, Jan 28, 2023 at 3:12 AM Yu-Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Hi Prabhakar,
>
> > +static unsigned long
> > +andes45_pma_setup(const struct andes45_pma_region *pma_region,
> > +               unsigned int entry_id)
> > +{
> > +     unsigned long size = pma_region->size;
> > +     unsigned long addr = pma_region->pa;
> > +     unsigned int pma_cfg_addr;
> > +     unsigned long pmacfg_val;
> > +     unsigned long pmaaddr;
> > +     char *pmaxcfg;
> > +
> > +     /* Check for 4KiB granularity */
> > +     if (size < (1 << 12))
> > +             return SBI_EINVAL;
> > +
> > +     /* Check size is power of 2 */
> > +     if (size & (size - 1))
> > +             return SBI_EINVAL;
> > +
> > +     if (pma_region->flags > 0xff || entry_id > 15)
>
> (pma_region->flags > 0xff) is always false as its type is unsigned char?
> I guess we can drop the condition and specify the bit-field of flags as
> 7-bit wide (the MSB is reserved), i.e.
>
> struct andes45_pma_region {
>     unsigned long pa;
>     unsigned long size;
>     u8 flags:7;
> [...]
>
> So if someone sets the flags to a value greater than 0x7f, it will cause
> a compilation error.
>
Good point, I will update this in the next version.

Cheers,
Prabhakar

[RFC PATCH v3 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Lad Prabhakar]

On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
and populate this region as PMA reserve DT node with shared DMA pool and
no-map flags set so that Linux drivers requesting any DMA'able memory go
through this region.

PMA node passed to the above stack:

        reserved-memory {
            #address-cells = <2>;
            #size-cells = <2>;
            ranges;

            pma_resv0 at 58000000 {
                compatible = "shared-dma-pool";
                reg = <0x0 0x58000000 0x0 0x08000000>;
                no-map;
                linux,dma-default;
            };
        };

Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
---
 platform/generic/Kconfig                 |  1 +
 platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
index 9ee9fe3..bc5c280 100644
--- a/platform/generic/Kconfig
+++ b/platform/generic/Kconfig
@@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
 
 config PLATFORM_RENESAS_RZFIVE
 	bool "Renesas RZ/Five support"
+	select ANDES45_PMA
 	default n
 
 config PLATFORM_SIFIVE_FU540
diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
index ee9c9c4..0a8c9f9 100644
--- a/platform/generic/renesas/rzfive/rzfive.c
+++ b/platform/generic/renesas/rzfive/rzfive.c
@@ -4,10 +4,30 @@
  *
  */
 
+#include <andes45-pma.h>
 #include <platform_override.h>
 #include <sbi/sbi_domain.h>
 #include <sbi_utils/fdt/fdt_helper.h>
 
+static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
+	{
+		.pa = 0x58000000,
+		.size = 0x8000000,
+		.flags = ANDES45_PMACFG_ETYP_NAPOT |
+			 ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF,
+		.dt_populate = true,
+		.shared_dma = true,
+		.no_map = true,
+		.dma_default = true,
+	},
+};
+
+static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *match)
+{
+	return andes45_pma_setup_regions(renesas_rzfive_pma_regions,
+					 array_size(renesas_rzfive_pma_regions));
+}
+
 int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
 {
 	/*
@@ -34,4 +54,5 @@ static const struct fdt_match renesas_rzfive_match[] = {
 const struct platform_override renesas_rzfive = {
 	.match_table = renesas_rzfive_match,
 	.early_init = renesas_rzfive_early_init,
+	.final_init = renesas_rzfive_final_init,
 };
-- 
2.17.1

[RFC PATCH v3 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Yu-Chien Peter Lin]

On Thu, Jan 26, 2023 at 09:58:31AM +0000, Lad Prabhakar wrote:
> On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
> ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
> and populate this region as PMA reserve DT node with shared DMA pool and
> no-map flags set so that Linux drivers requesting any DMA'able memory go
> through this region.
> 
> PMA node passed to the above stack:
> 
>         reserved-memory {
>             #address-cells = <2>;
>             #size-cells = <2>;
>             ranges;
> 
>             pma_resv0 at 58000000 {
>                 compatible = "shared-dma-pool";
>                 reg = <0x0 0x58000000 0x0 0x08000000>;
>                 no-map;
>                 linux,dma-default;
>             };
>         };
> 
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>

Reviewed-by: Yu Chien Peter Lin <peterlin@andestech.com>

Regards,
Peter Lin

> ---
>  platform/generic/Kconfig                 |  1 +
>  platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
>  2 files changed, 22 insertions(+)
> 
> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> index 9ee9fe3..bc5c280 100644
> --- a/platform/generic/Kconfig
> +++ b/platform/generic/Kconfig
> @@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
>  
>  config PLATFORM_RENESAS_RZFIVE
>  	bool "Renesas RZ/Five support"
> +	select ANDES45_PMA
>  	default n
>  
>  config PLATFORM_SIFIVE_FU540
> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> index ee9c9c4..0a8c9f9 100644
> --- a/platform/generic/renesas/rzfive/rzfive.c
> +++ b/platform/generic/renesas/rzfive/rzfive.c
> @@ -4,10 +4,30 @@
>   *
>   */
>  
> +#include <andes45-pma.h>
>  #include <platform_override.h>
>  #include <sbi/sbi_domain.h>
>  #include <sbi_utils/fdt/fdt_helper.h>
>  
> +static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
> +	{
> +		.pa = 0x58000000,
> +		.size = 0x8000000,
> +		.flags = ANDES45_PMACFG_ETYP_NAPOT |
> +			 ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF,
> +		.dt_populate = true,
> +		.shared_dma = true,
> +		.no_map = true,
> +		.dma_default = true,
> +	},
> +};
> +
> +static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *match)
> +{
> +	return andes45_pma_setup_regions(renesas_rzfive_pma_regions,
> +					 array_size(renesas_rzfive_pma_regions));
> +}
> +
>  int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
>  {
>  	/*
> @@ -34,4 +54,5 @@ static const struct fdt_match renesas_rzfive_match[] = {
>  const struct platform_override renesas_rzfive = {
>  	.match_table = renesas_rzfive_match,
>  	.early_init = renesas_rzfive_early_init,
> +	.final_init = renesas_rzfive_final_init,
>  };
> -- 
> 2.17.1
> 

[RFC PATCH v3 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Jessica Clarke]

On 26 Jan 2023, at 09:58, Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
> 
> On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
> ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
> and populate this region as PMA reserve DT node with shared DMA pool and
> no-map flags set so that Linux drivers requesting any DMA'able memory go
> through this region.
> 
> PMA node passed to the above stack:
> 
>        reserved-memory {
>            #address-cells = <2>;
>            #size-cells = <2>;
>            ranges;
> 
>            pma_resv0 at 58000000 {
>                compatible = "shared-dma-pool";
>                reg = <0x0 0x58000000 0x0 0x08000000>;
>                no-map;
>                linux,dma-default;
>            };
>        };
> 
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> ---
> platform/generic/Kconfig                 |  1 +
> platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
> 2 files changed, 22 insertions(+)
> 
> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> index 9ee9fe3..bc5c280 100644
> --- a/platform/generic/Kconfig
> +++ b/platform/generic/Kconfig
> @@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
> 
> config PLATFORM_RENESAS_RZFIVE
> 	bool "Renesas RZ/Five support"
> +	select ANDES45_PMA
> 	default n
> 
> config PLATFORM_SIFIVE_FU540
> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> index ee9c9c4..0a8c9f9 100644
> --- a/platform/generic/renesas/rzfive/rzfive.c
> +++ b/platform/generic/renesas/rzfive/rzfive.c
> @@ -4,10 +4,30 @@
>  *
>  */
> 
> +#include <andes45-pma.h>

This name looks inconsistent.

Jess

> #include <platform_override.h>
> #include <sbi/sbi_domain.h>
> #include <sbi_utils/fdt/fdt_helper.h>
> 
> +static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
> +	{
> +		.pa = 0x58000000,
> +		.size = 0x8000000,
> +		.flags = ANDES45_PMACFG_ETYP_NAPOT |
> +			 ANDES45_PMACFG_MTYP_MEM_NON_CACHE_BUF,
> +		.dt_populate = true,
> +		.shared_dma = true,
> +		.no_map = true,
> +		.dma_default = true,
> +	},
> +};
> +
> +static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *match)
> +{
> +	return andes45_pma_setup_regions(renesas_rzfive_pma_regions,
> +					 array_size(renesas_rzfive_pma_regions));
> +}
> +
> int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
> {
> 	/*
> @@ -34,4 +54,5 @@ static const struct fdt_match renesas_rzfive_match[] = {
> const struct platform_override renesas_rzfive = {
> 	.match_table = renesas_rzfive_match,
> 	.early_init = renesas_rzfive_early_init,
> +	.final_init = renesas_rzfive_final_init,
> };
> -- 
> 2.17.1
> 
> 
> -- 
> opensbi mailing list
> opensbi at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

[RFC PATCH v3 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Lad, Prabhakar]

Hi Jessica,

Thank you for the review.

On Sat, Jan 28, 2023 at 3:05 AM Jessica Clarke <jrtc27@jrtc27.com> wrote:
>
> On 26 Jan 2023, at 09:58, Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
> >
> > On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
> > ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
> > and populate this region as PMA reserve DT node with shared DMA pool and
> > no-map flags set so that Linux drivers requesting any DMA'able memory go
> > through this region.
> >
> > PMA node passed to the above stack:
> >
> >        reserved-memory {
> >            #address-cells = <2>;
> >            #size-cells = <2>;
> >            ranges;
> >
> >            pma_resv0 at 58000000 {
> >                compatible = "shared-dma-pool";
> >                reg = <0x0 0x58000000 0x0 0x08000000>;
> >                no-map;
> >                linux,dma-default;
> >            };
> >        };
> >
> > Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> > ---
> > platform/generic/Kconfig                 |  1 +
> > platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
> > 2 files changed, 22 insertions(+)
> >
> > diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> > index 9ee9fe3..bc5c280 100644
> > --- a/platform/generic/Kconfig
> > +++ b/platform/generic/Kconfig
> > @@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
> >
> > config PLATFORM_RENESAS_RZFIVE
> >       bool "Renesas RZ/Five support"
> > +     select ANDES45_PMA
> >       default n
> >
> > config PLATFORM_SIFIVE_FU540
> > diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> > index ee9c9c4..0a8c9f9 100644
> > --- a/platform/generic/renesas/rzfive/rzfive.c
> > +++ b/platform/generic/renesas/rzfive/rzfive.c
> > @@ -4,10 +4,30 @@
> >  *
> >  */
> >
> > +#include <andes45-pma.h>
>
> This name looks inconsistent.
>
As the RZ/Five has the Andes AX45MP core and the PMA code will be
re-used by other non-coherent  Andes AX45MP platforms hence the
andes45-pma.h (Also suggested by Andes maintainer)

Cheers,
Prabhakar

[RFC PATCH v3 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Jessica Clarke]

On 30 Jan 2023, at 15:09, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
> 
> Hi Jessica,
> 
> Thank you for the review.
> 
> On Sat, Jan 28, 2023 at 3:05 AM Jessica Clarke <jrtc27@jrtc27.com> wrote:
>> 
>> On 26 Jan 2023, at 09:58, Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>>> 
>>> On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
>>> ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
>>> and populate this region as PMA reserve DT node with shared DMA pool and
>>> no-map flags set so that Linux drivers requesting any DMA'able memory go
>>> through this region.
>>> 
>>> PMA node passed to the above stack:
>>> 
>>>       reserved-memory {
>>>           #address-cells = <2>;
>>>           #size-cells = <2>;
>>>           ranges;
>>> 
>>>           pma_resv0 at 58000000 {
>>>               compatible = "shared-dma-pool";
>>>               reg = <0x0 0x58000000 0x0 0x08000000>;
>>>               no-map;
>>>               linux,dma-default;
>>>           };
>>>       };
>>> 
>>> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
>>> ---
>>> platform/generic/Kconfig                 |  1 +
>>> platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
>>> 2 files changed, 22 insertions(+)
>>> 
>>> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
>>> index 9ee9fe3..bc5c280 100644
>>> --- a/platform/generic/Kconfig
>>> +++ b/platform/generic/Kconfig
>>> @@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
>>> 
>>> config PLATFORM_RENESAS_RZFIVE
>>>      bool "Renesas RZ/Five support"
>>> +     select ANDES45_PMA
>>>      default n
>>> 
>>> config PLATFORM_SIFIVE_FU540
>>> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
>>> index ee9c9c4..0a8c9f9 100644
>>> --- a/platform/generic/renesas/rzfive/rzfive.c
>>> +++ b/platform/generic/renesas/rzfive/rzfive.c
>>> @@ -4,10 +4,30 @@
>>> *
>>> */
>>> 
>>> +#include <andes45-pma.h>
>> 
>> This name looks inconsistent.
>> 
> As the RZ/Five has the Andes AX45MP core and the PMA code will be
> re-used by other non-coherent  Andes AX45MP platforms hence the
> andes45-pma.h (Also suggested by Andes maintainer)

I understand that. I meant the dash should be an underscore.

Jess

[RFC PATCH v3 2/3] platform: generic: renesas: rzfive: Configure the PMA region

[Lad, Prabhakar]

On Mon, Jan 30, 2023 at 4:29 PM Jessica Clarke <jrtc27@jrtc27.com> wrote:
>
> On 30 Jan 2023, at 15:09, Lad, Prabhakar <prabhakar.csengg@gmail.com> wrote:
> >
> > Hi Jessica,
> >
> > Thank you for the review.
> >
> > On Sat, Jan 28, 2023 at 3:05 AM Jessica Clarke <jrtc27@jrtc27.com> wrote:
> >>
> >> On 26 Jan 2023, at 09:58, Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
> >>>
> >>> On the Renesas RZ/Five SoC by default we want to configure 128MiB of memory
> >>> ranging from 0x58000000 as a non-cacheable + bufferable region in the PMA
> >>> and populate this region as PMA reserve DT node with shared DMA pool and
> >>> no-map flags set so that Linux drivers requesting any DMA'able memory go
> >>> through this region.
> >>>
> >>> PMA node passed to the above stack:
> >>>
> >>>       reserved-memory {
> >>>           #address-cells = <2>;
> >>>           #size-cells = <2>;
> >>>           ranges;
> >>>
> >>>           pma_resv0 at 58000000 {
> >>>               compatible = "shared-dma-pool";
> >>>               reg = <0x0 0x58000000 0x0 0x08000000>;
> >>>               no-map;
> >>>               linux,dma-default;
> >>>           };
> >>>       };
> >>>
> >>> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> >>> ---
> >>> platform/generic/Kconfig                 |  1 +
> >>> platform/generic/renesas/rzfive/rzfive.c | 21 +++++++++++++++++++++
> >>> 2 files changed, 22 insertions(+)
> >>>
> >>> diff --git a/platform/generic/Kconfig b/platform/generic/Kconfig
> >>> index 9ee9fe3..bc5c280 100644
> >>> --- a/platform/generic/Kconfig
> >>> +++ b/platform/generic/Kconfig
> >>> @@ -35,6 +35,7 @@ config PLATFORM_ANDES_AE350
> >>>
> >>> config PLATFORM_RENESAS_RZFIVE
> >>>      bool "Renesas RZ/Five support"
> >>> +     select ANDES45_PMA
> >>>      default n
> >>>
> >>> config PLATFORM_SIFIVE_FU540
> >>> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> >>> index ee9c9c4..0a8c9f9 100644
> >>> --- a/platform/generic/renesas/rzfive/rzfive.c
> >>> +++ b/platform/generic/renesas/rzfive/rzfive.c
> >>> @@ -4,10 +4,30 @@
> >>> *
> >>> */
> >>>
> >>> +#include <andes45-pma.h>
> >>
> >> This name looks inconsistent.
> >>
> > As the RZ/Five has the Andes AX45MP core and the PMA code will be
> > re-used by other non-coherent  Andes AX45MP platforms hence the
> > andes45-pma.h (Also suggested by Andes maintainer)
>
> I understand that. I meant the dash should be an underscore.
>
OK, I will rename it andes45_pma.h

Cheers,
Prabhakar

[RFC PATCH v3 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Lad Prabhakar]

I/O Coherence Port (IOCP) provides an AXI interface for connecting
external non-caching masters, such as DMA controllers. The accesses
from IOCP are coherent with D-Caches and L2 Cache.

IOCP is a specification option and is disabled on the Renesas RZ/Five
SoC due to this reason IP blocks using DMA will fail.

As a workaround for SoCs with IOCP disabled CMO needs to be handled by
software. Firstly OpenSBI configures the memory region as
"Memory, Non-cacheable, Bufferable" and passes this region as a global
shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
allocations happen from this region and synchronization callbacks are
implemented to synchronize when doing DMA transactions.

RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
errata should be applied to handle cache management.

Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
---
 platform/generic/renesas/rzfive/rzfive.c | 69 ++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
index 0a8c9f9..627df09 100644
--- a/platform/generic/renesas/rzfive/rzfive.c
+++ b/platform/generic/renesas/rzfive/rzfive.c
@@ -9,6 +9,34 @@
 #include <sbi/sbi_domain.h>
 #include <sbi_utils/fdt/fdt_helper.h>
 
+#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND	0
+
+/* AX45MP registers */
+#define AX45MP_CSR_MICM_CFG			0xfc0
+#define AX45MP_CSR_MDCM_CFG			0xfc1
+#define AX45MP_CSR_MMSC_CFG			0xfc2
+#define AX45MP_CSR_MCACHE_CTL			0x7ca
+
+/* AX45MP register bit offsets and masks */
+#define AX45MP_MISA_20_OFFSET			20
+#define AX45MP_MISA_20_MASK			(0x1 << AX45MP_MISA_20_OFFSET)
+
+#define AX45MP_MICM_CFG_ISZ_OFFSET		6
+#define AX45MP_MICM_CFG_ISZ_MASK		(0x7  << AX45MP_MICM_CFG_ISZ_OFFSET)
+
+#define AX45MP_MDCM_CFG_DSZ_OFFSET		6
+#define AX45MP_MDCM_CFG_DSZ_MASK		(0x7  << AX45MP_MDCM_CFG_DSZ_OFFSET)
+
+#define AX45MP_MMSC_CFG_CCTLCSR_OFFSET		16
+#define AX45MP_MMSC_CFG_CCTLCSR_MASK		(0x1 << AX45MP_MMSC_CFG_CCTLCSR_OFFSET)
+#define AX45MP_MMSC_IOCP_OFFSET			47
+#define AX45MP_MMSC_IOCP_MASK			(0x1ULL << AX45MP_MMSC_IOCP_OFFSET)
+
+#define AX45MP_MCACHE_CTL_CCTL_SUEN_OFFSET	8
+#define AX45MP_MCACHE_CTL_CCTL_SUEN_MASK	(0x1 << AX45MP_MCACHE_CTL_CCTL_SUEN_OFFSET)
+
+#define ANDESTECH_SBI_EXT_ANDES			0x900031e
+
 static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
 	{
 		.pa = 0x58000000,
@@ -28,6 +56,46 @@ static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *mat
 					 array_size(renesas_rzfive_pma_regions));
 }
 
+static bool renesas_rzfive_cpu_cache_controlable(void)
+{
+	return (((csr_read(AX45MP_CSR_MICM_CFG) & AX45MP_MICM_CFG_ISZ_MASK) ||
+		 (csr_read(AX45MP_CSR_MDCM_CFG) & AX45MP_MDCM_CFG_DSZ_MASK)) &&
+		(csr_read(CSR_MISA) & AX45MP_MISA_20_MASK) &&
+		(csr_read(AX45MP_CSR_MMSC_CFG) & AX45MP_MMSC_CFG_CCTLCSR_MASK) &&
+		(csr_read(AX45MP_CSR_MCACHE_CTL) & AX45MP_MCACHE_CTL_CCTL_SUEN_MASK));
+}
+
+static bool renesas_rzfive_cpu_iocp_disabled(void)
+{
+	return (csr_read(AX45MP_CSR_MMSC_CFG) & AX45MP_MMSC_IOCP_MASK) ? false : true;
+}
+
+static bool renesas_rzfive_apply_iocp_sw_workaround(void)
+{
+	return renesas_rzfive_cpu_cache_controlable() & renesas_rzfive_cpu_iocp_disabled();
+}
+
+static int renesas_rzfive_vendor_ext_provider(long extid, long funcid,
+					      const struct sbi_trap_regs *regs,
+					      unsigned long *out_value,
+					      struct sbi_trap_info *out_trap,
+					      const struct fdt_match *match)
+{
+	if (extid != ANDESTECH_SBI_EXT_ANDES)
+		return SBI_EINVAL;
+
+	switch (funcid) {
+	case RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND:
+		*out_value = renesas_rzfive_apply_iocp_sw_workaround();
+		break;
+
+	default:
+		break;
+	}
+
+	return 0;
+}
+
 int renesas_rzfive_early_init(bool cold_boot, const struct fdt_match *match)
 {
 	/*
@@ -55,4 +123,5 @@ const struct platform_override renesas_rzfive = {
 	.match_table = renesas_rzfive_match,
 	.early_init = renesas_rzfive_early_init,
 	.final_init = renesas_rzfive_final_init,
+	.vendor_ext_provider = renesas_rzfive_vendor_ext_provider,
 };
-- 
2.17.1

[RFC PATCH v3 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Yu-Chien Peter Lin]

Hi Prabhakar,

On Thu, Jan 26, 2023 at 09:58:32AM +0000, Lad Prabhakar wrote:
> I/O Coherence Port (IOCP) provides an AXI interface for connecting
> external non-caching masters, such as DMA controllers. The accesses
> from IOCP are coherent with D-Caches and L2 Cache.
> 
> IOCP is a specification option and is disabled on the Renesas RZ/Five
> SoC due to this reason IP blocks using DMA will fail.
> 
> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> software. Firstly OpenSBI configures the memory region as
> "Memory, Non-cacheable, Bufferable" and passes this region as a global
> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> allocations happen from this region and synchronization callbacks are
> implemented to synchronize when doing DMA transactions.
> 
> RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
> errata should be applied to handle cache management.
> 
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> ---
>  platform/generic/renesas/rzfive/rzfive.c | 69 ++++++++++++++++++++++++
>  1 file changed, 69 insertions(+)
> 
> diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> index 0a8c9f9..627df09 100644
> --- a/platform/generic/renesas/rzfive/rzfive.c
> +++ b/platform/generic/renesas/rzfive/rzfive.c
> @@ -9,6 +9,34 @@
>  #include <sbi/sbi_domain.h>
>  #include <sbi_utils/fdt/fdt_helper.h>
>  
> +#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND	0
> +
> +/* AX45MP registers */
> +#define AX45MP_CSR_MICM_CFG			0xfc0
> +#define AX45MP_CSR_MDCM_CFG			0xfc1
> +#define AX45MP_CSR_MMSC_CFG			0xfc2
> +#define AX45MP_CSR_MCACHE_CTL			0x7ca
> +
> +/* AX45MP register bit offsets and masks */
> +#define AX45MP_MISA_20_OFFSET			20
> +#define AX45MP_MISA_20_MASK			(0x1 << AX45MP_MISA_20_OFFSET)
> +
> +#define AX45MP_MICM_CFG_ISZ_OFFSET		6
> +#define AX45MP_MICM_CFG_ISZ_MASK		(0x7  << AX45MP_MICM_CFG_ISZ_OFFSET)
> +
> +#define AX45MP_MDCM_CFG_DSZ_OFFSET		6
> +#define AX45MP_MDCM_CFG_DSZ_MASK		(0x7  << AX45MP_MDCM_CFG_DSZ_OFFSET)
> +
> +#define AX45MP_MMSC_CFG_CCTLCSR_OFFSET		16
> +#define AX45MP_MMSC_CFG_CCTLCSR_MASK		(0x1 << AX45MP_MMSC_CFG_CCTLCSR_OFFSET)
> +#define AX45MP_MMSC_IOCP_OFFSET			47
> +#define AX45MP_MMSC_IOCP_MASK			(0x1ULL << AX45MP_MMSC_IOCP_OFFSET)
> +
> +#define AX45MP_MCACHE_CTL_CCTL_SUEN_OFFSET	8
> +#define AX45MP_MCACHE_CTL_CCTL_SUEN_MASK	(0x1 << AX45MP_MCACHE_CTL_CCTL_SUEN_OFFSET)

Could you move these CSR to platform/generic/include/andes/andes45.h
which is recently merged? Thanks.

> +
> +#define ANDESTECH_SBI_EXT_ANDES			0x900031e
> +
>  static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
>  	{
>  		.pa = 0x58000000,
> @@ -28,6 +56,46 @@ static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *mat
>  					 array_size(renesas_rzfive_pma_regions));
>  }
>  
> +static bool renesas_rzfive_cpu_cache_controlable(void)
> +{
> +	return (((csr_read(AX45MP_CSR_MICM_CFG) & AX45MP_MICM_CFG_ISZ_MASK) ||
> +		 (csr_read(AX45MP_CSR_MDCM_CFG) & AX45MP_MDCM_CFG_DSZ_MASK)) &&
> +		(csr_read(CSR_MISA) & AX45MP_MISA_20_MASK) &&
> +		(csr_read(AX45MP_CSR_MMSC_CFG) & AX45MP_MMSC_CFG_CCTLCSR_MASK) &&
> +		(csr_read(AX45MP_CSR_MCACHE_CTL) & AX45MP_MCACHE_CTL_CCTL_SUEN_MASK));
> +}
> +
> +static bool renesas_rzfive_cpu_iocp_disabled(void)
> +{
> +	return (csr_read(AX45MP_CSR_MMSC_CFG) & AX45MP_MMSC_IOCP_MASK) ? false : true;
> +}
> +
> +static bool renesas_rzfive_apply_iocp_sw_workaround(void)
> +{
> +	return renesas_rzfive_cpu_cache_controlable() & renesas_rzfive_cpu_iocp_disabled();
> +}
> +
> +static int renesas_rzfive_vendor_ext_provider(long extid, long funcid,
> +					      const struct sbi_trap_regs *regs,
> +					      unsigned long *out_value,
> +					      struct sbi_trap_info *out_trap,
> +					      const struct fdt_match *match)
> +{
> +	if (extid != ANDESTECH_SBI_EXT_ANDES)
> +		return SBI_EINVAL;

This causes compilation error without including sbi/sbi_error.h.

Regards,
Peter Lin

[RFC PATCH v3 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Lad, Prabhakar]

Hi Yu-Chien,

Thank you for the review.

On Sat, Jan 28, 2023 at 3:25 AM Yu-Chien Peter Lin
<peterlin@andestech.com> wrote:
>
> Hi Prabhakar,
>
> On Thu, Jan 26, 2023 at 09:58:32AM +0000, Lad Prabhakar wrote:
> > I/O Coherence Port (IOCP) provides an AXI interface for connecting
> > external non-caching masters, such as DMA controllers. The accesses
> > from IOCP are coherent with D-Caches and L2 Cache.
> >
> > IOCP is a specification option and is disabled on the Renesas RZ/Five
> > SoC due to this reason IP blocks using DMA will fail.
> >
> > As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> > software. Firstly OpenSBI configures the memory region as
> > "Memory, Non-cacheable, Bufferable" and passes this region as a global
> > shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> > allocations happen from this region and synchronization callbacks are
> > implemented to synchronize when doing DMA transactions.
> >
> > RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
> > errata should be applied to handle cache management.
> >
> > Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> > ---
> >  platform/generic/renesas/rzfive/rzfive.c | 69 ++++++++++++++++++++++++
> >  1 file changed, 69 insertions(+)
> >
> > diff --git a/platform/generic/renesas/rzfive/rzfive.c b/platform/generic/renesas/rzfive/rzfive.c
> > index 0a8c9f9..627df09 100644
> > --- a/platform/generic/renesas/rzfive/rzfive.c
> > +++ b/platform/generic/renesas/rzfive/rzfive.c
> > @@ -9,6 +9,34 @@
> >  #include <sbi/sbi_domain.h>
> >  #include <sbi_utils/fdt/fdt_helper.h>
> >
> > +#define RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND    0
> > +
> > +/* AX45MP registers */
> > +#define AX45MP_CSR_MICM_CFG                  0xfc0
> > +#define AX45MP_CSR_MDCM_CFG                  0xfc1
> > +#define AX45MP_CSR_MMSC_CFG                  0xfc2
> > +#define AX45MP_CSR_MCACHE_CTL                        0x7ca
> > +
> > +/* AX45MP register bit offsets and masks */
> > +#define AX45MP_MISA_20_OFFSET                        20
> > +#define AX45MP_MISA_20_MASK                  (0x1 << AX45MP_MISA_20_OFFSET)
> > +
> > +#define AX45MP_MICM_CFG_ISZ_OFFSET           6
> > +#define AX45MP_MICM_CFG_ISZ_MASK             (0x7  << AX45MP_MICM_CFG_ISZ_OFFSET)
> > +
> > +#define AX45MP_MDCM_CFG_DSZ_OFFSET           6
> > +#define AX45MP_MDCM_CFG_DSZ_MASK             (0x7  << AX45MP_MDCM_CFG_DSZ_OFFSET)
> > +
> > +#define AX45MP_MMSC_CFG_CCTLCSR_OFFSET               16
> > +#define AX45MP_MMSC_CFG_CCTLCSR_MASK         (0x1 << AX45MP_MMSC_CFG_CCTLCSR_OFFSET)
> > +#define AX45MP_MMSC_IOCP_OFFSET                      47
> > +#define AX45MP_MMSC_IOCP_MASK                        (0x1ULL << AX45MP_MMSC_IOCP_OFFSET)
> > +
> > +#define AX45MP_MCACHE_CTL_CCTL_SUEN_OFFSET   8
> > +#define AX45MP_MCACHE_CTL_CCTL_SUEN_MASK     (0x1 << AX45MP_MCACHE_CTL_CCTL_SUEN_OFFSET)
>
> Could you move these CSR to platform/generic/include/andes/andes45.h
> which is recently merged? Thanks.
>
Sure will do.

> > +
> > +#define ANDESTECH_SBI_EXT_ANDES                      0x900031e
> > +
> >  static const struct andes45_pma_region renesas_rzfive_pma_regions[] = {
> >       {
> >               .pa = 0x58000000,
> > @@ -28,6 +56,46 @@ static int renesas_rzfive_final_init(bool cold_boot, const struct fdt_match *mat
> >                                        array_size(renesas_rzfive_pma_regions));
> >  }
> >
> > +static bool renesas_rzfive_cpu_cache_controlable(void)
> > +{
> > +     return (((csr_read(AX45MP_CSR_MICM_CFG) & AX45MP_MICM_CFG_ISZ_MASK) ||
> > +              (csr_read(AX45MP_CSR_MDCM_CFG) & AX45MP_MDCM_CFG_DSZ_MASK)) &&
> > +             (csr_read(CSR_MISA) & AX45MP_MISA_20_MASK) &&
> > +             (csr_read(AX45MP_CSR_MMSC_CFG) & AX45MP_MMSC_CFG_CCTLCSR_MASK) &&
> > +             (csr_read(AX45MP_CSR_MCACHE_CTL) & AX45MP_MCACHE_CTL_CCTL_SUEN_MASK));
> > +}
> > +
> > +static bool renesas_rzfive_cpu_iocp_disabled(void)
> > +{
> > +     return (csr_read(AX45MP_CSR_MMSC_CFG) & AX45MP_MMSC_IOCP_MASK) ? false : true;
> > +}
> > +
> > +static bool renesas_rzfive_apply_iocp_sw_workaround(void)
> > +{
> > +     return renesas_rzfive_cpu_cache_controlable() & renesas_rzfive_cpu_iocp_disabled();
> > +}
> > +
> > +static int renesas_rzfive_vendor_ext_provider(long extid, long funcid,
> > +                                           const struct sbi_trap_regs *regs,
> > +                                           unsigned long *out_value,
> > +                                           struct sbi_trap_info *out_trap,
> > +                                           const struct fdt_match *match)
> > +{
> > +     if (extid != ANDESTECH_SBI_EXT_ANDES)
> > +             return SBI_EINVAL;
>
> This causes compilation error without including sbi/sbi_error.h.
>
Oops. I had a follow up patch which added this and I never hit this
issue. I will fix it.

Cheers,
Prabhakar

[RFC PATCH v3 3/3] platform: generic: renesas: rzfive: Add SBI EXT to check for enabling IOCP errata

[Lad, Prabhakar]

Hi Anup,

On Thu, Jan 26, 2023 at 9:58 AM Lad Prabhakar
<prabhakar.mahadev-lad.rj@bp.renesas.com> wrote:
>
> I/O Coherence Port (IOCP) provides an AXI interface for connecting
> external non-caching masters, such as DMA controllers. The accesses
> from IOCP are coherent with D-Caches and L2 Cache.
>
> IOCP is a specification option and is disabled on the Renesas RZ/Five
> SoC due to this reason IP blocks using DMA will fail.
>
> As a workaround for SoCs with IOCP disabled CMO needs to be handled by
> software. Firstly OpenSBI configures the memory region as
> "Memory, Non-cacheable, Bufferable" and passes this region as a global
> shared dma pool as a DT node. With DMA_GLOBAL_POOL enabled all DMA
> allocations happen from this region and synchronization callbacks are
> implemented to synchronize when doing DMA transactions.
>
> RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND SBI EXT checks if the IOCP
> errata should be applied to handle cache management.
>
> Signed-off-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com>
> ---
>  platform/generic/renesas/rzfive/rzfive.c | 69 ++++++++++++++++++++++++
>  1 file changed, 69 insertions(+)
>
<snip>
> +
> +static int renesas_rzfive_vendor_ext_provider(long extid, long funcid,
> +                                             const struct sbi_trap_regs *regs,
> +                                             unsigned long *out_value,
> +                                             struct sbi_trap_info *out_trap,
> +                                             const struct fdt_match *match)
> +{
> +       if (extid != ANDESTECH_SBI_EXT_ANDES)
Should we check extid here is this OK? Is returning SBI_EINVAL in case
of extid mismatch is ok or should we panic here?

> +               return SBI_EINVAL;
> +
> +       switch (funcid) {
> +       case RENESAS_RZFIVE_SBI_EXT_IOCP_SW_WORKAROUND:
> +               *out_value = renesas_rzfive_apply_iocp_sw_workaround();
> +               break;
> +
> +       default:
Is this a correct behavior or should we do a panic here?

Cheers,
Prabhakar