[PATCH v9 0/3] add zynqmp TCM bindings

[PATCH v9 0/3] add zynqmp TCM bindings

[Tanmay Shah]

Tightly-Coupled Memories(TCMs) are low-latency memory that provides
predictable instruction execution and predictable data load/store
timing. Each Cortex-R5F processor contains exclusive two 64 KB memory
banks on the ATCM and BTCM ports, for a total of 128 KB of memory.
In lockstep mode, both 128KB memory is accessible to the cluster.

As per ZynqMP Ultrascale+ Technical Reference Manual UG1085, following
is address space of TCM memory. The bindings in this patch series
introduces properties to accommodate following address space with
address translation between Linux and Cortex-R5 views.

|     |     |     |
| --- | --- | --- |
|      *Mode*        |   *R5 View* | *Linux view* |  Notes               |
| *Split Mode*       | *start addr*| *start addr* |                      |
| R5_0 ATCM (64 KB)  | 0x0000_0000 | 0xFFE0_0000  |                      |
| R5_0 BTCM (64 KB)  | 0x0002_0000 | 0xFFE2_0000  |                      |
| R5_1 ATCM (64 KB)  | 0x0000_0000 | 0xFFE9_0000  | alias of 0xFFE1_0000 |
| R5_1 BTCM (64 KB)  | 0x0002_0000 | 0xFFEB_0000  | alias of 0xFFE3_0000 |
|  ___               |     ___     |    ___       |                      |
| *Lockstep Mode*    |             |              |                      |
| R5_0 ATCM (128 KB) | 0x0000_0000 | 0xFFE0_0000  |                      |
| R5_0 BTCM (128 KB) | 0x0002_0000 | 0xFFE2_0000  |                      |

References:
UG1085 TCM address space:
https://docs.xilinx.com/r/en-US/ug1085-zynq-ultrascale-trm/Tightly-Coupled-Memory-Address-Map

Changes in v9:
  - Fix rproc lockstep dts
  - Introduce new API to request and release core1 TCM power-domains in
    lockstep mode. This will be used during prepare -> add_tcm_banks
    callback to enable TCM in lockstep mode.
  - Parse TCM from device-tree in lockstep mode and split mode in
    uniform way.
  - Fix TCM representation in device-tree in lockstep mode.
  - Fix comments as suggested

Changes in v8:
  - Remove use of pm_domains framework
  - Remove checking of pm_domain_id validation to power on/off tcm
  - Remove spurious change
  - parse power-domains property from device-tree and use EEMI calls
    to power on/off TCM instead of using pm domains framework

Changes in v7:
  - %s/pm_dev1/pm_dev_core0/r
  - %s/pm_dev_link1/pm_dev_core0_link/r
  - %s/pm_dev2/pm_dev_core1/r
  - %s/pm_dev_link2/pm_dev_core1_link/r
  - remove pm_domain_id check to move next patch
  - add comment about how 1st entry in pm domain list is used
  - fix loop when jump to fail_add_pm_domains loop
  - move checking of pm_domain_id from previous patch
  - fix mem_bank_data memory allocation

Changes in v6:
  - Introduce new node entry for r5f cluster split mode dts and
    keep it disabled by default.
  - Keep remoteproc lockstep mode enabled by default to maintian
    back compatibility.
  - Enable split mode only for zcu102 board to demo split mode use
  - Remove spurious change
  - Handle errors in add_pm_domains function
  - Remove redundant code to handle errors from remove_pm_domains
  - Missing . at the end of the commit message
  - remove redundant initialization of variables
  - remove fail_tcm label and relevant code to free memory
    acquired using devm_* API. As this will be freed when device free it
  - add extra check to see if "reg" property is supported or not

Changes in v5:
  - maintain Rob's Ack on bindings patch as no changes in bindings
  - split previous patch into multiple patches
  - Use pm domain framework to turn on/off TCM
  - Add support of parsing TCM information from device-tree
  - maintain backward compatibility with previous bindings without
    TCM information available in device-tree

This patch series continues previous effort to upstream ZynqMP
TCM bindings:
Previous v4 version link:
https://lore.kernel.org/all/20230829181900.2561194-1-tanmay.shah@amd.com/

Previous v3 version link:
https://lore.kernel.org/all/1689964908-22371-1-git-send-email-radhey.shyam.pandey@amd.com/
Radhey Shyam Pandey (1):
  dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings


Radhey Shyam Pandey (1):
  dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings

Tanmay Shah (2):
  dts: zynqmp: add properties for TCM in remoteproc
  remoteproc: zynqmp: parse TCM from device tree

 .../remoteproc/xlnx,zynqmp-r5fss.yaml         | 131 ++++++++--
 .../boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts  |   8 +
 arch/arm64/boot/dts/xilinx/zynqmp.dtsi        |  58 ++++-
 drivers/remoteproc/xlnx_r5_remoteproc.c       | 245 +++++++++++++++++-
 4 files changed, 413 insertions(+), 29 deletions(-)


base-commit: ff9af5732fe761fa8e7aa66cb482f93a37e284ee
-- 
2.25.1

[PATCH v9 1/3] dt-bindings: remoteproc: add Tightly Coupled Memory (TCM) bindings

[Tanmay Shah]

From: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>

Introduce bindings for TCM memory address space on AMD-xilinx Zynq
UltraScale+ platform. It will help in defining TCM in device-tree
and make it's access platform agnostic and data-driven.

Tightly-coupled memories(TCMs) are low-latency memory that provides
predictable instruction execution and predictable data load/store
timing. Each Cortex-R5F processor contains two 64-bit wide 64 KB memory
banks on the ATCM and BTCM ports, for a total of 128 KB of memory.

The TCM resources(reg, reg-names and power-domain) are documented for
each TCM in the R5 node. The reg and reg-names are made as required
properties as we don't want to hardcode TCM addresses for future
platforms and for zu+ legacy implementation will ensure that the
old dts w/o reg/reg-names works and stable ABI is maintained.

It also extends the examples for TCM split and lockstep modes.

Signed-off-by: Radhey Shyam Pandey <radhey.shyam.pandey@amd.com>
Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
Acked-by: Rob Herring <robh@kernel.org>
---
 .../remoteproc/xlnx,zynqmp-r5fss.yaml         | 131 +++++++++++++++---
 1 file changed, 113 insertions(+), 18 deletions(-)

diff --git a/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml b/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
index 78aac69f1060..9ecd63ea1b38 100644
--- a/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
+++ b/Documentation/devicetree/bindings/remoteproc/xlnx,zynqmp-r5fss.yaml
@@ -20,6 +20,17 @@ properties:
   compatible:
     const: xlnx,zynqmp-r5fss
 
+  "#address-cells":
+    const: 2
+
+  "#size-cells":
+    const: 2
+
+  ranges:
+    description: |
+      Standard ranges definition providing address translations for
+      local R5F TCM address spaces to bus addresses.
+
   xlnx,cluster-mode:
     $ref: /schemas/types.yaml#/definitions/uint32
     enum: [0, 1, 2]
@@ -37,7 +48,7 @@ properties:
       2: single cpu mode
 
 patternProperties:
-  "^r5f-[a-f0-9]+$":
+  "^r5f@[0-9a-f]+$":
     type: object
     description: |
       The RPU is located in the Low Power Domain of the Processor Subsystem.
@@ -54,8 +65,19 @@ patternProperties:
       compatible:
         const: xlnx,zynqmp-r5f
 
+      reg:
+        items:
+          - description: ATCM internal memory region
+          - description: BTCM internal memory region
+
+      reg-names:
+        items:
+          - const: atcm
+          - const: btcm
+
       power-domains:
-        maxItems: 1
+        minItems: 1
+        maxItems: 3
 
       mboxes:
         minItems: 1
@@ -102,34 +124,107 @@ patternProperties:
     required:
       - compatible
       - power-domains
+      - reg
+      - reg-names
 
     unevaluatedProperties: false
 
 required:
   - compatible
+  - "#address-cells"
+  - "#size-cells"
+  - ranges
 
 additionalProperties: false
 
 examples:
   - |
-    remoteproc {
-        compatible = "xlnx,zynqmp-r5fss";
-        xlnx,cluster-mode = <1>;
-
-        r5f-0 {
-            compatible = "xlnx,zynqmp-r5f";
-            power-domains = <&zynqmp_firmware 0x7>;
-            memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>, <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
-            mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
-            mbox-names = "tx", "rx";
+    #include <dt-bindings/power/xlnx-zynqmp-power.h>
+
+    //Split mode configuration
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        remoteproc@ffe00000 {
+            compatible = "xlnx,zynqmp-r5fss";
+            xlnx,cluster-mode = <0>;
+
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x10000>,
+                     <0x0 0x20000 0x0 0xffe20000 0x0 0x10000>,
+                     <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+                     <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+            r5f@0 {
+                compatible = "xlnx,zynqmp-r5f";
+                reg = <0x0 0x0 0x0 0x10000>, <0x0 0x20000 0x0 0x10000>;
+                reg-names = "atcm", "btcm";
+                power-domains = <&zynqmp_firmware PD_RPU_0>,
+                                <&zynqmp_firmware PD_R5_0_ATCM>,
+                                <&zynqmp_firmware PD_R5_0_BTCM>;
+                memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>,
+                                <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
+                mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
+                mbox-names = "tx", "rx";
+            };
+
+            r5f@1 {
+                compatible = "xlnx,zynqmp-r5f";
+                reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+                reg-names = "atcm", "btcm";
+                power-domains = <&zynqmp_firmware PD_RPU_1>,
+                                <&zynqmp_firmware PD_R5_1_ATCM>,
+                                <&zynqmp_firmware PD_R5_1_BTCM>;
+                memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>,
+                                <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
+                mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
+                mbox-names = "tx", "rx";
+            };
         };
+    };
 
-        r5f-1 {
-            compatible = "xlnx,zynqmp-r5f";
-            power-domains = <&zynqmp_firmware 0x8>;
-            memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>, <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
-            mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
-            mbox-names = "tx", "rx";
+  - |
+    //Lockstep configuration
+    soc {
+        #address-cells = <2>;
+        #size-cells = <2>;
+
+        remoteproc@ffe00000 {
+            compatible = "xlnx,zynqmp-r5fss";
+            xlnx,cluster-mode = <1>;
+
+            #address-cells = <2>;
+            #size-cells = <2>;
+            ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x20000>,
+                     <0x0 0x20000 0x0 0xffe20000 0x0 0x20000>;
+
+            r5f@0 {
+                compatible = "xlnx,zynqmp-r5f";
+                reg = <0x0 0x0 0x0 0x20000>, <0x0 0x20000 0x0 0x20000>;
+                reg-names = "atcm", "btcm";
+                power-domains = <&zynqmp_firmware PD_RPU_0>,
+                                <&zynqmp_firmware PD_R5_0_ATCM>,
+                                <&zynqmp_firmware PD_R5_0_BTCM>;
+                memory-region = <&rproc_0_fw_image>, <&rpu0vdev0buffer>,
+                                <&rpu0vdev0vring0>, <&rpu0vdev0vring1>;
+                mboxes = <&ipi_mailbox_rpu0 0>, <&ipi_mailbox_rpu0 1>;
+                mbox-names = "tx", "rx";
+            };
+
+            r5f@1 {
+                compatible = "xlnx,zynqmp-r5f";
+                reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+                reg-names = "atcm", "btcm";
+                power-domains = <&zynqmp_firmware PD_RPU_1>,
+                                <&zynqmp_firmware PD_R5_1_ATCM>,
+                                <&zynqmp_firmware PD_R5_1_BTCM>;
+                memory-region = <&rproc_1_fw_image>, <&rpu1vdev0buffer>,
+                                <&rpu1vdev0vring0>, <&rpu1vdev0vring1>;
+                mboxes = <&ipi_mailbox_rpu1 0>, <&ipi_mailbox_rpu1 1>;
+                mbox-names = "tx", "rx";
+            };
         };
     };
 ...
-- 
2.25.1

[PATCH v9 2/3] dts: zynqmp: add properties for TCM in remoteproc

[Tanmay Shah]

Add properties as per new bindings in zynqmp remoteproc node
to represent TCM address and size.

This patch also adds alternative remoteproc node to represent
remoteproc cluster in split mode. By default lockstep mode is
enabled and users should disable it before using split mode
dts. Both device-tree nodes can't be used simultaneously one
of them must be disabled. For zcu102-1.0 and zcu102-1.1 board
remoteproc split mode dts node is enabled and lockstep mode
dts is disabled.

Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---

Changes in v9:
  - fix rproc lockstep dts


 .../boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts  |  8 +++
 arch/arm64/boot/dts/xilinx/zynqmp.dtsi        | 58 +++++++++++++++++--
 2 files changed, 61 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
index c8f71a1aec89..495ca94b45db 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
+++ b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
@@ -14,6 +14,14 @@ / {
 	compatible = "xlnx,zynqmp-zcu102-rev1.0", "xlnx,zynqmp-zcu102", "xlnx,zynqmp";
 };
 
+&rproc_split {
+	status = "okay";
+};
+
+&rproc_lockstep {
+	status = "disabled";
+};
+
 &eeprom {
 	#address-cells = <1>;
 	#size-cells = <1>;
diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
index b61fc99cd911..cfdd1f68501f 100644
--- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
+++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
@@ -247,19 +247,67 @@ fpga_full: fpga-full {
 		ranges;
 	};
 
-	remoteproc {
+	rproc_lockstep: remoteproc@ffe00000 {
 		compatible = "xlnx,zynqmp-r5fss";
 		xlnx,cluster-mode = <1>;
 
-		r5f-0 {
+		#address-cells = <2>;
+		#size-cells = <2>;
+
+		ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x20000>,
+			 <0x0 0x20000 0x0 0xffe20000 0x0 0x20000>;
+
+		r5f@0 {
+			compatible = "xlnx,zynqmp-r5f";
+			reg = <0x0 0x0 0x0 0x20000>, <0x0 0x20000 0x0 0x20000>;
+			reg-names = "atcm", "btcm";
+			power-domains = <&zynqmp_firmware PD_RPU_0>,
+					<&zynqmp_firmware PD_R5_0_ATCM>,
+					<&zynqmp_firmware PD_R5_0_BTCM>;
+			memory-region = <&rproc_0_fw_image>;
+		};
+
+		r5f@1 {
+			compatible = "xlnx,zynqmp-r5f";
+			reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+			reg-names = "atcm", "btcm";
+			power-domains = <&zynqmp_firmware PD_RPU_1>,
+					<&zynqmp_firmware PD_R5_1_ATCM>,
+					<&zynqmp_firmware PD_R5_1_BTCM>;
+			memory-region = <&rproc_1_fw_image>;
+		};
+	};
+
+	rproc_split: remoteproc-split@ffe00000 {
+		status = "disabled";
+		compatible = "xlnx,zynqmp-r5fss";
+		xlnx,cluster-mode = <0>;
+
+		#address-cells = <2>;
+		#size-cells = <2>;
+
+		ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x10000>,
+			 <0x0 0x20000 0x0 0xffe20000 0x0 0x10000>,
+			 <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
+			 <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
+
+		r5f@0 {
 			compatible = "xlnx,zynqmp-r5f";
-			power-domains = <&zynqmp_firmware PD_RPU_0>;
+			reg = <0x0 0x0 0x0 0x10000>, <0x0 0x20000 0x0 0x10000>;
+			reg-names = "atcm", "btcm";
+			power-domains = <&zynqmp_firmware PD_RPU_0>,
+					<&zynqmp_firmware PD_R5_0_ATCM>,
+					<&zynqmp_firmware PD_R5_0_BTCM>;
 			memory-region = <&rproc_0_fw_image>;
 		};
 
-		r5f-1 {
+		r5f@1 {
 			compatible = "xlnx,zynqmp-r5f";
-			power-domains = <&zynqmp_firmware PD_RPU_1>;
+			reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
+			reg-names = "atcm", "btcm";
+			power-domains = <&zynqmp_firmware PD_RPU_1>,
+					<&zynqmp_firmware PD_R5_1_ATCM>,
+					<&zynqmp_firmware PD_R5_1_BTCM>;
 			memory-region = <&rproc_1_fw_image>;
 		};
 	};
-- 
2.25.1

[PATCH v9 3/3] remoteproc: zynqmp: parse TCM from device tree

[Tanmay Shah]

ZynqMP TCM information was fixed in driver. Now ZynqMP TCM information
is available in device-tree. Parse TCM information in driver
as per new bindings.

Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
---

Changes in v9:
  - Introduce new API to request and release core1 TCM power-domains in
    lockstep mode. This will be used during prepare -> add_tcm_banks
    callback to enable TCM in lockstep mode.
  - Parse TCM from device-tree in lockstep mode and split mode in
    uniform way.
  - Fix TCM representation in device-tree in lockstep mode.

Changes in v8:
  - Remove pm_domains framework
  - Remove checking of pm_domain_id validation to power on/off tcm
  - Remove spurious change
  - parse power-domains property from device-tree and use EEMI calls
    to power on/off TCM instead of using pm domains framework

Changes in v7:
  - move checking of pm_domain_id from previous patch
  - fix mem_bank_data memory allocation

 drivers/remoteproc/xlnx_r5_remoteproc.c | 245 +++++++++++++++++++++++-
 1 file changed, 239 insertions(+), 6 deletions(-)

diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
index 4395edea9a64..0f87b984850b 100644
--- a/drivers/remoteproc/xlnx_r5_remoteproc.c
+++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
@@ -74,8 +74,8 @@ struct mbox_info {
 };
 
 /*
- * Hardcoded TCM bank values. This will be removed once TCM bindings are
- * accepted for system-dt specifications and upstreamed in linux kernel
+ * Hardcoded TCM bank values. This will stay in driver to maintain backward
+ * compatibility with device-tree that does not have TCM information.
  */
 static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
 	{0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
@@ -102,6 +102,7 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
  * @rproc: rproc handle
  * @pm_domain_id: RPU CPU power domain id
  * @ipi: pointer to mailbox information
+ * @lockstep_core1_np: second core's device_node to use in lockstep mode
  */
 struct zynqmp_r5_core {
 	struct device *dev;
@@ -111,6 +112,7 @@ struct zynqmp_r5_core {
 	struct rproc *rproc;
 	u32 pm_domain_id;
 	struct mbox_info *ipi;
+	struct device_node *lockstep_core1_np;
 };
 
 /**
@@ -539,6 +541,110 @@ static int tcm_mem_map(struct rproc *rproc,
 	return 0;
 }
 
+int request_core1_tcm_lockstep(struct rproc *rproc)
+{
+	struct zynqmp_r5_core *r5_core = rproc->priv;
+	struct of_phandle_args out_args = {0};
+	int ret, i, num_pd, pd_id, ret_err;
+	struct device_node *np;
+
+	np = r5_core->lockstep_core1_np;
+
+	/* Get number of power-domains */
+	num_pd = of_count_phandle_with_args(np, "power-domains",
+					    "#power-domain-cells");
+	if (num_pd <= 0)
+		return -EINVAL;
+
+	/* Get individual power-domain id and enable TCM */
+	for (i = 1; i < num_pd; i++) {
+		ret = of_parse_phandle_with_args(np, "power-domains",
+						 "#power-domain-cells",
+						 i, &out_args);
+		if (ret) {
+			dev_warn(r5_core->dev,
+				 "failed to get tcm %d in power-domains list, ret %d\n",
+				 i, ret);
+			goto fail_request_core1_tcm;
+		}
+
+		pd_id = out_args.args[0];
+		of_node_put(out_args.np);
+
+		ret = zynqmp_pm_request_node(pd_id, ZYNQMP_PM_CAPABILITY_ACCESS, 0,
+					     ZYNQMP_PM_REQUEST_ACK_BLOCKING);
+		if (ret) {
+			dev_err(r5_core->dev, "failed to request TCM node 0x%x\n",
+				pd_id);
+			goto fail_request_core1_tcm;
+		}
+	}
+
+	return 0;
+
+fail_request_core1_tcm:
+
+	/* Cache actual error to return later */
+	ret_err = ret;
+
+	/* Release previously requested TCM in case of failure */
+	while (--i > 0) {
+		ret = of_parse_phandle_with_args(np, "power-domains",
+						 "#power-domain-cells",
+						 i, &out_args);
+		if (ret)
+			return ret;
+		pd_id = out_args.args[0];
+		of_node_put(out_args.np);
+		zynqmp_pm_release_node(pd_id);
+	}
+
+	return ret_err;
+}
+
+void release_core1_tcm_lockstep(struct rproc *rproc)
+{
+	struct zynqmp_r5_core *r5_core = rproc->priv;
+	struct of_phandle_args out_args = {0};
+	struct zynqmp_r5_cluster *cluster;
+	int ret, i, num_pd, pd_id;
+	struct device_node *np;
+
+	/* Get R5 core1 node */
+	cluster = dev_get_drvdata(r5_core->dev->parent);
+
+	if (cluster->mode != LOCKSTEP_MODE)
+		return;
+
+	np = r5_core->lockstep_core1_np;
+
+	/* Get number of power-domains */
+	num_pd = of_count_phandle_with_args(np, "power-domains",
+					    "#power-domain-cells");
+	if (num_pd <= 0)
+		return;
+
+	/* Get individual power-domain id and turn off each TCM */
+	for (i = 1; i < num_pd; i++) {
+		ret = of_parse_phandle_with_args(np, "power-domains",
+						 "#power-domain-cells",
+						 i, &out_args);
+		if (ret) {
+			dev_warn(r5_core->dev,
+				 "failed to get pd of core1 tcm %d in list, ret %d\n",
+				 i, ret);
+			continue;
+		}
+
+		pd_id = out_args.args[0];
+		of_node_put(out_args.np);
+
+		if (zynqmp_pm_release_node(pd_id))
+			dev_warn(r5_core->dev,
+				 "failed to release core1 tcm pd 0x%x\n", pd_id);
+	}
+}
+
 /*
  * add_tcm_carveout_split_mode()
  * @rproc: single R5 core's corresponding rproc instance
@@ -633,6 +739,21 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
 	r5_core = rproc->priv;
 	dev = r5_core->dev;
 
+	/*
+	 * In lockstep mode, R5 core0 uses TCM of R5 core1 via aliased addresses.
+	 * Aliased addresses are contiguous with core0 TCM and embedded in "reg"
+	 * property. However, R5 core1 TCM power-domains needs to be requested
+	 * from firmware to use R5 core1 TCM. Request core1 TCM power-domains
+	 * if TCM is parsed from device-tree.
+	 */
+	if (of_find_property(r5_core->np, "reg", NULL)) {
+		ret = request_core1_tcm_lockstep(rproc);
+		if (ret) {
+			dev_err(r5_core->dev, "failed to request core1 TCM power-domains\n");
+			return ret;
+		}
+	}
+
 	/* Go through zynqmp banks for r5 node */
 	num_banks = r5_core->tcm_bank_count;
 
@@ -689,6 +810,9 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
 		pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
 		zynqmp_pm_release_node(pm_domain_id);
 	}
+
+	release_core1_tcm_lockstep(rproc);
+
 	return ret;
 }
 
@@ -808,6 +932,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
 				 "can't turn off TCM bank 0x%x", pm_domain_id);
 	}
 
+	release_core1_tcm_lockstep(rproc);
+
 	return 0;
 }
 
@@ -878,6 +1004,95 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
 	return ERR_PTR(ret);
 }
 
+static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
+{
+	int i, j, tcm_bank_count, ret, tcm_pd_idx;
+	struct of_phandle_args out_args = {0};
+	struct zynqmp_r5_core *r5_core;
+	struct platform_device *cpdev;
+	struct mem_bank_data *tcm;
+	struct device_node *np;
+	struct resource *res;
+	u64 abs_addr, size;
+	struct device *dev;
+
+	for (i = 0; i < cluster->core_count; i++) {
+		r5_core = cluster->r5_cores[i];
+		dev = r5_core->dev;
+		np = r5_core->np;
+
+		/* we have address cell 2 and size cell as 2 */
+		tcm_bank_count = of_property_count_elems_of_size(np, "reg",
+								 4 * sizeof(u32));
+		if (tcm_bank_count <= 0) {
+			dev_err(dev, "can't get reg property err %d\n", tcm_bank_count);
+			return -EINVAL;
+		}
+
+		r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
+						  sizeof(struct mem_bank_data *),
+						  GFP_KERNEL);
+		if (!r5_core->tcm_banks)
+			ret = -ENOMEM;
+
+		r5_core->tcm_bank_count = tcm_bank_count;
+		for (j = 0, tcm_pd_idx = 1; j < tcm_bank_count; j++, tcm_pd_idx++) {
+			tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
+					   GFP_KERNEL);
+			if (!tcm)
+				return -ENOMEM;
+
+			r5_core->tcm_banks[j] = tcm;
+
+			/* Get power-domains id of TCM. */
+			ret = of_parse_phandle_with_args(np, "power-domains",
+							 "#power-domain-cells",
+							 tcm_pd_idx, &out_args);
+			if (ret) {
+				dev_err(r5_core->dev,
+					"failed to get tcm %d pm domain, ret %d\n",
+					tcm_pd_idx, ret);
+				return ret;
+			}
+			tcm->pm_domain_id = out_args.args[0];
+			of_node_put(out_args.np);
+
+			/* Get TCM address without translation. */
+			ret = of_property_read_reg(np, j, &abs_addr, &size);
+			if (ret) {
+				dev_err(dev, "failed to get reg property\n");
+				return ret;
+			}
+
+			/*
+			 * Remote processor can address only 32 bits
+			 * so convert 64-bits into 32-bits. This will discard
+			 * any unwanted upper 32-bits.
+			 */
+			tcm->da = (u32)abs_addr;
+			tcm->size = (u32)size;
+
+			cpdev = to_platform_device(dev);
+			res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
+			if (!res) {
+				dev_err(dev, "failed to get tcm resource\n");
+				return -EINVAL;
+			}
+
+			tcm->addr = (u32)res->start;
+			tcm->bank_name = (char *)res->name;
+			res = devm_request_mem_region(dev, tcm->addr, tcm->size,
+						      tcm->bank_name);
+			if (!res) {
+				dev_err(dev, "failed to request tcm resource\n");
+				return -EINVAL;
+			}
+		}
+	}
+
+	return 0;
+}
+
 /**
  * zynqmp_r5_get_tcm_node()
  * Ideally this function should parse tcm node and store information
@@ -956,9 +1171,14 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
 	struct zynqmp_r5_core *r5_core;
 	int ret, i;
 
-	ret = zynqmp_r5_get_tcm_node(cluster);
-	if (ret < 0) {
-		dev_err(dev, "can't get tcm node, err %d\n", ret);
+	r5_core = cluster->r5_cores[0];
+	if (of_find_property(r5_core->np, "reg", NULL))
+		ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
+	else
+		ret = zynqmp_r5_get_tcm_node(cluster);
+
+	if (ret) {
+		dev_err(dev, "can't get tcm, err %d\n", ret);
 		return ret;
 	}
 
@@ -1099,7 +1319,19 @@ static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
 		 * then ignore second child node.
 		 */
 		if (cluster_mode == LOCKSTEP_MODE) {
-			of_node_put(child);
+			/*
+			 * Get second core's device node only to use its power-domains.
+			 * Also, no need to use of_node_put on first core's device_node
+			 * as it is taken care by of_get_next_available_child.
+			 */
+			r5_cores[i]->lockstep_core1_np =
+				of_get_next_available_child(dev_node, child);
+
+			if (!r5_cores[i]->lockstep_core1_np) {
+				ret = -EINVAL;
+				goto release_r5_cores;
+			}
+
 			break;
 		}
 
@@ -1158,6 +1390,7 @@ static void zynqmp_r5_cluster_exit(void *data)
 		r5_core = cluster->r5_cores[i];
 		zynqmp_r5_free_mbox(r5_core->ipi);
 		of_reserved_mem_device_release(r5_core->dev);
+		of_node_put(r5_core->lockstep_core1_np);
 		put_device(r5_core->dev);
 		rproc_del(r5_core->rproc);
 		rproc_free(r5_core->rproc);
-- 
2.25.1

Re: [PATCH v9 2/3] dts: zynqmp: add properties for TCM in remoteproc

[Michal Simek]

On 1/10/24 22:35, Tanmay Shah wrote:
> Add properties as per new bindings in zynqmp remoteproc node
> to represent TCM address and size.
> 
> This patch also adds alternative remoteproc node to represent
> remoteproc cluster in split mode. By default lockstep mode is
> enabled and users should disable it before using split mode
> dts. Both device-tree nodes can't be used simultaneously one
> of them must be disabled. For zcu102-1.0 and zcu102-1.1 board
> remoteproc split mode dts node is enabled and lockstep mode
> dts is disabled.
> 
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
> 
> Changes in v9:
>    - fix rproc lockstep dts
> 
> 
>   .../boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts  |  8 +++
>   arch/arm64/boot/dts/xilinx/zynqmp.dtsi        | 58 +++++++++++++++++--
>   2 files changed, 61 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
> index c8f71a1aec89..495ca94b45db 100644
> --- a/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
> +++ b/arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev1.0.dts
> @@ -14,6 +14,14 @@ / {
>   	compatible = "xlnx,zynqmp-zcu102-rev1.0", "xlnx,zynqmp-zcu102", "xlnx,zynqmp";
>   };
> +&rproc_split {
> +	status = "okay";
> +};
> +
> +&rproc_lockstep {
> +	status = "disabled";
> +};
> +
>   &eeprom {
>   	#address-cells = <1>;
>   	#size-cells = <1>;
> diff --git a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> index b61fc99cd911..cfdd1f68501f 100644
> --- a/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> +++ b/arch/arm64/boot/dts/xilinx/zynqmp.dtsi
> @@ -247,19 +247,67 @@ fpga_full: fpga-full {
>   		ranges;
>   	};
>   
> -	remoteproc {
> +	rproc_lockstep: remoteproc@ffe00000 {
>   		compatible = "xlnx,zynqmp-r5fss";
>   		xlnx,cluster-mode = <1>;
>   
> -		r5f-0 {
> +		#address-cells = <2>;
> +		#size-cells = <2>;
> +
> +		ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x20000>,
> +			 <0x0 0x20000 0x0 0xffe20000 0x0 0x20000>;
> +
> +		r5f@0 {
> +			compatible = "xlnx,zynqmp-r5f";
> +			reg = <0x0 0x0 0x0 0x20000>, <0x0 0x20000 0x0 0x20000>;
> +			reg-names = "atcm", "btcm";
> +			power-domains = <&zynqmp_firmware PD_RPU_0>,
> +					<&zynqmp_firmware PD_R5_0_ATCM>,
> +					<&zynqmp_firmware PD_R5_0_BTCM>;
> +			memory-region = <&rproc_0_fw_image>;
> +		};
> +
> +		r5f@1 {
> +			compatible = "xlnx,zynqmp-r5f";
> +			reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
> +			reg-names = "atcm", "btcm";
> +			power-domains = <&zynqmp_firmware PD_RPU_1>,
> +					<&zynqmp_firmware PD_R5_1_ATCM>,
> +					<&zynqmp_firmware PD_R5_1_BTCM>;
> +			memory-region = <&rproc_1_fw_image>;
> +		};
> +	};
> +
> +	rproc_split: remoteproc-split@ffe00000 {
> +		status = "disabled";
> +		compatible = "xlnx,zynqmp-r5fss";
> +		xlnx,cluster-mode = <0>;
> +
> +		#address-cells = <2>;
> +		#size-cells = <2>;
> +
> +		ranges = <0x0 0x0 0x0 0xffe00000 0x0 0x10000>,
> +			 <0x0 0x20000 0x0 0xffe20000 0x0 0x10000>,
> +			 <0x1 0x0 0x0 0xffe90000 0x0 0x10000>,
> +			 <0x1 0x20000 0x0 0xffeb0000 0x0 0x10000>;
> +
> +		r5f@0 {
>   			compatible = "xlnx,zynqmp-r5f";
> -			power-domains = <&zynqmp_firmware PD_RPU_0>;
> +			reg = <0x0 0x0 0x0 0x10000>, <0x0 0x20000 0x0 0x10000>;
> +			reg-names = "atcm", "btcm";
> +			power-domains = <&zynqmp_firmware PD_RPU_0>,
> +					<&zynqmp_firmware PD_R5_0_ATCM>,
> +					<&zynqmp_firmware PD_R5_0_BTCM>;
>   			memory-region = <&rproc_0_fw_image>;
>   		};
>   
> -		r5f-1 {
> +		r5f@1 {
>   			compatible = "xlnx,zynqmp-r5f";
> -			power-domains = <&zynqmp_firmware PD_RPU_1>;
> +			reg = <0x1 0x0 0x0 0x10000>, <0x1 0x20000 0x0 0x10000>;
> +			reg-names = "atcm", "btcm";
> +			power-domains = <&zynqmp_firmware PD_RPU_1>,
> +					<&zynqmp_firmware PD_R5_1_ATCM>,
> +					<&zynqmp_firmware PD_R5_1_BTCM>;
>   			memory-region = <&rproc_1_fw_image>;
>   		};
>   	};

Acked-by: Michal Simek <michal.simek@amd.com>

Thanks,
Michal

Re: [PATCH v9 3/3] remoteproc: zynqmp: parse TCM from device tree

[Mathieu Poirier]

Alright, I spent several hours looking at this patchset and the driver as a
whole.  I certainly salute your efforts to heed my advice and make the code less
brittle but I'm afraid we are not there.

See below for a different way to proceed.

On Wed, Jan 10, 2024 at 01:35:05PM -0800, Tanmay Shah wrote:
> ZynqMP TCM information was fixed in driver. Now ZynqMP TCM information
> is available in device-tree. Parse TCM information in driver
> as per new bindings.
> 
> Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> ---
> 
> Changes in v9:
>   - Introduce new API to request and release core1 TCM power-domains in
>     lockstep mode. This will be used during prepare -> add_tcm_banks
>     callback to enable TCM in lockstep mode.
>   - Parse TCM from device-tree in lockstep mode and split mode in
>     uniform way.
>   - Fix TCM representation in device-tree in lockstep mode.
> 
> Changes in v8:
>   - Remove pm_domains framework
>   - Remove checking of pm_domain_id validation to power on/off tcm
>   - Remove spurious change
>   - parse power-domains property from device-tree and use EEMI calls
>     to power on/off TCM instead of using pm domains framework
> 
> Changes in v7:
>   - move checking of pm_domain_id from previous patch
>   - fix mem_bank_data memory allocation
> 
>  drivers/remoteproc/xlnx_r5_remoteproc.c | 245 +++++++++++++++++++++++-
>  1 file changed, 239 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> index 4395edea9a64..0f87b984850b 100644
> --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> @@ -74,8 +74,8 @@ struct mbox_info {
>  };
>  
>  /*
> - * Hardcoded TCM bank values. This will be removed once TCM bindings are
> - * accepted for system-dt specifications and upstreamed in linux kernel
> + * Hardcoded TCM bank values. This will stay in driver to maintain backward
> + * compatibility with device-tree that does not have TCM information.
>   */
>  static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
>  	{0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> @@ -102,6 +102,7 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
>   * @rproc: rproc handle
>   * @pm_domain_id: RPU CPU power domain id
>   * @ipi: pointer to mailbox information
> + * @lockstep_core1_np: second core's device_node to use in lockstep mode
>   */
>  struct zynqmp_r5_core {
>  	struct device *dev;
> @@ -111,6 +112,7 @@ struct zynqmp_r5_core {
>  	struct rproc *rproc;
>  	u32 pm_domain_id;
>  	struct mbox_info *ipi;
> +	struct device_node *lockstep_core1_np;
>  };
>  
>  /**
> @@ -539,6 +541,110 @@ static int tcm_mem_map(struct rproc *rproc,
>  	return 0;
>  }
>  
> +int request_core1_tcm_lockstep(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_core *r5_core = rproc->priv;
> +	struct of_phandle_args out_args = {0};
> +	int ret, i, num_pd, pd_id, ret_err;
> +	struct device_node *np;
> +
> +	np = r5_core->lockstep_core1_np;
> +
> +	/* Get number of power-domains */
> +	num_pd = of_count_phandle_with_args(np, "power-domains",
> +					    "#power-domain-cells");
> +	if (num_pd <= 0)
> +		return -EINVAL;
> +
> +	/* Get individual power-domain id and enable TCM */
> +	for (i = 1; i < num_pd; i++) {
> +		ret = of_parse_phandle_with_args(np, "power-domains",
> +						 "#power-domain-cells",
> +						 i, &out_args);
> +		if (ret) {
> +			dev_warn(r5_core->dev,
> +				 "failed to get tcm %d in power-domains list, ret %d\n",
> +				 i, ret);
> +			goto fail_request_core1_tcm;
> +		}
> +
> +		pd_id = out_args.args[0];
> +		of_node_put(out_args.np);
> +
> +		ret = zynqmp_pm_request_node(pd_id, ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> +					     ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> +		if (ret) {
> +			dev_err(r5_core->dev, "failed to request TCM node 0x%x\n",
> +				pd_id);
> +			goto fail_request_core1_tcm;
> +		}
> +	}
> +
> +	return 0;
> +
> +fail_request_core1_tcm:
> +
> +	/* Cache actual error to return later */
> +	ret_err = ret;
> +
> +	/* Release previously requested TCM in case of failure */
> +	while (--i > 0) {
> +		ret = of_parse_phandle_with_args(np, "power-domains",
> +						 "#power-domain-cells",
> +						 i, &out_args);
> +		if (ret)
> +			return ret;
> +		pd_id = out_args.args[0];
> +		of_node_put(out_args.np);
> +		zynqmp_pm_release_node(pd_id);
> +	}
> +
> +	return ret_err;
> +}
> +
> +void release_core1_tcm_lockstep(struct rproc *rproc)
> +{
> +	struct zynqmp_r5_core *r5_core = rproc->priv;
> +	struct of_phandle_args out_args = {0};
> +	struct zynqmp_r5_cluster *cluster;
> +	int ret, i, num_pd, pd_id;
> +	struct device_node *np;
> +
> +	/* Get R5 core1 node */
> +	cluster = dev_get_drvdata(r5_core->dev->parent);
> +
> +	if (cluster->mode != LOCKSTEP_MODE)
> +		return;
> +
> +	np = r5_core->lockstep_core1_np;
> +
> +	/* Get number of power-domains */
> +	num_pd = of_count_phandle_with_args(np, "power-domains",
> +					    "#power-domain-cells");
> +	if (num_pd <= 0)
> +		return;
> +
> +	/* Get individual power-domain id and turn off each TCM */
> +	for (i = 1; i < num_pd; i++) {
> +		ret = of_parse_phandle_with_args(np, "power-domains",
> +						 "#power-domain-cells",
> +						 i, &out_args);
> +		if (ret) {
> +			dev_warn(r5_core->dev,
> +				 "failed to get pd of core1 tcm %d in list, ret %d\n",
> +				 i, ret);
> +			continue;
> +		}
> +
> +		pd_id = out_args.args[0];
> +		of_node_put(out_args.np);
> +
> +		if (zynqmp_pm_release_node(pd_id))
> +			dev_warn(r5_core->dev,
> +				 "failed to release core1 tcm pd 0x%x\n", pd_id);
> +	}
> +}
> +
>  /*
>   * add_tcm_carveout_split_mode()
>   * @rproc: single R5 core's corresponding rproc instance
> @@ -633,6 +739,21 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>  	r5_core = rproc->priv;
>  	dev = r5_core->dev;
>  
> +	/*
> +	 * In lockstep mode, R5 core0 uses TCM of R5 core1 via aliased addresses.
> +	 * Aliased addresses are contiguous with core0 TCM and embedded in "reg"
> +	 * property. However, R5 core1 TCM power-domains needs to be requested
> +	 * from firmware to use R5 core1 TCM. Request core1 TCM power-domains
> +	 * if TCM is parsed from device-tree.
> +	 */
> +	if (of_find_property(r5_core->np, "reg", NULL)) {
> +		ret = request_core1_tcm_lockstep(rproc);
> +		if (ret) {
> +			dev_err(r5_core->dev, "failed to request core1 TCM power-domains\n");
> +			return ret;
> +		}
> +	}
> +
>  	/* Go through zynqmp banks for r5 node */
>  	num_banks = r5_core->tcm_bank_count;
>  
> @@ -689,6 +810,9 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
>  		pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
>  		zynqmp_pm_release_node(pm_domain_id);
>  	}
> +
> +	release_core1_tcm_lockstep(rproc);
> +
>  	return ret;
>  }
>  
> @@ -808,6 +932,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
>  				 "can't turn off TCM bank 0x%x", pm_domain_id);
>  	}
>  
> +	release_core1_tcm_lockstep(rproc);
> +
>  	return 0;
>  }
>  
> @@ -878,6 +1004,95 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
>  	return ERR_PTR(ret);
>  }
>  
> +static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
> +{
> +	int i, j, tcm_bank_count, ret, tcm_pd_idx;
> +	struct of_phandle_args out_args = {0};
> +	struct zynqmp_r5_core *r5_core;
> +	struct platform_device *cpdev;
> +	struct mem_bank_data *tcm;
> +	struct device_node *np;
> +	struct resource *res;
> +	u64 abs_addr, size;
> +	struct device *dev;
> +
> +	for (i = 0; i < cluster->core_count; i++) {
> +		r5_core = cluster->r5_cores[i];
> +		dev = r5_core->dev;
> +		np = r5_core->np;
> +

Using r5_core->np doesn't work because it deals with the specifics of a single
subnode when we need to deal with the subnodes of the entire cluster.


> +		/* we have address cell 2 and size cell as 2 */
> +		tcm_bank_count = of_property_count_elems_of_size(np, "reg",
> +								 4 * sizeof(u32));
> +		if (tcm_bank_count <= 0) {
> +			dev_err(dev, "can't get reg property err %d\n", tcm_bank_count);
> +			return -EINVAL;
> +		}
> +
> +		r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
> +						  sizeof(struct mem_bank_data *),
> +						  GFP_KERNEL);

Another problem is that when getting information from the DT, ->tcm_banks is
always 2 whereas it varies (2 or 4) when using the static mem_bank_data arrays.

We know the current driver works well when using static banks and everything is
already in place to address the mode of operation (lockstep vs split). As
such I suggest to reuse all that code by making function
zynqmp_r5_get_tcm_node_from_dt() return a mem_bank_data array of 4 elements.
That array would be instantiated using the information found in the DT,
regardless of the mode of operation.  Once we have that array it could simply be
inserted in function zynqmp_r5_get_tcm_node() and everything else in the driver
works the same way.

Note that for that work you will have to set the "reg" values of the second
core to 0 when in lockstep mode, which is fine because they are not used anyway.

Thanks,
Mathieu

> +		if (!r5_core->tcm_banks)
> +			ret = -ENOMEM;
> +
> +		r5_core->tcm_bank_count = tcm_bank_count;
> +		for (j = 0, tcm_pd_idx = 1; j < tcm_bank_count; j++, tcm_pd_idx++) {
> +			tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
> +					   GFP_KERNEL);
> +			if (!tcm)
> +				return -ENOMEM;
> +
> +			r5_core->tcm_banks[j] = tcm;
> +
> +			/* Get power-domains id of TCM. */
> +			ret = of_parse_phandle_with_args(np, "power-domains",
> +							 "#power-domain-cells",
> +							 tcm_pd_idx, &out_args);
> +			if (ret) {
> +				dev_err(r5_core->dev,
> +					"failed to get tcm %d pm domain, ret %d\n",
> +					tcm_pd_idx, ret);
> +				return ret;
> +			}
> +			tcm->pm_domain_id = out_args.args[0];
> +			of_node_put(out_args.np);
> +
> +			/* Get TCM address without translation. */
> +			ret = of_property_read_reg(np, j, &abs_addr, &size);
> +			if (ret) {
> +				dev_err(dev, "failed to get reg property\n");
> +				return ret;
> +			}
> +
> +			/*
> +			 * Remote processor can address only 32 bits
> +			 * so convert 64-bits into 32-bits. This will discard
> +			 * any unwanted upper 32-bits.
> +			 */
> +			tcm->da = (u32)abs_addr;
> +			tcm->size = (u32)size;
> +
> +			cpdev = to_platform_device(dev);
> +			res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
> +			if (!res) {
> +				dev_err(dev, "failed to get tcm resource\n");
> +				return -EINVAL;
> +			}
> +
> +			tcm->addr = (u32)res->start;
> +			tcm->bank_name = (char *)res->name;
> +			res = devm_request_mem_region(dev, tcm->addr, tcm->size,
> +						      tcm->bank_name);
> +			if (!res) {
> +				dev_err(dev, "failed to request tcm resource\n");
> +				return -EINVAL;
> +			}
> +		}
> +	}
> +
> +	return 0;
> +}
> +
>  /**
>   * zynqmp_r5_get_tcm_node()
>   * Ideally this function should parse tcm node and store information
> @@ -956,9 +1171,14 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
>  	struct zynqmp_r5_core *r5_core;
>  	int ret, i;
>  
> -	ret = zynqmp_r5_get_tcm_node(cluster);
> -	if (ret < 0) {
> -		dev_err(dev, "can't get tcm node, err %d\n", ret);
> +	r5_core = cluster->r5_cores[0];
> +	if (of_find_property(r5_core->np, "reg", NULL))
> +		ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
> +	else
> +		ret = zynqmp_r5_get_tcm_node(cluster);
> +
> +	if (ret) {
> +		dev_err(dev, "can't get tcm, err %d\n", ret);
>  		return ret;
>  	}
>  
> @@ -1099,7 +1319,19 @@ static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
>  		 * then ignore second child node.
>  		 */
>  		if (cluster_mode == LOCKSTEP_MODE) {
> -			of_node_put(child);
> +			/*
> +			 * Get second core's device node only to use its power-domains.
> +			 * Also, no need to use of_node_put on first core's device_node
> +			 * as it is taken care by of_get_next_available_child.
> +			 */
> +			r5_cores[i]->lockstep_core1_np =
> +				of_get_next_available_child(dev_node, child);
> +
> +			if (!r5_cores[i]->lockstep_core1_np) {
> +				ret = -EINVAL;
> +				goto release_r5_cores;
> +			}
> +
>  			break;
>  		}
>  
> @@ -1158,6 +1390,7 @@ static void zynqmp_r5_cluster_exit(void *data)
>  		r5_core = cluster->r5_cores[i];
>  		zynqmp_r5_free_mbox(r5_core->ipi);
>  		of_reserved_mem_device_release(r5_core->dev);
> +		of_node_put(r5_core->lockstep_core1_np);
>  		put_device(r5_core->dev);
>  		rproc_del(r5_core->rproc);
>  		rproc_free(r5_core->rproc);
> -- 
> 2.25.1
> 

Re: [PATCH v9 3/3] remoteproc: zynqmp: parse TCM from device tree

[Tanmay Shah]

On 1/17/24 12:58 PM, Mathieu Poirier wrote:
> Alright, I spent several hours looking at this patchset and the driver as a
> whole.  I certainly salute your efforts to heed my advice and make the code less
> brittle but I'm afraid we are not there.

Hi Mathieu,

I am back from vacation. Started looking into this. Thanks for spending time on this and helping

to make clean design of the driver. Please find my comments below.

> See below for a different way to proceed.
>
> On Wed, Jan 10, 2024 at 01:35:05PM -0800, Tanmay Shah wrote:
> > ZynqMP TCM information was fixed in driver. Now ZynqMP TCM information
> > is available in device-tree. Parse TCM information in driver
> > as per new bindings.
> > 
> > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > ---
> > 
> > Changes in v9:
> >   - Introduce new API to request and release core1 TCM power-domains in
> >     lockstep mode. This will be used during prepare -> add_tcm_banks
> >     callback to enable TCM in lockstep mode.
> >   - Parse TCM from device-tree in lockstep mode and split mode in
> >     uniform way.
> >   - Fix TCM representation in device-tree in lockstep mode.
> > 
> > Changes in v8:
> >   - Remove pm_domains framework
> >   - Remove checking of pm_domain_id validation to power on/off tcm
> >   - Remove spurious change
> >   - parse power-domains property from device-tree and use EEMI calls
> >     to power on/off TCM instead of using pm domains framework
> > 
> > Changes in v7:
> >   - move checking of pm_domain_id from previous patch
> >   - fix mem_bank_data memory allocation
> > 
> >  drivers/remoteproc/xlnx_r5_remoteproc.c | 245 +++++++++++++++++++++++-
> >  1 file changed, 239 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > index 4395edea9a64..0f87b984850b 100644
> > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > @@ -74,8 +74,8 @@ struct mbox_info {
> >  };
> >  
> >  /*
> > - * Hardcoded TCM bank values. This will be removed once TCM bindings are
> > - * accepted for system-dt specifications and upstreamed in linux kernel
> > + * Hardcoded TCM bank values. This will stay in driver to maintain backward
> > + * compatibility with device-tree that does not have TCM information.
> >   */
> >  static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
> >  	{0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> > @@ -102,6 +102,7 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> >   * @rproc: rproc handle
> >   * @pm_domain_id: RPU CPU power domain id
> >   * @ipi: pointer to mailbox information
> > + * @lockstep_core1_np: second core's device_node to use in lockstep mode
> >   */
> >  struct zynqmp_r5_core {
> >  	struct device *dev;
> > @@ -111,6 +112,7 @@ struct zynqmp_r5_core {
> >  	struct rproc *rproc;
> >  	u32 pm_domain_id;
> >  	struct mbox_info *ipi;
> > +	struct device_node *lockstep_core1_np;
> >  };
> >  
> >  /**
> > @@ -539,6 +541,110 @@ static int tcm_mem_map(struct rproc *rproc,
> >  	return 0;
> >  }
> >  
> > +int request_core1_tcm_lockstep(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_core *r5_core = rproc->priv;
> > +	struct of_phandle_args out_args = {0};
> > +	int ret, i, num_pd, pd_id, ret_err;
> > +	struct device_node *np;
> > +
> > +	np = r5_core->lockstep_core1_np;
> > +
> > +	/* Get number of power-domains */
> > +	num_pd = of_count_phandle_with_args(np, "power-domains",
> > +					    "#power-domain-cells");
> > +	if (num_pd <= 0)
> > +		return -EINVAL;
> > +
> > +	/* Get individual power-domain id and enable TCM */
> > +	for (i = 1; i < num_pd; i++) {
> > +		ret = of_parse_phandle_with_args(np, "power-domains",
> > +						 "#power-domain-cells",
> > +						 i, &out_args);
> > +		if (ret) {
> > +			dev_warn(r5_core->dev,
> > +				 "failed to get tcm %d in power-domains list, ret %d\n",
> > +				 i, ret);
> > +			goto fail_request_core1_tcm;
> > +		}
> > +
> > +		pd_id = out_args.args[0];
> > +		of_node_put(out_args.np);
> > +
> > +		ret = zynqmp_pm_request_node(pd_id, ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > +					     ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > +		if (ret) {
> > +			dev_err(r5_core->dev, "failed to request TCM node 0x%x\n",
> > +				pd_id);
> > +			goto fail_request_core1_tcm;
> > +		}
> > +	}
> > +
> > +	return 0;
> > +
> > +fail_request_core1_tcm:
> > +
> > +	/* Cache actual error to return later */
> > +	ret_err = ret;
> > +
> > +	/* Release previously requested TCM in case of failure */
> > +	while (--i > 0) {
> > +		ret = of_parse_phandle_with_args(np, "power-domains",
> > +						 "#power-domain-cells",
> > +						 i, &out_args);
> > +		if (ret)
> > +			return ret;
> > +		pd_id = out_args.args[0];
> > +		of_node_put(out_args.np);
> > +		zynqmp_pm_release_node(pd_id);
> > +	}
> > +
> > +	return ret_err;
> > +}
> > +
> > +void release_core1_tcm_lockstep(struct rproc *rproc)
> > +{
> > +	struct zynqmp_r5_core *r5_core = rproc->priv;
> > +	struct of_phandle_args out_args = {0};
> > +	struct zynqmp_r5_cluster *cluster;
> > +	int ret, i, num_pd, pd_id;
> > +	struct device_node *np;
> > +
> > +	/* Get R5 core1 node */
> > +	cluster = dev_get_drvdata(r5_core->dev->parent);
> > +
> > +	if (cluster->mode != LOCKSTEP_MODE)
> > +		return;
> > +
> > +	np = r5_core->lockstep_core1_np;
> > +
> > +	/* Get number of power-domains */
> > +	num_pd = of_count_phandle_with_args(np, "power-domains",
> > +					    "#power-domain-cells");
> > +	if (num_pd <= 0)
> > +		return;
> > +
> > +	/* Get individual power-domain id and turn off each TCM */
> > +	for (i = 1; i < num_pd; i++) {
> > +		ret = of_parse_phandle_with_args(np, "power-domains",
> > +						 "#power-domain-cells",
> > +						 i, &out_args);
> > +		if (ret) {
> > +			dev_warn(r5_core->dev,
> > +				 "failed to get pd of core1 tcm %d in list, ret %d\n",
> > +				 i, ret);
> > +			continue;
> > +		}
> > +
> > +		pd_id = out_args.args[0];
> > +		of_node_put(out_args.np);
> > +
> > +		if (zynqmp_pm_release_node(pd_id))
> > +			dev_warn(r5_core->dev,
> > +				 "failed to release core1 tcm pd 0x%x\n", pd_id);
> > +	}
> > +}
> > +
> >  /*
> >   * add_tcm_carveout_split_mode()
> >   * @rproc: single R5 core's corresponding rproc instance
> > @@ -633,6 +739,21 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> >  	r5_core = rproc->priv;
> >  	dev = r5_core->dev;
> >  
> > +	/*
> > +	 * In lockstep mode, R5 core0 uses TCM of R5 core1 via aliased addresses.
> > +	 * Aliased addresses are contiguous with core0 TCM and embedded in "reg"
> > +	 * property. However, R5 core1 TCM power-domains needs to be requested
> > +	 * from firmware to use R5 core1 TCM. Request core1 TCM power-domains
> > +	 * if TCM is parsed from device-tree.
> > +	 */
> > +	if (of_find_property(r5_core->np, "reg", NULL)) {
> > +		ret = request_core1_tcm_lockstep(rproc);
> > +		if (ret) {
> > +			dev_err(r5_core->dev, "failed to request core1 TCM power-domains\n");
> > +			return ret;
> > +		}
> > +	}
> > +
> >  	/* Go through zynqmp banks for r5 node */
> >  	num_banks = r5_core->tcm_bank_count;
> >  
> > @@ -689,6 +810,9 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> >  		pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> >  		zynqmp_pm_release_node(pm_domain_id);
> >  	}
> > +
> > +	release_core1_tcm_lockstep(rproc);
> > +
> >  	return ret;
> >  }
> >  
> > @@ -808,6 +932,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> >  				 "can't turn off TCM bank 0x%x", pm_domain_id);
> >  	}
> >  
> > +	release_core1_tcm_lockstep(rproc);
> > +
> >  	return 0;
> >  }
> >  
> > @@ -878,6 +1004,95 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
> >  	return ERR_PTR(ret);
> >  }
> >  
> > +static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
> > +{
> > +	int i, j, tcm_bank_count, ret, tcm_pd_idx;
> > +	struct of_phandle_args out_args = {0};
> > +	struct zynqmp_r5_core *r5_core;
> > +	struct platform_device *cpdev;
> > +	struct mem_bank_data *tcm;
> > +	struct device_node *np;
> > +	struct resource *res;
> > +	u64 abs_addr, size;
> > +	struct device *dev;
> > +
> > +	for (i = 0; i < cluster->core_count; i++) {
> > +		r5_core = cluster->r5_cores[i];
> > +		dev = r5_core->dev;
> > +		np = r5_core->np;
> > +
>
> Using r5_core->np doesn't work because it deals with the specifics of a single
> subnode when we need to deal with the subnodes of the entire cluster.

Correct. I think below mentioned design should resolve this problem.


>
> > +		/* we have address cell 2 and size cell as 2 */
> > +		tcm_bank_count = of_property_count_elems_of_size(np, "reg",
> > +								 4 * sizeof(u32));
> > +		if (tcm_bank_count <= 0) {
> > +			dev_err(dev, "can't get reg property err %d\n", tcm_bank_count);
> > +			return -EINVAL;
> > +		}
> > +
> > +		r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
> > +						  sizeof(struct mem_bank_data *),
> > +						  GFP_KERNEL);
>
> Another problem is that when getting information from the DT, ->tcm_banks is
> always 2 whereas it varies (2 or 4) when using the static mem_bank_data arrays.
>
> We know the current driver works well when using static banks and everything is
> already in place to address the mode of operation (lockstep vs split). As
> such I suggest to reuse all that code by making function
> zynqmp_r5_get_tcm_node_from_dt() return a mem_bank_data array of 4 elements.
> That array would be instantiated using the information found in the DT,
> regardless of the mode of operation.  Once we have that array it could simply be
> inserted in function zynqmp_r5_get_tcm_node() and everything else in the driver
> works the same way.
>
> Note that for that work you will have to set the "reg" values of the second
> core to 0 when in lockstep mode, which is fine because they are not used anyway.

I agree to most part. I will have to modify bindings as well to accommodate above design.

I think the problem is in lockstep mode, where two memory regions are combined and presented as single

memory-region in table: *zynqmp_tcm_banks_lockstep*. First I will have to fix that.

If that is fixed, then no need to have two functions for lockstep and split tcm carveouts. They both can be

handled uniformly. Once that is fixed, bindings needs to be fixed i.e.

For lockstep mode, "reg" will have 4 entries and split mode "reg" will have only two entries.

We don't have to make "reg" of core1 as 0 in lockstep mode, as it's not being used anyway.

Once I change bindings as above, then it's very easy to implement changes as suggested.

We parse TCM information from dt if "reg" is available, and fill up the table in r5_core->tcm_banks and rest of the

driver remain same.

I will implement and test v10 accordingly. I think this should make design clean.

Only downside is, we are spliting contiguous region in lockstep mode, so after this change, firmware's linker script

needs to be updated accordingly for lockstep mode (If contiguous regions are considered which is not the case for Xilinx's firmwares).

I would consider current implementation as bug, and will send Fixes tag accordingly where needed.


Thanks,

Tanmay


> Thanks,
> Mathieu
>
> > +		if (!r5_core->tcm_banks)
> > +			ret = -ENOMEM;
> > +
> > +		r5_core->tcm_bank_count = tcm_bank_count;
> > +		for (j = 0, tcm_pd_idx = 1; j < tcm_bank_count; j++, tcm_pd_idx++) {
> > +			tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
> > +					   GFP_KERNEL);
> > +			if (!tcm)
> > +				return -ENOMEM;
> > +
> > +			r5_core->tcm_banks[j] = tcm;
> > +
> > +			/* Get power-domains id of TCM. */
> > +			ret = of_parse_phandle_with_args(np, "power-domains",
> > +							 "#power-domain-cells",
> > +							 tcm_pd_idx, &out_args);
> > +			if (ret) {
> > +				dev_err(r5_core->dev,
> > +					"failed to get tcm %d pm domain, ret %d\n",
> > +					tcm_pd_idx, ret);
> > +				return ret;
> > +			}
> > +			tcm->pm_domain_id = out_args.args[0];
> > +			of_node_put(out_args.np);
> > +
> > +			/* Get TCM address without translation. */
> > +			ret = of_property_read_reg(np, j, &abs_addr, &size);
> > +			if (ret) {
> > +				dev_err(dev, "failed to get reg property\n");
> > +				return ret;
> > +			}
> > +
> > +			/*
> > +			 * Remote processor can address only 32 bits
> > +			 * so convert 64-bits into 32-bits. This will discard
> > +			 * any unwanted upper 32-bits.
> > +			 */
> > +			tcm->da = (u32)abs_addr;
> > +			tcm->size = (u32)size;
> > +
> > +			cpdev = to_platform_device(dev);
> > +			res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
> > +			if (!res) {
> > +				dev_err(dev, "failed to get tcm resource\n");
> > +				return -EINVAL;
> > +			}
> > +
> > +			tcm->addr = (u32)res->start;
> > +			tcm->bank_name = (char *)res->name;
> > +			res = devm_request_mem_region(dev, tcm->addr, tcm->size,
> > +						      tcm->bank_name);
> > +			if (!res) {
> > +				dev_err(dev, "failed to request tcm resource\n");
> > +				return -EINVAL;
> > +			}
> > +		}
> > +	}
> > +
> > +	return 0;
> > +}
> > +
> >  /**
> >   * zynqmp_r5_get_tcm_node()
> >   * Ideally this function should parse tcm node and store information
> > @@ -956,9 +1171,14 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
> >  	struct zynqmp_r5_core *r5_core;
> >  	int ret, i;
> >  
> > -	ret = zynqmp_r5_get_tcm_node(cluster);
> > -	if (ret < 0) {
> > -		dev_err(dev, "can't get tcm node, err %d\n", ret);
> > +	r5_core = cluster->r5_cores[0];
> > +	if (of_find_property(r5_core->np, "reg", NULL))
> > +		ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
> > +	else
> > +		ret = zynqmp_r5_get_tcm_node(cluster);
> > +
> > +	if (ret) {
> > +		dev_err(dev, "can't get tcm, err %d\n", ret);
> >  		return ret;
> >  	}
> >  
> > @@ -1099,7 +1319,19 @@ static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> >  		 * then ignore second child node.
> >  		 */
> >  		if (cluster_mode == LOCKSTEP_MODE) {
> > -			of_node_put(child);
> > +			/*
> > +			 * Get second core's device node only to use its power-domains.
> > +			 * Also, no need to use of_node_put on first core's device_node
> > +			 * as it is taken care by of_get_next_available_child.
> > +			 */
> > +			r5_cores[i]->lockstep_core1_np =
> > +				of_get_next_available_child(dev_node, child);
> > +
> > +			if (!r5_cores[i]->lockstep_core1_np) {
> > +				ret = -EINVAL;
> > +				goto release_r5_cores;
> > +			}
> > +
> >  			break;
> >  		}
> >  
> > @@ -1158,6 +1390,7 @@ static void zynqmp_r5_cluster_exit(void *data)
> >  		r5_core = cluster->r5_cores[i];
> >  		zynqmp_r5_free_mbox(r5_core->ipi);
> >  		of_reserved_mem_device_release(r5_core->dev);
> > +		of_node_put(r5_core->lockstep_core1_np);
> >  		put_device(r5_core->dev);
> >  		rproc_del(r5_core->rproc);
> >  		rproc_free(r5_core->rproc);
> > -- 
> > 2.25.1
> > 

Re: [PATCH v9 3/3] remoteproc: zynqmp: parse TCM from device tree

[Mathieu Poirier]

On Tue, Jan 30, 2024 at 02:47:07PM -0600, Tanmay Shah wrote:
> 
> On 1/17/24 12:58 PM, Mathieu Poirier wrote:
> > Alright, I spent several hours looking at this patchset and the driver as a
> > whole.  I certainly salute your efforts to heed my advice and make the code less
> > brittle but I'm afraid we are not there.
> 
> Hi Mathieu,
> 
> I am back from vacation. Started looking into this. Thanks for spending time on this and helping
> 
> to make clean design of the driver. Please find my comments below.
> 
> > See below for a different way to proceed.
> >
> > On Wed, Jan 10, 2024 at 01:35:05PM -0800, Tanmay Shah wrote:
> > > ZynqMP TCM information was fixed in driver. Now ZynqMP TCM information
> > > is available in device-tree. Parse TCM information in driver
> > > as per new bindings.
> > > 
> > > Signed-off-by: Tanmay Shah <tanmay.shah@amd.com>
> > > ---
> > > 
> > > Changes in v9:
> > >   - Introduce new API to request and release core1 TCM power-domains in
> > >     lockstep mode. This will be used during prepare -> add_tcm_banks
> > >     callback to enable TCM in lockstep mode.
> > >   - Parse TCM from device-tree in lockstep mode and split mode in
> > >     uniform way.
> > >   - Fix TCM representation in device-tree in lockstep mode.
> > > 
> > > Changes in v8:
> > >   - Remove pm_domains framework
> > >   - Remove checking of pm_domain_id validation to power on/off tcm
> > >   - Remove spurious change
> > >   - parse power-domains property from device-tree and use EEMI calls
> > >     to power on/off TCM instead of using pm domains framework
> > > 
> > > Changes in v7:
> > >   - move checking of pm_domain_id from previous patch
> > >   - fix mem_bank_data memory allocation
> > > 
> > >  drivers/remoteproc/xlnx_r5_remoteproc.c | 245 +++++++++++++++++++++++-
> > >  1 file changed, 239 insertions(+), 6 deletions(-)
> > > 
> > > diff --git a/drivers/remoteproc/xlnx_r5_remoteproc.c b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > index 4395edea9a64..0f87b984850b 100644
> > > --- a/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > +++ b/drivers/remoteproc/xlnx_r5_remoteproc.c
> > > @@ -74,8 +74,8 @@ struct mbox_info {
> > >  };
> > >  
> > >  /*
> > > - * Hardcoded TCM bank values. This will be removed once TCM bindings are
> > > - * accepted for system-dt specifications and upstreamed in linux kernel
> > > + * Hardcoded TCM bank values. This will stay in driver to maintain backward
> > > + * compatibility with device-tree that does not have TCM information.
> > >   */
> > >  static const struct mem_bank_data zynqmp_tcm_banks_split[] = {
> > >  	{0xffe00000UL, 0x0, 0x10000UL, PD_R5_0_ATCM, "atcm0"}, /* TCM 64KB each */
> > > @@ -102,6 +102,7 @@ static const struct mem_bank_data zynqmp_tcm_banks_lockstep[] = {
> > >   * @rproc: rproc handle
> > >   * @pm_domain_id: RPU CPU power domain id
> > >   * @ipi: pointer to mailbox information
> > > + * @lockstep_core1_np: second core's device_node to use in lockstep mode
> > >   */
> > >  struct zynqmp_r5_core {
> > >  	struct device *dev;
> > > @@ -111,6 +112,7 @@ struct zynqmp_r5_core {
> > >  	struct rproc *rproc;
> > >  	u32 pm_domain_id;
> > >  	struct mbox_info *ipi;
> > > +	struct device_node *lockstep_core1_np;
> > >  };
> > >  
> > >  /**
> > > @@ -539,6 +541,110 @@ static int tcm_mem_map(struct rproc *rproc,
> > >  	return 0;
> > >  }
> > >  
> > > +int request_core1_tcm_lockstep(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_core *r5_core = rproc->priv;
> > > +	struct of_phandle_args out_args = {0};
> > > +	int ret, i, num_pd, pd_id, ret_err;
> > > +	struct device_node *np;
> > > +
> > > +	np = r5_core->lockstep_core1_np;
> > > +
> > > +	/* Get number of power-domains */
> > > +	num_pd = of_count_phandle_with_args(np, "power-domains",
> > > +					    "#power-domain-cells");
> > > +	if (num_pd <= 0)
> > > +		return -EINVAL;
> > > +
> > > +	/* Get individual power-domain id and enable TCM */
> > > +	for (i = 1; i < num_pd; i++) {
> > > +		ret = of_parse_phandle_with_args(np, "power-domains",
> > > +						 "#power-domain-cells",
> > > +						 i, &out_args);
> > > +		if (ret) {
> > > +			dev_warn(r5_core->dev,
> > > +				 "failed to get tcm %d in power-domains list, ret %d\n",
> > > +				 i, ret);
> > > +			goto fail_request_core1_tcm;
> > > +		}
> > > +
> > > +		pd_id = out_args.args[0];
> > > +		of_node_put(out_args.np);
> > > +
> > > +		ret = zynqmp_pm_request_node(pd_id, ZYNQMP_PM_CAPABILITY_ACCESS, 0,
> > > +					     ZYNQMP_PM_REQUEST_ACK_BLOCKING);
> > > +		if (ret) {
> > > +			dev_err(r5_core->dev, "failed to request TCM node 0x%x\n",
> > > +				pd_id);
> > > +			goto fail_request_core1_tcm;
> > > +		}
> > > +	}
> > > +
> > > +	return 0;
> > > +
> > > +fail_request_core1_tcm:
> > > +
> > > +	/* Cache actual error to return later */
> > > +	ret_err = ret;
> > > +
> > > +	/* Release previously requested TCM in case of failure */
> > > +	while (--i > 0) {
> > > +		ret = of_parse_phandle_with_args(np, "power-domains",
> > > +						 "#power-domain-cells",
> > > +						 i, &out_args);
> > > +		if (ret)
> > > +			return ret;
> > > +		pd_id = out_args.args[0];
> > > +		of_node_put(out_args.np);
> > > +		zynqmp_pm_release_node(pd_id);
> > > +	}
> > > +
> > > +	return ret_err;
> > > +}
> > > +
> > > +void release_core1_tcm_lockstep(struct rproc *rproc)
> > > +{
> > > +	struct zynqmp_r5_core *r5_core = rproc->priv;
> > > +	struct of_phandle_args out_args = {0};
> > > +	struct zynqmp_r5_cluster *cluster;
> > > +	int ret, i, num_pd, pd_id;
> > > +	struct device_node *np;
> > > +
> > > +	/* Get R5 core1 node */
> > > +	cluster = dev_get_drvdata(r5_core->dev->parent);
> > > +
> > > +	if (cluster->mode != LOCKSTEP_MODE)
> > > +		return;
> > > +
> > > +	np = r5_core->lockstep_core1_np;
> > > +
> > > +	/* Get number of power-domains */
> > > +	num_pd = of_count_phandle_with_args(np, "power-domains",
> > > +					    "#power-domain-cells");
> > > +	if (num_pd <= 0)
> > > +		return;
> > > +
> > > +	/* Get individual power-domain id and turn off each TCM */
> > > +	for (i = 1; i < num_pd; i++) {
> > > +		ret = of_parse_phandle_with_args(np, "power-domains",
> > > +						 "#power-domain-cells",
> > > +						 i, &out_args);
> > > +		if (ret) {
> > > +			dev_warn(r5_core->dev,
> > > +				 "failed to get pd of core1 tcm %d in list, ret %d\n",
> > > +				 i, ret);
> > > +			continue;
> > > +		}
> > > +
> > > +		pd_id = out_args.args[0];
> > > +		of_node_put(out_args.np);
> > > +
> > > +		if (zynqmp_pm_release_node(pd_id))
> > > +			dev_warn(r5_core->dev,
> > > +				 "failed to release core1 tcm pd 0x%x\n", pd_id);
> > > +	}
> > > +}
> > > +
> > >  /*
> > >   * add_tcm_carveout_split_mode()
> > >   * @rproc: single R5 core's corresponding rproc instance
> > > @@ -633,6 +739,21 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > >  	r5_core = rproc->priv;
> > >  	dev = r5_core->dev;
> > >  
> > > +	/*
> > > +	 * In lockstep mode, R5 core0 uses TCM of R5 core1 via aliased addresses.
> > > +	 * Aliased addresses are contiguous with core0 TCM and embedded in "reg"
> > > +	 * property. However, R5 core1 TCM power-domains needs to be requested
> > > +	 * from firmware to use R5 core1 TCM. Request core1 TCM power-domains
> > > +	 * if TCM is parsed from device-tree.
> > > +	 */
> > > +	if (of_find_property(r5_core->np, "reg", NULL)) {
> > > +		ret = request_core1_tcm_lockstep(rproc);
> > > +		if (ret) {
> > > +			dev_err(r5_core->dev, "failed to request core1 TCM power-domains\n");
> > > +			return ret;
> > > +		}
> > > +	}
> > > +
> > >  	/* Go through zynqmp banks for r5 node */
> > >  	num_banks = r5_core->tcm_bank_count;
> > >  
> > > @@ -689,6 +810,9 @@ static int add_tcm_carveout_lockstep_mode(struct rproc *rproc)
> > >  		pm_domain_id = r5_core->tcm_banks[i]->pm_domain_id;
> > >  		zynqmp_pm_release_node(pm_domain_id);
> > >  	}
> > > +
> > > +	release_core1_tcm_lockstep(rproc);
> > > +
> > >  	return ret;
> > >  }
> > >  
> > > @@ -808,6 +932,8 @@ static int zynqmp_r5_rproc_unprepare(struct rproc *rproc)
> > >  				 "can't turn off TCM bank 0x%x", pm_domain_id);
> > >  	}
> > >  
> > > +	release_core1_tcm_lockstep(rproc);
> > > +
> > >  	return 0;
> > >  }
> > >  
> > > @@ -878,6 +1004,95 @@ static struct zynqmp_r5_core *zynqmp_r5_add_rproc_core(struct device *cdev)
> > >  	return ERR_PTR(ret);
> > >  }
> > >  
> > > +static int zynqmp_r5_get_tcm_node_from_dt(struct zynqmp_r5_cluster *cluster)
> > > +{
> > > +	int i, j, tcm_bank_count, ret, tcm_pd_idx;
> > > +	struct of_phandle_args out_args = {0};
> > > +	struct zynqmp_r5_core *r5_core;
> > > +	struct platform_device *cpdev;
> > > +	struct mem_bank_data *tcm;
> > > +	struct device_node *np;
> > > +	struct resource *res;
> > > +	u64 abs_addr, size;
> > > +	struct device *dev;
> > > +
> > > +	for (i = 0; i < cluster->core_count; i++) {
> > > +		r5_core = cluster->r5_cores[i];
> > > +		dev = r5_core->dev;
> > > +		np = r5_core->np;
> > > +
> >
> > Using r5_core->np doesn't work because it deals with the specifics of a single
> > subnode when we need to deal with the subnodes of the entire cluster.
> 
> Correct. I think below mentioned design should resolve this problem.
> 
> 
> >
> > > +		/* we have address cell 2 and size cell as 2 */
> > > +		tcm_bank_count = of_property_count_elems_of_size(np, "reg",
> > > +								 4 * sizeof(u32));
> > > +		if (tcm_bank_count <= 0) {
> > > +			dev_err(dev, "can't get reg property err %d\n", tcm_bank_count);
> > > +			return -EINVAL;
> > > +		}
> > > +
> > > +		r5_core->tcm_banks = devm_kcalloc(dev, tcm_bank_count,
> > > +						  sizeof(struct mem_bank_data *),
> > > +						  GFP_KERNEL);
> >
> > Another problem is that when getting information from the DT, ->tcm_banks is
> > always 2 whereas it varies (2 or 4) when using the static mem_bank_data arrays.
> >
> > We know the current driver works well when using static banks and everything is
> > already in place to address the mode of operation (lockstep vs split). As
> > such I suggest to reuse all that code by making function
> > zynqmp_r5_get_tcm_node_from_dt() return a mem_bank_data array of 4 elements.
> > That array would be instantiated using the information found in the DT,
> > regardless of the mode of operation.  Once we have that array it could simply be
> > inserted in function zynqmp_r5_get_tcm_node() and everything else in the driver
> > works the same way.
> >
> > Note that for that work you will have to set the "reg" values of the second
> > core to 0 when in lockstep mode, which is fine because they are not used anyway.
> 
> I agree to most part. I will have to modify bindings as well to accommodate above design.
> 
> I think the problem is in lockstep mode, where two memory regions are combined and presented as single
> 
> memory-region in table: *zynqmp_tcm_banks_lockstep*. First I will have to fix that.
> 
> If that is fixed, then no need to have two functions for lockstep and split tcm carveouts. They both can be
> 
> handled uniformly. Once that is fixed, bindings needs to be fixed i.e.
> 
> For lockstep mode, "reg" will have 4 entries and split mode "reg" will have only two entries.
> 
> We don't have to make "reg" of core1 as 0 in lockstep mode, as it's not being used anyway.
> 
> Once I change bindings as above, then it's very easy to implement changes as suggested.

I also came to the conclusion the bindings needed to change.  I'm not sure to
understand everything you write above and as such will wait for your next
patchset.

> 
> We parse TCM information from dt if "reg" is available, and fill up the table in r5_core->tcm_banks and rest of the
> 
> driver remain same.
> 
> I will implement and test v10 accordingly. I think this should make design clean.
> 
> Only downside is, we are spliting contiguous region in lockstep mode, so after this change, firmware's linker script
> 
> needs to be updated accordingly for lockstep mode (If contiguous regions are considered which is not the case for Xilinx's firmwares).
> 
> I would consider current implementation as bug, and will send Fixes tag accordingly where needed.

Here too I won't claim to understand exactly what you mean but that is likely
because my head is currently in another patchset.  That said, make sure to keep
things backward compatible when moving forward with your changes.

> 
> 
> Thanks,
> 
> Tanmay
> 
> 
> > Thanks,
> > Mathieu
> >
> > > +		if (!r5_core->tcm_banks)
> > > +			ret = -ENOMEM;
> > > +
> > > +		r5_core->tcm_bank_count = tcm_bank_count;
> > > +		for (j = 0, tcm_pd_idx = 1; j < tcm_bank_count; j++, tcm_pd_idx++) {
> > > +			tcm = devm_kzalloc(dev, sizeof(struct mem_bank_data),
> > > +					   GFP_KERNEL);
> > > +			if (!tcm)
> > > +				return -ENOMEM;
> > > +
> > > +			r5_core->tcm_banks[j] = tcm;
> > > +
> > > +			/* Get power-domains id of TCM. */
> > > +			ret = of_parse_phandle_with_args(np, "power-domains",
> > > +							 "#power-domain-cells",
> > > +							 tcm_pd_idx, &out_args);
> > > +			if (ret) {
> > > +				dev_err(r5_core->dev,
> > > +					"failed to get tcm %d pm domain, ret %d\n",
> > > +					tcm_pd_idx, ret);
> > > +				return ret;
> > > +			}
> > > +			tcm->pm_domain_id = out_args.args[0];
> > > +			of_node_put(out_args.np);
> > > +
> > > +			/* Get TCM address without translation. */
> > > +			ret = of_property_read_reg(np, j, &abs_addr, &size);
> > > +			if (ret) {
> > > +				dev_err(dev, "failed to get reg property\n");
> > > +				return ret;
> > > +			}
> > > +
> > > +			/*
> > > +			 * Remote processor can address only 32 bits
> > > +			 * so convert 64-bits into 32-bits. This will discard
> > > +			 * any unwanted upper 32-bits.
> > > +			 */
> > > +			tcm->da = (u32)abs_addr;
> > > +			tcm->size = (u32)size;
> > > +
> > > +			cpdev = to_platform_device(dev);
> > > +			res = platform_get_resource(cpdev, IORESOURCE_MEM, j);
> > > +			if (!res) {
> > > +				dev_err(dev, "failed to get tcm resource\n");
> > > +				return -EINVAL;
> > > +			}
> > > +
> > > +			tcm->addr = (u32)res->start;
> > > +			tcm->bank_name = (char *)res->name;
> > > +			res = devm_request_mem_region(dev, tcm->addr, tcm->size,
> > > +						      tcm->bank_name);
> > > +			if (!res) {
> > > +				dev_err(dev, "failed to request tcm resource\n");
> > > +				return -EINVAL;
> > > +			}
> > > +		}
> > > +	}
> > > +
> > > +	return 0;
> > > +}
> > > +
> > >  /**
> > >   * zynqmp_r5_get_tcm_node()
> > >   * Ideally this function should parse tcm node and store information
> > > @@ -956,9 +1171,14 @@ static int zynqmp_r5_core_init(struct zynqmp_r5_cluster *cluster,
> > >  	struct zynqmp_r5_core *r5_core;
> > >  	int ret, i;
> > >  
> > > -	ret = zynqmp_r5_get_tcm_node(cluster);
> > > -	if (ret < 0) {
> > > -		dev_err(dev, "can't get tcm node, err %d\n", ret);
> > > +	r5_core = cluster->r5_cores[0];
> > > +	if (of_find_property(r5_core->np, "reg", NULL))
> > > +		ret = zynqmp_r5_get_tcm_node_from_dt(cluster);
> > > +	else
> > > +		ret = zynqmp_r5_get_tcm_node(cluster);
> > > +
> > > +	if (ret) {
> > > +		dev_err(dev, "can't get tcm, err %d\n", ret);
> > >  		return ret;
> > >  	}
> > >  
> > > @@ -1099,7 +1319,19 @@ static int zynqmp_r5_cluster_init(struct zynqmp_r5_cluster *cluster)
> > >  		 * then ignore second child node.
> > >  		 */
> > >  		if (cluster_mode == LOCKSTEP_MODE) {
> > > -			of_node_put(child);
> > > +			/*
> > > +			 * Get second core's device node only to use its power-domains.
> > > +			 * Also, no need to use of_node_put on first core's device_node
> > > +			 * as it is taken care by of_get_next_available_child.
> > > +			 */
> > > +			r5_cores[i]->lockstep_core1_np =
> > > +				of_get_next_available_child(dev_node, child);
> > > +
> > > +			if (!r5_cores[i]->lockstep_core1_np) {
> > > +				ret = -EINVAL;
> > > +				goto release_r5_cores;
> > > +			}
> > > +
> > >  			break;
> > >  		}
> > >  
> > > @@ -1158,6 +1390,7 @@ static void zynqmp_r5_cluster_exit(void *data)
> > >  		r5_core = cluster->r5_cores[i];
> > >  		zynqmp_r5_free_mbox(r5_core->ipi);
> > >  		of_reserved_mem_device_release(r5_core->dev);
> > > +		of_node_put(r5_core->lockstep_core1_np);
> > >  		put_device(r5_core->dev);
> > >  		rproc_del(r5_core->rproc);
> > >  		rproc_free(r5_core->rproc);
> > > -- 
> > > 2.25.1
> > >