[PATCH v10 0/2] Dynamic Allocation of the reserved_mem array

[PATCH v10 0/2] Dynamic Allocation of the reserved_mem array

[Oreoluwa Babatunde]

The reserved_mem array is used to store data for the different
reserved memory regions defined in the DT of a device.  The array
stores information such as region name, node reference, start-address,
and size of the different reserved memory regions.

The array is currently statically allocated with a size of
MAX_RESERVED_REGIONS(64). This means that any system that specifies a
number of reserved memory regions greater than MAX_RESERVED_REGIONS(64)
will not have enough space to store the information for all the regions.

This can be fixed by making the reserved_mem array a dynamically sized
array which is allocated using memblock_alloc() based on the exact
number of reserved memory regions defined in the DT.

On architectures such as arm64, memblock allocated memory is not
writable until after the page tables have been setup.
This is an issue because the current implementation initializes the
reserved memory regions and stores their information in the array before
the page tables are setup. Hence, dynamically allocating the
reserved_mem array and attempting to write information to it at this
point will fail.

Therefore, the allocation of the reserved_mem array will need to be done
after the page tables have been setup, which means that the reserved
memory regions will also need to wait until after the page tables have
been setup to be stored in the array.

When processing the reserved memory regions defined in the DT, these
regions are marked as reserved by calling memblock_reserve(base, size).
Where:  base = base address of the reserved region.
        size = the size of the reserved memory region.

Depending on if that region is defined using the "no-map" property,
memblock_mark_nomap(base, size) is also called.

The "no-map" property is used to indicate to the operating system that a
mapping of the specified region must NOT be created. This also means
that no access (including speculative accesses) is allowed on this
region of memory except when it is coming from the device driver that
this region of memory is being reserved for.[1]

Therefore, it is important to call memblock_reserve() and
memblock_mark_nomap() on all the reserved memory regions before the
system sets up the page tables so that the system does not unknowingly
include any of the no-map reserved memory regions in the memory map.

There are two ways to define how/where a reserved memory region is
placed in memory:
i) Statically-placed reserved memory regions
i.e. regions defined with a set start address and size using the
     "reg" property in the DT.
ii) Dynamically-placed reserved memory regions.
i.e. regions defined by specifying a range of addresses where they can
     be placed in memory using the "alloc_ranges" and "size" properties
     in the DT.

The dynamically-placed reserved memory regions get assigned a start
address only at runtime. And this needs to  be done before the page
tables are setup so that memblock_reserve() and memblock_mark_nomap()
can be called on the allocated region as explained above.
Since the dynamically allocated reserved_mem array can only be
available after the page tables have been setup, the information for
the dynamically-placed reserved memory regions needs to be stored
somewhere temporarily until the reserved_mem array is available.

Therefore, this series makes use of a temporary static array to store
the information of the dynamically-placed reserved memory regions until
the reserved_mem array is allocated.
Once the reserved_mem array is available, the information is copied over
from the temporary array into the reserved_mem array, and the memory for
the temporary array is freed back to the system.

The information for the statically-placed reserved memory regions does
not need to be stored in a temporary array because their starting
address is already stored in the devicetree.
Once the reserved_mem array is allocated, the information for the
statically-placed reserved memory regions is added to the array.

Note:
Because of the use of a temporary array to store the information of the
dynamically-placed reserved memory regions, there still exists a
limitation of 64 for this particular kind of reserved memory regions.
From my observation, these regions are typically small in number and
hence I expect this to not be an issue for now.

Patch Versions:

v10:
- Rebase patchset on v6.12-rc2.

v9:
- fix issue reported from v8:
  https://lore.kernel.org/all/DU0PR04MB92999E9EEE959DBC3B1EAB6E80932@DU0PR04MB9299.eurprd04.prod.outlook.com/
  In v8, the rmem struct being passed into __reserved_mem_init_node()
  was not the same as what was being stored in the reserved_mem array.
  As a result, information such as rmem->ops was not being stored in
  the array for these regions.
  Make changes to pass the same reserved_mem struct into
  __reserved_mem_init_node() as what is being stored in the reserved_mem
  array.

v8:
https://lore.kernel.org/all/20240830162857.2821502-1-quic_obabatun@quicinc.com/
- Check the value of initial_boot_params in
  fdt_scan_reserved_mem_reg_nodes() to avoid breakage on architectures
  where this is not being used as was found to be the case for x86 in
  the issues reported below:
  https://lore.kernel.org/all/202408192157.8d8fe8a9-oliver.sang@intel.com/
  https://lore.kernel.org/all/ZsN_p9l8Pw2_X3j3@black.fi.intel.com/

v7:
https://lore.kernel.org/all/20240809184814.2703050-1-quic_obabatun@quicinc.com/
- Make changes to initialize the reserved memory regions earlier in
  response to issue reported in v6:
  https://lore.kernel.org/all/20240610213403.GA1697364@thelio-3990X/

- For the reserved regions to be setup properly,
  fdt_init_reserved_mem_node() needs to be called on each of the regions
  before the page tables are setup. Since the function requires a
  refernece to the devicetree node of each region, we are not able to
  use the unflattened_devicetree APIs since they are not available until
  after the page tables have been setup.
  Hence, revert the use of the unflatten_device APIs as a result of this
  limitation which was discovered in v6:
  https://lore.kernel.org/all/986361f4-f000-4129-8214-39f2fb4a90da@gmail.com/
  https://lore.kernel.org/all/DU0PR04MB9299C3EC247E1FE2C373440F80DE2@DU0PR04MB9299.eurprd04.prod.outlook.com/

v6:
https://lore.kernel.org/all/20240528223650.619532-1-quic_obabatun@quicinc.com/
- Rebased patchset on top of v6.10-rc1.
- Addressed comments received in v5 such as:
  1. Switched to using relevant typed functions such as
     of_property_read_u32(), of_property_present(), etc.
  2. Switched to using of_address_to_resource() to read the "reg"
     property of nodes.
  3. Renamed functions using "of_*" naming scheme instead of "dt_*".

v5:
https://lore.kernel.org/all/20240328211543.191876-1-quic_obabatun@quicinc.com/
- Rebased changes on top of v6.9-rc1.
- Addressed minor code comments from v4.

v4:
https://lore.kernel.org/all/20240308191204.819487-2-quic_obabatun@quicinc.com/
- Move fdt_init_reserved_mem() back into the unflatten_device_tree()
  function.
- Fix warnings found by Kernel test robot:
  https://lore.kernel.org/all/202401281219.iIhqs1Si-lkp@intel.com/
  https://lore.kernel.org/all/202401281304.tsu89Kcm-lkp@intel.com/
  https://lore.kernel.org/all/202401291128.e7tdNh5x-lkp@intel.com/

v3:
https://lore.kernel.org/all/20240126235425.12233-1-quic_obabatun@quicinc.com/
- Make use of __initdata to delete the temporary static array after
  dynamically allocating memory for reserved_mem array using memblock.
- Move call to fdt_init_reserved_mem() out of the
  unflatten_device_tree() function and into architecture specific setup
  code.
- Breaking up the changes for the individual architectures into separate
  patches.

v2:
https://lore.kernel.org/all/20231204041339.9902-1-quic_obabatun@quicinc.com/
- Extend changes to all other relevant architectures by moving
  fdt_init_reserved_mem() into the unflatten_device_tree() function.
- Add code to use unflatten devicetree APIs to process the reserved
  memory regions.

v1:
https://lore.kernel.org/all/20231019184825.9712-1-quic_obabatun@quicinc.com/

References:
[1] https://github.com/devicetree-org/dt-schema/blob/main/dtschema/schemas/reserved-memory/reserved-memory.yaml#L79

Oreoluwa Babatunde (2):
  of: reserved_mem: Restruture how the reserved memory regions are
    processed
  of: reserved_mem: Add code to dynamically allocate reserved_mem array

 drivers/of/fdt.c             |   5 +-
 drivers/of/of_private.h      |   3 +-
 drivers/of/of_reserved_mem.c | 227 +++++++++++++++++++++++++++--------
 3 files changed, 179 insertions(+), 56 deletions(-)

-- 
2.34.1

[PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Oreoluwa Babatunde]

Reserved memory regions defined in the devicetree can be broken up into
two groups:
i) Statically-placed reserved memory regions
i.e. regions defined with a static start address and size using the
     "reg" property.
ii) Dynamically-placed reserved memory regions.
i.e. regions defined by specifying an address range where they can be
     placed in memory using the "alloc_ranges" and "size" properties.

These regions are processed and set aside at boot time.
This is done in two stages as seen below:

Stage 1:
At this stage, fdt_scan_reserved_mem() scans through the child nodes of
the reserved_memory node using the flattened devicetree and does the
following:

1) If the node represents a statically-placed reserved memory region,
   i.e. if it is defined using the "reg" property:
   - Call memblock_reserve() or memblock_mark_nomap() as needed.
   - Add the information for that region into the reserved_mem array
     using fdt_reserved_mem_save_node().
     i.e. fdt_reserved_mem_save_node(node, name, base, size).

2) If the node represents a dynamically-placed reserved memory region,
   i.e. if it is defined using "alloc-ranges" and "size" properties:
   - Add the information for that region to the reserved_mem array with
     the starting address and size set to 0.
     i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
   Note: This region is saved to the array with a starting address of 0
   because a starting address is not yet allocated for it.

Stage 2:
After iterating through all the reserved memory nodes and storing their
relevant information in the reserved_mem array,fdt_init_reserved_mem() is
called and does the following:

1) For statically-placed reserved memory regions:
   - Call the region specific init function using
     __reserved_mem_init_node().
2) For dynamically-placed reserved memory regions:
   - Call __reserved_mem_alloc_size() which is used to allocate memory
     for each of these regions, and mark them as nomap if they have the
     nomap property specified in the DT.
   - Call the region specific init function.

The current size of the resvered_mem array is 64 as is defined by
MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
how many reserved memory regions can be specified on a system.
As systems continue to grow more and more complex, the number of
reserved memory regions needed are also growing and are starting to hit
this 64 count limit, hence the need to make the reserved_mem array
dynamically sized (i.e. dynamically allocating memory for the
reserved_mem array using membock_alloc_*).

On architectures such as arm64, memory allocated using memblock is
writable only after the page tables have been setup. This means that if
the reserved_mem array is going to be dynamically allocated, it needs to
happen after the page tables have been setup, not before.

Since the reserved memory regions are currently being processed and
added to the array before the page tables are setup, there is a need to
change the order in which some of the processing is done to allow for
the reserved_mem array to be dynamically sized.

It is possible to process the statically-placed reserved memory regions
without needing to store them in the reserved_mem array until after the
page tables have been setup because all the information stored in the
array is readily available in the devicetree and can be referenced at
any time.
Dynamically-placed reserved memory regions on the other hand get
assigned a start address only at runtime, and hence need a place to be
stored once they are allocated since there is no other referrence to the
start address for these regions.

Hence this patch changes the processing order of the reserved memory
regions in the following ways:

Step 1:
fdt_scan_reserved_mem() scans through the child nodes of
the reserved_memory node using the flattened devicetree and does the
following:

1) If the node represents a statically-placed reserved memory region,
   i.e. if it is defined using the "reg" property:
   - Call memblock_reserve() or memblock_mark_nomap() as needed.

2) If the node represents a dynamically-placed reserved memory region,
   i.e. if it is defined using "alloc-ranges" and "size" properties:
   - Call __reserved_mem_alloc_size() which will:
     i) Allocate memory for the reserved region and call
     memblock_mark_nomap() as needed.
     ii) Call the region specific initialization function using
     fdt_init_reserved_mem_node().
     iii) Save the region information in the reserved_mem array using
     fdt_reserved_mem_save_node().

Step 2:
1) This stage of the reserved memory processing is now only used to add
   the statically-placed reserved memory regions into the reserved_mem
   array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
   region specific initialization functions.

2) This step has also been moved to be after the page tables are
   setup. Moving this will allow us to replace the reserved_mem
   array with a dynamically sized array before storing the rest of
   these regions.

Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
---
 drivers/of/fdt.c             |   5 +-
 drivers/of/of_private.h      |   3 +-
 drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
 3 files changed, 122 insertions(+), 54 deletions(-)

diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
index 4d528c10df3a..d0dbc8183ac4 100644
--- a/drivers/of/fdt.c
+++ b/drivers/of/fdt.c
@@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
 			break;
 		memblock_reserve(base, size);
 	}
-
-	fdt_init_reserved_mem();
 }
 
 /**
@@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
 {
 	void *fdt = initial_boot_params;
 
+	/* Save the statically-placed regions in the reserved_mem array */
+	fdt_scan_reserved_mem_reg_nodes();
+
 	/* Don't use the bootloader provided DTB if ACPI is enabled */
 	if (!acpi_disabled)
 		fdt = NULL;
diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
index 04aa2a91f851..29525c0b9939 100644
--- a/drivers/of/of_private.h
+++ b/drivers/of/of_private.h
@@ -9,6 +9,7 @@
  */
 
 #define FDT_ALIGN_SIZE 8
+#define MAX_RESERVED_REGIONS    64
 
 /**
  * struct alias_prop - Alias property in 'aliases' node
@@ -180,7 +181,7 @@ static inline struct device_node *__of_get_dma_parent(const struct device_node *
 #endif
 
 int fdt_scan_reserved_mem(void);
-void fdt_init_reserved_mem(void);
+void __init fdt_scan_reserved_mem_reg_nodes(void);
 
 bool of_fdt_device_is_available(const void *blob, unsigned long node);
 
diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
index 46e1c3fbc769..2011174211f9 100644
--- a/drivers/of/of_reserved_mem.c
+++ b/drivers/of/of_reserved_mem.c
@@ -27,7 +27,6 @@
 
 #include "of_private.h"
 
-#define MAX_RESERVED_REGIONS	64
 static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
 static int reserved_mem_count;
 
@@ -56,6 +55,7 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
 	return err;
 }
 
+static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem);
 /*
  * fdt_reserved_mem_save_node() - save fdt node for second pass initialization
  */
@@ -74,6 +74,9 @@ static void __init fdt_reserved_mem_save_node(unsigned long node, const char *un
 	rmem->base = base;
 	rmem->size = size;
 
+	/* Call the region specific initialization function */
+	fdt_init_reserved_mem_node(rmem);
+
 	reserved_mem_count++;
 	return;
 }
@@ -106,7 +109,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
 	phys_addr_t base, size;
 	int len;
 	const __be32 *prop;
-	int first = 1;
 	bool nomap;
 
 	prop = of_get_flat_dt_prop(node, "reg", &len);
@@ -134,10 +136,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
 			       uname, &base, (unsigned long)(size / SZ_1M));
 
 		len -= t_len;
-		if (first) {
-			fdt_reserved_mem_save_node(node, uname, base, size);
-			first = 0;
-		}
 	}
 	return 0;
 }
@@ -165,12 +163,77 @@ static int __init __reserved_mem_check_root(unsigned long node)
 	return 0;
 }
 
+static void __init __rmem_check_for_overlap(void);
+
+/**
+ * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg" defined
+ * reserved memory regions.
+ *
+ * This function is used to scan through the DT and store the
+ * information for the reserved memory regions that are defined using
+ * the "reg" property. The region node number, name, base address, and
+ * size are all stored in the reserved_mem array by calling the
+ * fdt_reserved_mem_save_node() function.
+ */
+void __init fdt_scan_reserved_mem_reg_nodes(void)
+{
+	int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
+	const void *fdt = initial_boot_params;
+	phys_addr_t base, size;
+	const __be32 *prop;
+	int node, child;
+	int len;
+
+	if (!fdt)
+		return;
+
+	node = fdt_path_offset(fdt, "/reserved-memory");
+	if (node < 0) {
+		pr_info("Reserved memory: No reserved-memory node in the DT\n");
+		return;
+	}
+
+	if (__reserved_mem_check_root(node)) {
+		pr_err("Reserved memory: unsupported node format, ignoring\n");
+		return;
+	}
+
+	fdt_for_each_subnode(child, fdt, node) {
+		const char *uname;
+
+		prop = of_get_flat_dt_prop(child, "reg", &len);
+		if (!prop)
+			continue;
+		if (!of_fdt_device_is_available(fdt, child))
+			continue;
+
+		uname = fdt_get_name(fdt, child, NULL);
+		if (len && len % t_len != 0) {
+			pr_err("Reserved memory: invalid reg property in '%s', skipping node.\n",
+			       uname);
+			continue;
+		}
+		base = dt_mem_next_cell(dt_root_addr_cells, &prop);
+		size = dt_mem_next_cell(dt_root_size_cells, &prop);
+
+		if (size)
+			fdt_reserved_mem_save_node(child, uname, base, size);
+	}
+
+	/* check for overlapping reserved regions */
+	__rmem_check_for_overlap();
+}
+
+static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname);
+
 /*
  * fdt_scan_reserved_mem() - scan a single FDT node for reserved memory
  */
 int __init fdt_scan_reserved_mem(void)
 {
 	int node, child;
+	int dynamic_nodes_cnt = 0;
+	int dynamic_nodes[MAX_RESERVED_REGIONS];
 	const void *fdt = initial_boot_params;
 
 	node = fdt_path_offset(fdt, "/reserved-memory");
@@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
 		uname = fdt_get_name(fdt, child, NULL);
 
 		err = __reserved_mem_reserve_reg(child, uname);
-		if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL))
-			fdt_reserved_mem_save_node(child, uname, 0, 0);
+		/*
+		 * Save the nodes for the dynamically-placed regions
+		 * into an array which will be used for allocation right
+		 * after all the statically-placed regions are reserved
+		 * or marked as no-map. This is done to avoid dynamically
+		 * allocating from one of the statically-placed regions.
+		 */
+		if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL)) {
+			dynamic_nodes[dynamic_nodes_cnt] = child;
+			dynamic_nodes_cnt++;
+		}
+	}
+	for (int i = 0; i < dynamic_nodes_cnt; i++) {
+		const char *uname;
+
+		child = dynamic_nodes[i];
+		uname = fdt_get_name(fdt, child, NULL);
+		__reserved_mem_alloc_size(child, uname);
 	}
 	return 0;
 }
@@ -253,8 +332,7 @@ static int __init __reserved_mem_alloc_in_range(phys_addr_t size,
  * __reserved_mem_alloc_size() - allocate reserved memory described by
  *	'size', 'alignment'  and 'alloc-ranges' properties.
  */
-static int __init __reserved_mem_alloc_size(unsigned long node,
-	const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
+static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname)
 {
 	int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
 	phys_addr_t start = 0, end = 0;
@@ -334,9 +412,8 @@ static int __init __reserved_mem_alloc_size(unsigned long node,
 		return -ENOMEM;
 	}
 
-	*res_base = base;
-	*res_size = size;
-
+	/* Save region in the reserved_mem array */
+	fdt_reserved_mem_save_node(node, uname, base, size);
 	return 0;
 }
 
@@ -425,48 +502,37 @@ static void __init __rmem_check_for_overlap(void)
 }
 
 /**
- * fdt_init_reserved_mem() - allocate and init all saved reserved memory regions
+ * fdt_init_reserved_mem_node() - Initialize a reserved memory region
+ * @rmem: reserved_mem struct of the memory region to be initialized.
+ *
+ * This function is used to call the region specific initialization
+ * function for a reserved memory region.
  */
-void __init fdt_init_reserved_mem(void)
+static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem)
 {
-	int i;
-
-	/* check for overlapping reserved regions */
-	__rmem_check_for_overlap();
-
-	for (i = 0; i < reserved_mem_count; i++) {
-		struct reserved_mem *rmem = &reserved_mem[i];
-		unsigned long node = rmem->fdt_node;
-		int err = 0;
-		bool nomap;
+	unsigned long node = rmem->fdt_node;
+	int err = 0;
+	bool nomap;
 
-		nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
+	nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
 
-		if (rmem->size == 0)
-			err = __reserved_mem_alloc_size(node, rmem->name,
-						 &rmem->base, &rmem->size);
-		if (err == 0) {
-			err = __reserved_mem_init_node(rmem);
-			if (err != 0 && err != -ENOENT) {
-				pr_info("node %s compatible matching fail\n",
-					rmem->name);
-				if (nomap)
-					memblock_clear_nomap(rmem->base, rmem->size);
-				else
-					memblock_phys_free(rmem->base,
-							   rmem->size);
-			} else {
-				phys_addr_t end = rmem->base + rmem->size - 1;
-				bool reusable =
-					(of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
-
-				pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
-					&rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
-					nomap ? "nomap" : "map",
-					reusable ? "reusable" : "non-reusable",
-					rmem->name ? rmem->name : "unknown");
-			}
-		}
+	err = __reserved_mem_init_node(rmem);
+	if (err != 0 && err != -ENOENT) {
+		pr_info("node %s compatible matching fail\n", rmem->name);
+		if (nomap)
+			memblock_clear_nomap(rmem->base, rmem->size);
+		else
+			memblock_phys_free(rmem->base, rmem->size);
+	} else {
+		phys_addr_t end = rmem->base + rmem->size - 1;
+		bool reusable =
+			(of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
+
+		pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
+			&rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
+			nomap ? "nomap" : "map",
+			reusable ? "reusable" : "non-reusable",
+			rmem->name ? rmem->name : "unknown");
 	}
 }
 
-- 
2.34.1

[PATCH v10 2/2] of: reserved_mem: Add code to dynamically allocate reserved_mem array

[Oreoluwa Babatunde]

The reserved_mem array is statically allocated with a size of
MAX_RESERVED_REGIONS(64). Therefore, if the number of reserved_mem
regions exceeds this size, there will not be enough space to store
all the data.

Hence, extend the use of the static array by introducing a
dynamically allocated array based on the number of reserved memory
regions specified in the DT.

On architectures such as arm64, memblock allocated memory is not
writable until after the page tables have been setup. Hence, the
dynamic allocation of the reserved_mem array will need to be done only
after the page tables have been setup.

As a result, a temporary static array is still needed in the initial
stages to store the information of the dynamically-placed reserved
memory regions because the start address is selected only at run-time
and is not stored anywhere else.
It is not possible to wait until the reserved_mem array is allocated
because this is done after the page tables are setup and the reserved
memory regions need to be initialized before then.

After the reserved_mem array is allocated, all entries from the static
array is copied over to the new array, and the rest of the information
for the statically-placed reserved memory regions are read in from the
DT and stored in the new array as well.

Once the init process is completed, the temporary static array is
released back to the system because it is no longer needed. This is
achieved by marking it as __initdata.

Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
---
 drivers/of/of_reserved_mem.c | 63 +++++++++++++++++++++++++++++++++---
 1 file changed, 59 insertions(+), 4 deletions(-)

diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
index 2011174211f9..45517b9e57b1 100644
--- a/drivers/of/of_reserved_mem.c
+++ b/drivers/of/of_reserved_mem.c
@@ -27,7 +27,9 @@
 
 #include "of_private.h"
 
-static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
+static struct reserved_mem reserved_mem_array[MAX_RESERVED_REGIONS] __initdata;
+static struct reserved_mem *reserved_mem __refdata = reserved_mem_array;
+static int total_reserved_mem_cnt = MAX_RESERVED_REGIONS;
 static int reserved_mem_count;
 
 static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
@@ -55,6 +57,50 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
 	return err;
 }
 
+/*
+ * alloc_reserved_mem_array() - allocate memory for the reserved_mem
+ * array using memblock
+ *
+ * This function is used to allocate memory for the reserved_mem
+ * array according to the total number of reserved memory regions
+ * defined in the DT.
+ * After the new array is allocated, the information stored in
+ * the initial static array is copied over to this new array and
+ * the new array is used from this point on.
+ */
+static void __init alloc_reserved_mem_array(void)
+{
+	struct reserved_mem *new_array;
+	size_t alloc_size, copy_size, memset_size;
+
+	alloc_size = array_size(total_reserved_mem_cnt, sizeof(*new_array));
+	if (alloc_size == SIZE_MAX) {
+		pr_err("Failed to allocate memory for reserved_mem array with err: %d", -EOVERFLOW);
+		return;
+	}
+
+	new_array = memblock_alloc(alloc_size, SMP_CACHE_BYTES);
+	if (!new_array) {
+		pr_err("Failed to allocate memory for reserved_mem array with err: %d", -ENOMEM);
+		return;
+	}
+
+	copy_size = array_size(reserved_mem_count, sizeof(*new_array));
+	if (copy_size == SIZE_MAX) {
+		memblock_free(new_array, alloc_size);
+		total_reserved_mem_cnt = MAX_RESERVED_REGIONS;
+		pr_err("Failed to allocate memory for reserved_mem array with err: %d", -EOVERFLOW);
+		return;
+	}
+
+	memset_size = alloc_size - copy_size;
+
+	memcpy(new_array, reserved_mem, copy_size);
+	memset(new_array + reserved_mem_count, 0, memset_size);
+
+	reserved_mem = new_array;
+}
+
 static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem);
 /*
  * fdt_reserved_mem_save_node() - save fdt node for second pass initialization
@@ -64,7 +110,7 @@ static void __init fdt_reserved_mem_save_node(unsigned long node, const char *un
 {
 	struct reserved_mem *rmem = &reserved_mem[reserved_mem_count];
 
-	if (reserved_mem_count == ARRAY_SIZE(reserved_mem)) {
+	if (reserved_mem_count == total_reserved_mem_cnt) {
 		pr_err("not enough space for all defined regions.\n");
 		return;
 	}
@@ -193,6 +239,9 @@ void __init fdt_scan_reserved_mem_reg_nodes(void)
 		return;
 	}
 
+	/* Attempt dynamic allocation of a new reserved_mem array */
+	alloc_reserved_mem_array();
+
 	if (__reserved_mem_check_root(node)) {
 		pr_err("Reserved memory: unsupported node format, ignoring\n");
 		return;
@@ -232,7 +281,7 @@ static int __init __reserved_mem_alloc_size(unsigned long node, const char *unam
 int __init fdt_scan_reserved_mem(void)
 {
 	int node, child;
-	int dynamic_nodes_cnt = 0;
+	int dynamic_nodes_cnt = 0, count = 0;
 	int dynamic_nodes[MAX_RESERVED_REGIONS];
 	const void *fdt = initial_boot_params;
 
@@ -255,6 +304,8 @@ int __init fdt_scan_reserved_mem(void)
 		uname = fdt_get_name(fdt, child, NULL);
 
 		err = __reserved_mem_reserve_reg(child, uname);
+		if (!err)
+			count++;
 		/*
 		 * Save the nodes for the dynamically-placed regions
 		 * into an array which will be used for allocation right
@@ -269,11 +320,15 @@ int __init fdt_scan_reserved_mem(void)
 	}
 	for (int i = 0; i < dynamic_nodes_cnt; i++) {
 		const char *uname;
+		int err;
 
 		child = dynamic_nodes[i];
 		uname = fdt_get_name(fdt, child, NULL);
-		__reserved_mem_alloc_size(child, uname);
+		err = __reserved_mem_alloc_size(child, uname);
+		if (!err)
+			count++;
 	}
+	total_reserved_mem_cnt = count;
 	return 0;
 }
 
-- 
2.34.1

Re: [PATCH v10 0/2] Dynamic Allocation of the reserved_mem array

[Rob Herring]

On Tue, Oct 08, 2024 at 03:06:22PM -0700, Oreoluwa Babatunde wrote:
> The reserved_mem array is used to store data for the different
> reserved memory regions defined in the DT of a device.  The array
> stores information such as region name, node reference, start-address,
> and size of the different reserved memory regions.
> 
> The array is currently statically allocated with a size of
> MAX_RESERVED_REGIONS(64). This means that any system that specifies a
> number of reserved memory regions greater than MAX_RESERVED_REGIONS(64)
> will not have enough space to store the information for all the regions.
> 
> This can be fixed by making the reserved_mem array a dynamically sized
> array which is allocated using memblock_alloc() based on the exact
> number of reserved memory regions defined in the DT.
> 
> On architectures such as arm64, memblock allocated memory is not
> writable until after the page tables have been setup.
> This is an issue because the current implementation initializes the
> reserved memory regions and stores their information in the array before
> the page tables are setup. Hence, dynamically allocating the
> reserved_mem array and attempting to write information to it at this
> point will fail.
> 
> Therefore, the allocation of the reserved_mem array will need to be done
> after the page tables have been setup, which means that the reserved
> memory regions will also need to wait until after the page tables have
> been setup to be stored in the array.
> 
> When processing the reserved memory regions defined in the DT, these
> regions are marked as reserved by calling memblock_reserve(base, size).
> Where:  base = base address of the reserved region.
>         size = the size of the reserved memory region.
> 
> Depending on if that region is defined using the "no-map" property,
> memblock_mark_nomap(base, size) is also called.
> 
> The "no-map" property is used to indicate to the operating system that a
> mapping of the specified region must NOT be created. This also means
> that no access (including speculative accesses) is allowed on this
> region of memory except when it is coming from the device driver that
> this region of memory is being reserved for.[1]
> 
> Therefore, it is important to call memblock_reserve() and
> memblock_mark_nomap() on all the reserved memory regions before the
> system sets up the page tables so that the system does not unknowingly
> include any of the no-map reserved memory regions in the memory map.
> 
> There are two ways to define how/where a reserved memory region is
> placed in memory:
> i) Statically-placed reserved memory regions
> i.e. regions defined with a set start address and size using the
>      "reg" property in the DT.
> ii) Dynamically-placed reserved memory regions.
> i.e. regions defined by specifying a range of addresses where they can
>      be placed in memory using the "alloc_ranges" and "size" properties
>      in the DT.
> 
> The dynamically-placed reserved memory regions get assigned a start
> address only at runtime. And this needs to  be done before the page
> tables are setup so that memblock_reserve() and memblock_mark_nomap()
> can be called on the allocated region as explained above.
> Since the dynamically allocated reserved_mem array can only be
> available after the page tables have been setup, the information for
> the dynamically-placed reserved memory regions needs to be stored
> somewhere temporarily until the reserved_mem array is available.
> 
> Therefore, this series makes use of a temporary static array to store
> the information of the dynamically-placed reserved memory regions until
> the reserved_mem array is allocated.
> Once the reserved_mem array is available, the information is copied over
> from the temporary array into the reserved_mem array, and the memory for
> the temporary array is freed back to the system.
> 
> The information for the statically-placed reserved memory regions does
> not need to be stored in a temporary array because their starting
> address is already stored in the devicetree.
> Once the reserved_mem array is allocated, the information for the
> statically-placed reserved memory regions is added to the array.
> 
> Note:
> Because of the use of a temporary array to store the information of the
> dynamically-placed reserved memory regions, there still exists a
> limitation of 64 for this particular kind of reserved memory regions.
> >From my observation, these regions are typically small in number and
> hence I expect this to not be an issue for now.
> 
> Patch Versions:
> 
> v10:
> - Rebase patchset on v6.12-rc2.
> 
> v9:
> - fix issue reported from v8:
>   https://lore.kernel.org/all/DU0PR04MB92999E9EEE959DBC3B1EAB6E80932@DU0PR04MB9299.eurprd04.prod.outlook.com/
>   In v8, the rmem struct being passed into __reserved_mem_init_node()
>   was not the same as what was being stored in the reserved_mem array.
>   As a result, information such as rmem->ops was not being stored in
>   the array for these regions.
>   Make changes to pass the same reserved_mem struct into
>   __reserved_mem_init_node() as what is being stored in the reserved_mem
>   array.
> 
> v8:
> https://lore.kernel.org/all/20240830162857.2821502-1-quic_obabatun@quicinc.com/
> - Check the value of initial_boot_params in
>   fdt_scan_reserved_mem_reg_nodes() to avoid breakage on architectures
>   where this is not being used as was found to be the case for x86 in
>   the issues reported below:
>   https://lore.kernel.org/all/202408192157.8d8fe8a9-oliver.sang@intel.com/
>   https://lore.kernel.org/all/ZsN_p9l8Pw2_X3j3@black.fi.intel.com/
> 
> v7:
> https://lore.kernel.org/all/20240809184814.2703050-1-quic_obabatun@quicinc.com/
> - Make changes to initialize the reserved memory regions earlier in
>   response to issue reported in v6:
>   https://lore.kernel.org/all/20240610213403.GA1697364@thelio-3990X/
> 
> - For the reserved regions to be setup properly,
>   fdt_init_reserved_mem_node() needs to be called on each of the regions
>   before the page tables are setup. Since the function requires a
>   refernece to the devicetree node of each region, we are not able to
>   use the unflattened_devicetree APIs since they are not available until
>   after the page tables have been setup.
>   Hence, revert the use of the unflatten_device APIs as a result of this
>   limitation which was discovered in v6:
>   https://lore.kernel.org/all/986361f4-f000-4129-8214-39f2fb4a90da@gmail.com/
>   https://lore.kernel.org/all/DU0PR04MB9299C3EC247E1FE2C373440F80DE2@DU0PR04MB9299.eurprd04.prod.outlook.com/
> 
> v6:
> https://lore.kernel.org/all/20240528223650.619532-1-quic_obabatun@quicinc.com/
> - Rebased patchset on top of v6.10-rc1.
> - Addressed comments received in v5 such as:
>   1. Switched to using relevant typed functions such as
>      of_property_read_u32(), of_property_present(), etc.
>   2. Switched to using of_address_to_resource() to read the "reg"
>      property of nodes.
>   3. Renamed functions using "of_*" naming scheme instead of "dt_*".
> 
> v5:
> https://lore.kernel.org/all/20240328211543.191876-1-quic_obabatun@quicinc.com/
> - Rebased changes on top of v6.9-rc1.
> - Addressed minor code comments from v4.
> 
> v4:
> https://lore.kernel.org/all/20240308191204.819487-2-quic_obabatun@quicinc.com/
> - Move fdt_init_reserved_mem() back into the unflatten_device_tree()
>   function.
> - Fix warnings found by Kernel test robot:
>   https://lore.kernel.org/all/202401281219.iIhqs1Si-lkp@intel.com/
>   https://lore.kernel.org/all/202401281304.tsu89Kcm-lkp@intel.com/
>   https://lore.kernel.org/all/202401291128.e7tdNh5x-lkp@intel.com/
> 
> v3:
> https://lore.kernel.org/all/20240126235425.12233-1-quic_obabatun@quicinc.com/
> - Make use of __initdata to delete the temporary static array after
>   dynamically allocating memory for reserved_mem array using memblock.
> - Move call to fdt_init_reserved_mem() out of the
>   unflatten_device_tree() function and into architecture specific setup
>   code.
> - Breaking up the changes for the individual architectures into separate
>   patches.
> 
> v2:
> https://lore.kernel.org/all/20231204041339.9902-1-quic_obabatun@quicinc.com/
> - Extend changes to all other relevant architectures by moving
>   fdt_init_reserved_mem() into the unflatten_device_tree() function.
> - Add code to use unflatten devicetree APIs to process the reserved
>   memory regions.
> 
> v1:
> https://lore.kernel.org/all/20231019184825.9712-1-quic_obabatun@quicinc.com/
> 
> References:
> [1] https://github.com/devicetree-org/dt-schema/blob/main/dtschema/schemas/reserved-memory/reserved-memory.yaml#L79
> 
> Oreoluwa Babatunde (2):
>   of: reserved_mem: Restruture how the reserved memory regions are
>     processed
>   of: reserved_mem: Add code to dynamically allocate reserved_mem array
> 
>  drivers/of/fdt.c             |   5 +-
>  drivers/of/of_private.h      |   3 +-
>  drivers/of/of_reserved_mem.c | 227 +++++++++++++++++++++++++++--------
>  3 files changed, 179 insertions(+), 56 deletions(-)

Applied, thanks.

Rob

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Geert Uytterhoeven]

Hi Oreoluwa,

On Wed, Oct 9, 2024 at 12:08 AM Oreoluwa Babatunde
<quic_obabatun@quicinc.com> wrote:
> Reserved memory regions defined in the devicetree can be broken up into
> two groups:
> i) Statically-placed reserved memory regions
> i.e. regions defined with a static start address and size using the
>      "reg" property.
> ii) Dynamically-placed reserved memory regions.
> i.e. regions defined by specifying an address range where they can be
>      placed in memory using the "alloc_ranges" and "size" properties.
>
> These regions are processed and set aside at boot time.
> This is done in two stages as seen below:
>
> Stage 1:
> At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> the reserved_memory node using the flattened devicetree and does the
> following:
>
> 1) If the node represents a statically-placed reserved memory region,
>    i.e. if it is defined using the "reg" property:
>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
>    - Add the information for that region into the reserved_mem array
>      using fdt_reserved_mem_save_node().
>      i.e. fdt_reserved_mem_save_node(node, name, base, size).
>
> 2) If the node represents a dynamically-placed reserved memory region,
>    i.e. if it is defined using "alloc-ranges" and "size" properties:
>    - Add the information for that region to the reserved_mem array with
>      the starting address and size set to 0.
>      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>    Note: This region is saved to the array with a starting address of 0
>    because a starting address is not yet allocated for it.
>
> Stage 2:
> After iterating through all the reserved memory nodes and storing their
> relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> called and does the following:
>
> 1) For statically-placed reserved memory regions:
>    - Call the region specific init function using
>      __reserved_mem_init_node().
> 2) For dynamically-placed reserved memory regions:
>    - Call __reserved_mem_alloc_size() which is used to allocate memory
>      for each of these regions, and mark them as nomap if they have the
>      nomap property specified in the DT.
>    - Call the region specific init function.
>
> The current size of the resvered_mem array is 64 as is defined by
> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> how many reserved memory regions can be specified on a system.
> As systems continue to grow more and more complex, the number of
> reserved memory regions needed are also growing and are starting to hit
> this 64 count limit, hence the need to make the reserved_mem array
> dynamically sized (i.e. dynamically allocating memory for the
> reserved_mem array using membock_alloc_*).
>
> On architectures such as arm64, memory allocated using memblock is
> writable only after the page tables have been setup. This means that if
> the reserved_mem array is going to be dynamically allocated, it needs to
> happen after the page tables have been setup, not before.
>
> Since the reserved memory regions are currently being processed and
> added to the array before the page tables are setup, there is a need to
> change the order in which some of the processing is done to allow for
> the reserved_mem array to be dynamically sized.
>
> It is possible to process the statically-placed reserved memory regions
> without needing to store them in the reserved_mem array until after the
> page tables have been setup because all the information stored in the
> array is readily available in the devicetree and can be referenced at
> any time.
> Dynamically-placed reserved memory regions on the other hand get
> assigned a start address only at runtime, and hence need a place to be
> stored once they are allocated since there is no other referrence to the
> start address for these regions.
>
> Hence this patch changes the processing order of the reserved memory
> regions in the following ways:
>
> Step 1:
> fdt_scan_reserved_mem() scans through the child nodes of
> the reserved_memory node using the flattened devicetree and does the
> following:
>
> 1) If the node represents a statically-placed reserved memory region,
>    i.e. if it is defined using the "reg" property:
>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
>
> 2) If the node represents a dynamically-placed reserved memory region,
>    i.e. if it is defined using "alloc-ranges" and "size" properties:
>    - Call __reserved_mem_alloc_size() which will:
>      i) Allocate memory for the reserved region and call
>      memblock_mark_nomap() as needed.
>      ii) Call the region specific initialization function using
>      fdt_init_reserved_mem_node().
>      iii) Save the region information in the reserved_mem array using
>      fdt_reserved_mem_save_node().
>
> Step 2:
> 1) This stage of the reserved memory processing is now only used to add
>    the statically-placed reserved memory regions into the reserved_mem
>    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
>    region specific initialization functions.
>
> 2) This step has also been moved to be after the page tables are
>    setup. Moving this will allow us to replace the reserved_mem
>    array with a dynamically sized array before storing the rest of
>    these regions.
>
> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>

Thanks for your patch, which is now commit 8a6e02d0c00e7b62
("of: reserved_mem: Restructure how the reserved memory regions
are processed") in dt-rh/for-next.

I have bisected a boot issue on RZ/Five to this commit.
With "earlycon keep_bootcon" (else there is no output):

    Oops - store (or AMO) access fault [#1]
    CPU: 0 UID: 0 PID: 1 Comm: swapper Not tainted
6.12.0-rc1-00015-g8a6e02d0c00e #201
    Hardware name: Renesas SMARC EVK based on r9a07g043f01 (DT)
    epc : __memset+0x60/0x100
     ra : __dma_alloc_from_coherent+0x150/0x17a
    epc : ffffffff8062d2bc ra : ffffffff80053a94 sp : ffffffc60000ba20
     gp : ffffffff812e9938 tp : ffffffd601920000 t0 : ffffffc6000d0000
     t1 : 0000000000000000 t2 : ffffffffe9600000 s0 : ffffffc60000baa0
     s1 : ffffffc6000d0000 a0 : ffffffc6000d0000 a1 : 0000000000000000
     a2 : 0000000000001000 a3 : ffffffc6000d1000 a4 : 0000000000000000
     a5 : 0000000000000000 a6 : ffffffd601adacc0 a7 : ffffffd601a841a8
     s2 : ffffffd6018573c0 s3 : 0000000000001000 s4 : ffffffd6019541e0
     s5 : 0000000200000022 s6 : ffffffd6018f8410 s7 : ffffffd6018573e8
     s8 : 0000000000000001 s9 : 0000000000000001 s10: 0000000000000010
     s11: 0000000000000000 t3 : 0000000000000000 t4 : ffffffffdefe62d1
     t5 : 000000001cd6a3a9 t6 : ffffffd601b2aad6
    status: 0000000200000120 badaddr: ffffffc6000d0000 cause: 0000000000000007
    [<ffffffff8062d2bc>] __memset+0x60/0x100
    [<ffffffff80053e1a>] dma_alloc_from_global_coherent+0x1c/0x28
    [<ffffffff80053056>] dma_direct_alloc+0x98/0x112
    [<ffffffff8005238c>] dma_alloc_attrs+0x78/0x86
    [<ffffffff8035fdb4>] rz_dmac_probe+0x3f6/0x50a
    [<ffffffff803a0694>] platform_probe+0x4c/0x8a
    [<ffffffff8039ea16>] really_probe+0xe4/0x1c8
    [<ffffffff8039ebc4>] __driver_probe_device+0xca/0xce
    [<ffffffff8039ec48>] driver_probe_device+0x34/0x92
    [<ffffffff8039ede8>] __driver_attach+0xb4/0xbe
    [<ffffffff8039ce58>] bus_for_each_dev+0x60/0xa0
    [<ffffffff8039e26a>] driver_attach+0x1a/0x22
    [<ffffffff8039dc20>] bus_add_driver+0xa4/0x184
    [<ffffffff8039f65c>] driver_register+0x8a/0xb4
    [<ffffffff803a051c>] __platform_driver_register+0x1c/0x24
    [<ffffffff808202f6>] rz_dmac_driver_init+0x1a/0x22
    [<ffffffff80800ef6>] do_one_initcall+0x64/0x134
    [<ffffffff8080122e>] kernel_init_freeable+0x200/0x202
    [<ffffffff80638126>] kernel_init+0x1e/0x10a
    [<ffffffff8063d58e>] ret_from_fork+0xe/0x18
    Code: 1007 82b3 40e2 0797 0000 8793 00e7 8305 97ba 8782 (b023) 00b2
    ---[ end trace 0000000000000000 ]---
    Kernel panic - not syncing: Fatal exception in interrupt
    ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---

Nothing really stands out in the kernel log, except for a delayed
initialization of the reserved mem nodes (they are the same
before/after):

 printk: debug: ignoring loglevel setting.
-OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
nomap non-reusable mmode_resv0@30000
-OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
nomap non-reusable mmode_resv1@40000
-OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
nomap non-reusable mmode_resv3@44000000
-OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
nomap non-reusable mmode_resv2@44040000
+earlycon: scif0 at MMIO 0x000000001004b800 (options '115200n8')
+printk: legacy bootconsole [scif0] enabled
+printk: debug: skip boot console de-registration.
 Reserved memory: created DMA memory pool at 0x0000000058000000, size 128 MiB
 OF: reserved mem: initialized node pma_resv0@58000000, compatible id
shared-dma-pool
 OF: reserved mem: 0x0000000058000000..0x000000005fffffff (131072 KiB)
nomap non-reusable pma_resv0@58000000
+OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
nomap non-reusable mmode_resv0@30000
+OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
nomap non-reusable mmode_resv1@40000
+OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
nomap non-reusable mmode_resv2@44040000
+OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
nomap non-reusable mmode_resv3@44000000
 Zone ranges:
   DMA32    [mem 0x0000000048000000-0x000000007fffffff]
   Normal   empty

Reverting commits 00c9a452a235c61f ("of: reserved_mem: Add code to
dynamically allocate reserved_mem array") and 8a6e02d0c00e7b62 fixes
the issue.

root@smarc-rzfive:/sys/firmware/devicetree/base/reserved-memory# ls -l
total 0
-r--r--r-- 1 root root  4 Oct 29 12:37 #address-cells
-r--r--r-- 1 root root  4 Oct 29 12:37 #size-cells
drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv0@30000
drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv1@40000
drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv2@44040000
drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv3@44000000
-r--r--r-- 1 root root 16 Oct 29 12:37 name
drwxr-xr-x 2 root root  0 Oct 29 12:37 pma_resv0@58000000
-r--r--r-- 1 root root  0 Oct 29 12:37 ranges

> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
> index 4d528c10df3a..d0dbc8183ac4 100644
> --- a/drivers/of/fdt.c
> +++ b/drivers/of/fdt.c
> @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
>                         break;
>                 memblock_reserve(base, size);
>         }
> -
> -       fdt_init_reserved_mem();
>  }
>
>  /**
> @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
>  {
>         void *fdt = initial_boot_params;
>
> +       /* Save the statically-placed regions in the reserved_mem array */
> +       fdt_scan_reserved_mem_reg_nodes();
> +
>         /* Don't use the bootloader provided DTB if ACPI is enabled */
>         if (!acpi_disabled)
>                 fdt = NULL;
> diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
> index 04aa2a91f851..29525c0b9939 100644
> --- a/drivers/of/of_private.h
> +++ b/drivers/of/of_private.h
> @@ -9,6 +9,7 @@
>   */
>
>  #define FDT_ALIGN_SIZE 8
> +#define MAX_RESERVED_REGIONS    64
>
>  /**
>   * struct alias_prop - Alias property in 'aliases' node
> @@ -180,7 +181,7 @@ static inline struct device_node *__of_get_dma_parent(const struct device_node *
>  #endif
>
>  int fdt_scan_reserved_mem(void);
> -void fdt_init_reserved_mem(void);
> +void __init fdt_scan_reserved_mem_reg_nodes(void);
>
>  bool of_fdt_device_is_available(const void *blob, unsigned long node);
>
> diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
> index 46e1c3fbc769..2011174211f9 100644
> --- a/drivers/of/of_reserved_mem.c
> +++ b/drivers/of/of_reserved_mem.c
> @@ -27,7 +27,6 @@
>
>  #include "of_private.h"
>
> -#define MAX_RESERVED_REGIONS   64
>  static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
>  static int reserved_mem_count;
>
> @@ -56,6 +55,7 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
>         return err;
>  }
>
> +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem);
>  /*
>   * fdt_reserved_mem_save_node() - save fdt node for second pass initialization
>   */
> @@ -74,6 +74,9 @@ static void __init fdt_reserved_mem_save_node(unsigned long node, const char *un
>         rmem->base = base;
>         rmem->size = size;
>
> +       /* Call the region specific initialization function */
> +       fdt_init_reserved_mem_node(rmem);
> +
>         reserved_mem_count++;
>         return;
>  }
> @@ -106,7 +109,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
>         phys_addr_t base, size;
>         int len;
>         const __be32 *prop;
> -       int first = 1;
>         bool nomap;
>
>         prop = of_get_flat_dt_prop(node, "reg", &len);
> @@ -134,10 +136,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
>                                uname, &base, (unsigned long)(size / SZ_1M));
>
>                 len -= t_len;
> -               if (first) {
> -                       fdt_reserved_mem_save_node(node, uname, base, size);
> -                       first = 0;
> -               }
>         }
>         return 0;
>  }
> @@ -165,12 +163,77 @@ static int __init __reserved_mem_check_root(unsigned long node)
>         return 0;
>  }
>
> +static void __init __rmem_check_for_overlap(void);
> +
> +/**
> + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg" defined
> + * reserved memory regions.
> + *
> + * This function is used to scan through the DT and store the
> + * information for the reserved memory regions that are defined using
> + * the "reg" property. The region node number, name, base address, and
> + * size are all stored in the reserved_mem array by calling the
> + * fdt_reserved_mem_save_node() function.
> + */
> +void __init fdt_scan_reserved_mem_reg_nodes(void)
> +{
> +       int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
> +       const void *fdt = initial_boot_params;
> +       phys_addr_t base, size;
> +       const __be32 *prop;
> +       int node, child;
> +       int len;
> +
> +       if (!fdt)
> +               return;
> +
> +       node = fdt_path_offset(fdt, "/reserved-memory");
> +       if (node < 0) {
> +               pr_info("Reserved memory: No reserved-memory node in the DT\n");
> +               return;
> +       }
> +
> +       if (__reserved_mem_check_root(node)) {
> +               pr_err("Reserved memory: unsupported node format, ignoring\n");
> +               return;
> +       }
> +
> +       fdt_for_each_subnode(child, fdt, node) {
> +               const char *uname;
> +
> +               prop = of_get_flat_dt_prop(child, "reg", &len);
> +               if (!prop)
> +                       continue;
> +               if (!of_fdt_device_is_available(fdt, child))
> +                       continue;
> +
> +               uname = fdt_get_name(fdt, child, NULL);
> +               if (len && len % t_len != 0) {
> +                       pr_err("Reserved memory: invalid reg property in '%s', skipping node.\n",
> +                              uname);
> +                       continue;
> +               }
> +               base = dt_mem_next_cell(dt_root_addr_cells, &prop);
> +               size = dt_mem_next_cell(dt_root_size_cells, &prop);
> +
> +               if (size)
> +                       fdt_reserved_mem_save_node(child, uname, base, size);
> +       }
> +
> +       /* check for overlapping reserved regions */
> +       __rmem_check_for_overlap();
> +}
> +
> +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname);
> +
>  /*
>   * fdt_scan_reserved_mem() - scan a single FDT node for reserved memory
>   */
>  int __init fdt_scan_reserved_mem(void)
>  {
>         int node, child;
> +       int dynamic_nodes_cnt = 0;
> +       int dynamic_nodes[MAX_RESERVED_REGIONS];
>         const void *fdt = initial_boot_params;
>
>         node = fdt_path_offset(fdt, "/reserved-memory");
> @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
>                 uname = fdt_get_name(fdt, child, NULL);
>
>                 err = __reserved_mem_reserve_reg(child, uname);
> -               if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL))
> -                       fdt_reserved_mem_save_node(child, uname, 0, 0);
> +               /*
> +                * Save the nodes for the dynamically-placed regions
> +                * into an array which will be used for allocation right
> +                * after all the statically-placed regions are reserved
> +                * or marked as no-map. This is done to avoid dynamically
> +                * allocating from one of the statically-placed regions.
> +                */
> +               if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL)) {
> +                       dynamic_nodes[dynamic_nodes_cnt] = child;
> +                       dynamic_nodes_cnt++;
> +               }
> +       }
> +       for (int i = 0; i < dynamic_nodes_cnt; i++) {
> +               const char *uname;
> +
> +               child = dynamic_nodes[i];
> +               uname = fdt_get_name(fdt, child, NULL);
> +               __reserved_mem_alloc_size(child, uname);
>         }
>         return 0;
>  }
> @@ -253,8 +332,7 @@ static int __init __reserved_mem_alloc_in_range(phys_addr_t size,
>   * __reserved_mem_alloc_size() - allocate reserved memory described by
>   *     'size', 'alignment'  and 'alloc-ranges' properties.
>   */
> -static int __init __reserved_mem_alloc_size(unsigned long node,
> -       const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
> +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname)
>  {
>         int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
>         phys_addr_t start = 0, end = 0;
> @@ -334,9 +412,8 @@ static int __init __reserved_mem_alloc_size(unsigned long node,
>                 return -ENOMEM;
>         }
>
> -       *res_base = base;
> -       *res_size = size;
> -
> +       /* Save region in the reserved_mem array */
> +       fdt_reserved_mem_save_node(node, uname, base, size);
>         return 0;
>  }
>
> @@ -425,48 +502,37 @@ static void __init __rmem_check_for_overlap(void)
>  }
>
>  /**
> - * fdt_init_reserved_mem() - allocate and init all saved reserved memory regions
> + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
> + * @rmem: reserved_mem struct of the memory region to be initialized.
> + *
> + * This function is used to call the region specific initialization
> + * function for a reserved memory region.
>   */
> -void __init fdt_init_reserved_mem(void)
> +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem)
>  {
> -       int i;
> -
> -       /* check for overlapping reserved regions */
> -       __rmem_check_for_overlap();
> -
> -       for (i = 0; i < reserved_mem_count; i++) {
> -               struct reserved_mem *rmem = &reserved_mem[i];
> -               unsigned long node = rmem->fdt_node;
> -               int err = 0;
> -               bool nomap;
> +       unsigned long node = rmem->fdt_node;
> +       int err = 0;
> +       bool nomap;
>
> -               nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
> +       nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
>
> -               if (rmem->size == 0)
> -                       err = __reserved_mem_alloc_size(node, rmem->name,
> -                                                &rmem->base, &rmem->size);
> -               if (err == 0) {
> -                       err = __reserved_mem_init_node(rmem);
> -                       if (err != 0 && err != -ENOENT) {
> -                               pr_info("node %s compatible matching fail\n",
> -                                       rmem->name);
> -                               if (nomap)
> -                                       memblock_clear_nomap(rmem->base, rmem->size);
> -                               else
> -                                       memblock_phys_free(rmem->base,
> -                                                          rmem->size);
> -                       } else {
> -                               phys_addr_t end = rmem->base + rmem->size - 1;
> -                               bool reusable =
> -                                       (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> -
> -                               pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> -                                       &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> -                                       nomap ? "nomap" : "map",
> -                                       reusable ? "reusable" : "non-reusable",
> -                                       rmem->name ? rmem->name : "unknown");
> -                       }
> -               }
> +       err = __reserved_mem_init_node(rmem);
> +       if (err != 0 && err != -ENOENT) {
> +               pr_info("node %s compatible matching fail\n", rmem->name);
> +               if (nomap)
> +                       memblock_clear_nomap(rmem->base, rmem->size);
> +               else
> +                       memblock_phys_free(rmem->base, rmem->size);
> +       } else {
> +               phys_addr_t end = rmem->base + rmem->size - 1;
> +               bool reusable =
> +                       (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> +
> +               pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> +                       &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> +                       nomap ? "nomap" : "map",
> +                       reusable ? "reusable" : "non-reusable",
> +                       rmem->name ? rmem->name : "unknown");
>         }
>  }

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Geert Uytterhoeven]

CC linux-renesas-soc

On Tue, Oct 29, 2024 at 1:41 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Wed, Oct 9, 2024 at 12:08 AM Oreoluwa Babatunde
> <quic_obabatun@quicinc.com> wrote:
> > Reserved memory regions defined in the devicetree can be broken up into
> > two groups:
> > i) Statically-placed reserved memory regions
> > i.e. regions defined with a static start address and size using the
> >      "reg" property.
> > ii) Dynamically-placed reserved memory regions.
> > i.e. regions defined by specifying an address range where they can be
> >      placed in memory using the "alloc_ranges" and "size" properties.
> >
> > These regions are processed and set aside at boot time.
> > This is done in two stages as seen below:
> >
> > Stage 1:
> > At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> >
> > 1) If the node represents a statically-placed reserved memory region,
> >    i.e. if it is defined using the "reg" property:
> >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >    - Add the information for that region into the reserved_mem array
> >      using fdt_reserved_mem_save_node().
> >      i.e. fdt_reserved_mem_save_node(node, name, base, size).
> >
> > 2) If the node represents a dynamically-placed reserved memory region,
> >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >    - Add the information for that region to the reserved_mem array with
> >      the starting address and size set to 0.
> >      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
> >    Note: This region is saved to the array with a starting address of 0
> >    because a starting address is not yet allocated for it.
> >
> > Stage 2:
> > After iterating through all the reserved memory nodes and storing their
> > relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> > called and does the following:
> >
> > 1) For statically-placed reserved memory regions:
> >    - Call the region specific init function using
> >      __reserved_mem_init_node().
> > 2) For dynamically-placed reserved memory regions:
> >    - Call __reserved_mem_alloc_size() which is used to allocate memory
> >      for each of these regions, and mark them as nomap if they have the
> >      nomap property specified in the DT.
> >    - Call the region specific init function.
> >
> > The current size of the resvered_mem array is 64 as is defined by
> > MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> > how many reserved memory regions can be specified on a system.
> > As systems continue to grow more and more complex, the number of
> > reserved memory regions needed are also growing and are starting to hit
> > this 64 count limit, hence the need to make the reserved_mem array
> > dynamically sized (i.e. dynamically allocating memory for the
> > reserved_mem array using membock_alloc_*).
> >
> > On architectures such as arm64, memory allocated using memblock is
> > writable only after the page tables have been setup. This means that if
> > the reserved_mem array is going to be dynamically allocated, it needs to
> > happen after the page tables have been setup, not before.
> >
> > Since the reserved memory regions are currently being processed and
> > added to the array before the page tables are setup, there is a need to
> > change the order in which some of the processing is done to allow for
> > the reserved_mem array to be dynamically sized.
> >
> > It is possible to process the statically-placed reserved memory regions
> > without needing to store them in the reserved_mem array until after the
> > page tables have been setup because all the information stored in the
> > array is readily available in the devicetree and can be referenced at
> > any time.
> > Dynamically-placed reserved memory regions on the other hand get
> > assigned a start address only at runtime, and hence need a place to be
> > stored once they are allocated since there is no other referrence to the
> > start address for these regions.
> >
> > Hence this patch changes the processing order of the reserved memory
> > regions in the following ways:
> >
> > Step 1:
> > fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> >
> > 1) If the node represents a statically-placed reserved memory region,
> >    i.e. if it is defined using the "reg" property:
> >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >
> > 2) If the node represents a dynamically-placed reserved memory region,
> >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >    - Call __reserved_mem_alloc_size() which will:
> >      i) Allocate memory for the reserved region and call
> >      memblock_mark_nomap() as needed.
> >      ii) Call the region specific initialization function using
> >      fdt_init_reserved_mem_node().
> >      iii) Save the region information in the reserved_mem array using
> >      fdt_reserved_mem_save_node().
> >
> > Step 2:
> > 1) This stage of the reserved memory processing is now only used to add
> >    the statically-placed reserved memory regions into the reserved_mem
> >    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
> >    region specific initialization functions.
> >
> > 2) This step has also been moved to be after the page tables are
> >    setup. Moving this will allow us to replace the reserved_mem
> >    array with a dynamically sized array before storing the rest of
> >    these regions.
> >
> > Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>
> Thanks for your patch, which is now commit 8a6e02d0c00e7b62
> ("of: reserved_mem: Restructure how the reserved memory regions
> are processed") in dt-rh/for-next.
>
> I have bisected a boot issue on RZ/Five to this commit.
> With "earlycon keep_bootcon" (else there is no output):
>
>     Oops - store (or AMO) access fault [#1]
>     CPU: 0 UID: 0 PID: 1 Comm: swapper Not tainted
> 6.12.0-rc1-00015-g8a6e02d0c00e #201
>     Hardware name: Renesas SMARC EVK based on r9a07g043f01 (DT)
>     epc : __memset+0x60/0x100
>      ra : __dma_alloc_from_coherent+0x150/0x17a
>     epc : ffffffff8062d2bc ra : ffffffff80053a94 sp : ffffffc60000ba20
>      gp : ffffffff812e9938 tp : ffffffd601920000 t0 : ffffffc6000d0000
>      t1 : 0000000000000000 t2 : ffffffffe9600000 s0 : ffffffc60000baa0
>      s1 : ffffffc6000d0000 a0 : ffffffc6000d0000 a1 : 0000000000000000
>      a2 : 0000000000001000 a3 : ffffffc6000d1000 a4 : 0000000000000000
>      a5 : 0000000000000000 a6 : ffffffd601adacc0 a7 : ffffffd601a841a8
>      s2 : ffffffd6018573c0 s3 : 0000000000001000 s4 : ffffffd6019541e0
>      s5 : 0000000200000022 s6 : ffffffd6018f8410 s7 : ffffffd6018573e8
>      s8 : 0000000000000001 s9 : 0000000000000001 s10: 0000000000000010
>      s11: 0000000000000000 t3 : 0000000000000000 t4 : ffffffffdefe62d1
>      t5 : 000000001cd6a3a9 t6 : ffffffd601b2aad6
>     status: 0000000200000120 badaddr: ffffffc6000d0000 cause: 0000000000000007
>     [<ffffffff8062d2bc>] __memset+0x60/0x100
>     [<ffffffff80053e1a>] dma_alloc_from_global_coherent+0x1c/0x28
>     [<ffffffff80053056>] dma_direct_alloc+0x98/0x112
>     [<ffffffff8005238c>] dma_alloc_attrs+0x78/0x86
>     [<ffffffff8035fdb4>] rz_dmac_probe+0x3f6/0x50a
>     [<ffffffff803a0694>] platform_probe+0x4c/0x8a
>     [<ffffffff8039ea16>] really_probe+0xe4/0x1c8
>     [<ffffffff8039ebc4>] __driver_probe_device+0xca/0xce
>     [<ffffffff8039ec48>] driver_probe_device+0x34/0x92
>     [<ffffffff8039ede8>] __driver_attach+0xb4/0xbe
>     [<ffffffff8039ce58>] bus_for_each_dev+0x60/0xa0
>     [<ffffffff8039e26a>] driver_attach+0x1a/0x22
>     [<ffffffff8039dc20>] bus_add_driver+0xa4/0x184
>     [<ffffffff8039f65c>] driver_register+0x8a/0xb4
>     [<ffffffff803a051c>] __platform_driver_register+0x1c/0x24
>     [<ffffffff808202f6>] rz_dmac_driver_init+0x1a/0x22
>     [<ffffffff80800ef6>] do_one_initcall+0x64/0x134
>     [<ffffffff8080122e>] kernel_init_freeable+0x200/0x202
>     [<ffffffff80638126>] kernel_init+0x1e/0x10a
>     [<ffffffff8063d58e>] ret_from_fork+0xe/0x18
>     Code: 1007 82b3 40e2 0797 0000 8793 00e7 8305 97ba 8782 (b023) 00b2
>     ---[ end trace 0000000000000000 ]---
>     Kernel panic - not syncing: Fatal exception in interrupt
>     ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>
> Nothing really stands out in the kernel log, except for a delayed
> initialization of the reserved mem nodes (they are the same
> before/after):
>
>  printk: debug: ignoring loglevel setting.
> -OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
> nomap non-reusable mmode_resv0@30000
> -OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
> nomap non-reusable mmode_resv1@40000
> -OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
> nomap non-reusable mmode_resv3@44000000
> -OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
> nomap non-reusable mmode_resv2@44040000
> +earlycon: scif0 at MMIO 0x000000001004b800 (options '115200n8')
> +printk: legacy bootconsole [scif0] enabled
> +printk: debug: skip boot console de-registration.
>  Reserved memory: created DMA memory pool at 0x0000000058000000, size 128 MiB
>  OF: reserved mem: initialized node pma_resv0@58000000, compatible id
> shared-dma-pool
>  OF: reserved mem: 0x0000000058000000..0x000000005fffffff (131072 KiB)
> nomap non-reusable pma_resv0@58000000
> +OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
> nomap non-reusable mmode_resv0@30000
> +OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
> nomap non-reusable mmode_resv1@40000
> +OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
> nomap non-reusable mmode_resv2@44040000
> +OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
> nomap non-reusable mmode_resv3@44000000
>  Zone ranges:
>    DMA32    [mem 0x0000000048000000-0x000000007fffffff]
>    Normal   empty
>
> Reverting commits 00c9a452a235c61f ("of: reserved_mem: Add code to
> dynamically allocate reserved_mem array") and 8a6e02d0c00e7b62 fixes
> the issue.
>
> root@smarc-rzfive:/sys/firmware/devicetree/base/reserved-memory# ls -l
> total 0
> -r--r--r-- 1 root root  4 Oct 29 12:37 #address-cells
> -r--r--r-- 1 root root  4 Oct 29 12:37 #size-cells
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv0@30000
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv1@40000
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv2@44040000
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv3@44000000
> -r--r--r-- 1 root root 16 Oct 29 12:37 name
> drwxr-xr-x 2 root root  0 Oct 29 12:37 pma_resv0@58000000
> -r--r--r-- 1 root root  0 Oct 29 12:37 ranges
>
> > diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
> > index 4d528c10df3a..d0dbc8183ac4 100644
> > --- a/drivers/of/fdt.c
> > +++ b/drivers/of/fdt.c
> > @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
> >                         break;
> >                 memblock_reserve(base, size);
> >         }
> > -
> > -       fdt_init_reserved_mem();
> >  }
> >
> >  /**
> > @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
> >  {
> >         void *fdt = initial_boot_params;
> >
> > +       /* Save the statically-placed regions in the reserved_mem array */
> > +       fdt_scan_reserved_mem_reg_nodes();
> > +
> >         /* Don't use the bootloader provided DTB if ACPI is enabled */
> >         if (!acpi_disabled)
> >                 fdt = NULL;
> > diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
> > index 04aa2a91f851..29525c0b9939 100644
> > --- a/drivers/of/of_private.h
> > +++ b/drivers/of/of_private.h
> > @@ -9,6 +9,7 @@
> >   */
> >
> >  #define FDT_ALIGN_SIZE 8
> > +#define MAX_RESERVED_REGIONS    64
> >
> >  /**
> >   * struct alias_prop - Alias property in 'aliases' node
> > @@ -180,7 +181,7 @@ static inline struct device_node *__of_get_dma_parent(const struct device_node *
> >  #endif
> >
> >  int fdt_scan_reserved_mem(void);
> > -void fdt_init_reserved_mem(void);
> > +void __init fdt_scan_reserved_mem_reg_nodes(void);
> >
> >  bool of_fdt_device_is_available(const void *blob, unsigned long node);
> >
> > diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
> > index 46e1c3fbc769..2011174211f9 100644
> > --- a/drivers/of/of_reserved_mem.c
> > +++ b/drivers/of/of_reserved_mem.c
> > @@ -27,7 +27,6 @@
> >
> >  #include "of_private.h"
> >
> > -#define MAX_RESERVED_REGIONS   64
> >  static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
> >  static int reserved_mem_count;
> >
> > @@ -56,6 +55,7 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
> >         return err;
> >  }
> >
> > +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem);
> >  /*
> >   * fdt_reserved_mem_save_node() - save fdt node for second pass initialization
> >   */
> > @@ -74,6 +74,9 @@ static void __init fdt_reserved_mem_save_node(unsigned long node, const char *un
> >         rmem->base = base;
> >         rmem->size = size;
> >
> > +       /* Call the region specific initialization function */
> > +       fdt_init_reserved_mem_node(rmem);
> > +
> >         reserved_mem_count++;
> >         return;
> >  }
> > @@ -106,7 +109,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
> >         phys_addr_t base, size;
> >         int len;
> >         const __be32 *prop;
> > -       int first = 1;
> >         bool nomap;
> >
> >         prop = of_get_flat_dt_prop(node, "reg", &len);
> > @@ -134,10 +136,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
> >                                uname, &base, (unsigned long)(size / SZ_1M));
> >
> >                 len -= t_len;
> > -               if (first) {
> > -                       fdt_reserved_mem_save_node(node, uname, base, size);
> > -                       first = 0;
> > -               }
> >         }
> >         return 0;
> >  }
> > @@ -165,12 +163,77 @@ static int __init __reserved_mem_check_root(unsigned long node)
> >         return 0;
> >  }
> >
> > +static void __init __rmem_check_for_overlap(void);
> > +
> > +/**
> > + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg" defined
> > + * reserved memory regions.
> > + *
> > + * This function is used to scan through the DT and store the
> > + * information for the reserved memory regions that are defined using
> > + * the "reg" property. The region node number, name, base address, and
> > + * size are all stored in the reserved_mem array by calling the
> > + * fdt_reserved_mem_save_node() function.
> > + */
> > +void __init fdt_scan_reserved_mem_reg_nodes(void)
> > +{
> > +       int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
> > +       const void *fdt = initial_boot_params;
> > +       phys_addr_t base, size;
> > +       const __be32 *prop;
> > +       int node, child;
> > +       int len;
> > +
> > +       if (!fdt)
> > +               return;
> > +
> > +       node = fdt_path_offset(fdt, "/reserved-memory");
> > +       if (node < 0) {
> > +               pr_info("Reserved memory: No reserved-memory node in the DT\n");
> > +               return;
> > +       }
> > +
> > +       if (__reserved_mem_check_root(node)) {
> > +               pr_err("Reserved memory: unsupported node format, ignoring\n");
> > +               return;
> > +       }
> > +
> > +       fdt_for_each_subnode(child, fdt, node) {
> > +               const char *uname;
> > +
> > +               prop = of_get_flat_dt_prop(child, "reg", &len);
> > +               if (!prop)
> > +                       continue;
> > +               if (!of_fdt_device_is_available(fdt, child))
> > +                       continue;
> > +
> > +               uname = fdt_get_name(fdt, child, NULL);
> > +               if (len && len % t_len != 0) {
> > +                       pr_err("Reserved memory: invalid reg property in '%s', skipping node.\n",
> > +                              uname);
> > +                       continue;
> > +               }
> > +               base = dt_mem_next_cell(dt_root_addr_cells, &prop);
> > +               size = dt_mem_next_cell(dt_root_size_cells, &prop);
> > +
> > +               if (size)
> > +                       fdt_reserved_mem_save_node(child, uname, base, size);
> > +       }
> > +
> > +       /* check for overlapping reserved regions */
> > +       __rmem_check_for_overlap();
> > +}
> > +
> > +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname);
> > +
> >  /*
> >   * fdt_scan_reserved_mem() - scan a single FDT node for reserved memory
> >   */
> >  int __init fdt_scan_reserved_mem(void)
> >  {
> >         int node, child;
> > +       int dynamic_nodes_cnt = 0;
> > +       int dynamic_nodes[MAX_RESERVED_REGIONS];
> >         const void *fdt = initial_boot_params;
> >
> >         node = fdt_path_offset(fdt, "/reserved-memory");
> > @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
> >                 uname = fdt_get_name(fdt, child, NULL);
> >
> >                 err = __reserved_mem_reserve_reg(child, uname);
> > -               if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL))
> > -                       fdt_reserved_mem_save_node(child, uname, 0, 0);
> > +               /*
> > +                * Save the nodes for the dynamically-placed regions
> > +                * into an array which will be used for allocation right
> > +                * after all the statically-placed regions are reserved
> > +                * or marked as no-map. This is done to avoid dynamically
> > +                * allocating from one of the statically-placed regions.
> > +                */
> > +               if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL)) {
> > +                       dynamic_nodes[dynamic_nodes_cnt] = child;
> > +                       dynamic_nodes_cnt++;
> > +               }
> > +       }
> > +       for (int i = 0; i < dynamic_nodes_cnt; i++) {
> > +               const char *uname;
> > +
> > +               child = dynamic_nodes[i];
> > +               uname = fdt_get_name(fdt, child, NULL);
> > +               __reserved_mem_alloc_size(child, uname);
> >         }
> >         return 0;
> >  }
> > @@ -253,8 +332,7 @@ static int __init __reserved_mem_alloc_in_range(phys_addr_t size,
> >   * __reserved_mem_alloc_size() - allocate reserved memory described by
> >   *     'size', 'alignment'  and 'alloc-ranges' properties.
> >   */
> > -static int __init __reserved_mem_alloc_size(unsigned long node,
> > -       const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
> > +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname)
> >  {
> >         int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
> >         phys_addr_t start = 0, end = 0;
> > @@ -334,9 +412,8 @@ static int __init __reserved_mem_alloc_size(unsigned long node,
> >                 return -ENOMEM;
> >         }
> >
> > -       *res_base = base;
> > -       *res_size = size;
> > -
> > +       /* Save region in the reserved_mem array */
> > +       fdt_reserved_mem_save_node(node, uname, base, size);
> >         return 0;
> >  }
> >
> > @@ -425,48 +502,37 @@ static void __init __rmem_check_for_overlap(void)
> >  }
> >
> >  /**
> > - * fdt_init_reserved_mem() - allocate and init all saved reserved memory regions
> > + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
> > + * @rmem: reserved_mem struct of the memory region to be initialized.
> > + *
> > + * This function is used to call the region specific initialization
> > + * function for a reserved memory region.
> >   */
> > -void __init fdt_init_reserved_mem(void)
> > +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem)
> >  {
> > -       int i;
> > -
> > -       /* check for overlapping reserved regions */
> > -       __rmem_check_for_overlap();
> > -
> > -       for (i = 0; i < reserved_mem_count; i++) {
> > -               struct reserved_mem *rmem = &reserved_mem[i];
> > -               unsigned long node = rmem->fdt_node;
> > -               int err = 0;
> > -               bool nomap;
> > +       unsigned long node = rmem->fdt_node;
> > +       int err = 0;
> > +       bool nomap;
> >
> > -               nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
> > +       nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
> >
> > -               if (rmem->size == 0)
> > -                       err = __reserved_mem_alloc_size(node, rmem->name,
> > -                                                &rmem->base, &rmem->size);
> > -               if (err == 0) {
> > -                       err = __reserved_mem_init_node(rmem);
> > -                       if (err != 0 && err != -ENOENT) {
> > -                               pr_info("node %s compatible matching fail\n",
> > -                                       rmem->name);
> > -                               if (nomap)
> > -                                       memblock_clear_nomap(rmem->base, rmem->size);
> > -                               else
> > -                                       memblock_phys_free(rmem->base,
> > -                                                          rmem->size);
> > -                       } else {
> > -                               phys_addr_t end = rmem->base + rmem->size - 1;
> > -                               bool reusable =
> > -                                       (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> > -
> > -                               pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> > -                                       &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> > -                                       nomap ? "nomap" : "map",
> > -                                       reusable ? "reusable" : "non-reusable",
> > -                                       rmem->name ? rmem->name : "unknown");
> > -                       }
> > -               }
> > +       err = __reserved_mem_init_node(rmem);
> > +       if (err != 0 && err != -ENOENT) {
> > +               pr_info("node %s compatible matching fail\n", rmem->name);
> > +               if (nomap)
> > +                       memblock_clear_nomap(rmem->base, rmem->size);
> > +               else
> > +                       memblock_phys_free(rmem->base, rmem->size);
> > +       } else {
> > +               phys_addr_t end = rmem->base + rmem->size - 1;
> > +               bool reusable =
> > +                       (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> > +
> > +               pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> > +                       &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> > +                       nomap ? "nomap" : "map",
> > +                       reusable ? "reusable" : "non-reusable",
> > +                       rmem->name ? rmem->name : "unknown");
> >         }
> >  }

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Oreoluwa Babatunde]

On 10/29/2024 5:41 AM, Geert Uytterhoeven wrote:
> Hi Oreoluwa,
>
> On Wed, Oct 9, 2024 at 12:08 AM Oreoluwa Babatunde
> <quic_obabatun@quicinc.com> wrote:
>> Reserved memory regions defined in the devicetree can be broken up into
>> two groups:
>> i) Statically-placed reserved memory regions
>> i.e. regions defined with a static start address and size using the
>>      "reg" property.
>> ii) Dynamically-placed reserved memory regions.
>> i.e. regions defined by specifying an address range where they can be
>>      placed in memory using the "alloc_ranges" and "size" properties.
>>
>> These regions are processed and set aside at boot time.
>> This is done in two stages as seen below:
>>
>> Stage 1:
>> At this stage, fdt_scan_reserved_mem() scans through the child nodes of
>> the reserved_memory node using the flattened devicetree and does the
>> following:
>>
>> 1) If the node represents a statically-placed reserved memory region,
>>    i.e. if it is defined using the "reg" property:
>>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>    - Add the information for that region into the reserved_mem array
>>      using fdt_reserved_mem_save_node().
>>      i.e. fdt_reserved_mem_save_node(node, name, base, size).
>>
>> 2) If the node represents a dynamically-placed reserved memory region,
>>    i.e. if it is defined using "alloc-ranges" and "size" properties:
>>    - Add the information for that region to the reserved_mem array with
>>      the starting address and size set to 0.
>>      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>>    Note: This region is saved to the array with a starting address of 0
>>    because a starting address is not yet allocated for it.
>>
>> Stage 2:
>> After iterating through all the reserved memory nodes and storing their
>> relevant information in the reserved_mem array,fdt_init_reserved_mem() is
>> called and does the following:
>>
>> 1) For statically-placed reserved memory regions:
>>    - Call the region specific init function using
>>      __reserved_mem_init_node().
>> 2) For dynamically-placed reserved memory regions:
>>    - Call __reserved_mem_alloc_size() which is used to allocate memory
>>      for each of these regions, and mark them as nomap if they have the
>>      nomap property specified in the DT.
>>    - Call the region specific init function.
>>
>> The current size of the resvered_mem array is 64 as is defined by
>> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
>> how many reserved memory regions can be specified on a system.
>> As systems continue to grow more and more complex, the number of
>> reserved memory regions needed are also growing and are starting to hit
>> this 64 count limit, hence the need to make the reserved_mem array
>> dynamically sized (i.e. dynamically allocating memory for the
>> reserved_mem array using membock_alloc_*).
>>
>> On architectures such as arm64, memory allocated using memblock is
>> writable only after the page tables have been setup. This means that if
>> the reserved_mem array is going to be dynamically allocated, it needs to
>> happen after the page tables have been setup, not before.
>>
>> Since the reserved memory regions are currently being processed and
>> added to the array before the page tables are setup, there is a need to
>> change the order in which some of the processing is done to allow for
>> the reserved_mem array to be dynamically sized.
>>
>> It is possible to process the statically-placed reserved memory regions
>> without needing to store them in the reserved_mem array until after the
>> page tables have been setup because all the information stored in the
>> array is readily available in the devicetree and can be referenced at
>> any time.
>> Dynamically-placed reserved memory regions on the other hand get
>> assigned a start address only at runtime, and hence need a place to be
>> stored once they are allocated since there is no other referrence to the
>> start address for these regions.
>>
>> Hence this patch changes the processing order of the reserved memory
>> regions in the following ways:
>>
>> Step 1:
>> fdt_scan_reserved_mem() scans through the child nodes of
>> the reserved_memory node using the flattened devicetree and does the
>> following:
>>
>> 1) If the node represents a statically-placed reserved memory region,
>>    i.e. if it is defined using the "reg" property:
>>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>
>> 2) If the node represents a dynamically-placed reserved memory region,
>>    i.e. if it is defined using "alloc-ranges" and "size" properties:
>>    - Call __reserved_mem_alloc_size() which will:
>>      i) Allocate memory for the reserved region and call
>>      memblock_mark_nomap() as needed.
>>      ii) Call the region specific initialization function using
>>      fdt_init_reserved_mem_node().
>>      iii) Save the region information in the reserved_mem array using
>>      fdt_reserved_mem_save_node().
>>
>> Step 2:
>> 1) This stage of the reserved memory processing is now only used to add
>>    the statically-placed reserved memory regions into the reserved_mem
>>    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
>>    region specific initialization functions.
>>
>> 2) This step has also been moved to be after the page tables are
>>    setup. Moving this will allow us to replace the reserved_mem
>>    array with a dynamically sized array before storing the rest of
>>    these regions.
>>
>> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> Thanks for your patch, which is now commit 8a6e02d0c00e7b62
> ("of: reserved_mem: Restructure how the reserved memory regions
> are processed") in dt-rh/for-next.
>
> I have bisected a boot issue on RZ/Five to this commit.
> With "earlycon keep_bootcon" (else there is no output):
>
>     Oops - store (or AMO) access fault [#1]
>     CPU: 0 UID: 0 PID: 1 Comm: swapper Not tainted
> 6.12.0-rc1-00015-g8a6e02d0c00e #201
>     Hardware name: Renesas SMARC EVK based on r9a07g043f01 (DT)
>     epc : __memset+0x60/0x100
>      ra : __dma_alloc_from_coherent+0x150/0x17a
>     epc : ffffffff8062d2bc ra : ffffffff80053a94 sp : ffffffc60000ba20
>      gp : ffffffff812e9938 tp : ffffffd601920000 t0 : ffffffc6000d0000
>      t1 : 0000000000000000 t2 : ffffffffe9600000 s0 : ffffffc60000baa0
>      s1 : ffffffc6000d0000 a0 : ffffffc6000d0000 a1 : 0000000000000000
>      a2 : 0000000000001000 a3 : ffffffc6000d1000 a4 : 0000000000000000
>      a5 : 0000000000000000 a6 : ffffffd601adacc0 a7 : ffffffd601a841a8
>      s2 : ffffffd6018573c0 s3 : 0000000000001000 s4 : ffffffd6019541e0
>      s5 : 0000000200000022 s6 : ffffffd6018f8410 s7 : ffffffd6018573e8
>      s8 : 0000000000000001 s9 : 0000000000000001 s10: 0000000000000010
>      s11: 0000000000000000 t3 : 0000000000000000 t4 : ffffffffdefe62d1
>      t5 : 000000001cd6a3a9 t6 : ffffffd601b2aad6
>     status: 0000000200000120 badaddr: ffffffc6000d0000 cause: 0000000000000007
>     [<ffffffff8062d2bc>] __memset+0x60/0x100
>     [<ffffffff80053e1a>] dma_alloc_from_global_coherent+0x1c/0x28
>     [<ffffffff80053056>] dma_direct_alloc+0x98/0x112
>     [<ffffffff8005238c>] dma_alloc_attrs+0x78/0x86
>     [<ffffffff8035fdb4>] rz_dmac_probe+0x3f6/0x50a
>     [<ffffffff803a0694>] platform_probe+0x4c/0x8a
>     [<ffffffff8039ea16>] really_probe+0xe4/0x1c8
>     [<ffffffff8039ebc4>] __driver_probe_device+0xca/0xce
>     [<ffffffff8039ec48>] driver_probe_device+0x34/0x92
>     [<ffffffff8039ede8>] __driver_attach+0xb4/0xbe
>     [<ffffffff8039ce58>] bus_for_each_dev+0x60/0xa0
>     [<ffffffff8039e26a>] driver_attach+0x1a/0x22
>     [<ffffffff8039dc20>] bus_add_driver+0xa4/0x184
>     [<ffffffff8039f65c>] driver_register+0x8a/0xb4
>     [<ffffffff803a051c>] __platform_driver_register+0x1c/0x24
>     [<ffffffff808202f6>] rz_dmac_driver_init+0x1a/0x22
>     [<ffffffff80800ef6>] do_one_initcall+0x64/0x134
>     [<ffffffff8080122e>] kernel_init_freeable+0x200/0x202
>     [<ffffffff80638126>] kernel_init+0x1e/0x10a
>     [<ffffffff8063d58e>] ret_from_fork+0xe/0x18
>     Code: 1007 82b3 40e2 0797 0000 8793 00e7 8305 97ba 8782 (b023) 00b2
>     ---[ end trace 0000000000000000 ]---
>     Kernel panic - not syncing: Fatal exception in interrupt
>     ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
>
> Nothing really stands out in the kernel log, except for a delayed
> initialization of the reserved mem nodes (they are the same
> before/after):
>
>  printk: debug: ignoring loglevel setting.
> -OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
> nomap non-reusable mmode_resv0@30000
> -OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
> nomap non-reusable mmode_resv1@40000
> -OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
> nomap non-reusable mmode_resv3@44000000
> -OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
> nomap non-reusable mmode_resv2@44040000
> +earlycon: scif0 at MMIO 0x000000001004b800 (options '115200n8')
> +printk: legacy bootconsole [scif0] enabled
> +printk: debug: skip boot console de-registration.
>  Reserved memory: created DMA memory pool at 0x0000000058000000, size 128 MiB
>  OF: reserved mem: initialized node pma_resv0@58000000, compatible id
> shared-dma-pool
>  OF: reserved mem: 0x0000000058000000..0x000000005fffffff (131072 KiB)
> nomap non-reusable pma_resv0@58000000
> +OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
> nomap non-reusable mmode_resv0@30000
> +OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
> nomap non-reusable mmode_resv1@40000
> +OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
> nomap non-reusable mmode_resv2@44040000
> +OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
> nomap non-reusable mmode_resv3@44000000
>  Zone ranges:
>    DMA32    [mem 0x0000000048000000-0x000000007fffffff]
>    Normal   empty
>
> Reverting commits 00c9a452a235c61f ("of: reserved_mem: Add code to
> dynamically allocate reserved_mem array") and 8a6e02d0c00e7b62 fixes
> the issue.
>
> root@smarc-rzfive:/sys/firmware/devicetree/base/reserved-memory# ls -l
> total 0
> -r--r--r-- 1 root root  4 Oct 29 12:37 #address-cells
> -r--r--r-- 1 root root  4 Oct 29 12:37 #size-cells
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv0@30000
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv1@40000
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv2@44040000
> drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv3@44000000
> -r--r--r-- 1 root root 16 Oct 29 12:37 name
> drwxr-xr-x 2 root root  0 Oct 29 12:37 pma_resv0@58000000
> -r--r--r-- 1 root root  0 Oct 29 12:37 ranges
Hi Geert,

Thanks for reaching out and sorry you're seeing this issue.

Please can you provide reproduction steps? I tried booting up
risc-v arch with qemu but did not run into the issue you are
seeing.

Regards,
Oreoluwa

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Geert Uytterhoeven]

Hi Oreoluwa,

On Thu, Oct 31, 2024 at 10:06 PM Oreoluwa Babatunde
<quic_obabatun@quicinc.com> wrote:
> On 10/29/2024 5:41 AM, Geert Uytterhoeven wrote:
> > On Wed, Oct 9, 2024 at 12:08 AM Oreoluwa Babatunde
> > <quic_obabatun@quicinc.com> wrote:
> >> Reserved memory regions defined in the devicetree can be broken up into
> >> two groups:
> >> i) Statically-placed reserved memory regions
> >> i.e. regions defined with a static start address and size using the
> >>      "reg" property.
> >> ii) Dynamically-placed reserved memory regions.
> >> i.e. regions defined by specifying an address range where they can be
> >>      placed in memory using the "alloc_ranges" and "size" properties.
> >>
> >> These regions are processed and set aside at boot time.
> >> This is done in two stages as seen below:
> >>
> >> Stage 1:
> >> At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> >> the reserved_memory node using the flattened devicetree and does the
> >> following:
> >>
> >> 1) If the node represents a statically-placed reserved memory region,
> >>    i.e. if it is defined using the "reg" property:
> >>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >>    - Add the information for that region into the reserved_mem array
> >>      using fdt_reserved_mem_save_node().
> >>      i.e. fdt_reserved_mem_save_node(node, name, base, size).
> >>
> >> 2) If the node represents a dynamically-placed reserved memory region,
> >>    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >>    - Add the information for that region to the reserved_mem array with
> >>      the starting address and size set to 0.
> >>      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
> >>    Note: This region is saved to the array with a starting address of 0
> >>    because a starting address is not yet allocated for it.
> >>
> >> Stage 2:
> >> After iterating through all the reserved memory nodes and storing their
> >> relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> >> called and does the following:
> >>
> >> 1) For statically-placed reserved memory regions:
> >>    - Call the region specific init function using
> >>      __reserved_mem_init_node().
> >> 2) For dynamically-placed reserved memory regions:
> >>    - Call __reserved_mem_alloc_size() which is used to allocate memory
> >>      for each of these regions, and mark them as nomap if they have the
> >>      nomap property specified in the DT.
> >>    - Call the region specific init function.
> >>
> >> The current size of the resvered_mem array is 64 as is defined by
> >> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> >> how many reserved memory regions can be specified on a system.
> >> As systems continue to grow more and more complex, the number of
> >> reserved memory regions needed are also growing and are starting to hit
> >> this 64 count limit, hence the need to make the reserved_mem array
> >> dynamically sized (i.e. dynamically allocating memory for the
> >> reserved_mem array using membock_alloc_*).
> >>
> >> On architectures such as arm64, memory allocated using memblock is
> >> writable only after the page tables have been setup. This means that if
> >> the reserved_mem array is going to be dynamically allocated, it needs to
> >> happen after the page tables have been setup, not before.
> >>
> >> Since the reserved memory regions are currently being processed and
> >> added to the array before the page tables are setup, there is a need to
> >> change the order in which some of the processing is done to allow for
> >> the reserved_mem array to be dynamically sized.
> >>
> >> It is possible to process the statically-placed reserved memory regions
> >> without needing to store them in the reserved_mem array until after the
> >> page tables have been setup because all the information stored in the
> >> array is readily available in the devicetree and can be referenced at
> >> any time.
> >> Dynamically-placed reserved memory regions on the other hand get
> >> assigned a start address only at runtime, and hence need a place to be
> >> stored once they are allocated since there is no other referrence to the
> >> start address for these regions.
> >>
> >> Hence this patch changes the processing order of the reserved memory
> >> regions in the following ways:
> >>
> >> Step 1:
> >> fdt_scan_reserved_mem() scans through the child nodes of
> >> the reserved_memory node using the flattened devicetree and does the
> >> following:
> >>
> >> 1) If the node represents a statically-placed reserved memory region,
> >>    i.e. if it is defined using the "reg" property:
> >>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >>
> >> 2) If the node represents a dynamically-placed reserved memory region,
> >>    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >>    - Call __reserved_mem_alloc_size() which will:
> >>      i) Allocate memory for the reserved region and call
> >>      memblock_mark_nomap() as needed.
> >>      ii) Call the region specific initialization function using
> >>      fdt_init_reserved_mem_node().
> >>      iii) Save the region information in the reserved_mem array using
> >>      fdt_reserved_mem_save_node().
> >>
> >> Step 2:
> >> 1) This stage of the reserved memory processing is now only used to add
> >>    the statically-placed reserved memory regions into the reserved_mem
> >>    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
> >>    region specific initialization functions.
> >>
> >> 2) This step has also been moved to be after the page tables are
> >>    setup. Moving this will allow us to replace the reserved_mem
> >>    array with a dynamically sized array before storing the rest of
> >>    these regions.
> >>
> >> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> > Thanks for your patch, which is now commit 8a6e02d0c00e7b62
> > ("of: reserved_mem: Restructure how the reserved memory regions
> > are processed") in dt-rh/for-next.
> >
> > I have bisected a boot issue on RZ/Five to this commit.
> > With "earlycon keep_bootcon" (else there is no output):
> >
> >     Oops - store (or AMO) access fault [#1]
> >     CPU: 0 UID: 0 PID: 1 Comm: swapper Not tainted
> > 6.12.0-rc1-00015-g8a6e02d0c00e #201
> >     Hardware name: Renesas SMARC EVK based on r9a07g043f01 (DT)
> >     epc : __memset+0x60/0x100
> >      ra : __dma_alloc_from_coherent+0x150/0x17a
> >     epc : ffffffff8062d2bc ra : ffffffff80053a94 sp : ffffffc60000ba20
> >      gp : ffffffff812e9938 tp : ffffffd601920000 t0 : ffffffc6000d0000
> >      t1 : 0000000000000000 t2 : ffffffffe9600000 s0 : ffffffc60000baa0
> >      s1 : ffffffc6000d0000 a0 : ffffffc6000d0000 a1 : 0000000000000000
> >      a2 : 0000000000001000 a3 : ffffffc6000d1000 a4 : 0000000000000000
> >      a5 : 0000000000000000 a6 : ffffffd601adacc0 a7 : ffffffd601a841a8
> >      s2 : ffffffd6018573c0 s3 : 0000000000001000 s4 : ffffffd6019541e0
> >      s5 : 0000000200000022 s6 : ffffffd6018f8410 s7 : ffffffd6018573e8
> >      s8 : 0000000000000001 s9 : 0000000000000001 s10: 0000000000000010
> >      s11: 0000000000000000 t3 : 0000000000000000 t4 : ffffffffdefe62d1
> >      t5 : 000000001cd6a3a9 t6 : ffffffd601b2aad6
> >     status: 0000000200000120 badaddr: ffffffc6000d0000 cause: 0000000000000007
> >     [<ffffffff8062d2bc>] __memset+0x60/0x100
> >     [<ffffffff80053e1a>] dma_alloc_from_global_coherent+0x1c/0x28
> >     [<ffffffff80053056>] dma_direct_alloc+0x98/0x112
> >     [<ffffffff8005238c>] dma_alloc_attrs+0x78/0x86
> >     [<ffffffff8035fdb4>] rz_dmac_probe+0x3f6/0x50a
> >     [<ffffffff803a0694>] platform_probe+0x4c/0x8a
> >     [<ffffffff8039ea16>] really_probe+0xe4/0x1c8
> >     [<ffffffff8039ebc4>] __driver_probe_device+0xca/0xce
> >     [<ffffffff8039ec48>] driver_probe_device+0x34/0x92
> >     [<ffffffff8039ede8>] __driver_attach+0xb4/0xbe
> >     [<ffffffff8039ce58>] bus_for_each_dev+0x60/0xa0
> >     [<ffffffff8039e26a>] driver_attach+0x1a/0x22
> >     [<ffffffff8039dc20>] bus_add_driver+0xa4/0x184
> >     [<ffffffff8039f65c>] driver_register+0x8a/0xb4
> >     [<ffffffff803a051c>] __platform_driver_register+0x1c/0x24
> >     [<ffffffff808202f6>] rz_dmac_driver_init+0x1a/0x22
> >     [<ffffffff80800ef6>] do_one_initcall+0x64/0x134
> >     [<ffffffff8080122e>] kernel_init_freeable+0x200/0x202
> >     [<ffffffff80638126>] kernel_init+0x1e/0x10a
> >     [<ffffffff8063d58e>] ret_from_fork+0xe/0x18
> >     Code: 1007 82b3 40e2 0797 0000 8793 00e7 8305 97ba 8782 (b023) 00b2
> >     ---[ end trace 0000000000000000 ]---
> >     Kernel panic - not syncing: Fatal exception in interrupt
> >     ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
> >
> > Nothing really stands out in the kernel log, except for a delayed
> > initialization of the reserved mem nodes (they are the same
> > before/after):
> >
> >  printk: debug: ignoring loglevel setting.
> > -OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
> > nomap non-reusable mmode_resv0@30000
> > -OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
> > nomap non-reusable mmode_resv1@40000
> > -OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
> > nomap non-reusable mmode_resv3@44000000
> > -OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
> > nomap non-reusable mmode_resv2@44040000
> > +earlycon: scif0 at MMIO 0x000000001004b800 (options '115200n8')
> > +printk: legacy bootconsole [scif0] enabled
> > +printk: debug: skip boot console de-registration.
> >  Reserved memory: created DMA memory pool at 0x0000000058000000, size 128 MiB
> >  OF: reserved mem: initialized node pma_resv0@58000000, compatible id
> > shared-dma-pool
> >  OF: reserved mem: 0x0000000058000000..0x000000005fffffff (131072 KiB)
> > nomap non-reusable pma_resv0@58000000
> > +OF: reserved mem: 0x0000000000030000..0x000000000003ffff (64 KiB)
> > nomap non-reusable mmode_resv0@30000
> > +OF: reserved mem: 0x0000000000040000..0x000000000004ffff (64 KiB)
> > nomap non-reusable mmode_resv1@40000
> > +OF: reserved mem: 0x0000000044040000..0x000000004405ffff (128 KiB)
> > nomap non-reusable mmode_resv2@44040000
> > +OF: reserved mem: 0x0000000044000000..0x000000004403ffff (256 KiB)
> > nomap non-reusable mmode_resv3@44000000
> >  Zone ranges:
> >    DMA32    [mem 0x0000000048000000-0x000000007fffffff]
> >    Normal   empty
> >
> > Reverting commits 00c9a452a235c61f ("of: reserved_mem: Add code to
> > dynamically allocate reserved_mem array") and 8a6e02d0c00e7b62 fixes
> > the issue.
> >
> > root@smarc-rzfive:/sys/firmware/devicetree/base/reserved-memory# ls -l
> > total 0
> > -r--r--r-- 1 root root  4 Oct 29 12:37 #address-cells
> > -r--r--r-- 1 root root  4 Oct 29 12:37 #size-cells
> > drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv0@30000
> > drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv1@40000
> > drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv2@44040000
> > drwxr-xr-x 2 root root  0 Oct 29 12:37 mmode_resv3@44000000
> > -r--r--r-- 1 root root 16 Oct 29 12:37 name
> > drwxr-xr-x 2 root root  0 Oct 29 12:37 pma_resv0@58000000
> > -r--r--r-- 1 root root  0 Oct 29 12:37 ranges

> Thanks for reaching out and sorry you're seeing this issue.
>
> Please can you provide reproduction steps? I tried booting up
> risc-v arch with qemu but did not run into the issue you are
> seeing.

It indeed doesn't happen on other RISC-V platforms (tried MPFS/Icicle).
RZ/Five is special in that it is DMA non-coherent, and the crash
happens in dma_alloc_from_global_coherent().
According to the logs, the DMA pool at 0x58000000 is still being created.
I'll try to debug this further next week...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Geert Uytterhoeven]

Hi Oreoluwa,

On Fri, Nov 1, 2024 at 1:45 PM Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> On Thu, Oct 31, 2024 at 10:06 PM Oreoluwa Babatunde
> <quic_obabatun@quicinc.com> wrote:
> > On 10/29/2024 5:41 AM, Geert Uytterhoeven wrote:
> > > On Wed, Oct 9, 2024 at 12:08 AM Oreoluwa Babatunde
> > > <quic_obabatun@quicinc.com> wrote:
> > >> Reserved memory regions defined in the devicetree can be broken up into
> > >> two groups:
> > >> i) Statically-placed reserved memory regions
> > >> i.e. regions defined with a static start address and size using the
> > >>      "reg" property.
> > >> ii) Dynamically-placed reserved memory regions.
> > >> i.e. regions defined by specifying an address range where they can be
> > >>      placed in memory using the "alloc_ranges" and "size" properties.

[...]

> > >> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> > > Thanks for your patch, which is now commit 8a6e02d0c00e7b62
> > > ("of: reserved_mem: Restructure how the reserved memory regions
> > > are processed") in dt-rh/for-next.
> > >
> > > I have bisected a boot issue on RZ/Five to this commit.
> > > With "earlycon keep_bootcon" (else there is no output):
> > >
> > >     Oops - store (or AMO) access fault [#1]
> > >     CPU: 0 UID: 0 PID: 1 Comm: swapper Not tainted
> > > 6.12.0-rc1-00015-g8a6e02d0c00e #201
> > >     Hardware name: Renesas SMARC EVK based on r9a07g043f01 (DT)
> > >     epc : __memset+0x60/0x100
> > >      ra : __dma_alloc_from_coherent+0x150/0x17a
> > >     epc : ffffffff8062d2bc ra : ffffffff80053a94 sp : ffffffc60000ba20
> > >      gp : ffffffff812e9938 tp : ffffffd601920000 t0 : ffffffc6000d0000
> > >      t1 : 0000000000000000 t2 : ffffffffe9600000 s0 : ffffffc60000baa0
> > >      s1 : ffffffc6000d0000 a0 : ffffffc6000d0000 a1 : 0000000000000000
> > >      a2 : 0000000000001000 a3 : ffffffc6000d1000 a4 : 0000000000000000
> > >      a5 : 0000000000000000 a6 : ffffffd601adacc0 a7 : ffffffd601a841a8
> > >      s2 : ffffffd6018573c0 s3 : 0000000000001000 s4 : ffffffd6019541e0
> > >      s5 : 0000000200000022 s6 : ffffffd6018f8410 s7 : ffffffd6018573e8
> > >      s8 : 0000000000000001 s9 : 0000000000000001 s10: 0000000000000010
> > >      s11: 0000000000000000 t3 : 0000000000000000 t4 : ffffffffdefe62d1
> > >      t5 : 000000001cd6a3a9 t6 : ffffffd601b2aad6
> > >     status: 0000000200000120 badaddr: ffffffc6000d0000 cause: 0000000000000007
> > >     [<ffffffff8062d2bc>] __memset+0x60/0x100
> > >     [<ffffffff80053e1a>] dma_alloc_from_global_coherent+0x1c/0x28
> > >     [<ffffffff80053056>] dma_direct_alloc+0x98/0x112
> > >     [<ffffffff8005238c>] dma_alloc_attrs+0x78/0x86
> > >     [<ffffffff8035fdb4>] rz_dmac_probe+0x3f6/0x50a

> > Please can you provide reproduction steps? I tried booting up
> > risc-v arch with qemu but did not run into the issue you are
> > seeing.
>
> It indeed doesn't happen on other RISC-V platforms (tried MPFS/Icicle).
> RZ/Five is special in that it is DMA non-coherent, and the crash
> happens in dma_alloc_from_global_coherent().
> According to the logs, the DMA pool at 0x58000000 is still being created.
> I'll try to debug this further next week...

I found the issue, and have sent a fix
https://lore.kernel.org/r/f8cef6845a6141f0277e31a71fe153612daae776.1731436631.git.geert+renesas@glider.be

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Marco Felsch]

Hi,

On 24-10-08, Oreoluwa Babatunde wrote:
> Reserved memory regions defined in the devicetree can be broken up into
> two groups:
> i) Statically-placed reserved memory regions
> i.e. regions defined with a static start address and size using the
>      "reg" property.
> ii) Dynamically-placed reserved memory regions.
> i.e. regions defined by specifying an address range where they can be
>      placed in memory using the "alloc_ranges" and "size" properties.
> 
> These regions are processed and set aside at boot time.
> This is done in two stages as seen below:
> 
> Stage 1:
> At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> the reserved_memory node using the flattened devicetree and does the
> following:
> 
> 1) If the node represents a statically-placed reserved memory region,
>    i.e. if it is defined using the "reg" property:
>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
>    - Add the information for that region into the reserved_mem array
>      using fdt_reserved_mem_save_node().
>      i.e. fdt_reserved_mem_save_node(node, name, base, size).
> 
> 2) If the node represents a dynamically-placed reserved memory region,
>    i.e. if it is defined using "alloc-ranges" and "size" properties:
>    - Add the information for that region to the reserved_mem array with
>      the starting address and size set to 0.
>      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>    Note: This region is saved to the array with a starting address of 0
>    because a starting address is not yet allocated for it.
> 
> Stage 2:
> After iterating through all the reserved memory nodes and storing their
> relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> called and does the following:
> 
> 1) For statically-placed reserved memory regions:
>    - Call the region specific init function using
>      __reserved_mem_init_node().
> 2) For dynamically-placed reserved memory regions:
>    - Call __reserved_mem_alloc_size() which is used to allocate memory
>      for each of these regions, and mark them as nomap if they have the
>      nomap property specified in the DT.
>    - Call the region specific init function.
> 
> The current size of the resvered_mem array is 64 as is defined by
> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> how many reserved memory regions can be specified on a system.
> As systems continue to grow more and more complex, the number of
> reserved memory regions needed are also growing and are starting to hit
> this 64 count limit, hence the need to make the reserved_mem array
> dynamically sized (i.e. dynamically allocating memory for the
> reserved_mem array using membock_alloc_*).
> 
> On architectures such as arm64, memory allocated using memblock is
> writable only after the page tables have been setup. This means that if
> the reserved_mem array is going to be dynamically allocated, it needs to
> happen after the page tables have been setup, not before.
> 
> Since the reserved memory regions are currently being processed and
> added to the array before the page tables are setup, there is a need to
> change the order in which some of the processing is done to allow for
> the reserved_mem array to be dynamically sized.
> 
> It is possible to process the statically-placed reserved memory regions
> without needing to store them in the reserved_mem array until after the
> page tables have been setup because all the information stored in the
> array is readily available in the devicetree and can be referenced at
> any time.
> Dynamically-placed reserved memory regions on the other hand get
> assigned a start address only at runtime, and hence need a place to be
> stored once they are allocated since there is no other referrence to the
> start address for these regions.
> 
> Hence this patch changes the processing order of the reserved memory
> regions in the following ways:
> 
> Step 1:
> fdt_scan_reserved_mem() scans through the child nodes of
> the reserved_memory node using the flattened devicetree and does the
> following:
> 
> 1) If the node represents a statically-placed reserved memory region,
>    i.e. if it is defined using the "reg" property:
>    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> 
> 2) If the node represents a dynamically-placed reserved memory region,
>    i.e. if it is defined using "alloc-ranges" and "size" properties:
>    - Call __reserved_mem_alloc_size() which will:
>      i) Allocate memory for the reserved region and call
>      memblock_mark_nomap() as needed.
>      ii) Call the region specific initialization function using
>      fdt_init_reserved_mem_node().
>      iii) Save the region information in the reserved_mem array using
>      fdt_reserved_mem_save_node().
> 
> Step 2:
> 1) This stage of the reserved memory processing is now only used to add
>    the statically-placed reserved memory regions into the reserved_mem
>    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
>    region specific initialization functions.
> 
> 2) This step has also been moved to be after the page tables are
>    setup. Moving this will allow us to replace the reserved_mem
>    array with a dynamically sized array before storing the rest of
>    these regions.
> 
> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> ---
>  drivers/of/fdt.c             |   5 +-
>  drivers/of/of_private.h      |   3 +-
>  drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
>  3 files changed, 122 insertions(+), 54 deletions(-)

this patch got into stable kernel 6.12.13++ as part of Stable-dep-of.
The stable kernel commit is: 9a0fe62f93ede02c27aaca81112af1e59c8c0979.

With the patch applied I see that the cma area pool is misplaced which
cause my 4G device to fail to activate the cma pool. Below are some
logs:

*** Good case (6.12)

root@test:~# dmesg|grep -i cma
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
[    0.056915] Memory: 3695024K/4194304K available (15552K kernel code, 2510K rwdata, 5992K rodata, 6016K init, 489K bss, 231772K reserved, 262144K cma-reserved)

*** Bad (6.12.16)

root@test:~# dmesg|grep -i cma
[    0.000000] Reserved memory: created CMA memory pool at 0x00000000f2000000, size 256 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x00000000f2000000..0x0000000101ffffff (262144 KiB) map reusable linux,cma
[    0.056968] Memory: 3694896K/4194304K available (15616K kernel code, 2512K rwdata, 6012K rodata, 6080K init, 491K bss, 231900K reserved, 262144K cma-reserved)
[    0.116920] cma: CMA area linux,cma could not be activated

*** Good (6.12.16, revert 9a0fe62f93ed)

root@test:~# dmesg|grep -i cma
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
[    0.060976] Memory: 3694896K/4194304K available (15616K kernel code, 2512K rwdata, 6012K rodata, 6080K init, 491K bss, 231900K reserved, 262144K cma-reserved)

Below is our reserved-memory dts node:

reserved-memory {
	#address-cells = <2>; 
	#size-cells = <2>; 
	ranges;

	linux,cma {
		compatible = "shared-dma-pool";
		reusable;
		/*
		 * The CMA area must be in the lower 32-bit address range.
		 */
		alloc-ranges = <0x0 0x42000000 0 0xc0000000>;
		size = <0x0 0x10000000>;
		alignment = <0 0x2000>;
		linux,cma-default;
	};

	optee-core@40000000 {
		reg = <0 0x40000000 0 0x1e00000>;
		no-map;
	};

	optee-shm@41e00000 {
		reg = <0 0x41e00000 0 0x200000>;
		no-map;
	};

	m7_reserved: m7@80000000 {
		reg = <0 0x80000000 0 0x1000000>;
		no-map;
	};

	vdev0vring0: vdev0vring0@55000000 {
		reg = <0 0x55000000 0 0x8000>;
		no-map;
	};

	vdev0vring1: vdev0vring1@55008000 {
		reg = <0 0x55008000 0 0x8000>;
		no-map;
	};

	rsc_table: rsc-table@550ff000 {
		reg = <0 0x550ff000 0 0x1000>;
		no-map;
	};

	ram_console_buffer: ram-console-buffer@55100000 {
		reg = <0 0x55100000 0 0x1000>;
		no-map;
	};

	vdev0buffer: vdev0buffer@55400000 {
		compatible = "shared-dma-pool";
		reg = <0 0x55400000 0 0x100000>;
		no-map;
	};
};

My current workaround is to revert commit 9a0fe62f93ed and the
dep-chain: 2d1d620ff27b444 8de4e5a92282. But I would like to get a
proper solution without having revert commits in my downstream
patchstack.

Regards,
  Marco

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Marco Felsch]

On 25-02-26, Marco Felsch wrote:
> Hi,
> 
> On 24-10-08, Oreoluwa Babatunde wrote:
> > Reserved memory regions defined in the devicetree can be broken up into
> > two groups:
> > i) Statically-placed reserved memory regions
> > i.e. regions defined with a static start address and size using the
> >      "reg" property.
> > ii) Dynamically-placed reserved memory regions.
> > i.e. regions defined by specifying an address range where they can be
> >      placed in memory using the "alloc_ranges" and "size" properties.
> > 
> > These regions are processed and set aside at boot time.
> > This is done in two stages as seen below:
> > 
> > Stage 1:
> > At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> > 
> > 1) If the node represents a statically-placed reserved memory region,
> >    i.e. if it is defined using the "reg" property:
> >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >    - Add the information for that region into the reserved_mem array
> >      using fdt_reserved_mem_save_node().
> >      i.e. fdt_reserved_mem_save_node(node, name, base, size).
> > 
> > 2) If the node represents a dynamically-placed reserved memory region,
> >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >    - Add the information for that region to the reserved_mem array with
> >      the starting address and size set to 0.
> >      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
> >    Note: This region is saved to the array with a starting address of 0
> >    because a starting address is not yet allocated for it.
> > 
> > Stage 2:
> > After iterating through all the reserved memory nodes and storing their
> > relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> > called and does the following:
> > 
> > 1) For statically-placed reserved memory regions:
> >    - Call the region specific init function using
> >      __reserved_mem_init_node().
> > 2) For dynamically-placed reserved memory regions:
> >    - Call __reserved_mem_alloc_size() which is used to allocate memory
> >      for each of these regions, and mark them as nomap if they have the
> >      nomap property specified in the DT.
> >    - Call the region specific init function.
> > 
> > The current size of the resvered_mem array is 64 as is defined by
> > MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> > how many reserved memory regions can be specified on a system.
> > As systems continue to grow more and more complex, the number of
> > reserved memory regions needed are also growing and are starting to hit
> > this 64 count limit, hence the need to make the reserved_mem array
> > dynamically sized (i.e. dynamically allocating memory for the
> > reserved_mem array using membock_alloc_*).
> > 
> > On architectures such as arm64, memory allocated using memblock is
> > writable only after the page tables have been setup. This means that if
> > the reserved_mem array is going to be dynamically allocated, it needs to
> > happen after the page tables have been setup, not before.
> > 
> > Since the reserved memory regions are currently being processed and
> > added to the array before the page tables are setup, there is a need to
> > change the order in which some of the processing is done to allow for
> > the reserved_mem array to be dynamically sized.
> > 
> > It is possible to process the statically-placed reserved memory regions
> > without needing to store them in the reserved_mem array until after the
> > page tables have been setup because all the information stored in the
> > array is readily available in the devicetree and can be referenced at
> > any time.
> > Dynamically-placed reserved memory regions on the other hand get
> > assigned a start address only at runtime, and hence need a place to be
> > stored once they are allocated since there is no other referrence to the
> > start address for these regions.
> > 
> > Hence this patch changes the processing order of the reserved memory
> > regions in the following ways:
> > 
> > Step 1:
> > fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> > 
> > 1) If the node represents a statically-placed reserved memory region,
> >    i.e. if it is defined using the "reg" property:
> >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> > 
> > 2) If the node represents a dynamically-placed reserved memory region,
> >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >    - Call __reserved_mem_alloc_size() which will:
> >      i) Allocate memory for the reserved region and call
> >      memblock_mark_nomap() as needed.
> >      ii) Call the region specific initialization function using
> >      fdt_init_reserved_mem_node().
> >      iii) Save the region information in the reserved_mem array using
> >      fdt_reserved_mem_save_node().
> > 
> > Step 2:
> > 1) This stage of the reserved memory processing is now only used to add
> >    the statically-placed reserved memory regions into the reserved_mem
> >    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
> >    region specific initialization functions.
> > 
> > 2) This step has also been moved to be after the page tables are
> >    setup. Moving this will allow us to replace the reserved_mem
> >    array with a dynamically sized array before storing the rest of
> >    these regions.
> > 
> > Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> > ---
> >  drivers/of/fdt.c             |   5 +-
> >  drivers/of/of_private.h      |   3 +-
> >  drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
> >  3 files changed, 122 insertions(+), 54 deletions(-)
> 
> this patch got into stable kernel 6.12.13++ as part of Stable-dep-of.
> The stable kernel commit is: 9a0fe62f93ede02c27aaca81112af1e59c8c0979.
> 
> With the patch applied I see that the cma area pool is misplaced which
> cause my 4G device to fail to activate the cma pool. Below are some
> logs:
> 
> *** Good case (6.12)
> 
> root@test:~# dmesg|grep -i cma
> [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
> [    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
> [    0.056915] Memory: 3695024K/4194304K available (15552K kernel code, 2510K rwdata, 5992K rodata, 6016K init, 489K bss, 231772K reserved, 262144K cma-reserved)
> 
> *** Bad (6.12.16)
> 
> root@test:~# dmesg|grep -i cma
> [    0.000000] Reserved memory: created CMA memory pool at 0x00000000f2000000, size 256 MiB
> [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
> [    0.000000] OF: reserved mem: 0x00000000f2000000..0x0000000101ffffff (262144 KiB) map reusable linux,cma
> [    0.056968] Memory: 3694896K/4194304K available (15616K kernel code, 2512K rwdata, 6012K rodata, 6080K init, 491K bss, 231900K reserved, 262144K cma-reserved)
> [    0.116920] cma: CMA area linux,cma could not be activated
> 
> *** Good (6.12.16, revert 9a0fe62f93ed)
> 
> root@test:~# dmesg|grep -i cma
> [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
> [    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
> [    0.060976] Memory: 3694896K/4194304K available (15616K kernel code, 2512K rwdata, 6012K rodata, 6080K init, 491K bss, 231900K reserved, 262144K cma-reserved)
> 
> Below is our reserved-memory dts node:
> 
> reserved-memory {
> 	#address-cells = <2>; 
> 	#size-cells = <2>; 
> 	ranges;
> 
> 	linux,cma {
> 		compatible = "shared-dma-pool";
> 		reusable;
> 		/*
> 		 * The CMA area must be in the lower 32-bit address range.
> 		 */
> 		alloc-ranges = <0x0 0x42000000 0 0xc0000000>;
> 		size = <0x0 0x10000000>;
> 		alignment = <0 0x2000>;
> 		linux,cma-default;
> 	};
> 
> 	optee-core@40000000 {
> 		reg = <0 0x40000000 0 0x1e00000>;
> 		no-map;
> 	};
> 
> 	optee-shm@41e00000 {
> 		reg = <0 0x41e00000 0 0x200000>;
> 		no-map;
> 	};
> 
> 	m7_reserved: m7@80000000 {
> 		reg = <0 0x80000000 0 0x1000000>;
> 		no-map;
> 	};
> 
> 	vdev0vring0: vdev0vring0@55000000 {
> 		reg = <0 0x55000000 0 0x8000>;
> 		no-map;
> 	};
> 
> 	vdev0vring1: vdev0vring1@55008000 {
> 		reg = <0 0x55008000 0 0x8000>;
> 		no-map;
> 	};
> 
> 	rsc_table: rsc-table@550ff000 {
> 		reg = <0 0x550ff000 0 0x1000>;
> 		no-map;
> 	};
> 
> 	ram_console_buffer: ram-console-buffer@55100000 {
> 		reg = <0 0x55100000 0 0x1000>;
> 		no-map;
> 	};
> 
> 	vdev0buffer: vdev0buffer@55400000 {
> 		compatible = "shared-dma-pool";
> 		reg = <0 0x55400000 0 0x100000>;
> 		no-map;
> 	};
> };
> 
> My current workaround is to revert commit 9a0fe62f93ed and the
> dep-chain: 2d1d620ff27b444 8de4e5a92282. But I would like to get a

It's not 2d1d620ff27b444 but e61977c71494.

> proper solution without having revert commits in my downstream
> patchstack.
> 
> Regards,
>   Marco
> 
> 

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Rob Herring]

On Wed, Feb 26, 2025 at 5:51 AM Marco Felsch <m.felsch@pengutronix.de> wrote:
>
> Hi,
>
> On 24-10-08, Oreoluwa Babatunde wrote:
> > Reserved memory regions defined in the devicetree can be broken up into
> > two groups:
> > i) Statically-placed reserved memory regions
> > i.e. regions defined with a static start address and size using the
> >      "reg" property.
> > ii) Dynamically-placed reserved memory regions.
> > i.e. regions defined by specifying an address range where they can be
> >      placed in memory using the "alloc_ranges" and "size" properties.
> >
> > These regions are processed and set aside at boot time.
> > This is done in two stages as seen below:
> >
> > Stage 1:
> > At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> >
> > 1) If the node represents a statically-placed reserved memory region,
> >    i.e. if it is defined using the "reg" property:
> >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >    - Add the information for that region into the reserved_mem array
> >      using fdt_reserved_mem_save_node().
> >      i.e. fdt_reserved_mem_save_node(node, name, base, size).
> >
> > 2) If the node represents a dynamically-placed reserved memory region,
> >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >    - Add the information for that region to the reserved_mem array with
> >      the starting address and size set to 0.
> >      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
> >    Note: This region is saved to the array with a starting address of 0
> >    because a starting address is not yet allocated for it.
> >
> > Stage 2:
> > After iterating through all the reserved memory nodes and storing their
> > relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> > called and does the following:
> >
> > 1) For statically-placed reserved memory regions:
> >    - Call the region specific init function using
> >      __reserved_mem_init_node().
> > 2) For dynamically-placed reserved memory regions:
> >    - Call __reserved_mem_alloc_size() which is used to allocate memory
> >      for each of these regions, and mark them as nomap if they have the
> >      nomap property specified in the DT.
> >    - Call the region specific init function.
> >
> > The current size of the resvered_mem array is 64 as is defined by
> > MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> > how many reserved memory regions can be specified on a system.
> > As systems continue to grow more and more complex, the number of
> > reserved memory regions needed are also growing and are starting to hit
> > this 64 count limit, hence the need to make the reserved_mem array
> > dynamically sized (i.e. dynamically allocating memory for the
> > reserved_mem array using membock_alloc_*).
> >
> > On architectures such as arm64, memory allocated using memblock is
> > writable only after the page tables have been setup. This means that if
> > the reserved_mem array is going to be dynamically allocated, it needs to
> > happen after the page tables have been setup, not before.
> >
> > Since the reserved memory regions are currently being processed and
> > added to the array before the page tables are setup, there is a need to
> > change the order in which some of the processing is done to allow for
> > the reserved_mem array to be dynamically sized.
> >
> > It is possible to process the statically-placed reserved memory regions
> > without needing to store them in the reserved_mem array until after the
> > page tables have been setup because all the information stored in the
> > array is readily available in the devicetree and can be referenced at
> > any time.
> > Dynamically-placed reserved memory regions on the other hand get
> > assigned a start address only at runtime, and hence need a place to be
> > stored once they are allocated since there is no other referrence to the
> > start address for these regions.
> >
> > Hence this patch changes the processing order of the reserved memory
> > regions in the following ways:
> >
> > Step 1:
> > fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> >
> > 1) If the node represents a statically-placed reserved memory region,
> >    i.e. if it is defined using the "reg" property:
> >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >
> > 2) If the node represents a dynamically-placed reserved memory region,
> >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> >    - Call __reserved_mem_alloc_size() which will:
> >      i) Allocate memory for the reserved region and call
> >      memblock_mark_nomap() as needed.
> >      ii) Call the region specific initialization function using
> >      fdt_init_reserved_mem_node().
> >      iii) Save the region information in the reserved_mem array using
> >      fdt_reserved_mem_save_node().
> >
> > Step 2:
> > 1) This stage of the reserved memory processing is now only used to add
> >    the statically-placed reserved memory regions into the reserved_mem
> >    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
> >    region specific initialization functions.
> >
> > 2) This step has also been moved to be after the page tables are
> >    setup. Moving this will allow us to replace the reserved_mem
> >    array with a dynamically sized array before storing the rest of
> >    these regions.
> >
> > Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> > ---
> >  drivers/of/fdt.c             |   5 +-
> >  drivers/of/of_private.h      |   3 +-
> >  drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
> >  3 files changed, 122 insertions(+), 54 deletions(-)
>
> this patch got into stable kernel 6.12.13++ as part of Stable-dep-of.
> The stable kernel commit is: 9a0fe62f93ede02c27aaca81112af1e59c8c0979.
>
> With the patch applied I see that the cma area pool is misplaced which
> cause my 4G device to fail to activate the cma pool. Below are some
> logs:
>
> *** Good case (6.12)
>
> root@test:~# dmesg|grep -i cma
> [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
> [    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
> [    0.056915] Memory: 3695024K/4194304K available (15552K kernel code, 2510K rwdata, 5992K rodata, 6016K init, 489K bss, 231772K reserved, 262144K cma-reserved)
>
> *** Bad (6.12.16)
>
> root@test:~# dmesg|grep -i cma
> [    0.000000] Reserved memory: created CMA memory pool at 0x00000000f2000000, size 256 MiB
> [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
> [    0.000000] OF: reserved mem: 0x00000000f2000000..0x0000000101ffffff (262144 KiB) map reusable linux,cma

>                 /*
>                  * The CMA area must be in the lower 32-bit address range.
>                  */
>                 alloc-ranges = <0x0 0x42000000 0 0xc0000000>;

Are you expecting that 0xc0000000 is the end address rather than the
size? Because your range ends at 0x1_0200_0000. Looks to me like the
kernel correctly followed what the DT said was allowed. Why it moved,
I don't know. If you change 0xc0000000 to 0xbe000000, does it work?

Rob

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Marco Felsch]

On 25-02-28, Rob Herring wrote:
> On Wed, Feb 26, 2025 at 5:51 AM Marco Felsch <m.felsch@pengutronix.de> wrote:
> >
> > Hi,
> >
> > On 24-10-08, Oreoluwa Babatunde wrote:
> > > Reserved memory regions defined in the devicetree can be broken up into
> > > two groups:
> > > i) Statically-placed reserved memory regions
> > > i.e. regions defined with a static start address and size using the
> > >      "reg" property.
> > > ii) Dynamically-placed reserved memory regions.
> > > i.e. regions defined by specifying an address range where they can be
> > >      placed in memory using the "alloc_ranges" and "size" properties.
> > >
> > > These regions are processed and set aside at boot time.
> > > This is done in two stages as seen below:
> > >
> > > Stage 1:
> > > At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> > > the reserved_memory node using the flattened devicetree and does the
> > > following:
> > >
> > > 1) If the node represents a statically-placed reserved memory region,
> > >    i.e. if it is defined using the "reg" property:
> > >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> > >    - Add the information for that region into the reserved_mem array
> > >      using fdt_reserved_mem_save_node().
> > >      i.e. fdt_reserved_mem_save_node(node, name, base, size).
> > >
> > > 2) If the node represents a dynamically-placed reserved memory region,
> > >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> > >    - Add the information for that region to the reserved_mem array with
> > >      the starting address and size set to 0.
> > >      i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
> > >    Note: This region is saved to the array with a starting address of 0
> > >    because a starting address is not yet allocated for it.
> > >
> > > Stage 2:
> > > After iterating through all the reserved memory nodes and storing their
> > > relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> > > called and does the following:
> > >
> > > 1) For statically-placed reserved memory regions:
> > >    - Call the region specific init function using
> > >      __reserved_mem_init_node().
> > > 2) For dynamically-placed reserved memory regions:
> > >    - Call __reserved_mem_alloc_size() which is used to allocate memory
> > >      for each of these regions, and mark them as nomap if they have the
> > >      nomap property specified in the DT.
> > >    - Call the region specific init function.
> > >
> > > The current size of the resvered_mem array is 64 as is defined by
> > > MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> > > how many reserved memory regions can be specified on a system.
> > > As systems continue to grow more and more complex, the number of
> > > reserved memory regions needed are also growing and are starting to hit
> > > this 64 count limit, hence the need to make the reserved_mem array
> > > dynamically sized (i.e. dynamically allocating memory for the
> > > reserved_mem array using membock_alloc_*).
> > >
> > > On architectures such as arm64, memory allocated using memblock is
> > > writable only after the page tables have been setup. This means that if
> > > the reserved_mem array is going to be dynamically allocated, it needs to
> > > happen after the page tables have been setup, not before.
> > >
> > > Since the reserved memory regions are currently being processed and
> > > added to the array before the page tables are setup, there is a need to
> > > change the order in which some of the processing is done to allow for
> > > the reserved_mem array to be dynamically sized.
> > >
> > > It is possible to process the statically-placed reserved memory regions
> > > without needing to store them in the reserved_mem array until after the
> > > page tables have been setup because all the information stored in the
> > > array is readily available in the devicetree and can be referenced at
> > > any time.
> > > Dynamically-placed reserved memory regions on the other hand get
> > > assigned a start address only at runtime, and hence need a place to be
> > > stored once they are allocated since there is no other referrence to the
> > > start address for these regions.
> > >
> > > Hence this patch changes the processing order of the reserved memory
> > > regions in the following ways:
> > >
> > > Step 1:
> > > fdt_scan_reserved_mem() scans through the child nodes of
> > > the reserved_memory node using the flattened devicetree and does the
> > > following:
> > >
> > > 1) If the node represents a statically-placed reserved memory region,
> > >    i.e. if it is defined using the "reg" property:
> > >    - Call memblock_reserve() or memblock_mark_nomap() as needed.
> > >
> > > 2) If the node represents a dynamically-placed reserved memory region,
> > >    i.e. if it is defined using "alloc-ranges" and "size" properties:
> > >    - Call __reserved_mem_alloc_size() which will:
> > >      i) Allocate memory for the reserved region and call
> > >      memblock_mark_nomap() as needed.
> > >      ii) Call the region specific initialization function using
> > >      fdt_init_reserved_mem_node().
> > >      iii) Save the region information in the reserved_mem array using
> > >      fdt_reserved_mem_save_node().
> > >
> > > Step 2:
> > > 1) This stage of the reserved memory processing is now only used to add
> > >    the statically-placed reserved memory regions into the reserved_mem
> > >    array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
> > >    region specific initialization functions.
> > >
> > > 2) This step has also been moved to be after the page tables are
> > >    setup. Moving this will allow us to replace the reserved_mem
> > >    array with a dynamically sized array before storing the rest of
> > >    these regions.
> > >
> > > Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> > > ---
> > >  drivers/of/fdt.c             |   5 +-
> > >  drivers/of/of_private.h      |   3 +-
> > >  drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
> > >  3 files changed, 122 insertions(+), 54 deletions(-)
> >
> > this patch got into stable kernel 6.12.13++ as part of Stable-dep-of.
> > The stable kernel commit is: 9a0fe62f93ede02c27aaca81112af1e59c8c0979.
> >
> > With the patch applied I see that the cma area pool is misplaced which
> > cause my 4G device to fail to activate the cma pool. Below are some
> > logs:
> >
> > *** Good case (6.12)
> >
> > root@test:~# dmesg|grep -i cma
> > [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
> > [    0.000000] OF: reserved mem: 0x0000000044200000..0x00000000541fffff (262144 KiB) map reusable linux,cma
> > [    0.056915] Memory: 3695024K/4194304K available (15552K kernel code, 2510K rwdata, 5992K rodata, 6016K init, 489K bss, 231772K reserved, 262144K cma-reserved)
> >
> > *** Bad (6.12.16)
> >
> > root@test:~# dmesg|grep -i cma
> > [    0.000000] Reserved memory: created CMA memory pool at 0x00000000f2000000, size 256 MiB
> > [    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
> > [    0.000000] OF: reserved mem: 0x00000000f2000000..0x0000000101ffffff (262144 KiB) map reusable linux,cma
> 
> >                 /*
> >                  * The CMA area must be in the lower 32-bit address range.
> >                  */
> >                 alloc-ranges = <0x0 0x42000000 0 0xc0000000>;
> 
> Are you expecting that 0xc0000000 is the end address rather than the
> size? Because your range ends at 0x1_0200_0000. Looks to me like the
> kernel correctly followed what the DT said was allowed. Why it moved,
> I don't know. If you change 0xc0000000 to 0xbe000000, does it work?

Argh.. you're absolutely right, we had the wrong size specified and
didn't noticed it till the kernel update :/ I don't know why I didn't
noticed this earlier.

With the length/size set to 0xbe000000, the cma is now allocated
correctly:

root@test:~# dmesg|grep cma
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x00000000f0000000..0x00000000ffffffff (262144 KiB) map reusable linux,cma
[    0.060910] Memory: 3694896K/4194304K available (15616K kernel code, 2512K rwdata, 6016K rodata, 6080K init, 491K bss, 231900K reserved, 262144K cma-reserved)

As you said the the placing was changed too, not sure why but that's
okay. To be honest, placing it at the end and not somewhere in between
sounds far more reasonable to me.

Thanks for the support,
Marco

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[William Zhang]

Hi Oreoluwa,

On 10/8/2024 3:06 PM, Oreoluwa Babatunde wrote:
> Reserved memory regions defined in the devicetree can be broken up into
> two groups:
> i) Statically-placed reserved memory regions
> i.e. regions defined with a static start address and size using the
>       "reg" property.
> ii) Dynamically-placed reserved memory regions.
> i.e. regions defined by specifying an address range where they can be
>       placed in memory using the "alloc_ranges" and "size" properties.
> 
> These regions are processed and set aside at boot time.
> This is done in two stages as seen below:
> 
> Stage 1:
> At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> the reserved_memory node using the flattened devicetree and does the
> following:
> 
> 1) If the node represents a statically-placed reserved memory region,
>     i.e. if it is defined using the "reg" property:
>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
>     - Add the information for that region into the reserved_mem array
>       using fdt_reserved_mem_save_node().
>       i.e. fdt_reserved_mem_save_node(node, name, base, size).
> 
> 2) If the node represents a dynamically-placed reserved memory region,
>     i.e. if it is defined using "alloc-ranges" and "size" properties:
>     - Add the information for that region to the reserved_mem array with
>       the starting address and size set to 0.
>       i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>     Note: This region is saved to the array with a starting address of 0
>     because a starting address is not yet allocated for it.
> 
> Stage 2:
> After iterating through all the reserved memory nodes and storing their
> relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> called and does the following:
> 
> 1) For statically-placed reserved memory regions:
>     - Call the region specific init function using
>       __reserved_mem_init_node().
> 2) For dynamically-placed reserved memory regions:
>     - Call __reserved_mem_alloc_size() which is used to allocate memory
>       for each of these regions, and mark them as nomap if they have the
>       nomap property specified in the DT.
>     - Call the region specific init function.
> 
> The current size of the resvered_mem array is 64 as is defined by
> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> how many reserved memory regions can be specified on a system.
> As systems continue to grow more and more complex, the number of
> reserved memory regions needed are also growing and are starting to hit
> this 64 count limit, hence the need to make the reserved_mem array
> dynamically sized (i.e. dynamically allocating memory for the
> reserved_mem array using membock_alloc_*).
> 
> On architectures such as arm64, memory allocated using memblock is
> writable only after the page tables have been setup. This means that if
> the reserved_mem array is going to be dynamically allocated, it needs to
> happen after the page tables have been setup, not before.
> 
> Since the reserved memory regions are currently being processed and
> added to the array before the page tables are setup, there is a need to
> change the order in which some of the processing is done to allow for
> the reserved_mem array to be dynamically sized.
> 
> It is possible to process the statically-placed reserved memory regions
> without needing to store them in the reserved_mem array until after the
> page tables have been setup because all the information stored in the
> array is readily available in the devicetree and can be referenced at
> any time.
> Dynamically-placed reserved memory regions on the other hand get
> assigned a start address only at runtime, and hence need a place to be
> stored once they are allocated since there is no other referrence to the
> start address for these regions.
> 
> Hence this patch changes the processing order of the reserved memory
> regions in the following ways:
> 
> Step 1:
> fdt_scan_reserved_mem() scans through the child nodes of
> the reserved_memory node using the flattened devicetree and does the
> following:
> 
> 1) If the node represents a statically-placed reserved memory region,
>     i.e. if it is defined using the "reg" property:
>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
> 
> 2) If the node represents a dynamically-placed reserved memory region,
>     i.e. if it is defined using "alloc-ranges" and "size" properties:
>     - Call __reserved_mem_alloc_size() which will:
>       i) Allocate memory for the reserved region and call
>       memblock_mark_nomap() as needed.
>       ii) Call the region specific initialization function using
>       fdt_init_reserved_mem_node().
>       iii) Save the region information in the reserved_mem array using
>       fdt_reserved_mem_save_node().
> 
> Step 2:
> 1) This stage of the reserved memory processing is now only used to add
>     the statically-placed reserved memory regions into the reserved_mem
>     array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
>     region specific initialization functions.
> 
> 2) This step has also been moved to be after the page tables are
>     setup. Moving this will allow us to replace the reserved_mem
>     array with a dynamically sized array before storing the rest of
>     these regions.
I am running into a call trace with this order change on armv7 chip when 
I tried to allocate dma coherent memory from the device reserved memory. 
The issue does not happen on armv8 chips.

[    0.000000] Reserved memory: created CMA memory pool at 0x1e000000, 
size 32 MiB
[    0.000000] OF: reserved mem: initialized node dt_reserved_cma, 
compatible id shared-dma-pool
[    0.000000] OF: reserved mem: 0x1e000000..0x1fffffff (32768 KiB) map 
reusable dt_reserved_cma
....

[    0.445322] ------------[ cut here ]------------
[    0.445353] WARNING: CPU: 0 PID: 1 at mm/memory.c:3069 
__apply_to_page_range+0x380/0x388
[    0.488911] Modules linked in:
[    0.492027] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted 
6.16.0-rc1-g27605c8c0f69-dirty #3 NONE
[    0.501174] Hardware name: Generic DT based system
[    0.505965] Call trace:
[    0.505985]  unwind_backtrace from show_stack+0x10/0x14
[    0.513764]  show_stack from dump_stack_lvl+0x54/0x68
[    0.518834]  dump_stack_lvl from __warn+0x7c/0x128
[    0.523639]  __warn from warn_slowpath_fmt+0x184/0x18c
[    0.527676] Freeing initrd memory: 65536K
[    0.532788]  warn_slowpath_fmt from __apply_to_page_range+0x380/0x388
[    0.539242]  __apply_to_page_range from apply_to_page_range+0x1c/0x24
[    0.545689]  apply_to_page_range from __alloc_from_contiguous+0xc0/0x14c
[    0.552398]  __alloc_from_contiguous from cma_allocator_alloc+0x34/0x3c
[    0.559016]  cma_allocator_alloc from arch_dma_alloc+0x11c/0x2ac
[    0.565025]  arch_dma_alloc from dma_alloc_attrs+0x90/0x2e8
[    0.570603]  dma_alloc_attrs from dmydev_probe+0x8c/0xe8
[    0.575919]  dmydev_probe from platform_probe+0x5c/0xb0
[    0.581152]  platform_probe from really_probe+0xc8/0x2c8
[    0.586467]  really_probe from __driver_probe_device+0x88/0x19c
[    0.592387]  __driver_probe_device from driver_probe_device+0x30/0x104
[    0.598915]  driver_probe_device from __driver_attach+0x90/0x178
[    0.604921]  __driver_attach from bus_for_each_dev+0x7c/0xcc
[    0.610582]  bus_for_each_dev from bus_add_driver+0xcc/0x1ec
[    0.616241]  bus_add_driver from driver_register+0x7c/0x114
[    0.621814]  driver_register from dmydev_init+0x20/0x28
[    0.627045]  dmydev_init from do_one_initcall+0x58/0x200
[    0.632363]  do_one_initcall from kernel_init_freeable+0x1cc/0x228
[    0.638550]  kernel_init_freeable from kernel_init+0x1c/0x12c
[    0.644299]  kernel_init from ret_from_fork+0x14/0x28
[    0.649351] Exception stack(0xe0819fb0 to 0xe0819ff8)
[    0.654401] 9fa0:                                     00000000 
00000000 00000000 00000000
[    0.662575] 9fc0: 00000000 00000000 00000000 00000000 00000000 
00000000 00000000 00000000
[    0.670747] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[    0.677403] ---[ end trace 0000000000000000 ]---
[    0.682083] dmydev dmy_device: Allocate dma memory at 0xde000000 dma 
addr 0x1e000000

The reason is that now reserved memory's fixup function 
dma_contiguous_early_fixup is called after the page table is 
initialized. This fixup function increases the dma_mmu_remap count for 
each reserved memory. And the dma_contiguous_remap function depends on 
it to properly set up the reserved memory mmu table entry. Before this 
change, the paging_init function calls dma_contiguous_remap and it 
founds the reserved memory and set it up properly.  After the change, 
this function found there is no reserved memory so skip any 
initialization hence causes the crash later on when my driver tries to 
allocate dma memory from the reserved memory.

My workaround below is to move the dma_contiguous_remap out from the 
paging_init function to the place right after unflatten_device_tree 
where the dma_mmu_remap count is correctly set. But this is not ideal 
solution and would like to see if you have any better way to solve the 
issue.

diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
index a41c93988d2c..535d1bf44529 100644
--- a/arch/arm/kernel/setup.c
+++ b/arch/arm/kernel/setup.c
@@ -1079,6 +1079,7 @@ void __init hyp_mode_check(void)
  #endif
  }

+void __init dma_contiguous_remap(void);
  static void (*__arm_pm_restart)(enum reboot_mode reboot_mode, const 
char *cmd);

  static int arm_restart(struct notifier_block *nb, unsigned long action,
@@ -1164,6 +1165,7 @@ void __init setup_arch(char **cmdline_p)
         }

         unflatten_device_tree();
+       dma_contiguous_remap();

         arm_dt_init_cpu_maps();
         psci_dt_init();
diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
index edb7f56b7c91..1828c8737d70 100644
--- a/arch/arm/mm/mmu.c
+++ b/arch/arm/mm/mmu.c
@@ -1773,7 +1773,6 @@ void __init paging_init(const struct machine_desc 
*mdesc)
          * be used
          */
         map_kernel();
-       dma_contiguous_remap();
         early_fixmap_shutdown();
         devicemaps_init(mdesc);
         kmap_init();

You can reproduce the issue on any v7 devices by adding these nodes to 
the device tree
+       reserved-memory {
+               #address-cells = <1>;
+               #size-cells = <1>;
+               ranges;
+
+               dt_reserved_cma: dt_reserved_cma {
+                       compatible = "shared-dma-pool";
+                       reusable;
+
+                       reg = <0x1e000000 0x2000000>;
+               };
+       };
+
+       dmy_device {
+               compatible = "xyz,dmydev";
+               memory-region = <&dt_reserved_cma>;
+       };

And use this test driver to trigger the call stack:
diff --git a/drivers/char/dmydev.c b/drivers/char/dmydev.c
new file mode 100644
index 000000000000..1dd52ec492eb
--- /dev/null
+++ b/drivers/char/dmydev.c
@@ -0,0 +1,67 @@
+#include<linux/module.h>
+#include<linux/kernel.h>
+#include <linux/platform_device.h>
+#include <linux/of.h>
+#include <linux/of_reserved_mem.h>
+#include <linux/dma-mapping.h>
+
+static int dmydev_probe(struct platform_device *pdev)
+{
+       void* virt_addr;
+       dma_addr_t dma_addr;
+       int ret;
+
+       printk(KERN_ALERT "dmydev_probe called\n");
+
+       ret = of_reserved_mem_device_init(&pdev->dev);
+       if (ret && ret != -ENODEV) {
+               dev_err(&pdev->dev, "Couldn't assign reserve memory to 
device ret = %d\n", ret);
+                       return ret;
+       }
+
+       virt_addr = dma_alloc_coherent(&pdev->dev, 0x800000, &dma_addr, 
GFP_KERNEL);
+       if (virt_addr == NULL) {
+               dev_err(&pdev->dev,"Failed to allocated cma memory\n");
+               ret = -ENOMEM;
+       }
+       else
+               dev_err(&pdev->dev,"Allocate dma memory at 0x%px dma 
addr %pad\n", virt_addr, &dma_addr);
+
+       return ret;
+}
+
+static void dmydev_remove(struct platform_device *pdev)
+{
+}
+
+static const struct of_device_id dmydev_of_match[] = {
+       {.compatible = "xyz,dmydev"},
+       {}
+};
+MODULE_DEVICE_TABLE(of, dmydev_of_match);

Let me know if you need more info.

> 
> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> ---
>   drivers/of/fdt.c             |   5 +-
>   drivers/of/of_private.h      |   3 +-
>   drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
>   3 files changed, 122 insertions(+), 54 deletions(-)
> 
> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
> index 4d528c10df3a..d0dbc8183ac4 100644
> --- a/drivers/of/fdt.c
> +++ b/drivers/of/fdt.c
> @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
>   			break;
>   		memblock_reserve(base, size);
>   	}
> -
> -	fdt_init_reserved_mem();
>   }
>   
>   /**
> @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
>   {
>   	void *fdt = initial_boot_params;
>   
> +	/* Save the statically-placed regions in the reserved_mem array */
> +	fdt_scan_reserved_mem_reg_nodes();
> +
>   	/* Don't use the bootloader provided DTB if ACPI is enabled */
>   	if (!acpi_disabled)
>   		fdt = NULL;
> diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
> index 04aa2a91f851..29525c0b9939 100644
> --- a/drivers/of/of_private.h
> +++ b/drivers/of/of_private.h
> @@ -9,6 +9,7 @@
>    */
>   
>   #define FDT_ALIGN_SIZE 8
> +#define MAX_RESERVED_REGIONS    64
>   
>   /**
>    * struct alias_prop - Alias property in 'aliases' node
> @@ -180,7 +181,7 @@ static inline struct device_node *__of_get_dma_parent(const struct device_node *
>   #endif
>   
>   int fdt_scan_reserved_mem(void);
> -void fdt_init_reserved_mem(void);
> +void __init fdt_scan_reserved_mem_reg_nodes(void);
>   
>   bool of_fdt_device_is_available(const void *blob, unsigned long node);
>   
> diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
> index 46e1c3fbc769..2011174211f9 100644
> --- a/drivers/of/of_reserved_mem.c
> +++ b/drivers/of/of_reserved_mem.c
> @@ -27,7 +27,6 @@
>   
>   #include "of_private.h"
>   
> -#define MAX_RESERVED_REGIONS	64
>   static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
>   static int reserved_mem_count;
>   
> @@ -56,6 +55,7 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
>   	return err;
>   }
>   
> +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem);
>   /*
>    * fdt_reserved_mem_save_node() - save fdt node for second pass initialization
>    */
> @@ -74,6 +74,9 @@ static void __init fdt_reserved_mem_save_node(unsigned long node, const char *un
>   	rmem->base = base;
>   	rmem->size = size;
>   
> +	/* Call the region specific initialization function */
> +	fdt_init_reserved_mem_node(rmem);
> +
>   	reserved_mem_count++;
>   	return;
>   }
> @@ -106,7 +109,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
>   	phys_addr_t base, size;
>   	int len;
>   	const __be32 *prop;
> -	int first = 1;
>   	bool nomap;
>   
>   	prop = of_get_flat_dt_prop(node, "reg", &len);
> @@ -134,10 +136,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
>   			       uname, &base, (unsigned long)(size / SZ_1M));
>   
>   		len -= t_len;
> -		if (first) {
> -			fdt_reserved_mem_save_node(node, uname, base, size);
> -			first = 0;
> -		}
>   	}
>   	return 0;
>   }
> @@ -165,12 +163,77 @@ static int __init __reserved_mem_check_root(unsigned long node)
>   	return 0;
>   }
>   
> +static void __init __rmem_check_for_overlap(void);
> +
> +/**
> + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg" defined
> + * reserved memory regions.
> + *
> + * This function is used to scan through the DT and store the
> + * information for the reserved memory regions that are defined using
> + * the "reg" property. The region node number, name, base address, and
> + * size are all stored in the reserved_mem array by calling the
> + * fdt_reserved_mem_save_node() function.
> + */
> +void __init fdt_scan_reserved_mem_reg_nodes(void)
> +{
> +	int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
> +	const void *fdt = initial_boot_params;
> +	phys_addr_t base, size;
> +	const __be32 *prop;
> +	int node, child;
> +	int len;
> +
> +	if (!fdt)
> +		return;
> +
> +	node = fdt_path_offset(fdt, "/reserved-memory");
> +	if (node < 0) {
> +		pr_info("Reserved memory: No reserved-memory node in the DT\n");
> +		return;
> +	}
> +
> +	if (__reserved_mem_check_root(node)) {
> +		pr_err("Reserved memory: unsupported node format, ignoring\n");
> +		return;
> +	}
> +
> +	fdt_for_each_subnode(child, fdt, node) {
> +		const char *uname;
> +
> +		prop = of_get_flat_dt_prop(child, "reg", &len);
> +		if (!prop)
> +			continue;
> +		if (!of_fdt_device_is_available(fdt, child))
> +			continue;
> +
> +		uname = fdt_get_name(fdt, child, NULL);
> +		if (len && len % t_len != 0) {
> +			pr_err("Reserved memory: invalid reg property in '%s', skipping node.\n",
> +			       uname);
> +			continue;
> +		}
> +		base = dt_mem_next_cell(dt_root_addr_cells, &prop);
> +		size = dt_mem_next_cell(dt_root_size_cells, &prop);
> +
> +		if (size)
> +			fdt_reserved_mem_save_node(child, uname, base, size);
> +	}
> +
> +	/* check for overlapping reserved regions */
> +	__rmem_check_for_overlap();
> +}
> +
> +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname);
> +
>   /*
>    * fdt_scan_reserved_mem() - scan a single FDT node for reserved memory
>    */
>   int __init fdt_scan_reserved_mem(void)
>   {
>   	int node, child;
> +	int dynamic_nodes_cnt = 0;
> +	int dynamic_nodes[MAX_RESERVED_REGIONS];
>   	const void *fdt = initial_boot_params;
>   
>   	node = fdt_path_offset(fdt, "/reserved-memory");
> @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
>   		uname = fdt_get_name(fdt, child, NULL);
>   
>   		err = __reserved_mem_reserve_reg(child, uname);
> -		if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL))
> -			fdt_reserved_mem_save_node(child, uname, 0, 0);
> +		/*
> +		 * Save the nodes for the dynamically-placed regions
> +		 * into an array which will be used for allocation right
> +		 * after all the statically-placed regions are reserved
> +		 * or marked as no-map. This is done to avoid dynamically
> +		 * allocating from one of the statically-placed regions.
> +		 */
> +		if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL)) {
> +			dynamic_nodes[dynamic_nodes_cnt] = child;
> +			dynamic_nodes_cnt++;
> +		}
> +	}
> +	for (int i = 0; i < dynamic_nodes_cnt; i++) {
> +		const char *uname;
> +
> +		child = dynamic_nodes[i];
> +		uname = fdt_get_name(fdt, child, NULL);
> +		__reserved_mem_alloc_size(child, uname);
>   	}
>   	return 0;
>   }
> @@ -253,8 +332,7 @@ static int __init __reserved_mem_alloc_in_range(phys_addr_t size,
>    * __reserved_mem_alloc_size() - allocate reserved memory described by
>    *	'size', 'alignment'  and 'alloc-ranges' properties.
>    */
> -static int __init __reserved_mem_alloc_size(unsigned long node,
> -	const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
> +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname)
>   {
>   	int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
>   	phys_addr_t start = 0, end = 0;
> @@ -334,9 +412,8 @@ static int __init __reserved_mem_alloc_size(unsigned long node,
>   		return -ENOMEM;
>   	}
>   
> -	*res_base = base;
> -	*res_size = size;
> -
> +	/* Save region in the reserved_mem array */
> +	fdt_reserved_mem_save_node(node, uname, base, size);
>   	return 0;
>   }
>   
> @@ -425,48 +502,37 @@ static void __init __rmem_check_for_overlap(void)
>   }
>   
>   /**
> - * fdt_init_reserved_mem() - allocate and init all saved reserved memory regions
> + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
> + * @rmem: reserved_mem struct of the memory region to be initialized.
> + *
> + * This function is used to call the region specific initialization
> + * function for a reserved memory region.
>    */
> -void __init fdt_init_reserved_mem(void)
> +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem)
>   {
> -	int i;
> -
> -	/* check for overlapping reserved regions */
> -	__rmem_check_for_overlap();
> -
> -	for (i = 0; i < reserved_mem_count; i++) {
> -		struct reserved_mem *rmem = &reserved_mem[i];
> -		unsigned long node = rmem->fdt_node;
> -		int err = 0;
> -		bool nomap;
> +	unsigned long node = rmem->fdt_node;
> +	int err = 0;
> +	bool nomap;
>   
> -		nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
> +	nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
>   
> -		if (rmem->size == 0)
> -			err = __reserved_mem_alloc_size(node, rmem->name,
> -						 &rmem->base, &rmem->size);
> -		if (err == 0) {
> -			err = __reserved_mem_init_node(rmem);
> -			if (err != 0 && err != -ENOENT) {
> -				pr_info("node %s compatible matching fail\n",
> -					rmem->name);
> -				if (nomap)
> -					memblock_clear_nomap(rmem->base, rmem->size);
> -				else
> -					memblock_phys_free(rmem->base,
> -							   rmem->size);
> -			} else {
> -				phys_addr_t end = rmem->base + rmem->size - 1;
> -				bool reusable =
> -					(of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> -
> -				pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> -					&rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> -					nomap ? "nomap" : "map",
> -					reusable ? "reusable" : "non-reusable",
> -					rmem->name ? rmem->name : "unknown");
> -			}
> -		}
> +	err = __reserved_mem_init_node(rmem);
> +	if (err != 0 && err != -ENOENT) {
> +		pr_info("node %s compatible matching fail\n", rmem->name);
> +		if (nomap)
> +			memblock_clear_nomap(rmem->base, rmem->size);
> +		else
> +			memblock_phys_free(rmem->base, rmem->size);
> +	} else {
> +		phys_addr_t end = rmem->base + rmem->size - 1;
> +		bool reusable =
> +			(of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> +
> +		pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> +			&rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> +			nomap ? "nomap" : "map",
> +			reusable ? "reusable" : "non-reusable",
> +			rmem->name ? rmem->name : "unknown");
>   	}
>   }
>   

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[William Zhang]

[-- Attachment #1: Type: text/plain, Size: 25961 bytes --]

On Tue, Jun 17, 2025 at 10:15 AM William Zhang
<william.zhang@broadcom.com> wrote:
>
> Hi Oreoluwa,
>
> On 10/8/2024 3:06 PM, Oreoluwa Babatunde wrote:
> > Reserved memory regions defined in the devicetree can be broken up into
> > two groups:
> > i) Statically-placed reserved memory regions
> > i.e. regions defined with a static start address and size using the
> >       "reg" property.
> > ii) Dynamically-placed reserved memory regions.
> > i.e. regions defined by specifying an address range where they can be
> >       placed in memory using the "alloc_ranges" and "size" properties.
> >
> > These regions are processed and set aside at boot time.
> > This is done in two stages as seen below:
> >
> > Stage 1:
> > At this stage, fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> >
> > 1) If the node represents a statically-placed reserved memory region,
> >     i.e. if it is defined using the "reg" property:
> >     - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >     - Add the information for that region into the reserved_mem array
> >       using fdt_reserved_mem_save_node().
> >       i.e. fdt_reserved_mem_save_node(node, name, base, size).
> >
> > 2) If the node represents a dynamically-placed reserved memory region,
> >     i.e. if it is defined using "alloc-ranges" and "size" properties:
> >     - Add the information for that region to the reserved_mem array with
> >       the starting address and size set to 0.
> >       i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
> >     Note: This region is saved to the array with a starting address of 0
> >     because a starting address is not yet allocated for it.
> >
> > Stage 2:
> > After iterating through all the reserved memory nodes and storing their
> > relevant information in the reserved_mem array,fdt_init_reserved_mem() is
> > called and does the following:
> >
> > 1) For statically-placed reserved memory regions:
> >     - Call the region specific init function using
> >       __reserved_mem_init_node().
> > 2) For dynamically-placed reserved memory regions:
> >     - Call __reserved_mem_alloc_size() which is used to allocate memory
> >       for each of these regions, and mark them as nomap if they have the
> >       nomap property specified in the DT.
> >     - Call the region specific init function.
> >
> > The current size of the resvered_mem array is 64 as is defined by
> > MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> > how many reserved memory regions can be specified on a system.
> > As systems continue to grow more and more complex, the number of
> > reserved memory regions needed are also growing and are starting to hit
> > this 64 count limit, hence the need to make the reserved_mem array
> > dynamically sized (i.e. dynamically allocating memory for the
> > reserved_mem array using membock_alloc_*).
> >
> > On architectures such as arm64, memory allocated using memblock is
> > writable only after the page tables have been setup. This means that if
> > the reserved_mem array is going to be dynamically allocated, it needs to
> > happen after the page tables have been setup, not before.
> >
> > Since the reserved memory regions are currently being processed and
> > added to the array before the page tables are setup, there is a need to
> > change the order in which some of the processing is done to allow for
> > the reserved_mem array to be dynamically sized.
> >
> > It is possible to process the statically-placed reserved memory regions
> > without needing to store them in the reserved_mem array until after the
> > page tables have been setup because all the information stored in the
> > array is readily available in the devicetree and can be referenced at
> > any time.
> > Dynamically-placed reserved memory regions on the other hand get
> > assigned a start address only at runtime, and hence need a place to be
> > stored once they are allocated since there is no other referrence to the
> > start address for these regions.
> >
> > Hence this patch changes the processing order of the reserved memory
> > regions in the following ways:
> >
> > Step 1:
> > fdt_scan_reserved_mem() scans through the child nodes of
> > the reserved_memory node using the flattened devicetree and does the
> > following:
> >
> > 1) If the node represents a statically-placed reserved memory region,
> >     i.e. if it is defined using the "reg" property:
> >     - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >
> > 2) If the node represents a dynamically-placed reserved memory region,
> >     i.e. if it is defined using "alloc-ranges" and "size" properties:
> >     - Call __reserved_mem_alloc_size() which will:
> >       i) Allocate memory for the reserved region and call
> >       memblock_mark_nomap() as needed.
> >       ii) Call the region specific initialization function using
> >       fdt_init_reserved_mem_node().
> >       iii) Save the region information in the reserved_mem array using
> >       fdt_reserved_mem_save_node().
> >
> > Step 2:
> > 1) This stage of the reserved memory processing is now only used to add
> >     the statically-placed reserved memory regions into the reserved_mem
> >     array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
> >     region specific initialization functions.
> >
> > 2) This step has also been moved to be after the page tables are
> >     setup. Moving this will allow us to replace the reserved_mem
> >     array with a dynamically sized array before storing the rest of
> >     these regions.
> I am running into a call trace with this order change on armv7 chip when
> I tried to allocate dma coherent memory from the device reserved memory.
> The issue does not happen on armv8 chips.
>
> [    0.000000] Reserved memory: created CMA memory pool at 0x1e000000,
> size 32 MiB
> [    0.000000] OF: reserved mem: initialized node dt_reserved_cma,
> compatible id shared-dma-pool
> [    0.000000] OF: reserved mem: 0x1e000000..0x1fffffff (32768 KiB) map
> reusable dt_reserved_cma
> ....
>
> [    0.445322] ------------[ cut here ]------------
> [    0.445353] WARNING: CPU: 0 PID: 1 at mm/memory.c:3069
> __apply_to_page_range+0x380/0x388
> [    0.488911] Modules linked in:
> [    0.492027] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted
> 6.16.0-rc1-g27605c8c0f69-dirty #3 NONE
> [    0.501174] Hardware name: Generic DT based system
> [    0.505965] Call trace:
> [    0.505985]  unwind_backtrace from show_stack+0x10/0x14
> [    0.513764]  show_stack from dump_stack_lvl+0x54/0x68
> [    0.518834]  dump_stack_lvl from __warn+0x7c/0x128
> [    0.523639]  __warn from warn_slowpath_fmt+0x184/0x18c
> [    0.527676] Freeing initrd memory: 65536K
> [    0.532788]  warn_slowpath_fmt from __apply_to_page_range+0x380/0x388
> [    0.539242]  __apply_to_page_range from apply_to_page_range+0x1c/0x24
> [    0.545689]  apply_to_page_range from __alloc_from_contiguous+0xc0/0x14c
> [    0.552398]  __alloc_from_contiguous from cma_allocator_alloc+0x34/0x3c
> [    0.559016]  cma_allocator_alloc from arch_dma_alloc+0x11c/0x2ac
> [    0.565025]  arch_dma_alloc from dma_alloc_attrs+0x90/0x2e8
> [    0.570603]  dma_alloc_attrs from dmydev_probe+0x8c/0xe8
> [    0.575919]  dmydev_probe from platform_probe+0x5c/0xb0
> [    0.581152]  platform_probe from really_probe+0xc8/0x2c8
> [    0.586467]  really_probe from __driver_probe_device+0x88/0x19c
> [    0.592387]  __driver_probe_device from driver_probe_device+0x30/0x104
> [    0.598915]  driver_probe_device from __driver_attach+0x90/0x178
> [    0.604921]  __driver_attach from bus_for_each_dev+0x7c/0xcc
> [    0.610582]  bus_for_each_dev from bus_add_driver+0xcc/0x1ec
> [    0.616241]  bus_add_driver from driver_register+0x7c/0x114
> [    0.621814]  driver_register from dmydev_init+0x20/0x28
> [    0.627045]  dmydev_init from do_one_initcall+0x58/0x200
> [    0.632363]  do_one_initcall from kernel_init_freeable+0x1cc/0x228
> [    0.638550]  kernel_init_freeable from kernel_init+0x1c/0x12c
> [    0.644299]  kernel_init from ret_from_fork+0x14/0x28
> [    0.649351] Exception stack(0xe0819fb0 to 0xe0819ff8)
> [    0.654401] 9fa0:                                     00000000
> 00000000 00000000 00000000
> [    0.662575] 9fc0: 00000000 00000000 00000000 00000000 00000000
> 00000000 00000000 00000000
> [    0.670747] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
> [    0.677403] ---[ end trace 0000000000000000 ]---
> [    0.682083] dmydev dmy_device: Allocate dma memory at 0xde000000 dma
> addr 0x1e000000
>
> The reason is that now reserved memory's fixup function
> dma_contiguous_early_fixup is called after the page table is
> initialized. This fixup function increases the dma_mmu_remap count for
> each reserved memory. And the dma_contiguous_remap function depends on
> it to properly set up the reserved memory mmu table entry. Before this
> change, the paging_init function calls dma_contiguous_remap and it
> founds the reserved memory and set it up properly.  After the change,
> this function found there is no reserved memory so skip any
> initialization hence causes the crash later on when my driver tries to
> allocate dma memory from the reserved memory.
>
> My workaround below is to move the dma_contiguous_remap out from the
> paging_init function to the place right after unflatten_device_tree
> where the dma_mmu_remap count is correctly set. But this is not ideal
> solution and would like to see if you have any better way to solve the
> issue.
>
> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
> index a41c93988d2c..535d1bf44529 100644
> --- a/arch/arm/kernel/setup.c
> +++ b/arch/arm/kernel/setup.c
> @@ -1079,6 +1079,7 @@ void __init hyp_mode_check(void)
>   #endif
>   }
>
> +void __init dma_contiguous_remap(void);
>   static void (*__arm_pm_restart)(enum reboot_mode reboot_mode, const
> char *cmd);
>
>   static int arm_restart(struct notifier_block *nb, unsigned long action,
> @@ -1164,6 +1165,7 @@ void __init setup_arch(char **cmdline_p)
>          }
>
>          unflatten_device_tree();
> +       dma_contiguous_remap();
>
>          arm_dt_init_cpu_maps();
>          psci_dt_init();
> diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
> index edb7f56b7c91..1828c8737d70 100644
> --- a/arch/arm/mm/mmu.c
> +++ b/arch/arm/mm/mmu.c
> @@ -1773,7 +1773,6 @@ void __init paging_init(const struct machine_desc
> *mdesc)
>           * be used
>           */
>          map_kernel();
> -       dma_contiguous_remap();
>          early_fixmap_shutdown();
>          devicemaps_init(mdesc);
>          kmap_init();
>
> You can reproduce the issue on any v7 devices by adding these nodes to
> the device tree
> +       reserved-memory {
> +               #address-cells = <1>;
> +               #size-cells = <1>;
> +               ranges;
> +
> +               dt_reserved_cma: dt_reserved_cma {
> +                       compatible = "shared-dma-pool";
> +                       reusable;
> +
> +                       reg = <0x1e000000 0x2000000>;
> +               };
> +       };
> +
> +       dmy_device {
> +               compatible = "xyz,dmydev";
> +               memory-region = <&dt_reserved_cma>;
> +       };
>
> And use this test driver to trigger the call stack:
> diff --git a/drivers/char/dmydev.c b/drivers/char/dmydev.c
> new file mode 100644
> index 000000000000..1dd52ec492eb
> --- /dev/null
> +++ b/drivers/char/dmydev.c
> @@ -0,0 +1,67 @@
> +#include<linux/module.h>
> +#include<linux/kernel.h>
> +#include <linux/platform_device.h>
> +#include <linux/of.h>
> +#include <linux/of_reserved_mem.h>
> +#include <linux/dma-mapping.h>
> +
> +static int dmydev_probe(struct platform_device *pdev)
> +{
> +       void* virt_addr;
> +       dma_addr_t dma_addr;
> +       int ret;
> +
> +       printk(KERN_ALERT "dmydev_probe called\n");
> +
> +       ret = of_reserved_mem_device_init(&pdev->dev);
> +       if (ret && ret != -ENODEV) {
> +               dev_err(&pdev->dev, "Couldn't assign reserve memory to
> device ret = %d\n", ret);
> +                       return ret;
> +       }
> +
> +       virt_addr = dma_alloc_coherent(&pdev->dev, 0x800000, &dma_addr,
> GFP_KERNEL);
> +       if (virt_addr == NULL) {
> +               dev_err(&pdev->dev,"Failed to allocated cma memory\n");
> +               ret = -ENOMEM;
> +       }
> +       else
> +               dev_err(&pdev->dev,"Allocate dma memory at 0x%px dma
> addr %pad\n", virt_addr, &dma_addr);
> +
> +       return ret;
> +}
> +
> +static void dmydev_remove(struct platform_device *pdev)
> +{
> +}
> +
> +static const struct of_device_id dmydev_of_match[] = {
> +       {.compatible = "xyz,dmydev"},
> +       {}
> +};
> +MODULE_DEVICE_TABLE(of, dmydev_of_match);
>
> Let me know if you need more info.
>
> >
> > Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> > ---
> >   drivers/of/fdt.c             |   5 +-
> >   drivers/of/of_private.h      |   3 +-
> >   drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
> >   3 files changed, 122 insertions(+), 54 deletions(-)
> >
> > diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
> > index 4d528c10df3a..d0dbc8183ac4 100644
> > --- a/drivers/of/fdt.c
> > +++ b/drivers/of/fdt.c
> > @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
> >                       break;
> >               memblock_reserve(base, size);
> >       }
> > -
> > -     fdt_init_reserved_mem();
> >   }
> >
> >   /**
> > @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
> >   {
> >       void *fdt = initial_boot_params;
> >
> > +     /* Save the statically-placed regions in the reserved_mem array */
> > +     fdt_scan_reserved_mem_reg_nodes();
> > +
> >       /* Don't use the bootloader provided DTB if ACPI is enabled */
> >       if (!acpi_disabled)
> >               fdt = NULL;
> > diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
> > index 04aa2a91f851..29525c0b9939 100644
> > --- a/drivers/of/of_private.h
> > +++ b/drivers/of/of_private.h
> > @@ -9,6 +9,7 @@
> >    */
> >
> >   #define FDT_ALIGN_SIZE 8
> > +#define MAX_RESERVED_REGIONS    64
> >
> >   /**
> >    * struct alias_prop - Alias property in 'aliases' node
> > @@ -180,7 +181,7 @@ static inline struct device_node *__of_get_dma_parent(const struct device_node *
> >   #endif
> >
> >   int fdt_scan_reserved_mem(void);
> > -void fdt_init_reserved_mem(void);
> > +void __init fdt_scan_reserved_mem_reg_nodes(void);
> >
> >   bool of_fdt_device_is_available(const void *blob, unsigned long node);
> >
> > diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
> > index 46e1c3fbc769..2011174211f9 100644
> > --- a/drivers/of/of_reserved_mem.c
> > +++ b/drivers/of/of_reserved_mem.c
> > @@ -27,7 +27,6 @@
> >
> >   #include "of_private.h"
> >
> > -#define MAX_RESERVED_REGIONS 64
> >   static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
> >   static int reserved_mem_count;
> >
> > @@ -56,6 +55,7 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
> >       return err;
> >   }
> >
> > +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem);
> >   /*
> >    * fdt_reserved_mem_save_node() - save fdt node for second pass initialization
> >    */
> > @@ -74,6 +74,9 @@ static void __init fdt_reserved_mem_save_node(unsigned long node, const char *un
> >       rmem->base = base;
> >       rmem->size = size;
> >
> > +     /* Call the region specific initialization function */
> > +     fdt_init_reserved_mem_node(rmem);
> > +
> >       reserved_mem_count++;
> >       return;
> >   }
> > @@ -106,7 +109,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
> >       phys_addr_t base, size;
> >       int len;
> >       const __be32 *prop;
> > -     int first = 1;
> >       bool nomap;
> >
> >       prop = of_get_flat_dt_prop(node, "reg", &len);
> > @@ -134,10 +136,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
> >                              uname, &base, (unsigned long)(size / SZ_1M));
> >
> >               len -= t_len;
> > -             if (first) {
> > -                     fdt_reserved_mem_save_node(node, uname, base, size);
> > -                     first = 0;
> > -             }
> >       }
> >       return 0;
> >   }
> > @@ -165,12 +163,77 @@ static int __init __reserved_mem_check_root(unsigned long node)
> >       return 0;
> >   }
> >
> > +static void __init __rmem_check_for_overlap(void);
> > +
> > +/**
> > + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg" defined
> > + * reserved memory regions.
> > + *
> > + * This function is used to scan through the DT and store the
> > + * information for the reserved memory regions that are defined using
> > + * the "reg" property. The region node number, name, base address, and
> > + * size are all stored in the reserved_mem array by calling the
> > + * fdt_reserved_mem_save_node() function.
> > + */
> > +void __init fdt_scan_reserved_mem_reg_nodes(void)
> > +{
> > +     int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
> > +     const void *fdt = initial_boot_params;
> > +     phys_addr_t base, size;
> > +     const __be32 *prop;
> > +     int node, child;
> > +     int len;
> > +
> > +     if (!fdt)
> > +             return;
> > +
> > +     node = fdt_path_offset(fdt, "/reserved-memory");
> > +     if (node < 0) {
> > +             pr_info("Reserved memory: No reserved-memory node in the DT\n");
> > +             return;
> > +     }
> > +
> > +     if (__reserved_mem_check_root(node)) {
> > +             pr_err("Reserved memory: unsupported node format, ignoring\n");
> > +             return;
> > +     }
> > +
> > +     fdt_for_each_subnode(child, fdt, node) {
> > +             const char *uname;
> > +
> > +             prop = of_get_flat_dt_prop(child, "reg", &len);
> > +             if (!prop)
> > +                     continue;
> > +             if (!of_fdt_device_is_available(fdt, child))
> > +                     continue;
> > +
> > +             uname = fdt_get_name(fdt, child, NULL);
> > +             if (len && len % t_len != 0) {
> > +                     pr_err("Reserved memory: invalid reg property in '%s', skipping node.\n",
> > +                            uname);
> > +                     continue;
> > +             }
> > +             base = dt_mem_next_cell(dt_root_addr_cells, &prop);
> > +             size = dt_mem_next_cell(dt_root_size_cells, &prop);
> > +
> > +             if (size)
> > +                     fdt_reserved_mem_save_node(child, uname, base, size);
> > +     }
> > +
> > +     /* check for overlapping reserved regions */
> > +     __rmem_check_for_overlap();
> > +}
> > +
> > +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname);
> > +
> >   /*
> >    * fdt_scan_reserved_mem() - scan a single FDT node for reserved memory
> >    */
> >   int __init fdt_scan_reserved_mem(void)
> >   {
> >       int node, child;
> > +     int dynamic_nodes_cnt = 0;
> > +     int dynamic_nodes[MAX_RESERVED_REGIONS];
> >       const void *fdt = initial_boot_params;
> >
> >       node = fdt_path_offset(fdt, "/reserved-memory");
> > @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
> >               uname = fdt_get_name(fdt, child, NULL);
> >
> >               err = __reserved_mem_reserve_reg(child, uname);
> > -             if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL))
> > -                     fdt_reserved_mem_save_node(child, uname, 0, 0);
> > +             /*
> > +              * Save the nodes for the dynamically-placed regions
> > +              * into an array which will be used for allocation right
> > +              * after all the statically-placed regions are reserved
> > +              * or marked as no-map. This is done to avoid dynamically
> > +              * allocating from one of the statically-placed regions.
> > +              */
> > +             if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL)) {
> > +                     dynamic_nodes[dynamic_nodes_cnt] = child;
> > +                     dynamic_nodes_cnt++;
> > +             }
> > +     }
> > +     for (int i = 0; i < dynamic_nodes_cnt; i++) {
> > +             const char *uname;
> > +
> > +             child = dynamic_nodes[i];
> > +             uname = fdt_get_name(fdt, child, NULL);
> > +             __reserved_mem_alloc_size(child, uname);
> >       }
> >       return 0;
> >   }
> > @@ -253,8 +332,7 @@ static int __init __reserved_mem_alloc_in_range(phys_addr_t size,
> >    * __reserved_mem_alloc_size() - allocate reserved memory described by
> >    *  'size', 'alignment'  and 'alloc-ranges' properties.
> >    */
> > -static int __init __reserved_mem_alloc_size(unsigned long node,
> > -     const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
> > +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname)
> >   {
> >       int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
> >       phys_addr_t start = 0, end = 0;
> > @@ -334,9 +412,8 @@ static int __init __reserved_mem_alloc_size(unsigned long node,
> >               return -ENOMEM;
> >       }
> >
> > -     *res_base = base;
> > -     *res_size = size;
> > -
> > +     /* Save region in the reserved_mem array */
> > +     fdt_reserved_mem_save_node(node, uname, base, size);
> >       return 0;
> >   }
> >
> > @@ -425,48 +502,37 @@ static void __init __rmem_check_for_overlap(void)
> >   }
> >
> >   /**
> > - * fdt_init_reserved_mem() - allocate and init all saved reserved memory regions
> > + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
> > + * @rmem: reserved_mem struct of the memory region to be initialized.
> > + *
> > + * This function is used to call the region specific initialization
> > + * function for a reserved memory region.
> >    */
> > -void __init fdt_init_reserved_mem(void)
> > +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem)
> >   {
> > -     int i;
> > -
> > -     /* check for overlapping reserved regions */
> > -     __rmem_check_for_overlap();
> > -
> > -     for (i = 0; i < reserved_mem_count; i++) {
> > -             struct reserved_mem *rmem = &reserved_mem[i];
> > -             unsigned long node = rmem->fdt_node;
> > -             int err = 0;
> > -             bool nomap;
> > +     unsigned long node = rmem->fdt_node;
> > +     int err = 0;
> > +     bool nomap;
> >
> > -             nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
> > +     nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
> >
> > -             if (rmem->size == 0)
> > -                     err = __reserved_mem_alloc_size(node, rmem->name,
> > -                                              &rmem->base, &rmem->size);
> > -             if (err == 0) {
> > -                     err = __reserved_mem_init_node(rmem);
> > -                     if (err != 0 && err != -ENOENT) {
> > -                             pr_info("node %s compatible matching fail\n",
> > -                                     rmem->name);
> > -                             if (nomap)
> > -                                     memblock_clear_nomap(rmem->base, rmem->size);
> > -                             else
> > -                                     memblock_phys_free(rmem->base,
> > -                                                        rmem->size);
> > -                     } else {
> > -                             phys_addr_t end = rmem->base + rmem->size - 1;
> > -                             bool reusable =
> > -                                     (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> > -
> > -                             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> > -                                     &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> > -                                     nomap ? "nomap" : "map",
> > -                                     reusable ? "reusable" : "non-reusable",
> > -                                     rmem->name ? rmem->name : "unknown");
> > -                     }
> > -             }
> > +     err = __reserved_mem_init_node(rmem);
> > +     if (err != 0 && err != -ENOENT) {
> > +             pr_info("node %s compatible matching fail\n", rmem->name);
> > +             if (nomap)
> > +                     memblock_clear_nomap(rmem->base, rmem->size);
> > +             else
> > +                     memblock_phys_free(rmem->base, rmem->size);
> > +     } else {
> > +             phys_addr_t end = rmem->base + rmem->size - 1;
> > +             bool reusable =
> > +                     (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
> > +
> > +             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> > +                     &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
> > +                     nomap ? "nomap" : "map",
> > +                     reusable ? "reusable" : "non-reusable",
> > +                     rmem->name ? rmem->name : "unknown");
> >       }
> >   }
> >
>
Just want to follow up on this issue.  Do you need any further detail
or clarification?
Any ARM memory manage guru on this thread can comment?
Or is my workaround acceptable as a patch?

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4199 bytes --]

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Oreoluwa Babatunde]

On 6/22/2025 6:24 PM, William Zhang wrote:
> On Tue, Jun 17, 2025 at 10:15 AM William Zhang
> <william.zhang@broadcom.com> wrote:
>>
>> Hi Oreoluwa,
>>
>> On 10/8/2024 3:06 PM, Oreoluwa Babatunde wrote:
>>> Reserved memory regions defined in the devicetree can be broken up into
>>> two groups:
>>> i) Statically-placed reserved memory regions
>>> i.e. regions defined with a static start address and size using the
>>>       "reg" property.
>>> ii) Dynamically-placed reserved memory regions.
>>> i.e. regions defined by specifying an address range where they can be
>>>       placed in memory using the "alloc_ranges" and "size" properties.
>>>
>>> These regions are processed and set aside at boot time.
>>> This is done in two stages as seen below:
>>>
>>> Stage 1:
>>> At this stage, fdt_scan_reserved_mem() scans through the child nodes of
>>> the reserved_memory node using the flattened devicetree and does the
>>> following:
>>>
>>> 1) If the node represents a statically-placed reserved memory region,
>>>     i.e. if it is defined using the "reg" property:
>>>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>     - Add the information for that region into the reserved_mem array
>>>       using fdt_reserved_mem_save_node().
>>>       i.e. fdt_reserved_mem_save_node(node, name, base, size).
>>>
>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>     i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>     - Add the information for that region to the reserved_mem array with
>>>       the starting address and size set to 0.
>>>       i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>>>     Note: This region is saved to the array with a starting address of 0
>>>     because a starting address is not yet allocated for it.
>>>
>>> Stage 2:
>>> After iterating through all the reserved memory nodes and storing their
>>> relevant information in the reserved_mem array,fdt_init_reserved_mem() is
>>> called and does the following:
>>>
>>> 1) For statically-placed reserved memory regions:
>>>     - Call the region specific init function using
>>>       __reserved_mem_init_node().
>>> 2) For dynamically-placed reserved memory regions:
>>>     - Call __reserved_mem_alloc_size() which is used to allocate memory
>>>       for each of these regions, and mark them as nomap if they have the
>>>       nomap property specified in the DT.
>>>     - Call the region specific init function.
>>>
>>> The current size of the resvered_mem array is 64 as is defined by
>>> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
>>> how many reserved memory regions can be specified on a system.
>>> As systems continue to grow more and more complex, the number of
>>> reserved memory regions needed are also growing and are starting to hit
>>> this 64 count limit, hence the need to make the reserved_mem array
>>> dynamically sized (i.e. dynamically allocating memory for the
>>> reserved_mem array using membock_alloc_*).
>>>
>>> On architectures such as arm64, memory allocated using memblock is
>>> writable only after the page tables have been setup. This means that if
>>> the reserved_mem array is going to be dynamically allocated, it needs to
>>> happen after the page tables have been setup, not before.
>>>
>>> Since the reserved memory regions are currently being processed and
>>> added to the array before the page tables are setup, there is a need to
>>> change the order in which some of the processing is done to allow for
>>> the reserved_mem array to be dynamically sized.
>>>
>>> It is possible to process the statically-placed reserved memory regions
>>> without needing to store them in the reserved_mem array until after the
>>> page tables have been setup because all the information stored in the
>>> array is readily available in the devicetree and can be referenced at
>>> any time.
>>> Dynamically-placed reserved memory regions on the other hand get
>>> assigned a start address only at runtime, and hence need a place to be
>>> stored once they are allocated since there is no other referrence to the
>>> start address for these regions.
>>>
>>> Hence this patch changes the processing order of the reserved memory
>>> regions in the following ways:
>>>
>>> Step 1:
>>> fdt_scan_reserved_mem() scans through the child nodes of
>>> the reserved_memory node using the flattened devicetree and does the
>>> following:
>>>
>>> 1) If the node represents a statically-placed reserved memory region,
>>>     i.e. if it is defined using the "reg" property:
>>>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>
>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>     i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>     - Call __reserved_mem_alloc_size() which will:
>>>       i) Allocate memory for the reserved region and call
>>>       memblock_mark_nomap() as needed.
>>>       ii) Call the region specific initialization function using
>>>       fdt_init_reserved_mem_node().
>>>       iii) Save the region information in the reserved_mem array using
>>>       fdt_reserved_mem_save_node().
>>>
>>> Step 2:
>>> 1) This stage of the reserved memory processing is now only used to add
>>>     the statically-placed reserved memory regions into the reserved_mem
>>>     array using fdt_scan_reserved_mem_reg_nodes(), as well as call their
>>>     region specific initialization functions.
>>>
>>> 2) This step has also been moved to be after the page tables are
>>>     setup. Moving this will allow us to replace the reserved_mem
>>>     array with a dynamically sized array before storing the rest of
>>>     these regions.
>> I am running into a call trace with this order change on armv7 chip when
>> I tried to allocate dma coherent memory from the device reserved memory.
>> The issue does not happen on armv8 chips.
>>
>> [    0.000000] Reserved memory: created CMA memory pool at 0x1e000000,
>> size 32 MiB
>> [    0.000000] OF: reserved mem: initialized node dt_reserved_cma,
>> compatible id shared-dma-pool
>> [    0.000000] OF: reserved mem: 0x1e000000..0x1fffffff (32768 KiB) map
>> reusable dt_reserved_cma
>> ....
>>
>> [    0.445322] ------------[ cut here ]------------
>> [    0.445353] WARNING: CPU: 0 PID: 1 at mm/memory.c:3069
>> __apply_to_page_range+0x380/0x388
>> [    0.488911] Modules linked in:
>> [    0.492027] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted
>> 6.16.0-rc1-g27605c8c0f69-dirty #3 NONE
>> [    0.501174] Hardware name: Generic DT based system
>> [    0.505965] Call trace:
>> [    0.505985]  unwind_backtrace from show_stack+0x10/0x14
>> [    0.513764]  show_stack from dump_stack_lvl+0x54/0x68
>> [    0.518834]  dump_stack_lvl from __warn+0x7c/0x128
>> [    0.523639]  __warn from warn_slowpath_fmt+0x184/0x18c
>> [    0.527676] Freeing initrd memory: 65536K
>> [    0.532788]  warn_slowpath_fmt from __apply_to_page_range+0x380/0x388
>> [    0.539242]  __apply_to_page_range from apply_to_page_range+0x1c/0x24
>> [    0.545689]  apply_to_page_range from __alloc_from_contiguous+0xc0/0x14c
>> [    0.552398]  __alloc_from_contiguous from cma_allocator_alloc+0x34/0x3c
>> [    0.559016]  cma_allocator_alloc from arch_dma_alloc+0x11c/0x2ac
>> [    0.565025]  arch_dma_alloc from dma_alloc_attrs+0x90/0x2e8
>> [    0.570603]  dma_alloc_attrs from dmydev_probe+0x8c/0xe8
>> [    0.575919]  dmydev_probe from platform_probe+0x5c/0xb0
>> [    0.581152]  platform_probe from really_probe+0xc8/0x2c8
>> [    0.586467]  really_probe from __driver_probe_device+0x88/0x19c
>> [    0.592387]  __driver_probe_device from driver_probe_device+0x30/0x104
>> [    0.598915]  driver_probe_device from __driver_attach+0x90/0x178
>> [    0.604921]  __driver_attach from bus_for_each_dev+0x7c/0xcc
>> [    0.610582]  bus_for_each_dev from bus_add_driver+0xcc/0x1ec
>> [    0.616241]  bus_add_driver from driver_register+0x7c/0x114
>> [    0.621814]  driver_register from dmydev_init+0x20/0x28
>> [    0.627045]  dmydev_init from do_one_initcall+0x58/0x200
>> [    0.632363]  do_one_initcall from kernel_init_freeable+0x1cc/0x228
>> [    0.638550]  kernel_init_freeable from kernel_init+0x1c/0x12c
>> [    0.644299]  kernel_init from ret_from_fork+0x14/0x28
>> [    0.649351] Exception stack(0xe0819fb0 to 0xe0819ff8)
>> [    0.654401] 9fa0:                                     00000000
>> 00000000 00000000 00000000
>> [    0.662575] 9fc0: 00000000 00000000 00000000 00000000 00000000
>> 00000000 00000000 00000000
>> [    0.670747] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000
>> [    0.677403] ---[ end trace 0000000000000000 ]---
>> [    0.682083] dmydev dmy_device: Allocate dma memory at 0xde000000 dma
>> addr 0x1e000000
>>
>> The reason is that now reserved memory's fixup function
>> dma_contiguous_early_fixup is called after the page table is
>> initialized. This fixup function increases the dma_mmu_remap count for
>> each reserved memory. And the dma_contiguous_remap function depends on
>> it to properly set up the reserved memory mmu table entry. Before this
>> change, the paging_init function calls dma_contiguous_remap and it
>> founds the reserved memory and set it up properly.  After the change,
>> this function found there is no reserved memory so skip any
>> initialization hence causes the crash later on when my driver tries to
>> allocate dma memory from the reserved memory.
>>
>> My workaround below is to move the dma_contiguous_remap out from the
>> paging_init function to the place right after unflatten_device_tree
>> where the dma_mmu_remap count is correctly set. But this is not ideal
>> solution and would like to see if you have any better way to solve the
>> issue.
>>
>> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
>> index a41c93988d2c..535d1bf44529 100644
>> --- a/arch/arm/kernel/setup.c
>> +++ b/arch/arm/kernel/setup.c
>> @@ -1079,6 +1079,7 @@ void __init hyp_mode_check(void)
>>   #endif
>>   }
>>
>> +void __init dma_contiguous_remap(void);
>>   static void (*__arm_pm_restart)(enum reboot_mode reboot_mode, const
>> char *cmd);
>>
>>   static int arm_restart(struct notifier_block *nb, unsigned long action,
>> @@ -1164,6 +1165,7 @@ void __init setup_arch(char **cmdline_p)
>>          }
>>
>>          unflatten_device_tree();
>> +       dma_contiguous_remap();
>>
>>          arm_dt_init_cpu_maps();
>>          psci_dt_init();
>> diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
>> index edb7f56b7c91..1828c8737d70 100644
>> --- a/arch/arm/mm/mmu.c
>> +++ b/arch/arm/mm/mmu.c
>> @@ -1773,7 +1773,6 @@ void __init paging_init(const struct machine_desc
>> *mdesc)
>>           * be used
>>           */
>>          map_kernel();
>> -       dma_contiguous_remap();
>>          early_fixmap_shutdown();
>>          devicemaps_init(mdesc);
>>          kmap_init();
>>
>> You can reproduce the issue on any v7 devices by adding these nodes to
>> the device tree
>> +       reserved-memory {
>> +               #address-cells = <1>;
>> +               #size-cells = <1>;
>> +               ranges;
>> +
>> +               dt_reserved_cma: dt_reserved_cma {
>> +                       compatible = "shared-dma-pool";
>> +                       reusable;
>> +
>> +                       reg = <0x1e000000 0x2000000>;
>> +               };
>> +       };
>> +
>> +       dmy_device {
>> +               compatible = "xyz,dmydev";
>> +               memory-region = <&dt_reserved_cma>;
>> +       };
>>
>> And use this test driver to trigger the call stack:
>> diff --git a/drivers/char/dmydev.c b/drivers/char/dmydev.c
>> new file mode 100644
>> index 000000000000..1dd52ec492eb
>> --- /dev/null
>> +++ b/drivers/char/dmydev.c
>> @@ -0,0 +1,67 @@
>> +#include<linux/module.h>
>> +#include<linux/kernel.h>
>> +#include <linux/platform_device.h>
>> +#include <linux/of.h>
>> +#include <linux/of_reserved_mem.h>
>> +#include <linux/dma-mapping.h>
>> +
>> +static int dmydev_probe(struct platform_device *pdev)
>> +{
>> +       void* virt_addr;
>> +       dma_addr_t dma_addr;
>> +       int ret;
>> +
>> +       printk(KERN_ALERT "dmydev_probe called\n");
>> +
>> +       ret = of_reserved_mem_device_init(&pdev->dev);
>> +       if (ret && ret != -ENODEV) {
>> +               dev_err(&pdev->dev, "Couldn't assign reserve memory to
>> device ret = %d\n", ret);
>> +                       return ret;
>> +       }
>> +
>> +       virt_addr = dma_alloc_coherent(&pdev->dev, 0x800000, &dma_addr,
>> GFP_KERNEL);
>> +       if (virt_addr == NULL) {
>> +               dev_err(&pdev->dev,"Failed to allocated cma memory\n");
>> +               ret = -ENOMEM;
>> +       }
>> +       else
>> +               dev_err(&pdev->dev,"Allocate dma memory at 0x%px dma
>> addr %pad\n", virt_addr, &dma_addr);
>> +
>> +       return ret;
>> +}
>> +
>> +static void dmydev_remove(struct platform_device *pdev)
>> +{
>> +}
>> +
>> +static const struct of_device_id dmydev_of_match[] = {
>> +       {.compatible = "xyz,dmydev"},
>> +       {}
>> +};
>> +MODULE_DEVICE_TABLE(of, dmydev_of_match);
>>
>> Let me know if you need more info.
>>
>>>
>>> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>>> ---
>>>   drivers/of/fdt.c             |   5 +-
>>>   drivers/of/of_private.h      |   3 +-
>>>   drivers/of/of_reserved_mem.c | 168 ++++++++++++++++++++++++-----------
>>>   3 files changed, 122 insertions(+), 54 deletions(-)
>>>
>>> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
>>> index 4d528c10df3a..d0dbc8183ac4 100644
>>> --- a/drivers/of/fdt.c
>>> +++ b/drivers/of/fdt.c
>>> @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
>>>                       break;
>>>               memblock_reserve(base, size);
>>>       }
>>> -
>>> -     fdt_init_reserved_mem();
>>>   }
>>>
>>>   /**
>>> @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
>>>   {
>>>       void *fdt = initial_boot_params;
>>>
>>> +     /* Save the statically-placed regions in the reserved_mem array */
>>> +     fdt_scan_reserved_mem_reg_nodes();
>>> +
>>>       /* Don't use the bootloader provided DTB if ACPI is enabled */
>>>       if (!acpi_disabled)
>>>               fdt = NULL;
>>> diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
>>> index 04aa2a91f851..29525c0b9939 100644
>>> --- a/drivers/of/of_private.h
>>> +++ b/drivers/of/of_private.h
>>> @@ -9,6 +9,7 @@
>>>    */
>>>
>>>   #define FDT_ALIGN_SIZE 8
>>> +#define MAX_RESERVED_REGIONS    64
>>>
>>>   /**
>>>    * struct alias_prop - Alias property in 'aliases' node
>>> @@ -180,7 +181,7 @@ static inline struct device_node *__of_get_dma_parent(const struct device_node *
>>>   #endif
>>>
>>>   int fdt_scan_reserved_mem(void);
>>> -void fdt_init_reserved_mem(void);
>>> +void __init fdt_scan_reserved_mem_reg_nodes(void);
>>>
>>>   bool of_fdt_device_is_available(const void *blob, unsigned long node);
>>>
>>> diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
>>> index 46e1c3fbc769..2011174211f9 100644
>>> --- a/drivers/of/of_reserved_mem.c
>>> +++ b/drivers/of/of_reserved_mem.c
>>> @@ -27,7 +27,6 @@
>>>
>>>   #include "of_private.h"
>>>
>>> -#define MAX_RESERVED_REGIONS 64
>>>   static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
>>>   static int reserved_mem_count;
>>>
>>> @@ -56,6 +55,7 @@ static int __init early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
>>>       return err;
>>>   }
>>>
>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem);
>>>   /*
>>>    * fdt_reserved_mem_save_node() - save fdt node for second pass initialization
>>>    */
>>> @@ -74,6 +74,9 @@ static void __init fdt_reserved_mem_save_node(unsigned long node, const char *un
>>>       rmem->base = base;
>>>       rmem->size = size;
>>>
>>> +     /* Call the region specific initialization function */
>>> +     fdt_init_reserved_mem_node(rmem);
>>> +
>>>       reserved_mem_count++;
>>>       return;
>>>   }
>>> @@ -106,7 +109,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
>>>       phys_addr_t base, size;
>>>       int len;
>>>       const __be32 *prop;
>>> -     int first = 1;
>>>       bool nomap;
>>>
>>>       prop = of_get_flat_dt_prop(node, "reg", &len);
>>> @@ -134,10 +136,6 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
>>>                              uname, &base, (unsigned long)(size / SZ_1M));
>>>
>>>               len -= t_len;
>>> -             if (first) {
>>> -                     fdt_reserved_mem_save_node(node, uname, base, size);
>>> -                     first = 0;
>>> -             }
>>>       }
>>>       return 0;
>>>   }
>>> @@ -165,12 +163,77 @@ static int __init __reserved_mem_check_root(unsigned long node)
>>>       return 0;
>>>   }
>>>
>>> +static void __init __rmem_check_for_overlap(void);
>>> +
>>> +/**
>>> + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg" defined
>>> + * reserved memory regions.
>>> + *
>>> + * This function is used to scan through the DT and store the
>>> + * information for the reserved memory regions that are defined using
>>> + * the "reg" property. The region node number, name, base address, and
>>> + * size are all stored in the reserved_mem array by calling the
>>> + * fdt_reserved_mem_save_node() function.
>>> + */
>>> +void __init fdt_scan_reserved_mem_reg_nodes(void)
>>> +{
>>> +     int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
>>> +     const void *fdt = initial_boot_params;
>>> +     phys_addr_t base, size;
>>> +     const __be32 *prop;
>>> +     int node, child;
>>> +     int len;
>>> +
>>> +     if (!fdt)
>>> +             return;
>>> +
>>> +     node = fdt_path_offset(fdt, "/reserved-memory");
>>> +     if (node < 0) {
>>> +             pr_info("Reserved memory: No reserved-memory node in the DT\n");
>>> +             return;
>>> +     }
>>> +
>>> +     if (__reserved_mem_check_root(node)) {
>>> +             pr_err("Reserved memory: unsupported node format, ignoring\n");
>>> +             return;
>>> +     }
>>> +
>>> +     fdt_for_each_subnode(child, fdt, node) {
>>> +             const char *uname;
>>> +
>>> +             prop = of_get_flat_dt_prop(child, "reg", &len);
>>> +             if (!prop)
>>> +                     continue;
>>> +             if (!of_fdt_device_is_available(fdt, child))
>>> +                     continue;
>>> +
>>> +             uname = fdt_get_name(fdt, child, NULL);
>>> +             if (len && len % t_len != 0) {
>>> +                     pr_err("Reserved memory: invalid reg property in '%s', skipping node.\n",
>>> +                            uname);
>>> +                     continue;
>>> +             }
>>> +             base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>>> +             size = dt_mem_next_cell(dt_root_size_cells, &prop);
>>> +
>>> +             if (size)
>>> +                     fdt_reserved_mem_save_node(child, uname, base, size);
>>> +     }
>>> +
>>> +     /* check for overlapping reserved regions */
>>> +     __rmem_check_for_overlap();
>>> +}
>>> +
>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname);
>>> +
>>>   /*
>>>    * fdt_scan_reserved_mem() - scan a single FDT node for reserved memory
>>>    */
>>>   int __init fdt_scan_reserved_mem(void)
>>>   {
>>>       int node, child;
>>> +     int dynamic_nodes_cnt = 0;
>>> +     int dynamic_nodes[MAX_RESERVED_REGIONS];
>>>       const void *fdt = initial_boot_params;
>>>
>>>       node = fdt_path_offset(fdt, "/reserved-memory");
>>> @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
>>>               uname = fdt_get_name(fdt, child, NULL);
>>>
>>>               err = __reserved_mem_reserve_reg(child, uname);
>>> -             if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL))
>>> -                     fdt_reserved_mem_save_node(child, uname, 0, 0);
>>> +             /*
>>> +              * Save the nodes for the dynamically-placed regions
>>> +              * into an array which will be used for allocation right
>>> +              * after all the statically-placed regions are reserved
>>> +              * or marked as no-map. This is done to avoid dynamically
>>> +              * allocating from one of the statically-placed regions.
>>> +              */
>>> +             if (err == -ENOENT && of_get_flat_dt_prop(child, "size", NULL)) {
>>> +                     dynamic_nodes[dynamic_nodes_cnt] = child;
>>> +                     dynamic_nodes_cnt++;
>>> +             }
>>> +     }
>>> +     for (int i = 0; i < dynamic_nodes_cnt; i++) {
>>> +             const char *uname;
>>> +
>>> +             child = dynamic_nodes[i];
>>> +             uname = fdt_get_name(fdt, child, NULL);
>>> +             __reserved_mem_alloc_size(child, uname);
>>>       }
>>>       return 0;
>>>   }
>>> @@ -253,8 +332,7 @@ static int __init __reserved_mem_alloc_in_range(phys_addr_t size,
>>>    * __reserved_mem_alloc_size() - allocate reserved memory described by
>>>    *  'size', 'alignment'  and 'alloc-ranges' properties.
>>>    */
>>> -static int __init __reserved_mem_alloc_size(unsigned long node,
>>> -     const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const char *uname)
>>>   {
>>>       int t_len = (dt_root_addr_cells + dt_root_size_cells) * sizeof(__be32);
>>>       phys_addr_t start = 0, end = 0;
>>> @@ -334,9 +412,8 @@ static int __init __reserved_mem_alloc_size(unsigned long node,
>>>               return -ENOMEM;
>>>       }
>>>
>>> -     *res_base = base;
>>> -     *res_size = size;
>>> -
>>> +     /* Save region in the reserved_mem array */
>>> +     fdt_reserved_mem_save_node(node, uname, base, size);
>>>       return 0;
>>>   }
>>>
>>> @@ -425,48 +502,37 @@ static void __init __rmem_check_for_overlap(void)
>>>   }
>>>
>>>   /**
>>> - * fdt_init_reserved_mem() - allocate and init all saved reserved memory regions
>>> + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
>>> + * @rmem: reserved_mem struct of the memory region to be initialized.
>>> + *
>>> + * This function is used to call the region specific initialization
>>> + * function for a reserved memory region.
>>>    */
>>> -void __init fdt_init_reserved_mem(void)
>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem *rmem)
>>>   {
>>> -     int i;
>>> -
>>> -     /* check for overlapping reserved regions */
>>> -     __rmem_check_for_overlap();
>>> -
>>> -     for (i = 0; i < reserved_mem_count; i++) {
>>> -             struct reserved_mem *rmem = &reserved_mem[i];
>>> -             unsigned long node = rmem->fdt_node;
>>> -             int err = 0;
>>> -             bool nomap;
>>> +     unsigned long node = rmem->fdt_node;
>>> +     int err = 0;
>>> +     bool nomap;
>>>
>>> -             nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
>>> +     nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
>>>
>>> -             if (rmem->size == 0)
>>> -                     err = __reserved_mem_alloc_size(node, rmem->name,
>>> -                                              &rmem->base, &rmem->size);
>>> -             if (err == 0) {
>>> -                     err = __reserved_mem_init_node(rmem);
>>> -                     if (err != 0 && err != -ENOENT) {
>>> -                             pr_info("node %s compatible matching fail\n",
>>> -                                     rmem->name);
>>> -                             if (nomap)
>>> -                                     memblock_clear_nomap(rmem->base, rmem->size);
>>> -                             else
>>> -                                     memblock_phys_free(rmem->base,
>>> -                                                        rmem->size);
>>> -                     } else {
>>> -                             phys_addr_t end = rmem->base + rmem->size - 1;
>>> -                             bool reusable =
>>> -                                     (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
>>> -
>>> -                             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>> -                                     &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
>>> -                                     nomap ? "nomap" : "map",
>>> -                                     reusable ? "reusable" : "non-reusable",
>>> -                                     rmem->name ? rmem->name : "unknown");
>>> -                     }
>>> -             }
>>> +     err = __reserved_mem_init_node(rmem);
>>> +     if (err != 0 && err != -ENOENT) {
>>> +             pr_info("node %s compatible matching fail\n", rmem->name);
>>> +             if (nomap)
>>> +                     memblock_clear_nomap(rmem->base, rmem->size);
>>> +             else
>>> +                     memblock_phys_free(rmem->base, rmem->size);
>>> +     } else {
>>> +             phys_addr_t end = rmem->base + rmem->size - 1;
>>> +             bool reusable =
>>> +                     (of_get_flat_dt_prop(node, "reusable", NULL)) != NULL;
>>> +
>>> +             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>> +                     &rmem->base, &end, (unsigned long)(rmem->size / SZ_1K),
>>> +                     nomap ? "nomap" : "map",
>>> +                     reusable ? "reusable" : "non-reusable",
>>> +                     rmem->name ? rmem->name : "unknown");
>>>       }
>>>   }
>>>
>>
> Just want to follow up on this issue.  Do you need any further detail
> or clarification?
> Any ARM memory manage guru on this thread can comment?
> Or is my workaround acceptable as a patch?


Hi William,

Sorry about the delay in getting back to you.

Instead of moving dma_contiguous_remap(), I suggest moving  dma_contiguous_early_fixup()
to the function that parses the reserved regions so that it is done before paging_init.

Here is what that could look like. Can you please give this a try?

diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
index 77016c0cc296..132d2c66cafc 100644
--- a/drivers/of/of_reserved_mem.c
+++ b/drivers/of/of_reserved_mem.c
@@ -25,6 +25,7 @@
 #include <linux/memblock.h>
 #include <linux/kmemleak.h>
 #include <linux/cma.h>
+#include <linux/dma-map-ops.h>

 #include "of_private.h"

@@ -175,13 +176,17 @@ static int __init __reserved_mem_reserve_reg(unsigned long node,
                base = dt_mem_next_cell(dt_root_addr_cells, &prop);
                size = dt_mem_next_cell(dt_root_size_cells, &prop);

-               if (size &&
-                   early_init_dt_reserve_memory(base, size, nomap) == 0)
+               if (size && early_init_dt_reserve_memory(base, size, nomap) == 0) {
+                       /* Architecture specific contiguous memory fixup. */
+                       if (of_flat_dt_is_compatible(node, "shared-dma-pool"))
+                               dma_contiguous_early_fixup(base, size);
+
                        pr_debug("Reserved memory: reserved region for node '%s': base %pa, size %lu MiB\n",
                                uname, &base, (unsigned long)(size / SZ_1M));
-               else
+               } else {
                        pr_err("Reserved memory: failed to reserve memory for node '%s': base %pa, size %lu MiB\n",
                               uname, &base, (unsigned long)(size / SZ_1M));
+               }

                len -= t_len;
        }
@@ -472,6 +477,9 @@ static int __init __reserved_mem_alloc_size(unsigned long node, const char *unam
                       uname, (unsigned long)(size / SZ_1M));
                return -ENOMEM;
        }
+       /* Architecture specific contiguous memory fixup. */
+       if (of_flat_dt_is_compatible(node, "shared-dma-pool"))
+               dma_contiguous_early_fixup(base, size);
        /* Save region in the reserved_mem array */
        fdt_reserved_mem_save_node(node, uname, base, size);
diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
index 8df0dfaaca18..9e5d63efe7c5 100644
--- a/kernel/dma/contiguous.c
+++ b/kernel/dma/contiguous.c
@@ -480,8 +480,6 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem)
                pr_err("Reserved memory: unable to setup CMA region\n");
                return err;
        }
-       /* Architecture specific contiguous memory fixup. */
-       dma_contiguous_early_fixup(rmem->base, rmem->size);

        if (default_cma)
                dma_contiguous_default_area = cma;

Regards,
Oreoluwa

RE: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[William Zhang]

[-- Attachment #1: Type: text/plain, Size: 31739 bytes --]

> -----Original Message-----
> From: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> Sent: Friday, June 27, 2025 11:02 AM
> To: William Zhang <william.zhang@broadcom.com>; robh@kernel.org
> Cc: aisheng.dong@nxp.com; andy@black.fi.intel.com;
> catalin.marinas@arm.com; devicetree@vger.kernel.org; hch@lst.de;
> iommu@lists.linux.dev; kernel@quicinc.com; klarasmodin@gmail.com; linux-
> kernel@vger.kernel.org; m.szyprowski@samsung.com;
> quic_ninanaik@quicinc.com; robin.murphy@arm.com; saravanak@google.com;
> will@kernel.org; oreoluwa.babatunde@oss.qualcomm.com
> Subject: Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved
> memory regions are processed
>
>
>
> On 6/22/2025 6:24 PM, William Zhang wrote:
> > On Tue, Jun 17, 2025 at 10:15 AM William Zhang
> > <william.zhang@broadcom.com> wrote:
> >>
> >> Hi Oreoluwa,
> >>
> >> On 10/8/2024 3:06 PM, Oreoluwa Babatunde wrote:
> >>> Reserved memory regions defined in the devicetree can be broken up
> >>> into
> >>> two groups:
> >>> i) Statically-placed reserved memory regions
> >>> i.e. regions defined with a static start address and size using the
> >>>       "reg" property.
> >>> ii) Dynamically-placed reserved memory regions.
> >>> i.e. regions defined by specifying an address range where they can be
> >>>       placed in memory using the "alloc_ranges" and "size" properties.
> >>>
> >>> These regions are processed and set aside at boot time.
> >>> This is done in two stages as seen below:
> >>>
> >>> Stage 1:
> >>> At this stage, fdt_scan_reserved_mem() scans through the child nodes
> >>> of
> >>> the reserved_memory node using the flattened devicetree and does the
> >>> following:
> >>>
> >>> 1) If the node represents a statically-placed reserved memory region,
> >>>     i.e. if it is defined using the "reg" property:
> >>>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >>>     - Add the information for that region into the reserved_mem array
> >>>       using fdt_reserved_mem_save_node().
> >>>       i.e. fdt_reserved_mem_save_node(node, name, base, size).
> >>>
> >>> 2) If the node represents a dynamically-placed reserved memory region,
> >>>     i.e. if it is defined using "alloc-ranges" and "size" properties:
> >>>     - Add the information for that region to the reserved_mem array
> >>> with
> >>>       the starting address and size set to 0.
> >>>       i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
> >>>     Note: This region is saved to the array with a starting address of
> >>> 0
> >>>     because a starting address is not yet allocated for it.
> >>>
> >>> Stage 2:
> >>> After iterating through all the reserved memory nodes and storing
> >>> their
> >>> relevant information in the reserved_mem array,fdt_init_reserved_mem()
> >>> is
> >>> called and does the following:
> >>>
> >>> 1) For statically-placed reserved memory regions:
> >>>     - Call the region specific init function using
> >>>       __reserved_mem_init_node().
> >>> 2) For dynamically-placed reserved memory regions:
> >>>     - Call __reserved_mem_alloc_size() which is used to allocate
> >>> memory
> >>>       for each of these regions, and mark them as nomap if they have
> >>> the
> >>>       nomap property specified in the DT.
> >>>     - Call the region specific init function.
> >>>
> >>> The current size of the resvered_mem array is 64 as is defined by
> >>> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
> >>> how many reserved memory regions can be specified on a system.
> >>> As systems continue to grow more and more complex, the number of
> >>> reserved memory regions needed are also growing and are starting to
> >>> hit
> >>> this 64 count limit, hence the need to make the reserved_mem array
> >>> dynamically sized (i.e. dynamically allocating memory for the
> >>> reserved_mem array using membock_alloc_*).
> >>>
> >>> On architectures such as arm64, memory allocated using memblock is
> >>> writable only after the page tables have been setup. This means that
> >>> if
> >>> the reserved_mem array is going to be dynamically allocated, it needs
> >>> to
> >>> happen after the page tables have been setup, not before.
> >>>
> >>> Since the reserved memory regions are currently being processed and
> >>> added to the array before the page tables are setup, there is a need
> >>> to
> >>> change the order in which some of the processing is done to allow for
> >>> the reserved_mem array to be dynamically sized.
> >>>
> >>> It is possible to process the statically-placed reserved memory
> >>> regions
> >>> without needing to store them in the reserved_mem array until after
> >>> the
> >>> page tables have been setup because all the information stored in the
> >>> array is readily available in the devicetree and can be referenced at
> >>> any time.
> >>> Dynamically-placed reserved memory regions on the other hand get
> >>> assigned a start address only at runtime, and hence need a place to be
> >>> stored once they are allocated since there is no other referrence to
> >>> the
> >>> start address for these regions.
> >>>
> >>> Hence this patch changes the processing order of the reserved memory
> >>> regions in the following ways:
> >>>
> >>> Step 1:
> >>> fdt_scan_reserved_mem() scans through the child nodes of
> >>> the reserved_memory node using the flattened devicetree and does the
> >>> following:
> >>>
> >>> 1) If the node represents a statically-placed reserved memory region,
> >>>     i.e. if it is defined using the "reg" property:
> >>>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
> >>>
> >>> 2) If the node represents a dynamically-placed reserved memory region,
> >>>     i.e. if it is defined using "alloc-ranges" and "size" properties:
> >>>     - Call __reserved_mem_alloc_size() which will:
> >>>       i) Allocate memory for the reserved region and call
> >>>       memblock_mark_nomap() as needed.
> >>>       ii) Call the region specific initialization function using
> >>>       fdt_init_reserved_mem_node().
> >>>       iii) Save the region information in the reserved_mem array using
> >>>       fdt_reserved_mem_save_node().
> >>>
> >>> Step 2:
> >>> 1) This stage of the reserved memory processing is now only used to
> >>> add
> >>>     the statically-placed reserved memory regions into the
> >>> reserved_mem
> >>>     array using fdt_scan_reserved_mem_reg_nodes(), as well as call
> >>> their
> >>>     region specific initialization functions.
> >>>
> >>> 2) This step has also been moved to be after the page tables are
> >>>     setup. Moving this will allow us to replace the reserved_mem
> >>>     array with a dynamically sized array before storing the rest of
> >>>     these regions.
> >> I am running into a call trace with this order change on armv7 chip
> >> when
> >> I tried to allocate dma coherent memory from the device reserved
> >> memory.
> >> The issue does not happen on armv8 chips.
> >>
> >> [    0.000000] Reserved memory: created CMA memory pool at 0x1e000000,
> >> size 32 MiB
> >> [    0.000000] OF: reserved mem: initialized node dt_reserved_cma,
> >> compatible id shared-dma-pool
> >> [    0.000000] OF: reserved mem: 0x1e000000..0x1fffffff (32768 KiB) map
> >> reusable dt_reserved_cma
> >> ....
> >>
> >> [    0.445322] ------------[ cut here ]------------
> >> [    0.445353] WARNING: CPU: 0 PID: 1 at mm/memory.c:3069
> >> __apply_to_page_range+0x380/0x388
> >> [    0.488911] Modules linked in:
> >> [    0.492027] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted
> >> 6.16.0-rc1-g27605c8c0f69-dirty #3 NONE
> >> [    0.501174] Hardware name: Generic DT based system
> >> [    0.505965] Call trace:
> >> [    0.505985]  unwind_backtrace from show_stack+0x10/0x14
> >> [    0.513764]  show_stack from dump_stack_lvl+0x54/0x68
> >> [    0.518834]  dump_stack_lvl from __warn+0x7c/0x128
> >> [    0.523639]  __warn from warn_slowpath_fmt+0x184/0x18c
> >> [    0.527676] Freeing initrd memory: 65536K
> >> [    0.532788]  warn_slowpath_fmt from
> __apply_to_page_range+0x380/0x388
> >> [    0.539242]  __apply_to_page_range from
> apply_to_page_range+0x1c/0x24
> >> [    0.545689]  apply_to_page_range from
> __alloc_from_contiguous+0xc0/0x14c
> >> [    0.552398]  __alloc_from_contiguous from
> cma_allocator_alloc+0x34/0x3c
> >> [    0.559016]  cma_allocator_alloc from arch_dma_alloc+0x11c/0x2ac
> >> [    0.565025]  arch_dma_alloc from dma_alloc_attrs+0x90/0x2e8
> >> [    0.570603]  dma_alloc_attrs from dmydev_probe+0x8c/0xe8
> >> [    0.575919]  dmydev_probe from platform_probe+0x5c/0xb0
> >> [    0.581152]  platform_probe from really_probe+0xc8/0x2c8
> >> [    0.586467]  really_probe from __driver_probe_device+0x88/0x19c
> >> [    0.592387]  __driver_probe_device from
> >> driver_probe_device+0x30/0x104
> >> [    0.598915]  driver_probe_device from __driver_attach+0x90/0x178
> >> [    0.604921]  __driver_attach from bus_for_each_dev+0x7c/0xcc
> >> [    0.610582]  bus_for_each_dev from bus_add_driver+0xcc/0x1ec
> >> [    0.616241]  bus_add_driver from driver_register+0x7c/0x114
> >> [    0.621814]  driver_register from dmydev_init+0x20/0x28
> >> [    0.627045]  dmydev_init from do_one_initcall+0x58/0x200
> >> [    0.632363]  do_one_initcall from kernel_init_freeable+0x1cc/0x228
> >> [    0.638550]  kernel_init_freeable from kernel_init+0x1c/0x12c
> >> [    0.644299]  kernel_init from ret_from_fork+0x14/0x28
> >> [    0.649351] Exception stack(0xe0819fb0 to 0xe0819ff8)
> >> [    0.654401] 9fa0:                                     00000000
> >> 00000000 00000000 00000000
> >> [    0.662575] 9fc0: 00000000 00000000 00000000 00000000 00000000
> >> 00000000 00000000 00000000
> >> [    0.670747] 9fe0: 00000000 00000000 00000000 00000000 00000013
> 00000000
> >> [    0.677403] ---[ end trace 0000000000000000 ]---
> >> [    0.682083] dmydev dmy_device: Allocate dma memory at 0xde000000 dma
> >> addr 0x1e000000
> >>
> >> The reason is that now reserved memory's fixup function
> >> dma_contiguous_early_fixup is called after the page table is
> >> initialized. This fixup function increases the dma_mmu_remap count for
> >> each reserved memory. And the dma_contiguous_remap function depends
> on
> >> it to properly set up the reserved memory mmu table entry. Before this
> >> change, the paging_init function calls dma_contiguous_remap and it
> >> founds the reserved memory and set it up properly.  After the change,
> >> this function found there is no reserved memory so skip any
> >> initialization hence causes the crash later on when my driver tries to
> >> allocate dma memory from the reserved memory.
> >>
> >> My workaround below is to move the dma_contiguous_remap out from the
> >> paging_init function to the place right after unflatten_device_tree
> >> where the dma_mmu_remap count is correctly set. But this is not ideal
> >> solution and would like to see if you have any better way to solve the
> >> issue.
> >>
> >> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
> >> index a41c93988d2c..535d1bf44529 100644
> >> --- a/arch/arm/kernel/setup.c
> >> +++ b/arch/arm/kernel/setup.c
> >> @@ -1079,6 +1079,7 @@ void __init hyp_mode_check(void)
> >>   #endif
> >>   }
> >>
> >> +void __init dma_contiguous_remap(void);
> >>   static void (*__arm_pm_restart)(enum reboot_mode reboot_mode, const
> >> char *cmd);
> >>
> >>   static int arm_restart(struct notifier_block *nb, unsigned long
> >> action,
> >> @@ -1164,6 +1165,7 @@ void __init setup_arch(char **cmdline_p)
> >>          }
> >>
> >>          unflatten_device_tree();
> >> +       dma_contiguous_remap();
> >>
> >>          arm_dt_init_cpu_maps();
> >>          psci_dt_init();
> >> diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
> >> index edb7f56b7c91..1828c8737d70 100644
> >> --- a/arch/arm/mm/mmu.c
> >> +++ b/arch/arm/mm/mmu.c
> >> @@ -1773,7 +1773,6 @@ void __init paging_init(const struct machine_desc
> >> *mdesc)
> >>           * be used
> >>           */
> >>          map_kernel();
> >> -       dma_contiguous_remap();
> >>          early_fixmap_shutdown();
> >>          devicemaps_init(mdesc);
> >>          kmap_init();
> >>
> >> You can reproduce the issue on any v7 devices by adding these nodes to
> >> the device tree
> >> +       reserved-memory {
> >> +               #address-cells = <1>;
> >> +               #size-cells = <1>;
> >> +               ranges;
> >> +
> >> +               dt_reserved_cma: dt_reserved_cma {
> >> +                       compatible = "shared-dma-pool";
> >> +                       reusable;
> >> +
> >> +                       reg = <0x1e000000 0x2000000>;
> >> +               };
> >> +       };
> >> +
> >> +       dmy_device {
> >> +               compatible = "xyz,dmydev";
> >> +               memory-region = <&dt_reserved_cma>;
> >> +       };
> >>
> >> And use this test driver to trigger the call stack:
> >> diff --git a/drivers/char/dmydev.c b/drivers/char/dmydev.c
> >> new file mode 100644
> >> index 000000000000..1dd52ec492eb
> >> --- /dev/null
> >> +++ b/drivers/char/dmydev.c
> >> @@ -0,0 +1,67 @@
> >> +#include<linux/module.h>
> >> +#include<linux/kernel.h>
> >> +#include <linux/platform_device.h>
> >> +#include <linux/of.h>
> >> +#include <linux/of_reserved_mem.h>
> >> +#include <linux/dma-mapping.h>
> >> +
> >> +static int dmydev_probe(struct platform_device *pdev)
> >> +{
> >> +       void* virt_addr;
> >> +       dma_addr_t dma_addr;
> >> +       int ret;
> >> +
> >> +       printk(KERN_ALERT "dmydev_probe called\n");
> >> +
> >> +       ret = of_reserved_mem_device_init(&pdev->dev);
> >> +       if (ret && ret != -ENODEV) {
> >> +               dev_err(&pdev->dev, "Couldn't assign reserve memory to
> >> device ret = %d\n", ret);
> >> +                       return ret;
> >> +       }
> >> +
> >> +       virt_addr = dma_alloc_coherent(&pdev->dev, 0x800000, &dma_addr,
> >> GFP_KERNEL);
> >> +       if (virt_addr == NULL) {
> >> +               dev_err(&pdev->dev,"Failed to allocated cma memory\n");
> >> +               ret = -ENOMEM;
> >> +       }
> >> +       else
> >> +               dev_err(&pdev->dev,"Allocate dma memory at 0x%px dma
> >> addr %pad\n", virt_addr, &dma_addr);
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static void dmydev_remove(struct platform_device *pdev)
> >> +{
> >> +}
> >> +
> >> +static const struct of_device_id dmydev_of_match[] = {
> >> +       {.compatible = "xyz,dmydev"},
> >> +       {}
> >> +};
> >> +MODULE_DEVICE_TABLE(of, dmydev_of_match);
> >>
> >> Let me know if you need more info.
> >>
> >>>
> >>> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
> >>> ---
> >>>   drivers/of/fdt.c             |   5 +-
> >>>   drivers/of/of_private.h      |   3 +-
> >>>   drivers/of/of_reserved_mem.c | 168
> >>> ++++++++++++++++++++++++----------
> -
> >>>   3 files changed, 122 insertions(+), 54 deletions(-)
> >>>
> >>> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
> >>> index 4d528c10df3a..d0dbc8183ac4 100644
> >>> --- a/drivers/of/fdt.c
> >>> +++ b/drivers/of/fdt.c
> >>> @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
> >>>                       break;
> >>>               memblock_reserve(base, size);
> >>>       }
> >>> -
> >>> -     fdt_init_reserved_mem();
> >>>   }
> >>>
> >>>   /**
> >>> @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
> >>>   {
> >>>       void *fdt = initial_boot_params;
> >>>
> >>> +     /* Save the statically-placed regions in the reserved_mem array
> >>> */
> >>> +     fdt_scan_reserved_mem_reg_nodes();
> >>> +
> >>>       /* Don't use the bootloader provided DTB if ACPI is enabled */
> >>>       if (!acpi_disabled)
> >>>               fdt = NULL;
> >>> diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
> >>> index 04aa2a91f851..29525c0b9939 100644
> >>> --- a/drivers/of/of_private.h
> >>> +++ b/drivers/of/of_private.h
> >>> @@ -9,6 +9,7 @@
> >>>    */
> >>>
> >>>   #define FDT_ALIGN_SIZE 8
> >>> +#define MAX_RESERVED_REGIONS    64
> >>>
> >>>   /**
> >>>    * struct alias_prop - Alias property in 'aliases' node
> >>> @@ -180,7 +181,7 @@ static inline struct device_node
> *__of_get_dma_parent(const struct device_node *
> >>>   #endif
> >>>
> >>>   int fdt_scan_reserved_mem(void);
> >>> -void fdt_init_reserved_mem(void);
> >>> +void __init fdt_scan_reserved_mem_reg_nodes(void);
> >>>
> >>>   bool of_fdt_device_is_available(const void *blob, unsigned long
> >>> node);
> >>>
> >>> diff --git a/drivers/of/of_reserved_mem.c
> >>> b/drivers/of/of_reserved_mem.c
> >>> index 46e1c3fbc769..2011174211f9 100644
> >>> --- a/drivers/of/of_reserved_mem.c
> >>> +++ b/drivers/of/of_reserved_mem.c
> >>> @@ -27,7 +27,6 @@
> >>>
> >>>   #include "of_private.h"
> >>>
> >>> -#define MAX_RESERVED_REGIONS 64
> >>>   static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
> >>>   static int reserved_mem_count;
> >>>
> >>> @@ -56,6 +55,7 @@ static int __init
> early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
> >>>       return err;
> >>>   }
> >>>
> >>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
> *rmem);
> >>>   /*
> >>>    * fdt_reserved_mem_save_node() - save fdt node for second pass
> initialization
> >>>    */
> >>> @@ -74,6 +74,9 @@ static void __init
> fdt_reserved_mem_save_node(unsigned long node, const char *un
> >>>       rmem->base = base;
> >>>       rmem->size = size;
> >>>
> >>> +     /* Call the region specific initialization function */
> >>> +     fdt_init_reserved_mem_node(rmem);
> >>> +
> >>>       reserved_mem_count++;
> >>>       return;
> >>>   }
> >>> @@ -106,7 +109,6 @@ static int __init
> __reserved_mem_reserve_reg(unsigned long node,
> >>>       phys_addr_t base, size;
> >>>       int len;
> >>>       const __be32 *prop;
> >>> -     int first = 1;
> >>>       bool nomap;
> >>>
> >>>       prop = of_get_flat_dt_prop(node, "reg", &len);
> >>> @@ -134,10 +136,6 @@ static int __init
> __reserved_mem_reserve_reg(unsigned long node,
> >>>                              uname, &base, (unsigned long)(size /
> >>> SZ_1M));
> >>>
> >>>               len -= t_len;
> >>> -             if (first) {
> >>> -                     fdt_reserved_mem_save_node(node, uname, base,
> >>> size);
> >>> -                     first = 0;
> >>> -             }
> >>>       }
> >>>       return 0;
> >>>   }
> >>> @@ -165,12 +163,77 @@ static int __init
> __reserved_mem_check_root(unsigned long node)
> >>>       return 0;
> >>>   }
> >>>
> >>> +static void __init __rmem_check_for_overlap(void);
> >>> +
> >>> +/**
> >>> + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg"
> >>> defined
> >>> + * reserved memory regions.
> >>> + *
> >>> + * This function is used to scan through the DT and store the
> >>> + * information for the reserved memory regions that are defined using
> >>> + * the "reg" property. The region node number, name, base address,
> >>> and
> >>> + * size are all stored in the reserved_mem array by calling the
> >>> + * fdt_reserved_mem_save_node() function.
> >>> + */
> >>> +void __init fdt_scan_reserved_mem_reg_nodes(void)
> >>> +{
> >>> +     int t_len = (dt_root_addr_cells + dt_root_size_cells) *
> >>> sizeof(__be32);
> >>> +     const void *fdt = initial_boot_params;
> >>> +     phys_addr_t base, size;
> >>> +     const __be32 *prop;
> >>> +     int node, child;
> >>> +     int len;
> >>> +
> >>> +     if (!fdt)
> >>> +             return;
> >>> +
> >>> +     node = fdt_path_offset(fdt, "/reserved-memory");
> >>> +     if (node < 0) {
> >>> +             pr_info("Reserved memory: No reserved-memory node in the
> DT\n");
> >>> +             return;
> >>> +     }
> >>> +
> >>> +     if (__reserved_mem_check_root(node)) {
> >>> +             pr_err("Reserved memory: unsupported node format,
> >>> ignoring\n");
> >>> +             return;
> >>> +     }
> >>> +
> >>> +     fdt_for_each_subnode(child, fdt, node) {
> >>> +             const char *uname;
> >>> +
> >>> +             prop = of_get_flat_dt_prop(child, "reg", &len);
> >>> +             if (!prop)
> >>> +                     continue;
> >>> +             if (!of_fdt_device_is_available(fdt, child))
> >>> +                     continue;
> >>> +
> >>> +             uname = fdt_get_name(fdt, child, NULL);
> >>> +             if (len && len % t_len != 0) {
> >>> +                     pr_err("Reserved memory: invalid reg property in
> >>> '%s', skipping
> node.\n",
> >>> +                            uname);
> >>> +                     continue;
> >>> +             }
> >>> +             base = dt_mem_next_cell(dt_root_addr_cells, &prop);
> >>> +             size = dt_mem_next_cell(dt_root_size_cells, &prop);
> >>> +
> >>> +             if (size)
> >>> +                     fdt_reserved_mem_save_node(child, uname, base,
> >>> size);
> >>> +     }
> >>> +
> >>> +     /* check for overlapping reserved regions */
> >>> +     __rmem_check_for_overlap();
> >>> +}
> >>> +
> >>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
> char *uname);
> >>> +
> >>>   /*
> >>>    * fdt_scan_reserved_mem() - scan a single FDT node for reserved
> >>> memory
> >>>    */
> >>>   int __init fdt_scan_reserved_mem(void)
> >>>   {
> >>>       int node, child;
> >>> +     int dynamic_nodes_cnt = 0;
> >>> +     int dynamic_nodes[MAX_RESERVED_REGIONS];
> >>>       const void *fdt = initial_boot_params;
> >>>
> >>>       node = fdt_path_offset(fdt, "/reserved-memory");
> >>> @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
> >>>               uname = fdt_get_name(fdt, child, NULL);
> >>>
> >>>               err = __reserved_mem_reserve_reg(child, uname);
> >>> -             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
> >>> NULL))
> >>> -                     fdt_reserved_mem_save_node(child, uname, 0, 0);
> >>> +             /*
> >>> +              * Save the nodes for the dynamically-placed regions
> >>> +              * into an array which will be used for allocation right
> >>> +              * after all the statically-placed regions are reserved
> >>> +              * or marked as no-map. This is done to avoid
> >>> dynamically
> >>> +              * allocating from one of the statically-placed regions.
> >>> +              */
> >>> +             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
> >>> NULL)) {
> >>> +                     dynamic_nodes[dynamic_nodes_cnt] = child;
> >>> +                     dynamic_nodes_cnt++;
> >>> +             }
> >>> +     }
> >>> +     for (int i = 0; i < dynamic_nodes_cnt; i++) {
> >>> +             const char *uname;
> >>> +
> >>> +             child = dynamic_nodes[i];
> >>> +             uname = fdt_get_name(fdt, child, NULL);
> >>> +             __reserved_mem_alloc_size(child, uname);
> >>>       }
> >>>       return 0;
> >>>   }
> >>> @@ -253,8 +332,7 @@ static int __init
> __reserved_mem_alloc_in_range(phys_addr_t size,
> >>>    * __reserved_mem_alloc_size() - allocate reserved memory described
> >>> by
> >>>    *  'size', 'alignment'  and 'alloc-ranges' properties.
> >>>    */
> >>> -static int __init __reserved_mem_alloc_size(unsigned long node,
> >>> -     const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
> >>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
> char *uname)
> >>>   {
> >>>       int t_len = (dt_root_addr_cells + dt_root_size_cells) *
> >>> sizeof(__be32);
> >>>       phys_addr_t start = 0, end = 0;
> >>> @@ -334,9 +412,8 @@ static int __init
> __reserved_mem_alloc_size(unsigned long node,
> >>>               return -ENOMEM;
> >>>       }
> >>>
> >>> -     *res_base = base;
> >>> -     *res_size = size;
> >>> -
> >>> +     /* Save region in the reserved_mem array */
> >>> +     fdt_reserved_mem_save_node(node, uname, base, size);
> >>>       return 0;
> >>>   }
> >>>
> >>> @@ -425,48 +502,37 @@ static void __init
> __rmem_check_for_overlap(void)
> >>>   }
> >>>
> >>>   /**
> >>> - * fdt_init_reserved_mem() - allocate and init all saved reserved
> >>> memory
> regions
> >>> + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
> >>> + * @rmem: reserved_mem struct of the memory region to be initialized.
> >>> + *
> >>> + * This function is used to call the region specific initialization
> >>> + * function for a reserved memory region.
> >>>    */
> >>> -void __init fdt_init_reserved_mem(void)
> >>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
> *rmem)
> >>>   {
> >>> -     int i;
> >>> -
> >>> -     /* check for overlapping reserved regions */
> >>> -     __rmem_check_for_overlap();
> >>> -
> >>> -     for (i = 0; i < reserved_mem_count; i++) {
> >>> -             struct reserved_mem *rmem = &reserved_mem[i];
> >>> -             unsigned long node = rmem->fdt_node;
> >>> -             int err = 0;
> >>> -             bool nomap;
> >>> +     unsigned long node = rmem->fdt_node;
> >>> +     int err = 0;
> >>> +     bool nomap;
> >>>
> >>> -             nomap = of_get_flat_dt_prop(node, "no-map", NULL) !=
> >>> NULL;
> >>> +     nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
> >>>
> >>> -             if (rmem->size == 0)
> >>> -                     err = __reserved_mem_alloc_size(node,
> >>> rmem->name,
> >>> -                                              &rmem->base,
> >>> &rmem->size);
> >>> -             if (err == 0) {
> >>> -                     err = __reserved_mem_init_node(rmem);
> >>> -                     if (err != 0 && err != -ENOENT) {
> >>> -                             pr_info("node %s compatible matching
> >>> fail\n",
> >>> -                                     rmem->name);
> >>> -                             if (nomap)
> >>> -                                     memblock_clear_nomap(rmem->base,
> >>> rmem->size);
> >>> -                             else
> >>> -                                     memblock_phys_free(rmem->base,
> >>> -                                                        rmem->size);
> >>> -                     } else {
> >>> -                             phys_addr_t end = rmem->base +
> >>> rmem->size - 1;
> >>> -                             bool reusable =
> >>> -                                     (of_get_flat_dt_prop(node,
> >>> "reusable", NULL)) !=
> NULL;
> >>> -
> >>> -                             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> >>> -                                     &rmem->base, &end, (unsigned
> >>> long)(rmem->size /
> SZ_1K),
> >>> -                                     nomap ? "nomap" : "map",
> >>> -                                     reusable ? "reusable" :
> >>> "non-reusable",
> >>> -                                     rmem->name ? rmem->name :
> >>> "unknown");
> >>> -                     }
> >>> -             }
> >>> +     err = __reserved_mem_init_node(rmem);
> >>> +     if (err != 0 && err != -ENOENT) {
> >>> +             pr_info("node %s compatible matching fail\n",
> >>> rmem->name);
> >>> +             if (nomap)
> >>> +                     memblock_clear_nomap(rmem->base, rmem->size);
> >>> +             else
> >>> +                     memblock_phys_free(rmem->base, rmem->size);
> >>> +     } else {
> >>> +             phys_addr_t end = rmem->base + rmem->size - 1;
> >>> +             bool reusable =
> >>> +                     (of_get_flat_dt_prop(node, "reusable", NULL)) !=
> >>> NULL;
> >>> +
> >>> +             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
> >>> +                     &rmem->base, &end, (unsigned long)(rmem->size /
> >>> SZ_1K),
> >>> +                     nomap ? "nomap" : "map",
> >>> +                     reusable ? "reusable" : "non-reusable",
> >>> +                     rmem->name ? rmem->name : "unknown");
> >>>       }
> >>>   }
> >>>
> >>
> > Just want to follow up on this issue.  Do you need any further detail
> > or clarification?
> > Any ARM memory manage guru on this thread can comment?
> > Or is my workaround acceptable as a patch?
>
>
> Hi William,
>
> Sorry about the delay in getting back to you.
>
> Instead of moving dma_contiguous_remap(), I suggest moving
> dma_contiguous_early_fixup()
> to the function that parses the reserved regions so that it is done before
> paging_init.
>
> Here is what that could look like. Can you please give this a try?
>
> diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
> index 77016c0cc296..132d2c66cafc 100644
> --- a/drivers/of/of_reserved_mem.c
> +++ b/drivers/of/of_reserved_mem.c
> @@ -25,6 +25,7 @@
>  #include <linux/memblock.h>
>  #include <linux/kmemleak.h>
>  #include <linux/cma.h>
> +#include <linux/dma-map-ops.h>
>
>  #include "of_private.h"
>
> @@ -175,13 +176,17 @@ static int __init
> __reserved_mem_reserve_reg(unsigned long node,
>                 base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>                 size = dt_mem_next_cell(dt_root_size_cells, &prop);
>
> -               if (size &&
> -                   early_init_dt_reserve_memory(base, size, nomap) == 0)
> +               if (size && early_init_dt_reserve_memory(base, size,
> nomap) == 0) {
> +                       /* Architecture specific contiguous memory fixup.
> */
> +                       if (of_flat_dt_is_compatible(node,
> "shared-dma-pool"))
> +                               dma_contiguous_early_fixup(base, size);
> +
>                         pr_debug("Reserved memory: reserved region for
> node '%s':
> base %pa, size %lu MiB\n",
>                                 uname, &base, (unsigned long)(size /
> SZ_1M));
> -               else
> +               } else {
>                         pr_err("Reserved memory: failed to reserve memory
> for node '%s':
> base %pa, size %lu MiB\n",
>                                uname, &base, (unsigned long)(size /
> SZ_1M));
> +               }
>
>                 len -= t_len;
>         }
> @@ -472,6 +477,9 @@ static int __init __reserved_mem_alloc_size(unsigned
> long node, const char *unam
>                        uname, (unsigned long)(size / SZ_1M));
>                 return -ENOMEM;
>         }
> +       /* Architecture specific contiguous memory fixup. */
> +       if (of_flat_dt_is_compatible(node, "shared-dma-pool"))
> +               dma_contiguous_early_fixup(base, size);
>         /* Save region in the reserved_mem array */
>         fdt_reserved_mem_save_node(node, uname, base, size);
> diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
> index 8df0dfaaca18..9e5d63efe7c5 100644
> --- a/kernel/dma/contiguous.c
> +++ b/kernel/dma/contiguous.c
> @@ -480,8 +480,6 @@ static int __init rmem_cma_setup(struct reserved_mem
> *rmem)
>                 pr_err("Reserved memory: unable to setup CMA region\n");
>                 return err;
>         }
> -       /* Architecture specific contiguous memory fixup. */
> -       dma_contiguous_early_fixup(rmem->base, rmem->size);
>
>         if (default_cma)
>                 dma_contiguous_default_area = cma;
>
> Regards,
> Oreoluwa

Thank you Oreoluwa!  Your patch fixed the issue too and it looks a more
localized and better fix!

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4199 bytes --]

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Oreoluwa Babatunde]

On 6/28/2025 2:04 PM, William Zhang wrote:
>> -----Original Message-----
>> From: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>> Sent: Friday, June 27, 2025 11:02 AM
>> To: William Zhang <william.zhang@broadcom.com>; robh@kernel.org
>> Cc: aisheng.dong@nxp.com; andy@black.fi.intel.com;
>> catalin.marinas@arm.com; devicetree@vger.kernel.org; hch@lst.de;
>> iommu@lists.linux.dev; kernel@quicinc.com; klarasmodin@gmail.com; linux-
>> kernel@vger.kernel.org; m.szyprowski@samsung.com;
>> quic_ninanaik@quicinc.com; robin.murphy@arm.com; saravanak@google.com;
>> will@kernel.org; oreoluwa.babatunde@oss.qualcomm.com
>> Subject: Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved
>> memory regions are processed
>>
>>
>>
>> On 6/22/2025 6:24 PM, William Zhang wrote:
>>> On Tue, Jun 17, 2025 at 10:15 AM William Zhang
>>> <william.zhang@broadcom.com> wrote:
>>>>
>>>> Hi Oreoluwa,
>>>>
>>>> On 10/8/2024 3:06 PM, Oreoluwa Babatunde wrote:
>>>>> Reserved memory regions defined in the devicetree can be broken up
>>>>> into
>>>>> two groups:
>>>>> i) Statically-placed reserved memory regions
>>>>> i.e. regions defined with a static start address and size using the
>>>>>       "reg" property.
>>>>> ii) Dynamically-placed reserved memory regions.
>>>>> i.e. regions defined by specifying an address range where they can be
>>>>>       placed in memory using the "alloc_ranges" and "size" properties.
>>>>>
>>>>> These regions are processed and set aside at boot time.
>>>>> This is done in two stages as seen below:
>>>>>
>>>>> Stage 1:
>>>>> At this stage, fdt_scan_reserved_mem() scans through the child nodes
>>>>> of
>>>>> the reserved_memory node using the flattened devicetree and does the
>>>>> following:
>>>>>
>>>>> 1) If the node represents a statically-placed reserved memory region,
>>>>>     i.e. if it is defined using the "reg" property:
>>>>>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>>>     - Add the information for that region into the reserved_mem array
>>>>>       using fdt_reserved_mem_save_node().
>>>>>       i.e. fdt_reserved_mem_save_node(node, name, base, size).
>>>>>
>>>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>>>     i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>>>     - Add the information for that region to the reserved_mem array
>>>>> with
>>>>>       the starting address and size set to 0.
>>>>>       i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>>>>>     Note: This region is saved to the array with a starting address of
>>>>> 0
>>>>>     because a starting address is not yet allocated for it.
>>>>>
>>>>> Stage 2:
>>>>> After iterating through all the reserved memory nodes and storing
>>>>> their
>>>>> relevant information in the reserved_mem array,fdt_init_reserved_mem()
>>>>> is
>>>>> called and does the following:
>>>>>
>>>>> 1) For statically-placed reserved memory regions:
>>>>>     - Call the region specific init function using
>>>>>       __reserved_mem_init_node().
>>>>> 2) For dynamically-placed reserved memory regions:
>>>>>     - Call __reserved_mem_alloc_size() which is used to allocate
>>>>> memory
>>>>>       for each of these regions, and mark them as nomap if they have
>>>>> the
>>>>>       nomap property specified in the DT.
>>>>>     - Call the region specific init function.
>>>>>
>>>>> The current size of the resvered_mem array is 64 as is defined by
>>>>> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
>>>>> how many reserved memory regions can be specified on a system.
>>>>> As systems continue to grow more and more complex, the number of
>>>>> reserved memory regions needed are also growing and are starting to
>>>>> hit
>>>>> this 64 count limit, hence the need to make the reserved_mem array
>>>>> dynamically sized (i.e. dynamically allocating memory for the
>>>>> reserved_mem array using membock_alloc_*).
>>>>>
>>>>> On architectures such as arm64, memory allocated using memblock is
>>>>> writable only after the page tables have been setup. This means that
>>>>> if
>>>>> the reserved_mem array is going to be dynamically allocated, it needs
>>>>> to
>>>>> happen after the page tables have been setup, not before.
>>>>>
>>>>> Since the reserved memory regions are currently being processed and
>>>>> added to the array before the page tables are setup, there is a need
>>>>> to
>>>>> change the order in which some of the processing is done to allow for
>>>>> the reserved_mem array to be dynamically sized.
>>>>>
>>>>> It is possible to process the statically-placed reserved memory
>>>>> regions
>>>>> without needing to store them in the reserved_mem array until after
>>>>> the
>>>>> page tables have been setup because all the information stored in the
>>>>> array is readily available in the devicetree and can be referenced at
>>>>> any time.
>>>>> Dynamically-placed reserved memory regions on the other hand get
>>>>> assigned a start address only at runtime, and hence need a place to be
>>>>> stored once they are allocated since there is no other referrence to
>>>>> the
>>>>> start address for these regions.
>>>>>
>>>>> Hence this patch changes the processing order of the reserved memory
>>>>> regions in the following ways:
>>>>>
>>>>> Step 1:
>>>>> fdt_scan_reserved_mem() scans through the child nodes of
>>>>> the reserved_memory node using the flattened devicetree and does the
>>>>> following:
>>>>>
>>>>> 1) If the node represents a statically-placed reserved memory region,
>>>>>     i.e. if it is defined using the "reg" property:
>>>>>     - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>>>
>>>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>>>     i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>>>     - Call __reserved_mem_alloc_size() which will:
>>>>>       i) Allocate memory for the reserved region and call
>>>>>       memblock_mark_nomap() as needed.
>>>>>       ii) Call the region specific initialization function using
>>>>>       fdt_init_reserved_mem_node().
>>>>>       iii) Save the region information in the reserved_mem array using
>>>>>       fdt_reserved_mem_save_node().
>>>>>
>>>>> Step 2:
>>>>> 1) This stage of the reserved memory processing is now only used to
>>>>> add
>>>>>     the statically-placed reserved memory regions into the
>>>>> reserved_mem
>>>>>     array using fdt_scan_reserved_mem_reg_nodes(), as well as call
>>>>> their
>>>>>     region specific initialization functions.
>>>>>
>>>>> 2) This step has also been moved to be after the page tables are
>>>>>     setup. Moving this will allow us to replace the reserved_mem
>>>>>     array with a dynamically sized array before storing the rest of
>>>>>     these regions.
>>>> I am running into a call trace with this order change on armv7 chip
>>>> when
>>>> I tried to allocate dma coherent memory from the device reserved
>>>> memory.
>>>> The issue does not happen on armv8 chips.
>>>>
>>>> [    0.000000] Reserved memory: created CMA memory pool at 0x1e000000,
>>>> size 32 MiB
>>>> [    0.000000] OF: reserved mem: initialized node dt_reserved_cma,
>>>> compatible id shared-dma-pool
>>>> [    0.000000] OF: reserved mem: 0x1e000000..0x1fffffff (32768 KiB) map
>>>> reusable dt_reserved_cma
>>>> ....
>>>>
>>>> [    0.445322] ------------[ cut here ]------------
>>>> [    0.445353] WARNING: CPU: 0 PID: 1 at mm/memory.c:3069
>>>> __apply_to_page_range+0x380/0x388
>>>> [    0.488911] Modules linked in:
>>>> [    0.492027] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted
>>>> 6.16.0-rc1-g27605c8c0f69-dirty #3 NONE
>>>> [    0.501174] Hardware name: Generic DT based system
>>>> [    0.505965] Call trace:
>>>> [    0.505985]  unwind_backtrace from show_stack+0x10/0x14
>>>> [    0.513764]  show_stack from dump_stack_lvl+0x54/0x68
>>>> [    0.518834]  dump_stack_lvl from __warn+0x7c/0x128
>>>> [    0.523639]  __warn from warn_slowpath_fmt+0x184/0x18c
>>>> [    0.527676] Freeing initrd memory: 65536K
>>>> [    0.532788]  warn_slowpath_fmt from
>> __apply_to_page_range+0x380/0x388
>>>> [    0.539242]  __apply_to_page_range from
>> apply_to_page_range+0x1c/0x24
>>>> [    0.545689]  apply_to_page_range from
>> __alloc_from_contiguous+0xc0/0x14c
>>>> [    0.552398]  __alloc_from_contiguous from
>> cma_allocator_alloc+0x34/0x3c
>>>> [    0.559016]  cma_allocator_alloc from arch_dma_alloc+0x11c/0x2ac
>>>> [    0.565025]  arch_dma_alloc from dma_alloc_attrs+0x90/0x2e8
>>>> [    0.570603]  dma_alloc_attrs from dmydev_probe+0x8c/0xe8
>>>> [    0.575919]  dmydev_probe from platform_probe+0x5c/0xb0
>>>> [    0.581152]  platform_probe from really_probe+0xc8/0x2c8
>>>> [    0.586467]  really_probe from __driver_probe_device+0x88/0x19c
>>>> [    0.592387]  __driver_probe_device from
>>>> driver_probe_device+0x30/0x104
>>>> [    0.598915]  driver_probe_device from __driver_attach+0x90/0x178
>>>> [    0.604921]  __driver_attach from bus_for_each_dev+0x7c/0xcc
>>>> [    0.610582]  bus_for_each_dev from bus_add_driver+0xcc/0x1ec
>>>> [    0.616241]  bus_add_driver from driver_register+0x7c/0x114
>>>> [    0.621814]  driver_register from dmydev_init+0x20/0x28
>>>> [    0.627045]  dmydev_init from do_one_initcall+0x58/0x200
>>>> [    0.632363]  do_one_initcall from kernel_init_freeable+0x1cc/0x228
>>>> [    0.638550]  kernel_init_freeable from kernel_init+0x1c/0x12c
>>>> [    0.644299]  kernel_init from ret_from_fork+0x14/0x28
>>>> [    0.649351] Exception stack(0xe0819fb0 to 0xe0819ff8)
>>>> [    0.654401] 9fa0:                                     00000000
>>>> 00000000 00000000 00000000
>>>> [    0.662575] 9fc0: 00000000 00000000 00000000 00000000 00000000
>>>> 00000000 00000000 00000000
>>>> [    0.670747] 9fe0: 00000000 00000000 00000000 00000000 00000013
>> 00000000
>>>> [    0.677403] ---[ end trace 0000000000000000 ]---
>>>> [    0.682083] dmydev dmy_device: Allocate dma memory at 0xde000000 dma
>>>> addr 0x1e000000
>>>>
>>>> The reason is that now reserved memory's fixup function
>>>> dma_contiguous_early_fixup is called after the page table is
>>>> initialized. This fixup function increases the dma_mmu_remap count for
>>>> each reserved memory. And the dma_contiguous_remap function depends
>> on
>>>> it to properly set up the reserved memory mmu table entry. Before this
>>>> change, the paging_init function calls dma_contiguous_remap and it
>>>> founds the reserved memory and set it up properly.  After the change,
>>>> this function found there is no reserved memory so skip any
>>>> initialization hence causes the crash later on when my driver tries to
>>>> allocate dma memory from the reserved memory.
>>>>
>>>> My workaround below is to move the dma_contiguous_remap out from the
>>>> paging_init function to the place right after unflatten_device_tree
>>>> where the dma_mmu_remap count is correctly set. But this is not ideal
>>>> solution and would like to see if you have any better way to solve the
>>>> issue.
>>>>
>>>> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
>>>> index a41c93988d2c..535d1bf44529 100644
>>>> --- a/arch/arm/kernel/setup.c
>>>> +++ b/arch/arm/kernel/setup.c
>>>> @@ -1079,6 +1079,7 @@ void __init hyp_mode_check(void)
>>>>   #endif
>>>>   }
>>>>
>>>> +void __init dma_contiguous_remap(void);
>>>>   static void (*__arm_pm_restart)(enum reboot_mode reboot_mode, const
>>>> char *cmd);
>>>>
>>>>   static int arm_restart(struct notifier_block *nb, unsigned long
>>>> action,
>>>> @@ -1164,6 +1165,7 @@ void __init setup_arch(char **cmdline_p)
>>>>          }
>>>>
>>>>          unflatten_device_tree();
>>>> +       dma_contiguous_remap();
>>>>
>>>>          arm_dt_init_cpu_maps();
>>>>          psci_dt_init();
>>>> diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
>>>> index edb7f56b7c91..1828c8737d70 100644
>>>> --- a/arch/arm/mm/mmu.c
>>>> +++ b/arch/arm/mm/mmu.c
>>>> @@ -1773,7 +1773,6 @@ void __init paging_init(const struct machine_desc
>>>> *mdesc)
>>>>           * be used
>>>>           */
>>>>          map_kernel();
>>>> -       dma_contiguous_remap();
>>>>          early_fixmap_shutdown();
>>>>          devicemaps_init(mdesc);
>>>>          kmap_init();
>>>>
>>>> You can reproduce the issue on any v7 devices by adding these nodes to
>>>> the device tree
>>>> +       reserved-memory {
>>>> +               #address-cells = <1>;
>>>> +               #size-cells = <1>;
>>>> +               ranges;
>>>> +
>>>> +               dt_reserved_cma: dt_reserved_cma {
>>>> +                       compatible = "shared-dma-pool";
>>>> +                       reusable;
>>>> +
>>>> +                       reg = <0x1e000000 0x2000000>;
>>>> +               };
>>>> +       };
>>>> +
>>>> +       dmy_device {
>>>> +               compatible = "xyz,dmydev";
>>>> +               memory-region = <&dt_reserved_cma>;
>>>> +       };
>>>>
>>>> And use this test driver to trigger the call stack:
>>>> diff --git a/drivers/char/dmydev.c b/drivers/char/dmydev.c
>>>> new file mode 100644
>>>> index 000000000000..1dd52ec492eb
>>>> --- /dev/null
>>>> +++ b/drivers/char/dmydev.c
>>>> @@ -0,0 +1,67 @@
>>>> +#include<linux/module.h>
>>>> +#include<linux/kernel.h>
>>>> +#include <linux/platform_device.h>
>>>> +#include <linux/of.h>
>>>> +#include <linux/of_reserved_mem.h>
>>>> +#include <linux/dma-mapping.h>
>>>> +
>>>> +static int dmydev_probe(struct platform_device *pdev)
>>>> +{
>>>> +       void* virt_addr;
>>>> +       dma_addr_t dma_addr;
>>>> +       int ret;
>>>> +
>>>> +       printk(KERN_ALERT "dmydev_probe called\n");
>>>> +
>>>> +       ret = of_reserved_mem_device_init(&pdev->dev);
>>>> +       if (ret && ret != -ENODEV) {
>>>> +               dev_err(&pdev->dev, "Couldn't assign reserve memory to
>>>> device ret = %d\n", ret);
>>>> +                       return ret;
>>>> +       }
>>>> +
>>>> +       virt_addr = dma_alloc_coherent(&pdev->dev, 0x800000, &dma_addr,
>>>> GFP_KERNEL);
>>>> +       if (virt_addr == NULL) {
>>>> +               dev_err(&pdev->dev,"Failed to allocated cma memory\n");
>>>> +               ret = -ENOMEM;
>>>> +       }
>>>> +       else
>>>> +               dev_err(&pdev->dev,"Allocate dma memory at 0x%px dma
>>>> addr %pad\n", virt_addr, &dma_addr);
>>>> +
>>>> +       return ret;
>>>> +}
>>>> +
>>>> +static void dmydev_remove(struct platform_device *pdev)
>>>> +{
>>>> +}
>>>> +
>>>> +static const struct of_device_id dmydev_of_match[] = {
>>>> +       {.compatible = "xyz,dmydev"},
>>>> +       {}
>>>> +};
>>>> +MODULE_DEVICE_TABLE(of, dmydev_of_match);
>>>>
>>>> Let me know if you need more info.
>>>>
>>>>>
>>>>> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>>>>> ---
>>>>>   drivers/of/fdt.c             |   5 +-
>>>>>   drivers/of/of_private.h      |   3 +-
>>>>>   drivers/of/of_reserved_mem.c | 168
>>>>> ++++++++++++++++++++++++----------
>> -
>>>>>   3 files changed, 122 insertions(+), 54 deletions(-)
>>>>>
>>>>> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
>>>>> index 4d528c10df3a..d0dbc8183ac4 100644
>>>>> --- a/drivers/of/fdt.c
>>>>> +++ b/drivers/of/fdt.c
>>>>> @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
>>>>>                       break;
>>>>>               memblock_reserve(base, size);
>>>>>       }
>>>>> -
>>>>> -     fdt_init_reserved_mem();
>>>>>   }
>>>>>
>>>>>   /**
>>>>> @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
>>>>>   {
>>>>>       void *fdt = initial_boot_params;
>>>>>
>>>>> +     /* Save the statically-placed regions in the reserved_mem array
>>>>> */
>>>>> +     fdt_scan_reserved_mem_reg_nodes();
>>>>> +
>>>>>       /* Don't use the bootloader provided DTB if ACPI is enabled */
>>>>>       if (!acpi_disabled)
>>>>>               fdt = NULL;
>>>>> diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
>>>>> index 04aa2a91f851..29525c0b9939 100644
>>>>> --- a/drivers/of/of_private.h
>>>>> +++ b/drivers/of/of_private.h
>>>>> @@ -9,6 +9,7 @@
>>>>>    */
>>>>>
>>>>>   #define FDT_ALIGN_SIZE 8
>>>>> +#define MAX_RESERVED_REGIONS    64
>>>>>
>>>>>   /**
>>>>>    * struct alias_prop - Alias property in 'aliases' node
>>>>> @@ -180,7 +181,7 @@ static inline struct device_node
>> *__of_get_dma_parent(const struct device_node *
>>>>>   #endif
>>>>>
>>>>>   int fdt_scan_reserved_mem(void);
>>>>> -void fdt_init_reserved_mem(void);
>>>>> +void __init fdt_scan_reserved_mem_reg_nodes(void);
>>>>>
>>>>>   bool of_fdt_device_is_available(const void *blob, unsigned long
>>>>> node);
>>>>>
>>>>> diff --git a/drivers/of/of_reserved_mem.c
>>>>> b/drivers/of/of_reserved_mem.c
>>>>> index 46e1c3fbc769..2011174211f9 100644
>>>>> --- a/drivers/of/of_reserved_mem.c
>>>>> +++ b/drivers/of/of_reserved_mem.c
>>>>> @@ -27,7 +27,6 @@
>>>>>
>>>>>   #include "of_private.h"
>>>>>
>>>>> -#define MAX_RESERVED_REGIONS 64
>>>>>   static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
>>>>>   static int reserved_mem_count;
>>>>>
>>>>> @@ -56,6 +55,7 @@ static int __init
>> early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
>>>>>       return err;
>>>>>   }
>>>>>
>>>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
>> *rmem);
>>>>>   /*
>>>>>    * fdt_reserved_mem_save_node() - save fdt node for second pass
>> initialization
>>>>>    */
>>>>> @@ -74,6 +74,9 @@ static void __init
>> fdt_reserved_mem_save_node(unsigned long node, const char *un
>>>>>       rmem->base = base;
>>>>>       rmem->size = size;
>>>>>
>>>>> +     /* Call the region specific initialization function */
>>>>> +     fdt_init_reserved_mem_node(rmem);
>>>>> +
>>>>>       reserved_mem_count++;
>>>>>       return;
>>>>>   }
>>>>> @@ -106,7 +109,6 @@ static int __init
>> __reserved_mem_reserve_reg(unsigned long node,
>>>>>       phys_addr_t base, size;
>>>>>       int len;
>>>>>       const __be32 *prop;
>>>>> -     int first = 1;
>>>>>       bool nomap;
>>>>>
>>>>>       prop = of_get_flat_dt_prop(node, "reg", &len);
>>>>> @@ -134,10 +136,6 @@ static int __init
>> __reserved_mem_reserve_reg(unsigned long node,
>>>>>                              uname, &base, (unsigned long)(size /
>>>>> SZ_1M));
>>>>>
>>>>>               len -= t_len;
>>>>> -             if (first) {
>>>>> -                     fdt_reserved_mem_save_node(node, uname, base,
>>>>> size);
>>>>> -                     first = 0;
>>>>> -             }
>>>>>       }
>>>>>       return 0;
>>>>>   }
>>>>> @@ -165,12 +163,77 @@ static int __init
>> __reserved_mem_check_root(unsigned long node)
>>>>>       return 0;
>>>>>   }
>>>>>
>>>>> +static void __init __rmem_check_for_overlap(void);
>>>>> +
>>>>> +/**
>>>>> + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg"
>>>>> defined
>>>>> + * reserved memory regions.
>>>>> + *
>>>>> + * This function is used to scan through the DT and store the
>>>>> + * information for the reserved memory regions that are defined using
>>>>> + * the "reg" property. The region node number, name, base address,
>>>>> and
>>>>> + * size are all stored in the reserved_mem array by calling the
>>>>> + * fdt_reserved_mem_save_node() function.
>>>>> + */
>>>>> +void __init fdt_scan_reserved_mem_reg_nodes(void)
>>>>> +{
>>>>> +     int t_len = (dt_root_addr_cells + dt_root_size_cells) *
>>>>> sizeof(__be32);
>>>>> +     const void *fdt = initial_boot_params;
>>>>> +     phys_addr_t base, size;
>>>>> +     const __be32 *prop;
>>>>> +     int node, child;
>>>>> +     int len;
>>>>> +
>>>>> +     if (!fdt)
>>>>> +             return;
>>>>> +
>>>>> +     node = fdt_path_offset(fdt, "/reserved-memory");
>>>>> +     if (node < 0) {
>>>>> +             pr_info("Reserved memory: No reserved-memory node in the
>> DT\n");
>>>>> +             return;
>>>>> +     }
>>>>> +
>>>>> +     if (__reserved_mem_check_root(node)) {
>>>>> +             pr_err("Reserved memory: unsupported node format,
>>>>> ignoring\n");
>>>>> +             return;
>>>>> +     }
>>>>> +
>>>>> +     fdt_for_each_subnode(child, fdt, node) {
>>>>> +             const char *uname;
>>>>> +
>>>>> +             prop = of_get_flat_dt_prop(child, "reg", &len);
>>>>> +             if (!prop)
>>>>> +                     continue;
>>>>> +             if (!of_fdt_device_is_available(fdt, child))
>>>>> +                     continue;
>>>>> +
>>>>> +             uname = fdt_get_name(fdt, child, NULL);
>>>>> +             if (len && len % t_len != 0) {
>>>>> +                     pr_err("Reserved memory: invalid reg property in
>>>>> '%s', skipping
>> node.\n",
>>>>> +                            uname);
>>>>> +                     continue;
>>>>> +             }
>>>>> +             base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>>>>> +             size = dt_mem_next_cell(dt_root_size_cells, &prop);
>>>>> +
>>>>> +             if (size)
>>>>> +                     fdt_reserved_mem_save_node(child, uname, base,
>>>>> size);
>>>>> +     }
>>>>> +
>>>>> +     /* check for overlapping reserved regions */
>>>>> +     __rmem_check_for_overlap();
>>>>> +}
>>>>> +
>>>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
>> char *uname);
>>>>> +
>>>>>   /*
>>>>>    * fdt_scan_reserved_mem() - scan a single FDT node for reserved
>>>>> memory
>>>>>    */
>>>>>   int __init fdt_scan_reserved_mem(void)
>>>>>   {
>>>>>       int node, child;
>>>>> +     int dynamic_nodes_cnt = 0;
>>>>> +     int dynamic_nodes[MAX_RESERVED_REGIONS];
>>>>>       const void *fdt = initial_boot_params;
>>>>>
>>>>>       node = fdt_path_offset(fdt, "/reserved-memory");
>>>>> @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
>>>>>               uname = fdt_get_name(fdt, child, NULL);
>>>>>
>>>>>               err = __reserved_mem_reserve_reg(child, uname);
>>>>> -             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
>>>>> NULL))
>>>>> -                     fdt_reserved_mem_save_node(child, uname, 0, 0);
>>>>> +             /*
>>>>> +              * Save the nodes for the dynamically-placed regions
>>>>> +              * into an array which will be used for allocation right
>>>>> +              * after all the statically-placed regions are reserved
>>>>> +              * or marked as no-map. This is done to avoid
>>>>> dynamically
>>>>> +              * allocating from one of the statically-placed regions.
>>>>> +              */
>>>>> +             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
>>>>> NULL)) {
>>>>> +                     dynamic_nodes[dynamic_nodes_cnt] = child;
>>>>> +                     dynamic_nodes_cnt++;
>>>>> +             }
>>>>> +     }
>>>>> +     for (int i = 0; i < dynamic_nodes_cnt; i++) {
>>>>> +             const char *uname;
>>>>> +
>>>>> +             child = dynamic_nodes[i];
>>>>> +             uname = fdt_get_name(fdt, child, NULL);
>>>>> +             __reserved_mem_alloc_size(child, uname);
>>>>>       }
>>>>>       return 0;
>>>>>   }
>>>>> @@ -253,8 +332,7 @@ static int __init
>> __reserved_mem_alloc_in_range(phys_addr_t size,
>>>>>    * __reserved_mem_alloc_size() - allocate reserved memory described
>>>>> by
>>>>>    *  'size', 'alignment'  and 'alloc-ranges' properties.
>>>>>    */
>>>>> -static int __init __reserved_mem_alloc_size(unsigned long node,
>>>>> -     const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
>>>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
>> char *uname)
>>>>>   {
>>>>>       int t_len = (dt_root_addr_cells + dt_root_size_cells) *
>>>>> sizeof(__be32);
>>>>>       phys_addr_t start = 0, end = 0;
>>>>> @@ -334,9 +412,8 @@ static int __init
>> __reserved_mem_alloc_size(unsigned long node,
>>>>>               return -ENOMEM;
>>>>>       }
>>>>>
>>>>> -     *res_base = base;
>>>>> -     *res_size = size;
>>>>> -
>>>>> +     /* Save region in the reserved_mem array */
>>>>> +     fdt_reserved_mem_save_node(node, uname, base, size);
>>>>>       return 0;
>>>>>   }
>>>>>
>>>>> @@ -425,48 +502,37 @@ static void __init
>> __rmem_check_for_overlap(void)
>>>>>   }
>>>>>
>>>>>   /**
>>>>> - * fdt_init_reserved_mem() - allocate and init all saved reserved
>>>>> memory
>> regions
>>>>> + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
>>>>> + * @rmem: reserved_mem struct of the memory region to be initialized.
>>>>> + *
>>>>> + * This function is used to call the region specific initialization
>>>>> + * function for a reserved memory region.
>>>>>    */
>>>>> -void __init fdt_init_reserved_mem(void)
>>>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
>> *rmem)
>>>>>   {
>>>>> -     int i;
>>>>> -
>>>>> -     /* check for overlapping reserved regions */
>>>>> -     __rmem_check_for_overlap();
>>>>> -
>>>>> -     for (i = 0; i < reserved_mem_count; i++) {
>>>>> -             struct reserved_mem *rmem = &reserved_mem[i];
>>>>> -             unsigned long node = rmem->fdt_node;
>>>>> -             int err = 0;
>>>>> -             bool nomap;
>>>>> +     unsigned long node = rmem->fdt_node;
>>>>> +     int err = 0;
>>>>> +     bool nomap;
>>>>>
>>>>> -             nomap = of_get_flat_dt_prop(node, "no-map", NULL) !=
>>>>> NULL;
>>>>> +     nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
>>>>>
>>>>> -             if (rmem->size == 0)
>>>>> -                     err = __reserved_mem_alloc_size(node,
>>>>> rmem->name,
>>>>> -                                              &rmem->base,
>>>>> &rmem->size);
>>>>> -             if (err == 0) {
>>>>> -                     err = __reserved_mem_init_node(rmem);
>>>>> -                     if (err != 0 && err != -ENOENT) {
>>>>> -                             pr_info("node %s compatible matching
>>>>> fail\n",
>>>>> -                                     rmem->name);
>>>>> -                             if (nomap)
>>>>> -                                     memblock_clear_nomap(rmem->base,
>>>>> rmem->size);
>>>>> -                             else
>>>>> -                                     memblock_phys_free(rmem->base,
>>>>> -                                                        rmem->size);
>>>>> -                     } else {
>>>>> -                             phys_addr_t end = rmem->base +
>>>>> rmem->size - 1;
>>>>> -                             bool reusable =
>>>>> -                                     (of_get_flat_dt_prop(node,
>>>>> "reusable", NULL)) !=
>> NULL;
>>>>> -
>>>>> -                             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>>>> -                                     &rmem->base, &end, (unsigned
>>>>> long)(rmem->size /
>> SZ_1K),
>>>>> -                                     nomap ? "nomap" : "map",
>>>>> -                                     reusable ? "reusable" :
>>>>> "non-reusable",
>>>>> -                                     rmem->name ? rmem->name :
>>>>> "unknown");
>>>>> -                     }
>>>>> -             }
>>>>> +     err = __reserved_mem_init_node(rmem);
>>>>> +     if (err != 0 && err != -ENOENT) {
>>>>> +             pr_info("node %s compatible matching fail\n",
>>>>> rmem->name);
>>>>> +             if (nomap)
>>>>> +                     memblock_clear_nomap(rmem->base, rmem->size);
>>>>> +             else
>>>>> +                     memblock_phys_free(rmem->base, rmem->size);
>>>>> +     } else {
>>>>> +             phys_addr_t end = rmem->base + rmem->size - 1;
>>>>> +             bool reusable =
>>>>> +                     (of_get_flat_dt_prop(node, "reusable", NULL)) !=
>>>>> NULL;
>>>>> +
>>>>> +             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>>>> +                     &rmem->base, &end, (unsigned long)(rmem->size /
>>>>> SZ_1K),
>>>>> +                     nomap ? "nomap" : "map",
>>>>> +                     reusable ? "reusable" : "non-reusable",
>>>>> +                     rmem->name ? rmem->name : "unknown");
>>>>>       }
>>>>>   }
>>>>>
>>>>
>>> Just want to follow up on this issue.  Do you need any further detail
>>> or clarification?
>>> Any ARM memory manage guru on this thread can comment?
>>> Or is my workaround acceptable as a patch?
>>
>>
>> Hi William,
>>
>> Sorry about the delay in getting back to you.
>>
>> Instead of moving dma_contiguous_remap(), I suggest moving
>> dma_contiguous_early_fixup()
>> to the function that parses the reserved regions so that it is done before
>> paging_init.
>>
>> Here is what that could look like. Can you please give this a try?
>>
>> diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
>> index 77016c0cc296..132d2c66cafc 100644
>> --- a/drivers/of/of_reserved_mem.c
>> +++ b/drivers/of/of_reserved_mem.c
>> @@ -25,6 +25,7 @@
>>  #include <linux/memblock.h>
>>  #include <linux/kmemleak.h>
>>  #include <linux/cma.h>
>> +#include <linux/dma-map-ops.h>
>>
>>  #include "of_private.h"
>>
>> @@ -175,13 +176,17 @@ static int __init
>> __reserved_mem_reserve_reg(unsigned long node,
>>                 base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>>                 size = dt_mem_next_cell(dt_root_size_cells, &prop);
>>
>> -               if (size &&
>> -                   early_init_dt_reserve_memory(base, size, nomap) == 0)
>> +               if (size && early_init_dt_reserve_memory(base, size,
>> nomap) == 0) {
>> +                       /* Architecture specific contiguous memory fixup.
>> */
>> +                       if (of_flat_dt_is_compatible(node,
>> "shared-dma-pool"))
>> +                               dma_contiguous_early_fixup(base, size);
>> +
>>                         pr_debug("Reserved memory: reserved region for
>> node '%s':
>> base %pa, size %lu MiB\n",
>>                                 uname, &base, (unsigned long)(size /
>> SZ_1M));
>> -               else
>> +               } else {
>>                         pr_err("Reserved memory: failed to reserve memory
>> for node '%s':
>> base %pa, size %lu MiB\n",
>>                                uname, &base, (unsigned long)(size /
>> SZ_1M));
>> +               }
>>
>>                 len -= t_len;
>>         }
>> @@ -472,6 +477,9 @@ static int __init __reserved_mem_alloc_size(unsigned
>> long node, const char *unam
>>                        uname, (unsigned long)(size / SZ_1M));
>>                 return -ENOMEM;
>>         }
>> +       /* Architecture specific contiguous memory fixup. */
>> +       if (of_flat_dt_is_compatible(node, "shared-dma-pool"))
>> +               dma_contiguous_early_fixup(base, size);
>>         /* Save region in the reserved_mem array */
>>         fdt_reserved_mem_save_node(node, uname, base, size);
>> diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
>> index 8df0dfaaca18..9e5d63efe7c5 100644
>> --- a/kernel/dma/contiguous.c
>> +++ b/kernel/dma/contiguous.c
>> @@ -480,8 +480,6 @@ static int __init rmem_cma_setup(struct reserved_mem
>> *rmem)
>>                 pr_err("Reserved memory: unable to setup CMA region\n");
>>                 return err;
>>         }
>> -       /* Architecture specific contiguous memory fixup. */
>> -       dma_contiguous_early_fixup(rmem->base, rmem->size);
>>
>>         if (default_cma)
>>                 dma_contiguous_default_area = cma;
>>
>> Regards,
>> Oreoluwa
> 
> Thank you Oreoluwa!  Your patch fixed the issue too and it looks a more
> localized and better fix!

Thank you for testing it out! I'm glad it worked for you.
I can work on an offical patch for this and submit it in
the next few days.

Regards,
Oreoluwa

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Marek Szyprowski]

On 03.07.2025 19:27, Oreoluwa Babatunde wrote:
> On 6/28/2025 2:04 PM, William Zhang wrote:
>>> -----Original Message-----
>>> From: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>>> Sent: Friday, June 27, 2025 11:02 AM
>>> To: William Zhang <william.zhang@broadcom.com>; robh@kernel.org
>>> Cc: aisheng.dong@nxp.com; andy@black.fi.intel.com;
>>> catalin.marinas@arm.com; devicetree@vger.kernel.org; hch@lst.de;
>>> iommu@lists.linux.dev; kernel@quicinc.com; klarasmodin@gmail.com; linux-
>>> kernel@vger.kernel.org; m.szyprowski@samsung.com;
>>> quic_ninanaik@quicinc.com; robin.murphy@arm.com; saravanak@google.com;
>>> will@kernel.org; oreoluwa.babatunde@oss.qualcomm.com
>>> Subject: Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved
>>> memory regions are processed
>>>
>>>
>>>
>>> On 6/22/2025 6:24 PM, William Zhang wrote:
>>>> On Tue, Jun 17, 2025 at 10:15 AM William Zhang
>>>> <william.zhang@broadcom.com> wrote:
>>>>> Hi Oreoluwa,
>>>>>
>>>>> On 10/8/2024 3:06 PM, Oreoluwa Babatunde wrote:
>>>>>> Reserved memory regions defined in the devicetree can be broken up
>>>>>> into
>>>>>> two groups:
>>>>>> i) Statically-placed reserved memory regions
>>>>>> i.e. regions defined with a static start address and size using the
>>>>>>        "reg" property.
>>>>>> ii) Dynamically-placed reserved memory regions.
>>>>>> i.e. regions defined by specifying an address range where they can be
>>>>>>        placed in memory using the "alloc_ranges" and "size" properties.
>>>>>>
>>>>>> These regions are processed and set aside at boot time.
>>>>>> This is done in two stages as seen below:
>>>>>>
>>>>>> Stage 1:
>>>>>> At this stage, fdt_scan_reserved_mem() scans through the child nodes
>>>>>> of
>>>>>> the reserved_memory node using the flattened devicetree and does the
>>>>>> following:
>>>>>>
>>>>>> 1) If the node represents a statically-placed reserved memory region,
>>>>>>      i.e. if it is defined using the "reg" property:
>>>>>>      - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>>>>      - Add the information for that region into the reserved_mem array
>>>>>>        using fdt_reserved_mem_save_node().
>>>>>>        i.e. fdt_reserved_mem_save_node(node, name, base, size).
>>>>>>
>>>>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>>>>      i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>>>>      - Add the information for that region to the reserved_mem array
>>>>>> with
>>>>>>        the starting address and size set to 0.
>>>>>>        i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>>>>>>      Note: This region is saved to the array with a starting address of
>>>>>> 0
>>>>>>      because a starting address is not yet allocated for it.
>>>>>>
>>>>>> Stage 2:
>>>>>> After iterating through all the reserved memory nodes and storing
>>>>>> their
>>>>>> relevant information in the reserved_mem array,fdt_init_reserved_mem()
>>>>>> is
>>>>>> called and does the following:
>>>>>>
>>>>>> 1) For statically-placed reserved memory regions:
>>>>>>      - Call the region specific init function using
>>>>>>        __reserved_mem_init_node().
>>>>>> 2) For dynamically-placed reserved memory regions:
>>>>>>      - Call __reserved_mem_alloc_size() which is used to allocate
>>>>>> memory
>>>>>>        for each of these regions, and mark them as nomap if they have
>>>>>> the
>>>>>>        nomap property specified in the DT.
>>>>>>      - Call the region specific init function.
>>>>>>
>>>>>> The current size of the resvered_mem array is 64 as is defined by
>>>>>> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
>>>>>> how many reserved memory regions can be specified on a system.
>>>>>> As systems continue to grow more and more complex, the number of
>>>>>> reserved memory regions needed are also growing and are starting to
>>>>>> hit
>>>>>> this 64 count limit, hence the need to make the reserved_mem array
>>>>>> dynamically sized (i.e. dynamically allocating memory for the
>>>>>> reserved_mem array using membock_alloc_*).
>>>>>>
>>>>>> On architectures such as arm64, memory allocated using memblock is
>>>>>> writable only after the page tables have been setup. This means that
>>>>>> if
>>>>>> the reserved_mem array is going to be dynamically allocated, it needs
>>>>>> to
>>>>>> happen after the page tables have been setup, not before.
>>>>>>
>>>>>> Since the reserved memory regions are currently being processed and
>>>>>> added to the array before the page tables are setup, there is a need
>>>>>> to
>>>>>> change the order in which some of the processing is done to allow for
>>>>>> the reserved_mem array to be dynamically sized.
>>>>>>
>>>>>> It is possible to process the statically-placed reserved memory
>>>>>> regions
>>>>>> without needing to store them in the reserved_mem array until after
>>>>>> the
>>>>>> page tables have been setup because all the information stored in the
>>>>>> array is readily available in the devicetree and can be referenced at
>>>>>> any time.
>>>>>> Dynamically-placed reserved memory regions on the other hand get
>>>>>> assigned a start address only at runtime, and hence need a place to be
>>>>>> stored once they are allocated since there is no other referrence to
>>>>>> the
>>>>>> start address for these regions.
>>>>>>
>>>>>> Hence this patch changes the processing order of the reserved memory
>>>>>> regions in the following ways:
>>>>>>
>>>>>> Step 1:
>>>>>> fdt_scan_reserved_mem() scans through the child nodes of
>>>>>> the reserved_memory node using the flattened devicetree and does the
>>>>>> following:
>>>>>>
>>>>>> 1) If the node represents a statically-placed reserved memory region,
>>>>>>      i.e. if it is defined using the "reg" property:
>>>>>>      - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>>>>
>>>>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>>>>      i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>>>>      - Call __reserved_mem_alloc_size() which will:
>>>>>>        i) Allocate memory for the reserved region and call
>>>>>>        memblock_mark_nomap() as needed.
>>>>>>        ii) Call the region specific initialization function using
>>>>>>        fdt_init_reserved_mem_node().
>>>>>>        iii) Save the region information in the reserved_mem array using
>>>>>>        fdt_reserved_mem_save_node().
>>>>>>
>>>>>> Step 2:
>>>>>> 1) This stage of the reserved memory processing is now only used to
>>>>>> add
>>>>>>      the statically-placed reserved memory regions into the
>>>>>> reserved_mem
>>>>>>      array using fdt_scan_reserved_mem_reg_nodes(), as well as call
>>>>>> their
>>>>>>      region specific initialization functions.
>>>>>>
>>>>>> 2) This step has also been moved to be after the page tables are
>>>>>>      setup. Moving this will allow us to replace the reserved_mem
>>>>>>      array with a dynamically sized array before storing the rest of
>>>>>>      these regions.
>>>>> I am running into a call trace with this order change on armv7 chip
>>>>> when
>>>>> I tried to allocate dma coherent memory from the device reserved
>>>>> memory.
>>>>> The issue does not happen on armv8 chips.
>>>>>
>>>>> [    0.000000] Reserved memory: created CMA memory pool at 0x1e000000,
>>>>> size 32 MiB
>>>>> [    0.000000] OF: reserved mem: initialized node dt_reserved_cma,
>>>>> compatible id shared-dma-pool
>>>>> [    0.000000] OF: reserved mem: 0x1e000000..0x1fffffff (32768 KiB) map
>>>>> reusable dt_reserved_cma
>>>>> ....
>>>>>
>>>>> [    0.445322] ------------[ cut here ]------------
>>>>> [    0.445353] WARNING: CPU: 0 PID: 1 at mm/memory.c:3069
>>>>> __apply_to_page_range+0x380/0x388
>>>>> [    0.488911] Modules linked in:
>>>>> [    0.492027] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted
>>>>> 6.16.0-rc1-g27605c8c0f69-dirty #3 NONE
>>>>> [    0.501174] Hardware name: Generic DT based system
>>>>> [    0.505965] Call trace:
>>>>> [    0.505985]  unwind_backtrace from show_stack+0x10/0x14
>>>>> [    0.513764]  show_stack from dump_stack_lvl+0x54/0x68
>>>>> [    0.518834]  dump_stack_lvl from __warn+0x7c/0x128
>>>>> [    0.523639]  __warn from warn_slowpath_fmt+0x184/0x18c
>>>>> [    0.527676] Freeing initrd memory: 65536K
>>>>> [    0.532788]  warn_slowpath_fmt from
>>> __apply_to_page_range+0x380/0x388
>>>>> [    0.539242]  __apply_to_page_range from
>>> apply_to_page_range+0x1c/0x24
>>>>> [    0.545689]  apply_to_page_range from
>>> __alloc_from_contiguous+0xc0/0x14c
>>>>> [    0.552398]  __alloc_from_contiguous from
>>> cma_allocator_alloc+0x34/0x3c
>>>>> [    0.559016]  cma_allocator_alloc from arch_dma_alloc+0x11c/0x2ac
>>>>> [    0.565025]  arch_dma_alloc from dma_alloc_attrs+0x90/0x2e8
>>>>> [    0.570603]  dma_alloc_attrs from dmydev_probe+0x8c/0xe8
>>>>> [    0.575919]  dmydev_probe from platform_probe+0x5c/0xb0
>>>>> [    0.581152]  platform_probe from really_probe+0xc8/0x2c8
>>>>> [    0.586467]  really_probe from __driver_probe_device+0x88/0x19c
>>>>> [    0.592387]  __driver_probe_device from
>>>>> driver_probe_device+0x30/0x104
>>>>> [    0.598915]  driver_probe_device from __driver_attach+0x90/0x178
>>>>> [    0.604921]  __driver_attach from bus_for_each_dev+0x7c/0xcc
>>>>> [    0.610582]  bus_for_each_dev from bus_add_driver+0xcc/0x1ec
>>>>> [    0.616241]  bus_add_driver from driver_register+0x7c/0x114
>>>>> [    0.621814]  driver_register from dmydev_init+0x20/0x28
>>>>> [    0.627045]  dmydev_init from do_one_initcall+0x58/0x200
>>>>> [    0.632363]  do_one_initcall from kernel_init_freeable+0x1cc/0x228
>>>>> [    0.638550]  kernel_init_freeable from kernel_init+0x1c/0x12c
>>>>> [    0.644299]  kernel_init from ret_from_fork+0x14/0x28
>>>>> [    0.649351] Exception stack(0xe0819fb0 to 0xe0819ff8)
>>>>> [    0.654401] 9fa0:                                     00000000
>>>>> 00000000 00000000 00000000
>>>>> [    0.662575] 9fc0: 00000000 00000000 00000000 00000000 00000000
>>>>> 00000000 00000000 00000000
>>>>> [    0.670747] 9fe0: 00000000 00000000 00000000 00000000 00000013
>>> 00000000
>>>>> [    0.677403] ---[ end trace 0000000000000000 ]---
>>>>> [    0.682083] dmydev dmy_device: Allocate dma memory at 0xde000000 dma
>>>>> addr 0x1e000000
>>>>>
>>>>> The reason is that now reserved memory's fixup function
>>>>> dma_contiguous_early_fixup is called after the page table is
>>>>> initialized. This fixup function increases the dma_mmu_remap count for
>>>>> each reserved memory. And the dma_contiguous_remap function depends
>>> on
>>>>> it to properly set up the reserved memory mmu table entry. Before this
>>>>> change, the paging_init function calls dma_contiguous_remap and it
>>>>> founds the reserved memory and set it up properly.  After the change,
>>>>> this function found there is no reserved memory so skip any
>>>>> initialization hence causes the crash later on when my driver tries to
>>>>> allocate dma memory from the reserved memory.
>>>>>
>>>>> My workaround below is to move the dma_contiguous_remap out from the
>>>>> paging_init function to the place right after unflatten_device_tree
>>>>> where the dma_mmu_remap count is correctly set. But this is not ideal
>>>>> solution and would like to see if you have any better way to solve the
>>>>> issue.
>>>>>
>>>>> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
>>>>> index a41c93988d2c..535d1bf44529 100644
>>>>> --- a/arch/arm/kernel/setup.c
>>>>> +++ b/arch/arm/kernel/setup.c
>>>>> @@ -1079,6 +1079,7 @@ void __init hyp_mode_check(void)
>>>>>    #endif
>>>>>    }
>>>>>
>>>>> +void __init dma_contiguous_remap(void);
>>>>>    static void (*__arm_pm_restart)(enum reboot_mode reboot_mode, const
>>>>> char *cmd);
>>>>>
>>>>>    static int arm_restart(struct notifier_block *nb, unsigned long
>>>>> action,
>>>>> @@ -1164,6 +1165,7 @@ void __init setup_arch(char **cmdline_p)
>>>>>           }
>>>>>
>>>>>           unflatten_device_tree();
>>>>> +       dma_contiguous_remap();
>>>>>
>>>>>           arm_dt_init_cpu_maps();
>>>>>           psci_dt_init();
>>>>> diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
>>>>> index edb7f56b7c91..1828c8737d70 100644
>>>>> --- a/arch/arm/mm/mmu.c
>>>>> +++ b/arch/arm/mm/mmu.c
>>>>> @@ -1773,7 +1773,6 @@ void __init paging_init(const struct machine_desc
>>>>> *mdesc)
>>>>>            * be used
>>>>>            */
>>>>>           map_kernel();
>>>>> -       dma_contiguous_remap();
>>>>>           early_fixmap_shutdown();
>>>>>           devicemaps_init(mdesc);
>>>>>           kmap_init();
>>>>>
>>>>> You can reproduce the issue on any v7 devices by adding these nodes to
>>>>> the device tree
>>>>> +       reserved-memory {
>>>>> +               #address-cells = <1>;
>>>>> +               #size-cells = <1>;
>>>>> +               ranges;
>>>>> +
>>>>> +               dt_reserved_cma: dt_reserved_cma {
>>>>> +                       compatible = "shared-dma-pool";
>>>>> +                       reusable;
>>>>> +
>>>>> +                       reg = <0x1e000000 0x2000000>;
>>>>> +               };
>>>>> +       };
>>>>> +
>>>>> +       dmy_device {
>>>>> +               compatible = "xyz,dmydev";
>>>>> +               memory-region = <&dt_reserved_cma>;
>>>>> +       };
>>>>>
>>>>> And use this test driver to trigger the call stack:
>>>>> diff --git a/drivers/char/dmydev.c b/drivers/char/dmydev.c
>>>>> new file mode 100644
>>>>> index 000000000000..1dd52ec492eb
>>>>> --- /dev/null
>>>>> +++ b/drivers/char/dmydev.c
>>>>> @@ -0,0 +1,67 @@
>>>>> +#include<linux/module.h>
>>>>> +#include<linux/kernel.h>
>>>>> +#include <linux/platform_device.h>
>>>>> +#include <linux/of.h>
>>>>> +#include <linux/of_reserved_mem.h>
>>>>> +#include <linux/dma-mapping.h>
>>>>> +
>>>>> +static int dmydev_probe(struct platform_device *pdev)
>>>>> +{
>>>>> +       void* virt_addr;
>>>>> +       dma_addr_t dma_addr;
>>>>> +       int ret;
>>>>> +
>>>>> +       printk(KERN_ALERT "dmydev_probe called\n");
>>>>> +
>>>>> +       ret = of_reserved_mem_device_init(&pdev->dev);
>>>>> +       if (ret && ret != -ENODEV) {
>>>>> +               dev_err(&pdev->dev, "Couldn't assign reserve memory to
>>>>> device ret = %d\n", ret);
>>>>> +                       return ret;
>>>>> +       }
>>>>> +
>>>>> +       virt_addr = dma_alloc_coherent(&pdev->dev, 0x800000, &dma_addr,
>>>>> GFP_KERNEL);
>>>>> +       if (virt_addr == NULL) {
>>>>> +               dev_err(&pdev->dev,"Failed to allocated cma memory\n");
>>>>> +               ret = -ENOMEM;
>>>>> +       }
>>>>> +       else
>>>>> +               dev_err(&pdev->dev,"Allocate dma memory at 0x%px dma
>>>>> addr %pad\n", virt_addr, &dma_addr);
>>>>> +
>>>>> +       return ret;
>>>>> +}
>>>>> +
>>>>> +static void dmydev_remove(struct platform_device *pdev)
>>>>> +{
>>>>> +}
>>>>> +
>>>>> +static const struct of_device_id dmydev_of_match[] = {
>>>>> +       {.compatible = "xyz,dmydev"},
>>>>> +       {}
>>>>> +};
>>>>> +MODULE_DEVICE_TABLE(of, dmydev_of_match);
>>>>>
>>>>> Let me know if you need more info.
>>>>>
>>>>>> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>>>>>> ---
>>>>>>    drivers/of/fdt.c             |   5 +-
>>>>>>    drivers/of/of_private.h      |   3 +-
>>>>>>    drivers/of/of_reserved_mem.c | 168
>>>>>> ++++++++++++++++++++++++----------
>>> -
>>>>>>    3 files changed, 122 insertions(+), 54 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
>>>>>> index 4d528c10df3a..d0dbc8183ac4 100644
>>>>>> --- a/drivers/of/fdt.c
>>>>>> +++ b/drivers/of/fdt.c
>>>>>> @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
>>>>>>                        break;
>>>>>>                memblock_reserve(base, size);
>>>>>>        }
>>>>>> -
>>>>>> -     fdt_init_reserved_mem();
>>>>>>    }
>>>>>>
>>>>>>    /**
>>>>>> @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
>>>>>>    {
>>>>>>        void *fdt = initial_boot_params;
>>>>>>
>>>>>> +     /* Save the statically-placed regions in the reserved_mem array
>>>>>> */
>>>>>> +     fdt_scan_reserved_mem_reg_nodes();
>>>>>> +
>>>>>>        /* Don't use the bootloader provided DTB if ACPI is enabled */
>>>>>>        if (!acpi_disabled)
>>>>>>                fdt = NULL;
>>>>>> diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
>>>>>> index 04aa2a91f851..29525c0b9939 100644
>>>>>> --- a/drivers/of/of_private.h
>>>>>> +++ b/drivers/of/of_private.h
>>>>>> @@ -9,6 +9,7 @@
>>>>>>     */
>>>>>>
>>>>>>    #define FDT_ALIGN_SIZE 8
>>>>>> +#define MAX_RESERVED_REGIONS    64
>>>>>>
>>>>>>    /**
>>>>>>     * struct alias_prop - Alias property in 'aliases' node
>>>>>> @@ -180,7 +181,7 @@ static inline struct device_node
>>> *__of_get_dma_parent(const struct device_node *
>>>>>>    #endif
>>>>>>
>>>>>>    int fdt_scan_reserved_mem(void);
>>>>>> -void fdt_init_reserved_mem(void);
>>>>>> +void __init fdt_scan_reserved_mem_reg_nodes(void);
>>>>>>
>>>>>>    bool of_fdt_device_is_available(const void *blob, unsigned long
>>>>>> node);
>>>>>>
>>>>>> diff --git a/drivers/of/of_reserved_mem.c
>>>>>> b/drivers/of/of_reserved_mem.c
>>>>>> index 46e1c3fbc769..2011174211f9 100644
>>>>>> --- a/drivers/of/of_reserved_mem.c
>>>>>> +++ b/drivers/of/of_reserved_mem.c
>>>>>> @@ -27,7 +27,6 @@
>>>>>>
>>>>>>    #include "of_private.h"
>>>>>>
>>>>>> -#define MAX_RESERVED_REGIONS 64
>>>>>>    static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
>>>>>>    static int reserved_mem_count;
>>>>>>
>>>>>> @@ -56,6 +55,7 @@ static int __init
>>> early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
>>>>>>        return err;
>>>>>>    }
>>>>>>
>>>>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
>>> *rmem);
>>>>>>    /*
>>>>>>     * fdt_reserved_mem_save_node() - save fdt node for second pass
>>> initialization
>>>>>>     */
>>>>>> @@ -74,6 +74,9 @@ static void __init
>>> fdt_reserved_mem_save_node(unsigned long node, const char *un
>>>>>>        rmem->base = base;
>>>>>>        rmem->size = size;
>>>>>>
>>>>>> +     /* Call the region specific initialization function */
>>>>>> +     fdt_init_reserved_mem_node(rmem);
>>>>>> +
>>>>>>        reserved_mem_count++;
>>>>>>        return;
>>>>>>    }
>>>>>> @@ -106,7 +109,6 @@ static int __init
>>> __reserved_mem_reserve_reg(unsigned long node,
>>>>>>        phys_addr_t base, size;
>>>>>>        int len;
>>>>>>        const __be32 *prop;
>>>>>> -     int first = 1;
>>>>>>        bool nomap;
>>>>>>
>>>>>>        prop = of_get_flat_dt_prop(node, "reg", &len);
>>>>>> @@ -134,10 +136,6 @@ static int __init
>>> __reserved_mem_reserve_reg(unsigned long node,
>>>>>>                               uname, &base, (unsigned long)(size /
>>>>>> SZ_1M));
>>>>>>
>>>>>>                len -= t_len;
>>>>>> -             if (first) {
>>>>>> -                     fdt_reserved_mem_save_node(node, uname, base,
>>>>>> size);
>>>>>> -                     first = 0;
>>>>>> -             }
>>>>>>        }
>>>>>>        return 0;
>>>>>>    }
>>>>>> @@ -165,12 +163,77 @@ static int __init
>>> __reserved_mem_check_root(unsigned long node)
>>>>>>        return 0;
>>>>>>    }
>>>>>>
>>>>>> +static void __init __rmem_check_for_overlap(void);
>>>>>> +
>>>>>> +/**
>>>>>> + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg"
>>>>>> defined
>>>>>> + * reserved memory regions.
>>>>>> + *
>>>>>> + * This function is used to scan through the DT and store the
>>>>>> + * information for the reserved memory regions that are defined using
>>>>>> + * the "reg" property. The region node number, name, base address,
>>>>>> and
>>>>>> + * size are all stored in the reserved_mem array by calling the
>>>>>> + * fdt_reserved_mem_save_node() function.
>>>>>> + */
>>>>>> +void __init fdt_scan_reserved_mem_reg_nodes(void)
>>>>>> +{
>>>>>> +     int t_len = (dt_root_addr_cells + dt_root_size_cells) *
>>>>>> sizeof(__be32);
>>>>>> +     const void *fdt = initial_boot_params;
>>>>>> +     phys_addr_t base, size;
>>>>>> +     const __be32 *prop;
>>>>>> +     int node, child;
>>>>>> +     int len;
>>>>>> +
>>>>>> +     if (!fdt)
>>>>>> +             return;
>>>>>> +
>>>>>> +     node = fdt_path_offset(fdt, "/reserved-memory");
>>>>>> +     if (node < 0) {
>>>>>> +             pr_info("Reserved memory: No reserved-memory node in the
>>> DT\n");
>>>>>> +             return;
>>>>>> +     }
>>>>>> +
>>>>>> +     if (__reserved_mem_check_root(node)) {
>>>>>> +             pr_err("Reserved memory: unsupported node format,
>>>>>> ignoring\n");
>>>>>> +             return;
>>>>>> +     }
>>>>>> +
>>>>>> +     fdt_for_each_subnode(child, fdt, node) {
>>>>>> +             const char *uname;
>>>>>> +
>>>>>> +             prop = of_get_flat_dt_prop(child, "reg", &len);
>>>>>> +             if (!prop)
>>>>>> +                     continue;
>>>>>> +             if (!of_fdt_device_is_available(fdt, child))
>>>>>> +                     continue;
>>>>>> +
>>>>>> +             uname = fdt_get_name(fdt, child, NULL);
>>>>>> +             if (len && len % t_len != 0) {
>>>>>> +                     pr_err("Reserved memory: invalid reg property in
>>>>>> '%s', skipping
>>> node.\n",
>>>>>> +                            uname);
>>>>>> +                     continue;
>>>>>> +             }
>>>>>> +             base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>>>>>> +             size = dt_mem_next_cell(dt_root_size_cells, &prop);
>>>>>> +
>>>>>> +             if (size)
>>>>>> +                     fdt_reserved_mem_save_node(child, uname, base,
>>>>>> size);
>>>>>> +     }
>>>>>> +
>>>>>> +     /* check for overlapping reserved regions */
>>>>>> +     __rmem_check_for_overlap();
>>>>>> +}
>>>>>> +
>>>>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
>>> char *uname);
>>>>>> +
>>>>>>    /*
>>>>>>     * fdt_scan_reserved_mem() - scan a single FDT node for reserved
>>>>>> memory
>>>>>>     */
>>>>>>    int __init fdt_scan_reserved_mem(void)
>>>>>>    {
>>>>>>        int node, child;
>>>>>> +     int dynamic_nodes_cnt = 0;
>>>>>> +     int dynamic_nodes[MAX_RESERVED_REGIONS];
>>>>>>        const void *fdt = initial_boot_params;
>>>>>>
>>>>>>        node = fdt_path_offset(fdt, "/reserved-memory");
>>>>>> @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
>>>>>>                uname = fdt_get_name(fdt, child, NULL);
>>>>>>
>>>>>>                err = __reserved_mem_reserve_reg(child, uname);
>>>>>> -             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
>>>>>> NULL))
>>>>>> -                     fdt_reserved_mem_save_node(child, uname, 0, 0);
>>>>>> +             /*
>>>>>> +              * Save the nodes for the dynamically-placed regions
>>>>>> +              * into an array which will be used for allocation right
>>>>>> +              * after all the statically-placed regions are reserved
>>>>>> +              * or marked as no-map. This is done to avoid
>>>>>> dynamically
>>>>>> +              * allocating from one of the statically-placed regions.
>>>>>> +              */
>>>>>> +             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
>>>>>> NULL)) {
>>>>>> +                     dynamic_nodes[dynamic_nodes_cnt] = child;
>>>>>> +                     dynamic_nodes_cnt++;
>>>>>> +             }
>>>>>> +     }
>>>>>> +     for (int i = 0; i < dynamic_nodes_cnt; i++) {
>>>>>> +             const char *uname;
>>>>>> +
>>>>>> +             child = dynamic_nodes[i];
>>>>>> +             uname = fdt_get_name(fdt, child, NULL);
>>>>>> +             __reserved_mem_alloc_size(child, uname);
>>>>>>        }
>>>>>>        return 0;
>>>>>>    }
>>>>>> @@ -253,8 +332,7 @@ static int __init
>>> __reserved_mem_alloc_in_range(phys_addr_t size,
>>>>>>     * __reserved_mem_alloc_size() - allocate reserved memory described
>>>>>> by
>>>>>>     *  'size', 'alignment'  and 'alloc-ranges' properties.
>>>>>>     */
>>>>>> -static int __init __reserved_mem_alloc_size(unsigned long node,
>>>>>> -     const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
>>>>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
>>> char *uname)
>>>>>>    {
>>>>>>        int t_len = (dt_root_addr_cells + dt_root_size_cells) *
>>>>>> sizeof(__be32);
>>>>>>        phys_addr_t start = 0, end = 0;
>>>>>> @@ -334,9 +412,8 @@ static int __init
>>> __reserved_mem_alloc_size(unsigned long node,
>>>>>>                return -ENOMEM;
>>>>>>        }
>>>>>>
>>>>>> -     *res_base = base;
>>>>>> -     *res_size = size;
>>>>>> -
>>>>>> +     /* Save region in the reserved_mem array */
>>>>>> +     fdt_reserved_mem_save_node(node, uname, base, size);
>>>>>>        return 0;
>>>>>>    }
>>>>>>
>>>>>> @@ -425,48 +502,37 @@ static void __init
>>> __rmem_check_for_overlap(void)
>>>>>>    }
>>>>>>
>>>>>>    /**
>>>>>> - * fdt_init_reserved_mem() - allocate and init all saved reserved
>>>>>> memory
>>> regions
>>>>>> + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
>>>>>> + * @rmem: reserved_mem struct of the memory region to be initialized.
>>>>>> + *
>>>>>> + * This function is used to call the region specific initialization
>>>>>> + * function for a reserved memory region.
>>>>>>     */
>>>>>> -void __init fdt_init_reserved_mem(void)
>>>>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
>>> *rmem)
>>>>>>    {
>>>>>> -     int i;
>>>>>> -
>>>>>> -     /* check for overlapping reserved regions */
>>>>>> -     __rmem_check_for_overlap();
>>>>>> -
>>>>>> -     for (i = 0; i < reserved_mem_count; i++) {
>>>>>> -             struct reserved_mem *rmem = &reserved_mem[i];
>>>>>> -             unsigned long node = rmem->fdt_node;
>>>>>> -             int err = 0;
>>>>>> -             bool nomap;
>>>>>> +     unsigned long node = rmem->fdt_node;
>>>>>> +     int err = 0;
>>>>>> +     bool nomap;
>>>>>>
>>>>>> -             nomap = of_get_flat_dt_prop(node, "no-map", NULL) !=
>>>>>> NULL;
>>>>>> +     nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
>>>>>>
>>>>>> -             if (rmem->size == 0)
>>>>>> -                     err = __reserved_mem_alloc_size(node,
>>>>>> rmem->name,
>>>>>> -                                              &rmem->base,
>>>>>> &rmem->size);
>>>>>> -             if (err == 0) {
>>>>>> -                     err = __reserved_mem_init_node(rmem);
>>>>>> -                     if (err != 0 && err != -ENOENT) {
>>>>>> -                             pr_info("node %s compatible matching
>>>>>> fail\n",
>>>>>> -                                     rmem->name);
>>>>>> -                             if (nomap)
>>>>>> -                                     memblock_clear_nomap(rmem->base,
>>>>>> rmem->size);
>>>>>> -                             else
>>>>>> -                                     memblock_phys_free(rmem->base,
>>>>>> -                                                        rmem->size);
>>>>>> -                     } else {
>>>>>> -                             phys_addr_t end = rmem->base +
>>>>>> rmem->size - 1;
>>>>>> -                             bool reusable =
>>>>>> -                                     (of_get_flat_dt_prop(node,
>>>>>> "reusable", NULL)) !=
>>> NULL;
>>>>>> -
>>>>>> -                             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>>>>> -                                     &rmem->base, &end, (unsigned
>>>>>> long)(rmem->size /
>>> SZ_1K),
>>>>>> -                                     nomap ? "nomap" : "map",
>>>>>> -                                     reusable ? "reusable" :
>>>>>> "non-reusable",
>>>>>> -                                     rmem->name ? rmem->name :
>>>>>> "unknown");
>>>>>> -                     }
>>>>>> -             }
>>>>>> +     err = __reserved_mem_init_node(rmem);
>>>>>> +     if (err != 0 && err != -ENOENT) {
>>>>>> +             pr_info("node %s compatible matching fail\n",
>>>>>> rmem->name);
>>>>>> +             if (nomap)
>>>>>> +                     memblock_clear_nomap(rmem->base, rmem->size);
>>>>>> +             else
>>>>>> +                     memblock_phys_free(rmem->base, rmem->size);
>>>>>> +     } else {
>>>>>> +             phys_addr_t end = rmem->base + rmem->size - 1;
>>>>>> +             bool reusable =
>>>>>> +                     (of_get_flat_dt_prop(node, "reusable", NULL)) !=
>>>>>> NULL;
>>>>>> +
>>>>>> +             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>>>>> +                     &rmem->base, &end, (unsigned long)(rmem->size /
>>>>>> SZ_1K),
>>>>>> +                     nomap ? "nomap" : "map",
>>>>>> +                     reusable ? "reusable" : "non-reusable",
>>>>>> +                     rmem->name ? rmem->name : "unknown");
>>>>>>        }
>>>>>>    }
>>>>>>
>>>> Just want to follow up on this issue.  Do you need any further detail
>>>> or clarification?
>>>> Any ARM memory manage guru on this thread can comment?
>>>> Or is my workaround acceptable as a patch?
>>>
>>> Hi William,
>>>
>>> Sorry about the delay in getting back to you.
>>>
>>> Instead of moving dma_contiguous_remap(), I suggest moving
>>> dma_contiguous_early_fixup()
>>> to the function that parses the reserved regions so that it is done before
>>> paging_init.
>>>
>>> Here is what that could look like. Can you please give this a try?
>>>
>>> diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
>>> index 77016c0cc296..132d2c66cafc 100644
>>> --- a/drivers/of/of_reserved_mem.c
>>> +++ b/drivers/of/of_reserved_mem.c
>>> @@ -25,6 +25,7 @@
>>>   #include <linux/memblock.h>
>>>   #include <linux/kmemleak.h>
>>>   #include <linux/cma.h>
>>> +#include <linux/dma-map-ops.h>
>>>
>>>   #include "of_private.h"
>>>
>>> @@ -175,13 +176,17 @@ static int __init
>>> __reserved_mem_reserve_reg(unsigned long node,
>>>                  base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>>>                  size = dt_mem_next_cell(dt_root_size_cells, &prop);
>>>
>>> -               if (size &&
>>> -                   early_init_dt_reserve_memory(base, size, nomap) == 0)
>>> +               if (size && early_init_dt_reserve_memory(base, size,
>>> nomap) == 0) {
>>> +                       /* Architecture specific contiguous memory fixup.
>>> */
>>> +                       if (of_flat_dt_is_compatible(node,
>>> "shared-dma-pool"))
>>> +                               dma_contiguous_early_fixup(base, size);
>>> +
>>>                          pr_debug("Reserved memory: reserved region for
>>> node '%s':
>>> base %pa, size %lu MiB\n",
>>>                                  uname, &base, (unsigned long)(size /
>>> SZ_1M));
>>> -               else
>>> +               } else {
>>>                          pr_err("Reserved memory: failed to reserve memory
>>> for node '%s':
>>> base %pa, size %lu MiB\n",
>>>                                 uname, &base, (unsigned long)(size /
>>> SZ_1M));
>>> +               }
>>>
>>>                  len -= t_len;
>>>          }
>>> @@ -472,6 +477,9 @@ static int __init __reserved_mem_alloc_size(unsigned
>>> long node, const char *unam
>>>                         uname, (unsigned long)(size / SZ_1M));
>>>                  return -ENOMEM;
>>>          }
>>> +       /* Architecture specific contiguous memory fixup. */
>>> +       if (of_flat_dt_is_compatible(node, "shared-dma-pool"))
>>> +               dma_contiguous_early_fixup(base, size);
>>>          /* Save region in the reserved_mem array */
>>>          fdt_reserved_mem_save_node(node, uname, base, size);
>>> diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
>>> index 8df0dfaaca18..9e5d63efe7c5 100644
>>> --- a/kernel/dma/contiguous.c
>>> +++ b/kernel/dma/contiguous.c
>>> @@ -480,8 +480,6 @@ static int __init rmem_cma_setup(struct reserved_mem
>>> *rmem)
>>>                  pr_err("Reserved memory: unable to setup CMA region\n");
>>>                  return err;
>>>          }
>>> -       /* Architecture specific contiguous memory fixup. */
>>> -       dma_contiguous_early_fixup(rmem->base, rmem->size);
>>>
>>>          if (default_cma)
>>>                  dma_contiguous_default_area = cma;
>>>
>>> Regards,
>>> Oreoluwa
>> Thank you Oreoluwa!  Your patch fixed the issue too and it looks a more
>> localized and better fix!
> Thank you for testing it out! I'm glad it worked for you.
> I can work on an offical patch for this and submit it in
> the next few days.

I'm waiting for the final patch then.

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved memory regions are processed

[Oreoluwa Babatunde]

On 7/8/2025 3:47 AM, Marek Szyprowski wrote:
> On 03.07.2025 19:27, Oreoluwa Babatunde wrote:
>> On 6/28/2025 2:04 PM, William Zhang wrote:
>>>> -----Original Message-----
>>>> From: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>>>> Sent: Friday, June 27, 2025 11:02 AM
>>>> To: William Zhang <william.zhang@broadcom.com>; robh@kernel.org
>>>> Cc: aisheng.dong@nxp.com; andy@black.fi.intel.com;
>>>> catalin.marinas@arm.com; devicetree@vger.kernel.org; hch@lst.de;
>>>> iommu@lists.linux.dev; kernel@quicinc.com; klarasmodin@gmail.com; linux-
>>>> kernel@vger.kernel.org; m.szyprowski@samsung.com;
>>>> quic_ninanaik@quicinc.com; robin.murphy@arm.com; saravanak@google.com;
>>>> will@kernel.org; oreoluwa.babatunde@oss.qualcomm.com
>>>> Subject: Re: [PATCH v10 1/2] of: reserved_mem: Restruture how the reserved
>>>> memory regions are processed
>>>>
>>>>
>>>>
>>>> On 6/22/2025 6:24 PM, William Zhang wrote:
>>>>> On Tue, Jun 17, 2025 at 10:15 AM William Zhang
>>>>> <william.zhang@broadcom.com> wrote:
>>>>>> Hi Oreoluwa,
>>>>>>
>>>>>> On 10/8/2024 3:06 PM, Oreoluwa Babatunde wrote:
>>>>>>> Reserved memory regions defined in the devicetree can be broken up
>>>>>>> into
>>>>>>> two groups:
>>>>>>> i) Statically-placed reserved memory regions
>>>>>>> i.e. regions defined with a static start address and size using the
>>>>>>>        "reg" property.
>>>>>>> ii) Dynamically-placed reserved memory regions.
>>>>>>> i.e. regions defined by specifying an address range where they can be
>>>>>>>        placed in memory using the "alloc_ranges" and "size" properties.
>>>>>>>
>>>>>>> These regions are processed and set aside at boot time.
>>>>>>> This is done in two stages as seen below:
>>>>>>>
>>>>>>> Stage 1:
>>>>>>> At this stage, fdt_scan_reserved_mem() scans through the child nodes
>>>>>>> of
>>>>>>> the reserved_memory node using the flattened devicetree and does the
>>>>>>> following:
>>>>>>>
>>>>>>> 1) If the node represents a statically-placed reserved memory region,
>>>>>>>      i.e. if it is defined using the "reg" property:
>>>>>>>      - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>>>>>      - Add the information for that region into the reserved_mem array
>>>>>>>        using fdt_reserved_mem_save_node().
>>>>>>>        i.e. fdt_reserved_mem_save_node(node, name, base, size).
>>>>>>>
>>>>>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>>>>>      i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>>>>>      - Add the information for that region to the reserved_mem array
>>>>>>> with
>>>>>>>        the starting address and size set to 0.
>>>>>>>        i.e. fdt_reserved_mem_save_node(node, name, 0, 0).
>>>>>>>      Note: This region is saved to the array with a starting address of
>>>>>>> 0
>>>>>>>      because a starting address is not yet allocated for it.
>>>>>>>
>>>>>>> Stage 2:
>>>>>>> After iterating through all the reserved memory nodes and storing
>>>>>>> their
>>>>>>> relevant information in the reserved_mem array,fdt_init_reserved_mem()
>>>>>>> is
>>>>>>> called and does the following:
>>>>>>>
>>>>>>> 1) For statically-placed reserved memory regions:
>>>>>>>      - Call the region specific init function using
>>>>>>>        __reserved_mem_init_node().
>>>>>>> 2) For dynamically-placed reserved memory regions:
>>>>>>>      - Call __reserved_mem_alloc_size() which is used to allocate
>>>>>>> memory
>>>>>>>        for each of these regions, and mark them as nomap if they have
>>>>>>> the
>>>>>>>        nomap property specified in the DT.
>>>>>>>      - Call the region specific init function.
>>>>>>>
>>>>>>> The current size of the resvered_mem array is 64 as is defined by
>>>>>>> MAX_RESERVED_REGIONS. This means that there is a limitation of 64 for
>>>>>>> how many reserved memory regions can be specified on a system.
>>>>>>> As systems continue to grow more and more complex, the number of
>>>>>>> reserved memory regions needed are also growing and are starting to
>>>>>>> hit
>>>>>>> this 64 count limit, hence the need to make the reserved_mem array
>>>>>>> dynamically sized (i.e. dynamically allocating memory for the
>>>>>>> reserved_mem array using membock_alloc_*).
>>>>>>>
>>>>>>> On architectures such as arm64, memory allocated using memblock is
>>>>>>> writable only after the page tables have been setup. This means that
>>>>>>> if
>>>>>>> the reserved_mem array is going to be dynamically allocated, it needs
>>>>>>> to
>>>>>>> happen after the page tables have been setup, not before.
>>>>>>>
>>>>>>> Since the reserved memory regions are currently being processed and
>>>>>>> added to the array before the page tables are setup, there is a need
>>>>>>> to
>>>>>>> change the order in which some of the processing is done to allow for
>>>>>>> the reserved_mem array to be dynamically sized.
>>>>>>>
>>>>>>> It is possible to process the statically-placed reserved memory
>>>>>>> regions
>>>>>>> without needing to store them in the reserved_mem array until after
>>>>>>> the
>>>>>>> page tables have been setup because all the information stored in the
>>>>>>> array is readily available in the devicetree and can be referenced at
>>>>>>> any time.
>>>>>>> Dynamically-placed reserved memory regions on the other hand get
>>>>>>> assigned a start address only at runtime, and hence need a place to be
>>>>>>> stored once they are allocated since there is no other referrence to
>>>>>>> the
>>>>>>> start address for these regions.
>>>>>>>
>>>>>>> Hence this patch changes the processing order of the reserved memory
>>>>>>> regions in the following ways:
>>>>>>>
>>>>>>> Step 1:
>>>>>>> fdt_scan_reserved_mem() scans through the child nodes of
>>>>>>> the reserved_memory node using the flattened devicetree and does the
>>>>>>> following:
>>>>>>>
>>>>>>> 1) If the node represents a statically-placed reserved memory region,
>>>>>>>      i.e. if it is defined using the "reg" property:
>>>>>>>      - Call memblock_reserve() or memblock_mark_nomap() as needed.
>>>>>>>
>>>>>>> 2) If the node represents a dynamically-placed reserved memory region,
>>>>>>>      i.e. if it is defined using "alloc-ranges" and "size" properties:
>>>>>>>      - Call __reserved_mem_alloc_size() which will:
>>>>>>>        i) Allocate memory for the reserved region and call
>>>>>>>        memblock_mark_nomap() as needed.
>>>>>>>        ii) Call the region specific initialization function using
>>>>>>>        fdt_init_reserved_mem_node().
>>>>>>>        iii) Save the region information in the reserved_mem array using
>>>>>>>        fdt_reserved_mem_save_node().
>>>>>>>
>>>>>>> Step 2:
>>>>>>> 1) This stage of the reserved memory processing is now only used to
>>>>>>> add
>>>>>>>      the statically-placed reserved memory regions into the
>>>>>>> reserved_mem
>>>>>>>      array using fdt_scan_reserved_mem_reg_nodes(), as well as call
>>>>>>> their
>>>>>>>      region specific initialization functions.
>>>>>>>
>>>>>>> 2) This step has also been moved to be after the page tables are
>>>>>>>      setup. Moving this will allow us to replace the reserved_mem
>>>>>>>      array with a dynamically sized array before storing the rest of
>>>>>>>      these regions.
>>>>>> I am running into a call trace with this order change on armv7 chip
>>>>>> when
>>>>>> I tried to allocate dma coherent memory from the device reserved
>>>>>> memory.
>>>>>> The issue does not happen on armv8 chips.
>>>>>>
>>>>>> [    0.000000] Reserved memory: created CMA memory pool at 0x1e000000,
>>>>>> size 32 MiB
>>>>>> [    0.000000] OF: reserved mem: initialized node dt_reserved_cma,
>>>>>> compatible id shared-dma-pool
>>>>>> [    0.000000] OF: reserved mem: 0x1e000000..0x1fffffff (32768 KiB) map
>>>>>> reusable dt_reserved_cma
>>>>>> ....
>>>>>>
>>>>>> [    0.445322] ------------[ cut here ]------------
>>>>>> [    0.445353] WARNING: CPU: 0 PID: 1 at mm/memory.c:3069
>>>>>> __apply_to_page_range+0x380/0x388
>>>>>> [    0.488911] Modules linked in:
>>>>>> [    0.492027] CPU: 0 UID: 0 PID: 1 Comm: swapper/0 Not tainted
>>>>>> 6.16.0-rc1-g27605c8c0f69-dirty #3 NONE
>>>>>> [    0.501174] Hardware name: Generic DT based system
>>>>>> [    0.505965] Call trace:
>>>>>> [    0.505985]  unwind_backtrace from show_stack+0x10/0x14
>>>>>> [    0.513764]  show_stack from dump_stack_lvl+0x54/0x68
>>>>>> [    0.518834]  dump_stack_lvl from __warn+0x7c/0x128
>>>>>> [    0.523639]  __warn from warn_slowpath_fmt+0x184/0x18c
>>>>>> [    0.527676] Freeing initrd memory: 65536K
>>>>>> [    0.532788]  warn_slowpath_fmt from
>>>> __apply_to_page_range+0x380/0x388
>>>>>> [    0.539242]  __apply_to_page_range from
>>>> apply_to_page_range+0x1c/0x24
>>>>>> [    0.545689]  apply_to_page_range from
>>>> __alloc_from_contiguous+0xc0/0x14c
>>>>>> [    0.552398]  __alloc_from_contiguous from
>>>> cma_allocator_alloc+0x34/0x3c
>>>>>> [    0.559016]  cma_allocator_alloc from arch_dma_alloc+0x11c/0x2ac
>>>>>> [    0.565025]  arch_dma_alloc from dma_alloc_attrs+0x90/0x2e8
>>>>>> [    0.570603]  dma_alloc_attrs from dmydev_probe+0x8c/0xe8
>>>>>> [    0.575919]  dmydev_probe from platform_probe+0x5c/0xb0
>>>>>> [    0.581152]  platform_probe from really_probe+0xc8/0x2c8
>>>>>> [    0.586467]  really_probe from __driver_probe_device+0x88/0x19c
>>>>>> [    0.592387]  __driver_probe_device from
>>>>>> driver_probe_device+0x30/0x104
>>>>>> [    0.598915]  driver_probe_device from __driver_attach+0x90/0x178
>>>>>> [    0.604921]  __driver_attach from bus_for_each_dev+0x7c/0xcc
>>>>>> [    0.610582]  bus_for_each_dev from bus_add_driver+0xcc/0x1ec
>>>>>> [    0.616241]  bus_add_driver from driver_register+0x7c/0x114
>>>>>> [    0.621814]  driver_register from dmydev_init+0x20/0x28
>>>>>> [    0.627045]  dmydev_init from do_one_initcall+0x58/0x200
>>>>>> [    0.632363]  do_one_initcall from kernel_init_freeable+0x1cc/0x228
>>>>>> [    0.638550]  kernel_init_freeable from kernel_init+0x1c/0x12c
>>>>>> [    0.644299]  kernel_init from ret_from_fork+0x14/0x28
>>>>>> [    0.649351] Exception stack(0xe0819fb0 to 0xe0819ff8)
>>>>>> [    0.654401] 9fa0:                                     00000000
>>>>>> 00000000 00000000 00000000
>>>>>> [    0.662575] 9fc0: 00000000 00000000 00000000 00000000 00000000
>>>>>> 00000000 00000000 00000000
>>>>>> [    0.670747] 9fe0: 00000000 00000000 00000000 00000000 00000013
>>>> 00000000
>>>>>> [    0.677403] ---[ end trace 0000000000000000 ]---
>>>>>> [    0.682083] dmydev dmy_device: Allocate dma memory at 0xde000000 dma
>>>>>> addr 0x1e000000
>>>>>>
>>>>>> The reason is that now reserved memory's fixup function
>>>>>> dma_contiguous_early_fixup is called after the page table is
>>>>>> initialized. This fixup function increases the dma_mmu_remap count for
>>>>>> each reserved memory. And the dma_contiguous_remap function depends
>>>> on
>>>>>> it to properly set up the reserved memory mmu table entry. Before this
>>>>>> change, the paging_init function calls dma_contiguous_remap and it
>>>>>> founds the reserved memory and set it up properly.  After the change,
>>>>>> this function found there is no reserved memory so skip any
>>>>>> initialization hence causes the crash later on when my driver tries to
>>>>>> allocate dma memory from the reserved memory.
>>>>>>
>>>>>> My workaround below is to move the dma_contiguous_remap out from the
>>>>>> paging_init function to the place right after unflatten_device_tree
>>>>>> where the dma_mmu_remap count is correctly set. But this is not ideal
>>>>>> solution and would like to see if you have any better way to solve the
>>>>>> issue.
>>>>>>
>>>>>> diff --git a/arch/arm/kernel/setup.c b/arch/arm/kernel/setup.c
>>>>>> index a41c93988d2c..535d1bf44529 100644
>>>>>> --- a/arch/arm/kernel/setup.c
>>>>>> +++ b/arch/arm/kernel/setup.c
>>>>>> @@ -1079,6 +1079,7 @@ void __init hyp_mode_check(void)
>>>>>>    #endif
>>>>>>    }
>>>>>>
>>>>>> +void __init dma_contiguous_remap(void);
>>>>>>    static void (*__arm_pm_restart)(enum reboot_mode reboot_mode, const
>>>>>> char *cmd);
>>>>>>
>>>>>>    static int arm_restart(struct notifier_block *nb, unsigned long
>>>>>> action,
>>>>>> @@ -1164,6 +1165,7 @@ void __init setup_arch(char **cmdline_p)
>>>>>>           }
>>>>>>
>>>>>>           unflatten_device_tree();
>>>>>> +       dma_contiguous_remap();
>>>>>>
>>>>>>           arm_dt_init_cpu_maps();
>>>>>>           psci_dt_init();
>>>>>> diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c
>>>>>> index edb7f56b7c91..1828c8737d70 100644
>>>>>> --- a/arch/arm/mm/mmu.c
>>>>>> +++ b/arch/arm/mm/mmu.c
>>>>>> @@ -1773,7 +1773,6 @@ void __init paging_init(const struct machine_desc
>>>>>> *mdesc)
>>>>>>            * be used
>>>>>>            */
>>>>>>           map_kernel();
>>>>>> -       dma_contiguous_remap();
>>>>>>           early_fixmap_shutdown();
>>>>>>           devicemaps_init(mdesc);
>>>>>>           kmap_init();
>>>>>>
>>>>>> You can reproduce the issue on any v7 devices by adding these nodes to
>>>>>> the device tree
>>>>>> +       reserved-memory {
>>>>>> +               #address-cells = <1>;
>>>>>> +               #size-cells = <1>;
>>>>>> +               ranges;
>>>>>> +
>>>>>> +               dt_reserved_cma: dt_reserved_cma {
>>>>>> +                       compatible = "shared-dma-pool";
>>>>>> +                       reusable;
>>>>>> +
>>>>>> +                       reg = <0x1e000000 0x2000000>;
>>>>>> +               };
>>>>>> +       };
>>>>>> +
>>>>>> +       dmy_device {
>>>>>> +               compatible = "xyz,dmydev";
>>>>>> +               memory-region = <&dt_reserved_cma>;
>>>>>> +       };
>>>>>>
>>>>>> And use this test driver to trigger the call stack:
>>>>>> diff --git a/drivers/char/dmydev.c b/drivers/char/dmydev.c
>>>>>> new file mode 100644
>>>>>> index 000000000000..1dd52ec492eb
>>>>>> --- /dev/null
>>>>>> +++ b/drivers/char/dmydev.c
>>>>>> @@ -0,0 +1,67 @@
>>>>>> +#include<linux/module.h>
>>>>>> +#include<linux/kernel.h>
>>>>>> +#include <linux/platform_device.h>
>>>>>> +#include <linux/of.h>
>>>>>> +#include <linux/of_reserved_mem.h>
>>>>>> +#include <linux/dma-mapping.h>
>>>>>> +
>>>>>> +static int dmydev_probe(struct platform_device *pdev)
>>>>>> +{
>>>>>> +       void* virt_addr;
>>>>>> +       dma_addr_t dma_addr;
>>>>>> +       int ret;
>>>>>> +
>>>>>> +       printk(KERN_ALERT "dmydev_probe called\n");
>>>>>> +
>>>>>> +       ret = of_reserved_mem_device_init(&pdev->dev);
>>>>>> +       if (ret && ret != -ENODEV) {
>>>>>> +               dev_err(&pdev->dev, "Couldn't assign reserve memory to
>>>>>> device ret = %d\n", ret);
>>>>>> +                       return ret;
>>>>>> +       }
>>>>>> +
>>>>>> +       virt_addr = dma_alloc_coherent(&pdev->dev, 0x800000, &dma_addr,
>>>>>> GFP_KERNEL);
>>>>>> +       if (virt_addr == NULL) {
>>>>>> +               dev_err(&pdev->dev,"Failed to allocated cma memory\n");
>>>>>> +               ret = -ENOMEM;
>>>>>> +       }
>>>>>> +       else
>>>>>> +               dev_err(&pdev->dev,"Allocate dma memory at 0x%px dma
>>>>>> addr %pad\n", virt_addr, &dma_addr);
>>>>>> +
>>>>>> +       return ret;
>>>>>> +}
>>>>>> +
>>>>>> +static void dmydev_remove(struct platform_device *pdev)
>>>>>> +{
>>>>>> +}
>>>>>> +
>>>>>> +static const struct of_device_id dmydev_of_match[] = {
>>>>>> +       {.compatible = "xyz,dmydev"},
>>>>>> +       {}
>>>>>> +};
>>>>>> +MODULE_DEVICE_TABLE(of, dmydev_of_match);
>>>>>>
>>>>>> Let me know if you need more info.
>>>>>>
>>>>>>> Signed-off-by: Oreoluwa Babatunde <quic_obabatun@quicinc.com>
>>>>>>> ---
>>>>>>>    drivers/of/fdt.c             |   5 +-
>>>>>>>    drivers/of/of_private.h      |   3 +-
>>>>>>>    drivers/of/of_reserved_mem.c | 168
>>>>>>> ++++++++++++++++++++++++----------
>>>> -
>>>>>>>    3 files changed, 122 insertions(+), 54 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
>>>>>>> index 4d528c10df3a..d0dbc8183ac4 100644
>>>>>>> --- a/drivers/of/fdt.c
>>>>>>> +++ b/drivers/of/fdt.c
>>>>>>> @@ -511,8 +511,6 @@ void __init early_init_fdt_scan_reserved_mem(void)
>>>>>>>                        break;
>>>>>>>                memblock_reserve(base, size);
>>>>>>>        }
>>>>>>> -
>>>>>>> -     fdt_init_reserved_mem();
>>>>>>>    }
>>>>>>>
>>>>>>>    /**
>>>>>>> @@ -1212,6 +1210,9 @@ void __init unflatten_device_tree(void)
>>>>>>>    {
>>>>>>>        void *fdt = initial_boot_params;
>>>>>>>
>>>>>>> +     /* Save the statically-placed regions in the reserved_mem array
>>>>>>> */
>>>>>>> +     fdt_scan_reserved_mem_reg_nodes();
>>>>>>> +
>>>>>>>        /* Don't use the bootloader provided DTB if ACPI is enabled */
>>>>>>>        if (!acpi_disabled)
>>>>>>>                fdt = NULL;
>>>>>>> diff --git a/drivers/of/of_private.h b/drivers/of/of_private.h
>>>>>>> index 04aa2a91f851..29525c0b9939 100644
>>>>>>> --- a/drivers/of/of_private.h
>>>>>>> +++ b/drivers/of/of_private.h
>>>>>>> @@ -9,6 +9,7 @@
>>>>>>>     */
>>>>>>>
>>>>>>>    #define FDT_ALIGN_SIZE 8
>>>>>>> +#define MAX_RESERVED_REGIONS    64
>>>>>>>
>>>>>>>    /**
>>>>>>>     * struct alias_prop - Alias property in 'aliases' node
>>>>>>> @@ -180,7 +181,7 @@ static inline struct device_node
>>>> *__of_get_dma_parent(const struct device_node *
>>>>>>>    #endif
>>>>>>>
>>>>>>>    int fdt_scan_reserved_mem(void);
>>>>>>> -void fdt_init_reserved_mem(void);
>>>>>>> +void __init fdt_scan_reserved_mem_reg_nodes(void);
>>>>>>>
>>>>>>>    bool of_fdt_device_is_available(const void *blob, unsigned long
>>>>>>> node);
>>>>>>>
>>>>>>> diff --git a/drivers/of/of_reserved_mem.c
>>>>>>> b/drivers/of/of_reserved_mem.c
>>>>>>> index 46e1c3fbc769..2011174211f9 100644
>>>>>>> --- a/drivers/of/of_reserved_mem.c
>>>>>>> +++ b/drivers/of/of_reserved_mem.c
>>>>>>> @@ -27,7 +27,6 @@
>>>>>>>
>>>>>>>    #include "of_private.h"
>>>>>>>
>>>>>>> -#define MAX_RESERVED_REGIONS 64
>>>>>>>    static struct reserved_mem reserved_mem[MAX_RESERVED_REGIONS];
>>>>>>>    static int reserved_mem_count;
>>>>>>>
>>>>>>> @@ -56,6 +55,7 @@ static int __init
>>>> early_init_dt_alloc_reserved_memory_arch(phys_addr_t size,
>>>>>>>        return err;
>>>>>>>    }
>>>>>>>
>>>>>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
>>>> *rmem);
>>>>>>>    /*
>>>>>>>     * fdt_reserved_mem_save_node() - save fdt node for second pass
>>>> initialization
>>>>>>>     */
>>>>>>> @@ -74,6 +74,9 @@ static void __init
>>>> fdt_reserved_mem_save_node(unsigned long node, const char *un
>>>>>>>        rmem->base = base;
>>>>>>>        rmem->size = size;
>>>>>>>
>>>>>>> +     /* Call the region specific initialization function */
>>>>>>> +     fdt_init_reserved_mem_node(rmem);
>>>>>>> +
>>>>>>>        reserved_mem_count++;
>>>>>>>        return;
>>>>>>>    }
>>>>>>> @@ -106,7 +109,6 @@ static int __init
>>>> __reserved_mem_reserve_reg(unsigned long node,
>>>>>>>        phys_addr_t base, size;
>>>>>>>        int len;
>>>>>>>        const __be32 *prop;
>>>>>>> -     int first = 1;
>>>>>>>        bool nomap;
>>>>>>>
>>>>>>>        prop = of_get_flat_dt_prop(node, "reg", &len);
>>>>>>> @@ -134,10 +136,6 @@ static int __init
>>>> __reserved_mem_reserve_reg(unsigned long node,
>>>>>>>                               uname, &base, (unsigned long)(size /
>>>>>>> SZ_1M));
>>>>>>>
>>>>>>>                len -= t_len;
>>>>>>> -             if (first) {
>>>>>>> -                     fdt_reserved_mem_save_node(node, uname, base,
>>>>>>> size);
>>>>>>> -                     first = 0;
>>>>>>> -             }
>>>>>>>        }
>>>>>>>        return 0;
>>>>>>>    }
>>>>>>> @@ -165,12 +163,77 @@ static int __init
>>>> __reserved_mem_check_root(unsigned long node)
>>>>>>>        return 0;
>>>>>>>    }
>>>>>>>
>>>>>>> +static void __init __rmem_check_for_overlap(void);
>>>>>>> +
>>>>>>> +/**
>>>>>>> + * fdt_scan_reserved_mem_reg_nodes() - Store info for the "reg"
>>>>>>> defined
>>>>>>> + * reserved memory regions.
>>>>>>> + *
>>>>>>> + * This function is used to scan through the DT and store the
>>>>>>> + * information for the reserved memory regions that are defined using
>>>>>>> + * the "reg" property. The region node number, name, base address,
>>>>>>> and
>>>>>>> + * size are all stored in the reserved_mem array by calling the
>>>>>>> + * fdt_reserved_mem_save_node() function.
>>>>>>> + */
>>>>>>> +void __init fdt_scan_reserved_mem_reg_nodes(void)
>>>>>>> +{
>>>>>>> +     int t_len = (dt_root_addr_cells + dt_root_size_cells) *
>>>>>>> sizeof(__be32);
>>>>>>> +     const void *fdt = initial_boot_params;
>>>>>>> +     phys_addr_t base, size;
>>>>>>> +     const __be32 *prop;
>>>>>>> +     int node, child;
>>>>>>> +     int len;
>>>>>>> +
>>>>>>> +     if (!fdt)
>>>>>>> +             return;
>>>>>>> +
>>>>>>> +     node = fdt_path_offset(fdt, "/reserved-memory");
>>>>>>> +     if (node < 0) {
>>>>>>> +             pr_info("Reserved memory: No reserved-memory node in the
>>>> DT\n");
>>>>>>> +             return;
>>>>>>> +     }
>>>>>>> +
>>>>>>> +     if (__reserved_mem_check_root(node)) {
>>>>>>> +             pr_err("Reserved memory: unsupported node format,
>>>>>>> ignoring\n");
>>>>>>> +             return;
>>>>>>> +     }
>>>>>>> +
>>>>>>> +     fdt_for_each_subnode(child, fdt, node) {
>>>>>>> +             const char *uname;
>>>>>>> +
>>>>>>> +             prop = of_get_flat_dt_prop(child, "reg", &len);
>>>>>>> +             if (!prop)
>>>>>>> +                     continue;
>>>>>>> +             if (!of_fdt_device_is_available(fdt, child))
>>>>>>> +                     continue;
>>>>>>> +
>>>>>>> +             uname = fdt_get_name(fdt, child, NULL);
>>>>>>> +             if (len && len % t_len != 0) {
>>>>>>> +                     pr_err("Reserved memory: invalid reg property in
>>>>>>> '%s', skipping
>>>> node.\n",
>>>>>>> +                            uname);
>>>>>>> +                     continue;
>>>>>>> +             }
>>>>>>> +             base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>>>>>>> +             size = dt_mem_next_cell(dt_root_size_cells, &prop);
>>>>>>> +
>>>>>>> +             if (size)
>>>>>>> +                     fdt_reserved_mem_save_node(child, uname, base,
>>>>>>> size);
>>>>>>> +     }
>>>>>>> +
>>>>>>> +     /* check for overlapping reserved regions */
>>>>>>> +     __rmem_check_for_overlap();
>>>>>>> +}
>>>>>>> +
>>>>>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
>>>> char *uname);
>>>>>>> +
>>>>>>>    /*
>>>>>>>     * fdt_scan_reserved_mem() - scan a single FDT node for reserved
>>>>>>> memory
>>>>>>>     */
>>>>>>>    int __init fdt_scan_reserved_mem(void)
>>>>>>>    {
>>>>>>>        int node, child;
>>>>>>> +     int dynamic_nodes_cnt = 0;
>>>>>>> +     int dynamic_nodes[MAX_RESERVED_REGIONS];
>>>>>>>        const void *fdt = initial_boot_params;
>>>>>>>
>>>>>>>        node = fdt_path_offset(fdt, "/reserved-memory");
>>>>>>> @@ -192,8 +255,24 @@ int __init fdt_scan_reserved_mem(void)
>>>>>>>                uname = fdt_get_name(fdt, child, NULL);
>>>>>>>
>>>>>>>                err = __reserved_mem_reserve_reg(child, uname);
>>>>>>> -             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
>>>>>>> NULL))
>>>>>>> -                     fdt_reserved_mem_save_node(child, uname, 0, 0);
>>>>>>> +             /*
>>>>>>> +              * Save the nodes for the dynamically-placed regions
>>>>>>> +              * into an array which will be used for allocation right
>>>>>>> +              * after all the statically-placed regions are reserved
>>>>>>> +              * or marked as no-map. This is done to avoid
>>>>>>> dynamically
>>>>>>> +              * allocating from one of the statically-placed regions.
>>>>>>> +              */
>>>>>>> +             if (err == -ENOENT && of_get_flat_dt_prop(child, "size",
>>>>>>> NULL)) {
>>>>>>> +                     dynamic_nodes[dynamic_nodes_cnt] = child;
>>>>>>> +                     dynamic_nodes_cnt++;
>>>>>>> +             }
>>>>>>> +     }
>>>>>>> +     for (int i = 0; i < dynamic_nodes_cnt; i++) {
>>>>>>> +             const char *uname;
>>>>>>> +
>>>>>>> +             child = dynamic_nodes[i];
>>>>>>> +             uname = fdt_get_name(fdt, child, NULL);
>>>>>>> +             __reserved_mem_alloc_size(child, uname);
>>>>>>>        }
>>>>>>>        return 0;
>>>>>>>    }
>>>>>>> @@ -253,8 +332,7 @@ static int __init
>>>> __reserved_mem_alloc_in_range(phys_addr_t size,
>>>>>>>     * __reserved_mem_alloc_size() - allocate reserved memory described
>>>>>>> by
>>>>>>>     *  'size', 'alignment'  and 'alloc-ranges' properties.
>>>>>>>     */
>>>>>>> -static int __init __reserved_mem_alloc_size(unsigned long node,
>>>>>>> -     const char *uname, phys_addr_t *res_base, phys_addr_t *res_size)
>>>>>>> +static int __init __reserved_mem_alloc_size(unsigned long node, const
>>>> char *uname)
>>>>>>>    {
>>>>>>>        int t_len = (dt_root_addr_cells + dt_root_size_cells) *
>>>>>>> sizeof(__be32);
>>>>>>>        phys_addr_t start = 0, end = 0;
>>>>>>> @@ -334,9 +412,8 @@ static int __init
>>>> __reserved_mem_alloc_size(unsigned long node,
>>>>>>>                return -ENOMEM;
>>>>>>>        }
>>>>>>>
>>>>>>> -     *res_base = base;
>>>>>>> -     *res_size = size;
>>>>>>> -
>>>>>>> +     /* Save region in the reserved_mem array */
>>>>>>> +     fdt_reserved_mem_save_node(node, uname, base, size);
>>>>>>>        return 0;
>>>>>>>    }
>>>>>>>
>>>>>>> @@ -425,48 +502,37 @@ static void __init
>>>> __rmem_check_for_overlap(void)
>>>>>>>    }
>>>>>>>
>>>>>>>    /**
>>>>>>> - * fdt_init_reserved_mem() - allocate and init all saved reserved
>>>>>>> memory
>>>> regions
>>>>>>> + * fdt_init_reserved_mem_node() - Initialize a reserved memory region
>>>>>>> + * @rmem: reserved_mem struct of the memory region to be initialized.
>>>>>>> + *
>>>>>>> + * This function is used to call the region specific initialization
>>>>>>> + * function for a reserved memory region.
>>>>>>>     */
>>>>>>> -void __init fdt_init_reserved_mem(void)
>>>>>>> +static void __init fdt_init_reserved_mem_node(struct reserved_mem
>>>> *rmem)
>>>>>>>    {
>>>>>>> -     int i;
>>>>>>> -
>>>>>>> -     /* check for overlapping reserved regions */
>>>>>>> -     __rmem_check_for_overlap();
>>>>>>> -
>>>>>>> -     for (i = 0; i < reserved_mem_count; i++) {
>>>>>>> -             struct reserved_mem *rmem = &reserved_mem[i];
>>>>>>> -             unsigned long node = rmem->fdt_node;
>>>>>>> -             int err = 0;
>>>>>>> -             bool nomap;
>>>>>>> +     unsigned long node = rmem->fdt_node;
>>>>>>> +     int err = 0;
>>>>>>> +     bool nomap;
>>>>>>>
>>>>>>> -             nomap = of_get_flat_dt_prop(node, "no-map", NULL) !=
>>>>>>> NULL;
>>>>>>> +     nomap = of_get_flat_dt_prop(node, "no-map", NULL) != NULL;
>>>>>>>
>>>>>>> -             if (rmem->size == 0)
>>>>>>> -                     err = __reserved_mem_alloc_size(node,
>>>>>>> rmem->name,
>>>>>>> -                                              &rmem->base,
>>>>>>> &rmem->size);
>>>>>>> -             if (err == 0) {
>>>>>>> -                     err = __reserved_mem_init_node(rmem);
>>>>>>> -                     if (err != 0 && err != -ENOENT) {
>>>>>>> -                             pr_info("node %s compatible matching
>>>>>>> fail\n",
>>>>>>> -                                     rmem->name);
>>>>>>> -                             if (nomap)
>>>>>>> -                                     memblock_clear_nomap(rmem->base,
>>>>>>> rmem->size);
>>>>>>> -                             else
>>>>>>> -                                     memblock_phys_free(rmem->base,
>>>>>>> -                                                        rmem->size);
>>>>>>> -                     } else {
>>>>>>> -                             phys_addr_t end = rmem->base +
>>>>>>> rmem->size - 1;
>>>>>>> -                             bool reusable =
>>>>>>> -                                     (of_get_flat_dt_prop(node,
>>>>>>> "reusable", NULL)) !=
>>>> NULL;
>>>>>>> -
>>>>>>> -                             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>>>>>> -                                     &rmem->base, &end, (unsigned
>>>>>>> long)(rmem->size /
>>>> SZ_1K),
>>>>>>> -                                     nomap ? "nomap" : "map",
>>>>>>> -                                     reusable ? "reusable" :
>>>>>>> "non-reusable",
>>>>>>> -                                     rmem->name ? rmem->name :
>>>>>>> "unknown");
>>>>>>> -                     }
>>>>>>> -             }
>>>>>>> +     err = __reserved_mem_init_node(rmem);
>>>>>>> +     if (err != 0 && err != -ENOENT) {
>>>>>>> +             pr_info("node %s compatible matching fail\n",
>>>>>>> rmem->name);
>>>>>>> +             if (nomap)
>>>>>>> +                     memblock_clear_nomap(rmem->base, rmem->size);
>>>>>>> +             else
>>>>>>> +                     memblock_phys_free(rmem->base, rmem->size);
>>>>>>> +     } else {
>>>>>>> +             phys_addr_t end = rmem->base + rmem->size - 1;
>>>>>>> +             bool reusable =
>>>>>>> +                     (of_get_flat_dt_prop(node, "reusable", NULL)) !=
>>>>>>> NULL;
>>>>>>> +
>>>>>>> +             pr_info("%pa..%pa (%lu KiB) %s %s %s\n",
>>>>>>> +                     &rmem->base, &end, (unsigned long)(rmem->size /
>>>>>>> SZ_1K),
>>>>>>> +                     nomap ? "nomap" : "map",
>>>>>>> +                     reusable ? "reusable" : "non-reusable",
>>>>>>> +                     rmem->name ? rmem->name : "unknown");
>>>>>>>        }
>>>>>>>    }
>>>>>>>
>>>>> Just want to follow up on this issue.  Do you need any further detail
>>>>> or clarification?
>>>>> Any ARM memory manage guru on this thread can comment?
>>>>> Or is my workaround acceptable as a patch?
>>>>
>>>> Hi William,
>>>>
>>>> Sorry about the delay in getting back to you.
>>>>
>>>> Instead of moving dma_contiguous_remap(), I suggest moving
>>>> dma_contiguous_early_fixup()
>>>> to the function that parses the reserved regions so that it is done before
>>>> paging_init.
>>>>
>>>> Here is what that could look like. Can you please give this a try?
>>>>
>>>> diff --git a/drivers/of/of_reserved_mem.c b/drivers/of/of_reserved_mem.c
>>>> index 77016c0cc296..132d2c66cafc 100644
>>>> --- a/drivers/of/of_reserved_mem.c
>>>> +++ b/drivers/of/of_reserved_mem.c
>>>> @@ -25,6 +25,7 @@
>>>>   #include <linux/memblock.h>
>>>>   #include <linux/kmemleak.h>
>>>>   #include <linux/cma.h>
>>>> +#include <linux/dma-map-ops.h>
>>>>
>>>>   #include "of_private.h"
>>>>
>>>> @@ -175,13 +176,17 @@ static int __init
>>>> __reserved_mem_reserve_reg(unsigned long node,
>>>>                  base = dt_mem_next_cell(dt_root_addr_cells, &prop);
>>>>                  size = dt_mem_next_cell(dt_root_size_cells, &prop);
>>>>
>>>> -               if (size &&
>>>> -                   early_init_dt_reserve_memory(base, size, nomap) == 0)
>>>> +               if (size && early_init_dt_reserve_memory(base, size,
>>>> nomap) == 0) {
>>>> +                       /* Architecture specific contiguous memory fixup.
>>>> */
>>>> +                       if (of_flat_dt_is_compatible(node,
>>>> "shared-dma-pool"))
>>>> +                               dma_contiguous_early_fixup(base, size);
>>>> +
>>>>                          pr_debug("Reserved memory: reserved region for
>>>> node '%s':
>>>> base %pa, size %lu MiB\n",
>>>>                                  uname, &base, (unsigned long)(size /
>>>> SZ_1M));
>>>> -               else
>>>> +               } else {
>>>>                          pr_err("Reserved memory: failed to reserve memory
>>>> for node '%s':
>>>> base %pa, size %lu MiB\n",
>>>>                                 uname, &base, (unsigned long)(size /
>>>> SZ_1M));
>>>> +               }
>>>>
>>>>                  len -= t_len;
>>>>          }
>>>> @@ -472,6 +477,9 @@ static int __init __reserved_mem_alloc_size(unsigned
>>>> long node, const char *unam
>>>>                         uname, (unsigned long)(size / SZ_1M));
>>>>                  return -ENOMEM;
>>>>          }
>>>> +       /* Architecture specific contiguous memory fixup. */
>>>> +       if (of_flat_dt_is_compatible(node, "shared-dma-pool"))
>>>> +               dma_contiguous_early_fixup(base, size);
>>>>          /* Save region in the reserved_mem array */
>>>>          fdt_reserved_mem_save_node(node, uname, base, size);
>>>> diff --git a/kernel/dma/contiguous.c b/kernel/dma/contiguous.c
>>>> index 8df0dfaaca18..9e5d63efe7c5 100644
>>>> --- a/kernel/dma/contiguous.c
>>>> +++ b/kernel/dma/contiguous.c
>>>> @@ -480,8 +480,6 @@ static int __init rmem_cma_setup(struct reserved_mem
>>>> *rmem)
>>>>                  pr_err("Reserved memory: unable to setup CMA region\n");
>>>>                  return err;
>>>>          }
>>>> -       /* Architecture specific contiguous memory fixup. */
>>>> -       dma_contiguous_early_fixup(rmem->base, rmem->size);
>>>>
>>>>          if (default_cma)
>>>>                  dma_contiguous_default_area = cma;
>>>>
>>>> Regards,
>>>> Oreoluwa
>>> Thank you Oreoluwa!  Your patch fixed the issue too and it looks a more
>>> localized and better fix!
>> Thank you for testing it out! I'm glad it worked for you.
>> I can work on an offical patch for this and submit it in
>> the next few days.
> 
> I'm waiting for the final patch then.
> 
> Best regards

I have uploaded the final patch here:
https://lore.kernel.org/all/20250708165627.845295-1-oreoluwa.babatunde@oss.qualcomm.com/

Thanks!