[PATCH v2 0/3] OpenSBI domain initialization order

[PATCH v2 0/3] OpenSBI domain initialization order

[Anup Patel]

Currently, we rely on the "opensbi-domain" DT property of CPU DT nodes
to select the domain assigned to each HART at boot-time but unfortunately
this is only suitable if we have two non-ROOT domains (aka trused and
non-trusted. For more than two non-ROOT domains, the domain initialization
order should be much more predictable where users should define the exact
order via some DT property under domain DT nodes.

These patches can also be found in sbi_domain_init_order_v2 branch
at: https://github.com/avpatel/opensbi.git

Changes since v1:
- Removed unwanted "break" from the loop in sbi_domain_context_exit()
- Use root values when "next-arg1", "next-addr" and "next-mode" DT
  properties are absent in the domain DT node.

Anup Patel (3):
  lib: sbi_domain: Check possible harts in
    sbi_domain_context_enter/exit()
  lib: sbi_domain: Re-work boot-time assignment of HARTs to non-ROOT
    domains
  lib: sbi_domain: Introduce domain intialization order

 docs/domain_support.md             |  25 ++++---
 include/sbi/sbi_domain.h           |   6 +-
 include/sbi_utils/fdt/fdt_domain.h |   3 +-
 lib/sbi/sbi_domain.c               |  45 ++++++++-----
 lib/sbi/sbi_domain_context.c       |  31 ++++++---
 lib/utils/fdt/fdt_domain.c         | 103 +++++------------------------
 6 files changed, 79 insertions(+), 134 deletions(-)

-- 
2.43.0


-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH v2 1/3] lib: sbi_domain: Check possible harts in sbi_domain_context_enter/exit()

[Anup Patel]

When context switching to a domain the current hart MUST be part
of the possible harts of that domain. Add appropriate checks in
sbi_domain_context_enter/exit() along these lines.

Signed-off-by: Anup Patel <anup.patel@oss.qualcomm.com>
Reviewed-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
Tested-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
---
 lib/sbi/sbi_domain_context.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/lib/sbi/sbi_domain_context.c b/lib/sbi/sbi_domain_context.c
index 0861d541..837ff54f 100644
--- a/lib/sbi/sbi_domain_context.c
+++ b/lib/sbi/sbi_domain_context.c
@@ -231,18 +231,23 @@ int sbi_domain_context_enter(struct sbi_domain *dom)
 	int rc;
 	struct hart_context *dom_ctx;
 	struct hart_context *ctx = hart_context_thishart_get();
+	u32 hartindex = current_hartindex();
 
 	/* Target domain must not be same as the current domain */
 	if (!dom || dom == sbi_domain_thishart_ptr())
 		return SBI_EINVAL;
 
+	/* Target domain must have current hart as a possible harts */
+	if (!sbi_hartmask_test_hartindex(hartindex, dom->possible_harts))
+		return SBI_EINVAL;
+
 	/*
 	 * If it's first time to call `enter` on the current hart, no
 	 * context allocated before. Allocate context for each valid
 	 * domain on the current hart.
 	 */
 	if (!ctx) {
-		rc = hart_context_init(current_hartindex());
+		rc = hart_context_init(hartindex);
 		if (rc)
 			return rc;
 
@@ -251,7 +256,7 @@ int sbi_domain_context_enter(struct sbi_domain *dom)
 			return SBI_EINVAL;
 	}
 
-	dom_ctx = hart_context_get(dom, current_hartindex());
+	dom_ctx = hart_context_get(dom, hartindex);
 	/* Validate the domain context existence */
 	if (!dom_ctx)
 		return SBI_EINVAL;
@@ -276,7 +281,7 @@ int sbi_domain_context_exit(void)
 	 * its context on the current hart if valid.
 	 */
 	if (!ctx) {
-		rc = hart_context_init(current_hartindex());
+		rc = hart_context_init(hartindex);
 		if (rc)
 			return rc;
 
@@ -294,6 +299,9 @@ int sbi_domain_context_exit(void)
 			if (dom == &root || dom == sbi_domain_thishart_ptr())
 				continue;
 
+			if (!sbi_hartmask_test_hartindex(hartindex, dom->possible_harts))
+				continue;
+
 			tmp = hart_context_get(dom, hartindex);
 			if (tmp && !tmp->initialized) {
 				dom_ctx = tmp;
-- 
2.43.0


-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH v2 2/3] lib: sbi_domain: Re-work boot-time assignment of HARTs to non-ROOT domains

[Anup Patel]

Assign a non-ROOT domain to a HART on first come first serve basis if the
HART is listed as a possible HART of the non-ROOT domain. If no non-ROOT
domain list a HART as possible HART then the HART is assigned to the ROOT
domain.

This allows us to drop the OpenSBI specific DT property from each CPU DT
node (aka "opensbi-domain" Dt property).

Signed-off-by: Anup Patel <anup.patel@oss.qualcomm.com>
---
 docs/domain_support.md             | 20 +++----
 include/sbi/sbi_domain.h           |  4 +-
 include/sbi_utils/fdt/fdt_domain.h |  3 +-
 lib/sbi/sbi_domain.c               | 34 ++++++-----
 lib/utils/fdt/fdt_domain.c         | 95 +++---------------------------
 5 files changed, 35 insertions(+), 121 deletions(-)

diff --git a/docs/domain_support.md b/docs/domain_support.md
index e267a9f7..82f155e1 100644
--- a/docs/domain_support.md
+++ b/docs/domain_support.md
@@ -207,15 +207,14 @@ The DT properties of a domain instance DT node are as follows:
 
 ### Assigning HART To Domain Instance
 
-By default, all HARTs are assigned to **the ROOT domain**. The OpenSBI
-platform support can provide the HART to domain instance assignment using
-platform specific callback.
+At boot-time, a HART is assigned to a non-ROOT domain on first come
+first serve basis if the HART is listed as a possible HART of the
+non-ROOT domain. If no non-ROOT domain list a HART as possible HART
+then the HART is assigned to **the ROOT domain**.
 
-The HART to domain instance assignment can be parsed from the device tree
-using optional DT property **opensbi-domain** in each CPU DT node. The
-value of DT property **opensbi-domain** is the DT phandle of the domain
-instance DT node. If **opensbi-domain** DT property is not specified then
-corresponding HART is assigned to **the ROOT domain**.
+At runtime, the assignment of a HART can change from one domain to
+another domain as long as the HART is listed in possible HARTs of
+both domains.
 
 ### Domain Configuration Only Accessible to OpenSBI
 
@@ -289,7 +288,6 @@ be done:
             device_type = "cpu";
             reg = <0x00>;
             compatible = "riscv";
-            opensbi-domain = <&tdomain>;
             ...
         };
 
@@ -297,7 +295,6 @@ be done:
             device_type = "cpu";
             reg = <0x01>;
             compatible = "riscv";
-            opensbi-domain = <&udomain>;
             ...
         };
 
@@ -305,7 +302,6 @@ be done:
             device_type = "cpu";
             reg = <0x02>;
             compatible = "riscv";
-            opensbi-domain = <&udomain>;
             ...
         };
 
@@ -313,7 +309,6 @@ be done:
             device_type = "cpu";
             reg = <0x03>;
             compatible = "riscv";
-            opensbi-domain = <&udomain>;
             ...
         };
 
@@ -321,7 +316,6 @@ be done:
             device_type = "cpu";
             reg = <0x04>;
             compatible = "riscv";
-            opensbi-domain = <&udomain>;
             ...
         };
     };
diff --git a/include/sbi/sbi_domain.h b/include/sbi/sbi_domain.h
index 16edd4ce..b6731c04 100644
--- a/include/sbi/sbi_domain.h
+++ b/include/sbi/sbi_domain.h
@@ -328,12 +328,10 @@ void sbi_domain_dump_all(const char *suffix);
 /**
  * Register a new domain
  * @param dom pointer to domain
- * @param assign_mask pointer to HART mask of HARTs assigned to the domain
  *
  * @return 0 on success and negative error code on failure
  */
-int sbi_domain_register(struct sbi_domain *dom,
-			const struct sbi_hartmask *assign_mask);
+int sbi_domain_register(struct sbi_domain *dom);
 
 /**
  * Add a memory range with its flags to the root domain
diff --git a/include/sbi_utils/fdt/fdt_domain.h b/include/sbi_utils/fdt/fdt_domain.h
index 8c2dee09..a366b271 100644
--- a/include/sbi_utils/fdt/fdt_domain.h
+++ b/include/sbi_utils/fdt/fdt_domain.h
@@ -50,8 +50,7 @@ int fdt_iterate_each_memregion(void *fdt, int domain_offset, void *opaque,
  *
  * This routine:
  * 1. Disables MMIO devices not accessible to the coldboot HART domain
- * 2. Removes "opensbi-domain" DT property from CPU DT nodes
- * 3. Removes domain configuration DT node under /chosen DT node
+ * 2. Removes domain configuration DT node under /chosen DT node
  *
  * It is recommended that platform support call this function in
  * their final_init() platform operation.
diff --git a/lib/sbi/sbi_domain.c b/lib/sbi/sbi_domain.c
index fa69170b..af753459 100644
--- a/lib/sbi/sbi_domain.c
+++ b/lib/sbi/sbi_domain.c
@@ -625,16 +625,14 @@ void sbi_domain_dump_all(const char *suffix)
 	}
 }
 
-int sbi_domain_register(struct sbi_domain *dom,
-			const struct sbi_hartmask *assign_mask)
+int sbi_domain_register(struct sbi_domain *dom)
 {
-	u32 i;
-	int rc;
+	u32 i, cold_hartid = current_hartid();
 	struct sbi_domain *tdom;
-	u32 cold_hartid = current_hartid();
+	int rc;
 
 	/* Sanity checks */
-	if (!dom || !assign_mask || domain_finalized)
+	if (!dom || domain_finalized)
 		return SBI_EINVAL;
 
 	/* Check if domain already discovered */
@@ -663,15 +661,21 @@ int sbi_domain_register(struct sbi_domain *dom,
 	/* Clear assigned HARTs of domain */
 	sbi_hartmask_clear_all(&dom->assigned_harts);
 
-	/* Assign domain to HART if HART is a possible HART */
-	sbi_hartmask_for_each_hartindex(i, assign_mask) {
-		if (!sbi_hartmask_test_hartindex(i, dom->possible_harts))
-			continue;
-
+	/*
+	 * Assign a non-ROOT domain to a HART on first come first serve
+	 * basis if the HART is listed as a possible HART of the non-ROOT
+	 * domain. If no non-ROOT domain list a HART as possible HART then
+	 * the HART is assigned to the ROOT domain.
+	 */
+	sbi_hartmask_for_each_hartindex(i, dom->possible_harts) {
 		tdom = sbi_hartindex_to_domain(i);
-		if (tdom)
-			sbi_hartmask_clear_hartindex(i,
-					&tdom->assigned_harts);
+		if (tdom) {
+			if (tdom == &root)
+				sbi_hartmask_clear_hartindex(i, &tdom->assigned_harts);
+			else
+				continue;
+		}
+
 		sbi_update_hartindex_to_domain(i, dom);
 		sbi_hartmask_set_hartindex(i, &dom->assigned_harts);
 
@@ -975,7 +979,7 @@ int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)
 		sbi_hartmask_set_hartindex(i, root_hmask);
 
 	/* Finally register the root domain */
-	rc = sbi_domain_register(&root, root_hmask);
+	rc = sbi_domain_register(&root);
 	if (rc)
 		goto fail_free_root_hmask;
 
diff --git a/lib/utils/fdt/fdt_domain.c b/lib/utils/fdt/fdt_domain.c
index 61627db3..3857831d 100644
--- a/lib/utils/fdt/fdt_domain.c
+++ b/lib/utils/fdt/fdt_domain.c
@@ -170,26 +170,11 @@ static int __fixup_disable_devices(void *fdt, int doff, int roff,
 
 void fdt_domain_fixup(void *fdt)
 {
-	u32 i, dcount;
+	u32 dcount;
 	int err, poffset, doffset;
 	struct sbi_domain *dom = sbi_domain_thishart_ptr();
 	struct __fixup_find_domain_offset_info fdo;
 
-	/* Remove the domain assignment DT property from CPU DT nodes */
-	poffset = fdt_path_offset(fdt, "/cpus");
-	if (poffset < 0)
-		return;
-	fdt_for_each_subnode(doffset, fdt, poffset) {
-		err = fdt_parse_hart_id(fdt, doffset, &i);
-		if (err)
-			continue;
-
-		if (!fdt_node_is_enabled(fdt, doffset))
-			continue;
-
-		fdt_nop_property(fdt, doffset, "opensbi-domain");
-	}
-
 	/* Skip device disable for root domain */
 	if (!dom->index)
 		goto skip_device_disable;
@@ -315,12 +300,10 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
 	const char *inherit;
 	struct sbi_domain *dom;
 	struct sbi_hartmask *mask;
-	struct sbi_hartmask assign_mask;
 	struct parse_region_data preg;
-	int *cold_domain_offset = opaque;
 	struct sbi_domain_memregion *reg;
 	int inheritance_mode = FDT_ROOT_REGION_INHERIT_M_ONLY;
-	int i, err = 0, len, cpus_offset, cpu_offset, doffset;
+	int i, err = 0, len, cpus_offset, cpu_offset;
 
 	dom = sbi_zalloc(sizeof(*dom));
 	if (!dom)
@@ -440,7 +423,7 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
 		val64 = fdt32_to_cpu(val[0]);
 		val64 = (val64 << 32) | fdt32_to_cpu(val[1]);
 	} else {
-		val64 = sbi_scratch_thishart_ptr()->next_arg1;
+		val64 = root.next_arg1;
 	}
 	dom->next_arg1 = val64;
 
@@ -451,8 +434,7 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
 		val64 = fdt32_to_cpu(val[0]);
 		val64 = (val64 << 32) | fdt32_to_cpu(val[1]);
 	} else {
-		if (domain_offset == *cold_domain_offset)
-			val64 = sbi_scratch_thishart_ptr()->next_addr;
+		val64 = root.next_addr;
 	}
 	dom->next_addr = val64;
 
@@ -464,8 +446,7 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
 		if (val32 != 0x0 && val32 != 0x1)
 			val32 = 0x1;
 	} else {
-		if (domain_offset == *cold_domain_offset)
-			val32 = sbi_scratch_thishart_ptr()->next_mode;
+		val32 = root.next_mode;
 	}
 	dom->next_mode = val32;
 
@@ -490,37 +471,8 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
 		goto fail_free_all;
 	}
 
-	/* HART to domain assignment mask based on CPU DT nodes */
-	sbi_hartmask_clear_all(&assign_mask);
-	fdt_for_each_subnode(cpu_offset, fdt, cpus_offset) {
-		err = fdt_parse_hart_id(fdt, cpu_offset, &val32);
-		if (err)
-			continue;
-
-		if (SBI_HARTMASK_MAX_BITS <= sbi_hartid_to_hartindex(val32))
-			continue;
-
-		if (!fdt_node_is_enabled(fdt, cpu_offset))
-			continue;
-
-		/* This is an optional property */
-		val = fdt_getprop(fdt, cpu_offset, "opensbi-domain", &len);
-		if (!val || len < 4)
-			continue;
-
-		/* However, it should be valid if specified */
-		doffset = fdt_node_offset_by_phandle(fdt, fdt32_to_cpu(*val));
-		if (doffset < 0) {
-			err = doffset;
-			goto fail_free_all;
-		}
-
-		if (doffset == domain_offset)
-			sbi_hartmask_set_hartid(val32, &assign_mask);
-	}
-
 	/* Register the domain */
-	err = sbi_domain_register(dom, &assign_mask);
+	err = sbi_domain_register(dom);
 	if (err)
 		goto fail_free_all;
 
@@ -537,43 +489,10 @@ fail_free_domain:
 
 int fdt_domains_populate(const void *fdt)
 {
-	const u32 *val;
-	int cold_domain_offset;
-	u32 hartid, cold_hartid;
-	int err, len, cpus_offset, cpu_offset;
-
 	/* Sanity checks */
 	if (!fdt)
 		return SBI_EINVAL;
 
-	/* Find /cpus DT node */
-	cpus_offset = fdt_path_offset(fdt, "/cpus");
-	if (cpus_offset < 0)
-		return cpus_offset;
-
-	/* Find coldboot HART domain DT node offset */
-	cold_domain_offset = -1;
-	cold_hartid = current_hartid();
-	fdt_for_each_subnode(cpu_offset, fdt, cpus_offset) {
-		err = fdt_parse_hart_id(fdt, cpu_offset, &hartid);
-		if (err)
-			continue;
-
-		if (hartid != cold_hartid)
-			continue;
-
-		if (!fdt_node_is_enabled(fdt, cpu_offset))
-			continue;
-
-		val = fdt_getprop(fdt, cpu_offset, "opensbi-domain", &len);
-		if (val && len >= 4)
-			cold_domain_offset = fdt_node_offset_by_phandle(fdt,
-							   fdt32_to_cpu(*val));
-
-		break;
-	}
-
 	/* Iterate over each domain in FDT and populate details */
-	return fdt_iterate_each_domain_ro(fdt, &cold_domain_offset,
-					  __fdt_parse_domain);
+	return fdt_iterate_each_domain_ro(fdt, NULL, __fdt_parse_domain);
 }
-- 
2.43.0


-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

[PATCH v2 3/3] lib: sbi_domain: Introduce domain intialization order

[Anup Patel]

Currently, the domain initialization order is implied by the order
in which domains are populated by sbi_platform_domains_init() from
sbi_domain_finalize(). This is not documented anywhere and forces
unecessary ordering between domain DT nodes.

To address the above, introduce per-domain 32-bit integer to represent
intialization order (aka "init_order") where domain with a lower
initialization order will be booted first and two domains must not
have same initialization order. For DT based domain creation, new
"init-order" DT property can be used in domain DT node to specify
the initialization order. The ROOT domain is assumed to have lowest
initialization order (aka 0xffffffff).

Signed-off-by: Anup Patel <anup.patel@oss.qualcomm.com>
Reviewed-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
Tested-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
---
 docs/domain_support.md       |  5 +++++
 include/sbi/sbi_domain.h     |  2 ++
 lib/sbi/sbi_domain.c         | 21 ++++++++++++++-------
 lib/sbi/sbi_domain_context.c | 17 +++++++++--------
 lib/utils/fdt/fdt_domain.c   |  8 ++++++++
 5 files changed, 38 insertions(+), 15 deletions(-)

diff --git a/docs/domain_support.md b/docs/domain_support.md
index 82f155e1..655bf474 100644
--- a/docs/domain_support.md
+++ b/docs/domain_support.md
@@ -39,6 +39,9 @@ has following details:
 
 * **index** - Logical index of this domain
 * **name** - Name of this domain
+* **init_order** - Initialization order of this domain. Domain with a
+  lower initialization order will be booted first and two domains must
+  not have same initialization order.
 * **assigned_harts** - HARTs assigned to this domain
 * **possible_harts** - HARTs possible in this domain
 * **regions** - Array of memory regions terminated by a memory region
@@ -75,6 +78,8 @@ following manner:
 
 * **index** - Logical index of the ROOT domain is always zero
 * **name** - Name of the ROOT domain is "root"
+* **init_order** - Initialization order of the ROOT domain is always
+  0xffffffff (aka maximum possible 32-bit value)
 * **assigned_harts** - At boot-time all valid HARTs of a RISC-V platform
   are assigned the ROOT domain which changes later based on OpenSBI
   platform support
diff --git a/include/sbi/sbi_domain.h b/include/sbi/sbi_domain.h
index b6731c04..b35ea99f 100644
--- a/include/sbi/sbi_domain.h
+++ b/include/sbi/sbi_domain.h
@@ -199,6 +199,8 @@ struct sbi_domain {
 	spinlock_t assigned_harts_lock;
 	/** Name of this domain */
 	char name[64];
+	/** Initialization order of this domain */
+	u32 init_order;
 	/** Possible HARTs in this domain */
 	const struct sbi_hartmask *possible_harts;
 	/** Array of memory regions terminated by a region with order zero */
diff --git a/lib/sbi/sbi_domain.c b/lib/sbi/sbi_domain.c
index af753459..747c346c 100644
--- a/lib/sbi/sbi_domain.c
+++ b/lib/sbi/sbi_domain.c
@@ -28,6 +28,7 @@ static bool domain_finalized = false;
 
 struct sbi_domain root = {
 	.name = "root",
+	.init_order = -1U,
 	.possible_harts = NULL,
 	.regions = NULL,
 	.system_reset_allowed = true,
@@ -537,6 +538,9 @@ void sbi_domain_dump(const struct sbi_domain *dom, const char *suffix)
 	sbi_printf("Domain%d Name        %s: %s\n",
 		   dom->index, suffix, dom->name);
 
+	sbi_printf("Domain%d Init Order  %s: 0x%x\n",
+		   dom->index, suffix, dom->init_order);
+
 	sbi_printf("Domain%d Boot HART   %s: %d\n",
 		   dom->index, suffix, dom->boot_hartid);
 
@@ -635,9 +639,14 @@ int sbi_domain_register(struct sbi_domain *dom)
 	if (!dom || domain_finalized)
 		return SBI_EINVAL;
 
-	/* Check if domain already discovered */
+	/*
+	 * Ensure that:
+	 *  1) Domain not already registered
+	 *  2) Initialization order is unique
+	 */
 	sbi_domain_for_each(tdom) {
-		if (tdom == dom)
+		if (tdom == dom ||
+		    tdom->init_order == dom->init_order)
 			return SBI_EALREADY;
 	}
 
@@ -662,15 +671,13 @@ int sbi_domain_register(struct sbi_domain *dom)
 	sbi_hartmask_clear_all(&dom->assigned_harts);
 
 	/*
-	 * Assign a non-ROOT domain to a HART on first come first serve
-	 * basis if the HART is listed as a possible HART of the non-ROOT
-	 * domain. If no non-ROOT domain list a HART as possible HART then
-	 * the HART is assigned to the ROOT domain.
+	 * Assign HART to a domain with the least initialization order
+	 * where the HART is listed as a possible HART of the domain.
 	 */
 	sbi_hartmask_for_each_hartindex(i, dom->possible_harts) {
 		tdom = sbi_hartindex_to_domain(i);
 		if (tdom) {
-			if (tdom == &root)
+			if (tdom->init_order > dom->init_order)
 				sbi_hartmask_clear_hartindex(i, &tdom->assigned_harts);
 			else
 				continue;
diff --git a/lib/sbi/sbi_domain_context.c b/lib/sbi/sbi_domain_context.c
index 837ff54f..2b504300 100644
--- a/lib/sbi/sbi_domain_context.c
+++ b/lib/sbi/sbi_domain_context.c
@@ -291,28 +291,29 @@ int sbi_domain_context_exit(void)
 	}
 
 	dom_ctx = ctx->prev_ctx;
+	ctx->prev_ctx = NULL;
 
 	/* If no previous caller context */
 	if (!dom_ctx) {
-		/* Try to find next uninitialized user-defined domain's context */
+		/* Try to find next uninitialized domain with least initialization order */
+		dom_ctx = NULL;
 		sbi_domain_for_each(dom) {
-			if (dom == &root || dom == sbi_domain_thishart_ptr())
+			if (dom == sbi_domain_thishart_ptr())
 				continue;
 
 			if (!sbi_hartmask_test_hartindex(hartindex, dom->possible_harts))
 				continue;
 
 			tmp = hart_context_get(dom, hartindex);
-			if (tmp && !tmp->initialized) {
+			if (tmp && tmp->initialized)
+				continue;
+
+			if (!dom_ctx || tmp->dom->init_order < dom_ctx->dom->init_order)
 				dom_ctx = tmp;
-				break;
-			}
 		}
 	}
-
-	/* Take the root domain context if fail to find */
 	if (!dom_ctx)
-		dom_ctx = hart_context_get(&root, hartindex);
+		return SBI_ENOENT;
 
 	return switch_to_next_domain_context(ctx, dom_ctx);
 }
diff --git a/lib/utils/fdt/fdt_domain.c b/lib/utils/fdt/fdt_domain.c
index 3857831d..a649330c 100644
--- a/lib/utils/fdt/fdt_domain.c
+++ b/lib/utils/fdt/fdt_domain.c
@@ -330,6 +330,14 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
 		    sizeof(dom->name));
 	dom->name[sizeof(dom->name) - 1] = '\0';
 
+	/* Read initalization order */
+	val = fdt_getprop(fdt, domain_offset, "init-order", &len);
+	len = len / sizeof(u32);
+	if (val && len)
+		dom->init_order = fdt32_to_cpu(val[0]);
+	else
+		dom->init_order = (u32)domain_offset;
+
 	/* Setup possible HARTs mask */
 	SBI_HARTMASK_INIT(mask);
 	dom->possible_harts = mask;
-- 
2.43.0


-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Re: [PATCH v2 0/3] OpenSBI domain initialization order

[Pawandeep Oza]

On Mon, Aug 17, 2026 at 7:32 AM Anup Patel <anup.patel@oss.qualcomm.com> wrote:
>
> Currently, we rely on the "opensbi-domain" DT property of CPU DT nodes
> to select the domain assigned to each HART at boot-time but unfortunately
> this is only suitable if we have two non-ROOT domains (aka trused and
> non-trusted. For more than two non-ROOT domains, the domain initialization
> order should be much more predictable where users should define the exact
> order via some DT property under domain DT nodes.
>
> These patches can also be found in sbi_domain_init_order_v2 branch
> at: https://github.com/avpatel/opensbi.git
>
> Changes since v1:
> - Removed unwanted "break" from the loop in sbi_domain_context_exit()
> - Use root values when "next-arg1", "next-addr" and "next-mode" DT
>   properties are absent in the domain DT node.
>
> Anup Patel (3):
>   lib: sbi_domain: Check possible harts in
>     sbi_domain_context_enter/exit()
>   lib: sbi_domain: Re-work boot-time assignment of HARTs to non-ROOT
>     domains
>   lib: sbi_domain: Introduce domain intialization order
>
>  docs/domain_support.md             |  25 ++++---
>  include/sbi/sbi_domain.h           |   6 +-
>  include/sbi_utils/fdt/fdt_domain.h |   3 +-
>  lib/sbi/sbi_domain.c               |  45 ++++++++-----
>  lib/sbi/sbi_domain_context.c       |  31 ++++++---
>  lib/utils/fdt/fdt_domain.c         | 103 +++++------------------------
>  6 files changed, 79 insertions(+), 134 deletions(-)
>
> --
> 2.43.0
>
>
> --
> opensbi mailing list
> opensbi@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/opensbi

LGTM

Reviewed-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
Tested-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>

-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Re: [PATCH v2 1/3] lib: sbi_domain: Check possible harts in sbi_domain_context_enter/exit()

[Yu-Chien Peter Lin]

On Mon, Aug 17, 2026 at 08:00:23PM +0530, Anup Patel wrote:
> When context switching to a domain the current hart MUST be part
> of the possible harts of that domain. Add appropriate checks in
> sbi_domain_context_enter/exit() along these lines.
> 
> Signed-off-by: Anup Patel <anup.patel@oss.qualcomm.com>
> Reviewed-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
> Tested-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>

Reviewed-by: Yu-Chien Peter Lin <peter.lin@sifive.com>

-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Re: [PATCH v2 3/3] lib: sbi_domain: Introduce domain intialization order

[Yu-Chien Peter Lin]

Hi Anup,

On Mon, Aug 17, 2026 at 08:00:25PM +0530, Anup Patel wrote:
> Currently, the domain initialization order is implied by the order
> in which domains are populated by sbi_platform_domains_init() from
> sbi_domain_finalize(). This is not documented anywhere and forces
> unecessary ordering between domain DT nodes.
> 
> To address the above, introduce per-domain 32-bit integer to represent
> intialization order (aka "init_order") where domain with a lower
> initialization order will be booted first and two domains must not
> have same initialization order. For DT based domain creation, new
> "init-order" DT property can be used in domain DT node to specify
> the initialization order. The ROOT domain is assumed to have lowest
> initialization order (aka 0xffffffff).
> 
> Signed-off-by: Anup Patel <anup.patel@oss.qualcomm.com>
> Reviewed-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
> Tested-by: Pawandeep Oza <pawandeep.oza@oss.qualcomm.com>
> ---
>  docs/domain_support.md       |  5 +++++
>  include/sbi/sbi_domain.h     |  2 ++
>  lib/sbi/sbi_domain.c         | 21 ++++++++++++++-------
>  lib/sbi/sbi_domain_context.c | 17 +++++++++--------
>  lib/utils/fdt/fdt_domain.c   |  8 ++++++++
>  5 files changed, 38 insertions(+), 15 deletions(-)
> 
> diff --git a/docs/domain_support.md b/docs/domain_support.md
> index 82f155e1..655bf474 100644
> --- a/docs/domain_support.md
> +++ b/docs/domain_support.md
> @@ -39,6 +39,9 @@ has following details:
>  
>  * **index** - Logical index of this domain
>  * **name** - Name of this domain
> +* **init_order** - Initialization order of this domain. Domain with a
> +  lower initialization order will be booted first and two domains must
> +  not have same initialization order.

Maybe the init-order default behavior (FDT offset fallback) is
worth noting here.

Regards,
Peter Lin

-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi

Re: [PATCH v2 2/3] lib: sbi_domain: Re-work boot-time assignment of HARTs to non-ROOT domains

[Yu-Chien Peter Lin]

Hi Anup,

On Mon, Aug 17, 2026 at 08:00:24PM +0530, Anup Patel wrote:
> Assign a non-ROOT domain to a HART on first come first serve basis if the
> HART is listed as a possible HART of the non-ROOT domain. If no non-ROOT
> domain list a HART as possible HART then the HART is assigned to the ROOT
> domain.
> 
> This allows us to drop the OpenSBI specific DT property from each CPU DT
> node (aka "opensbi-domain" Dt property).
> 
> Signed-off-by: Anup Patel <anup.patel@oss.qualcomm.com>
> ---
>  docs/domain_support.md             | 20 +++----
>  include/sbi/sbi_domain.h           |  4 +-
>  include/sbi_utils/fdt/fdt_domain.h |  3 +-
>  lib/sbi/sbi_domain.c               | 34 ++++++-----
>  lib/utils/fdt/fdt_domain.c         | 95 +++---------------------------
>  5 files changed, 35 insertions(+), 121 deletions(-)
> 
> diff --git a/docs/domain_support.md b/docs/domain_support.md
> index e267a9f7..82f155e1 100644
> --- a/docs/domain_support.md
> +++ b/docs/domain_support.md
> @@ -207,15 +207,14 @@ The DT properties of a domain instance DT node are as follows:
>  
>  ### Assigning HART To Domain Instance
>  
> -By default, all HARTs are assigned to **the ROOT domain**. The OpenSBI
> -platform support can provide the HART to domain instance assignment using
> -platform specific callback.
> +At boot-time, a HART is assigned to a non-ROOT domain on first come
> +first serve basis if the HART is listed as a possible HART of the
> +non-ROOT domain. If no non-ROOT domain list a HART as possible HART
> +then the HART is assigned to **the ROOT domain**.
>  
> -The HART to domain instance assignment can be parsed from the device tree
> -using optional DT property **opensbi-domain** in each CPU DT node. The
> -value of DT property **opensbi-domain** is the DT phandle of the domain
> -instance DT node. If **opensbi-domain** DT property is not specified then
> -corresponding HART is assigned to **the ROOT domain**.
> +At runtime, the assignment of a HART can change from one domain to
> +another domain as long as the HART is listed in possible HARTs of
> +both domains.
>  
>  ### Domain Configuration Only Accessible to OpenSBI
>  
> @@ -289,7 +288,6 @@ be done:
>              device_type = "cpu";
>              reg = <0x00>;
>              compatible = "riscv";
> -            opensbi-domain = <&tdomain>;
>              ...
>          };
>  
> @@ -297,7 +295,6 @@ be done:
>              device_type = "cpu";
>              reg = <0x01>;
>              compatible = "riscv";
> -            opensbi-domain = <&udomain>;
>              ...
>          };
>  
> @@ -305,7 +302,6 @@ be done:
>              device_type = "cpu";
>              reg = <0x02>;
>              compatible = "riscv";
> -            opensbi-domain = <&udomain>;
>              ...
>          };
>  
> @@ -313,7 +309,6 @@ be done:
>              device_type = "cpu";
>              reg = <0x03>;
>              compatible = "riscv";
> -            opensbi-domain = <&udomain>;
>              ...
>          };
>  
> @@ -321,7 +316,6 @@ be done:
>              device_type = "cpu";
>              reg = <0x04>;
>              compatible = "riscv";
> -            opensbi-domain = <&udomain>;
>              ...
>          };
>      };
> diff --git a/include/sbi/sbi_domain.h b/include/sbi/sbi_domain.h
> index 16edd4ce..b6731c04 100644
> --- a/include/sbi/sbi_domain.h
> +++ b/include/sbi/sbi_domain.h
> @@ -328,12 +328,10 @@ void sbi_domain_dump_all(const char *suffix);
>  /**
>   * Register a new domain
>   * @param dom pointer to domain
> - * @param assign_mask pointer to HART mask of HARTs assigned to the domain
>   *
>   * @return 0 on success and negative error code on failure
>   */
> -int sbi_domain_register(struct sbi_domain *dom,
> -			const struct sbi_hartmask *assign_mask);
> +int sbi_domain_register(struct sbi_domain *dom);
>  
>  /**
>   * Add a memory range with its flags to the root domain
> diff --git a/include/sbi_utils/fdt/fdt_domain.h b/include/sbi_utils/fdt/fdt_domain.h
> index 8c2dee09..a366b271 100644
> --- a/include/sbi_utils/fdt/fdt_domain.h
> +++ b/include/sbi_utils/fdt/fdt_domain.h
> @@ -50,8 +50,7 @@ int fdt_iterate_each_memregion(void *fdt, int domain_offset, void *opaque,
>   *
>   * This routine:
>   * 1. Disables MMIO devices not accessible to the coldboot HART domain
> - * 2. Removes "opensbi-domain" DT property from CPU DT nodes
> - * 3. Removes domain configuration DT node under /chosen DT node
> + * 2. Removes domain configuration DT node under /chosen DT node
>   *
>   * It is recommended that platform support call this function in
>   * their final_init() platform operation.
> diff --git a/lib/sbi/sbi_domain.c b/lib/sbi/sbi_domain.c
> index fa69170b..af753459 100644
> --- a/lib/sbi/sbi_domain.c
> +++ b/lib/sbi/sbi_domain.c
> @@ -625,16 +625,14 @@ void sbi_domain_dump_all(const char *suffix)
>  	}
>  }
>  
> -int sbi_domain_register(struct sbi_domain *dom,
> -			const struct sbi_hartmask *assign_mask)
> +int sbi_domain_register(struct sbi_domain *dom)
>  {
> -	u32 i;
> -	int rc;
> +	u32 i, cold_hartid = current_hartid();
>  	struct sbi_domain *tdom;
> -	u32 cold_hartid = current_hartid();
> +	int rc;
>  
>  	/* Sanity checks */
> -	if (!dom || !assign_mask || domain_finalized)
> +	if (!dom || domain_finalized)
>  		return SBI_EINVAL;
>  
>  	/* Check if domain already discovered */
> @@ -663,15 +661,21 @@ int sbi_domain_register(struct sbi_domain *dom,
>  	/* Clear assigned HARTs of domain */
>  	sbi_hartmask_clear_all(&dom->assigned_harts);
>  
> -	/* Assign domain to HART if HART is a possible HART */
> -	sbi_hartmask_for_each_hartindex(i, assign_mask) {
> -		if (!sbi_hartmask_test_hartindex(i, dom->possible_harts))
> -			continue;
> -
> +	/*
> +	 * Assign a non-ROOT domain to a HART on first come first serve
> +	 * basis if the HART is listed as a possible HART of the non-ROOT
> +	 * domain. If no non-ROOT domain list a HART as possible HART then
> +	 * the HART is assigned to the ROOT domain.
> +	 */
> +	sbi_hartmask_for_each_hartindex(i, dom->possible_harts) {
>  		tdom = sbi_hartindex_to_domain(i);
> -		if (tdom)
> -			sbi_hartmask_clear_hartindex(i,
> -					&tdom->assigned_harts);
> +		if (tdom) {
> +			if (tdom == &root)
> +				sbi_hartmask_clear_hartindex(i, &tdom->assigned_harts);
> +			else
> +				continue;
> +		}
> +
>  		sbi_update_hartindex_to_domain(i, dom);
>  		sbi_hartmask_set_hartindex(i, &dom->assigned_harts);
>  
> @@ -975,7 +979,7 @@ int sbi_domain_init(struct sbi_scratch *scratch, u32 cold_hartid)
>  		sbi_hartmask_set_hartindex(i, root_hmask);
>  
>  	/* Finally register the root domain */
> -	rc = sbi_domain_register(&root, root_hmask);
> +	rc = sbi_domain_register(&root);
>  	if (rc)
>  		goto fail_free_root_hmask;
>  
> diff --git a/lib/utils/fdt/fdt_domain.c b/lib/utils/fdt/fdt_domain.c
> index 61627db3..3857831d 100644
> --- a/lib/utils/fdt/fdt_domain.c
> +++ b/lib/utils/fdt/fdt_domain.c
> @@ -170,26 +170,11 @@ static int __fixup_disable_devices(void *fdt, int doff, int roff,
>  
>  void fdt_domain_fixup(void *fdt)
>  {
> -	u32 i, dcount;
> +	u32 dcount;
>  	int err, poffset, doffset;
>  	struct sbi_domain *dom = sbi_domain_thishart_ptr();
>  	struct __fixup_find_domain_offset_info fdo;
>  
> -	/* Remove the domain assignment DT property from CPU DT nodes */
> -	poffset = fdt_path_offset(fdt, "/cpus");
> -	if (poffset < 0)
> -		return;
> -	fdt_for_each_subnode(doffset, fdt, poffset) {
> -		err = fdt_parse_hart_id(fdt, doffset, &i);
> -		if (err)
> -			continue;
> -
> -		if (!fdt_node_is_enabled(fdt, doffset))
> -			continue;
> -
> -		fdt_nop_property(fdt, doffset, "opensbi-domain");
> -	}
> -
>  	/* Skip device disable for root domain */
>  	if (!dom->index)
>  		goto skip_device_disable;
> @@ -315,12 +300,10 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
>  	const char *inherit;
>  	struct sbi_domain *dom;
>  	struct sbi_hartmask *mask;
> -	struct sbi_hartmask assign_mask;
>  	struct parse_region_data preg;
> -	int *cold_domain_offset = opaque;
>  	struct sbi_domain_memregion *reg;
>  	int inheritance_mode = FDT_ROOT_REGION_INHERIT_M_ONLY;
> -	int i, err = 0, len, cpus_offset, cpu_offset, doffset;
> +	int i, err = 0, len, cpus_offset, cpu_offset;
>  
>  	dom = sbi_zalloc(sizeof(*dom));
>  	if (!dom)
> @@ -440,7 +423,7 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
>  		val64 = fdt32_to_cpu(val[0]);
>  		val64 = (val64 << 32) | fdt32_to_cpu(val[1]);
>  	} else {
> -		val64 = sbi_scratch_thishart_ptr()->next_arg1;
> +		val64 = root.next_arg1;
>  	}
>  	dom->next_arg1 = val64;
>  
> @@ -451,8 +434,7 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
>  		val64 = fdt32_to_cpu(val[0]);
>  		val64 = (val64 << 32) | fdt32_to_cpu(val[1]);
>  	} else {
> -		if (domain_offset == *cold_domain_offset)
> -			val64 = sbi_scratch_thishart_ptr()->next_addr;
> +		val64 = root.next_addr;
>  	}
>  	dom->next_addr = val64;
>  
> @@ -464,8 +446,7 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
>  		if (val32 != 0x0 && val32 != 0x1)
>  			val32 = 0x1;
>  	} else {
> -		if (domain_offset == *cold_domain_offset)
> -			val32 = sbi_scratch_thishart_ptr()->next_mode;
> +		val32 = root.next_mode;
>  	}
>  	dom->next_mode = val32;

We may update domain_support.md accordingly to reflect these changes.
https://github.com/riscv-software-src/opensbi/blob/4e79fd7de59f1b2899092c1a84ce68c8ebc68f93/docs/domain_support.md?plain=1#L186-L202

>  
> @@ -490,37 +471,8 @@ static int __fdt_parse_domain(const void *fdt, int domain_offset, void *opaque)
>  		goto fail_free_all;
>  	}
>  
> -	/* HART to domain assignment mask based on CPU DT nodes */
> -	sbi_hartmask_clear_all(&assign_mask);
> -	fdt_for_each_subnode(cpu_offset, fdt, cpus_offset) {

cpus_offset is unused now in __fdt_parse_domain().

Best regards,
Peter Lin

> -		err = fdt_parse_hart_id(fdt, cpu_offset, &val32);
> -		if (err)
> -			continue;
> -
> -		if (SBI_HARTMASK_MAX_BITS <= sbi_hartid_to_hartindex(val32))
> -			continue;
> -
> -		if (!fdt_node_is_enabled(fdt, cpu_offset))
> -			continue;
> -
> -		/* This is an optional property */
> -		val = fdt_getprop(fdt, cpu_offset, "opensbi-domain", &len);
> -		if (!val || len < 4)
> -			continue;
> -
> -		/* However, it should be valid if specified */
> -		doffset = fdt_node_offset_by_phandle(fdt, fdt32_to_cpu(*val));
> -		if (doffset < 0) {
> -			err = doffset;
> -			goto fail_free_all;
> -		}
> -
> -		if (doffset == domain_offset)
> -			sbi_hartmask_set_hartid(val32, &assign_mask);
> -	}
> -
>  	/* Register the domain */
> -	err = sbi_domain_register(dom, &assign_mask);
> +	err = sbi_domain_register(dom);
>  	if (err)
>  		goto fail_free_all;
>  
> @@ -537,43 +489,10 @@ fail_free_domain:
>  
>  int fdt_domains_populate(const void *fdt)
>  {
> -	const u32 *val;
> -	int cold_domain_offset;
> -	u32 hartid, cold_hartid;
> -	int err, len, cpus_offset, cpu_offset;
> -
>  	/* Sanity checks */
>  	if (!fdt)
>  		return SBI_EINVAL;
>  
> -	/* Find /cpus DT node */
> -	cpus_offset = fdt_path_offset(fdt, "/cpus");
> -	if (cpus_offset < 0)
> -		return cpus_offset;
> -
> -	/* Find coldboot HART domain DT node offset */
> -	cold_domain_offset = -1;
> -	cold_hartid = current_hartid();
> -	fdt_for_each_subnode(cpu_offset, fdt, cpus_offset) {
> -		err = fdt_parse_hart_id(fdt, cpu_offset, &hartid);
> -		if (err)
> -			continue;
> -
> -		if (hartid != cold_hartid)
> -			continue;
> -
> -		if (!fdt_node_is_enabled(fdt, cpu_offset))
> -			continue;
> -
> -		val = fdt_getprop(fdt, cpu_offset, "opensbi-domain", &len);
> -		if (val && len >= 4)
> -			cold_domain_offset = fdt_node_offset_by_phandle(fdt,
> -							   fdt32_to_cpu(*val));
> -
> -		break;
> -	}
> -
>  	/* Iterate over each domain in FDT and populate details */
> -	return fdt_iterate_each_domain_ro(fdt, &cold_domain_offset,
> -					  __fdt_parse_domain);
> +	return fdt_iterate_each_domain_ro(fdt, NULL, __fdt_parse_domain);
>  }

-- 
opensbi mailing list
opensbi@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/opensbi