[PATCH v4 0/2] of: populate of_root_node if not set (alternate)

[PATCH v4 0/2] of: populate of_root_node if not set (alternate)

[Frank Rowand]

This series is a different implementation to achieve the goals of
https://lore.kernel.org/r/20220623105044.152832-1-clement.leger@bootlin.com

In order to apply overlays or create new nodes under the root node, the
kernel expects of_root to be set. On some system where a device-tree was
not provided by firmware (x86 for instance) if CONFIG_OF is enabled,
then we will end up with a null of_root. This series adds support to
create this root node using a builtin dtb and removes the manual
creation of the root node done in unittests.c.

Changes since version 3: (all)
  - also tested on UML (previously only tested on arm)

Changes since version 3: (patch 1/2)
  - refresh for 6.3-rc1
  - unflatten_device_tree() - calculate of_fdt_crc32 if setting
    initial_boot_params to __dtb_empty_root_begin so CRC check
    in of_fdt_raw_init() will not fail

Changes since version 3: (patch 2/2)
  - refresh for 6.3-rc1
  - remove the CONFIG_UML case of populating the devicetree
  - unittest_data_add() - move an EXPECT_BEGIN() to after an error
    check that can result in an early return

Changes since version 2: (patch 1/2)
  - change of __dtb_empty_root_* from "void *" to "uint8_t []"

Changes since version 1: (patch 1/2)
  - refresh for 6.2-rc1
  - update Signed-off-by
  - fix typo in of_fdt.h: s/of_setup/setup_of
  - unflatten_device_tree(): validate size in header field dtb_empty_root
    that will be used to copy dtb_empty_root
  - add Kconfig option to manually select CONFIG_OF_EARLY_FLATTREE

Changes since version 1: (patch 2/2)
  - refresh for 6.2-rc1
  - update Signed-off-by
  - fix formatting error (leading space) in patch comment



Frank Rowand (2):
  of: create of_root if no dtb provided
  of: unittest: treat missing of_root as error instead of fixing up

 drivers/of/Kconfig        |  7 ++++++-
 drivers/of/Makefile       |  2 +-
 drivers/of/empty_root.dts |  6 ++++++
 drivers/of/fdt.c          | 29 ++++++++++++++++++++++++++++-
 drivers/of/unittest.c     | 16 ++++------------
 include/linux/of_fdt.h    |  2 ++
 init/main.c               |  2 ++
 7 files changed, 49 insertions(+), 15 deletions(-)
 create mode 100644 drivers/of/empty_root.dts

-- 
Frank Rowand <frowand.list@gmail.com>

[PATCH v4 1/2] of: create of_root if no dtb provided

[Frank Rowand]

When enabling CONFIG_OF on a platform where of_root is not populated by
firmware, we end up without a root node. In order to apply overlays and
create subnodes of the root node, we need one. Create this root node
by unflattening an empty builtin dtb.

If firmware provides a flattened device tree (FDT) then the FDT is
unflattened via setup_arch().  Otherwise setup_of(), which is called
immediately after setup_arch(), will create the default root node
if it does not exist.

Signed-off-by: Frank Rowand <frowand.list@gmail.com>
---

Please wait for test results from Clement before accepting.

Changes since version 3:
  - refresh for 6.3-rc1
  - unflatten_device_tree() - calculate of_fdt_crc32 if setting
    initial_boot_params to __dtb_empty_root_begin so CRC check
    in of_fdt_raw_init() will not fail

Changes since version 2:
  - change of __dtb_empty_root_* from "void *" to "uint8_t []"

Changes since version 1:
  - refresh for 6.2-rc1
  - update Signed-off-by
  - fix typo in of_fdt.h: s/of_setup/setup_of
  - unflatten_device_tree(): validate size in header field dtb_empty_root
    that will be used to copy dtb_empty_root
  - add Kconfig option to manually select CONFIG_OF_EARLY_FLATTREE

 drivers/of/Kconfig        |  7 ++++++-
 drivers/of/Makefile       |  2 +-
 drivers/of/empty_root.dts |  6 ++++++
 drivers/of/fdt.c          | 29 ++++++++++++++++++++++++++++-
 include/linux/of_fdt.h    |  2 ++
 init/main.c               |  2 ++
 6 files changed, 45 insertions(+), 3 deletions(-)
 create mode 100644 drivers/of/empty_root.dts

diff --git a/drivers/of/Kconfig b/drivers/of/Kconfig
index 644386833a7b..194e090ceee8 100644
--- a/drivers/of/Kconfig
+++ b/drivers/of/Kconfig
@@ -54,9 +54,14 @@ config OF_FLATTREE
 	select CRC32
 
 config OF_EARLY_FLATTREE
-	bool
+	bool "Functions for accessing Flat Devicetree (FDT) early in boot"
 	select DMA_DECLARE_COHERENT if HAS_DMA
 	select OF_FLATTREE
+	help
+	  Normally selected by platforms that process an FDT that has been
+	  passed to the kernel by the bootloader.  If the bootloader does not
+	  pass an FDT to the kernel and you need an empty devicetree that
+	  contains only a root node to exist, then say Y here.
 
 config OF_PROMTREE
 	bool
diff --git a/drivers/of/Makefile b/drivers/of/Makefile
index e0360a44306e..cbae92c5ed02 100644
--- a/drivers/of/Makefile
+++ b/drivers/of/Makefile
@@ -2,7 +2,7 @@
 obj-y = base.o device.o platform.o property.o
 obj-$(CONFIG_OF_KOBJ) += kobj.o
 obj-$(CONFIG_OF_DYNAMIC) += dynamic.o
-obj-$(CONFIG_OF_FLATTREE) += fdt.o
+obj-$(CONFIG_OF_FLATTREE) += fdt.o empty_root.dtb.o
 obj-$(CONFIG_OF_EARLY_FLATTREE) += fdt_address.o
 obj-$(CONFIG_OF_PROMTREE) += pdt.o
 obj-$(CONFIG_OF_ADDRESS)  += address.o
diff --git a/drivers/of/empty_root.dts b/drivers/of/empty_root.dts
new file mode 100644
index 000000000000..cf9e97a60f48
--- /dev/null
+++ b/drivers/of/empty_root.dts
@@ -0,0 +1,6 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/dts-v1/;
+
+/ {
+
+};
diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
index d1a68b6d03b3..f65016c2e79f 100644
--- a/drivers/of/fdt.c
+++ b/drivers/of/fdt.c
@@ -32,6 +32,13 @@
 
 #include "of_private.h"
 
+/*
+ * __dtb_empty_root_begin[] and __dtb_empty_root_end[] magically created by
+ * cmd_dt_S_dtb in scripts/Makefile.lib
+ */
+extern uint8_t __dtb_empty_root_begin[];
+extern uint8_t __dtb_empty_root_end[];
+
 /*
  * of_fdt_limit_memory - limit the number of regions in the /memory node
  * @limit: maximum entries
@@ -1326,8 +1333,21 @@ bool __init early_init_dt_scan(void *params)
  */
 void __init unflatten_device_tree(void)
 {
-	__unflatten_device_tree(initial_boot_params, NULL, &of_root,
+	if (!initial_boot_params) {
+		initial_boot_params = (void *) __dtb_empty_root_begin;
+		/* fdt_totalsize() will be used for copy size */
+		if (fdt_totalsize(initial_boot_params) >
+		    __dtb_empty_root_end - __dtb_empty_root_begin) {
+			pr_err("invalid size in dtb_empty_root\n");
+			return;
+		}
+		of_fdt_crc32 = crc32_be(~0, initial_boot_params,
+					fdt_totalsize(initial_boot_params));
+		unflatten_and_copy_device_tree();
+	} else {
+		__unflatten_device_tree(initial_boot_params, NULL, &of_root,
 				early_init_dt_alloc_memory_arch, false);
+	}
 
 	/* Get pointer to "/chosen" and "/aliases" nodes for use everywhere */
 	of_alias_scan(early_init_dt_alloc_memory_arch);
@@ -1367,6 +1387,13 @@ void __init unflatten_and_copy_device_tree(void)
 	unflatten_device_tree();
 }
 
+void __init setup_of(void)
+{
+	/* if architecture did not unflatten devicetree, do it now */
+	if (!of_root)
+		unflatten_device_tree();
+}
+
 #ifdef CONFIG_SYSFS
 static ssize_t of_fdt_raw_read(struct file *filp, struct kobject *kobj,
 			       struct bin_attribute *bin_attr,
diff --git a/include/linux/of_fdt.h b/include/linux/of_fdt.h
index d69ad5bb1eb1..f0dc46d576da 100644
--- a/include/linux/of_fdt.h
+++ b/include/linux/of_fdt.h
@@ -81,6 +81,7 @@ extern const void *of_flat_dt_match_machine(const void *default_match,
 /* Other Prototypes */
 extern void unflatten_device_tree(void);
 extern void unflatten_and_copy_device_tree(void);
+extern void setup_of(void);
 extern void early_init_devtree(void *);
 extern void early_get_first_memblock_info(void *, phys_addr_t *);
 #else /* CONFIG_OF_EARLY_FLATTREE */
@@ -91,6 +92,7 @@ static inline void early_init_fdt_reserve_self(void) {}
 static inline const char *of_flat_dt_get_machine_name(void) { return NULL; }
 static inline void unflatten_device_tree(void) {}
 static inline void unflatten_and_copy_device_tree(void) {}
+static inline void setup_of(void) {}
 #endif /* CONFIG_OF_EARLY_FLATTREE */
 
 #endif /* __ASSEMBLY__ */
diff --git a/init/main.c b/init/main.c
index 4425d1783d5c..3c443e0c5927 100644
--- a/init/main.c
+++ b/init/main.c
@@ -101,6 +101,7 @@
 #include <linux/init_syscalls.h>
 #include <linux/stackdepot.h>
 #include <linux/randomize_kstack.h>
+#include <linux/of_fdt.h>
 #include <net/net_namespace.h>
 
 #include <asm/io.h>
@@ -961,6 +962,7 @@ asmlinkage __visible void __init __no_sanitize_address start_kernel(void)
 	pr_notice("%s", linux_banner);
 	early_security_init();
 	setup_arch(&command_line);
+	setup_of();
 	setup_boot_config();
 	setup_command_line(command_line);
 	setup_nr_cpu_ids();
-- 
Frank Rowand <frowand.list@gmail.com>

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Rob Herring]

On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> When enabling CONFIG_OF on a platform where of_root is not populated by
> firmware, we end up without a root node. In order to apply overlays and
> create subnodes of the root node, we need one. Create this root node
> by unflattening an empty builtin dtb.
> 
> If firmware provides a flattened device tree (FDT) then the FDT is
> unflattened via setup_arch().  Otherwise setup_of(), which is called
> immediately after setup_arch(), will create the default root node
> if it does not exist.

I thought of another way to handle this. Every arch except IIRC sparc, 
s390, and ia64 calls unflatten(_and_copy)?_device_tree already. At 
least any arch anyone is going to care about for this stuff does. It's 
just conditional in some cases. So why not make the existing calls 
unconditional?

Either way, I think that of_have_populated_dt() calls will need to be 
checked whether this change.

Rob

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Guenter Roeck]

Hi,

On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> When enabling CONFIG_OF on a platform where of_root is not populated by
> firmware, we end up without a root node. In order to apply overlays and
> create subnodes of the root node, we need one. Create this root node
> by unflattening an empty builtin dtb.
> 
> If firmware provides a flattened device tree (FDT) then the FDT is
> unflattened via setup_arch().  Otherwise setup_of(), which is called
> immediately after setup_arch(), will create the default root node
> if it does not exist.
> 
> Signed-off-by: Frank Rowand <frowand.list@gmail.com>

This patch results in a crash on nios2.

Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
------------
qemu log:
earlycon: uart8250 at MMIO32 0x18001600 (options '')
printk: legacy bootconsole [uart8250] enabled
Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---

Reverting this patch fixes the problem.

Guenter

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Rob Herring]

+Stephen

On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
>
> Hi,
>
> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> > When enabling CONFIG_OF on a platform where of_root is not populated by
> > firmware, we end up without a root node. In order to apply overlays and
> > create subnodes of the root node, we need one. Create this root node
> > by unflattening an empty builtin dtb.
> >
> > If firmware provides a flattened device tree (FDT) then the FDT is
> > unflattened via setup_arch().  Otherwise setup_of(), which is called
> > immediately after setup_arch(), will create the default root node
> > if it does not exist.
> >
> > Signed-off-by: Frank Rowand <frowand.list@gmail.com>
>
> This patch results in a crash on nios2.

This patch was never applied. I assume you meant a later version of it
that did get applied.

>
> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)

Booting with DT?

> ------------
> qemu log:
> earlycon: uart8250 at MMIO32 0x18001600 (options '')
> printk: legacy bootconsole [uart8250] enabled
> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---

nios2 looks utterly broken to me. This change should be a nop unless
initial_boot_params is NULL. It looks like it is possible for r6 (dtb
address) to be 0 depending on kconfig options, but that would have
skipped copying and unflattening which would then panic in
setup_cpuinfo(). If initial_boot_params is not NULL, then the same
early_init_dt_alloc_memory_arch() calls should fail when copying the
DT. So I don't see how nios2 booting with DT ever worked.

Rob

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Guenter Roeck]

On 3/18/24 12:26, Rob Herring wrote:
> +Stephen
> 
> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> Hi,
>>
>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
>>> When enabling CONFIG_OF on a platform where of_root is not populated by
>>> firmware, we end up without a root node. In order to apply overlays and
>>> create subnodes of the root node, we need one. Create this root node
>>> by unflattening an empty builtin dtb.
>>>
>>> If firmware provides a flattened device tree (FDT) then the FDT is
>>> unflattened via setup_arch().  Otherwise setup_of(), which is called
>>> immediately after setup_arch(), will create the default root node
>>> if it does not exist.
>>>
>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
>>
>> This patch results in a crash on nios2.
> 
> This patch was never applied. I assume you meant a later version of it
> that did get applied.
> 

It is the patch in the link that was provided with the patch. What else
should we use as reference ? FWIW, I did look for a more recent version,
but I must have missed it. My bad.

>>
>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
> 
> Booting with DT?
> 

Yes, with arch/nios2/boot/dts/10m50_devboard.dtb.

>> ------------
>> qemu log: >> earlycon: uart8250 at MMIO32 0x18001600 (options '')
>> printk: legacy bootconsole [uart8250] enabled
>> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
>> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
>> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---
> 
> nios2 looks utterly broken to me. This change should be a nop unless
> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
> address) to be 0 depending on kconfig options, but that would have
> skipped copying and unflattening which would then panic in
> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
> early_init_dt_alloc_memory_arch() calls should fail when copying the
> DT. So I don't see how nios2 booting with DT ever worked.
> 

All I can say that it did work with devicetree until this patch was
applied. It doesn't boot without it. I tried without devicetree file
after reverting this patch. That results in

Kernel panic - not syncing: setup_cpuinfo: No CPU found in devicetree!

Guenter

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Guenter Roeck]

On 3/18/24 12:26, Rob Herring wrote:
> +Stephen
> 
> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> Hi,
>>
>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
>>> When enabling CONFIG_OF on a platform where of_root is not populated by
>>> firmware, we end up without a root node. In order to apply overlays and
>>> create subnodes of the root node, we need one. Create this root node
>>> by unflattening an empty builtin dtb.
>>>
>>> If firmware provides a flattened device tree (FDT) then the FDT is
>>> unflattened via setup_arch().  Otherwise setup_of(), which is called
>>> immediately after setup_arch(), will create the default root node
>>> if it does not exist.
>>>
>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
>>
>> This patch results in a crash on nios2.
> 
> This patch was never applied. I assume you meant a later version of it
> that did get applied.
> 
>>
>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
> 
> Booting with DT?
> 
>> ------------
>> qemu log:
>> earlycon: uart8250 at MMIO32 0x18001600 (options '')
>> printk: legacy bootconsole [uart8250] enabled
>> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
>> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
>> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---
> 
> nios2 looks utterly broken to me. This change should be a nop unless
> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
> address) to be 0 depending on kconfig options, but that would have
> skipped copying and unflattening which would then panic in
> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
> early_init_dt_alloc_memory_arch() calls should fail when copying the
> DT. So I don't see how nios2 booting with DT ever worked.
> 

For nios2, in early_init_devtree():

void __init early_init_devtree(void *params)
{
         __be32 *dtb = (u32 *)__dtb_start;
	...
         if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
                 params = (void *)__dtb_start;

That worked fine until this patch. Starting with this patch, __dtb_start
always points to a valid empty devicetree blob, which overrides the
devicetree blob passed to early_init_devtree(). This causes the problem.

Guenter

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Rob Herring]

On Mon, Mar 18, 2024 at 4:31 PM Guenter Roeck <linux@roeck-us.net> wrote:
>
> On 3/18/24 12:26, Rob Herring wrote:
> > +Stephen
> >
> > On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >>
> >> Hi,
> >>
> >> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> >>> When enabling CONFIG_OF on a platform where of_root is not populated by
> >>> firmware, we end up without a root node. In order to apply overlays and
> >>> create subnodes of the root node, we need one. Create this root node
> >>> by unflattening an empty builtin dtb.
> >>>
> >>> If firmware provides a flattened device tree (FDT) then the FDT is
> >>> unflattened via setup_arch().  Otherwise setup_of(), which is called
> >>> immediately after setup_arch(), will create the default root node
> >>> if it does not exist.
> >>>
> >>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
> >>
> >> This patch results in a crash on nios2.
> >
> > This patch was never applied. I assume you meant a later version of it
> > that did get applied.
> >
> >>
> >> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
> >
> > Booting with DT?
> >
> >> ------------
> >> qemu log:
> >> earlycon: uart8250 at MMIO32 0x18001600 (options '')
> >> printk: legacy bootconsole [uart8250] enabled
> >> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
> >> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
> >> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---
> >
> > nios2 looks utterly broken to me. This change should be a nop unless
> > initial_boot_params is NULL. It looks like it is possible for r6 (dtb
> > address) to be 0 depending on kconfig options, but that would have
> > skipped copying and unflattening which would then panic in
> > setup_cpuinfo(). If initial_boot_params is not NULL, then the same
> > early_init_dt_alloc_memory_arch() calls should fail when copying the
> > DT. So I don't see how nios2 booting with DT ever worked.
> >
>
> For nios2, in early_init_devtree():
>
> void __init early_init_devtree(void *params)
> {
>          __be32 *dtb = (u32 *)__dtb_start;
>         ...
>          if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
>                  params = (void *)__dtb_start;
>
> That worked fine until this patch. Starting with this patch, __dtb_start
> always points to a valid empty devicetree blob, which overrides the
> devicetree blob passed to early_init_devtree(). This causes the problem.

With an external DTB, it doesn't boot with or without this patch. It
just dies in different spots. Before it just skipped any memory
allocs. With 'CONFIG_NIOS2_DTB_SOURCE="10m50_devboard.dts"', it boots
fine for me with or without this patch. This is what I'm running:

qemu-system-nios2 -kernel .build-nios2/vmlinux -nographic -dtb
.build-nios2/arch/nios2/boot/dts/10m50_devboard.dtb -append
"earlycon=uart8250,mmio32,0x18001600,115200n8"

Rob

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Guenter Roeck]

On 3/20/24 12:14, Rob Herring wrote:
> On Mon, Mar 18, 2024 at 4:31 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> On 3/18/24 12:26, Rob Herring wrote:
>>> +Stephen
>>>
>>> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
>>>>> When enabling CONFIG_OF on a platform where of_root is not populated by
>>>>> firmware, we end up without a root node. In order to apply overlays and
>>>>> create subnodes of the root node, we need one. Create this root node
>>>>> by unflattening an empty builtin dtb.
>>>>>
>>>>> If firmware provides a flattened device tree (FDT) then the FDT is
>>>>> unflattened via setup_arch().  Otherwise setup_of(), which is called
>>>>> immediately after setup_arch(), will create the default root node
>>>>> if it does not exist.
>>>>>
>>>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
>>>>
>>>> This patch results in a crash on nios2.
>>>
>>> This patch was never applied. I assume you meant a later version of it
>>> that did get applied.
>>>
>>>>
>>>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
>>>
>>> Booting with DT?
>>>
>>>> ------------
>>>> qemu log:
>>>> earlycon: uart8250 at MMIO32 0x18001600 (options '')
>>>> printk: legacy bootconsole [uart8250] enabled
>>>> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
>>>> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
>>>> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---
>>>
>>> nios2 looks utterly broken to me. This change should be a nop unless
>>> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
>>> address) to be 0 depending on kconfig options, but that would have
>>> skipped copying and unflattening which would then panic in
>>> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
>>> early_init_dt_alloc_memory_arch() calls should fail when copying the
>>> DT. So I don't see how nios2 booting with DT ever worked.
>>>
>>
>> For nios2, in early_init_devtree():
>>
>> void __init early_init_devtree(void *params)
>> {
>>           __be32 *dtb = (u32 *)__dtb_start;
>>          ...
>>           if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
>>                   params = (void *)__dtb_start;
>>
>> That worked fine until this patch. Starting with this patch, __dtb_start
>> always points to a valid empty devicetree blob, which overrides the
>> devicetree blob passed to early_init_devtree(). This causes the problem.
> 
> With an external DTB, it doesn't boot with or without this patch. It
> just dies in different spots. Before it just skipped any memory

No, that is incorrect. Up to this patch it booted just fine with an
external dtb using the "-initrd" command line argument, and I explained
to you above why this is the case.

I guess you are saying I should stop boot testing nios2. I'll do just that.
Saves me resources and avoids discussions like this.

Guenter

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Rob Herring]

On Wed, Mar 20, 2024 at 3:06 PM Guenter Roeck <linux@roeck-us.net> wrote:
>
> On 3/20/24 12:14, Rob Herring wrote:
> > On Mon, Mar 18, 2024 at 4:31 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >>
> >> On 3/18/24 12:26, Rob Herring wrote:
> >>> +Stephen
> >>>
> >>> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> >>>>> When enabling CONFIG_OF on a platform where of_root is not populated by
> >>>>> firmware, we end up without a root node. In order to apply overlays and
> >>>>> create subnodes of the root node, we need one. Create this root node
> >>>>> by unflattening an empty builtin dtb.
> >>>>>
> >>>>> If firmware provides a flattened device tree (FDT) then the FDT is
> >>>>> unflattened via setup_arch().  Otherwise setup_of(), which is called
> >>>>> immediately after setup_arch(), will create the default root node
> >>>>> if it does not exist.
> >>>>>
> >>>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
> >>>>
> >>>> This patch results in a crash on nios2.
> >>>
> >>> This patch was never applied. I assume you meant a later version of it
> >>> that did get applied.
> >>>
> >>>>
> >>>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
> >>>
> >>> Booting with DT?
> >>>
> >>>> ------------
> >>>> qemu log:
> >>>> earlycon: uart8250 at MMIO32 0x18001600 (options '')
> >>>> printk: legacy bootconsole [uart8250] enabled
> >>>> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
> >>>> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
> >>>> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---
> >>>
> >>> nios2 looks utterly broken to me. This change should be a nop unless
> >>> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
> >>> address) to be 0 depending on kconfig options, but that would have
> >>> skipped copying and unflattening which would then panic in
> >>> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
> >>> early_init_dt_alloc_memory_arch() calls should fail when copying the
> >>> DT. So I don't see how nios2 booting with DT ever worked.
> >>>
> >>
> >> For nios2, in early_init_devtree():
> >>
> >> void __init early_init_devtree(void *params)
> >> {
> >>           __be32 *dtb = (u32 *)__dtb_start;
> >>          ...
> >>           if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
> >>                   params = (void *)__dtb_start;
> >>
> >> That worked fine until this patch. Starting with this patch, __dtb_start
> >> always points to a valid empty devicetree blob, which overrides the
> >> devicetree blob passed to early_init_devtree(). This causes the problem.
> >
> > With an external DTB, it doesn't boot with or without this patch. It
> > just dies in different spots. Before it just skipped any memory
>
> No, that is incorrect.

Well, I can tell you it doesn't boot for me. So I must be doing
something different from your setup.

> Up to this patch it booted just fine with an
> external dtb using the "-initrd" command line argument, and I explained
> to you above why this is the case.

What does -initrd have to do with anything? Does that shift where the
external dtb is placed or something?

I think I see the issue. __dtb_start points to the start of *all*
built-in DTBs, not a specific one. In this case, arc, csky, loongarch,
mips, openrisc, riscv, sh, and xtensa may all be broken too (if one
picks the magic combination of booting modes and kconfig options). I
would expect all these cases have been broken forever if the DT
unittest is enabled as it too adds a built-in dtb. But I would also
expect that arch code gets linked first and link order would save us
here.

Rob

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Guenter Roeck]

On 3/27/24 06:11, Rob Herring wrote:
> On Wed, Mar 20, 2024 at 3:06 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> On 3/20/24 12:14, Rob Herring wrote:
>>> On Mon, Mar 18, 2024 at 4:31 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>>>
>>>> On 3/18/24 12:26, Rob Herring wrote:
>>>>> +Stephen
>>>>>
>>>>> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
>>>>>>> When enabling CONFIG_OF on a platform where of_root is not populated by
>>>>>>> firmware, we end up without a root node. In order to apply overlays and
>>>>>>> create subnodes of the root node, we need one. Create this root node
>>>>>>> by unflattening an empty builtin dtb.
>>>>>>>
>>>>>>> If firmware provides a flattened device tree (FDT) then the FDT is
>>>>>>> unflattened via setup_arch().  Otherwise setup_of(), which is called
>>>>>>> immediately after setup_arch(), will create the default root node
>>>>>>> if it does not exist.
>>>>>>>
>>>>>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
>>>>>>
>>>>>> This patch results in a crash on nios2.
>>>>>
>>>>> This patch was never applied. I assume you meant a later version of it
>>>>> that did get applied.
>>>>>
>>>>>>
>>>>>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
>>>>>
>>>>> Booting with DT?
>>>>>
>>>>>> ------------
>>>>>> qemu log:
>>>>>> earlycon: uart8250 at MMIO32 0x18001600 (options '')
>>>>>> printk: legacy bootconsole [uart8250] enabled
>>>>>> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
>>>>>> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
>>>>>> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---
>>>>>
>>>>> nios2 looks utterly broken to me. This change should be a nop unless
>>>>> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
>>>>> address) to be 0 depending on kconfig options, but that would have
>>>>> skipped copying and unflattening which would then panic in
>>>>> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
>>>>> early_init_dt_alloc_memory_arch() calls should fail when copying the
>>>>> DT. So I don't see how nios2 booting with DT ever worked.
>>>>>
>>>>
>>>> For nios2, in early_init_devtree():
>>>>
>>>> void __init early_init_devtree(void *params)
>>>> {
>>>>            __be32 *dtb = (u32 *)__dtb_start;
>>>>           ...
>>>>            if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
>>>>                    params = (void *)__dtb_start;
>>>>
>>>> That worked fine until this patch. Starting with this patch, __dtb_start
>>>> always points to a valid empty devicetree blob, which overrides the
>>>> devicetree blob passed to early_init_devtree(). This causes the problem.
>>>
>>> With an external DTB, it doesn't boot with or without this patch. It
>>> just dies in different spots. Before it just skipped any memory
>>
>> No, that is incorrect.
> 
> Well, I can tell you it doesn't boot for me. So I must be doing
> something different from your setup.
> 

Maybe you have OF_UNITTEST enabled and it indeed results in the
problem you mention below. I don't have it enabled because it produces
various backtraces which would hide real problems.

>> Up to this patch it booted just fine with an
>> external dtb using the "-initrd" command line argument, and I explained
>> to you above why this is the case.
> 
> What does -initrd have to do with anything? Does that shift where the
> external dtb is placed or something?
> 

Nothing. I meant to say -dtb.

> I think I see the issue. __dtb_start points to the start of *all*
> built-in DTBs, not a specific one. In this case, arc, csky, loongarch,
> mips, openrisc, riscv, sh, and xtensa may all be broken too (if one
> picks the magic combination of booting modes and kconfig options). I

No.

- arc only picks the internal dtb if use_embedded_dtb is true. This flag
   is only set if there is no external dtb, or if the external dtb does
   not provide a valid machine description.
- openrisc only picks the internal dtb if no external dtb is provided.
- riscv only picks the internal dtb if CONFIG_BUILTIN_DTB is enabled.
- sh only used the internal dtb if CONFIG_USE_BUILTIN_DTB is enabled.
- xtensa only picks the internal dtb if there is no external dtb.

However, nios2 picks the internal dtb _even if_ an external dtb
is provided if there is an internal dtb. In other words, it prefers
the internal dtb over the external dtb. All other architectures
prefer the external dtb over the internal dtb.

> would expect all these cases have been broken forever if the DT
> unittest is enabled as it too adds a built-in dtb. But I would also

Even if that is correct for nios2, that hardly seems to be an argument
to break nios2 boot with external dtb unconditionally.

Thanks,
Guenter

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Rob Herring]

On Wed, Mar 27, 2024 at 9:40 AM Guenter Roeck <linux@roeck-us.net> wrote:
>
> On 3/27/24 06:11, Rob Herring wrote:
> > On Wed, Mar 20, 2024 at 3:06 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >>
> >> On 3/20/24 12:14, Rob Herring wrote:
> >>> On Mon, Mar 18, 2024 at 4:31 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >>>>
> >>>> On 3/18/24 12:26, Rob Herring wrote:
> >>>>> +Stephen
> >>>>>
> >>>>> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> >>>>>>> When enabling CONFIG_OF on a platform where of_root is not populated by
> >>>>>>> firmware, we end up without a root node. In order to apply overlays and
> >>>>>>> create subnodes of the root node, we need one. Create this root node
> >>>>>>> by unflattening an empty builtin dtb.
> >>>>>>>
> >>>>>>> If firmware provides a flattened device tree (FDT) then the FDT is
> >>>>>>> unflattened via setup_arch().  Otherwise setup_of(), which is called
> >>>>>>> immediately after setup_arch(), will create the default root node
> >>>>>>> if it does not exist.
> >>>>>>>
> >>>>>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
> >>>>>>
> >>>>>> This patch results in a crash on nios2.
> >>>>>
> >>>>> This patch was never applied. I assume you meant a later version of it
> >>>>> that did get applied.
> >>>>>
> >>>>>>
> >>>>>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ... running ...R failed (crashed)
> >>>>>
> >>>>> Booting with DT?
> >>>>>
> >>>>>> ------------
> >>>>>> qemu log:
> >>>>>> earlycon: uart8250 at MMIO32 0x18001600 (options '')
> >>>>>> printk: legacy bootconsole [uart8250] enabled
> >>>>>> Linux version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT 2024
> >>>>>> Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40
> >>>>>> ---[ end Kernel panic - not syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72 bytes align=0x40 ]---
> >>>>>
> >>>>> nios2 looks utterly broken to me. This change should be a nop unless
> >>>>> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
> >>>>> address) to be 0 depending on kconfig options, but that would have
> >>>>> skipped copying and unflattening which would then panic in
> >>>>> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
> >>>>> early_init_dt_alloc_memory_arch() calls should fail when copying the
> >>>>> DT. So I don't see how nios2 booting with DT ever worked.
> >>>>>
> >>>>
> >>>> For nios2, in early_init_devtree():
> >>>>
> >>>> void __init early_init_devtree(void *params)
> >>>> {
> >>>>            __be32 *dtb = (u32 *)__dtb_start;
> >>>>           ...
> >>>>            if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
> >>>>                    params = (void *)__dtb_start;
> >>>>
> >>>> That worked fine until this patch. Starting with this patch, __dtb_start
> >>>> always points to a valid empty devicetree blob, which overrides the
> >>>> devicetree blob passed to early_init_devtree(). This causes the problem.
> >>>
> >>> With an external DTB, it doesn't boot with or without this patch. It
> >>> just dies in different spots. Before it just skipped any memory
> >>
> >> No, that is incorrect.
> >
> > Well, I can tell you it doesn't boot for me. So I must be doing
> > something different from your setup.
> >
>
> Maybe you have OF_UNITTEST enabled and it indeed results in the
> problem you mention below. I don't have it enabled because it produces
> various backtraces which would hide real problems.

I thought of that, but I don't think I did. What I suspect is the
external dtb is at address 0.

> >> Up to this patch it booted just fine with an
> >> external dtb using the "-initrd" command line argument, and I explained
> >> to you above why this is the case.
> >
> > What does -initrd have to do with anything? Does that shift where the
> > external dtb is placed or something?
> >
>
> Nothing. I meant to say -dtb.
>
> > I think I see the issue. __dtb_start points to the start of *all*
> > built-in DTBs, not a specific one. In this case, arc, csky, loongarch,
> > mips, openrisc, riscv, sh, and xtensa may all be broken too (if one
> > picks the magic combination of booting modes and kconfig options). I
>
> No.
>
> - arc only picks the internal dtb if use_embedded_dtb is true. This flag
>    is only set if there is no external dtb, or if the external dtb does
>    not provide a valid machine description.

Right, but when it does pick the internal dtb, it is expecting its dtb
at __dtb_start. What I'm saying is that's never been a good or safe
assumption. We just happened to add another case to trigger it. The
only reliable way to get a built-in DTB is if foo.dtb is built-in,
then use __dtb_foo_begin to get its address. That's what some MIPS
platforms with multiple DTBs do.

> - openrisc only picks the internal dtb if no external dtb is provided.
> - riscv only picks the internal dtb if CONFIG_BUILTIN_DTB is enabled.
> - sh only used the internal dtb if CONFIG_USE_BUILTIN_DTB is enabled.
> - xtensa only picks the internal dtb if there is no external dtb.
>
> However, nios2 picks the internal dtb _even if_ an external dtb
> is provided if there is an internal dtb. In other words, it prefers
> the internal dtb over the external dtb. All other architectures
> prefer the external dtb over the internal dtb.

Thanks for the analysis. I had started and abandoned common support
(mostly Kconfig options) for built-in dtbs years ago. I decided
against it because it is not something we want to encourage (as the
boot dtb). In the meantime, we've gained new architectures that have
added it. Sigh. So now I'm reconsidering something common (though not
for v6.9).

>
> > would expect all these cases have been broken forever if the DT
> > unittest is enabled as it too adds a built-in dtb. But I would also
>
> Even if that is correct for nios2, that hardly seems to be an argument
> to break nios2 boot with external dtb unconditionally.

That wasn't an argument for breaking it. Using an external dtb should
really be the default and strongly preferred though.

I'm still not sure how to fix this easily for 6.9. Something like what
microblaze does which puts the boot dtb under a consistent symbol
name. Or perhaps we could iterate thru the built-in dtbs and skip ones
without top-level compatible.

Rob

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Guenter Roeck]

On Wed, Mar 27, 2024 at 01:38:13PM -0500, Rob Herring wrote:
> On Wed, Mar 27, 2024 at 9:40 AM Guenter Roeck <linux@roeck-us.net> wrote:
> >
> > On 3/27/24 06:11, Rob Herring wrote:
> > > On Wed, Mar 20, 2024 at 3:06 PM Guenter Roeck <linux@roeck-us.net> wrote:
> > >>
> > >> On 3/20/24 12:14, Rob Herring wrote:
> > >>> On Mon, Mar 18, 2024 at 4:31 PM Guenter Roeck <linux@roeck-us.net>
> > >>> wrote:
> > >>>>
> > >>>> On 3/18/24 12:26, Rob Herring wrote:
> > >>>>> +Stephen
> > >>>>>
> > >>>>> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net>
> > >>>>> wrote:
> > >>>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> > >>>>>>> When enabling CONFIG_OF on a platform where of_root is not
> > >>>>>>> populated by firmware, we end up without a root node. In order to
> > >>>>>>> apply overlays and create subnodes of the root node, we need one.
> > >>>>>>> Create this root node by unflattening an empty builtin dtb.
> > >>>>>>>
> > >>>>>>> If firmware provides a flattened device tree (FDT) then the FDT is
> > >>>>>>> unflattened via setup_arch().  Otherwise setup_of(), which is
> > >>>>>>> called immediately after setup_arch(), will create the default root
> > >>>>>>> node if it does not exist.
> > >>>>>>>
> > >>>>>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
> > >>>>>>
> > >>>>>> This patch results in a crash on nios2.
> > >>>>>
> > >>>>> This patch was never applied. I assume you meant a later version of
> > >>>>> it that did get applied.
> > >>>>>
> > >>>>>>
> > >>>>>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ...
> > >>>>>> running ...R failed (crashed)
> > >>>>>
> > >>>>> Booting with DT?
> > >>>>>
> > >>>>>> ------------ qemu log: earlycon: uart8250 at MMIO32 0x18001600
> > >>>>>> (options '') printk: legacy bootconsole [uart8250] enabled Linux
> > >>>>>> version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc
> > >>>>>> (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT
> > >>>>>> 2024 Kernel panic - not syncing: early_init_dt_alloc_memory_arch:
> > >>>>>> Failed to allocate 72 bytes align=0x40 ---[ end Kernel panic - not
> > >>>>>> syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72
> > >>>>>> bytes align=0x40 ]---
> > >>>>>
> > >>>>> nios2 looks utterly broken to me. This change should be a nop unless
> > >>>>> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
> > >>>>> address) to be 0 depending on kconfig options, but that would have
> > >>>>> skipped copying and unflattening which would then panic in
> > >>>>> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
> > >>>>> early_init_dt_alloc_memory_arch() calls should fail when copying the
> > >>>>> DT. So I don't see how nios2 booting with DT ever worked.
> > >>>>>
> > >>>>
> > >>>> For nios2, in early_init_devtree():
> > >>>>
> > >>>> void __init early_init_devtree(void *params) { __be32 *dtb = (u32
> > >>>> *)__dtb_start; ...  if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
> > >>>> params = (void *)__dtb_start;
> > >>>>
> > >>>> That worked fine until this patch. Starting with this patch,
> > >>>> __dtb_start always points to a valid empty devicetree blob, which
> > >>>> overrides the devicetree blob passed to early_init_devtree(). This
> > >>>> causes the problem.
> > >>>
> > >>> With an external DTB, it doesn't boot with or without this patch. It
> > >>> just dies in different spots. Before it just skipped any memory
> > >>
> > >> No, that is incorrect.
> > >
> > > Well, I can tell you it doesn't boot for me. So I must be doing something
> > > different from your setup.
> > >
> >
> > Maybe you have OF_UNITTEST enabled and it indeed results in the problem you
> > mention below. I don't have it enabled because it produces various
> > backtraces which would hide real problems.
> 
> I thought of that, but I don't think I did. What I suspect is the external
> dtb is at address 0.
> 
> > >> Up to this patch it booted just fine with an external dtb using the
> > >> "-initrd" command line argument, and I explained to you above why this
> > >> is the case.
> > >
> > > What does -initrd have to do with anything? Does that shift where the
> > > external dtb is placed or something?
> > >
> >
> > Nothing. I meant to say -dtb.
> >
> > > I think I see the issue. __dtb_start points to the start of *all*
> > > built-in DTBs, not a specific one. In this case, arc, csky, loongarch,
> > > mips, openrisc, riscv, sh, and xtensa may all be broken too (if one picks
> > > the magic combination of booting modes and kconfig options). I
> >
> > No.
> >
> > - arc only picks the internal dtb if use_embedded_dtb is true. This flag is
> > only set if there is no external dtb, or if the external dtb does not
> > provide a valid machine description.
> 
> Right, but when it does pick the internal dtb, it is expecting its dtb at
> __dtb_start. What I'm saying is that's never been a good or safe assumption.
> We just happened to add another case to trigger it. The only reliable way to
> get a built-in DTB is if foo.dtb is built-in, then use __dtb_foo_begin to get
> its address. That's what some MIPS platforms with multiple DTBs do.
> 

I may be missing something, but it seems to me that most if not all
platforms with support for configurable built-in dtbs use __dtb_start
to get its address.

> > - openrisc only picks the internal dtb if no external dtb is provided.  -
> > riscv only picks the internal dtb if CONFIG_BUILTIN_DTB is enabled.  - sh
> > only used the internal dtb if CONFIG_USE_BUILTIN_DTB is enabled.  - xtensa
> > only picks the internal dtb if there is no external dtb.
> >
> > However, nios2 picks the internal dtb _even if_ an external dtb is provided
> > if there is an internal dtb. In other words, it prefers the internal dtb
> > over the external dtb. All other architectures prefer the external dtb over
> > the internal dtb.
> 
> Thanks for the analysis. I had started and abandoned common support (mostly
> Kconfig options) for built-in dtbs years ago. I decided against it because it
> is not something we want to encourage (as the boot dtb). In the meantime,
> we've gained new architectures that have added it. Sigh. So now I'm
> reconsidering something common (though not for v6.9).
> 
> >
> > > would expect all these cases have been broken forever if the DT unittest
> > > is enabled as it too adds a built-in dtb. But I would also
> >
> > Even if that is correct for nios2, that hardly seems to be an argument to
> > break nios2 boot with external dtb unconditionally.
> 
> That wasn't an argument for breaking it. Using an external dtb should really
> be the default and strongly preferred though.
> 
> I'm still not sure how to fix this easily for 6.9. Something like what
> microblaze does which puts the boot dtb under a consistent symbol name. Or
> perhaps we could iterate thru the built-in dtbs and skip ones without
> top-level compatible.
> 

I did submit a patch to only override the external dtb if support for the
internal dtb is enabled. I copied you on it, so you should have seen it.

https://lore.kernel.org/linux-kernel/20240322065419.162416-1-linux@roeck-us.net/

Maybe that is less than perfect, but it is minimalistic, and I didn't want
to change code behavior more than absolutely necessary without guidance
from the nios2 maintainer.

Guenter

Re: [PATCH v4 1/2] of: create of_root if no dtb provided

[Rob Herring]

On Wed, Mar 27, 2024 at 2:47 PM Guenter Roeck <linux@roeck-us.net> wrote:
>
> On Wed, Mar 27, 2024 at 01:38:13PM -0500, Rob Herring wrote:
> > On Wed, Mar 27, 2024 at 9:40 AM Guenter Roeck <linux@roeck-us.net> wrote:
> > >
> > > On 3/27/24 06:11, Rob Herring wrote:
> > > > On Wed, Mar 20, 2024 at 3:06 PM Guenter Roeck <linux@roeck-us.net> wrote:
> > > >>
> > > >> On 3/20/24 12:14, Rob Herring wrote:
> > > >>> On Mon, Mar 18, 2024 at 4:31 PM Guenter Roeck <linux@roeck-us.net>
> > > >>> wrote:
> > > >>>>
> > > >>>> On 3/18/24 12:26, Rob Herring wrote:
> > > >>>>> +Stephen
> > > >>>>>
> > > >>>>> On Mon, Mar 18, 2024 at 12:09 PM Guenter Roeck <linux@roeck-us.net>
> > > >>>>> wrote:
> > > >>>>>>
> > > >>>>>> Hi,
> > > >>>>>>
> > > >>>>>> On Fri, Mar 17, 2023 at 12:34:14AM -0500, Frank Rowand wrote:
> > > >>>>>>> When enabling CONFIG_OF on a platform where of_root is not
> > > >>>>>>> populated by firmware, we end up without a root node. In order to
> > > >>>>>>> apply overlays and create subnodes of the root node, we need one.
> > > >>>>>>> Create this root node by unflattening an empty builtin dtb.
> > > >>>>>>>
> > > >>>>>>> If firmware provides a flattened device tree (FDT) then the FDT is
> > > >>>>>>> unflattened via setup_arch().  Otherwise setup_of(), which is
> > > >>>>>>> called immediately after setup_arch(), will create the default root
> > > >>>>>>> node if it does not exist.
> > > >>>>>>>
> > > >>>>>>> Signed-off-by: Frank Rowand <frowand.list@gmail.com>
> > > >>>>>>
> > > >>>>>> This patch results in a crash on nios2.
> > > >>>>>
> > > >>>>> This patch was never applied. I assume you meant a later version of
> > > >>>>> it that did get applied.
> > > >>>>>
> > > >>>>>>
> > > >>>>>> Building nios2:10m50-ghrd:10m50_defconfig:10m50_devboard.dts ...
> > > >>>>>> running ...R failed (crashed)
> > > >>>>>
> > > >>>>> Booting with DT?
> > > >>>>>
> > > >>>>>> ------------ qemu log: earlycon: uart8250 at MMIO32 0x18001600
> > > >>>>>> (options '') printk: legacy bootconsole [uart8250] enabled Linux
> > > >>>>>> version 6.8.0-11409-gf6cef5f8c37f (groeck@desktop) (nios2-linux-gcc
> > > >>>>>> (GCC) 11.4.0, GNU ld (GNU Binutils) 2.40) #1 Sun Mar 17 23:38:59 PDT
> > > >>>>>> 2024 Kernel panic - not syncing: early_init_dt_alloc_memory_arch:
> > > >>>>>> Failed to allocate 72 bytes align=0x40 ---[ end Kernel panic - not
> > > >>>>>> syncing: early_init_dt_alloc_memory_arch: Failed to allocate 72
> > > >>>>>> bytes align=0x40 ]---
> > > >>>>>
> > > >>>>> nios2 looks utterly broken to me. This change should be a nop unless
> > > >>>>> initial_boot_params is NULL. It looks like it is possible for r6 (dtb
> > > >>>>> address) to be 0 depending on kconfig options, but that would have
> > > >>>>> skipped copying and unflattening which would then panic in
> > > >>>>> setup_cpuinfo(). If initial_boot_params is not NULL, then the same
> > > >>>>> early_init_dt_alloc_memory_arch() calls should fail when copying the
> > > >>>>> DT. So I don't see how nios2 booting with DT ever worked.
> > > >>>>>
> > > >>>>
> > > >>>> For nios2, in early_init_devtree():
> > > >>>>
> > > >>>> void __init early_init_devtree(void *params) { __be32 *dtb = (u32
> > > >>>> *)__dtb_start; ...  if (be32_to_cpu((__be32) *dtb) == OF_DT_HEADER)
> > > >>>> params = (void *)__dtb_start;
> > > >>>>
> > > >>>> That worked fine until this patch. Starting with this patch,
> > > >>>> __dtb_start always points to a valid empty devicetree blob, which
> > > >>>> overrides the devicetree blob passed to early_init_devtree(). This
> > > >>>> causes the problem.
> > > >>>
> > > >>> With an external DTB, it doesn't boot with or without this patch. It
> > > >>> just dies in different spots. Before it just skipped any memory
> > > >>
> > > >> No, that is incorrect.
> > > >
> > > > Well, I can tell you it doesn't boot for me. So I must be doing something
> > > > different from your setup.
> > > >
> > >
> > > Maybe you have OF_UNITTEST enabled and it indeed results in the problem you
> > > mention below. I don't have it enabled because it produces various
> > > backtraces which would hide real problems.
> >
> > I thought of that, but I don't think I did. What I suspect is the external
> > dtb is at address 0.

Testing again, I built 10m50_defconfig without modification and ran
qemu (from debian testing):

qemu-system-nios2 -dtb
.build-nios2/arch/nios2/boot/dts/10m50_devboard.dtb -kernel
.build-nios2/vmlinux --nographic

I had to enable CONFIG_NIOS2_PASS_CMDLINE (which really means pass
cmdline, dtb, and initrd from bootloader) and it works and regresses
as you report.

> > > >> Up to this patch it booted just fine with an external dtb using the
> > > >> "-initrd" command line argument, and I explained to you above why this
> > > >> is the case.
> > > >
> > > > What does -initrd have to do with anything? Does that shift where the
> > > > external dtb is placed or something?
> > > >
> > >
> > > Nothing. I meant to say -dtb.
> > >
> > > > I think I see the issue. __dtb_start points to the start of *all*
> > > > built-in DTBs, not a specific one. In this case, arc, csky, loongarch,
> > > > mips, openrisc, riscv, sh, and xtensa may all be broken too (if one picks
> > > > the magic combination of booting modes and kconfig options). I
> > >
> > > No.
> > >
> > > - arc only picks the internal dtb if use_embedded_dtb is true. This flag is
> > > only set if there is no external dtb, or if the external dtb does not
> > > provide a valid machine description.
> >
> > Right, but when it does pick the internal dtb, it is expecting its dtb at
> > __dtb_start. What I'm saying is that's never been a good or safe assumption.
> > We just happened to add another case to trigger it. The only reliable way to
> > get a built-in DTB is if foo.dtb is built-in, then use __dtb_foo_begin to get
> > its address. That's what some MIPS platforms with multiple DTBs do.
> >
>
> I may be missing something, but it seems to me that most if not all
> platforms with support for configurable built-in dtbs use __dtb_start
> to get its address.

That was my concern as that only points to the 1st DTB and nothing
prevents there being more than one. But I think link order saves all
of them after all.

> > > - openrisc only picks the internal dtb if no external dtb is provided.  -
> > > riscv only picks the internal dtb if CONFIG_BUILTIN_DTB is enabled.  - sh
> > > only used the internal dtb if CONFIG_USE_BUILTIN_DTB is enabled.  - xtensa
> > > only picks the internal dtb if there is no external dtb.
> > >
> > > However, nios2 picks the internal dtb _even if_ an external dtb is provided
> > > if there is an internal dtb. In other words, it prefers the internal dtb
> > > over the external dtb. All other architectures prefer the external dtb over
> > > the internal dtb.
> >
> > Thanks for the analysis. I had started and abandoned common support (mostly
> > Kconfig options) for built-in dtbs years ago. I decided against it because it
> > is not something we want to encourage (as the boot dtb). In the meantime,
> > we've gained new architectures that have added it. Sigh. So now I'm
> > reconsidering something common (though not for v6.9).
> >
> > >
> > > > would expect all these cases have been broken forever if the DT unittest
> > > > is enabled as it too adds a built-in dtb. But I would also
> > >
> > > Even if that is correct for nios2, that hardly seems to be an argument to
> > > break nios2 boot with external dtb unconditionally.
> >
> > That wasn't an argument for breaking it. Using an external dtb should really
> > be the default and strongly preferred though.
> >
> > I'm still not sure how to fix this easily for 6.9. Something like what
> > microblaze does which puts the boot dtb under a consistent symbol name. Or
> > perhaps we could iterate thru the built-in dtbs and skip ones without
> > top-level compatible.
> >
>
> I did submit a patch to only override the external dtb if support for the
> internal dtb is enabled. I copied you on it, so you should have seen it.
>
> https://lore.kernel.org/linux-kernel/20240322065419.162416-1-linux@roeck-us.net/

Now reviewed, thanks. Sadly, I rarely see anything outside the normal
workflow of what I can filter out. I'm copied pretty much 1:1 with
what is sent to the DT list which is a fire hose.

Rob

[PATCH v4 2/2] of: unittest: treat missing of_root as error instead of fixing up

[Frank Rowand]

setup_of() now ensures that of_root node is populated with the
root of a default devicetree. Remove the unittest code that
created of_root if it was missing.  Verify that of_root is
valid before attempting to attach the testcase-data subtree.
Remove the unittest code that unflattens the unittest overlay
base if architecture is UML.

Signed-off-by: Frank Rowand <frowand.list@gmail.com>
---

Changes since version 3:
  - refresh for 6.3-rc1
  - remove the CONFIG_UML case of populating the devicetree
  - unittest_data_add() - move an EXPECT_BEGIN() to after an error
    check that can result in an early return

Changes since version 2:
  - none

Changes since version 1:
  - refresh for 6.2-rc1
  - update Signed-off-by
  - fix formatting error (leading space) in patch comment

 drivers/of/unittest.c | 16 ++++------------
 1 file changed, 4 insertions(+), 12 deletions(-)

diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c
index b5a7a31d8bd2..8dc293ac08b7 100644
--- a/drivers/of/unittest.c
+++ b/drivers/of/unittest.c
@@ -1476,20 +1476,16 @@ static int __init unittest_data_add(void)
 		return -EINVAL;
 	}
 
+	/* attach the sub-tree to live tree */
 	if (!of_root) {
-		of_root = unittest_data_node;
-		for_each_of_allnodes(np)
-			__of_attach_node_sysfs(np);
-		of_aliases = of_find_node_by_path("/aliases");
-		of_chosen = of_find_node_by_path("/chosen");
-		of_overlay_mutex_unlock();
-		return 0;
+		pr_warn("%s: no live tree to attach sub-tree\n", __func__);
+		kfree(unittest_data);
+		return -ENODEV;
 	}
 
 	EXPECT_BEGIN(KERN_INFO,
 		     "Duplicate name in testcase-data, renamed to \"duplicate-name#1\"");
 
-	/* attach the sub-tree to live tree */
 	np = unittest_data_node->child;
 	while (np) {
 		struct device_node *next = np->sibling;
@@ -3612,10 +3608,6 @@ static int __init of_unittest(void)
 	add_taint(TAINT_TEST, LOCKDEP_STILL_OK);
 
 	/* adding data for unittest */
-
-	if (IS_ENABLED(CONFIG_UML))
-		unittest_unflatten_overlay_base();
-
 	res = unittest_data_add();
 	if (res)
 		return res;
-- 
Frank Rowand <frowand.list@gmail.com>