From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C2AD4C46467 for ; Wed, 11 Jan 2023 21:10:28 +0000 (UTC) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 6E90585560; Wed, 11 Jan 2023 22:10:26 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=reject dis=none) header.from=diskos.nl Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (2048-bit key; secure) header.d=diskos.nl header.i=@diskos.nl header.b="ZKJN2kus"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 9E7BC855A6; Wed, 11 Jan 2023 22:09:47 +0100 (CET) Received: from smtp.delft.diskos.nl (smtp.delft.diskos.nl [45.80.169.141]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 180BE853B1 for ; Wed, 11 Jan 2023 22:09:22 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=reject dis=none) header.from=diskos.nl Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=joost@diskos.nl Received: from [10.135.128.1] (smtp.delft.diskos.nl [45.80.169.141]) by smtp.delft.diskos.nl (Postfix) with ESMTPSA id AF4848BDDE; Wed, 11 Jan 2023 22:09:04 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=diskos.nl; s=mail; t=1673471344; bh=QVZJSV3a/sx1IYTlHbtV2wJ/sPwZOgUaRVhe2UdZBEg=; h=Date:From:Subject:To:Cc:In-Reply-To:References:From; b=ZKJN2kusXB3ia4U00fT9CN40YO6FecR60GfLDnqgHTTKu2Z+L9e8+nq91WzTeMgMA yh1ZujweoNZ8vDAC4HPyEMgcsRWmmXXHciEdLGi1V8MiElxO2EgIBkwAvADcsU/hiZ ZELlly+EHJ+PdqV02sCKo0VURHS8zGjTnsKyVzfEG0dj0ZSVHMFG8sc3twzTjlCzir WFPt/yH8GlD+nSN0nbfGcVuwG5Z5LN2pqeqyWrnRVabeb9n5OYAP0idEV8VUXtjeN5 wCXBumAHhzGGsr3VYhQCcQJESyttgRoWaUxqbZmNK9cyOQSgpFEnImdg83p6tr/SZC 0EsWH5jCexW7g== Date: Wed, 11 Jan 2023 22:08:59 +0100 From: Joost van Zwieten Subject: Re: kernel doesn't start on Odroid U2 unless setting initrd_high To: Tom Rini Cc: Jaehoon Chung , u-boot@lists.denx.de Message-Id: In-Reply-To: <20230111203844.GO3787616@bill-the-cat> References: <20230110184101.GO3787616@bill-the-cat> <20230110230824.GR3787616@bill-the-cat> <20230110232915.GS3787616@bill-the-cat> <20230111203844.GO3787616@bill-the-cat> X-Mailer: geary/3.38.1 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean On Wed, Jan 11, 2023 at 15:38, Tom Rini wrote: > On Wed, Jan 11, 2023 at 09:28:42PM +0100, Joost van Zwieten wrote: >> >> >> On Tue, Jan 10, 2023 at 18:29, Tom Rini wrote: >> > On Wed, Jan 11, 2023 at 12:10:42AM +0100, Joost van Zwieten wrote: >> > > >> > > >> > > On Tue, Jan 10, 2023 at 18:08, Tom Rini >> wrote: >> > > > On Wed, Jan 11, 2023 at 12:01:46AM +0100, Joost van Zwieten >> wrote: >> > > > > >> > > > > >> > > > > On Tue, Jan 10, 2023 at 13:41, Tom Rini >> >> > > wrote: >> > > > > > On Tue, Jan 10, 2023 at 09:13:32AM +0100, Joost van >> Zwieten >> > > wrote: >> > > > > > >> > > > > > > Dear maintainers, >> > > > > > > >> > > > > > > As of commit 4963f63fe61f15329d77472a762b1d8bf754d24b >> > > U-Boot >> > > > > fails >> > > > > > > to start >> > > > > > > a kernel (with `bootz`) on my Odroid U2 unless I >> force >> > > > > > > `initrd_high`, e.g. >> > > > > > > to `0x50000000`. With commit 4963f63f and >> `initrd_high` >> > > > > *unset* I >> > > > > > > get the >> > > > > > > following output: >> > > > > > > >> > > > > > > U-Boot 2020.10-rc2-00314-g4963f63fe6 (Jan 09 2023 >> - >> > > 23:59:31 >> > > > > > > +0100) >> > > > > > > >> > > > > > > CPU: Exynos4412 @ 1 GHz >> > > > > > > Model: Odroid based on Exynos4412 >> > > > > > > Type: u3 >> > > > > > > DRAM: 2 GiB >> > > > > > > LDO20@VDDQ_EMMC_1.8V: set 1800000 uV; enabling >> > > > > > > LDO22@VDDQ_EMMC_2.8V: set 2800000 uV; enabling >> > > > > > > LDO21@TFLASH_2.8V: set 2800000 uV; enabling >> > > > > > > MMC: SAMSUNG SDHCI: 2, EXYNOS DWMMC: 0 >> > > > > > > Loading Environment from MMC... *** Warning - bad >> CRC, >> > > using >> > > > > > > default >> > > > > > > environment >> > > > > > > >> > > > > > > In: serial >> > > > > > > Out: serial >> > > > > > > Err: serial >> > > > > > > Boot device: MMC(2) >> > > > > > > Net: No ethernet found. >> > > > > > > Hit any key to stop autoboot: 0 >> > > > > > > Odroid # env set fk_kvers 5.10.0-20-armmp >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > ${fdt_addr_r} >> > > > > > > /boot/dtbs/${fk_kvers}/exynos4412-odroidu3.dtb >> > > > > > > 53440 bytes read in 50 ms (1 MiB/s) >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > > > ${kernel_addr_r} >> > > > > > > /boot/vmlinuz-${fk_kvers} >> > > > > > > 4973056 bytes read in 182 ms (26.1 MiB/s) >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > > > ${ramdisk_addr_r} >> > > > > > > /boot/initrd.img-${fk_kvers} >> > > > > > > 22231585 bytes read in 777 ms (27.3 MiB/s) >> > > > > > > Odroid # env set bootargs console=ttySAC1,115200n8 >> > > > > > > Odroid # bootz ${kernel_addr_r} >> > > > > ${ramdisk_addr_r}:${filesize} >> > > > > > > ${fdt_addr_r} >> > > > > > > Kernel image @ 0x41000000 [ 0x000000 - 0x4be200 ] >> > > > > > > ## Flattened Device Tree blob at 40800000 >> > > > > > > Booting using the fdt blob at 0x40800000 >> > > > > > > Loading Ramdisk to b9947000, end bae7aa21 ... >> OK >> > > > > > > Loading Device Tree to b9936000, end b99460bf >> ... OK >> > > > > > > >> > > > > > > Starting kernel ... >> > > > > > > >> > > > > > > And that's all I ever see. Normally the initrd loads >> a >> > > module >> > > > > that >> > > > > > > causes an >> > > > > > > LED on the Odroid to blink, and this is not happening >> > > either, >> > > > > so >> > > > > > > I'm pretty >> > > > > > > confident the kernel doesn't start or at least >> crashes >> > > before >> > > > > > > producing >> > > > > > > output. If I set `initrd_high` to `0x50000000` (or >> > > something >> > > > > in the >> > > > > > > neighborhood) the kernel starts just fine: >> > > > > > > >> > > > > > > U-Boot 2020.10-rc2-00314-g4963f63fe6 (Jan 09 2023 >> - >> > > 23:59:31 >> > > > > > > +0100) >> > > > > > > >> > > > > > > CPU: Exynos4412 @ 1 GHz >> > > > > > > Model: Odroid based on Exynos4412 >> > > > > > > Type: u3 >> > > > > > > DRAM: 2 GiB >> > > > > > > LDO20@VDDQ_EMMC_1.8V: set 1800000 uV; enabling >> > > > > > > LDO22@VDDQ_EMMC_2.8V: set 2800000 uV; enabling >> > > > > > > LDO21@TFLASH_2.8V: set 2800000 uV; enabling >> > > > > > > MMC: SAMSUNG SDHCI: 2, EXYNOS DWMMC: 0 >> > > > > > > Loading Environment from MMC... *** Warning - bad >> CRC, >> > > using >> > > > > > > default >> > > > > > > environment >> > > > > > > >> > > > > > > In: serial >> > > > > > > Out: serial >> > > > > > > Err: serial >> > > > > > > Boot device: MMC(2) >> > > > > > > Net: No ethernet found. >> > > > > > > Hit any key to stop autoboot: 0 >> > > > > > > Odroid # env set fk_kvers 5.10.0-20-armmp >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > ${fdt_addr_r} >> > > > > > > /boot/dtbs/${fk_kvers}/exynos4412-odroidu3.dtb >> > > > > > > 53440 bytes read in 49 ms (1 MiB/s) >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > > > ${kernel_addr_r} >> > > > > > > /boot/vmlinuz-${fk_kvers} >> > > > > > > 4973056 bytes read in 181 ms (26.2 MiB/s) >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > > > ${ramdisk_addr_r} >> > > > > > > /boot/initrd.img-${fk_kvers} >> > > > > > > 22231585 bytes read in 777 ms (27.3 MiB/s) >> > > > > > > Odroid # env set bootargs console=ttySAC1,115200n8 >> > > > > > > Odroid # env set initrd_high 0x50000000 >> > > > > > > Odroid # bootz ${kernel_addr_r} >> > > > > ${ramdisk_addr_r}:${filesize} >> > > > > > > ${fdt_addr_r} >> > > > > > > Kernel image @ 0x41000000 [ 0x000000 - 0x4be200 ] >> > > > > > > ## Flattened Device Tree blob at 40800000 >> > > > > > > Booting using the fdt blob at 0x40800000 >> > > > > > > Loading Ramdisk to 4eacc000, end 4ffffa21 ... >> OK >> > > > > > > Loading Device Tree to bae6a000, end bae7a0bf >> ... OK >> > > > > > > >> > > > > > > Starting kernel ... >> > > > > > > >> > > > > > > [ 0.000000] Booting Linux on physical CPU 0xa00 >> > > > > > > >> > > > > > > >> > > > > > > The difference between those two runs is the location >> > > where >> > > > > U-Boot >> > > > > > > loads the >> > > > > > > initrd. The parent commit of 4963f63f boots fine >> without >> > > > > setting >> > > > > > > `initrd_high`: >> > > > > > > >> > > > > > > U-Boot 2020.10-rc2-00313-gdfaf6a5797 (Jan 10 2023 >> - >> > > 00:13:19 >> > > > > > > +0100) >> > > > > > > >> > > > > > > CPU: Exynos4412 @ 1 GHz >> > > > > > > Model: Odroid based on Exynos4412 >> > > > > > > Type: u3 >> > > > > > > DRAM: 2 GiB >> > > > > > > LDO20@VDDQ_EMMC_1.8V: set 1800000 uV; enabling >> > > > > > > LDO22@VDDQ_EMMC_2.8V: set 2800000 uV; enabling >> > > > > > > LDO21@TFLASH_2.8V: set 2800000 uV; enabling >> > > > > > > MMC: SAMSUNG SDHCI: 2, EXYNOS DWMMC: 0 >> > > > > > > Loading Environment from MMC... *** Warning - bad >> CRC, >> > > using >> > > > > > > default >> > > > > > > environment >> > > > > > > >> > > > > > > In: serial >> > > > > > > Out: serial >> > > > > > > Err: serial >> > > > > > > Boot device: MMC(2) >> > > > > > > Net: No ethernet found. >> > > > > > > Hit any key to stop autoboot: 0 >> > > > > > > Odroid # env set fk_kvers 5.10.0-20-armmp >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > ${fdt_addr_r} >> > > > > > > /boot/dtbs/${fk_kvers}/exynos4412-odroidu3.dtb >> > > > > > > 53440 bytes read in 49 ms (1 MiB/s) >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > > > ${kernel_addr_r} >> > > > > > > /boot/vmlinuz-${fk_kvers} >> > > > > > > 4973056 bytes read in 181 ms (26.2 MiB/s) >> > > > > > > Odroid # load mmc ${mmcbootdev}:${mmcbootpart} >> > > > > ${ramdisk_addr_r} >> > > > > > > /boot/initrd.img-${fk_kvers} >> > > > > > > 22231585 bytes read in 776 ms (27.3 MiB/s) >> > > > > > > Odroid # env set bootargs console=ttySAC1,115200n8 >> > > > > > > Odroid # bootz ${kernel_addr_r} >> > > > > ${ramdisk_addr_r}:${filesize} >> > > > > > > ${fdt_addr_r} >> > > > > > > Kernel image @ 0x41000000 [ 0x000000 - 0x4be200 ] >> > > > > > > ## Flattened Device Tree blob at 40800000 >> > > > > > > Booting using the fdt blob at 0x40800000 >> > > > > > > Loading Ramdisk to 4eacc000, end 4ffffa21 ... >> OK >> > > > > > > Loading Device Tree to 4eabb000, end 4eacb0bf >> ... OK >> > > > > > > >> > > > > > > Starting kernel ... >> > > > > > > >> > > > > > > [ 0.000000] Booting Linux on physical CPU 0xa00 >> > > > > > > >> > > > > > > >> > > > > > > I've tested two different kernels from Debian >> (buster and >> > > > > bullseye) >> > > > > > > and both >> > > > > > > have the same problem. I'm building U-Boot using the >> > > default >> > > > > config >> > > > > > > for >> > > > > > > board `odroid` and boot the Odroid from an SD card >> with >> > > > > firmware >> > > > > > > (`bl1`, >> > > > > > > `bl2` and `tzsw`) from Hardkernel's clone of the >> u-boot >> > > > > repository >> > > > > > > [1]. >> > > > > > > There are no peripherals connected apart from the SD >> card >> > > and >> > > > > the >> > > > > > > serial >> > > > > > > console. Please let me know if you need more >> information. >> > > > > > >> > > > > > Interesting. So what does "bdi" show, both in the older >> good >> > > and >> > > > > > then current tree? Specifically the information about >> where >> > > DRAM >> > > > > is, and >> > > > > > how large it is? Thanks. >> > > > > > >> > > > > > -- >> > > > > > Tom >> > > > > >> > > > > Both 4963f63f (not working) and its parent (working) >> produce >> > > the >> > > > > same >> > > > > output: >> > > > > >> > > > > boot_params = 0x40000100 >> > > > > DRAM bank = 0x00000000 >> > > > > -> start = 0x40000000 >> > > > > -> size = 0x10000000 >> > > > > DRAM bank = 0x00000001 >> > > > > -> start = 0x50000000 >> > > > > -> size = 0x10000000 >> > > > > DRAM bank = 0x00000002 >> > > > > -> start = 0x60000000 >> > > > > -> size = 0x10000000 >> > > > > DRAM bank = 0x00000003 >> > > > > -> start = 0x70000000 >> > > > > -> size = 0x10000000 >> > > > > DRAM bank = 0x00000004 >> > > > > -> start = 0x80000000 >> > > > > -> size = 0x10000000 >> > > > > DRAM bank = 0x00000005 >> > > > > -> start = 0x90000000 >> > > > > -> size = 0x10000000 >> > > > > DRAM bank = 0x00000006 >> > > > > -> start = 0xa0000000 >> > > > > -> size = 0x10000000 >> > > > > DRAM bank = 0x00000007 >> > > > > -> start = 0xb0000000 >> > > > > -> size = 0x0ff00000 >> > > > > memstart = 0x40000000 >> > > > > memsize = 0x7ff00000 >> > > > > flashstart = 0x00000000 >> > > > > flashsize = 0x00000000 >> > > > > flashoffset = 0x00000000 >> > > > > baudrate = 115200 bps >> > > > > relocaddr = 0xbfe83000 >> > > > > reloc off = 0x7c083000 >> > > > > Build = 32-bit >> > > > > current eth = unknown >> > > > > ethaddr = (not set) >> > > > > IP addr = >> > > > > fdt_blob = 0xbae7bf00 >> > > > > new_fdt = 0xbae7bf00 >> > > > > fdt_size = 0x00002fa0 >> > > > > lmb_dump_all: >> > > > > memory.cnt = 0x1 >> > > > > memory.size = 0x0 >> > > > > memory.reg[0x0].base = 0x40000000 >> > > > > .size = 0x7ff00000 >> > > > > >> > > > > reserved.cnt = 0x1 >> > > > > reserved.size = 0x0 >> > > > > reserved.reg[0x0].base = 0xbae7acd0 >> > > > > .size = 0x5085330 >> > > > > arch_number = 0x000010c1 >> > > > > TLB addr = 0xbfef0000 >> > > > > irq_sp = 0xbae7bef0 >> > > > > sp start = 0xbae7bee0 >> > > > > Board Type = 0 >> > > > > Early malloc usage: f0 / 400 >> > > > > >> > > > > I'm not sure if this is relevant, but with the failing >> commit >> > > I can >> > > > > get the >> > > > > kernel to start with `initrd_high` set to `0x70000000` or >> > > lower, >> > > > > but not >> > > > > higher. >> > > > >> > > > OK, so 8 whole banks, of 256MB each. Does this platform have >> 2GB >> > > of >> > > > memory for real? >> > > > >> > > > -- >> > > > Tom >> > > >> > > Yes, it does. See https://www.hardkernel.com/shop/odroid-u3/ . >> > >> > Thanks. I guess my current thought is that in the absence of >> adding >> > earlycon to the kernel args, to perhaps get more info out of the >> kernel >> > as to when / where it's dying, that bootm_size should be set to >> > 0x10000000 (256MB) to replicate the old behavior, on this >> platform. >> > Jaehoon, do you have any other ideas? >> > >> > -- >> > Tom >> >> I was not aware of the existence of `earlycon`. Running the first >> failing >> commit 4963f63f with `earlycon` and with `initrd_high` *unset* >> reveals a >> kernel panic: >> >> Kernel image @ 0x41000000 [ 0x000000 - 0x4be200 ] >> ## Flattened Device Tree blob at 40800000 >> Booting using the fdt blob at 0x40800000 >> Loading Ramdisk to b9947000, end bae7aa21 ... OK >> Loading Device Tree to b9936000, end b99460bf ... OK >> >> Starting kernel ... >> >> [ 0.000000] Booting Linux on physical CPU 0xa00 >> [ 0.000000] Linux version 5.10.0-20-armmp >> (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 >> 20210110, >> GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.158-2 >> (2022-12-13) >> [ 0.000000] CPU: ARMv7 Processor [413fc090] revision 0 >> (ARMv7), >> cr=10c5387d >> [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT >> aliasing >> instruction cache >> [ 0.000000] OF: fdt: Machine model: Hardkernel ODROID-U3 >> board based >> on Exynos4412 >> [ 0.000000] earlycon: exynos4210 at MMIO 0x13810000 (options >> '') >> [ 0.000000] printk: bootconsole [exynos4210] enabled >> [ 0.000000] Memory policy: Data cache writealloc >> [ 0.000000] efi: UEFI not found. >> [ 0.000000] Reserved memory: created DMA memory pool at >> 0xbf700000, >> size 8 MiB >> [ 0.000000] OF: reserved mem: initialized node >> region_mfc_right, >> compatible id shared-dma-pool >> [ 0.000000] Reserved memory: created DMA memory pool at >> 0xbd300000, >> size 36 MiB >> [ 0.000000] OF: reserved mem: initialized node >> region_mfc_left, >> compatible id shared-dma-pool >> [ 0.000000] cma: Reserved 16 MiB at 0xbc000000 >> [ 0.000000] Samsung CPU ID: 0xe4412220 >> [ 0.000000] Zone ranges: >> [ 0.000000] DMA [mem >> 0x0000000040000000-0x000000006fffffff] >> [ 0.000000] Normal empty >> [ 0.000000] HighMem [mem >> 0x0000000070000000-0x00000000bd2fffff] > > So here we go. Initrd cannot be in highmem. I cannot find this > documented, quickly, under https://www.kernel.org/doc/html/latest/ so > I > guess this is more just background knowledge than formal knowledge. > This > also matches up with your noting that initrd_high=0x70000000 works. > > -- > Tom I found this [1]. Section 5 mentions placing the initrd in lowmem, if I understand everything correctly. Setting `bootm_size` to ` 0x10000000` works for me. I can write a patch that adds `bootm_size=0x100000000` to `CFG_EXTRA_ENV_SETTINGS` in `include/configs/odroid.h` if you want, but I don't understand the fix, high mem and low mem well enough to write a decent commit message. I can certainly test (a backport of, see bug reports [2], [3]) a patch. Thanks for identifying the problem and proposing a fix! Best, Joost PS: I didn't subscribe to the mailing list, so if someone sends a patch, please CC me. [1]: https://www.kernel.org/doc/Documentation/arm/booting.rst [2]: https://lists.denx.de/pipermail/u-boot/2023-January/504115.html [3]: https://lists.denx.de/pipermail/u-boot/2023-January/504116.html