RPi4 U-Boot freeze

[Stefan Agner <stefan@agner.ch>]

On 2020-09-14 10:15, Matthias Brugger wrote:
> On 10/09/2020 23:12, Stefan Agner wrote:
>> On 2020-09-07 16:36, Peter Robinson wrote:
>>>> Any thoughts on this issue?
>>>
>>> Any reason why you're using 2020.01 and not at least 2020.07, or at
>>> least seeing if it's reproducible on 2020.10-rc3? The RPi4 support has
>>> changed quite a bit since then I suspect.
>>>
>>
>> Hi Peter,
>>
>> It's a stable release and we support a couple of devices with the same
>> U-Boot version. I'd rather prefer to stay with 2020.01 for RPi4 as well.
>>
>> We are on 2020.07 on development branch, and it does work fine there. So
>> I thought it can't be that hard, just bisect and backport whatever fixes
>> it... Unfortunately, it seems that there is no particular commit which
>> fixes it (the bisect ended up in a random unrelated commit, and it seems
>> that the issue appears/disappears depending on alignment/size...).
>>
>> I also did applied pretty much every RPi4 related commit made after
>> 2020.01 up until master back to 2020.01, no success either.
>>
> 
> Which version of the Raspberry Pi firmware did you use?
> Unfortunately changes in the FW breaks stuff on U-Boot from time to time.

The 4GB I left untouched so far, it came with the following setup:

pi at raspberrypi:~$ sudo rpi-eeprom-update
BCM2711 detected
Dedicated VL805 EEPROM detected
*** UPDATE AVAILABLE ***
BOOTLOADER: update available
CURRENT: Mon 15 Jul 12:59:55 UTC 2019 (1563195595)
 LATEST: Thu 16 Apr 17:11:26 UTC 2020 (1587057086)
 FW DIR: /lib/firmware/raspberrypi/bootloader/critical
VL805: update available
CURRENT: 00013701
 LATEST: 000137ad

The 2GB I did some firmware updates already, currently I ran it with the
following settings:

pi at raspberrypi:~$ sudo rpi-eeprom-update 
BCM2711 detected
Dedicated VL805 EEPROM detected
BOOTLOADER: up-to-date
CURRENT: Thu 16 Apr 17:11:26 UTC 2020 (1587057086)
 LATEST: Thu 16 Apr 17:11:26 UTC 2020 (1587057086)
 FW DIR: /lib/firmware/raspberrypi/bootloader/critical
VL805: up-to-date
CURRENT: 000138a1
 LATEST: 000137ad

I was able to reproduce the issue with U-Boot 2020.07, but I still have
two non-upstream patches ontop (I really can't see how they can affect
relocation, but they seem to cause a state which makes the issue
appear). I try to find a configuration which shows it without
non-upstream code.

--
Stefan


> 
> Regards,
> Mathias
> 
>> I fear that the problem in fact is still in master, but only appears if
>> certain things align a certain way... That is why I thought I bring it
>> up, to see if anybody else has noticed something along this lines. We do
>> have a rather trimmed down configuration, which might make the problem
>> appear more (fit in a D cache...).
>>
>> I probably will just disable cached around relocation for 2020.01 and
>> see if it resurfaces on development branch.
>>
>> --
>> Stefan
>>
>>
>>>> Just to be sure, I did some memory testing on the 2GB module, but no
>>>> issues found.
>>>>
>>>> I still somehow suspected that something else might be wrong with my
>>>> hardware, so I bought a new RPi4 (this time with 4GB of RAM) but the
>>>> very same with that:
>>>>
>>>> U-Boot 2020.01 (Aug 23 2020 - 22:02:31 +0000)
>>>>
>>>> DRAM:  3.9 GiB
>>>> <freeze>
>>>>
>>>> I still think there is something wrong with caching. From what I
>>>> understand caches are enabled by the RPi (4) firmware. Is it safe to run
>>>> 32-bit ARM U-Boot with enabled caches?
>>>>
>>>> --
>>>> Stefan
>>>>
>>>> On 2020-08-23 19:06, Stefan Agner wrote:
>>>>> Hi,
>>>>>
>>>>> I noticed a quite common freeze when running 32-bit U-Boot 2020.01
>>>>> (rpi_4_32b_defconfig) on a 2GB RPi4 model:
>>>>>
>>>>> U-Boot 2020.01 (Aug 07 2020 - 13:00:23 +0000)
>>>>>
>>>>> DRAM:  1.9 GiB
>>>>> <freeze, no more output>
>>>>>
>>>>> This happens fairly often, I would say 4 out of 5 boot tries. However,
>>>>> if it boots, everything seems to run fine.
>>>>>
>>>>> The issue seems to go away when using 2020.04 or any newer release,
>>>>> however, when trying to find the actual patch fixing the issue using git
>>>>> bisect I ended up with a MMC merge request which really seems unrelated
>>>>> (36bdcf7f3b). It seems that the problem is quite evasive and disappears
>>>>> if certain structure are aligned differently etc.
>>>>>
>>>>> Enabling initcall debugging showed that U-Boot crashes right after
>>>>> relocation:
>>>>>
>>>>> ...
>>>>> initcall: 00016f2c
>>>>>
>>>>> RAM Configuration:
>>>>> Bank #0: 0 948 MiB
>>>>> Bank #1: 40000000 1 GiB
>>>>> Bank #2: 0 0 Bytes
>>>>> Bank #3: 0 0 Bytes
>>>>>
>>>>> DRAM:  1.9 GiB
>>>>> initcall: 00016bb8
>>>>> New Stack Pointer is: 3af6d9e0
>>>>> initcall: 00016da4
>>>>> initcall: 00016ef0
>>>>> initcall: 00016ef8
>>>>> initcall: 00016d38
>>>>> Relocation Offset is: 3b375000
>>>>> Relocating to 3b37d000, new gd at 3af78ec0, sp at 3af6d9e0
>>>>> initcall: 00016ec8 [clear_bss]
>>>>> initcall: 0004465c [display_options?? only appears sometimes]
>>>>> <freeze>
>>>>>
>>>>> I realized when using CONFIG_SYS_(I|D)CACHE_OFF=y the problem seems to
>>>>> disappear. But to be 100% certain that it is cache related, I used my
>>>>> original configuration (which is known to "reliably" freeze), and
>>>>> replaced 00016ec8 with 00008688 manually in the binary, essentially
>>>>> swapping out function pointers in "init_sequence_f" [00008688 is
>>>>> cleanup_before_linux, which flushes and disables I-cache/D-cache]. And
>>>>> indeed, that hacked up binary does boot reliably every time:
>>>>>
>>>>> ...
>>>>> initcall: 00016f2c
>>>>>
>>>>> RAM Configuration:
>>>>> Bank #0: 0 948 MiB
>>>>> Bank #1: 40000000 1 GiB
>>>>> Bank #2: 0 0 Bytes
>>>>> Bank #3: 0 0 Bytes
>>>>>
>>>>> DRAM:  1.9 GiB
>>>>> initcall: 00016bb8
>>>>> New Stack Pointer is: 3af6d9e0
>>>>> initcall: 00016da4
>>>>> initcall: 00016ef0
>>>>> initcall: 00016ef8
>>>>> initcall: 00016d38
>>>>> Relocation Offset is: 3b375000
>>>>> Relocating to 3b37d000, new gd at 3af78ec0, sp at 3af6d9e0
>>>>> initcall: 00008688
>>>>> initcall: 3b38c10c
>>>>> initcall: 3b38c114
>>>>> initcall: 000172e0 (relocated to 3b38c2e0)
>>>>> initcall: 0001712c (relocated to 3b38c12c)
>>>>> ...
>>>>>
>>>>>  From what I understand on RPi4 caches are enabled when entering U-Boot.
>>>>> I was wondering if the relocation code really can handle that?
>>>>>
>>>>> --
>>>>> Stefan
>>