Question regarding OpenBMC crashing and messages of U-Boot SPL Already initialized

Question regarding OpenBMC crashing and messages of U-Boot SPL Already initialized

[Jeff Johnson]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

Greetings,

I'm seeing random crashing of openbmc 2.16.0-dev on an ASpeed AST2600. I am
trying to make sense of what I am seeing. Most of my experience is in the
x86 arch world, not Arm.

OpenBMC is running on ASpeed AST2600 devices, not QEMU. If I log console,
when a device crashes I see the below in the console log. Why would the SPL
be repeated trying to load when the system (BMC) is already up at init
level 3?

Thanks in advance for any feedback.

--Jeff

#--start console output--
[  OK  ] Started Phosphor Inband IPMI.
[  OK  ] Started Phosphor Time Manager daemon.
         Starting Phosphor IPMI KCS DBus Bridge...
[  OK  ] Started Network IPMI daemon.
         Starting VES Ethernet over USB gadget device...
[  OK  ] Started Phosphor IPMI KCS DBus Bridge.
[   32.252854] configfs-gadget.usbnet gadget.0: HOST MAC 4e:96:48:66:73:d7
[   32.260416] configfs-gadget.usbnet gadget.0: MAC d6:a8:1f:95:d9:0a

Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) nodistro.0
localhost ttyS4

localhost login: root
Password:
root@localhost:~# BP0c00

U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
already initialized,
U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
already initialized,
U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
already initialized,
U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
already initialized,
U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
already initialized,
U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
already initialized,
U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
already initialized,

[-- Attachment #2: Type: text/html, Size: 1848 bytes --]

Re: Question regarding OpenBMC crashing and messages of U-Boot SPL Already initialized

[Andrew Jeffery]

Hello,

On Fri, 2025-01-31 at 14:55 -0800, Jeff Johnson wrote:
> Greetings,
> 
> I'm seeing random crashing of openbmc 2.16.0-dev on an ASpeed
> AST2600. I am
> trying to make sense of what I am seeing. Most of my experience is in
> the
> x86 arch world, not Arm.
> 
> OpenBMC is running on ASpeed AST2600 devices, not QEMU. If I log
> console,
> when a device crashes I see the below in the console log. Why would
> the SPL
> be repeated trying to load when the system (BMC) is already up at
> init
> level 3?

It's likely that one of the hardware watchdogs is running and nothing
is feeding it, leading to a SoC reset.

> 
> Thanks in advance for any feedback.
> 
> --Jeff
> 
> #--start console output--
> [  OK  ] Started Phosphor Inband IPMI.
> [  OK  ] Started Phosphor Time Manager daemon.
>          Starting Phosphor IPMI KCS DBus Bridge...
> [  OK  ] Started Network IPMI daemon.
>          Starting VES Ethernet over USB gadget device...
> [  OK  ] Started Phosphor IPMI KCS DBus Bridge.
> [   32.252854] configfs-gadget.usbnet gadget.0: HOST MAC
> 4e:96:48:66:73:d7
> [   32.260416] configfs-gadget.usbnet gadget.0: MAC d6:a8:1f:95:d9:0a
> 
> Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro)
> nodistro.0
> localhost ttyS4
> 
> localhost login: root
> Password:
> root@localhost:~# BP0c00
> 
> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,
> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,
> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,
> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,
> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,
> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,
> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,

It's unusual that there's not other output interleaved here, which
suggests u-boot is getting stuck somewhere it shouldn't?

Can you capture the entire console output from BMC reset to the
occurrence of this condition, and then send it in reply here?

Some other details would help:

 * What platform are you testing on?
 * What's the upstream commit of openbmc/openbmc that you're testing?

Andrew

Re: Question regarding OpenBMC crashing and messages of U-Boot SPL Already initialized

[Joel Stanley]

On Mon, 3 Feb 2025 at 11:30, Andrew Jeffery <andrew@codeconstruct.com.au> wrote:
> On Fri, 2025-01-31 at 14:55 -0800, Jeff Johnson wrote:

> > localhost login: root
> > Password:
> > root@localhost:~# BP0c00

The string BP0c00 is from the BMC's bootrom. I forget how to decode
it, so I'm not sure if there is any useful info there, except that we
know the BMC is getting reset.

> U-Boot SPL 2019.04 (Oct 08 2021 - 00:31:50 +0000)
> already initialized,

The SPL knows that DRAM is already up ("already initialized") and is
trying to move to the next stage of booting. It doesn't make it there,
and we see it starting over. We don't see the message from the bootrom
though, so it's not a BMC reset occurring.

Cheers,

Joel