* OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
@ 2024-05-20 17:32 Johannes Truschnigg
2024-05-21 1:05 ` Andrew Jeffery
2024-07-07 9:19 ` Johannes Truschnigg
0 siblings, 2 replies; 11+ messages in thread
From: Johannes Truschnigg @ 2024-05-20 17:32 UTC (permalink / raw)
To: openbmc
[-- Attachment #1: Type: text/plain, Size: 3709 bytes --]
Dear list,
I am new to this whole BMC-innards thing, and the very opposite of familiar
with embedded devices, so please bear with me on this :)
I have a Gigabyte MC12-LE0[0] mainboard in hand, which has an AST2500-based
integrated BMC running AMI-supplied (mildly customized, I gather) BMC firmware.
I would very much like to try to get OpenBMC to boot on this BMC instead of the
stock fw, but have hit a number of walls during my first baby steps with this
over the last few days, which is why I turn to you to seek help and guidance.
The BMC implementation seems lean very much onto the AST2500 Evaluation Board
(the unchanged name pops up in a number of places in the stock firmware), so I
would guess that the OpenBMC evb-ast2500 machine should be able to get
somewhere (and even if it's just to see OpenBMC's instead of AMI's Linux start,
and then crash hard due to some subtle incompatibility :)) at least.
Afaict, the AMI BMC firmware lineage uses another (custom, undisclosed?) image
format that seems to be called "FMH" internally. While there are some third
party tools[1] and patches to support it[2] floating around on the net, I
failed to get anywhere with either until now.
My understanding is that there are two paths forward:
1.) To make OpenBMC's build artifacts into FMH-style BLOBs, and find a way to
feed them to the stock fw's u-boot, which would ideally result in OpenBMC being
able to boot this way.
2.) To replace the stock fw's u-boot release with an OpenBMC u-boot that can
load the evb-ast2500 artifacts natively, and have the BMC boot OpenBMC that
way.
Can you please confirm that this assessment is sane, or correct me if it isn't?
If so, I would presume option 2.) to be the easier road to travel, but I am
somewhat stumped as to how I can get OpenBMC's u-boot onto the BMC, *ideally*
(but not necessarily) in a non-destructive way (as I do not know how to or have
a way to revert to the original state without access to the bootloader or even
stock firmware running).
So far, I tried to put the u-boot-evb-ast2500-v2019.04+git-r0.bin artifact from
my OpenBMC build results into a memory location on the BMC using TFTP, and `go`
to that address afterwards - but that effectively reloaded the *original*
u-boot from the vendor fw. I don't know enough about the inner workings of an
AST2500 or u-boot (or any other embedded bootloader, really), but if you could
enlighten me as to what has happened there, I'd be very grateful (... maybe
u-boot does a check if it's already reloacted itself towards end of phys.
memory, and directly jump there if that is deemed the case?) Do you have any
hints how I could achieve what I am trying to do? Is this even feasible?
I did find a Github issue[3] from 2017 which hints at others having tried
something like this, but no documented successes.
I'd be very grateful for your input on this matter. Please keep my address CC'd
as I am currently not subscribed to this list.
Thank you all very much for your time reading this! :)
PS: I've included some potentially useful u-boot monitor output from the stock
firmware at [4], as I am not sure if attachments of this size (or inline text)
would be OK on-list.
[0]: https://www.gigabyte.com/Enterprise/Server-Motherboard/MC12-LE0-rev-1x
[1]: https://github.com/ya-mouse/bmc-ami
[2]: https://github.com/ami-megarac/OSSW/blob/master/SPX-12/Kernel_amiext_ex-src/12-drivers_mtd_maps_fmh.h
[3]: https://github.com/openbmc/openbmc/issues/1649
[4]: https://paste.debian.net/hidden/43509e70/
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: https://johannes.truschnigg.info/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-20 17:32 OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions Johannes Truschnigg
@ 2024-05-21 1:05 ` Andrew Jeffery
2024-05-21 17:05 ` Johannes Truschnigg
2024-07-07 9:19 ` Johannes Truschnigg
1 sibling, 1 reply; 11+ messages in thread
From: Andrew Jeffery @ 2024-05-21 1:05 UTC (permalink / raw)
To: Johannes Truschnigg, openbmc
Hello,
On Mon, 2024-05-20 at 19:32 +0200, Johannes Truschnigg wrote:
> Dear list,
>
> I am new to this whole BMC-innards thing, and the very opposite of familiar
> with embedded devices, so please bear with me on this :)
:)
>
> The BMC implementation seems lean very much onto the AST2500 Evaluation Board
> (the unchanged name pops up in a number of places in the stock firmware), so I
> would guess that the OpenBMC evb-ast2500 machine should be able to get
> somewhere (and even if it's just to see OpenBMC's instead of AMI's Linux start,
> and then crash hard due to some subtle incompatibility :)) at least.
Perhaps. While the SoC itself is obviously supported, how it's
integrated into the rest of the platform will likely be very different
to how things are wired up on the EVB.
> My understanding is that there are two paths forward:
>
> 1.) To make OpenBMC's build artifacts into FMH-style BLOBs, and find a way to
> feed them to the stock fw's u-boot, which would ideally result in OpenBMC being
> able to boot this way.
>
> 2.) To replace the stock fw's u-boot release with an OpenBMC u-boot that can
> load the evb-ast2500 artifacts natively, and have the BMC boot OpenBMC that
> way.
>
> Can you please confirm that this assessment is sane, or correct me if it isn't?
Yeah, that's sane. I'm not familiar with FMH so can't really comment on
how feasible that approach might be, but my intuition is it's probably
a bunch of pain.
>
> If so, I would presume option 2.) to be the easier road to travel, but I am
> somewhat stumped as to how I can get OpenBMC's u-boot onto the BMC, *ideally*
> (but not necessarily) in a non-destructive way (as I do not know how to or have
> a way to revert to the original state without access to the bootloader or even
> stock firmware running).
Something to consider is using qemu to boot the proprietary firmware,
and do your experiments there. That way it's non-destructive in the
sense that it's all just software and you won't need to break out a
flash programmer.
A quick look at the BMC firmware from your [1] shows Gigabyte do their
usual thing and ship the entire flash image. I downloaded the zip file,
unpacked it, and booted it like
```
0 andrew@heihei:~$ mkdir /tmp/mc12-le0
0 andrew@heihei:~$ cd !$
cd /tmp/mc12-le0
0 andrew@heihei:/tmp/mc12-le0$ unzip ~/Downloads/server_firmware_ast2500_AMI_12.x.zip
Archive: /home/andrew/Downloads/server_firmware_ast2500_AMI_12.x.zip
creating: 126121/
inflating: 126121/bmc_fw_update_linux.sh
inflating: 126121/bmc_fw_update_uefi.nsh
inflating: 126121/bmc_fw_update_win.bat
inflating: 126121/BMC_Release_Note_126121.doc
creating: 126121/fw/
inflating: 126121/fw/126121.bin
inflating: 126121/fw/rom.ima_enc
inflating: 126121/GBT_BMC_Firmware_Upgrade_User_Guide_v006_pre.pdf
inflating: 126121/projects.txt
creating: 126121/Tools/
inflating: 126121/Tools/gigaflash
inflating: 126121/Tools/gigaflash.efi
inflating: 126121/Tools/gigaflash.exe
inflating: 126121/Tools/gigaflash_arm
inflating: 126121/Tools/gigaflash_arm.efi
inflating: 126121/Tools/gigaflash_x64
inflating: 126121/Tools/gigaflash_x64.exe
creating: 126121/Tools/x64/
inflating: 126121/Tools/x64/astio.cat
inflating: 126121/Tools/x64/astio.sys
creating: 126121/Tools/x64_Win8/
inflating: 126121/Tools/x64_Win8/astio.cat
inflating: 126121/Tools/x64_Win8/astio.Inf
inflating: 126121/Tools/x64_Win8/astio.sys
creating: 126121/Tools/x86/
inflating: 126121/Tools/x86/astio.sys
0 andrew@heihei:/tmp/mc12-le0$ cd 126121/fw
0 andrew@heihei:/tmp/mc12-le0/126121/fw$ binwalk 126121.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
163848 0x28008 CRC32 polynomial table, little endian
213604 0x34264 CRC32 polynomial table, little endian
393216 0x60000 JFFS2 filesystem, little endian
5636096 0x560000 CramFS filesystem, little endian, size: 40951808, version 2, sorted_dirs, CRC 0x417607BC, edition 0, 27801 blocks, 6575 files
46596160 0x2C70040 uImage header, header size: 64 bytes, header CRC: 0x596F847A, created: 2024-03-12 06:08:06, image size: 2792592 bytes, Data Address: 0x80008000, Entry Point: 0x80008000, data CRC: 0xC9C2F025, OS: Linux, CPU: ARM, image type: OS Kernel Image, compression type: none, image name: "Linux-3.14.17-ami"
46596224 0x2C70080 Linux kernel ARM boot executable zImage (little-endian)
46613079 0x2C74257 gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
49479680 0x2F30000 JFFS2 filesystem, little endian
50003968 0x2FB0000 CramFS filesystem, little endian, size: 5963776, version 2, sorted_dirs, CRC 0xBCBB0E63, edition 0, 1566 blocks, 131 files
0 andrew@heihei:/tmp/mc12-le0/126121/fw$ truncate -s 64M 126121.bin
0 andrew@heihei:/tmp/mc12-le0/126121/fw$ qemu-system-arm -M ast2500-evb,fmc-model=n25q512a -drive file=126121.bin,if=mtd,format=raw -nographic
qemu-system-arm: warning: Aspeed iBT has no chardev backend
U-Boot 2013.07 (Mar 12 2024 - 14:08:49)
I2C: ready
DRAM: 424 MiB
eSPI Handshake complete
OEM_BOARD_INIT - Start (BMC)
LPC mode
OEM_BOARD_INIT - End
Flash: Found SPI Chip Micron/Numonyx N25Q512A(0x20ba) 2x I/O READ, NORMAL WRITE
64 MiB
MMC:
*** Warning - bad CRC, using default environment
Un-Protected 1 sectors
Erasing Flash...
Erasing sector 4 ... ok.
Erased 1 sectors
Writing to Flash... done
Protected 1 sectors
Net: RTL8211E, EEECR = 0x00
RTL8211E, EEEAR = 0x00
RTL8211E, EEELPAR = 0x00
RTL8211E, LACR = 0x00
RTL8211E, LCR = 0x00
ast_eth0, ast_eth1
DRAM ECC enabled
Hit any key to stop autoboot: 0
Image to be booted is 1
EMMC and EXT4 is not enabled - Cannot locate kernel file in Root
Initing KCS...done
Uboot waiting for firmware update to start...
Uboot waiting for fwupdate to start timed out
Disabling Watchdog 2 Timer
AST2500EVB>
```
Not sure how productive that is, but maybe you can experiment.
If you enable networking and tftp on the qemu command-line you could
potentially boot your own kernels. Or you could mess with booting the
OpenBMC firmware as well.
If you want to do something more destructive and overwrite the firmware
on the flash, then you should probably have a flash programmer handy,
and/or become familiar with something like culvert (which I maintain):
https://github.com/amboar/culvert
Note that gigabyte do ship "gigaflash" in that zip file above, which is
a closed-source application similar to culvert, that derives from
Aspeed's own "socflash" tool.
>
> So far, I tried to put the u-boot-evb-ast2500-v2019.04+git-r0.bin artifact from
> my OpenBMC build results into a memory location on the BMC using TFTP, and `go`
> to that address afterwards - but that effectively reloaded the *original*
> u-boot from the vendor fw. I don't know enough about the inner workings of an
> AST2500 or u-boot (or any other embedded bootloader, really), but if you could
> enlighten me as to what has happened there, I'd be very grateful (... maybe
> u-boot does a check if it's already reloacted itself towards end of phys.
> memory, and directly jump there if that is deemed the case?)
>
u-boot is linked at 0 and uses XIP from the SPI-NOR (which is memory-
mapped at 0), so I suspect it has read out the vendor copy again.
> Do you have any
> hints how I could achieve what I am trying to do? Is this even feasible?
Given the above, you'd have to replace u-boot on the hardware.
>
> I did find a Github issue[3] from 2017 which hints at others having tried
> something like this, but no documented successes.
>
> I'd be very grateful for your input on this matter. Please keep my address CC'd
> as I am currently not subscribed to this list.
A potential collaborator on Gigabyte stuff is bielids. I don't have
their contact details, but you can find some of our past discussion
here:
https://github.com/amboar/culvert/issues/51
Andrew
>
> Thank you all very much for your time reading this! :)
>
>
> PS: I've included some potentially useful u-boot monitor output from the stock
> firmware at [4], as I am not sure if attachments of this size (or inline text)
> would be OK on-list.
>
>
> [0]: https://www.gigabyte.com/Enterprise/Server-Motherboard/MC12-LE0-rev-1x
> [1]: https://github.com/ya-mouse/bmc-ami
> [2]: https://github.com/ami-megarac/OSSW/blob/master/SPX-12/Kernel_amiext_ex-src/12-drivers_mtd_maps_fmh.h
> [3]: https://github.com/openbmc/openbmc/issues/1649
> [4]: https://paste.debian.net/hidden/43509e70/
>
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-21 1:05 ` Andrew Jeffery
@ 2024-05-21 17:05 ` Johannes Truschnigg
2024-05-22 1:29 ` Andrew Jeffery
0 siblings, 1 reply; 11+ messages in thread
From: Johannes Truschnigg @ 2024-05-21 17:05 UTC (permalink / raw)
To: Andrew Jeffery; +Cc: openbmc
[-- Attachment #1: Type: text/plain, Size: 7930 bytes --]
Hi Andrew,
thanks so much for your kind response! I've come across your project (culvert)
on my little quest to liberate my board's BMC already, and I'm eager to try
working with it as soon as I manage to build it (which will leventually
happen)!
But first, I need to share the good news: Yesterday, shortly after I had sent
my initial post to this list, my friend Paul on IRC supplied me with ideas and
instructions to get the OpenBMC stuff to fly on the Gigabyte-supplied AST2500,
and we had smashing success after a rather short while! We patched OpenBMC's
u-boot to assume a CONFIG_SYS_TEXT_BASE of 0x83000000, tftp'd the resulting
binary to that address, issued
`go 0x83000000`
in the AMI u-boot monitor, and... had successfully chainloaded the OpenBMC
u-boot on the Gigabyte BMC!
From there, I could load the OpenBMC FIT and went to a initrd rescue shell
rather quickly. The sequence of commands was, for posterity (assuming a TFTP
server on 192.168.2.123 can supply the mentioned artifacts after successful
DHCPv4 autoconfig):
```
### original/stock AMI uboot monitor:
AST2500EVB> dhcp
AST2500EVB> tftp 0x83000000 192.168.2.123:hacked-uboot
AST2500EVB> go 0x83000000
### chainloaded OpenBMC u-boot:
ast# setenv ethact ethernet@1e680000
ast# dhcp
ast# tftpboot 0x84000000 192.168.2.123:openbmc-fit-image
ast# setenv bootargs console=ttyS4,115200n8 root=/dev/ram rw enable-initrd-debug-sh debug-init-sh
ast# bootm 0x84000000
```
I have provided a copy of the `hacked-uboot` file contents at [0]. Its
sha256sum hexdigest reads
2937963c78f577355dc3bd15b8b8e57dcacf72fd74cd82854d1136bb26356e68
From there, I could reconfigure the BMC's Ethernet interface for my LAN,
shuttle in the OpenBMC squashfs-xz image via tftp onto tmpfs, loop-mount that,
and chroot into the resulting directory tree - the userland seems to work,
too! :)
What I am struggling with (as other plans leave me little time today
unfortunately) is how to boot into this environment after manually massaging
the rescue environment into the half-working state. If you have any advice on
how to streamline my boot process, I'd be eager to read about it!
On Tue, May 21, 2024 at 10:35:34AM +0930, Andrew Jeffery wrote:
> [...]
> > The BMC implementation seems lean very much onto the AST2500 Evaluation Board
> > (the unchanged name pops up in a number of places in the stock firmware), so I
> > would guess that the OpenBMC evb-ast2500 machine should be able to get
> > somewhere (and even if it's just to see OpenBMC's instead of AMI's Linux start,
> > and then crash hard due to some subtle incompatibility :)) at least.
>
> Perhaps. While the SoC itself is obviously supported, how it's
> integrated into the rest of the platform will likely be very different
> to how things are wired up on the EVB.
I think I've already discovered some of the rough edges that are still coming
for me and this attempt at liberating the Gigabyte BMC, as the host does not
seem to want to power on without the (stock) BMC running. That, and all the
other obstacles that are sure to come, I will try to tackle in time :)
> > My understanding is that there are two paths forward:
> >
> > 1.) To make OpenBMC's build artifacts into FMH-style BLOBs, and find a way to
> > feed them to the stock fw's u-boot, which would ideally result in OpenBMC being
> > able to boot this way.
> >
> > 2.) To replace the stock fw's u-boot release with an OpenBMC u-boot that can
> > load the evb-ast2500 artifacts natively, and have the BMC boot OpenBMC that
> > way.
> >
> > Can you please confirm that this assessment is sane, or correct me if it isn't?
>
> Yeah, that's sane. I'm not familiar with FMH so can't really comment on
> how feasible that approach might be, but my intuition is it's probably
> a bunch of pain.
Because of the successfull attempt at chainloading a hacked u-boot, I think we
can safely discard trying to travel down path #1 and skip learning more about
FMH than absolutely necessary :)
> > If so, I would presume option 2.) to be the easier road to travel, but I am
> > somewhat stumped as to how I can get OpenBMC's u-boot onto the BMC, *ideally*
> > (but not necessarily) in a non-destructive way (as I do not know how to or have
> > a way to revert to the original state without access to the bootloader or even
> > stock firmware running).
>
> Something to consider is using qemu to boot the proprietary firmware,
[...]
> ast_eth0, ast_eth1
> DRAM ECC enabled
> Hit any key to stop autoboot: 0
> Image to be booted is 1
> EMMC and EXT4 is not enabled - Cannot locate kernel file in Root
> Initing KCS...done
> Uboot waiting for firmware update to start...
> Uboot waiting for fwupdate to start timed out
> Disabling Watchdog 2 Timer
> AST2500EVB>
> ```
Thanks for this; while I've had success in booting the evb-2500ast artifacts
in QEMU (as documented by OpenBMC and QEMU), I had not successfully tried with
the AMI-supplied artifacts yet. This saved me a ton of time trying to figure
that out!
However, what I do not really understand about the situation in the emulator
is that `fmh` will not list any FMH-style blocks/partitions/area in that case,
whilst it will on the physical BMC. As a consequence, I cannot actually boot
the stock firmware in QEMU passt the u-boot stage, as that relies on `bootfmh`
and the FMH areas to be properly recognized.
Maybe the munging of the image size with `truncate` changes something that the
FMH format can't cope with? I am unsure, and haven't yet looked into the
public FMH bits to confirm this theory's merit.
> If you want to do something more destructive and overwrite the firmware
> on the flash, then you should probably have a flash programmer handy,
> and/or become familiar with something like culvert (which I maintain):
>
> https://github.com/amboar/culvert
I'll be sure to try my luck with culvert as soon as I am able, and I would
have tried working with it earlier more eagerly if I had not made such
unexpectedly quick progress with the other ways tried ;)
> Note that gigabyte do ship "gigaflash" in that zip file above, which is
> a closed-source application similar to culvert, that derives from
> Aspeed's own "socflash" tool.
I noticed that (but did not know about its lineage), but cannot get it to
actually work for me on my Linux-based host system (grml 2024.02 for amd64 at
the moment), neither for updating nor for dumping the BMC ROM.
Since the binary is not stripped (and seems to have some understanding of FMH
and whatever funny signature business that is going on in the `rom.ima_enc`
parts of the BMC FW ZIP archives), it might still be useful in other ways
after all...
> > [...]
>
> u-boot is linked at 0 and uses XIP from the SPI-NOR (which is memory-
> mapped at 0), so I suspect it has read out the vendor copy again.
That's what Paul and me guessed on IRC yesterday, and the first attempt at
working around that seems to have been successful, as stated above! :)
> [...]
> A potential collaborator on Gigabyte stuff is bielids. I don't have
> their contact details, but you can find some of our past discussion
> here:
>
> https://github.com/amboar/culvert/issues/51
Thanks, I've had this issue bookmarked since the start of last weekend
already, and your patient and helpful approach there actually was one of the
main motivating factors for me to actually embark on this journey :) It's kind
people like you who make FOSS a great world to engage in - so let me say thank
you for your service to the general public! :)
Have a nice day!
- Johannes
[0]: https://johannes.truschnigg.info/assets/AMI_ast2500_hacked_u-boot_0x8300000
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: https://johannes.truschnigg.info/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-21 17:05 ` Johannes Truschnigg
@ 2024-05-22 1:29 ` Andrew Jeffery
2024-05-27 18:09 ` Johannes Truschnigg
0 siblings, 1 reply; 11+ messages in thread
From: Andrew Jeffery @ 2024-05-22 1:29 UTC (permalink / raw)
To: Johannes Truschnigg; +Cc: openbmc
On Tue, 2024-05-21 at 19:05 +0200, Johannes Truschnigg wrote:
> Hi Andrew,
>
> thanks so much for your kind response! I've come across your project (culvert)
> on my little quest to liberate my board's BMC already, and I'm eager to try
> working with it as soon as I manage to build it (which will leventually
> happen)!
>
> But first, I need to share the good news: Yesterday, shortly after I had sent
> my initial post to this list, my friend Paul on IRC supplied me with ideas and
> instructions to get the OpenBMC stuff to fly on the Gigabyte-supplied AST2500,
> and we had smashing success after a rather short while! We patched OpenBMC's
> u-boot to assume a CONFIG_SYS_TEXT_BASE of 0x83000000, tftp'd the resulting
> binary to that address, issued
>
> `go 0x83000000`
>
> in the AMI u-boot monitor, and... had successfully chainloaded the OpenBMC
> u-boot on the Gigabyte BMC!
Nice!
>
> From there, I could load the OpenBMC FIT and went to a initrd rescue shell
> rather quickly. The sequence of commands was, for posterity (assuming a TFTP
> server on 192.168.2.123 can supply the mentioned artifacts after successful
> DHCPv4 autoconfig):
>
> ```
> ### original/stock AMI uboot monitor:
> AST2500EVB> dhcp
> AST2500EVB> tftp 0x83000000 192.168.2.123:hacked-uboot
> AST2500EVB> go 0x83000000
>
> ### chainloaded OpenBMC u-boot:
> ast# setenv ethact ethernet@1e680000
> ast# dhcp
> ast# tftpboot 0x84000000 192.168.2.123:openbmc-fit-image
> ast# setenv bootargs console=ttyS4,115200n8 root=/dev/ram rw enable-initrd-debug-sh debug-init-sh
> ast# bootm 0x84000000
> ```
>
> I have provided a copy of the `hacked-uboot` file contents at [0]. Its
> sha256sum hexdigest reads
> 2937963c78f577355dc3bd15b8b8e57dcacf72fd74cd82854d1136bb26356e68
>
> From there, I could reconfigure the BMC's Ethernet interface for my LAN,
> shuttle in the OpenBMC squashfs-xz image via tftp onto tmpfs, loop-mount that,
> and chroot into the resulting directory tree - the userland seems to work,
> too! :)
>
>
> What I am struggling with (as other plans leave me little time today
> unfortunately) is how to boot into this environment after manually massaging
> the rescue environment into the half-working state. If you have any advice on
> how to streamline my boot process, I'd be eager to read about it!
Perhaps using `setenv` to write a u-boot script with the commands above
(executed via the `run` command`) along with `saveenv` might get you
some way to automation?
Anyway, nice work.
>
>
> On Tue, May 21, 2024 at 10:35:34AM +0930, Andrew Jeffery wrote:
> > [...]
> > > The BMC implementation seems lean very much onto the AST2500 Evaluation Board
> > > (the unchanged name pops up in a number of places in the stock firmware), so I
> > > would guess that the OpenBMC evb-ast2500 machine should be able to get
> > > somewhere (and even if it's just to see OpenBMC's instead of AMI's Linux start,
> > > and then crash hard due to some subtle incompatibility :)) at least.
> >
> > Perhaps. While the SoC itself is obviously supported, how it's
> > integrated into the rest of the platform will likely be very different
> > to how things are wired up on the EVB.
>
> I think I've already discovered some of the rough edges that are still coming
> for me and this attempt at liberating the Gigabyte BMC, as the host does not
> seem to want to power on without the (stock) BMC running. That, and all the
> other obstacles that are sure to come, I will try to tackle in time :)
:)
>
> > > If so, I would presume option 2.) to be the easier road to travel, but I am
> > > somewhat stumped as to how I can get OpenBMC's u-boot onto the BMC, *ideally*
> > > (but not necessarily) in a non-destructive way (as I do not know how to or have
> > > a way to revert to the original state without access to the bootloader or even
> > > stock firmware running).
> >
> > Something to consider is using qemu to boot the proprietary firmware,
> [...]
> > ast_eth0, ast_eth1
> > DRAM ECC enabled
> > Hit any key to stop autoboot: 0
> > Image to be booted is 1
> > EMMC and EXT4 is not enabled - Cannot locate kernel file in Root
> > Initing KCS...done
> > Uboot waiting for firmware update to start...
> > Uboot waiting for fwupdate to start timed out
> > Disabling Watchdog 2 Timer
> > AST2500EVB>
> > ```
>
> Thanks for this; while I've had success in booting the evb-2500ast artifacts
> in QEMU (as documented by OpenBMC and QEMU), I had not successfully tried with
> the AMI-supplied artifacts yet. This saved me a ton of time trying to figure
> that out!
>
> However, what I do not really understand about the situation in the emulator
> is that `fmh` will not list any FMH-style blocks/partitions/area in that case,
> whilst it will on the physical BMC. As a consequence, I cannot actually boot
> the stock firmware in QEMU passt the u-boot stage, as that relies on `bootfmh`
> and the FMH areas to be properly recognized.
>
> Maybe the munging of the image size with `truncate` changes something that the
> FMH format can't cope with?
>
I doubt it; the `truncate` command I used actually extends the image
out to the full 64M. The image supplied by Gigabyte is a few bytes
short, and qemu barfs if the flash image size does not exactly match
the size of the emulated SPI-NOR part.
But yeah, I'm also unsure about the FMH stuff. Perhaps something
interesting is that the u-boot output under qemu talks about eMMC not
being enabled:
> EMMC and EXT4 is not enabled - Cannot locate kernel file in Root
Not sure of what influences that output, but maybe try hooking up the
eMMC as well?
>
> > Note that gigabyte do ship "gigaflash" in that zip file above, which is
> > a closed-source application similar to culvert, that derives from
> > Aspeed's own "socflash" tool.
>
> I noticed that (but did not know about its lineage), but cannot get it to
> actually work for me on my Linux-based host system (grml 2024.02 for amd64 at
> the moment), neither for updating nor for dumping the BMC ROM.
>
> Since the binary is not stripped (and seems to have some understanding of FMH
> and whatever funny signature business that is going on in the `rom.ima_enc`
> parts of the BMC FW ZIP archives), it might still be useful in other ways
> after all...
>
Nice.
>
> > > [...]
> >
> > u-boot is linked at 0 and uses XIP from the SPI-NOR (which is memory-
> > mapped at 0), so I suspect it has read out the vendor copy again.
>
> That's what Paul and me guessed on IRC yesterday, and the first attempt at
> working around that seems to have been successful, as stated above! :)
Glad we were all on the right track!
>
>
> > [...]
> > A potential collaborator on Gigabyte stuff is bielids. I don't have
> > their contact details, but you can find some of our past discussion
> > here:
> >
> > https://github.com/amboar/culvert/issues/51
>
> Thanks, I've had this issue bookmarked since the start of last weekend
> already, and your patient and helpful approach there actually was one of the
> main motivating factors for me to actually embark on this journey :) It's kind
> people like you who make FOSS a great world to engage in - so let me say thank
> you for your service to the general public! :)
Ah, thanks for the kind words :)
Andrew
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-22 1:29 ` Andrew Jeffery
@ 2024-05-27 18:09 ` Johannes Truschnigg
2024-05-28 0:26 ` Andrew Jeffery
0 siblings, 1 reply; 11+ messages in thread
From: Johannes Truschnigg @ 2024-05-27 18:09 UTC (permalink / raw)
To: openbmc
[-- Attachment #1: Type: text/plain, Size: 20865 bytes --]
Hello again, dear list members!
I've spent some time with my/Gigabyte's/AMI's AST2500 today, and I might have
messed it up for good in the process, unwillingly. However, the result might
also present a fresh opportunity - I reall do not know! :) But first things
first!
Because I could not find any documentation about potantial OpenBMC nfsroot
support, I decided to "just try it", with unforseen consequences (anyone
remember Half-Life? :)) Since `picocom` was kind enough to preserve a session
log of the whole ordeal, the excerpt below should contain the gist of how I
messed up:
```
U-Boot 2013.07 (Mar 12 2024 - 14:08:49)
I2C: ready
DRAM: 424 MiB
eSPI Handshake complete
OEM_BOARD_INIT - Start (BMC)
LPC mode
OEM_BOARD_INIT - End
Flash: Found SPI Chip Micron/Numonyx N25Q512A(0x20ba) 2x I/O READ, NORMAL WRITE
64 MiB
MMC:
Net: RTL8211E, EEECR = 0x06
RTL8211E, EEEAR = 0x00
RTL8211E, EEELPAR = 0x06
RTL8211E, LACR = 0xc1
RTL8211E, LCR = 0x9742
ast_eth0, ast_eth1
DRAM ECC enabled
Hit any key to stop autoboot: 0
AST2500EVB>set autoload no
AST2500EVB>dhcp
BOOTP broadcast 1
[...]
BOOTP broadcast 25
Retry count exceeded; starting again
BOOTP broadcast 1
BOOTP broadcast 2
DHCP client bound to address 172.18.3.62
AST2500EVB>tftp 0x83000000 172.18.3.199:uboot
Using ast_eth1 device
TFTP from server 172.18.3.199; our IP address is 172.18.3.62
Filename 'uboot'.
Load address: 0x83000000
Loading: #################################################################
#########
366.7 KiB
done
Bytes transferred = 375501 (5bacd hex)
AST2500EVB>go 0x83000000
## Starting application at 0x83000000 ...
U-Boot 2019.04 (Mar 25 2024 - 04:43:19 +0000)
SOC : AST2500-A2
RST : WDT1
RST : WDT3 - Boot
LPC Mode : SIO:Disable
Eth : MAC0: RMII/NCSI, , MAC1: RGMII,
Model: AST2500 EVB
DRAM: 424 MiB (capacity:512 MiB, VGA:32 MiB, ECC:on, ECC size:424 MiB)
MMC: sdhci_slot0@100: 0, sdhci_slot1@200: 1
Loading Environment from SPI Flash... SF: Detected n25q512ax3 with page size 256 Bytes, erase size 4 KiB, total 64 MiB
*** Warning - bad CRC, using default environment
In: serial@1e784000
Out: serial@1e784000
Err: serial@1e784000
Net:
Warning: ethernet@1e660000 (eth0) using random MAC address - 56:4b:8e:16:48:eb
eth0: ethernet@1e660000
Warning: ethernet@1e680000 (eth1) using random MAC address - 32:f3:07:0a:d2:ad
, eth1: ethernet@1e680000
Hit any key to stop autoboot: 0
ast# setenv ethact ethernet@1e680000
ast# setenv autoload no
ast# dhcp
ethernet@1e680000: link up, 1000 Mbps full-duplex mac:32:f3:07:0a:d2:ad
BOOTP broadcast 1
BOOTP broadcast 2
BOOTP broadcast 3
BOOTP broadcast 4
DHCP client bound to address 172.18.3.83 (3007 ms)
ast# tftpboot 0x84000000 172.18.3.199:fit.bin
ethernet@1e680000: link up, 1000 Mbps full-duplex mac:32:f3:07:0a:d2:ad
Using ethernet@1e680000 device
TFTP from server 172.18.3.199; our IP address is 172.18.3.83
Filename 'fit.bin'.
Load address: 0x84000000
Loading: #################################################################
#######################################
5.6 MiB/s
done
Bytes transferred = 4375688 (42c488 hex)
ast# setenv bootargs console=ttyS4,115200n8 root=/dev/nfs rootfstype=nfs ip=172.18.3.173:172.18.3.138:172.18.3.254:255.255.255.0:bmctest:eth1:off:172.18.3.254::172.18.3.254 nfsroot=172.18.3.138:/,mountport=13337
ast# bootm 0x84000000
## Loading kernel from FIT Image at 84000000 ...
Using 'conf-aspeed-ast2500-evb.dtb' configuration
Trying 'kernel-1' kernel subimage
Description: Linux kernel
Type: Kernel Image
Compression: uncompressed
Data Start: 0x84000118
Data Size: 3257440 Bytes = 3.1 MiB
Architecture: ARM
OS: Linux
Load Address: 0x80001000
Entry Point: 0x80001000
Hash algo: sha256
Hash value: 0c874e335b61b661a7f1d638258ee1652728f021921b74bdcbad2bbb4ceef3f9
Verifying Hash Integrity ... sha256+ OK
## Loading ramdisk from FIT Image at 84000000 ...
Using 'conf-aspeed-ast2500-evb.dtb' configuration
Trying 'ramdisk-1' ramdisk subimage
Description: obmc-phosphor-initramfs
Type: RAMDisk Image
Compression: uncompressed
Data Start: 0x84322138
Data Size: 1088964 Bytes = 1 MiB
Architecture: ARM
OS: Linux
Load Address: unavailable
Entry Point: unavailable
Hash algo: sha256
Hash value: 94cc4badfdacec0b053b68954c69c55af5e50b3d8bc15b2852005fd6c84c238f
Verifying Hash Integrity ... sha256+ OK
## Loading fdt from FIT Image at 84000000 ...
Using 'conf-aspeed-ast2500-evb.dtb' configuration
Trying 'fdt-aspeed-ast2500-evb.dtb' fdt subimage
Description: Flattened Device Tree blob
Type: Flat Device Tree
Compression: uncompressed
Data Start: 0x8431b68c
Data Size: 27103 Bytes = 26.5 KiB
Architecture: ARM
Hash algo: sha256
Hash value: ca0066dfb087e1b1e022dec0c9aef3ac1c1ac350a49f5959e056acf64383b637
Verifying Hash Integrity ... sha256+ OK
Booting using the fdt blob at 0x8431b68c
Loading Kernel Image ... OK
Loading Ramdisk to 8fef6000, end 8ffffdc4 ... OK
Loading Device Tree to 8feec000, end 8fef59de ... OK
Starting kernel ...
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 6.6.30-f013890-00167-gf013890407d8 (oe-user@oe-host) (arm-openbmc-linux-gnueabi-gcc (GCC) 13.2.0, GNU ld (GNU Binutils) 2.42.0.20240216) #1 Mon May 13 00:55:18 UTC 2024
[ 0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
[ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
[ 0.000000] OF: fdt: Machine model: AST2500 EVB
[ 0.000000] Memory policy: Data cache writeback
[ 0.000000] Reserved memory: created CMA memory pool at 0x99000000, size 16 MiB
[ 0.000000] OF: reserved mem: initialized node framebuffer, compatible id shared-dma-pool
[ 0.000000] OF: reserved mem: 0x99000000..0x99ffffff (16384 KiB) map reusable framebuffer
[ 0.000000] cma: Reserved 16 MiB at 0x98000000 on node -1
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000080000000-0x000000009a7fffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000080000000-0x000000009a7fffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000009a7fffff]
[ 0.000000] Kernel command line: console=ttyS4,115200n8 root=/dev/nfs rootfstype=nfs ip=172.18.3.173:172.18.3.138:172.18.3.254:255.255.255.0:bmctest:eth1:off:172.18.3.254::172.18.3.254 nfsroot=172.18.3.138:/,mountport=13337
[ 0.000000] Unknown kernel command line parameters "ip=172.18.3.173:172.18.3.138:172.18.3.254:255.255.255.0:bmctest:eth1:off:172.18.3.254::172.18.3.254 nfsroot=172.18.3.138:/,mountport=13337", will be passed to user space.
[ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 107696
[ 0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[ 0.000000] Memory: 385236K/434176K available (7168K kernel code, 686K rwdata, 1664K rodata, 1024K init, 141K bss, 16172K reserved, 32768K cma-reserved)
[ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.000000] ftrace: allocating 24994 entries in 49 pages
[ 0.000000] ftrace: allocated 49 pages with 3 groups
[ 0.000000] trace event string verifier disabled
[ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[ 0.000000] i2c controller registered, irq 17
[ 0.000000] clocksource: FTTMR010-TIMER2: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 77222644334 ns
[ 0.000004] sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 86767015915ns
[ 0.000079] Switching to timer-based delay loop, resolution 40ns
[ 0.001275] Calibrating delay loop (skipped), value calculated using timer frequency.. 49.50 BogoMIPS (lpj=247500)
[ 0.001340] CPU: Testing write buffer coherency: ok
[ 0.001456] pid_max: default: 32768 minimum: 301
[ 0.011962] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.012029] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.027599] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1.
[ 0.028143] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1.
[ 0.028703] Setting up static identity map for 0x80100000 - 0x80100038
[ 0.030465] ASPEED AST2500 rev A2 (04030303)
[ 0.032008] devtmpfs: initialized
[ 0.053913] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.053996] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[ 0.060628] pinctrl core: initialized pinctrl subsystem
[ 0.065337] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[ 0.067525] DMA: preallocated 256 KiB pool for atomic coherent allocations
[ 0.070964] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
[ 0.071009] hw-breakpoint: maximum watchpoint size is 4 bytes.
[ 0.132580] mc: Linux media interface: v0.10
[ 0.132897] videodev: Linux video capture interface: v2.00
[ 0.133060] pps_core: LinuxPPS API ver. 1 registered
[ 0.133085] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 0.133161] PTP clock support registered
[ 0.142848] clocksource: Switched to clocksource FTTMR010-TIMER2
[ 0.174185] NET: Registered PF_INET protocol family
[ 0.174819] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.177558] tcp_listen_portaddr_hash hash table entries: 1024 (order: 0, 4096 bytes, linear)
[ 0.177651] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[ 0.177714] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
[ 0.177806] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
[ 0.178014] TCP: Hash tables configured (established 4096 bind 4096)
[ 0.178317] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[ 0.178411] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[ 0.179584] NET: Registered PF_UNIX/PF_LOCAL protocol family
[ 0.186070] Unpacking initramfs...
[ 0.213300] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[ 0.214001] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[ 0.214060] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc.
[ 0.263220] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
[ 0.273969] printk: console [ttyS4] disabled
[ 0.275132] 1e784000.serial: ttyS4 at MMIO 0x1e784000 (irq = 20, base_baud = 1500000) is a 16550A
[ 0.275268] printk: console [ttyS4] enabled
[ 0.935649] timeriomem_rng 1e6e2078.hwrng: 32bits from 0x(ptrval) @ 1us
[ 0.943808] aspeed_gfx 1e6e6000.display: assigned reserved memory node framebuffer
[ 0.967762] random: crng init done
[ 0.984563] [drm] Initialized aspeed-gfx-drm 1.0.0 20180319 for 1e6e6000.display on minor 0
[ 1.007161] aspeed_gfx 1e6e6000.display: [drm] fb0: aspeed-gfx-drmd frame buffer device
[ 1.055838] loop: module loaded
[ 1.141860] spi-nor spi0.0: mt25ql512a (65536 Kbytes)
[ 1.274576] spi-aspeed-smc 1e620000.spi: CE0 read buswidth:2 [0x203c0641]
[ 1.346136] 5 fixed-partitions partitions found on MTD device bmc
[ 1.352312] Creating 5 MTD partitions on "bmc":
[ 1.356984] 0x000000000000-0x000000060000 : "u-boot"
[ 1.385686] 0x000000060000-0x000000080000 : "u-boot-env"
[ 1.406591] 0x000000080000-0x0000004c0000 : "kernel"
[ 1.414849] 0x0000004c0000-0x000001c00000 : "rofs"
[ 1.436205] 0x000001c00000-0x000002000000 : "rwfs"
[ 1.448668] spi-nor spi1.0: unrecognized JEDEC id bytes: 00 03 00 00 00 00
[ 1.463133] ftgmac100 1e660000.ethernet: Error applying setting, reverse things back
[ 1.471391] ftgmac100 1e660000.ethernet: Read MAC address 56:4b:8e:16:48:eb from chip
[ 1.510983] Generic PHY 1e660000.ethernet--1:00: attached PHY driver (mii_bus:phy_addr=1e660000.ethernet--1:00, irq=POLL)
[ 1.523568] ftgmac100 1e660000.ethernet eth0: irq 22, mapped at 96ea64c4
[ 1.531266] ftgmac100 1e680000.ethernet: Read MAC address 32:f3:07:0a:d2:ad from chip
[ 1.570263] RTL8211E Gigabit Ethernet 1e680000.ethernet--1:00: attached PHY driver (mii_bus:phy_addr=1e680000.ethernet--1:00, irq=POLL)
[ 1.583921] ftgmac100 1e680000.ethernet eth1: irq 23, mapped at ec62a3ab
[ 1.746308] aspeed_vhub 1e6a0000.usb-vhub: Initialized virtual hub in USB2 mode
[ 1.754377] Mass Storage Function, version: 2009/09/11
[ 1.759586] LUN: removable file: (no medium)
[ 1.764058] no file given for LUN0
[ 1.767569] udc 1e6a0000.usb-vhub:p1: failed to start g_mass_storage: -22
[ 1.774490] g_mass_storage: probe of gadget.0 failed with error -22
[ 1.780961] Mass Storage Function, version: 2009/09/11
[ 1.786234] LUN: removable file: (no medium)
[ 1.790615] no file given for LUN0
[ 1.794196] udc 1e6a0000.usb-vhub:p2: failed to start g_mass_storage: -22
[ 1.801044] g_mass_storage: probe of gadget.1 failed with error -22
[ 1.807565] Mass Storage Function, version: 2009/09/11
[ 1.812765] LUN: removable file: (no medium)
[ 1.817213] no file given for LUN0
[ 1.820709] udc 1e6a0000.usb-vhub:p3: failed to start g_mass_storage: -22
[ 1.827628] g_mass_storage: probe of gadget.2 failed with error -22
[ 1.834157] Mass Storage Function, version: 2009/09/11
[ 1.839354] LUN: removable file: (no medium)
[ 1.843823] no file given for LUN0
[ 1.847327] udc 1e6a0000.usb-vhub:p4: failed to start g_mass_storage: -22
[ 1.854243] g_mass_storage: probe of gadget.3 failed with error -22
[ 1.860710] Mass Storage Function, version: 2009/09/11
[ 1.865982] LUN: removable file: (no medium)
[ 1.870363] no file given for LUN0
[ 1.873934] udc 1e6a0000.usb-vhub:p5: failed to start g_mass_storage: -22
[ 1.880778] g_mass_storage: probe of gadget.4 failed with error -22
[ 1.887290] UDC core: g_mass_storage: couldn't find an available UDC
[ 1.894510] i2c_dev: i2c /dev entries driver
[ 2.009530] aspeed-i2c-bus 1e78a100.i2c-bus: i2c bus 3 registered, irq 25
[ 2.033760] aspeed-i2c-bus 1e78a300.i2c-bus: i2c bus 7 registered, irq 26
[ 2.041576] Driver for 1-wire Dallas network protocol.
[ 2.080643] NET: Registered PF_INET6 protocol family
[ 2.114646] Segment Routing with IPv6
[ 2.118479] In-situ OAM (IOAM) with IPv6
[ 2.122989] NET: Registered PF_PACKET protocol family
[ 2.128116] 8021q: 802.1Q VLAN Support v1.8
[ 2.173478] printk: console [netcon0] enabled
[ 2.177927] netconsole: network logging started
[ 2.183614] clk: Disabling unused clocks
[ 2.368191] Freeing initrd memory: 1064K
[ 2.378419] Freeing unused kernel image (initmem) memory: 1024K
[ 2.386528] Checked W+X mappings: passed, no W+X pages found
[ 2.392272] Run /init as init process
rofs = mtd4 squashfs rwfs = mtd5 jffs2
mount: mounting /dev/mtdblock4 on run/initramfs/ro failed: Invalid argument
[ 3.427189] jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000000: 0x49f3 instead
[ 3.436958] jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000004: 0xe73d instead
[... XXX *many* more of those XXX at various offsets...]
[ 10.907145] jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x003f0024: 0x68b6 instead
[ 10.916740] jffs2: Further such events for this erase block will not be printed
[ 10.934585] jffs2: Cowardly refusing to erase blocks on filesystem with no valid JFFS2 nodes
[ 10.943154] jffs2: empty_blocks 0, bad_blocks 0, c->nr_blocks 64
mount: mounting /dev/mtdblock5 on run/initramfs/rw failed: Input/output error
Mounting read-write /dev/mtdblock5 filesystem failed. Please fix and run
mount /dev/mtdblock5 run/initramfs/rw -t jffs2 -o rw
or perform a factory reset with the clean-rwfs-filesystem option.
Fatal error, tri[ 10.990385] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[ 10.999145] CPU: 0 PID: 1 Comm: init Not tainted 6.6.30-f013890-00167-gf013890407d8 #1
[ 11.007102] Hardware name: Generic DT based system
[ 11.011934] unwind_backtrace from show_stack+0x18/0x1c
[ 11.017265] show_stack from dump_stack_lvl+0x24/0x2c
[ 11.022404] dump_stack_lvl from panic+0xf4/0x30c
[ 11.027180] panic from do_exit+0x8a8/0x8d0
[ 11.031441] do_exit from do_group_exit+0x40/0x84
[ 11.036209] do_group_exit from __wake_up_parent+0x0/0x1c
U-Boot 2013.07 (Mar 12 2024 - 14:08:49)
I2C: ready
DRAM: 424 MiB
eSPI Handshake complete
OEM_BOARD_INIT - Start (BMC)
LPC mode
OEM_BOARD_INIT - End
Flash: Found SPI Chip Micron/Numonyx N25Q512A(0x20ba) 2x I/O READ, NORMAL WRITE
64 MiB
MMC:
*** Warning - bad CRC, using default environment
Un-Protected 1 sectors
Erasing Flash...
Erasing sector 4 ... ok.
Erased 1 sectors
Writing to Flash... done
Protected 1 sectors
Net: RTL8211E, EEECR = 0x06
RTL8211E, EEEAR = 0x00
RTL8211E, EEELPAR = 0x06
RTL8211E, LACR = 0xc1
RTL8211E, LCR = 0x9742
ast_eth0, ast_eth1
DRAM ECC enabled
Hit any key to stop autoboot: 0
AST2500EVB>
```
So afaiui my attempt misfired, the kernel panic'd, reset the system, and
then... u-boot kinda decided to erase flash sector 4 (why?), which seems to have contained something important for (the original u-boot from Gigabyte on SPI flash) to boot. I did not realize that and cut power, waiting for a few secs before powering up again... only to notice that my UART serial console was staying dark this time. Meanwhile, the BMC NIC LEDs blink in a strange loop that hints at some restart procedure that is repeated ad nauseam.
To my luck, however, the host itself still willingly boots into the live system that I had set up before, but with a subtle difference once it's loaded: The AST2500 seems to be in a state much more amenable to Andrew's `culvert` utility now, which reports its neighborhood BMC like this now:
```
root@grml ~ # /tmp/culvert probe
[*] failed to initialise devmem bridge: -1
[*] Accessing the BMC's AHB via the p2a bridge
debug: Permissive
Debug UART port: UART5
xdma: Restricted
BMC: Disabled
VGA: Enabled
XDMA on VGA: Enabled
XDMA is constrained: Yes
p2a: Permissive
BMC: Disabled
VGA: Enabled
MMIO on VGA: Enabled
[0x00000000 - 0x0fffffff] Firmware: Writable
[0x10000000 - 0x1fffffff] SoC IO: Writable
[0x20000000 - 0x2fffffff] BMC Flash: Writable
[0x30000000 - 0x3fffffff] Host Flash: Writable
[0x40000000 - 0x5fffffff] Reserved: Writable
[0x60000000 - 0x7fffffff] LPC Host: Writable
[0x80000000 - 0xffffffff] DRAM: Writable
ilpc: Permissive
SuperIO address: 0x2e
```
(Before my mishap, whenever I tried to run that command, all but `ilpc` yielded
nothing, and even `ilpc` reported "Restricted" - none of which I know how to
interpret at all, by the way! :D)
So I guess I have at least six questions now:
1.) What happened when the kernel called it quits, u-boot reloaded and decided
to format some of its flash?
2.) Can I influence this behavior on my (spare) board before I try again?
3.) Can I restore the original firmware/SPI content on my board by any means
from the now running host OS? If so, what way would you suggest I try first?
4.) Does having `culvert` have this new level of access have any new advantages
or open possibilities that I should be aware of?
5.) Suppose I can restore the BMC's original SPI content and behavior - what's
a recommended way to have the TFPT'd kernel boot into an OpenBMC rootfs
*without* having it store on the BMC's main storage/overwriting SPI?
6.) Assuming this cannot be recovered in software - what are my chances of
identifying the SPI flash on my board as such, and re-writing its contents
using an affordable SPI programmer solution, given that I've never done
anything like this with hardware before? :^)
If any of you could answer me any of these, I would be very grateful indeed.
Thanks a lot for your time reading this far! :)
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: https://johannes.truschnigg.info/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-27 18:09 ` Johannes Truschnigg
@ 2024-05-28 0:26 ` Andrew Jeffery
2024-05-28 18:37 ` Johannes Truschnigg
0 siblings, 1 reply; 11+ messages in thread
From: Andrew Jeffery @ 2024-05-28 0:26 UTC (permalink / raw)
To: Johannes Truschnigg, openbmc
On Mon, 2024-05-27 at 20:09 +0200, Johannes Truschnigg wrote:
> Hello again, dear list members!
>
> I've spent some time with my/Gigabyte's/AMI's AST2500 today, and I might have
> messed it up for good in the process, unwillingly. However, the result might
> also present a fresh opportunity - I reall do not know! :) But first things
> first!
>
> Because I could not find any documentation about potantial OpenBMC nfsroot
> support, I decided to "just try it", with unforseen consequences (anyone
> remember Half-Life? :)) Since `picocom` was kind enough to preserve a session
> log of the whole ordeal, the excerpt below should contain the gist of how I
> messed up:
>
>
> ```
>
> U-Boot 2013.07 (Mar 12 2024 - 14:08:49)
>
> I2C: ready
> DRAM: 424 MiB
> eSPI Handshake complete
> OEM_BOARD_INIT - Start (BMC)
> LPC mode
> OEM_BOARD_INIT - End
> Flash: Found SPI Chip Micron/Numonyx N25Q512A(0x20ba) 2x I/O READ, NORMAL WRITE
> 64 MiB
> MMC:
> Net: RTL8211E, EEECR = 0x06
> RTL8211E, EEEAR = 0x00
> RTL8211E, EEELPAR = 0x06
> RTL8211E, LACR = 0xc1
> RTL8211E, LCR = 0x9742
> ast_eth0, ast_eth1
> DRAM ECC enabled
> Hit any key to stop autoboot: 0
> AST2500EVB>set autoload no
> AST2500EVB>dhcp
> BOOTP broadcast 1
> [...]
> BOOTP broadcast 25
>
> Retry count exceeded; starting again
> BOOTP broadcast 1
> BOOTP broadcast 2
> DHCP client bound to address 172.18.3.62
> AST2500EVB>tftp 0x83000000 172.18.3.199:uboot
> Using ast_eth1 device
> TFTP from server 172.18.3.199; our IP address is 172.18.3.62
> Filename 'uboot'.
> Load address: 0x83000000
> Loading: #################################################################
> #########
> 366.7 KiB
> done
> Bytes transferred = 375501 (5bacd hex)
> AST2500EVB>go 0x83000000
> ## Starting application at 0x83000000 ...
>
>
> U-Boot 2019.04 (Mar 25 2024 - 04:43:19 +0000)
>
> SOC : AST2500-A2
> RST : WDT1
> RST : WDT3 - Boot
> LPC Mode : SIO:Disable
> Eth : MAC0: RMII/NCSI, , MAC1: RGMII,
> Model: AST2500 EVB
> DRAM: 424 MiB (capacity:512 MiB, VGA:32 MiB, ECC:on, ECC size:424 MiB)
> MMC: sdhci_slot0@100: 0, sdhci_slot1@200: 1
> Loading Environment from SPI Flash... SF: Detected n25q512ax3 with page size 256 Bytes, erase size 4 KiB, total 64 MiB
> *** Warning - bad CRC, using default environment
>
> In: serial@1e784000
> Out: serial@1e784000
> Err: serial@1e784000
> Net:
> Warning: ethernet@1e660000 (eth0) using random MAC address - 56:4b:8e:16:48:eb
> eth0: ethernet@1e660000
> Warning: ethernet@1e680000 (eth1) using random MAC address - 32:f3:07:0a:d2:ad
> , eth1: ethernet@1e680000
> Hit any key to stop autoboot: 0
> ast# setenv ethact ethernet@1e680000
> ast# setenv autoload no
> ast# dhcp
> ethernet@1e680000: link up, 1000 Mbps full-duplex mac:32:f3:07:0a:d2:ad
> BOOTP broadcast 1
> BOOTP broadcast 2
> BOOTP broadcast 3
> BOOTP broadcast 4
> DHCP client bound to address 172.18.3.83 (3007 ms)
> ast# tftpboot 0x84000000 172.18.3.199:fit.bin
> ethernet@1e680000: link up, 1000 Mbps full-duplex mac:32:f3:07:0a:d2:ad
> Using ethernet@1e680000 device
> TFTP from server 172.18.3.199; our IP address is 172.18.3.83
> Filename 'fit.bin'.
> Load address: 0x84000000
> Loading: #################################################################
> #######################################
> 5.6 MiB/s
> done
> Bytes transferred = 4375688 (42c488 hex)
> ast# setenv bootargs console=ttyS4,115200n8 root=/dev/nfs rootfstype=nfs ip=172.18.3.173:172.18.3.138:172.18.3.254:255.255.255.0:bmctest:eth1:off:172.18.3.254::172.18.3.254 nfsroot=172.18.3.138:/,mountport=13337
> ast# bootm 0x84000000
> ## Loading kernel from FIT Image at 84000000 ...
> Using 'conf-aspeed-ast2500-evb.dtb' configuration
> Trying 'kernel-1' kernel subimage
> Description: Linux kernel
> Type: Kernel Image
> Compression: uncompressed
> Data Start: 0x84000118
> Data Size: 3257440 Bytes = 3.1 MiB
> Architecture: ARM
> OS: Linux
> Load Address: 0x80001000
> Entry Point: 0x80001000
> Hash algo: sha256
> Hash value: 0c874e335b61b661a7f1d638258ee1652728f021921b74bdcbad2bbb4ceef3f9
> Verifying Hash Integrity ... sha256+ OK
> ## Loading ramdisk from FIT Image at 84000000 ...
> Using 'conf-aspeed-ast2500-evb.dtb' configuration
> Trying 'ramdisk-1' ramdisk subimage
> Description: obmc-phosphor-initramfs
> Type: RAMDisk Image
> Compression: uncompressed
> Data Start: 0x84322138
> Data Size: 1088964 Bytes = 1 MiB
> Architecture: ARM
> OS: Linux
> Load Address: unavailable
> Entry Point: unavailable
> Hash algo: sha256
> Hash value: 94cc4badfdacec0b053b68954c69c55af5e50b3d8bc15b2852005fd6c84c238f
> Verifying Hash Integrity ... sha256+ OK
> ## Loading fdt from FIT Image at 84000000 ...
> Using 'conf-aspeed-ast2500-evb.dtb' configuration
> Trying 'fdt-aspeed-ast2500-evb.dtb' fdt subimage
> Description: Flattened Device Tree blob
> Type: Flat Device Tree
> Compression: uncompressed
> Data Start: 0x8431b68c
> Data Size: 27103 Bytes = 26.5 KiB
> Architecture: ARM
> Hash algo: sha256
> Hash value: ca0066dfb087e1b1e022dec0c9aef3ac1c1ac350a49f5959e056acf64383b637
> Verifying Hash Integrity ... sha256+ OK
> Booting using the fdt blob at 0x8431b68c
> Loading Kernel Image ... OK
> Loading Ramdisk to 8fef6000, end 8ffffdc4 ... OK
> Loading Device Tree to 8feec000, end 8fef59de ... OK
>
> Starting kernel ...
>
> [ 0.000000] Booting Linux on physical CPU 0x0
> [ 0.000000] Linux version 6.6.30-f013890-00167-gf013890407d8 (oe-user@oe-host) (arm-openbmc-linux-gnueabi-gcc (GCC) 13.2.0, GNU ld (GNU Binutils) 2.42.0.20240216) #1 Mon May 13 00:55:18 UTC 2024
> [ 0.000000] CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
> [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
> [ 0.000000] OF: fdt: Machine model: AST2500 EVB
> [ 0.000000] Memory policy: Data cache writeback
> [ 0.000000] Reserved memory: created CMA memory pool at 0x99000000, size 16 MiB
> [ 0.000000] OF: reserved mem: initialized node framebuffer, compatible id shared-dma-pool
> [ 0.000000] OF: reserved mem: 0x99000000..0x99ffffff (16384 KiB) map reusable framebuffer
> [ 0.000000] cma: Reserved 16 MiB at 0x98000000 on node -1
> [ 0.000000] Zone ranges:
> [ 0.000000] Normal [mem 0x0000000080000000-0x000000009a7fffff]
> [ 0.000000] Movable zone start for each node
> [ 0.000000] Early memory node ranges
> [ 0.000000] node 0: [mem 0x0000000080000000-0x000000009a7fffff]
> [ 0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x000000009a7fffff]
> [ 0.000000] Kernel command line: console=ttyS4,115200n8 root=/dev/nfs rootfstype=nfs ip=172.18.3.173:172.18.3.138:172.18.3.254:255.255.255.0:bmctest:eth1:off:172.18.3.254::172.18.3.254 nfsroot=172.18.3.138:/,mountport=13337
> [ 0.000000] Unknown kernel command line parameters "ip=172.18.3.173:172.18.3.138:172.18.3.254:255.255.255.0:bmctest:eth1:off:172.18.3.254::172.18.3.254 nfsroot=172.18.3.138:/,mountport=13337", will be passed to user space.
> [ 0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
> [ 0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
> [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 107696
> [ 0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
> [ 0.000000] Memory: 385236K/434176K available (7168K kernel code, 686K rwdata, 1664K rodata, 1024K init, 141K bss, 16172K reserved, 32768K cma-reserved)
> [ 0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
> [ 0.000000] ftrace: allocating 24994 entries in 49 pages
> [ 0.000000] ftrace: allocated 49 pages with 3 groups
> [ 0.000000] trace event string verifier disabled
> [ 0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
> [ 0.000000] i2c controller registered, irq 17
> [ 0.000000] clocksource: FTTMR010-TIMER2: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 77222644334 ns
> [ 0.000004] sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 86767015915ns
> [ 0.000079] Switching to timer-based delay loop, resolution 40ns
> [ 0.001275] Calibrating delay loop (skipped), value calculated using timer frequency.. 49.50 BogoMIPS (lpj=247500)
> [ 0.001340] CPU: Testing write buffer coherency: ok
> [ 0.001456] pid_max: default: 32768 minimum: 301
> [ 0.011962] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
> [ 0.012029] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
> [ 0.027599] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1.
> [ 0.028143] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1.
> [ 0.028703] Setting up static identity map for 0x80100000 - 0x80100038
> [ 0.030465] ASPEED AST2500 rev A2 (04030303)
> [ 0.032008] devtmpfs: initialized
> [ 0.053913] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
> [ 0.053996] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
> [ 0.060628] pinctrl core: initialized pinctrl subsystem
> [ 0.065337] NET: Registered PF_NETLINK/PF_ROUTE protocol family
> [ 0.067525] DMA: preallocated 256 KiB pool for atomic coherent allocations
> [ 0.070964] hw-breakpoint: found 6 breakpoint and 1 watchpoint registers.
> [ 0.071009] hw-breakpoint: maximum watchpoint size is 4 bytes.
> [ 0.132580] mc: Linux media interface: v0.10
> [ 0.132897] videodev: Linux video capture interface: v2.00
> [ 0.133060] pps_core: LinuxPPS API ver. 1 registered
> [ 0.133085] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
> [ 0.133161] PTP clock support registered
> [ 0.142848] clocksource: Switched to clocksource FTTMR010-TIMER2
> [ 0.174185] NET: Registered PF_INET protocol family
> [ 0.174819] IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
> [ 0.177558] tcp_listen_portaddr_hash hash table entries: 1024 (order: 0, 4096 bytes, linear)
> [ 0.177651] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
> [ 0.177714] TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
> [ 0.177806] TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
> [ 0.178014] TCP: Hash tables configured (established 4096 bind 4096)
> [ 0.178317] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
> [ 0.178411] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
> [ 0.179584] NET: Registered PF_UNIX/PF_LOCAL protocol family
> [ 0.186070] Unpacking initramfs...
> [ 0.213300] workingset: timestamp_bits=30 max_order=17 bucket_order=0
> [ 0.214001] squashfs: version 4.0 (2009/01/31) Phillip Lougher
> [ 0.214060] jffs2: version 2.2. (SUMMARY) © 2001-2006 Red Hat, Inc.
> [ 0.263220] Serial: 8250/16550 driver, 6 ports, IRQ sharing enabled
> [ 0.273969] printk: console [ttyS4] disabled
> [ 0.275132] 1e784000.serial: ttyS4 at MMIO 0x1e784000 (irq = 20, base_baud = 1500000) is a 16550A
> [ 0.275268] printk: console [ttyS4] enabled
> [ 0.935649] timeriomem_rng 1e6e2078.hwrng: 32bits from 0x(ptrval) @ 1us
> [ 0.943808] aspeed_gfx 1e6e6000.display: assigned reserved memory node framebuffer
> [ 0.967762] random: crng init done
> [ 0.984563] [drm] Initialized aspeed-gfx-drm 1.0.0 20180319 for 1e6e6000.display on minor 0
> [ 1.007161] aspeed_gfx 1e6e6000.display: [drm] fb0: aspeed-gfx-drmd frame buffer device
> [ 1.055838] loop: module loaded
> [ 1.141860] spi-nor spi0.0: mt25ql512a (65536 Kbytes)
> [ 1.274576] spi-aspeed-smc 1e620000.spi: CE0 read buswidth:2 [0x203c0641]
> [ 1.346136] 5 fixed-partitions partitions found on MTD device bmc
> [ 1.352312] Creating 5 MTD partitions on "bmc":
> [ 1.356984] 0x000000000000-0x000000060000 : "u-boot"
> [ 1.385686] 0x000000060000-0x000000080000 : "u-boot-env"
> [ 1.406591] 0x000000080000-0x0000004c0000 : "kernel"
> [ 1.414849] 0x0000004c0000-0x000001c00000 : "rofs"
> [ 1.436205] 0x000001c00000-0x000002000000 : "rwfs"
> [ 1.448668] spi-nor spi1.0: unrecognized JEDEC id bytes: 00 03 00 00 00 00
> [ 1.463133] ftgmac100 1e660000.ethernet: Error applying setting, reverse things back
> [ 1.471391] ftgmac100 1e660000.ethernet: Read MAC address 56:4b:8e:16:48:eb from chip
> [ 1.510983] Generic PHY 1e660000.ethernet--1:00: attached PHY driver (mii_bus:phy_addr=1e660000.ethernet--1:00, irq=POLL)
> [ 1.523568] ftgmac100 1e660000.ethernet eth0: irq 22, mapped at 96ea64c4
> [ 1.531266] ftgmac100 1e680000.ethernet: Read MAC address 32:f3:07:0a:d2:ad from chip
> [ 1.570263] RTL8211E Gigabit Ethernet 1e680000.ethernet--1:00: attached PHY driver (mii_bus:phy_addr=1e680000.ethernet--1:00, irq=POLL)
> [ 1.583921] ftgmac100 1e680000.ethernet eth1: irq 23, mapped at ec62a3ab
> [ 1.746308] aspeed_vhub 1e6a0000.usb-vhub: Initialized virtual hub in USB2 mode
> [ 1.754377] Mass Storage Function, version: 2009/09/11
> [ 1.759586] LUN: removable file: (no medium)
> [ 1.764058] no file given for LUN0
> [ 1.767569] udc 1e6a0000.usb-vhub:p1: failed to start g_mass_storage: -22
> [ 1.774490] g_mass_storage: probe of gadget.0 failed with error -22
> [ 1.780961] Mass Storage Function, version: 2009/09/11
> [ 1.786234] LUN: removable file: (no medium)
> [ 1.790615] no file given for LUN0
> [ 1.794196] udc 1e6a0000.usb-vhub:p2: failed to start g_mass_storage: -22
> [ 1.801044] g_mass_storage: probe of gadget.1 failed with error -22
> [ 1.807565] Mass Storage Function, version: 2009/09/11
> [ 1.812765] LUN: removable file: (no medium)
> [ 1.817213] no file given for LUN0
> [ 1.820709] udc 1e6a0000.usb-vhub:p3: failed to start g_mass_storage: -22
> [ 1.827628] g_mass_storage: probe of gadget.2 failed with error -22
> [ 1.834157] Mass Storage Function, version: 2009/09/11
> [ 1.839354] LUN: removable file: (no medium)
> [ 1.843823] no file given for LUN0
> [ 1.847327] udc 1e6a0000.usb-vhub:p4: failed to start g_mass_storage: -22
> [ 1.854243] g_mass_storage: probe of gadget.3 failed with error -22
> [ 1.860710] Mass Storage Function, version: 2009/09/11
> [ 1.865982] LUN: removable file: (no medium)
> [ 1.870363] no file given for LUN0
> [ 1.873934] udc 1e6a0000.usb-vhub:p5: failed to start g_mass_storage: -22
> [ 1.880778] g_mass_storage: probe of gadget.4 failed with error -22
> [ 1.887290] UDC core: g_mass_storage: couldn't find an available UDC
> [ 1.894510] i2c_dev: i2c /dev entries driver
> [ 2.009530] aspeed-i2c-bus 1e78a100.i2c-bus: i2c bus 3 registered, irq 25
> [ 2.033760] aspeed-i2c-bus 1e78a300.i2c-bus: i2c bus 7 registered, irq 26
> [ 2.041576] Driver for 1-wire Dallas network protocol.
> [ 2.080643] NET: Registered PF_INET6 protocol family
> [ 2.114646] Segment Routing with IPv6
> [ 2.118479] In-situ OAM (IOAM) with IPv6
> [ 2.122989] NET: Registered PF_PACKET protocol family
> [ 2.128116] 8021q: 802.1Q VLAN Support v1.8
> [ 2.173478] printk: console [netcon0] enabled
> [ 2.177927] netconsole: network logging started
> [ 2.183614] clk: Disabling unused clocks
> [ 2.368191] Freeing initrd memory: 1064K
> [ 2.378419] Freeing unused kernel image (initmem) memory: 1024K
> [ 2.386528] Checked W+X mappings: passed, no W+X pages found
> [ 2.392272] Run /init as init process
> rofs = mtd4 squashfs rwfs = mtd5 jffs2
> mount: mounting /dev/mtdblock4 on run/initramfs/ro failed: Invalid argument
> [ 3.427189] jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000000: 0x49f3 instead
> [ 3.436958] jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00000004: 0xe73d instead
>
> [... XXX *many* more of those XXX at various offsets...]
>
> [ 10.907145] jffs2: jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x003f0024: 0x68b6 instead
> [ 10.916740] jffs2: Further such events for this erase block will not be printed
> [ 10.934585] jffs2: Cowardly refusing to erase blocks on filesystem with no valid JFFS2 nodes
> [ 10.943154] jffs2: empty_blocks 0, bad_blocks 0, c->nr_blocks 64
> mount: mounting /dev/mtdblock5 on run/initramfs/rw failed: Input/output error
>
> Mounting read-write /dev/mtdblock5 filesystem failed. Please fix and run
> mount /dev/mtdblock5 run/initramfs/rw -t jffs2 -o rw
> or perform a factory reset with the clean-rwfs-filesystem option.
> Fatal error, tri[ 10.990385] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
> [ 10.999145] CPU: 0 PID: 1 Comm: init Not tainted 6.6.30-f013890-00167-gf013890407d8 #1
> [ 11.007102] Hardware name: Generic DT based system
> [ 11.011934] unwind_backtrace from show_stack+0x18/0x1c
> [ 11.017265] show_stack from dump_stack_lvl+0x24/0x2c
> [ 11.022404] dump_stack_lvl from panic+0xf4/0x30c
> [ 11.027180] panic from do_exit+0x8a8/0x8d0
> [ 11.031441] do_exit from do_group_exit+0x40/0x84
> [ 11.036209] do_group_exit from __wake_up_parent+0x0/0x1c
>
>
> U-Boot 2013.07 (Mar 12 2024 - 14:08:49)
>
> I2C: ready
> DRAM: 424 MiB
> eSPI Handshake complete
> OEM_BOARD_INIT - Start (BMC)
> LPC mode
> OEM_BOARD_INIT - End
> Flash: Found SPI Chip Micron/Numonyx N25Q512A(0x20ba) 2x I/O READ, NORMAL WRITE
> 64 MiB
> MMC:
> *** Warning - bad CRC, using default environment
>
> Un-Protected 1 sectors
> Erasing Flash...
> Erasing sector 4 ... ok.
> Erased 1 sectors
> Writing to Flash... done
> Protected 1 sectors
> Net: RTL8211E, EEECR = 0x06
> RTL8211E, EEEAR = 0x00
> RTL8211E, EEELPAR = 0x06
> RTL8211E, LACR = 0xc1
> RTL8211E, LCR = 0x9742
> ast_eth0, ast_eth1
> DRAM ECC enabled
> Hit any key to stop autoboot: 0
> AST2500EVB>
> ```
>
> So afaiui my attempt misfired, the kernel panic'd, reset the system, and
> then... u-boot kinda decided to erase flash sector 4 (why?),
>
Not sure, especially because it's the vendor u-boot :)
> which seems to have contained something important for (the original u-boot from Gigabyte on SPI flash) to boot. I did not realize that and cut power, waiting for a few secs before powering up again... only to notice that my UART serial console was staying dark this time. Meanwhile, the BMC NIC LEDs blink in a strange loop that hints at some restart procedure that is repeated ad nauseam.
>
> To my luck, however, the host itself still willingly boots into the live system that I had set up before, but with a subtle difference once it's loaded: The AST2500 seems to be in a state much more amenable to Andrew's `culvert` utility now, which reports its neighborhood BMC like this now:
>
> ```
> root@grml ~ # /tmp/culvert probe
> [*] failed to initialise devmem bridge: -1
> [*] Accessing the BMC's AHB via the p2a bridge
> debug: Permissive
> Debug UART port: UART5
> xdma: Restricted
> BMC: Disabled
> VGA: Enabled
> XDMA on VGA: Enabled
> XDMA is constrained: Yes
> p2a: Permissive
> BMC: Disabled
> VGA: Enabled
> MMIO on VGA: Enabled
> [0x00000000 - 0x0fffffff] Firmware: Writable
> [0x10000000 - 0x1fffffff] SoC IO: Writable
> [0x20000000 - 0x2fffffff] BMC Flash: Writable
> [0x30000000 - 0x3fffffff] Host Flash: Writable
> [0x40000000 - 0x5fffffff] Reserved: Writable
> [0x60000000 - 0x7fffffff] LPC Host: Writable
> [0x80000000 - 0xffffffff] DRAM: Writable
> ilpc: Permissive
> SuperIO address: 0x2e
> ```
>
> (Before my mishap, whenever I tried to run that command, all but `ilpc` yielded
> nothing, and even `ilpc` reported "Restricted" - none of which I know how to
> interpret at all, by the way! :D)
So I have a bit of an educated guess here:
1. For AST SoCs prior to the AST2600, disabling the hardware backdoors
must be done in firmware
2. Your process above has corrupted u-boot
3. The corruption is such that u-boot fails prior to applying the
backdoor mitigations (if that's where the mitigation was done to begin
with - however it's where we do the mitigation in OpenBMC)
As for the culvert 'probe' output and the
Permissive/Restricted/Disabled states:
- Permissive means there are no constraints on the bridge - culvert can
read and write any AHB address
- Restricted means some constraints apply to the bridge - either the
address space is restricted (e.g. XDMA is constrained to the VGA
reserved memory), or write access is disabled for some portion or all
of the AHB address space (e.g. the P2A write filters for the listed
regions)
- Disabled means what it says on the tin, the BMC's AHB address space
cannot be accessed via the bridge at hand, we cannot read or write.
>
>
> So I guess I have at least six questions now:
>
> 1.) What happened when the kernel called it quits, u-boot reloaded and decided
> to format some of its flash?
Not sure, but you'd do well to boot a kernel that doesn't try to mount
partitions from the flash.
>
> 2.) Can I influence this behavior on my (spare) board before I try again?
With the above suggestion it largely comes back to how you assemble the
initrd used by the kernel
>
> 3.) Can I restore the original firmware/SPI content on my board by any means
> from the now running host OS? If so, what way would you suggest I try first?
> 4.) Does having `culvert` have this new level of access have any new advantages
> or open possibilities that I should be aware of?
So one of the motivations for culvert is to reflash the BMC over the
AHB bridges reported in the probe output. This works regardless of the
state of the BMC firmware, so long as it hasn't disabled the hardware
backdoors. You can try it with e.g.
```
# culvert -vv write firmware < $firmware_image
```
(you may want to experiment with `culvert -vv read firmware` first).
That said, experience in [1] suggests Gigabyte have introduced some
gremlins that aren't accounted for by culvert, and that you might have
more success with gigaflash.
[1]:
https://github.com/amboar/culvert/issues/51#issuecomment-2129043859
>
> 5.) Suppose I can restore the BMC's original SPI content and behavior - what's
> a recommended way to have the TFPT'd kernel boot into an OpenBMC rootfs
> *without* having it store on the BMC's main storage/overwriting SPI?
If you're looking to deal with OpenBMC directly then this collection of
patches from Patrick will probably help:
https://gerrit.openbmc.org/q/topic:%22no-rootfs%22
Otherwise, using buildroot to create a self-contained kernel and
initramfs is a quick way to make progress.
>
> 6.) Assuming this cannot be recovered in software - what are my chances of
> identifying the SPI flash on my board as such, and re-writing its contents
> using an affordable SPI programmer solution, given that I've never done
> anything like this with hardware before? :^)
From the manual[2] I expect it's in the unmarked socket between the
PCIe x4 (27), M.2 (28) and PCIe x16 (29) slots.
[2]:
https://download.gigabyte.com/FileList/Manual/server_manual_MC12-LE0_e_v1.0.pdf?v=24c023a8081dd64c15be33bf2abf5220
Andrew
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-28 0:26 ` Andrew Jeffery
@ 2024-05-28 18:37 ` Johannes Truschnigg
2024-05-29 0:52 ` Andrew Jeffery
0 siblings, 1 reply; 11+ messages in thread
From: Johannes Truschnigg @ 2024-05-28 18:37 UTC (permalink / raw)
To: openbmc
[-- Attachment #1: Type: text/plain, Size: 21458 bytes --]
Hi again! :)
On Tue, May 28, 2024 at 09:56:38AM +0930, Andrew Jeffery wrote:
> [...]
> > ```
> > root@grml ~ # /tmp/culvert probe
> > [*] failed to initialise devmem bridge: -1
> > [*] Accessing the BMC's AHB via the p2a bridge
> > debug: Permissive
> > Debug UART port: UART5
> > xdma: Restricted
> > BMC: Disabled
> > VGA: Enabled
> > XDMA on VGA: Enabled
> > XDMA is constrained: Yes
> > p2a: Permissive
> > BMC: Disabled
> > VGA: Enabled
> > MMIO on VGA: Enabled
> > [0x00000000 - 0x0fffffff] Firmware: Writable
> > [0x10000000 - 0x1fffffff] SoC IO: Writable
> > [0x20000000 - 0x2fffffff] BMC Flash: Writable
> > [0x30000000 - 0x3fffffff] Host Flash: Writable
> > [0x40000000 - 0x5fffffff] Reserved: Writable
> > [0x60000000 - 0x7fffffff] LPC Host: Writable
> > [0x80000000 - 0xffffffff] DRAM: Writable
> > ilpc: Permissive
> > SuperIO address: 0x2e
> > ```
> >
> > (Before my mishap, whenever I tried to run that command, all but `ilpc` yielded
> > nothing, and even `ilpc` reported "Restricted" - none of which I know how to
> > interpret at all, by the way! :D)
>
> So I have a bit of an educated guess here:
>
> 1. For AST SoCs prior to the AST2600, disabling the hardware backdoors
> must be done in firmware
> 2. Your process above has corrupted u-boot
> 3. The corruption is such that u-boot fails prior to applying the
> backdoor mitigations (if that's where the mitigation was done to begin
> with - however it's where we do the mitigation in OpenBMC)
>
> As for the culvert 'probe' output and the
> Permissive/Restricted/Disabled states:
>
> - Permissive means there are no constraints on the bridge - culvert can
> read and write any AHB address
> - Restricted means some constraints apply to the bridge - either the
> address space is restricted (e.g. XDMA is constrained to the VGA
> reserved memory), or write access is disabled for some portion or all
> of the AHB address space (e.g. the P2A write filters for the listed
> regions)
> - Disabled means what it says on the tin, the BMC's AHB address space
> cannot be accessed via the bridge at hand, we cannot read or write.
Thanks for this summary and explanation. The "I know some of these words" is
strong in me still, esp. during the later paragraphs, but I think I have a
chance to get to understand more about how it all comes together eventually ;)
> > So I guess I have at least six questions now:
> >
> > 1.) What happened when the kernel called it quits, u-boot reloaded and decided
> > to format some of its flash?
>
> Not sure, but you'd do well to boot a kernel that doesn't try to mount
> partitions from the flash.
Understood. Is that always the case for OpenBMC kernel images with default
config?
> > 3.) Can I restore the original firmware/SPI content on my board by any means
> > from the now running host OS? If so, what way would you suggest I try first?
>
> > 4.) Does having `culvert` have this new level of access have any new advantages
> > or open possibilities that I should be aware of?
>
> So one of the motivations for culvert is to reflash the BMC over the
> AHB bridges reported in the probe output. This works regardless of the
> state of the BMC firmware, so long as it hasn't disabled the hardware
> backdoors. You can try it with e.g.
>
> ```
> # culvert -vv write firmware < $firmware_image
> ```
>
> (you may want to experiment with `culvert -vv read firmware` first).
>
> That said, experience in [1] suggests Gigabyte have introduced some
> gremlins that aren't accounted for by culvert, and that you might have
> more success with gigaflash.
>
> [1]:
> https://github.com/amboar/culvert/issues/51#issuecomment-2129043859
So I did play around with this after I'd read the above github issue's latest
additions (how nice it was to learn that someone actually could use what
little info I had produced so far already! :)), and I concluded I would want
to try to get `culvert` to not flash back stock fw, but the OpenBMC phosphor
image I had prepared a while ago.
And indeed, it can be done, and I did it. I am just nor exactly sure *how* or
*why* I could do it, but maybe you can figure that out with the information I
gathered.
First, I used `gigaflash` to dump BMC ROM to a file, and that worked somewhat
unreliably at first, to my surprise - the first `gigaflash_x64 -dump somefile`
attempt caused my host to hang, but it recovered to its original state (BMC
dead, live distro (grml amd64) booted OK) after a power cycle.
The subsequent attempts using a more up-to-date release of gigaflash worked,
and while fooling around with culvert and gigaflash (to check if gigaflash
always produced the same result and to find out if the switches that Gigabyte's
provided wrapper shellscript used to updaye fw actually made a difference), I
noticed that at some point, culvert suddenly could (correctly, bit-by-bit
identically!) dump the ROM, too!
For posterity, this is what it looked like when gigaflash successfully dumped
the image for me:
```
# Tools/gigaflash_x64 -dump test.img -cs 0 -2500
gigaflash v2.0.10
Failed to connect BMC, try to dump image!
--- Dump image from BMC...
Find ASPEED Device 1a03:2000 on 4:0.0
MMIO Virtual Address: a4e84000
Relocate IO Base: f000
Found ASPEED Device 1a03:2500 rev. 41
Static Memory Controller Information:
CS0 Flash Type is SPI
CS1 Flash Type is SPI
CS2 Flash Type is SPI
CS3 Flash Type is NOR
CS4 Flash Type is NOR
Boot CS is 0
Option Information:
CS: 0
Flash Type: SPI
[Warning] Don't AC OFF or Reboot System During BMC Firmware Update!!
[SOCFLASH] Flash ID : 20ba20
Find Flash Chip #1: Numonyx N25Q512
Backup Flash Chip O.K.
--- Dump image finished
--- Wait 90 secs for BMC ready...
```
I tried to arrive at a minimal reproducer on what I had to do after a power
cut to get `culvert read firmware` to work, and in the end, it seems to be
that one needs to run
`gigaflash -dump some_filename -2500`
... once, and *only then* culvert can read the data as well.
A session log I kept when it successfully dumped looked like this:
```
[*] Found 5 registered bridge drivers
[*] Trying bridge driver l2a
[*] Failed to initialise L2A bridge: -95
[*] Trying bridge driver ilpc
[*] Probing ilpc
[*] Probing 0x2e for SuperIO
[*] Unlocking SuperIO: 0
[*] Selecting SuperIO device 2 (SUART1): 0
[*] Found device 2 selected: 0
[*] Selecting SuperIO device 12 (SUART4): 0
[*] Found device 12 selected: 0
[*] Locking SuperIO
[*] Found SuperIO device at 0x2e
[*] Probing for SoC revision registers
[*] ahb_readl: 0x1e6e2004: 0xf7cffedc
[*] ahb_readl: 0x1e6e207c: 0x04030303
[*] Found revision 0x4030303
[*] Trying bridge driver devmem
[*] failed to initialise devmem bridge: -1
[*] Trying bridge driver debug-uart
[*] Unrecognised argument list for debug interface (0)
[*] Trying bridge driver p2a
[*] Probing p2a
[*] Probing for SoC revision registers
[*] ahb_readl: 0x1e6e2004: 0xf7cffedc
[*] ahb_readl: 0x1e6e207c: 0x04030303
[*] Found revision 0x4030303
[*] Accessing the BMC's AHB via the p2a bridge
[*] Probing for SoC revision registers
[*] ahb_readl: 0x1e6e2004: 0xf7cffedc
[*] ahb_readl: 0x1e6e207c: 0x04030303
[*] Found revision 0x4030303
[*] Selected devicetree for SoC 'aspeed,ast2500'
[*] Found 15 registered drivers
[*] Processing devicetree node at /aliases
[*] Processing devicetree node at /memory@80000000
[*] Processing devicetree node at /ahb
[*] Processing devicetree node at /ahb/sram@1e720000
[*] Processing devicetree node at /ahb/bus-controller@1e600000
[*] Bound trace driver to /ahb/bus-controller@1e600000
[*] Processing devicetree node at /ahb/apb
[*] Processing devicetree node at /ahb/apb/spi@1e620000
[*] Bound sfc driver to /ahb/apb/spi@1e620000
[*] Processing devicetree node at /ahb/apb/spi@1e630000
[*] Bound sfc driver to /ahb/apb/spi@1e630000
[*] Processing devicetree node at /ahb/apb/spi@1e631000
[*] Bound sfc driver to /ahb/apb/spi@1e631000
[*] Processing devicetree node at /ahb/apb/memory-controller@1e6e0000
[*] Bound sdmc driver to /ahb/apb/memory-controller@1e6e0000
[*] Processing devicetree node at /ahb/apb/syscon@1e6e2000
[*] Processing devicetree node at /ahb/apb/syscon@1e6e2000/clock
[*] Bound clk driver to /ahb/apb/syscon@1e6e2000/clock
[*] Processing devicetree node at /ahb/apb/syscon@1e6e2000/strapping
[*] Bound strap driver to /ahb/apb/syscon@1e6e2000/strapping
[*] Processing devicetree node at /ahb/apb/syscon@1e6e2000/superio
[*] Bound sioctl driver to /ahb/apb/syscon@1e6e2000/superio
[*] Processing devicetree node at /ahb/apb/syscon@1e6e2000/bridge-controller
[*] Bound bridge-controller driver to /ahb/apb/syscon@1e6e2000/bridge-controller
[*] Processing devicetree node at /ahb/apb/syscon@1e6e2000/debug-bridge-controller
[*] Bound debugctl driver to /ahb/apb/syscon@1e6e2000/debug-bridge-controller
[*] Processing devicetree node at /ahb/apb/syscon@1e6e2000/pcie-bridge-controller
[*] Bound pciectl driver to /ahb/apb/syscon@1e6e2000/pcie-bridge-controller
[*] Bound scu driver to /ahb/apb/syscon@1e6e2000
[*] Processing devicetree node at /ahb/apb/watchdog@1e785000
[*] Bound wdt driver to /ahb/apb/watchdog@1e785000
[*] Processing devicetree node at /ahb/apb/watchdog@1e785020
[*] Bound wdt driver to /ahb/apb/watchdog@1e785020
[*] Processing devicetree node at /ahb/apb/watchdog@1e785040
[*] Bound wdt driver to /ahb/apb/watchdog@1e785040
[*] Processing devicetree node at /ahb/apb/serial@1e787000
[*] Bound vuart driver to /ahb/apb/serial@1e787000
[*] Processing devicetree node at /ahb/apb/lpc@1e789000
[*] Processing devicetree node at /ahb/apb/lpc@1e789000/bridge-controller
[*] Bound ilpcctl driver to /ahb/apb/lpc@1e789000/bridge-controller
[*] Bound uart-mux driver to /ahb/apb/lpc@1e789000
[*] Initialising flash controller
[*] fdt: Looking up device name 'fmc'
[*] fdt: Locating node with device path '/ahb/apb/spi@1e620000'
[*] ahb_readl: 0x1e6e2000: 0x00000001
[*] Initialised scu driver
[*] Initialised clk driver
[*] ahb_readl: 0x1e6e2070: 0xf120f287
[*] ahb_readl: 0x1e620010: 0x00002400
[*] ahb_readl: 0x1e620000: 0x8007002a
[*] ahb_writel: 0x1e620000: 0x8007002a
[*] ahb_writel: 0x1e620010: 0x00000400
[*] ahb_writel: 0x1e620094: 0x00000000
[*] Initialised sfc driver
[*] Initialising flash chip
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000403
[*] ahb_readl: 0x20000000: 0x02020202
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000400
[*] LIBFLASH: Init status: 02
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000403
[*] ahb_readl: 0x20000000: 0x1020ba20
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000400
[*] LIBFLASH: Flash ID: 20.ba.20 (20ba20)
[*] LIBFLASH: Found chip Micron N25Qx512Ax size 64M erase granule: 4K
[*] LIBFLASH: Flash >16MB, enabling 4B mode...
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000403
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000400
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000403
[*] ahb_readl: 0x20000000: 0x02020202
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000400
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000403
[*] ahb_writel: 0x1e620010: 0x00000407
[*] ahb_writel: 0x1e620010: 0x00000400
[*] LIBFLASH: Enabling controller 4B mode...
[*] ahb_readl: 0x1e620004: 0x00000701
[*] ahb_writel: 0x1e620010: 0x00002400
[*] ahb_writel: 0x1e620004: 0x00000701
[*] Write-protecting all chip-selects
[*] ahb_readl: 0x1e620000: 0x8007002a
[*] ahb_writel: 0x1e620000: 0x8007002a
[*] Exfiltrating BMC flash to stdout
................................................................
[*] ahb_readl: 0x1e620000: 0x8007002a
[*] ahb_writel: 0x1e620000: 0x8007002a
[*] Unbound instance of driver uart-mux
[*] Unbound instance of driver ilpcctl
[*] Unbound instance of driver vuart
[*] Unbound instance of driver wdt
[*] Unbound instance of driver wdt
[*] Unbound instance of driver wdt
[*] Unbound instance of driver scu
[*] Unbound instance of driver pciectl
[*] Unbound instance of driver debugctl
[*] Unbound instance of driver bridge-controller
[*] Unbound instance of driver sioctl
[*] Unbound instance of driver strap
[*] Unbound instance of driver clk
[*] Unbound instance of driver sdmc
[*] Unbound instance of driver sfc
[*] Unbound instance of driver sfc
[*] ahb_writel: 0x1e620010: 0x00002400
[*] Unbound instance of driver sfc
[*] Unbound instance of driver trace
```
I did also try to get mmiotrace, which you mentioned in the GH issue (I have
never worked with it before, but I am familiar with ftrace, so I am not 100%
but reasonably certain that I did not hold it wrong), to work, but I could not
make it emit any tracing data while either culvert or gigaflash were dumping
ROM. Only when (un)loading the `ast` driver, lots of tracing data could be
collected.
I do have strace capture of `gigaflash` running for the first time after a
reboot, but all the juicy bits seem to hide behind mmap() anyway, so I will
only provide it upon request.
Then it dawned on me that `culvert probe` before and after the gigaflash
"unlock" might hold the key, and when diffing two runs' results (one before,
when culvert could not dump anything, and one after, when it very well could)
yielded this result:
```
$ diff -u1 culvert_probe_initial culvert_probe_after_dump
--- culvert_probe_initial
+++ culvert_probe_after_dump
@@ -121,4 +121,4 @@
[*] ahb_readl: 0x1e6e2070: 0xf120f286
-[*] ahb_readl: 0x1e789100: 0x00000000
-ilpc: Permissive
+[*] ahb_readl: 0x1e789100: 0x00000040
+ilpc: Restricted
[*] ahb_readl: 0x1e6e2070: 0xf120f286
```
I do not know how to interpret this, but here's goping you can tell if this
might help solve the mystery also reported in the GH issue? :)
> > 5.) Suppose I can restore the BMC's original SPI content and behavior - what's
> > a recommended way to have the TFPT'd kernel boot into an OpenBMC rootfs
> > *without* having it store on the BMC's main storage/overwriting SPI?
>
> If you're looking to deal with OpenBMC directly then this collection of
> patches from Patrick will probably help:
>
> https://gerrit.openbmc.org/q/topic:%22no-rootfs%22
Thanks a bunch, this seems like a very useful reading list for when I sorted my
immediate trouble with the board/BMC resulting of what I am about to detail
at the end of this mail! ;)
> > [...]
> > 6.) Assuming this cannot be recovered in software - what are my chances of
> > identifying the SPI flash on my board as such, and re-writing its contents
> > using an affordable SPI programmer solution, given that I've never done
> > anything like this with hardware before? :^)
>
> From the manual[2] I expect it's in the unmarked socket between the
> PCIe x4 (27), M.2 (28) and PCIe x16 (29) slots.
Thanks, I was afraid so - despite the diagram in the manual, the IC is not
socketed, so I guess I'll have to find someone who'd be able to desolder it for
me, or find a test clip that is compatible with that kind of "socket".
Anyhow, like I hinted at above, that's not the end of today's episode: I did
try if I could get culvert to *write* my OpenBMC flash in from my root prompt,
and it readily complied on first try (after I'd gotten it to dump first).
Unfortunately, I ran it with `-v -v` in effect, LOTS of debug output
overwhelmed my generous scrollback buffer, and I could only preserve the last
few thousands of lines it put out while writing and verifying flash. It looks a
lot like this:
```
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002400
.[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002403
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002400
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002403
[*] ahb_readl: 0x20000000: 0x02020202
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002400
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002403
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002400
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002403
[*] ahb_readl: 0x20000000: 0x03030303
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002400
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002403
[*] ahb_readl: 0x20000000: 0x00000000
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002400
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002403
[*] ahb_readl: 0x20000000: 0x81818181
[*] ahb_writel: 0x1e620010: 0x00002407
[*] ahb_writel: 0x1e620010: 0x00002400
.
[*] LIBFLASH: Verifying...
................................................................................................................................................................................................................................................................
[*] Performing SoC reset
[*] fdt: Looking up device name 'wdt2'
[*] fdt: Locating node with device path '/ahb/apb/watchdog@1e785020'
[*] ahb_readl: 0x1e78502c: 0x00000010
[*] wdt_readl: base: 0x1e785020, reg: 0x0c, val: 0x00000010
[*] wdt_writel: base: 0x1e785020, reg: 0x0c, val: 0x00000010
[*] ahb_writel: 0x1e78502c: 0x00000010
[*] ahb_readl: 0x1e78502c: 0x00000010
[*] wdt_readl: base: 0x1e785020, reg: 0x0c, val: 0x00000010
[*] wdt_writel: base: 0x1e785020, reg: 0x0c, val: 0x00000010
[*] ahb_writel: 0x1e78502c: 0x00000010
[*] wdt_writel: base: 0x1e785020, reg: 0x1c, val: 0x023ffffb
[*] ahb_writel: 0x1e78503c: 0x023ffffb
[*] ahb_readl: 0x1e78502c: 0x00000010
[*] wdt_readl: base: 0x1e785020, reg: 0x0c, val: 0x00000010
[*] wdt_writel: base: 0x1e785020, reg: 0x04, val: 0x004c4b40
[*] ahb_writel: 0x1e785024: 0x004c4b40
[*] wdt_writel: base: 0x1e785020, reg: 0x08, val: 0x00004755
[*] ahb_writel: 0x1e785028: 0x00004755
[*] ahb_readl: 0x1e78502c: 0x00000010
[*] wdt_readl: base: 0x1e785020, reg: 0x0c, val: 0x00000010
[*] wdt_writel: base: 0x1e785020, reg: 0x0c, val: 0x00000013
[*] ahb_writel: 0x1e78502c: 0x00000013
[*] Waiting 6000000 microseconds for watchdog timer to expire
[*] ahb_writel: 0x1e6e207c: 0x00000001
[*] wdt_writel: base: 0x1e785020, reg: 0x04, val: 0x00000000
[*] ahb_writel: 0x1e785024: 0x00000000
[*] Unbound instance of driver uart-mux
[*] Unbound instance of driver ilpcctl
[*] Unbound instance of driver vuart
[*] Unbound instance of driver wdt
[*] Unbound instance of driver wdt
[*] Unbound instance of driver wdt
[*] Unbound instance of driver scu
[*] Unbound instance of driver pciectl
[*] Unbound instance of driver debugctl
[*] Unbound instance of driver bridge-controller
[*] Unbound instance of driver sioctl
[*] Unbound instance of driver strap
[*] Unbound instance of driver clk
[*] Unbound instance of driver sdmc
[*] Unbound instance of driver sfc
[*] Unbound instance of driver sfc
[*] ahb_writel: 0x1e620010: 0x00002400
[*] Unbound instance of driver sfc
[*] Unbound instance of driver trace
/tmp/culvert -v -v write firmware < 190.48s user 69.45s system 97% cpu 4:25.94 total
```
The result is both very encouraging and also disappointing, because as you
intially theorized, the BMC boots fine with OpenBMC flashed onto it - but none
of its host system manmagement capabilities actually work. I do have the
vanilla OpenBMC web application available now, with DHCP, DNS, NTP et al.
working fine for the BMC, can log in via SSH, but all the peripherals the BMC
ought to be able to manage and hook into do not work at present.
The next step that I would want to take is that I find a way to revert to the
BMC stock fw with having only OpenBMC running, since the host apparently cannot
boot any more (same situation as with the stock BMC fw when u-boot had
initialized, but no BMC system was allowed to boot up, afaict - the power
button/contacts just do nothing) in this state. After that, I would like to
establish a sane (and hopefully easy) way to convert the board's BMC firmware
from OpenBMC to stock, and vice versa.
Once I have established a surefire and straightfoward way to do what I have
done in such meandering and clumsy attempts, I would like to learn more about
how the "M" is actually put into this whole "BMC" thing, and see how far I can
take that. The stock fw has some interesting description files regarding i2c
configs that might come in handy, but I am just not educated enough (yet, I
hope) to make real sense of it :)
Can you perhaps offer me advice on how to flash arbitrary new SPI flash
contents from either OpenBMC's u-boot or an OpenBMC root shell, or what I would
need to look at in detail to learn how to do that?
As always, I am very grateful for anyone's advice and time. Thank you! :)
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: https://johannes.truschnigg.info/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-28 18:37 ` Johannes Truschnigg
@ 2024-05-29 0:52 ` Andrew Jeffery
2024-05-30 15:37 ` Johannes Truschnigg
0 siblings, 1 reply; 11+ messages in thread
From: Andrew Jeffery @ 2024-05-29 0:52 UTC (permalink / raw)
To: Johannes Truschnigg, openbmc
On Tue, 2024-05-28 at 20:37 +0200, Johannes Truschnigg wrote:
> Hi again! :)
>
> On Tue, May 28, 2024 at 09:56:38AM +0930, Andrew Jeffery wrote:
>
> > > So I guess I have at least six questions now:
> > >
> > > 1.) What happened when the kernel called it quits, u-boot reloaded and decided
> > > to format some of its flash?
> >
> > Not sure, but you'd do well to boot a kernel that doesn't try to mount
> > partitions from the flash.
>
> Understood. Is that always the case for OpenBMC kernel images with default
> config?
Depends on what you mean by default. Which platform did you build for?
>
> The subsequent attempts using a more up-to-date release of gigaflash worked,
> and while fooling around with culvert and gigaflash (to check if gigaflash
> always produced the same result and to find out if the switches that Gigabyte's
> provided wrapper shellscript used to updaye fw actually made a difference), I
> noticed that at some point, culvert suddenly could (correctly, bit-by-bit
> identically!) dump the ROM, too!
Right, culvert is likely missing some trick with initialising the flash
controller. I'm trying to understand what that might be :)
>
> For posterity, this is what it looked like when gigaflash successfully dumped
> the image for me:
>
> ```
> # Tools/gigaflash_x64 -dump test.img -cs 0 -2500
> gigaflash v2.0.10
> Failed to connect BMC, try to dump image!
> --- Dump image from BMC...
> Find ASPEED Device 1a03:2000 on 4:0.0
> MMIO Virtual Address: a4e84000
> Relocate IO Base: f000
> Found ASPEED Device 1a03:2500 rev. 41
> Static Memory Controller Information:
> CS0 Flash Type is SPI
> CS1 Flash Type is SPI
> CS2 Flash Type is SPI
> CS3 Flash Type is NOR
> CS4 Flash Type is NOR
> Boot CS is 0
> Option Information:
> CS: 0
> Flash Type: SPI
> [Warning] Don't AC OFF or Reboot System During BMC Firmware Update!!
> [SOCFLASH] Flash ID : 20ba20
> Find Flash Chip #1: Numonyx N25Q512
> Backup Flash Chip O.K.
> --- Dump image finished
> --- Wait 90 secs for BMC ready...
> ```
>
>
> I tried to arrive at a minimal reproducer on what I had to do after a power
> cut to get `culvert read firmware` to work, and in the end, it seems to be
> that one needs to run
>
> `gigaflash -dump some_filename -2500`
>
> ... once, and *only then* culvert can read the data as well.
Okay, handy to know. That jives with my comment above.
>
> I did also try to get mmiotrace, which you mentioned in the GH issue (I have
> never worked with it before, but I am familiar with ftrace, so I am not 100%
> but reasonably certain that I did not hold it wrong), to work, but I could not
> make it emit any tracing data while either culvert or gigaflash were dumping
> ROM. Only when (un)loading the `ast` driver, lots of tracing data could be
> collected.
>
> I do have strace capture of `gigaflash` running for the first time after a
> reboot, but all the juicy bits seem to hide behind mmap() anyway, so I will
> only provide it upon request.
Well, consider this a request :D What it's mapping is of interest to
me, along with what it's opening more broadly, and any calls to
ioperm() or iopl().
>
> Then it dawned on me that `culvert probe` before and after the gigaflash
> "unlock" might hold the key, and when diffing two runs' results (one before,
> when culvert could not dump anything, and one after, when it very well could)
> yielded this result:
>
> ```
> $ diff -u1 culvert_probe_initial culvert_probe_after_dump
> --- culvert_probe_initial
> +++ culvert_probe_after_dump
> @@ -121,4 +121,4 @@
> [*] ahb_readl: 0x1e6e2070: 0xf120f286
> -[*] ahb_readl: 0x1e789100: 0x00000000
> -ilpc: Permissive
> +[*] ahb_readl: 0x1e789100: 0x00000040
> +ilpc: Restricted
> [*] ahb_readl: 0x1e6e2070: 0xf120f286
> ```
So something (it's unclear to me whether the BMC has rebooted in your
test here) has set the bit to disable writes via the iLPC2AHB bridge.
The reason I ask for the strace above is the mmiotrace output in the
github issue only shows a couple of accesses to the P2A. I'm wondering
whether gigaflash is exploiting iLPC2AHB and LPC FW/MEM cycles instead
(or perhaps I need to improve my understanding of the limits of
mmiotrace).
>
> The result is both very encouraging and also disappointing, because as you
> intially theorized, the BMC boots fine with OpenBMC flashed onto it - but none
> of its host system manmagement capabilities actually work. I do have the
> vanilla OpenBMC web application available now, with DHCP, DNS, NTP et al.
> working fine for the BMC, can log in via SSH, but all the peripherals the BMC
> ought to be able to manage and hook into do not work at present.
Yes, this is expected. Configuring the BMC applications to match the
integration into the board is the biggest challenge.
>
>
> The next step that I would want to take is that I find a way to revert to the
> BMC stock fw with having only OpenBMC running, since the host apparently cannot
> boot any more (same situation as with the stock BMC fw when u-boot had
> initialized, but no BMC system was allowed to boot up, afaict - the power
> button/contacts just do nothing) in this state. After that, I would like to
> establish a sane (and hopefully easy) way to convert the board's BMC firmware
> from OpenBMC to stock, and vice versa.
The least-effort hack would be to place the stock firmware at
/run/initramfs/image-bmc and reboot OpenBMC - this will write the stock
firmware image back to the flash for you.
>
> Once I have established a surefire and straightfoward way to do what I have
> done in such meandering and clumsy attempts, I would like to learn more about
> how the "M" is actually put into this whole "BMC" thing, and see how far I can
> take that. The stock fw has some interesting description files regarding i2c
> configs that might come in handy, but I am just not educated enough (yet, I
> hope) to make real sense of it :)
Yeah, the I2C configs will certainly help.
Breaking into the stock firmware on the hardware and tracing things
like GPIO accesses would go a long way. With the tools at hand it
shouldn't be too difficult :)
>
> Can you perhaps offer me advice on how to flash arbitrary new SPI flash
> contents from either OpenBMC's u-boot or an OpenBMC root shell, or what I would
> need to look at in detail to learn how to do that?
See the comment above regarding /run/initramfs/image-bmc. However you
can also boot to an initrd and use flashcp. The main thing is you need
to make sure no other accesses to the mtd device are taking place
(hence suggesting the initrd environment).
Andrew
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-29 0:52 ` Andrew Jeffery
@ 2024-05-30 15:37 ` Johannes Truschnigg
2024-05-31 1:06 ` Andrew Jeffery
0 siblings, 1 reply; 11+ messages in thread
From: Johannes Truschnigg @ 2024-05-30 15:37 UTC (permalink / raw)
To: openbmc
[-- Attachment #1: Type: text/plain, Size: 6133 bytes --]
Hi list!
First things first, I wanted to share the good news that I can now boot the
host with OpenBMC running the show on the BMC: Fiddling with GPIO #539 on the
OpenBMC root shell in the right way makes the host power up a few seconds
later. I do it like this for now:
```
echo 539 > /sys/class/gpio/export
echo out > /sys/class/gpio/gpio539/direction
echo 0 > /sys/class/gpio/gpio539/value
sleep 1
echo 1 > /sys/class/gpio/gpio539/value
```
I'd like to say thanks to my friend Paul on IRC here in public, because
without your patient guidance and steady flow of information and ideas, this
100% wouldn't have been possible!! :)
For posterity and completeness, this is what culvert has to say about the
BMC's state now that it runs OpenBMC:
```
root@grml ~ # /tmp/culvert -v -v probe
[*] Found 5 registered bridge drivers
[*] Trying bridge driver l2a
[*] Failed to initialise L2A bridge: -95
[*] Trying bridge driver ilpc
[*] Probing ilpc
[*] Probing 0x2e for SuperIO
[*] Unlocking SuperIO: 0
[*] Selecting SuperIO device 2 (SUART1): 0
[*] Found device 255 selected: 0
[*] Locking SuperIO
[*] Probing 0x4e for SuperIO
[*] Unlocking SuperIO: 0
[*] Selecting SuperIO device 2 (SUART1): 0
[*] Found device 255 selected: 0
[*] Locking SuperIO
[*] SuperIO disabled
[*] Trying bridge driver devmem
[*] failed to initialise devmem bridge: -1
[*] Trying bridge driver debug-uart
[*] Unrecognised argument list for debug interface (0)
[*] Trying bridge driver p2a
[*] Probing p2a
[*] Probing for SoC revision registers
[*] ahb_readl: 0x1e6e2004: 0xffffffff
[*] ahb_readl: 0x1e6e207c: 0xffffffff
[*] Found revision 0xffffffff
[*] Revision 0xffffffff is unsupported
[*] Failed P2A probe: -19
[*] Accessing the BMC's AHB via the ilpc bridge
[*] Probing for SoC revision registers
[*] ahb_readl: 0x1e6e2004: 0xffffffff
[*] ahb_readl: 0x1e6e207c: 0xffffffff
[*] Found revision 0xffffffff
[*] Revision 0xffffffff is unsupported
[*] Failed to probe SoC revision: -19
[*] Failed to probe SoC, exiting: -19
```
I haven't tried to execute gigaflash on the booted host yet, but it is on my
list of things to try next.
On Wed, May 29, 2024 at 10:22:09AM +0930, Andrew Jeffery wrote:
> On Tue, 2024-05-28 at 20:37 +0200, Johannes Truschnigg wrote:
> > [...]
> > Understood. Is that always the case for OpenBMC kernel images with default
> > config?
>
> Depends on what you mean by default. Which platform did you build for?
I built obmc-phosphor-image for evb-ast2500.
> > [...]
>
> Right, culvert is likely missing some trick with initialising the flash
> controller. I'm trying to understand what that might be :)
I was hoping you were going to say that! 8-)
> > [...]
> > I do have strace capture of `gigaflash` running for the first time after a
> > reboot, but all the juicy bits seem to hide behind mmap() anyway, so I will
> > only provide it upon request.
>
> Well, consider this a request :D What it's mapping is of interest to
> me, along with what it's opening more broadly, and any calls to
> ioperm() or iopl().
I've uploaded the zstd-compressed trace result (it contains a lot of the
actual ROM content dumped with how I called strace/gigaflash) to [0]. I really
hope it helps chasing down what you're looking for! :)
> > [...]
>
> The least-effort hack would be to place the stock firmware at
> /run/initramfs/image-bmc and reboot OpenBMC - this will write the stock
> firmware image back to the flash for you.
Thanks; this reads like a very easy way to revert to stock - I will play
around with OpenBMC on the physical thing some more before actually trying
that out, though!
> > Once I have established a surefire and straightfoward way to do what I have
> > done in such meandering and clumsy attempts, I would like to learn more about
> > how the "M" is actually put into this whole "BMC" thing, and see how far I can
> > take that. The stock fw has some interesting description files regarding i2c
> > configs that might come in handy, but I am just not educated enough (yet, I
> > hope) to make real sense of it :)
>
> Yeah, the I2C configs will certainly help.
>
> Breaking into the stock firmware on the hardware and tracing things
> like GPIO accesses would go a long way. With the tools at hand it
> shouldn't be too difficult :)
Yeah that would be cool, but I had established with Paul that the stock fw
kernel doesn't come with debugfs and missing userspace tooling for tracing
syscalls, so it will certainly be a challenge (at least for me, equipped with
my current skillset, and the time constraints of having a day job with some
other meatspace activity on top of that :D).
> > Can you perhaps offer me advice on how to flash arbitrary new SPI flash
> > contents from either OpenBMC's u-boot or an OpenBMC root shell, or what I would
> > need to look at in detail to learn how to do that?
>
> See the comment above regarding /run/initramfs/image-bmc. However you
> can also boot to an initrd and use flashcp. The main thing is you need
> to make sure no other accesses to the mtd device are taking place
> (hence suggesting the initrd environment).
Thanks again also for this! One of my problems is that I have no idea what
userland support utils/components even *exist* for that kind of embedded hw, so
I am very much in the dark what my options could be without such hints.
Meanwhile, due to a friendly user on a German web forum who led me to similar
matters discussed about an ASRock Rack-series motherboard (which also comes
with AST2500 for its BMC purposes), I think I have properly identified the
relevant IC where the BMC's firmware actually resides, and it looks like
SOIC-8 in direct vicinity to the AST2500 itself. I will have some new gear for
my electronics drawer delivered soon to see how/if I can deal with it! :)
[0]: https://johannes.truschnigg.info/tmp/strace_gigaflash_initial_.3305.zst
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: https://johannes.truschnigg.info/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-30 15:37 ` Johannes Truschnigg
@ 2024-05-31 1:06 ` Andrew Jeffery
0 siblings, 0 replies; 11+ messages in thread
From: Andrew Jeffery @ 2024-05-31 1:06 UTC (permalink / raw)
To: Johannes Truschnigg, openbmc; +Cc: Zev Weiss
On Thu, 2024-05-30 at 17:37 +0200, Johannes Truschnigg wrote:
> Hi list!
>
> First things first, I wanted to share the good news that I can now boot the
> host with OpenBMC running the show on the BMC: Fiddling with GPIO #539 on the
> OpenBMC root shell in the right way makes the host power up a few seconds
> later. I do it like this for now:
>
> ```
> echo 539 > /sys/class/gpio/export
> echo out > /sys/class/gpio/gpio539/direction
> echo 0 > /sys/class/gpio/gpio539/value
> sleep 1
> echo 1 > /sys/class/gpio/gpio539/value
> ```
>
> I'd like to say thanks to my friend Paul on IRC here in public, because
> without your patient guidance and steady flow of information and ideas, this
> 100% wouldn't have been possible!! :)
Nice work. For the record, the GPIO sysfs interface is deprecated and
likely to go away soon. Useful for PoCs, but longer term you should
look to the libgpiod APIs and tools[1].
[1]: https://git.kernel.org/pub/scm/libs/libgpiod/libgpiod.git/about/
>
>
> For posterity and completeness, this is what culvert has to say about the
> BMC's state now that it runs OpenBMC:
>
> ```
> root@grml ~ # /tmp/culvert -v -v probe
> [*] Found 5 registered bridge drivers
> [*] Trying bridge driver l2a
> [*] Failed to initialise L2A bridge: -95
> [*] Trying bridge driver ilpc
> [*] Probing ilpc
> [*] Probing 0x2e for SuperIO
> [*] Unlocking SuperIO: 0
> [*] Selecting SuperIO device 2 (SUART1): 0
> [*] Found device 255 selected: 0
> [*] Locking SuperIO
> [*] Probing 0x4e for SuperIO
> [*] Unlocking SuperIO: 0
> [*] Selecting SuperIO device 2 (SUART1): 0
> [*] Found device 255 selected: 0
> [*] Locking SuperIO
> [*] SuperIO disabled
> [*] Trying bridge driver devmem
> [*] failed to initialise devmem bridge: -1
> [*] Trying bridge driver debug-uart
> [*] Unrecognised argument list for debug interface (0)
> [*] Trying bridge driver p2a
> [*] Probing p2a
> [*] Probing for SoC revision registers
> [*] ahb_readl: 0x1e6e2004: 0xffffffff
> [*] ahb_readl: 0x1e6e207c: 0xffffffff
> [*] Found revision 0xffffffff
> [*] Revision 0xffffffff is unsupported
> [*] Failed P2A probe: -19
> [*] Accessing the BMC's AHB via the ilpc bridge
> [*] Probing for SoC revision registers
> [*] ahb_readl: 0x1e6e2004: 0xffffffff
> [*] ahb_readl: 0x1e6e207c: 0xffffffff
> [*] Found revision 0xffffffff
> [*] Revision 0xffffffff is unsupported
> [*] Failed to probe SoC revision: -19
> [*] Failed to probe SoC, exiting: -19
>
> ```
>
> I haven't tried to execute gigaflash on the booted host yet, but it is on my
> list of things to try next.
What you can see from the output above is that OpenBMC has disabled all
the hardware backdoors that culvert tries to exploit.
The consequence of this is that gigaflash will (... should?) now also
fail to function.
You may want to play with these u-boot options:
https://github.com/openbmc/u-boot/blob/v2019.04-aspeed-openbmc/arch/arm/mach-aspeed/Kconfig#L48-L85
Or we can add support to culvert for enabling and disabling the
backdoors at runtime, and run it with /dev/mem enabled on the BMC (set
mem.devmem=1 on the kernel commandline). Adding support to culvert
should just be a matter of wiring up some command-line UI - what bits
to flip and the necessary abstractions are already provided in the
implementation.
>
>
> > > [...]
> > > I do have strace capture of `gigaflash` running for the first time after a
> > > reboot, but all the juicy bits seem to hide behind mmap() anyway, so I will
> > > only provide it upon request.
> >
> > Well, consider this a request :D What it's mapping is of interest to
> > me, along with what it's opening more broadly, and any calls to
> > ioperm() or iopl().
>
> I've uploaded the zstd-compressed trace result (it contains a lot of the
> actual ROM content dumped with how I called strace/gigaflash) to [0]. I really
> hope it helps chasing down what you're looking for! :)
Given some of the discussion on the culvert issue, it may not, but I'll
take a look. And indeed they're using the P2A, so we'll need to
consider another approach to tracing the accesses.
>
>
> > > Once I have established a surefire and straightfoward way to do what I have
> > > done in such meandering and clumsy attempts, I would like to learn more about
> > > how the "M" is actually put into this whole "BMC" thing, and see how far I can
> > > take that. The stock fw has some interesting description files regarding i2c
> > > configs that might come in handy, but I am just not educated enough (yet, I
> > > hope) to make real sense of it :)
> >
> > Yeah, the I2C configs will certainly help.
> >
> > Breaking into the stock firmware on the hardware and tracing things
> > like GPIO accesses would go a long way. With the tools at hand it
> > shouldn't be too difficult :)
>
> Yeah that would be cool, but I had established with Paul that the stock fw
> kernel doesn't come with debugfs and missing userspace tooling for tracing
> syscalls, so it will certainly be a challenge (at least for me, equipped with
> my current skillset, and the time constraints of having a day job with some
> other meatspace activity on top of that :D).
If you can coax it past u-boot under qemu you can trace the hardware
accesses via the qemu models (without requiring support from the OS).
> Meanwhile, due to a friendly user on a German web forum who led me to similar
> matters discussed about an ASRock Rack-series motherboard (which also comes
> with AST2500 for its BMC purposes), I think I have properly identified the
> relevant IC where the BMC's firmware actually resides, and it looks like
> SOIC-8 in direct vicinity to the AST2500 itself. I will have some new gear for
> my electronics drawer delivered soon to see how/if I can deal with it! :)
For the record, Zev (who also helps out with culvert) is the OpenBMC
maintainer for the ASRock Rack platforms, so should be able to assist
you there as well.
Andrew
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions
2024-05-20 17:32 OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions Johannes Truschnigg
2024-05-21 1:05 ` Andrew Jeffery
@ 2024-07-07 9:19 ` Johannes Truschnigg
1 sibling, 0 replies; 11+ messages in thread
From: Johannes Truschnigg @ 2024-07-07 9:19 UTC (permalink / raw)
To: openbmc
[-- Attachment #1: Type: text/plain, Size: 5325 bytes --]
Hey list!
Sorry for my long radio silence -- I've been busy (with other things, but also)
quietly working at this porting project of mine, and made some imho rather cool
progress. I have also, however, hit a few roadblocks I cannot seem to surmount
on my own, and so I turn to you, dear reader, for advice and/or consolation.
I have my own "meta-gigabyte" layer including a board-specific DTS for u-boot
and linux-aspeed, with results of those that seem mostly OK. The
ultra-barebones bmcweb function is there, and I have iKVM working.
My plan was to get host power control and the ID button/LED working as a next
objective - that way, the most important features from my POV would be there,
and I would feel OK-ish with publishing what I got. For the past few weeks, I
had been living under the impression that poking GPIO 27 the right way did just
that (power up the host), and so I went on to reverse engineer the ID
button/LED stuff with `gpiomon`. I figured out a number of functions this way:
The Front Panel Header (FPH) Power Button corresponds to GPIO 24.
The FPH Reset Button to GPIO 26.
The FPH ID Button (as well as the board-mounted ID button) to GPIO 134.
HDD LED activity seems to be wired to GPIO 111.
The ID LED on the board itself is controlled via GPIO offset 49.
Having poked at all these GPIOs somehow made my GPIO 27 interaction to boot up
the host ineffective, however. That change persisted and survived even a cycle
of going back to the stock firmware, and then flashing OpenBMC again. I have no
idea what went wrong there, but I might have since managed to recover/revise
the procedure, and can get the host to power again as a consequence: After a
*ton* of experimentation, I think I also have figure out the proper minimal
sequence of gpioset and gpioget invocations to make the host boot when the BMC
is online; it seems to be this magic spell:
# gpioget 0 25; gpioset -m time -D open-source -s 1 0 27=1; \
gpioset 0 25=0; gpioget 0 25
(I cannot help but notice that these pins are adjacent to the RESET and POWER
buttons!)
What I am struggling with now is how I should best make x86-power-control or a
functional equivalent benefit from this knowledge, too, and how I can make it
manage host power state in my stead. I was hoping that someone on this list
would be both knowledgeable and kind enough to give me advice on how to
proceed. The OpenBMC tree has a number of other machines where a shellscript
that's a machine-specific part of "phosphor-state-manager" does the required
`gpioget`/`gpioget` dance and then uses `busctl` to let the rest of the system
know how things are (supposed to be) after that - is that what I should be
purusing, too? Or would it just be a matter of properly naming (*is* there a
canonical (sub)set of GPIO names?) the numeric IDs in the DTS, and I could
expect things to magically fall into place on their own? If there's some
implementation I should be looking at in particular, I'd be happy to do and
then try mimic that.
Also, since right now the FPH pins/buttons don't do anything apart from
registering in gpiomon, I was wondering if it's normal to be the BMC's job to
wait for these event firing, and then doing the proper GPIO magic to perform
what the user asked for pressing the respective button. I guess that is what
has to happen, right? If so, can anyone maybe tell me where to look at for an
example of how this is set up for an existing, working machine?
Furthermore, I am not sure which other operations I'll still have to figure out
GPIO sequences for. Graceful shutdown (I expect this to have to generate an
ACPI Power Button event in the host, or is there some other magic involved?)
probably, "forced" shutdown/cutting power - but what else? And does anyone have
any concrete advice on how to methodically tackle that (find out which GPIO
needs which treatment to generate which event)?
What I also wanted to learn is how I should best integrate my custom .dts and
resulting .dtb in my machine-specific layer(s), so that both the kernel and
u-boot "pick it up" correctly. The kernel part was relatively easy, since the
build infra seems to automatically include any additional .dts named files
placed in the proper location and be able to use that via setting
'KERNEL_DEVICETREE = "gigabyte-mc12-le0.dtb"' (in my instance), but u-boot does
not, and I ended up overwriting arch/arm/dts/ast2500-evb.dts with my own in a
`do_patch:append() {}` block and setting 'UBOOT_DEVICETREE = "ast2500-evb"' in
my machine config.
I guess the *intended*/preferred way is to upstream the (correct and sufficient
- which I do not have yet) DTS artifacts into both kernel and u-boot, and not
to have this problem in the first place? Assuming that's not an option for now,
how should I *properly* integrate my local modification (actually: addition) so
that OpenBMC upstream would be in a position to accept the "meta-gigabyte"
layer including it with good conscience? (That is my eventual goal :))
Thanks a lot for reading this far, and for all the support provided to me off
and on list to get this closer to actually working! :)
--
with best regards:
- Johannes Truschnigg ( johannes@truschnigg.info )
www: https://johannes.truschnigg.info/
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-07-08 23:21 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-20 17:32 OpenBMC replacing AMI AST2500 BMC fw on Gigabyte MC12-LE0 - questions Johannes Truschnigg
2024-05-21 1:05 ` Andrew Jeffery
2024-05-21 17:05 ` Johannes Truschnigg
2024-05-22 1:29 ` Andrew Jeffery
2024-05-27 18:09 ` Johannes Truschnigg
2024-05-28 0:26 ` Andrew Jeffery
2024-05-28 18:37 ` Johannes Truschnigg
2024-05-29 0:52 ` Andrew Jeffery
2024-05-30 15:37 ` Johannes Truschnigg
2024-05-31 1:06 ` Andrew Jeffery
2024-07-07 9:19 ` Johannes Truschnigg
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.