Gitlab runner timeout on evb-ast2600 ?

Gitlab runner timeout on evb-ast2600 ?

[Eugen Hristev]

Hi everyone,

I keep getting an error on running the CI loop on -next on my tree :

https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517

! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
tests will run !

Any clue ? It's been like this for few days. It doesn't look like a
faulty patch.

Thanks,
Eugen

Re: Gitlab runner timeout on evb-ast2600 ?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 671 bytes --]

On Mon, Apr 07, 2025 at 04:55:16PM +0300, Eugen Hristev wrote:

> Hi everyone,
> 
> I keep getting an error on running the CI loop on -next on my tree :
> 
> https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
> 
> ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> tests will run !
> 
> Any clue ? It's been like this for few days. It doesn't look like a
> faulty patch.

Try adding:
TEST_PY_EXTRA: "--capture=tee-sys"

To the stanza for the board in .gitlab-ci.yml and see if there's more
details in the output. The failure means that QEMU didn't run or we
crashed on start or something I believe.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Gitlab runner timeout on evb-ast2600 ?

[Joel Stanley]

On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
>
> Hi everyone,
>
> I keep getting an error on running the CI loop on -next on my tree :
>
> https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517

Thanks for the report.

>
> ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> tests will run !
>
> Any clue ? It's been like this for few days. It doesn't look like a
> faulty patch.

I can't see that link you sent me, it 403s.

Do you have any other traces or information to share?

When did this start failing?

Cheers,

Joel

Re: Gitlab runner timeout on evb-ast2600 ?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
> On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
> >
> > Hi everyone,
> >
> > I keep getting an error on running the CI loop on -next on my tree :
> >
> > https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
> 
> Thanks for the report.
> 
> >
> > ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> > tests will run !
> >
> > Any clue ? It's been like this for few days. It doesn't look like a
> > faulty patch.
> 
> I can't see that link you sent me, it 403s.
> 
> Do you have any other traces or information to share?

It's been intermittent recently. I was able to put qemu + gdb together
and see it's one of the early initcalls that fails and hang()s but
didn't see why. You should be able to use commit 1f0281294d8b ("Merge
patch series "Annotate switch/case fallthrough cases"") as a starting
point of failure.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Gitlab runner timeout on evb-ast2600 ?

[Cédric Le Goater]

Hi,

On 4/11/25 01:14, Tom Rini wrote:
> On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
>> On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
>>>
>>> Hi everyone,
>>>
>>> I keep getting an error on running the CI loop on -next on my tree :
>>>
>>> https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
>>
>> Thanks for the report.
>>
>>>
>>> ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
>>> tests will run !
>>>
>>> Any clue ? It's been like this for few days. It doesn't look like a
>>> faulty patch.
>>
>> I can't see that link you sent me, it 403s.
>>
>> Do you have any other traces or information to share?
> 
> It's been intermittent recently. I was able to put qemu + gdb together
> and see it's one of the early initcalls that fails and hang()s but
> didn't see why. You should be able to use commit 1f0281294d8b ("Merge
> patch series "Annotate switch/case fallthrough cases"") as a starting
> point of failure.
> 

Do you have logs ?

Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :
     
     uboot_build=/path/to/u-boot/
     flashsize=65536
     flash=./flash-ast2600-evb-uboot
     dd of=$flash bs=1K if=/dev/zero count=$flashsize
     dd of=$flash conv=notrunc bs=1K if=$uboot_build/spl/u-boot-spl.bin
     dd of=$flash conv=notrunc bs=1K if=$uboot_build/u-boot.img seek=64
     
     $ qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
     qemu-system-arm: warning: nic ftgmac100.1 has no peer
     qemu-system-arm: warning: nic ftgmac100.2 has no peer
     qemu-system-arm: warning: nic ftgmac100.3 has no peer
     
     U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
     Trying to boot from RAM
     ## Checking hash(es) for config conf-1 ... OK
     ## Checking hash(es) for Image firmware-1 ... crc32+ OK
     ## Checking hash(es) for Image fdt-1 ... crc32+ OK
     
     
     U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
     
     Model: Aspeed BMC
     DRAM:  already initialized, 1008 MiB (capacity:1024 MiB, VGA:64 MiB), ECC off
     Core:  266 devices, 24 uclasses, devicetree: separate
     WDT:   Started watchdog@1e785000 with servicing every 1000ms (60s timeout)
     MMC:   sdhci@1e750100: 0
     Loading Environment from SPIFlash... SF: Detected mx25u51245g with page size 256 Bytes, erase size 64 KiB, total 64 MiB
     *** Warning - bad CRC, using default environment
     
     In:    serial@1e784000
     Out:   serial@1e784000
     Err:   serial@1e784000
     Net:   eth0: ftgmac@1e660000, eth1: ftgmac@1e680000, eth2: ftgmac@1e670000, eth3: ftgmac@1e690000
     Hit any key to stop autoboot:  0
     libfdt fdt_check_header(): FDT_ERR_BADMAGIC
     Error loading kernel FIT image
     =>
     

Thanks,

C.

Re: Gitlab runner timeout on evb-ast2600 ?

[Joel Stanley]

On Fri, 11 Apr 2025 at 16:54, Cédric Le Goater <clg@kaod.org> wrote:
>
> Hi,
>
> On 4/11/25 01:14, Tom Rini wrote:
> > On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
> >> On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> I keep getting an error on running the CI loop on -next on my tree :
> >>>
> >>> https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
> >>
> >> Thanks for the report.
> >>
> >>>
> >>> ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> >>> tests will run !
> >>>
> >>> Any clue ? It's been like this for few days. It doesn't look like a
> >>> faulty patch.
> >>
> >> I can't see that link you sent me, it 403s.
> >>
> >> Do you have any other traces or information to share?
> >
> > It's been intermittent recently. I was able to put qemu + gdb together
> > and see it's one of the early initcalls that fails and hang()s but
> > didn't see why. You should be able to use commit 1f0281294d8b ("Merge
> > patch series "Annotate switch/case fallthrough cases"") as a starting
> > point of failure.
> >
>
> Do you have logs ?
>
> Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :

I brefily tried to run the steps from CI locally, but couldn't work
out how to get the flash image built. Instead I used the instructions
from 154cffa16a7b6647e180ed6cef55342b39580aff.

Testing master with Debian's qemu 9.2.2:

makearm evb-ast2600_defconfig
make u-boot-with-spl.bin
truncate -s 64M u-boot-with-spl.bin
qemu-system-arm -nographic -M ast2600-evb \
        -drive file=u-boot-with-spl.bin,if=mtd,format=raw


U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 17:05:20 +0930)
Trying to boot from RAM
## Checking hash(es) for config conf-1 ... OK
## Checking hash(es) for Image firmware-1 ... crc32+ OK
## Checking hash(es) for Image fdt-1 ... crc32+ OK


U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 17:05:20 +0930)

Model: Aspeed BMC
DRAM:  already initialized, 1008 MiB (capacity:1024 MiB, VGA:64 MiB), ECC off
Core:  266 devices, 24 uclasses, devicetree: separate
WDT:   Started watchdog@1e785000 with servicing every 1000ms (60s timeout)
MMC:   sdhci@1e750100: 0
Loading Environment from SPIFlash... SF: Detected mx25u51245g with
page size 256 Bytes, erase size 64 KiB, total 64 MiB
*** Warning - bad CRC, using default environment

In:    serial@1e784000
Out:   serial@1e784000
Err:   serial@1e784000
Net:   eth0: ftgmac@1e660000, eth1: ftgmac@1e680000, eth2:
ftgmac@1e670000, eth3: ftgmac@1e690000
Hit any key to stop autoboot:  0
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error loading kernel FIT image
=>

I tried to run qemu using the recipie from CI but I couldn't see where
the step to create the flash image happens.

>
>      uboot_build=/path/to/u-boot/
>      flashsize=65536
>      flash=./flash-ast2600-evb-uboot
>      dd of=$flash bs=1K if=/dev/zero count=$flashsize
>      dd of=$flash conv=notrunc bs=1K if=$uboot_build/spl/u-boot-spl.bin
>      dd of=$flash conv=notrunc bs=1K if=$uboot_build/u-boot.img seek=64
>
>      $ qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
>      qemu-system-arm: warning: nic ftgmac100.1 has no peer
>      qemu-system-arm: warning: nic ftgmac100.2 has no peer
>      qemu-system-arm: warning: nic ftgmac100.3 has no peer
>
>      U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
>      Trying to boot from RAM
>      ## Checking hash(es) for config conf-1 ... OK
>      ## Checking hash(es) for Image firmware-1 ... crc32+ OK
>      ## Checking hash(es) for Image fdt-1 ... crc32+ OK
>
>
>      U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
>
>      Model: Aspeed BMC
>      DRAM:  already initialized, 1008 MiB (capacity:1024 MiB, VGA:64 MiB), ECC off
>      Core:  266 devices, 24 uclasses, devicetree: separate
>      WDT:   Started watchdog@1e785000 with servicing every 1000ms (60s timeout)
>      MMC:   sdhci@1e750100: 0
>      Loading Environment from SPIFlash... SF: Detected mx25u51245g with page size 256 Bytes, erase size 64 KiB, total 64 MiB
>      *** Warning - bad CRC, using default environment
>
>      In:    serial@1e784000
>      Out:   serial@1e784000
>      Err:   serial@1e784000
>      Net:   eth0: ftgmac@1e660000, eth1: ftgmac@1e680000, eth2: ftgmac@1e670000, eth3: ftgmac@1e690000
>      Hit any key to stop autoboot:  0
>      libfdt fdt_check_header(): FDT_ERR_BADMAGIC
>      Error loading kernel FIT image
>      =>
>
>
> Thanks,
>
> C.
>
>

Re: Gitlab runner timeout on evb-ast2600 ?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2705 bytes --]

On Fri, Apr 11, 2025 at 05:26:49PM +0930, Joel Stanley wrote:
> On Fri, 11 Apr 2025 at 16:54, Cédric Le Goater <clg@kaod.org> wrote:
> >
> > Hi,
> >
> > On 4/11/25 01:14, Tom Rini wrote:
> > > On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
> > >> On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
> > >>>
> > >>> Hi everyone,
> > >>>
> > >>> I keep getting an error on running the CI loop on -next on my tree :
> > >>>
> > >>> https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
> > >>
> > >> Thanks for the report.
> > >>
> > >>>
> > >>> ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> > >>> tests will run !
> > >>>
> > >>> Any clue ? It's been like this for few days. It doesn't look like a
> > >>> faulty patch.
> > >>
> > >> I can't see that link you sent me, it 403s.
> > >>
> > >> Do you have any other traces or information to share?
> > >
> > > It's been intermittent recently. I was able to put qemu + gdb together
> > > and see it's one of the early initcalls that fails and hang()s but
> > > didn't see why. You should be able to use commit 1f0281294d8b ("Merge
> > > patch series "Annotate switch/case fallthrough cases"") as a starting
> > > point of failure.
> > >
> >
> > Do you have logs ?
> >
> > Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :
> 
> I brefily tried to run the steps from CI locally, but couldn't work
> out how to get the flash image built. Instead I used the instructions
> from 154cffa16a7b6647e180ed6cef55342b39580aff.

It's a bit of abstractions, sorry. It's this:
https://source.denx.de/u-boot/u-boot-test-hooks/-/blob/master/bin/flash.qemu_gen_padded_image?ref_type=heads
with variables filled from:
https://source.denx.de/u-boot/u-boot-test-hooks/-/blob/master/bin/travis-ci/conf.evb-ast2600_qemu?ref_type=heads

> Testing master with Debian's qemu 9.2.2:
> 
> makearm evb-ast2600_defconfig
> make u-boot-with-spl.bin
> truncate -s 64M u-boot-with-spl.bin
> qemu-system-arm -nographic -M ast2600-evb \
>         -drive file=u-boot-with-spl.bin,if=mtd,format=raw
> 
> 
> U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 17:05:20 +0930)
> Trying to boot from RAM
> ## Checking hash(es) for config conf-1 ... OK
> ## Checking hash(es) for Image firmware-1 ... crc32+ OK
> ## Checking hash(es) for Image fdt-1 ... crc32+ OK
> 
> 
> U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 17:05:20 +0930)

Just please note I don't know if this rev of U-Boot was a failing one,
or not. That's part of the challenge here is seemingly arbitrary changes
lead to this platform passing/failing.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Gitlab runner timeout on evb-ast2600 ?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2636 bytes --]

On Fri, Apr 11, 2025 at 09:23:59AM +0200, Cédric Le Goater wrote:
> Hi,
> 
> On 4/11/25 01:14, Tom Rini wrote:
> > On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
> > > On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
> > > > 
> > > > Hi everyone,
> > > > 
> > > > I keep getting an error on running the CI loop on -next on my tree :
> > > > 
> > > > https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
> > > 
> > > Thanks for the report.
> > > 
> > > > 
> > > > ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> > > > tests will run !
> > > > 
> > > > Any clue ? It's been like this for few days. It doesn't look like a
> > > > faulty patch.
> > > 
> > > I can't see that link you sent me, it 403s.
> > > 
> > > Do you have any other traces or information to share?
> > 
> > It's been intermittent recently. I was able to put qemu + gdb together
> > and see it's one of the early initcalls that fails and hang()s but
> > didn't see why. You should be able to use commit 1f0281294d8b ("Merge
> > patch series "Annotate switch/case fallthrough cases"") as a starting
> > point of failure.
> > 
> 
> Do you have logs ?
> 
> Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :
>     uboot_build=/path/to/u-boot/
>     flashsize=65536
>     flash=./flash-ast2600-evb-uboot
>     dd of=$flash bs=1K if=/dev/zero count=$flashsize
>     dd of=$flash conv=notrunc bs=1K if=$uboot_build/spl/u-boot-spl.bin
>     dd of=$flash conv=notrunc bs=1K if=$uboot_build/u-boot.img seek=64
>     $ qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
>     qemu-system-arm: warning: nic ftgmac100.1 has no peer
>     qemu-system-arm: warning: nic ftgmac100.2 has no peer
>     qemu-system-arm: warning: nic ftgmac100.3 has no peer
>     U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
>     Trying to boot from RAM
>     ## Checking hash(es) for config conf-1 ... OK
>     ## Checking hash(es) for Image firmware-1 ... crc32+ OK
>     ## Checking hash(es) for Image fdt-1 ... crc32+ OK
>     U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)

Please note that it's inconsistent about what commit fails. I would
expect 1f0281294d8b in U-Boot to fail but I don't know about
cb7555e93075.

Also can you please check QEMU v8.2.0? That's what we're on, and I can
try and test 9.1.3 or 10.0.0-rc and see if it's a QEMU problem instead.
Thanks!

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Gitlab runner timeout on evb-ast2600 ?

[Cédric Le Goater]

On 4/11/25 17:01, Tom Rini wrote:
> On Fri, Apr 11, 2025 at 09:23:59AM +0200, Cédric Le Goater wrote:
>> Hi,
>>
>> On 4/11/25 01:14, Tom Rini wrote:
>>> On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
>>>> On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> I keep getting an error on running the CI loop on -next on my tree :
>>>>>
>>>>> https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
>>>>
>>>> Thanks for the report.
>>>>
>>>>>
>>>>> ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
>>>>> tests will run !
>>>>>
>>>>> Any clue ? It's been like this for few days. It doesn't look like a
>>>>> faulty patch.
>>>>
>>>> I can't see that link you sent me, it 403s.
>>>>
>>>> Do you have any other traces or information to share?
>>>
>>> It's been intermittent recently. I was able to put qemu + gdb together
>>> and see it's one of the early initcalls that fails and hang()s but
>>> didn't see why. You should be able to use commit 1f0281294d8b ("Merge
>>> patch series "Annotate switch/case fallthrough cases"") as a starting
>>> point of failure.
>>>
>>
>> Do you have logs ?
>>
>> Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :
>>      uboot_build=/path/to/u-boot/
>>      flashsize=65536
>>      flash=./flash-ast2600-evb-uboot
>>      dd of=$flash bs=1K if=/dev/zero count=$flashsize
>>      dd of=$flash conv=notrunc bs=1K if=$uboot_build/spl/u-boot-spl.bin
>>      dd of=$flash conv=notrunc bs=1K if=$uboot_build/u-boot.img seek=64
>>      $ qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
>>      qemu-system-arm: warning: nic ftgmac100.1 has no peer
>>      qemu-system-arm: warning: nic ftgmac100.2 has no peer
>>      qemu-system-arm: warning: nic ftgmac100.3 has no peer
>>      U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
>>      Trying to boot from RAM
>>      ## Checking hash(es) for config conf-1 ... OK
>>      ## Checking hash(es) for Image firmware-1 ... crc32+ OK
>>      ## Checking hash(es) for Image fdt-1 ... crc32+ OK
>>      U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
> 
> Please note that it's inconsistent about what commit fails. I would
> expect 1f0281294d8b in U-Boot to fail but I don't know about
> cb7555e93075.
> 
> Also can you please check QEMU v8.2.0? That's what we're on, and I can
> try and test 9.1.3 or 10.0.0-rc and see if it's a QEMU problem instead.
> Thanks!

Same result :

qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
qemu-system-arm: warning: nic ftgmac100.1 has no peer
qemu-system-arm: warning: nic ftgmac100.2 has no peer
qemu-system-arm: warning: nic ftgmac100.3 has no peer

U-Boot SPL 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)
Trying to boot from RAM
## Checking hash(es) for config conf-1 ... OK
## Checking hash(es) for Image firmware-1 ... crc32+ OK
## Checking hash(es) for Image fdt-1 ... crc32+ OK


U-Boot 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)

Model: Aspeed BMC
DRAM:  already initialized, 1008 MiB (capacity:1024 MiB, VGA:64 MiB), ECC off
Core:  266 devices, 24 uclasses, devicetree: separate
WDT:   Started watchdog@1e785000 with servicing every 1000ms (60s timeout)
MMC:   sdhci@1e750100: 0
Loading Environment from SPIFlash... SF: Detected mx25u51245g with page size 256 Bytes, erase size 64 KiB, total 64 MiB
*** Warning - bad CRC, using default environment

In:    serial@1e784000
Out:   serial@1e784000
Err:   serial@1e784000
Net:   eth0: ftgmac@1e660000, eth1: ftgmac@1e680000, eth2: ftgmac@1e670000, eth3: ftgmac@1e690000
Hit any key to stop autoboot:  0
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Error loading kernel FIT image
=> QEMU 8.2.10 monitor - type 'help' for more information
(qemu) q


What about the compiler ?


Thanks,

C.

Re: Gitlab runner timeout on evb-ast2600 ?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 4683 bytes --]

On Fri, Apr 11, 2025 at 05:26:55PM +0200, Cédric Le Goater wrote:
> On 4/11/25 17:01, Tom Rini wrote:
> > On Fri, Apr 11, 2025 at 09:23:59AM +0200, Cédric Le Goater wrote:
> > > Hi,
> > > 
> > > On 4/11/25 01:14, Tom Rini wrote:
> > > > On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
> > > > > On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
> > > > > > 
> > > > > > Hi everyone,
> > > > > > 
> > > > > > I keep getting an error on running the CI loop on -next on my tree :
> > > > > > 
> > > > > > https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
> > > > > 
> > > > > Thanks for the report.
> > > > > 
> > > > > > 
> > > > > > ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> > > > > > tests will run !
> > > > > > 
> > > > > > Any clue ? It's been like this for few days. It doesn't look like a
> > > > > > faulty patch.
> > > > > 
> > > > > I can't see that link you sent me, it 403s.
> > > > > 
> > > > > Do you have any other traces or information to share?
> > > > 
> > > > It's been intermittent recently. I was able to put qemu + gdb together
> > > > and see it's one of the early initcalls that fails and hang()s but
> > > > didn't see why. You should be able to use commit 1f0281294d8b ("Merge
> > > > patch series "Annotate switch/case fallthrough cases"") as a starting
> > > > point of failure.
> > > > 
> > > 
> > > Do you have logs ?
> > > 
> > > Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :
> > >      uboot_build=/path/to/u-boot/
> > >      flashsize=65536
> > >      flash=./flash-ast2600-evb-uboot
> > >      dd of=$flash bs=1K if=/dev/zero count=$flashsize
> > >      dd of=$flash conv=notrunc bs=1K if=$uboot_build/spl/u-boot-spl.bin
> > >      dd of=$flash conv=notrunc bs=1K if=$uboot_build/u-boot.img seek=64
> > >      $ qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
> > >      qemu-system-arm: warning: nic ftgmac100.1 has no peer
> > >      qemu-system-arm: warning: nic ftgmac100.2 has no peer
> > >      qemu-system-arm: warning: nic ftgmac100.3 has no peer
> > >      U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
> > >      Trying to boot from RAM
> > >      ## Checking hash(es) for config conf-1 ... OK
> > >      ## Checking hash(es) for Image firmware-1 ... crc32+ OK
> > >      ## Checking hash(es) for Image fdt-1 ... crc32+ OK
> > >      U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
> > 
> > Please note that it's inconsistent about what commit fails. I would
> > expect 1f0281294d8b in U-Boot to fail but I don't know about
> > cb7555e93075.
> > 
> > Also can you please check QEMU v8.2.0? That's what we're on, and I can
> > try and test 9.1.3 or 10.0.0-rc and see if it's a QEMU problem instead.
> > Thanks!
> 
> Same result :
> 
> qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
> qemu-system-arm: warning: nic ftgmac100.1 has no peer
> qemu-system-arm: warning: nic ftgmac100.2 has no peer
> qemu-system-arm: warning: nic ftgmac100.3 has no peer
> 
> U-Boot SPL 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)
> Trying to boot from RAM
> ## Checking hash(es) for config conf-1 ... OK
> ## Checking hash(es) for Image firmware-1 ... crc32+ OK
> ## Checking hash(es) for Image fdt-1 ... crc32+ OK
> 
> 
> U-Boot 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)
> 
> Model: Aspeed BMC
> DRAM:  already initialized, 1008 MiB (capacity:1024 MiB, VGA:64 MiB), ECC off
> Core:  266 devices, 24 uclasses, devicetree: separate
> WDT:   Started watchdog@1e785000 with servicing every 1000ms (60s timeout)
> MMC:   sdhci@1e750100: 0
> Loading Environment from SPIFlash... SF: Detected mx25u51245g with page size 256 Bytes, erase size 64 KiB, total 64 MiB
> *** Warning - bad CRC, using default environment
> 
> In:    serial@1e784000
> Out:   serial@1e784000
> Err:   serial@1e784000
> Net:   eth0: ftgmac@1e660000, eth1: ftgmac@1e680000, eth2: ftgmac@1e670000, eth3: ftgmac@1e690000
> Hit any key to stop autoboot:  0
> libfdt fdt_check_header(): FDT_ERR_BADMAGIC
> Error loading kernel FIT image
> => QEMU 8.2.10 monitor - type 'help' for more information
> (qemu) q
> 
> 
> What about the compiler ?

It's:
https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-aarch64-linux.tar.xz

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Gitlab runner timeout on evb-ast2600 ?

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 5278 bytes --]

On Fri, Apr 11, 2025 at 10:42:48AM -0600, Tom Rini wrote:
> On Fri, Apr 11, 2025 at 05:26:55PM +0200, Cédric Le Goater wrote:
> > On 4/11/25 17:01, Tom Rini wrote:
> > > On Fri, Apr 11, 2025 at 09:23:59AM +0200, Cédric Le Goater wrote:
> > > > Hi,
> > > > 
> > > > On 4/11/25 01:14, Tom Rini wrote:
> > > > > On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
> > > > > > On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
> > > > > > > 
> > > > > > > Hi everyone,
> > > > > > > 
> > > > > > > I keep getting an error on running the CI loop on -next on my tree :
> > > > > > > 
> > > > > > > https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
> > > > > > 
> > > > > > Thanks for the report.
> > > > > > 
> > > > > > > 
> > > > > > > ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
> > > > > > > tests will run !
> > > > > > > 
> > > > > > > Any clue ? It's been like this for few days. It doesn't look like a
> > > > > > > faulty patch.
> > > > > > 
> > > > > > I can't see that link you sent me, it 403s.
> > > > > > 
> > > > > > Do you have any other traces or information to share?
> > > > > 
> > > > > It's been intermittent recently. I was able to put qemu + gdb together
> > > > > and see it's one of the early initcalls that fails and hang()s but
> > > > > didn't see why. You should be able to use commit 1f0281294d8b ("Merge
> > > > > patch series "Annotate switch/case fallthrough cases"") as a starting
> > > > > point of failure.
> > > > > 
> > > > 
> > > > Do you have logs ?
> > > > 
> > > > Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :
> > > >      uboot_build=/path/to/u-boot/
> > > >      flashsize=65536
> > > >      flash=./flash-ast2600-evb-uboot
> > > >      dd of=$flash bs=1K if=/dev/zero count=$flashsize
> > > >      dd of=$flash conv=notrunc bs=1K if=$uboot_build/spl/u-boot-spl.bin
> > > >      dd of=$flash conv=notrunc bs=1K if=$uboot_build/u-boot.img seek=64
> > > >      $ qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
> > > >      qemu-system-arm: warning: nic ftgmac100.1 has no peer
> > > >      qemu-system-arm: warning: nic ftgmac100.2 has no peer
> > > >      qemu-system-arm: warning: nic ftgmac100.3 has no peer
> > > >      U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
> > > >      Trying to boot from RAM
> > > >      ## Checking hash(es) for config conf-1 ... OK
> > > >      ## Checking hash(es) for Image firmware-1 ... crc32+ OK
> > > >      ## Checking hash(es) for Image fdt-1 ... crc32+ OK
> > > >      U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
> > > 
> > > Please note that it's inconsistent about what commit fails. I would
> > > expect 1f0281294d8b in U-Boot to fail but I don't know about
> > > cb7555e93075.
> > > 
> > > Also can you please check QEMU v8.2.0? That's what we're on, and I can
> > > try and test 9.1.3 or 10.0.0-rc and see if it's a QEMU problem instead.
> > > Thanks!
> > 
> > Same result :
> > 
> > qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
> > qemu-system-arm: warning: nic ftgmac100.1 has no peer
> > qemu-system-arm: warning: nic ftgmac100.2 has no peer
> > qemu-system-arm: warning: nic ftgmac100.3 has no peer
> > 
> > U-Boot SPL 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)
> > Trying to boot from RAM
> > ## Checking hash(es) for config conf-1 ... OK
> > ## Checking hash(es) for Image firmware-1 ... crc32+ OK
> > ## Checking hash(es) for Image fdt-1 ... crc32+ OK
> > 
> > 
> > U-Boot 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)
> > 
> > Model: Aspeed BMC
> > DRAM:  already initialized, 1008 MiB (capacity:1024 MiB, VGA:64 MiB), ECC off
> > Core:  266 devices, 24 uclasses, devicetree: separate
> > WDT:   Started watchdog@1e785000 with servicing every 1000ms (60s timeout)
> > MMC:   sdhci@1e750100: 0
> > Loading Environment from SPIFlash... SF: Detected mx25u51245g with page size 256 Bytes, erase size 64 KiB, total 64 MiB
> > *** Warning - bad CRC, using default environment
> > 
> > In:    serial@1e784000
> > Out:   serial@1e784000
> > Err:   serial@1e784000
> > Net:   eth0: ftgmac@1e660000, eth1: ftgmac@1e680000, eth2: ftgmac@1e670000, eth3: ftgmac@1e690000
> > Hit any key to stop autoboot:  0
> > libfdt fdt_check_header(): FDT_ERR_BADMAGIC
> > Error loading kernel FIT image
> > => QEMU 8.2.10 monitor - type 'help' for more information
> > (qemu) q
> > 
> > 
> > What about the compiler ?
> 
> It's:
> https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-aarch64-linux.tar.xz

And part of what makes this a pain to debug is that trivial changes
break/fail it here. For example, ff0b7d741dec fails for me but if I make
a tiny source change so the version string is ff0b7d741dec-dirty it
passes. So you may need something longer for your ${uboot_build}
directory, mine is /home/trini/work/u-boot/u-boot.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 659 bytes --]

Re: Gitlab runner timeout on evb-ast2600 ?

[Cédric Le Goater]

On 4/11/25 18:42, Tom Rini wrote:
> On Fri, Apr 11, 2025 at 05:26:55PM +0200, Cédric Le Goater wrote:
>> On 4/11/25 17:01, Tom Rini wrote:
>>> On Fri, Apr 11, 2025 at 09:23:59AM +0200, Cédric Le Goater wrote:
>>>> Hi,
>>>>
>>>> On 4/11/25 01:14, Tom Rini wrote:
>>>>> On Fri, Apr 11, 2025 at 08:07:15AM +0930, Joel Stanley wrote:
>>>>>> On Mon, 7 Apr 2025 at 23:25, Eugen Hristev <eugen.hristev@linaro.org> wrote:
>>>>>>>
>>>>>>> Hi everyone,
>>>>>>>
>>>>>>> I keep getting an error on running the CI loop on -next on my tree :
>>>>>>>
>>>>>>> https://source.denx.de/u-boot/custodians/u-boot-at91/-/jobs/1090517
>>>>>>
>>>>>> Thanks for the report.
>>>>>>
>>>>>>>
>>>>>>> ! _pytest.outcomes.Exit: Lab timeout: Marking connection bad - no other
>>>>>>> tests will run !
>>>>>>>
>>>>>>> Any clue ? It's been like this for few days. It doesn't look like a
>>>>>>> faulty patch.
>>>>>>
>>>>>> I can't see that link you sent me, it 403s.
>>>>>>
>>>>>> Do you have any other traces or information to share?
>>>>>
>>>>> It's been intermittent recently. I was able to put qemu + gdb together
>>>>> and see it's one of the early initcalls that fails and hang()s but
>>>>> didn't see why. You should be able to use commit 1f0281294d8b ("Merge
>>>>> patch series "Annotate switch/case fallthrough cases"") as a starting
>>>>> point of failure.
>>>>>
>>>>
>>>> Do you have logs ?
>>>>
>>>> Things look ok from the QEMU side (latest 10.0.0-rc and 9.1) :
>>>>       uboot_build=/path/to/u-boot/
>>>>       flashsize=65536
>>>>       flash=./flash-ast2600-evb-uboot
>>>>       dd of=$flash bs=1K if=/dev/zero count=$flashsize
>>>>       dd of=$flash conv=notrunc bs=1K if=$uboot_build/spl/u-boot-spl.bin
>>>>       dd of=$flash conv=notrunc bs=1K if=$uboot_build/u-boot.img seek=64
>>>>       $ qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
>>>>       qemu-system-arm: warning: nic ftgmac100.1 has no peer
>>>>       qemu-system-arm: warning: nic ftgmac100.2 has no peer
>>>>       qemu-system-arm: warning: nic ftgmac100.3 has no peer
>>>>       U-Boot SPL 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
>>>>       Trying to boot from RAM
>>>>       ## Checking hash(es) for config conf-1 ... OK
>>>>       ## Checking hash(es) for Image firmware-1 ... crc32+ OK
>>>>       ## Checking hash(es) for Image fdt-1 ... crc32+ OK
>>>>       U-Boot 2025.04-00921-gcb7555e93075 (Apr 11 2025 - 09:08:40 +0200)
>>>
>>> Please note that it's inconsistent about what commit fails. I would
>>> expect 1f0281294d8b in U-Boot to fail but I don't know about
>>> cb7555e93075.
>>>
>>> Also can you please check QEMU v8.2.0? That's what we're on, and I can
>>> try and test 9.1.3 or 10.0.0-rc and see if it's a QEMU problem instead.
>>> Thanks!
>>
>> Same result :
>>
>> qemu-system-arm -M ast2600-evb -net nic,netdev=net0 -netdev user,id=net0 -nographic -snapshot -serial mon:stdio   -drive file=flash-ast2600-evb-uboot,format=raw,if=mtd
>> qemu-system-arm: warning: nic ftgmac100.1 has no peer
>> qemu-system-arm: warning: nic ftgmac100.2 has no peer
>> qemu-system-arm: warning: nic ftgmac100.3 has no peer
>>
>> U-Boot SPL 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)
>> Trying to boot from RAM
>> ## Checking hash(es) for config conf-1 ... OK
>> ## Checking hash(es) for Image firmware-1 ... crc32+ OK
>> ## Checking hash(es) for Image fdt-1 ... crc32+ OK
>>
>>
>> U-Boot 2025.04-00881-g1f0281294d8b (Apr 11 2025 - 17:23:25 +0200)
>>
>> Model: Aspeed BMC
>> DRAM:  already initialized, 1008 MiB (capacity:1024 MiB, VGA:64 MiB), ECC off
>> Core:  266 devices, 24 uclasses, devicetree: separate
>> WDT:   Started watchdog@1e785000 with servicing every 1000ms (60s timeout)
>> MMC:   sdhci@1e750100: 0
>> Loading Environment from SPIFlash... SF: Detected mx25u51245g with page size 256 Bytes, erase size 64 KiB, total 64 MiB
>> *** Warning - bad CRC, using default environment
>>
>> In:    serial@1e784000
>> Out:   serial@1e784000
>> Err:   serial@1e784000
>> Net:   eth0: ftgmac@1e660000, eth1: ftgmac@1e680000, eth2: ftgmac@1e670000, eth3: ftgmac@1e690000
>> Hit any key to stop autoboot:  0
>> libfdt fdt_check_header(): FDT_ERR_BADMAGIC
>> Error loading kernel FIT image
>> => QEMU 8.2.10 monitor - type 'help' for more information
>> (qemu) q
>>
>>
>> What about the compiler ?
> 
> It's:
> https://www.kernel.org/pub/tools/crosstool/files/bin/x86_64/13.2.0/x86_64-gcc-13.2.0-nolibc-aarch64-linux.tar.xz
> 

1f0281294d8b fails to compile :

   cc1: error: unknown value 'generic-armv7-a' for '-mtune'

I use most of the time :

arm-none-eabi-gcc (Fedora 14.1.0-2.fc41) 14.1.0

Thanks,

C.