arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

[Arnd Bergmann]

On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
> arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
> 
> Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
> Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
> 
> Tree: arm-soc
> Branch: local/for-next
> Git Describe: v4.4-rc3-483-gb48b5ceb7227
> Git Commit: b48b5ceb7227a8e6e548dae7a44c80ad85719808
> Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc.git
> Tested: 67 unique boards, 18 SoC families, 22 builds out of 135
> 
> Boot Failures Detected: https://kernelci.org/boot/?v4.4-rc3-483-gb48b5ceb7227&fail

We got three regressions in the arm-soc for-next branch today, compared to yesterday:

* I did not merge any rockchip changes, but it looks like multi_v7_defconfig is
  now broken on rk3288-rock2-square and rk3288-veyron-jerry (not in the list below,
  but in the web interface). I put a diff of the defconfig file on
  http://pastebin.com/DHgzz1jr, but I don't anything that may have cause the
  regression.

* bcm4708-smartrg-sr400ac doesn't find its rootfs any more

[   16.670994] No filesystem could mount root, tried:  ext3 ext2 ext4 squashfs vfat msdos ntfs
[   16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)

  I merged some patches from Florian yesterday, but it could also be a problem with 
  the defconfig changes

* Juno seems to be sick and doesn't like LAVA today:

Connected to serial03.lavalab.
Escape character is '^]'.
Command sent successfully.
Command sent successfully.
ErrorMessage: Failed to boot test image.
Lava failed at action boot_linaro_image with error:Failed to boot test image.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lava_dispatcher/job.py", line 381, in run
    action.run(**params)
  File "/usr/lib/python2.7/dist-packages/lava_dispatcher/actions/boot_control.py", line 156, in run
    raise CriticalError("Failed to boot test image.")
CriticalError: Failed to boot test image.
Command sent successfully.

	Arnd

> arm:
> 
>     multi_v7_defconfig+CONFIG_THUMB2_KERNEL=y:
>         bcm4708-smartrg-sr400ac: 1 failed lab
>         rk3288-rock2-square: 1 failed lab
> 
>     multi_v7_defconfig+CONFIG_LKDTM=y:
>         at91-sama5d3_xplained: 1 failed lab
>         bcm4708-smartrg-sr400ac: 1 failed lab
>         exynos5422-odroidxu3: 1 failed lab
>         rk3288-rock2-square: 1 failed lab
> 
>     multi_v7_defconfig+CONFIG_PROVE_LOCKING=y:
>         rk3288-rock2-square: 1 failed lab
> 
>     multi_v7_defconfig+CONFIG_ARM_LPAE=y:
>         rk3288-rock2-square: 1 failed lab
> 
>     multi_v7_defconfig:
>         bcm4708-smartrg-sr400ac: 1 failed lab
>         rk3288-rock2-square: 1 failed lab
>         rk3288-rock2-square_rootfs:nfs: 1 failed lab
> 
> arm64:
> 
>     defconfig+CONFIG_LKDTM=y:
>         juno: 1 failed lab
> 

arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

[Jon Medhurst (Tixy)]

On Wed, 2015-12-16 at 11:17 +0100, Arnd Bergmann wrote:
> On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
> > arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
> > 
> > Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
> > Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
> > 
> >
[...]
> * Juno seems to be sick and doesn't like LAVA today:

Looks like a LAVA hickup, I downloaded the Image and btb and they appear
to work fine on my board. Just to make sure I was running the right
kernel...

# uname -a
Linux genericarmv8 4.4.0-rc3+ #1 SMP PREEMPT Wed Dec 16 05:51:25 CET 2015 aarch64 GNU/Linux

-- 
Tixy

arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

[Heiko Stübner]

Am Mittwoch, 16. Dezember 2015, 11:17:30 schrieb Arnd Bergmann:
> On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
> > arm-soc boot: 320 boots: 12 failed, 308 passed
> > (v4.4-rc3-483-gb48b5ceb7227)
> > 
> > Full Boot Summary:
> > https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb72
> > 27/ Full Build Summary:
> > https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
> > 
> > Tree: arm-soc
> > Branch: local/for-next
> > Git Describe: v4.4-rc3-483-gb48b5ceb7227
> > Git Commit: b48b5ceb7227a8e6e548dae7a44c80ad85719808
> > Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc.git
> > Tested: 67 unique boards, 18 SoC families, 22 builds out of 135
> > 
> > Boot Failures Detected:
> > https://kernelci.org/boot/?v4.4-rc3-483-gb48b5ceb7227&fail
> We got three regressions in the arm-soc for-next branch today, compared to
> yesterday:
> 
> * I did not merge any rockchip changes, but it looks like multi_v7_defconfig
> is now broken on rk3288-rock2-square and rk3288-veyron-jerry (not in the
> list below, but in the web interface). I put a diff of the defconfig file
> on http://pastebin.com/DHgzz1jr, but I don't anything that may have cause
> the regression.

I would also think more of a hickup of the kernelci systems. When looking at 
veyron-jerry it shows the tftp kernel transfer not completing
(line "Waiting for the transfer"...) and then the board getting reset and 
booting the original chromeos after that.

On the rock2 there is also nothing standing out in terms of errors or so.
So I guess it might be best to simply check if that is still present tomorrow.


Heiko

arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

[Tyler Baker]

On 16 December 2015 at 02:17, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 16 December 2015 01:02:44 kernelci. org bot wrote:
>> arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)
>>
>> Full Boot Summary: https://kernelci.org/boot/all/job/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
>> Full Build Summary: https://kernelci.org/build/arm-soc/kernel/v4.4-rc3-483-gb48b5ceb7227/
>>
>> Tree: arm-soc
>> Branch: local/for-next
>> Git Describe: v4.4-rc3-483-gb48b5ceb7227
>> Git Commit: b48b5ceb7227a8e6e548dae7a44c80ad85719808
>> Git URL: git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc.git
>> Tested: 67 unique boards, 18 SoC families, 22 builds out of 135
>>
>> Boot Failures Detected: https://kernelci.org/boot/?v4.4-rc3-483-gb48b5ceb7227&fail
>
> We got three regressions in the arm-soc for-next branch today, compared to yesterday:

I had the bot bisect these failures, results below.

>
> * I did not merge any rockchip changes, but it looks like multi_v7_defconfig is
>   now broken on rk3288-rock2-square and rk3288-veyron-jerry (not in the list below,
>   but in the web interface). I put a diff of the defconfig file on
>   http://pastebin.com/DHgzz1jr, but I don't anything that may have cause the
>   regression.

The bisection pointed d2318f340253 ("arm64: dts: berlin4ct: add
watchdog nodes") which is clearly a red herring. kernelci.org inserts
the modules from the build into the ramdisk, and eudev will probe
them. With the modules inserted into the ramdisk, it fails to reach a
user space prompt. If I remove the modules from the ramdisk it boots
fine[1]. I may manually bisect if we cannot spot the issue easily.

> * bcm4708-smartrg-sr400ac doesn't find its rootfs any more
>
> [   16.670994] No filesystem could mount root, tried:  ext3 ext2 ext4 squashfs vfat msdos ntfs
> [   16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
>
>   I merged some patches from Florian yesterday, but it could also be a problem with
>   the defconfig changes

The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig:
Enable some drivers for LS1021A")

I've rebased my tree with arm-soc for-next, and reverted the commits
listed above. Should know shortly if the reverts fix the boot issues.

> * Juno seems to be sick and doesn't like LAVA today:
>
> Connected to serial03.lavalab.
> Escape character is '^]'.
> Command sent successfully.
> Command sent successfully.
> ErrorMessage: Failed to boot test image.
> Lava failed at action boot_linaro_image with error:Failed to boot test image.
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/dist-packages/lava_dispatcher/job.py", line 381, in run
>     action.run(**params)
>   File "/usr/lib/python2.7/dist-packages/lava_dispatcher/actions/boot_control.py", line 156, in run
>     raise CriticalError("Failed to boot test image.")
> CriticalError: Failed to boot test image.
> Command sent successfully.

Indeed, I'll look into this. Thanks for reporting.

Cheers,

Tyler

[1] http://nwdrone.com/scheduler/job/13662/log_file

arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

[Arnd Bergmann]

On Wednesday 16 December 2015 11:34:30 Tyler Baker wrote:
> > * bcm4708-smartrg-sr400ac doesn't find its rootfs any more
> >
> > [   16.670994] No filesystem could mount root, tried:  ext3 ext2 ext4 squashfs vfat msdos ntfs
> > [   16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
> >
> >   I merged some patches from Florian yesterday, but it could also be a problem with
> >   the defconfig changes
> 
> The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig:
> Enable some drivers for LS1021A")
> 
> I've rebased my tree with arm-soc for-next, and reverted the commits
> listed above. Should know shortly if the reverts fix the boot issues.

I've looked at the built-in drivers to see if they might do something
on unrelated hardware (we've had bugs like that before), but didn't
find anything: the watchdog and cpufreq drivers all look sane, everything
else is a loadable module.

One possible explanation would be a buggy bootloader that fails to pass
a correct initrd and/or dtb if the kernel image gets too big.

Do you know if there is an initrd that is passed on this platform?

	Arnd

arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

[Tyler Baker]

On 16 December 2015 at 12:16, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday 16 December 2015 11:34:30 Tyler Baker wrote:
>> > * bcm4708-smartrg-sr400ac doesn't find its rootfs any more
>> >
>> > [   16.670994] No filesystem could mount root, tried:  ext3 ext2 ext4 squashfs vfat msdos ntfs
>> > [   16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
>> >
>> >   I merged some patches from Florian yesterday, but it could also be a problem with
>> >   the defconfig changes
>>
>> The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig:
>> Enable some drivers for LS1021A")
>>
>> I've rebased my tree with arm-soc for-next, and reverted the commits
>> listed above. Should know shortly if the reverts fix the boot issues.
>
> I've looked at the built-in drivers to see if they might do something
> on unrelated hardware (we've had bugs like that before), but didn't
> find anything: the watchdog and cpufreq drivers all look sane, everything
> else is a loadable module.
>
> One possible explanation would be a buggy bootloader that fails to pass
> a correct initrd and/or dtb if the kernel image gets too big.
>
> Do you know if there is an initrd that is passed on this platform?

There is, and if I disable only CONFIG_BLK_DEV_RAM the
bcm4708-smartrg-sr400ac boots again. This platform runs a CFE
bootloader, and this is these are the kernel arguments injected into
the DT - "console=ttyS0,115200 initrd=0x4000000,8M root=/dev/ram0
ip=dhcp".

I'm still investigating the rk3288 failures, and will report back when
I find something.

Cheers,

Tyler

arm-soc boot: 320 boots: 12 failed, 308 passed (v4.4-rc3-483-gb48b5ceb7227)

[Arnd Bergmann]

On Wednesday 16 December 2015 13:51:10 Tyler Baker wrote:
> On 16 December 2015 at 12:16, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Wednesday 16 December 2015 11:34:30 Tyler Baker wrote:
> >> > * bcm4708-smartrg-sr400ac doesn't find its rootfs any more
> >> >
> >> > [   16.670994] No filesystem could mount root, tried:  ext3 ext2 ext4 squashfs vfat msdos ntfs
> >> > [   16.679388] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
> >> >
> >> >   I merged some patches from Florian yesterday, but it could also be a problem with
> >> >   the defconfig changes
> >>
> >> The bisection pointed to b02ec7658c98 ("ARM: multi_v7_defconfig:
> >> Enable some drivers for LS1021A")
> >>
> >> I've rebased my tree with arm-soc for-next, and reverted the commits
> >> listed above. Should know shortly if the reverts fix the boot issues.
> >
> > I've looked at the built-in drivers to see if they might do something
> > on unrelated hardware (we've had bugs like that before), but didn't
> > find anything: the watchdog and cpufreq drivers all look sane, everything
> > else is a loadable module.
> >
> > One possible explanation would be a buggy bootloader that fails to pass
> > a correct initrd and/or dtb if the kernel image gets too big.
> >
> > Do you know if there is an initrd that is passed on this platform?
> 
> There is, and if I disable only CONFIG_BLK_DEV_RAM the
> bcm4708-smartrg-sr400ac boots again. This platform runs a CFE
> bootloader, and this is these are the kernel arguments injected into
> the DT - "console=ttyS0,115200 initrd=0x4000000,8M root=/dev/ram0
> ip=dhcp".

Ok, so it's probably the initrd= argument that causes the problem: as
long as CONFIG_BLK_DEV_INITRD was disabled, the code in arm_memblock_init
would ignore the initrd that gets passed by the bootloader and just
use the initramfs that is appended to the kernel.

There may also be some problem related to how the bootloader uses
a command line rather than the appropriate ATAG to pass the initrd
location.

	Arnd