mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Guillaume Tucker]

Hi Linus,

Please see below, the kernelci.org automated boot bisection found
that the OMAP3 BeagleBoard-xM is failing to boot with SMP
disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
log output after the bootloader so the kernel seems to be
crashing very early on.


Here are the details and log of the "bad" boot:

  https://kernelci.org/boot/id/5bde6b0459b51468bcda93e0/

  https://storage.kernelci.org/mainline/master/v4.19-12838-g71e56028173b/arm/multi_v7_defconfig+CONFIG_SMP=n/lab-baylibre/boot-omap3-beagle-xm.html

and the history of all the bisection steps and checks:

  http://lava.baylibre.com:10080/scheduler/device/omap3-beagle-xm-sea?recent_length=20&recent_search=lava-bisect-9615#recent_


Please note that the board went offline after that so we don't
know if it's already fixed in v4.20-rc1, although that would seem
quite unlikely.


As always, please take the automated bisection results with a
pinch of salt: the root cause often lies somewhere else than in
the "breaking" commit found, and there has been no further
investigation beyond the automated bisection results.

Hope this helps!

Best wishes,
Guillaume


On 04/11/2018 09:40, kernelci.org bot wrote:
> Bisection result for mainline/master (v4.19-12838-g71e56028173b) on beagle-xm
> 
>   Good:       83650fd58a93 Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
>   Bad:        71e56028173b Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>   Found:      513eb9859552 ARM: defconfig: Update multi_v7 to use PREEMPT
> 
> Details:
>   Good:       https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.19-12735-g83650fd58a93/
>   Bad:        https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.19-12838-g71e56028173b/
> 
> Checks:
>   revert:     PASS
>   verify:     PASS
> 
> Parameters:
>   Tree:       mainline
>   URL:        http://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
>   Branch:     master
>   Target:     beagle-xm
>   Lab:        lab-baylibre
>   Config:     multi_v7_defconfig+CONFIG_SMP=n
>   Plan:       boot
> 
> Breaking commit found:
> 
> -------------------------------------------------------------------------------
> commit 513eb98595522bc0cb83831a9daee1d5738e66f1
> Author: Linus Walleij <linus.walleij@linaro.org>
> Date:   Fri Oct 19 11:24:26 2018 +0200
> 
>     ARM: defconfig: Update multi_v7 to use PREEMPT
>     
>     Using CONFIG_PREEMPT as preemption model for ARMv7 systems
>     appear to be the most reasonable default.
>     
>     Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
>     Signed-off-by: Olof Johansson <olof@lixom.net>
> 
> diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig
> index 63af6234c1b6..1c7616815a86 100644
> --- a/arch/arm/configs/multi_v7_defconfig
> +++ b/arch/arm/configs/multi_v7_defconfig
> @@ -1,6 +1,7 @@
>  CONFIG_SYSVIPC=y
>  CONFIG_NO_HZ=y
>  CONFIG_HIGH_RES_TIMERS=y
> +CONFIG_PREEMPT=y
>  CONFIG_CGROUPS=y
>  CONFIG_BLK_DEV_INITRD=y
>  CONFIG_EMBEDDED=y
> -------------------------------------------------------------------------------
> 
> 
> Git bisection log:
> 
> -------------------------------------------------------------------------------
> git bisect start
> # good: [83650fd58a934fad5b7735fea96905ef986c0821] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
> git bisect good 83650fd58a934fad5b7735fea96905ef986c0821
> # bad: [71e56028173bc84f01456a5679d8be9d681b49f1] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 71e56028173bc84f01456a5679d8be9d681b49f1
> # good: [4e303fbe2d95806c875f5ebfcb3d980e20b4bd83] perf top: Allow disabling the overwrite mode
> git bisect good 4e303fbe2d95806c875f5ebfcb3d980e20b4bd83
> # bad: [01897f3e05ede4d66c0f9df465fde1d67a1d733f] Merge branch 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
> git bisect bad 01897f3e05ede4d66c0f9df465fde1d67a1d733f
> # good: [ff27a06af6ffd3f49b9e193eb68f487ad76651e1] perf top: Start display thread earlier
> git bisect good ff27a06af6ffd3f49b9e193eb68f487ad76651e1
> # bad: [d2ff0ff2c23f1bacd35073bf50e6c18298a8d530] Merge tag 'armsoc-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc
> git bisect bad d2ff0ff2c23f1bacd35073bf50e6c18298a8d530
> # bad: [d4f79cb872bd4faa44eb133c1aee5f3411342c54] ARM: orion: avoid VLA in orion_mpp_conf
> git bisect bad d4f79cb872bd4faa44eb133c1aee5f3411342c54
> # good: [7f3d08f5255b964827214e69967b4d5008d6ea18] arm64: defconfig: Enable some IPMI configs
> git bisect good 7f3d08f5255b964827214e69967b4d5008d6ea18
> # bad: [513eb98595522bc0cb83831a9daee1d5738e66f1] ARM: defconfig: Update multi_v7 to use PREEMPT
> git bisect bad 513eb98595522bc0cb83831a9daee1d5738e66f1
> # first bad commit: [513eb98595522bc0cb83831a9daee1d5738e66f1] ARM: defconfig: Update multi_v7 to use PREEMPT
> -------------------------------------------------------------------------------
> 

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Olof Johansson]

On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:
>
> Hi Linus,
>
> Please see below, the kernelci.org automated boot bisection found
> that the OMAP3 BeagleBoard-xM is failing to boot with SMP
> disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
> log output after the bootloader so the kernel seems to be
> crashing very early on.

Can you enable DEBUG_LL to capture what goes wrong?

The regression is on me. Linus sent the patch, but I was trigger happy
and applied it to fixes instead of letting it soak in -next longer.

I'll revert it in our fixes branch and send it in quickly, and apply
it to a branch picked up by -next instead.


-Olof

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Guillaume Tucker]

On 05/11/2018 14:33, Olof Johansson wrote:
> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker
> <guillaume.tucker@collabora.com> wrote:
>>
>> Hi Linus,
>>
>> Please see below, the kernelci.org automated boot bisection found
>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP
>> disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
>> log output after the bootloader so the kernel seems to be
>> crashing very early on.
> 
> Can you enable DEBUG_LL to capture what goes wrong?

Yes I've scheduled a re-run of the automated bisection with
DEBUG_LL turned on, but the beagle-xm board is currently offline
in lab-baylibre.  I'll keep you posted when I get some results.

> The regression is on me. Linus sent the patch, but I was trigger happy
> and applied it to fixes instead of letting it soak in -next longer.
> 
> I'll revert it in our fixes branch and send it in quickly, and apply
> it to a branch picked up by -next instead.

Ack - will also keep an eye on boot reports.

Guillaume

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Guillaume Tucker]

On 06/11/2018 10:54, Guillaume Tucker wrote:
> On 05/11/2018 14:33, Olof Johansson wrote:
>> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker
>> <guillaume.tucker@collabora.com> wrote:
>>>
>>> Hi Linus,
>>>
>>> Please see below, the kernelci.org automated boot bisection found
>>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP
>>> disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
>>> log output after the bootloader so the kernel seems to be
>>> crashing very early on.
>>
>> Can you enable DEBUG_LL to capture what goes wrong?
> 
> Yes I've scheduled a re-run of the automated bisection with
> DEBUG_LL turned on, but the beagle-xm board is currently offline
> in lab-baylibre.  I'll keep you posted when I get some results.

Unfortunately, enabling the DEBUG_LL config didn't help.  There's
still no visible kernel log at all from the failing LAVA jobs:

    http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table


This might be a hard one to debug without physical access to the
board, but please let me know if you think there's anything else
I can try.  For example, if there's a known older kernel revision
where the combination of SMP=n and PREEMPT=y worked then I could
try to run an automated boot bisection based on that.

Guillaume

>> The regression is on me. Linus sent the patch, but I was trigger happy
>> and applied it to fixes instead of letting it soak in -next longer.
>>
>> I'll revert it in our fixes branch and send it in quickly, and apply
>> it to a branch picked up by -next instead.
> 
> Ack - will also keep an eye on boot reports.
> 
> Guillaume
> 

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Olof Johansson]

On Wed, Nov 7, 2018 at 8:09 AM Guillaume Tucker
<guillaume.tucker@collabora.com> wrote:
>
> On 06/11/2018 10:54, Guillaume Tucker wrote:
> > On 05/11/2018 14:33, Olof Johansson wrote:
> >> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker
> >> <guillaume.tucker@collabora.com> wrote:
> >>>
> >>> Hi Linus,
> >>>
> >>> Please see below, the kernelci.org automated boot bisection found
> >>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP
> >>> disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
> >>> log output after the bootloader so the kernel seems to be
> >>> crashing very early on.
> >>
> >> Can you enable DEBUG_LL to capture what goes wrong?
> >
> > Yes I've scheduled a re-run of the automated bisection with
> > DEBUG_LL turned on, but the beagle-xm board is currently offline
> > in lab-baylibre.  I'll keep you posted when I get some results.
>
> Unfortunately, enabling the DEBUG_LL config didn't help.  There's
> still no visible kernel log at all from the failing LAVA jobs:
>
>     http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table
>
>
> This might be a hard one to debug without physical access to the
> board, but please let me know if you think there's anything else
> I can try.  For example, if there's a known older kernel revision
> where the combination of SMP=n and PREEMPT=y worked then I could
> try to run an automated boot bisection based on that.

Is this only beagle-xm, or do we see it on all OMAP3 hardware?


-Olof

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Tony Lindgren]

* Olof Johansson <olof@lixom.net> [181107 16:17]:
> On Wed, Nov 7, 2018 at 8:09 AM Guillaume Tucker
> <guillaume.tucker@collabora.com> wrote:
> >
> > On 06/11/2018 10:54, Guillaume Tucker wrote:
> > > On 05/11/2018 14:33, Olof Johansson wrote:
> > >> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker
> > >> <guillaume.tucker@collabora.com> wrote:
> > >>>
> > >>> Hi Linus,
> > >>>
> > >>> Please see below, the kernelci.org automated boot bisection found
> > >>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP
> > >>> disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
> > >>> log output after the bootloader so the kernel seems to be
> > >>> crashing very early on.
> > >>
> > >> Can you enable DEBUG_LL to capture what goes wrong?
> > >
> > > Yes I've scheduled a re-run of the automated bisection with
> > > DEBUG_LL turned on, but the beagle-xm board is currently offline
> > > in lab-baylibre.  I'll keep you posted when I get some results.
> >
> > Unfortunately, enabling the DEBUG_LL config didn't help.  There's
> > still no visible kernel log at all from the failing LAVA jobs:
> >
> >     http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table
> >
> >
> > This might be a hard one to debug without physical access to the
> > board, but please let me know if you think there's anything else
> > I can try.  For example, if there's a known older kernel revision
> > where the combination of SMP=n and PREEMPT=y worked then I could
> > try to run an automated boot bisection based on that.
> 
> Is this only beagle-xm, or do we see it on all OMAP3 hardware?

Seems to not happen on three other omap3 boards I've tried with.

Regards,

Tony

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Kevin Hilman]

Guillaume Tucker <guillaume.tucker@collabora.com> writes:

> On 06/11/2018 10:54, Guillaume Tucker wrote:
>> On 05/11/2018 14:33, Olof Johansson wrote:
>>> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker
>>> <guillaume.tucker@collabora.com> wrote:
>>>>
>>>> Hi Linus,
>>>>
>>>> Please see below, the kernelci.org automated boot bisection found
>>>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP
>>>> disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
>>>> log output after the bootloader so the kernel seems to be
>>>> crashing very early on.
>>>
>>> Can you enable DEBUG_LL to capture what goes wrong?
>> 
>> Yes I've scheduled a re-run of the automated bisection with
>> DEBUG_LL turned on, but the beagle-xm board is currently offline
>> in lab-baylibre.  I'll keep you posted when I get some results.
>
> Unfortunately, enabling the DEBUG_LL config didn't help.  There's
> still no visible kernel log at all from the failing LAVA jobs:
>
>     http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table
>
>
> This might be a hard one to debug without physical access to the
> board, but please let me know if you think there's anything else
> I can try.  For example, if there's a known older kernel revision
> where the combination of SMP=n and PREEMPT=y worked then I could
> try to run an automated boot bisection based on that.

The reason DEBUG_LL doesn't show anything is because it also needs
a specific UART enabled for the output.  For the case of beagle-xm, it
needs CONFIG_DEBUG_OMAP3UART3=y also.

It did that locally, and the error on the console is this:

   Error: invalid dtb and unrecognized/unsupported machine ID
     r1=0x0000060a, r2=0x00000000
   Available machine support:

Upon closer inspection, the load addrs are such that it appears the DTB
is getting overwritten during zImage compression.

So bascially, it's just an (un)lucky fluke that CONFIG_PREEMPT=y changed
the kernel size just enough to cause this problem.

I did a local test (with pyboot, not LAVA) with all the same load
addresses, but without initrd_high and fdt_high, and it boots fine[1]

Guillaume, I think the proper solution is to fixup the LAVA device-type
for beagle-xm.  In particular, it's setting the u-boot variables
'initrd_high' and 'fdt_high' which means it's not letting uboot move
them.

Looks like that can be overridden in the device-dict using
"set base_high_limits = false", so I've done that now to this device[2]

Kevin

[1] http://termbin.com/9mm6
[2] http://lava.baylibre.com:10080/scheduler/device/omap3-beagle-xm-sea/devicedict

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Olof Johansson]

On Thu, Nov 8, 2018 at 2:34 PM Kevin Hilman <khilman@baylibre.com> wrote:
>
> Guillaume Tucker <guillaume.tucker@collabora.com> writes:
>
> > On 06/11/2018 10:54, Guillaume Tucker wrote:
> >> On 05/11/2018 14:33, Olof Johansson wrote:
> >>> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker
> >>> <guillaume.tucker@collabora.com> wrote:
> >>>>
> >>>> Hi Linus,
> >>>>
> >>>> Please see below, the kernelci.org automated boot bisection found
> >>>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP
> >>>> disabled and PREEMPT enabled in multi_v7_defconfig.  There is no
> >>>> log output after the bootloader so the kernel seems to be
> >>>> crashing very early on.
> >>>
> >>> Can you enable DEBUG_LL to capture what goes wrong?
> >>
> >> Yes I've scheduled a re-run of the automated bisection with
> >> DEBUG_LL turned on, but the beagle-xm board is currently offline
> >> in lab-baylibre.  I'll keep you posted when I get some results.
> >
> > Unfortunately, enabling the DEBUG_LL config didn't help.  There's
> > still no visible kernel log at all from the failing LAVA jobs:
> >
> >     http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table
> >
> >
> > This might be a hard one to debug without physical access to the
> > board, but please let me know if you think there's anything else
> > I can try.  For example, if there's a known older kernel revision
> > where the combination of SMP=n and PREEMPT=y worked then I could
> > try to run an automated boot bisection based on that.
>
> The reason DEBUG_LL doesn't show anything is because it also needs
> a specific UART enabled for the output.  For the case of beagle-xm, it
> needs CONFIG_DEBUG_OMAP3UART3=y also.
>
> It did that locally, and the error on the console is this:
>
>    Error: invalid dtb and unrecognized/unsupported machine ID
>      r1=0x0000060a, r2=0x00000000
>    Available machine support:
>
> Upon closer inspection, the load addrs are such that it appears the DTB
> is getting overwritten during zImage compression.
>
> So bascially, it's just an (un)lucky fluke that CONFIG_PREEMPT=y changed
> the kernel size just enough to cause this problem.
>
> I did a local test (with pyboot, not LAVA) with all the same load
> addresses, but without initrd_high and fdt_high, and it boots fine[1]
>
> Guillaume, I think the proper solution is to fixup the LAVA device-type
> for beagle-xm.  In particular, it's setting the u-boot variables
> 'initrd_high' and 'fdt_high' which means it's not letting uboot move
> them.
>
> Looks like that can be overridden in the device-dict using
> "set base_high_limits = false", so I've done that now to this device[2]

Argh, I remember hitting these issues a few times through the years.
Thanks for finding the problem, Kevin.


-Olof

mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm

[Linus Walleij]

On Thu, Nov 8, 2018 at 11:34 PM Kevin Hilman <khilman@baylibre.com> wrote:

> Upon closer inspection, the load addrs are such that it appears the DTB
> is getting overwritten during zImage compression.
>
> So bascially, it's just an (un)lucky fluke that CONFIG_PREEMPT=y changed
> the kernel size just enough to cause this problem.

Oh that kind of bug. It has bitten me more than once :(

Thanks a lot for rootcausing this Kevin!

Yours,
Linus Walleij