* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm [not found] <5bdebe88.1c69fb81.3857a.05d1@mx.google.com> @ 2018-11-05 9:43 ` Guillaume Tucker 2018-11-05 14:33 ` Olof Johansson 0 siblings, 1 reply; 9+ messages in thread From: Guillaume Tucker @ 2018-11-05 9:43 UTC (permalink / raw) To: linux-arm-kernel Hi Linus, Please see below, the kernelci.org automated boot bisection found that the OMAP3 BeagleBoard-xM is failing to boot with SMP disabled and PREEMPT enabled in multi_v7_defconfig. There is no log output after the bootloader so the kernel seems to be crashing very early on. Here are the details and log of the "bad" boot: https://kernelci.org/boot/id/5bde6b0459b51468bcda93e0/ https://storage.kernelci.org/mainline/master/v4.19-12838-g71e56028173b/arm/multi_v7_defconfig+CONFIG_SMP=n/lab-baylibre/boot-omap3-beagle-xm.html and the history of all the bisection steps and checks: http://lava.baylibre.com:10080/scheduler/device/omap3-beagle-xm-sea?recent_length=20&recent_search=lava-bisect-9615#recent_ Please note that the board went offline after that so we don't know if it's already fixed in v4.20-rc1, although that would seem quite unlikely. As always, please take the automated bisection results with a pinch of salt: the root cause often lies somewhere else than in the "breaking" commit found, and there has been no further investigation beyond the automated bisection results. Hope this helps! Best wishes, Guillaume On 04/11/2018 09:40, kernelci.org bot wrote: > Bisection result for mainline/master (v4.19-12838-g71e56028173b) on beagle-xm > > Good: 83650fd58a93 Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux > Bad: 71e56028173b Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > Found: 513eb9859552 ARM: defconfig: Update multi_v7 to use PREEMPT > > Details: > Good: https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.19-12735-g83650fd58a93/ > Bad: https://kernelci.org/boot/all/job/mainline/branch/master/kernel/v4.19-12838-g71e56028173b/ > > Checks: > revert: PASS > verify: PASS > > Parameters: > Tree: mainline > URL: http://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > Branch: master > Target: beagle-xm > Lab: lab-baylibre > Config: multi_v7_defconfig+CONFIG_SMP=n > Plan: boot > > Breaking commit found: > > ------------------------------------------------------------------------------- > commit 513eb98595522bc0cb83831a9daee1d5738e66f1 > Author: Linus Walleij <linus.walleij@linaro.org> > Date: Fri Oct 19 11:24:26 2018 +0200 > > ARM: defconfig: Update multi_v7 to use PREEMPT > > Using CONFIG_PREEMPT as preemption model for ARMv7 systems > appear to be the most reasonable default. > > Signed-off-by: Linus Walleij <linus.walleij@linaro.org> > Signed-off-by: Olof Johansson <olof@lixom.net> > > diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig > index 63af6234c1b6..1c7616815a86 100644 > --- a/arch/arm/configs/multi_v7_defconfig > +++ b/arch/arm/configs/multi_v7_defconfig > @@ -1,6 +1,7 @@ > CONFIG_SYSVIPC=y > CONFIG_NO_HZ=y > CONFIG_HIGH_RES_TIMERS=y > +CONFIG_PREEMPT=y > CONFIG_CGROUPS=y > CONFIG_BLK_DEV_INITRD=y > CONFIG_EMBEDDED=y > ------------------------------------------------------------------------------- > > > Git bisection log: > > ------------------------------------------------------------------------------- > git bisect start > # good: [83650fd58a934fad5b7735fea96905ef986c0821] Merge tag 'arm64-upstream' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux > git bisect good 83650fd58a934fad5b7735fea96905ef986c0821 > # bad: [71e56028173bc84f01456a5679d8be9d681b49f1] Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect bad 71e56028173bc84f01456a5679d8be9d681b49f1 > # good: [4e303fbe2d95806c875f5ebfcb3d980e20b4bd83] perf top: Allow disabling the overwrite mode > git bisect good 4e303fbe2d95806c875f5ebfcb3d980e20b4bd83 > # bad: [01897f3e05ede4d66c0f9df465fde1d67a1d733f] Merge branch 'perf-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect bad 01897f3e05ede4d66c0f9df465fde1d67a1d733f > # good: [ff27a06af6ffd3f49b9e193eb68f487ad76651e1] perf top: Start display thread earlier > git bisect good ff27a06af6ffd3f49b9e193eb68f487ad76651e1 > # bad: [d2ff0ff2c23f1bacd35073bf50e6c18298a8d530] Merge tag 'armsoc-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm/arm-soc > git bisect bad d2ff0ff2c23f1bacd35073bf50e6c18298a8d530 > # bad: [d4f79cb872bd4faa44eb133c1aee5f3411342c54] ARM: orion: avoid VLA in orion_mpp_conf > git bisect bad d4f79cb872bd4faa44eb133c1aee5f3411342c54 > # good: [7f3d08f5255b964827214e69967b4d5008d6ea18] arm64: defconfig: Enable some IPMI configs > git bisect good 7f3d08f5255b964827214e69967b4d5008d6ea18 > # bad: [513eb98595522bc0cb83831a9daee1d5738e66f1] ARM: defconfig: Update multi_v7 to use PREEMPT > git bisect bad 513eb98595522bc0cb83831a9daee1d5738e66f1 > # first bad commit: [513eb98595522bc0cb83831a9daee1d5738e66f1] ARM: defconfig: Update multi_v7 to use PREEMPT > ------------------------------------------------------------------------------- > ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-05 9:43 ` mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm Guillaume Tucker @ 2018-11-05 14:33 ` Olof Johansson 2018-11-06 10:54 ` Guillaume Tucker 0 siblings, 1 reply; 9+ messages in thread From: Olof Johansson @ 2018-11-05 14:33 UTC (permalink / raw) To: linux-arm-kernel On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker <guillaume.tucker@collabora.com> wrote: > > Hi Linus, > > Please see below, the kernelci.org automated boot bisection found > that the OMAP3 BeagleBoard-xM is failing to boot with SMP > disabled and PREEMPT enabled in multi_v7_defconfig. There is no > log output after the bootloader so the kernel seems to be > crashing very early on. Can you enable DEBUG_LL to capture what goes wrong? The regression is on me. Linus sent the patch, but I was trigger happy and applied it to fixes instead of letting it soak in -next longer. I'll revert it in our fixes branch and send it in quickly, and apply it to a branch picked up by -next instead. -Olof ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-05 14:33 ` Olof Johansson @ 2018-11-06 10:54 ` Guillaume Tucker 2018-11-07 16:09 ` Guillaume Tucker 0 siblings, 1 reply; 9+ messages in thread From: Guillaume Tucker @ 2018-11-06 10:54 UTC (permalink / raw) To: linux-arm-kernel On 05/11/2018 14:33, Olof Johansson wrote: > On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker > <guillaume.tucker@collabora.com> wrote: >> >> Hi Linus, >> >> Please see below, the kernelci.org automated boot bisection found >> that the OMAP3 BeagleBoard-xM is failing to boot with SMP >> disabled and PREEMPT enabled in multi_v7_defconfig. There is no >> log output after the bootloader so the kernel seems to be >> crashing very early on. > > Can you enable DEBUG_LL to capture what goes wrong? Yes I've scheduled a re-run of the automated bisection with DEBUG_LL turned on, but the beagle-xm board is currently offline in lab-baylibre. I'll keep you posted when I get some results. > The regression is on me. Linus sent the patch, but I was trigger happy > and applied it to fixes instead of letting it soak in -next longer. > > I'll revert it in our fixes branch and send it in quickly, and apply > it to a branch picked up by -next instead. Ack - will also keep an eye on boot reports. Guillaume ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-06 10:54 ` Guillaume Tucker @ 2018-11-07 16:09 ` Guillaume Tucker 2018-11-07 16:16 ` Olof Johansson 2018-11-08 22:34 ` Kevin Hilman 0 siblings, 2 replies; 9+ messages in thread From: Guillaume Tucker @ 2018-11-07 16:09 UTC (permalink / raw) To: linux-arm-kernel On 06/11/2018 10:54, Guillaume Tucker wrote: > On 05/11/2018 14:33, Olof Johansson wrote: >> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker >> <guillaume.tucker@collabora.com> wrote: >>> >>> Hi Linus, >>> >>> Please see below, the kernelci.org automated boot bisection found >>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP >>> disabled and PREEMPT enabled in multi_v7_defconfig. There is no >>> log output after the bootloader so the kernel seems to be >>> crashing very early on. >> >> Can you enable DEBUG_LL to capture what goes wrong? > > Yes I've scheduled a re-run of the automated bisection with > DEBUG_LL turned on, but the beagle-xm board is currently offline > in lab-baylibre. I'll keep you posted when I get some results. Unfortunately, enabling the DEBUG_LL config didn't help. There's still no visible kernel log at all from the failing LAVA jobs: http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table This might be a hard one to debug without physical access to the board, but please let me know if you think there's anything else I can try. For example, if there's a known older kernel revision where the combination of SMP=n and PREEMPT=y worked then I could try to run an automated boot bisection based on that. Guillaume >> The regression is on me. Linus sent the patch, but I was trigger happy >> and applied it to fixes instead of letting it soak in -next longer. >> >> I'll revert it in our fixes branch and send it in quickly, and apply >> it to a branch picked up by -next instead. > > Ack - will also keep an eye on boot reports. > > Guillaume > ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-07 16:09 ` Guillaume Tucker @ 2018-11-07 16:16 ` Olof Johansson 2018-11-08 15:54 ` Tony Lindgren 2018-11-08 22:34 ` Kevin Hilman 1 sibling, 1 reply; 9+ messages in thread From: Olof Johansson @ 2018-11-07 16:16 UTC (permalink / raw) To: linux-arm-kernel On Wed, Nov 7, 2018 at 8:09 AM Guillaume Tucker <guillaume.tucker@collabora.com> wrote: > > On 06/11/2018 10:54, Guillaume Tucker wrote: > > On 05/11/2018 14:33, Olof Johansson wrote: > >> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker > >> <guillaume.tucker@collabora.com> wrote: > >>> > >>> Hi Linus, > >>> > >>> Please see below, the kernelci.org automated boot bisection found > >>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP > >>> disabled and PREEMPT enabled in multi_v7_defconfig. There is no > >>> log output after the bootloader so the kernel seems to be > >>> crashing very early on. > >> > >> Can you enable DEBUG_LL to capture what goes wrong? > > > > Yes I've scheduled a re-run of the automated bisection with > > DEBUG_LL turned on, but the beagle-xm board is currently offline > > in lab-baylibre. I'll keep you posted when I get some results. > > Unfortunately, enabling the DEBUG_LL config didn't help. There's > still no visible kernel log at all from the failing LAVA jobs: > > http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table > > > This might be a hard one to debug without physical access to the > board, but please let me know if you think there's anything else > I can try. For example, if there's a known older kernel revision > where the combination of SMP=n and PREEMPT=y worked then I could > try to run an automated boot bisection based on that. Is this only beagle-xm, or do we see it on all OMAP3 hardware? -Olof ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-07 16:16 ` Olof Johansson @ 2018-11-08 15:54 ` Tony Lindgren 0 siblings, 0 replies; 9+ messages in thread From: Tony Lindgren @ 2018-11-08 15:54 UTC (permalink / raw) To: linux-arm-kernel * Olof Johansson <olof@lixom.net> [181107 16:17]: > On Wed, Nov 7, 2018 at 8:09 AM Guillaume Tucker > <guillaume.tucker@collabora.com> wrote: > > > > On 06/11/2018 10:54, Guillaume Tucker wrote: > > > On 05/11/2018 14:33, Olof Johansson wrote: > > >> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker > > >> <guillaume.tucker@collabora.com> wrote: > > >>> > > >>> Hi Linus, > > >>> > > >>> Please see below, the kernelci.org automated boot bisection found > > >>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP > > >>> disabled and PREEMPT enabled in multi_v7_defconfig. There is no > > >>> log output after the bootloader so the kernel seems to be > > >>> crashing very early on. > > >> > > >> Can you enable DEBUG_LL to capture what goes wrong? > > > > > > Yes I've scheduled a re-run of the automated bisection with > > > DEBUG_LL turned on, but the beagle-xm board is currently offline > > > in lab-baylibre. I'll keep you posted when I get some results. > > > > Unfortunately, enabling the DEBUG_LL config didn't help. There's > > still no visible kernel log at all from the failing LAVA jobs: > > > > http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table > > > > > > This might be a hard one to debug without physical access to the > > board, but please let me know if you think there's anything else > > I can try. For example, if there's a known older kernel revision > > where the combination of SMP=n and PREEMPT=y worked then I could > > try to run an automated boot bisection based on that. > > Is this only beagle-xm, or do we see it on all OMAP3 hardware? Seems to not happen on three other omap3 boards I've tried with. Regards, Tony ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-07 16:09 ` Guillaume Tucker 2018-11-07 16:16 ` Olof Johansson @ 2018-11-08 22:34 ` Kevin Hilman 2018-11-09 0:32 ` Olof Johansson 2018-11-09 8:52 ` Linus Walleij 1 sibling, 2 replies; 9+ messages in thread From: Kevin Hilman @ 2018-11-08 22:34 UTC (permalink / raw) To: linux-arm-kernel Guillaume Tucker <guillaume.tucker@collabora.com> writes: > On 06/11/2018 10:54, Guillaume Tucker wrote: >> On 05/11/2018 14:33, Olof Johansson wrote: >>> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker >>> <guillaume.tucker@collabora.com> wrote: >>>> >>>> Hi Linus, >>>> >>>> Please see below, the kernelci.org automated boot bisection found >>>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP >>>> disabled and PREEMPT enabled in multi_v7_defconfig. There is no >>>> log output after the bootloader so the kernel seems to be >>>> crashing very early on. >>> >>> Can you enable DEBUG_LL to capture what goes wrong? >> >> Yes I've scheduled a re-run of the automated bisection with >> DEBUG_LL turned on, but the beagle-xm board is currently offline >> in lab-baylibre. I'll keep you posted when I get some results. > > Unfortunately, enabling the DEBUG_LL config didn't help. There's > still no visible kernel log at all from the failing LAVA jobs: > > http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table > > > This might be a hard one to debug without physical access to the > board, but please let me know if you think there's anything else > I can try. For example, if there's a known older kernel revision > where the combination of SMP=n and PREEMPT=y worked then I could > try to run an automated boot bisection based on that. The reason DEBUG_LL doesn't show anything is because it also needs a specific UART enabled for the output. For the case of beagle-xm, it needs CONFIG_DEBUG_OMAP3UART3=y also. It did that locally, and the error on the console is this: Error: invalid dtb and unrecognized/unsupported machine ID r1=0x0000060a, r2=0x00000000 Available machine support: Upon closer inspection, the load addrs are such that it appears the DTB is getting overwritten during zImage compression. So bascially, it's just an (un)lucky fluke that CONFIG_PREEMPT=y changed the kernel size just enough to cause this problem. I did a local test (with pyboot, not LAVA) with all the same load addresses, but without initrd_high and fdt_high, and it boots fine[1] Guillaume, I think the proper solution is to fixup the LAVA device-type for beagle-xm. In particular, it's setting the u-boot variables 'initrd_high' and 'fdt_high' which means it's not letting uboot move them. Looks like that can be overridden in the device-dict using "set base_high_limits = false", so I've done that now to this device[2] Kevin [1] http://termbin.com/9mm6 [2] http://lava.baylibre.com:10080/scheduler/device/omap3-beagle-xm-sea/devicedict ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-08 22:34 ` Kevin Hilman @ 2018-11-09 0:32 ` Olof Johansson 2018-11-09 8:52 ` Linus Walleij 1 sibling, 0 replies; 9+ messages in thread From: Olof Johansson @ 2018-11-09 0:32 UTC (permalink / raw) To: linux-arm-kernel On Thu, Nov 8, 2018 at 2:34 PM Kevin Hilman <khilman@baylibre.com> wrote: > > Guillaume Tucker <guillaume.tucker@collabora.com> writes: > > > On 06/11/2018 10:54, Guillaume Tucker wrote: > >> On 05/11/2018 14:33, Olof Johansson wrote: > >>> On Mon, Nov 5, 2018 at 1:43 AM Guillaume Tucker > >>> <guillaume.tucker@collabora.com> wrote: > >>>> > >>>> Hi Linus, > >>>> > >>>> Please see below, the kernelci.org automated boot bisection found > >>>> that the OMAP3 BeagleBoard-xM is failing to boot with SMP > >>>> disabled and PREEMPT enabled in multi_v7_defconfig. There is no > >>>> log output after the bootloader so the kernel seems to be > >>>> crashing very early on. > >>> > >>> Can you enable DEBUG_LL to capture what goes wrong? > >> > >> Yes I've scheduled a re-run of the automated bisection with > >> DEBUG_LL turned on, but the beagle-xm board is currently offline > >> in lab-baylibre. I'll keep you posted when I get some results. > > > > Unfortunately, enabling the DEBUG_LL config didn't help. There's > > still no visible kernel log at all from the failing LAVA jobs: > > > > http://lava.baylibre.com:10080/scheduler/alljobs?length=25&search=lava-bisect-9638#table > > > > > > This might be a hard one to debug without physical access to the > > board, but please let me know if you think there's anything else > > I can try. For example, if there's a known older kernel revision > > where the combination of SMP=n and PREEMPT=y worked then I could > > try to run an automated boot bisection based on that. > > The reason DEBUG_LL doesn't show anything is because it also needs > a specific UART enabled for the output. For the case of beagle-xm, it > needs CONFIG_DEBUG_OMAP3UART3=y also. > > It did that locally, and the error on the console is this: > > Error: invalid dtb and unrecognized/unsupported machine ID > r1=0x0000060a, r2=0x00000000 > Available machine support: > > Upon closer inspection, the load addrs are such that it appears the DTB > is getting overwritten during zImage compression. > > So bascially, it's just an (un)lucky fluke that CONFIG_PREEMPT=y changed > the kernel size just enough to cause this problem. > > I did a local test (with pyboot, not LAVA) with all the same load > addresses, but without initrd_high and fdt_high, and it boots fine[1] > > Guillaume, I think the proper solution is to fixup the LAVA device-type > for beagle-xm. In particular, it's setting the u-boot variables > 'initrd_high' and 'fdt_high' which means it's not letting uboot move > them. > > Looks like that can be overridden in the device-dict using > "set base_high_limits = false", so I've done that now to this device[2] Argh, I remember hitting these issues a few times through the years. Thanks for finding the problem, Kevin. -Olof ^ permalink raw reply [flat|nested] 9+ messages in thread
* mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm 2018-11-08 22:34 ` Kevin Hilman 2018-11-09 0:32 ` Olof Johansson @ 2018-11-09 8:52 ` Linus Walleij 1 sibling, 0 replies; 9+ messages in thread From: Linus Walleij @ 2018-11-09 8:52 UTC (permalink / raw) To: linux-arm-kernel On Thu, Nov 8, 2018 at 11:34 PM Kevin Hilman <khilman@baylibre.com> wrote: > Upon closer inspection, the load addrs are such that it appears the DTB > is getting overwritten during zImage compression. > > So bascially, it's just an (un)lucky fluke that CONFIG_PREEMPT=y changed > the kernel size just enough to cause this problem. Oh that kind of bug. It has bitten me more than once :( Thanks a lot for rootcausing this Kevin! Yours, Linus Walleij ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2018-11-09 8:52 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <5bdebe88.1c69fb81.3857a.05d1@mx.google.com>
2018-11-05 9:43 ` mainline/master boot bisection: v4.19-12838-g71e56028173b on beagle-xm Guillaume Tucker
2018-11-05 14:33 ` Olof Johansson
2018-11-06 10:54 ` Guillaume Tucker
2018-11-07 16:09 ` Guillaume Tucker
2018-11-07 16:16 ` Olof Johansson
2018-11-08 15:54 ` Tony Lindgren
2018-11-08 22:34 ` Kevin Hilman
2018-11-09 0:32 ` Olof Johansson
2018-11-09 8:52 ` Linus Walleij
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).