* v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output @ 2025-04-02 12:49 Naresh Kamboju 2025-04-03 1:18 ` Nathan Chancellor 0 siblings, 1 reply; 6+ messages in thread From: Naresh Kamboju @ 2025-04-02 12:49 UTC (permalink / raw) To: open list, Linux ARM, lkft-triage, Linux Regressions, clang-built-linux Cc: Arnd Bergmann, Ard Biesheuvel, Dan Carpenter, Anders Roxell Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c the lto-thing, hardening and lto-full config boot failed with toolchain clang-nightly on the mainline master branch with no console output. First seen on the v6.14-12245-g91e5bfe317d8 Good: v6.14-11270-g08733088b566 Bad: v6.14-12245-g91e5bfe317d8 Regressions found on rk3399-rock-pi-4b: - boot/clang-nightly-lkftconfig-kselftest - boot/clang-nightly-lkftconfig-lto-thing - boot/clang-nightly-lkftconfig-hardening - boot/clang-nightly-lkftconfig-lto-full Regressions found on dragonboard-410c: - boot/clang-nightly-lkftconfig-lto-thing - boot/clang-nightly-lkftconfig-lto-full - boot/clang-nightly-lkftconfig-hardening Regressions found on dragonboard-845c: - boot/clang-nightly-lkftconfig-hardening - boot/clang-nightly-lkftconfig-lto-thing Regression Analysis: - New regression? Yes - Reproducibility? Yes Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output Reported-by: Linux Kernel Functional Testing <lkft@linaro.org> ## Boot log Starting kernel ... <No console output> ## Source * Kernel version: 6.14.0 * Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git * Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516 * Git describe: v6.14-12245-g91e5bfe317d8 * Project details: https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/ * Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c) * Toolchains: clang-nightly (Debian clang version 21.0.0 ) * Kconfigs: lto-thing, hardening, lto-full ## Build * Build log: https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/testrun/27859063/suite/boot/test/clang-nightly-lkftconfig-lto-thing/log * Build history: https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/testrun/27859063/suite/boot/test/clang-nightly-lkftconfig-lto-thing/history/ * Build details: https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/testrun/27859063/suite/boot/test/clang-nightly-lkftconfig-lto-thing/details/ * Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/2v9EdkI9AGWWvuT5OzYdgSgXEeH/ * Kernel config: https://storage.tuxsuite.com/public/linaro/lkft/builds/2v9EdkI9AGWWvuT5OzYdgSgXEeH/config -- Linaro LKFT https://lkft.linaro.org ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output 2025-04-02 12:49 v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output Naresh Kamboju @ 2025-04-03 1:18 ` Nathan Chancellor 2025-04-03 10:47 ` Naresh Kamboju 0 siblings, 1 reply; 6+ messages in thread From: Nathan Chancellor @ 2025-04-03 1:18 UTC (permalink / raw) To: Naresh Kamboju Cc: open list, Linux ARM, lkft-triage, Linux Regressions, clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter, Anders Roxell Hi Naresh, On Wed, Apr 02, 2025 at 06:19:19PM +0530, Naresh Kamboju wrote: > Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c > the lto-thing, hardening and lto-full config boot failed with toolchain > clang-nightly on the mainline master branch with no console output. > > First seen on the v6.14-12245-g91e5bfe317d8 > Good: v6.14-11270-g08733088b566 > Bad: v6.14-12245-g91e5bfe317d8 > > Regressions found on rk3399-rock-pi-4b: > - boot/clang-nightly-lkftconfig-kselftest > - boot/clang-nightly-lkftconfig-lto-thing > - boot/clang-nightly-lkftconfig-hardening > - boot/clang-nightly-lkftconfig-lto-full > > Regressions found on dragonboard-410c: > - boot/clang-nightly-lkftconfig-lto-thing > - boot/clang-nightly-lkftconfig-lto-full > - boot/clang-nightly-lkftconfig-hardening > > Regressions found on dragonboard-845c: > - boot/clang-nightly-lkftconfig-hardening > - boot/clang-nightly-lkftconfig-lto-thing > > Regression Analysis: > - New regression? Yes > - Reproducibility? Yes > > Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c > no console output > > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org> > > ## Boot log > Starting kernel > ... > <No console output> > > > ## Source > * Kernel version: 6.14.0 > * Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > * Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516 > * Git describe: v6.14-12245-g91e5bfe317d8 > * Project details: > https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/ > * Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c) > * Toolchains: clang-nightly (Debian clang version 21.0.0 ) The version string for the toolchain seems to be slightly truncated but from the configuration it is: Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357) which can be parsed with our parse-debian-clang.py [1] to get more information: $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)' clang checkout date: 2025-03-30 10:54 UTC (3 days, 14:17:24.134272 ago) clang revision: 3b3d1a5c2614 clang revision link: https://github.com/llvm/llvm-project/commit/3b3d1a5c2614 Our CI is using a slightly newer version: $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)' clang checkout date: 2025-04-01 11:25 UTC (1 day, 13:49:39.394836 ago) clang revision: 290d7b82cb5d clang revision link: https://github.com/llvm/llvm-project/commit/290d7b82cb5d Can you see if it is reproducible with that revision? For what it's worth, both of the arm64 boxes I have can boot a ThinLTO kernel compiled with a version of LLVM @ 749535ba2808e133682074f712ac6829335f8875, so it could be something that was broken for a little bit but Debian happened to sync before the fix was committed. [1]: https://github.com/ClangBuiltLinux/continuous-integration2/blob/fe4844afc1be91d469fc162c8a179f23fafb9384/scripts/parse-debian-clang.py Cheers, Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output 2025-04-03 1:18 ` Nathan Chancellor @ 2025-04-03 10:47 ` Naresh Kamboju 2025-04-05 0:27 ` Nathan Chancellor 0 siblings, 1 reply; 6+ messages in thread From: Naresh Kamboju @ 2025-04-03 10:47 UTC (permalink / raw) To: Nathan Chancellor Cc: open list, Linux ARM, lkft-triage, Linux Regressions, clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter, Anders Roxell On Thu, 3 Apr 2025 at 06:48, Nathan Chancellor <nathan@kernel.org> wrote: > > Hi Naresh, > > On Wed, Apr 02, 2025 at 06:19:19PM +0530, Naresh Kamboju wrote: > > Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c > > the lto-thing, hardening and lto-full config boot failed with toolchain > > clang-nightly on the mainline master branch with no console output. > > > > First seen on the v6.14-12245-g91e5bfe317d8 > > Good: v6.14-11270-g08733088b566 > > Bad: v6.14-12245-g91e5bfe317d8 > > > > Regressions found on rk3399-rock-pi-4b: > > - boot/clang-nightly-lkftconfig-kselftest > > - boot/clang-nightly-lkftconfig-lto-thing > > - boot/clang-nightly-lkftconfig-hardening > > - boot/clang-nightly-lkftconfig-lto-full > > > > Regressions found on dragonboard-410c: > > - boot/clang-nightly-lkftconfig-lto-thing > > - boot/clang-nightly-lkftconfig-lto-full > > - boot/clang-nightly-lkftconfig-hardening > > > > Regressions found on dragonboard-845c: > > - boot/clang-nightly-lkftconfig-hardening > > - boot/clang-nightly-lkftconfig-lto-thing > > > > Regression Analysis: > > - New regression? Yes > > - Reproducibility? Yes > > > > Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c > > no console output > > > > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org> > > > > ## Boot log > > Starting kernel > > ... > > <No console output> > > > > > > ## Source > > * Kernel version: 6.14.0 > > * Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git > > * Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516 > > * Git describe: v6.14-12245-g91e5bfe317d8 > > * Project details: > > https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/ > > * Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c) > > * Toolchains: clang-nightly (Debian clang version 21.0.0 ) > > The version string for the toolchain seems to be slightly truncated but > from the configuration it is: > > Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357) > > which can be parsed with our parse-debian-clang.py [1] to get more > information: > > $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)' > clang checkout date: 2025-03-30 10:54 UTC (3 days, 14:17:24.134272 ago) > clang revision: 3b3d1a5c2614 > clang revision link: https://github.com/llvm/llvm-project/commit/3b3d1a5c2614 > > Our CI is using a slightly newer version: > > $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)' > clang checkout date: 2025-04-01 11:25 UTC (1 day, 13:49:39.394836 ago) > clang revision: 290d7b82cb5d > clang revision link: https://github.com/llvm/llvm-project/commit/290d7b82cb5d > > Can you see if it is reproducible with that revision? I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board with different Clang nightly versions and the latest mainline kernel. The combination using clang-nightly:20250319 successfully booted the Rock Pi 4 board. However, the combination using clang-nightly:20250401 failed to boot the same board. "name": "clang", "version": "21.0.0", "version_full": "Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)" Reference: - https://lkft.validation.linaro.org/scheduler/job/8196258 - https://lkft.validation.linaro.org/scheduler/job/8196275 - Naresh > For what it's > worth, both of the arm64 boxes I have can boot a ThinLTO kernel compiled > with a version of LLVM @ 749535ba2808e133682074f712ac6829335f8875, so it > could be something that was broken for a little bit but Debian happened > to sync before the fix was committed. > > [1]: https://github.com/ClangBuiltLinux/continuous-integration2/blob/fe4844afc1be91d469fc162c8a179f23fafb9384/scripts/parse-debian-clang.py > > Cheers, > Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output 2025-04-03 10:47 ` Naresh Kamboju @ 2025-04-05 0:27 ` Nathan Chancellor 2025-04-10 19:19 ` Naresh Kamboju 0 siblings, 1 reply; 6+ messages in thread From: Nathan Chancellor @ 2025-04-05 0:27 UTC (permalink / raw) To: Naresh Kamboju Cc: open list, Linux ARM, lkft-triage, Linux Regressions, clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter, Anders Roxell On Thu, Apr 03, 2025 at 04:17:44PM +0530, Naresh Kamboju wrote: > I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board > with different Clang nightly versions and the latest mainline kernel. > > The combination using clang-nightly:20250319 successfully booted the > Rock Pi 4 board. > However, the combination using clang-nightly:20250401 failed to boot > the same board. > > "name": "clang", > "version": "21.0.0", > "version_full": "Debian clang version 21.0.0 > (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)" > > Reference: > - https://lkft.validation.linaro.org/scheduler/job/8196258 > - https://lkft.validation.linaro.org/scheduler/job/8196275 Thank you for verifying. Do you test with clang-20 and if so, are tests at that same revision passing? I want to make sure this is an LLVM regression before trying to get into figuring out how to bisect with your setup. Another way to do that is to see if clang-nightly boots are failing on the stable branches, which would point to Linux not being at fault. Cheers, Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output 2025-04-05 0:27 ` Nathan Chancellor @ 2025-04-10 19:19 ` Naresh Kamboju 2025-04-12 1:45 ` Nathan Chancellor 0 siblings, 1 reply; 6+ messages in thread From: Naresh Kamboju @ 2025-04-10 19:19 UTC (permalink / raw) To: Nathan Chancellor Cc: open list, Linux ARM, lkft-triage, Linux Regressions, clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter, Anders Roxell, Kees Cook On Sat, 5 Apr 2025 at 05:58, Nathan Chancellor <nathan@kernel.org> wrote: > > On Thu, Apr 03, 2025 at 04:17:44PM +0530, Naresh Kamboju wrote: > > I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board > > with different Clang nightly versions and the latest mainline kernel. > > > > The combination using clang-nightly:20250319 successfully booted the > > Rock Pi 4 board. > > However, the combination using clang-nightly:20250401 failed to boot > > the same board. > > > > "name": "clang", > > "version": "21.0.0", > > "version_full": "Debian clang version 21.0.0 > > (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)" > > > > Reference: > > - https://lkft.validation.linaro.org/scheduler/job/8196258 > > - https://lkft.validation.linaro.org/scheduler/job/8196275 > > Thank you for verifying. Do you test with clang-20 and if so, are tests > at that same revision passing? I want to make sure this is an LLVM > regression before trying to get into figuring out how to bisect with > your setup. Another way to do that is to see if clang-nightly boots are > failing on the stable branches, which would point to Linux not being at > fault. clang-nightly boot failed with stable-rc 6.12, 6.13 and 6.14 and mainline and next on following devices and virtual platforms. - dragonboard-410c - dragonboard-845c - rk3399-rock-pi-4b - Juno-r2 - e850-96 - FVP (Fast Models Fixed Virtual Platforms) Boot Test links, - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076019/suite/boot/test/clang-nightly-lkftconfig-kselftest/history/ - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076006/suite/boot/test/clang-nightly-lkftconfig-lto-thing/history/ - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076009/suite/boot/test/clang-nightly-lkftconfig-hardening/history/ Lore link, - https://lore.kernel.org/all/CA+G9fYve7+nXJNoV48TksXoMeVjgJuP8Gs=+1br+Qur1DPWV4A@mail.gmail.com/ > > Cheers, > Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output 2025-04-10 19:19 ` Naresh Kamboju @ 2025-04-12 1:45 ` Nathan Chancellor 0 siblings, 0 replies; 6+ messages in thread From: Nathan Chancellor @ 2025-04-12 1:45 UTC (permalink / raw) To: Naresh Kamboju Cc: open list, Linux ARM, lkft-triage, Linux Regressions, clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter, Anders Roxell, Kees Cook Hi Naresh, On Fri, Apr 11, 2025 at 12:49:17AM +0530, Naresh Kamboju wrote: > clang-nightly boot failed with stable-rc 6.12, 6.13 and 6.14 and > mainline and next on following devices and virtual platforms. > - dragonboard-410c > - dragonboard-845c > - rk3399-rock-pi-4b > - Juno-r2 > - e850-96 > - FVP (Fast Models Fixed Virtual Platforms) > > Boot Test links, > - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076019/suite/boot/test/clang-nightly-lkftconfig-kselftest/history/ > - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076006/suite/boot/test/clang-nightly-lkftconfig-lto-thing/history/ > - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076009/suite/boot/test/clang-nightly-lkftconfig-hardening/history/ Thanks again for confirming. As this is reproducible with the FVP, I was able to cobble together how to boot it locally with tuxrun and your configuration (which felt rather convoluted to get from the website above) and use that to bisect LLVM: https://github.com/ClangBuiltLinux/linux/issues/2082 I will continue to look into this next week. Cheers, Nathan ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-04-12 1:45 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-04-02 12:49 v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output Naresh Kamboju 2025-04-03 1:18 ` Nathan Chancellor 2025-04-03 10:47 ` Naresh Kamboju 2025-04-05 0:27 ` Nathan Chancellor 2025-04-10 19:19 ` Naresh Kamboju 2025-04-12 1:45 ` Nathan Chancellor
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).