* v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output
@ 2025-04-02 12:49 Naresh Kamboju
2025-04-03 1:18 ` Nathan Chancellor
0 siblings, 1 reply; 6+ messages in thread
From: Naresh Kamboju @ 2025-04-02 12:49 UTC (permalink / raw)
To: open list, Linux ARM, lkft-triage, Linux Regressions,
clang-built-linux
Cc: Arnd Bergmann, Ard Biesheuvel, Dan Carpenter, Anders Roxell
Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c
the lto-thing, hardening and lto-full config boot failed with toolchain
clang-nightly on the mainline master branch with no console output.
First seen on the v6.14-12245-g91e5bfe317d8
Good: v6.14-11270-g08733088b566
Bad: v6.14-12245-g91e5bfe317d8
Regressions found on rk3399-rock-pi-4b:
- boot/clang-nightly-lkftconfig-kselftest
- boot/clang-nightly-lkftconfig-lto-thing
- boot/clang-nightly-lkftconfig-hardening
- boot/clang-nightly-lkftconfig-lto-full
Regressions found on dragonboard-410c:
- boot/clang-nightly-lkftconfig-lto-thing
- boot/clang-nightly-lkftconfig-lto-full
- boot/clang-nightly-lkftconfig-hardening
Regressions found on dragonboard-845c:
- boot/clang-nightly-lkftconfig-hardening
- boot/clang-nightly-lkftconfig-lto-thing
Regression Analysis:
- New regression? Yes
- Reproducibility? Yes
Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c
no console output
Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
## Boot log
Starting kernel
...
<No console output>
## Source
* Kernel version: 6.14.0
* Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
* Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516
* Git describe: v6.14-12245-g91e5bfe317d8
* Project details:
https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/
* Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c)
* Toolchains: clang-nightly (Debian clang version 21.0.0 )
* Kconfigs: lto-thing, hardening, lto-full
## Build
* Build log: https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/testrun/27859063/suite/boot/test/clang-nightly-lkftconfig-lto-thing/log
* Build history:
https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/testrun/27859063/suite/boot/test/clang-nightly-lkftconfig-lto-thing/history/
* Build details:
https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/testrun/27859063/suite/boot/test/clang-nightly-lkftconfig-lto-thing/details/
* Build link: https://storage.tuxsuite.com/public/linaro/lkft/builds/2v9EdkI9AGWWvuT5OzYdgSgXEeH/
* Kernel config:
https://storage.tuxsuite.com/public/linaro/lkft/builds/2v9EdkI9AGWWvuT5OzYdgSgXEeH/config
--
Linaro LKFT
https://lkft.linaro.org
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output
2025-04-02 12:49 v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output Naresh Kamboju
@ 2025-04-03 1:18 ` Nathan Chancellor
2025-04-03 10:47 ` Naresh Kamboju
0 siblings, 1 reply; 6+ messages in thread
From: Nathan Chancellor @ 2025-04-03 1:18 UTC (permalink / raw)
To: Naresh Kamboju
Cc: open list, Linux ARM, lkft-triage, Linux Regressions,
clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter,
Anders Roxell
Hi Naresh,
On Wed, Apr 02, 2025 at 06:19:19PM +0530, Naresh Kamboju wrote:
> Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c
> the lto-thing, hardening and lto-full config boot failed with toolchain
> clang-nightly on the mainline master branch with no console output.
>
> First seen on the v6.14-12245-g91e5bfe317d8
> Good: v6.14-11270-g08733088b566
> Bad: v6.14-12245-g91e5bfe317d8
>
> Regressions found on rk3399-rock-pi-4b:
> - boot/clang-nightly-lkftconfig-kselftest
> - boot/clang-nightly-lkftconfig-lto-thing
> - boot/clang-nightly-lkftconfig-hardening
> - boot/clang-nightly-lkftconfig-lto-full
>
> Regressions found on dragonboard-410c:
> - boot/clang-nightly-lkftconfig-lto-thing
> - boot/clang-nightly-lkftconfig-lto-full
> - boot/clang-nightly-lkftconfig-hardening
>
> Regressions found on dragonboard-845c:
> - boot/clang-nightly-lkftconfig-hardening
> - boot/clang-nightly-lkftconfig-lto-thing
>
> Regression Analysis:
> - New regression? Yes
> - Reproducibility? Yes
>
> Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c
> no console output
>
> Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
>
> ## Boot log
> Starting kernel
> ...
> <No console output>
>
>
> ## Source
> * Kernel version: 6.14.0
> * Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> * Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516
> * Git describe: v6.14-12245-g91e5bfe317d8
> * Project details:
> https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/
> * Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c)
> * Toolchains: clang-nightly (Debian clang version 21.0.0 )
The version string for the toolchain seems to be slightly truncated but
from the configuration it is:
Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)
which can be parsed with our parse-debian-clang.py [1] to get more
information:
$ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)'
clang checkout date: 2025-03-30 10:54 UTC (3 days, 14:17:24.134272 ago)
clang revision: 3b3d1a5c2614
clang revision link: https://github.com/llvm/llvm-project/commit/3b3d1a5c2614
Our CI is using a slightly newer version:
$ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)'
clang checkout date: 2025-04-01 11:25 UTC (1 day, 13:49:39.394836 ago)
clang revision: 290d7b82cb5d
clang revision link: https://github.com/llvm/llvm-project/commit/290d7b82cb5d
Can you see if it is reproducible with that revision? For what it's
worth, both of the arm64 boxes I have can boot a ThinLTO kernel compiled
with a version of LLVM @ 749535ba2808e133682074f712ac6829335f8875, so it
could be something that was broken for a little bit but Debian happened
to sync before the fix was committed.
[1]: https://github.com/ClangBuiltLinux/continuous-integration2/blob/fe4844afc1be91d469fc162c8a179f23fafb9384/scripts/parse-debian-clang.py
Cheers,
Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output
2025-04-03 1:18 ` Nathan Chancellor
@ 2025-04-03 10:47 ` Naresh Kamboju
2025-04-05 0:27 ` Nathan Chancellor
0 siblings, 1 reply; 6+ messages in thread
From: Naresh Kamboju @ 2025-04-03 10:47 UTC (permalink / raw)
To: Nathan Chancellor
Cc: open list, Linux ARM, lkft-triage, Linux Regressions,
clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter,
Anders Roxell
On Thu, 3 Apr 2025 at 06:48, Nathan Chancellor <nathan@kernel.org> wrote:
>
> Hi Naresh,
>
> On Wed, Apr 02, 2025 at 06:19:19PM +0530, Naresh Kamboju wrote:
> > Regressions on rk3399-rock-pi-4b, dragonboard-410c and dragonboard-845c
> > the lto-thing, hardening and lto-full config boot failed with toolchain
> > clang-nightly on the mainline master branch with no console output.
> >
> > First seen on the v6.14-12245-g91e5bfe317d8
> > Good: v6.14-11270-g08733088b566
> > Bad: v6.14-12245-g91e5bfe317d8
> >
> > Regressions found on rk3399-rock-pi-4b:
> > - boot/clang-nightly-lkftconfig-kselftest
> > - boot/clang-nightly-lkftconfig-lto-thing
> > - boot/clang-nightly-lkftconfig-hardening
> > - boot/clang-nightly-lkftconfig-lto-full
> >
> > Regressions found on dragonboard-410c:
> > - boot/clang-nightly-lkftconfig-lto-thing
> > - boot/clang-nightly-lkftconfig-lto-full
> > - boot/clang-nightly-lkftconfig-hardening
> >
> > Regressions found on dragonboard-845c:
> > - boot/clang-nightly-lkftconfig-hardening
> > - boot/clang-nightly-lkftconfig-lto-thing
> >
> > Regression Analysis:
> > - New regression? Yes
> > - Reproducibility? Yes
> >
> > Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c
> > no console output
> >
> > Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
> >
> > ## Boot log
> > Starting kernel
> > ...
> > <No console output>
> >
> >
> > ## Source
> > * Kernel version: 6.14.0
> > * Git tree: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> > * Git sha: 91e5bfe317d8f8471fbaa3e70cf66cae1314a516
> > * Git describe: v6.14-12245-g91e5bfe317d8
> > * Project details:
> > https://qa-reports.linaro.org/lkft/linux-mainline-master/build/v6.14-12245-g91e5bfe317d8/
> > * Architectures: arm64 (rk3399-rock-pi-4b, dragonboard-410c, dragonboard-845c)
> > * Toolchains: clang-nightly (Debian clang version 21.0.0 )
>
> The version string for the toolchain seems to be slightly truncated but
> from the configuration it is:
>
> Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)
>
> which can be parsed with our parse-debian-clang.py [1] to get more
> information:
>
> $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250330105456+3b3d1a5c2614-1~exp1~20250330225508.1357)'
> clang checkout date: 2025-03-30 10:54 UTC (3 days, 14:17:24.134272 ago)
> clang revision: 3b3d1a5c2614
> clang revision link: https://github.com/llvm/llvm-project/commit/3b3d1a5c2614
>
> Our CI is using a slightly newer version:
>
> $ scripts/parse-debian-clang.py -p -v 'Debian clang version 21.0.0 (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)'
> clang checkout date: 2025-04-01 11:25 UTC (1 day, 13:49:39.394836 ago)
> clang revision: 290d7b82cb5d
> clang revision link: https://github.com/llvm/llvm-project/commit/290d7b82cb5d
>
> Can you see if it is reproducible with that revision?
I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board
with different Clang nightly versions and the latest mainline kernel.
The combination using clang-nightly:20250319 successfully booted the
Rock Pi 4 board.
However, the combination using clang-nightly:20250401 failed to boot
the same board.
"name": "clang",
"version": "21.0.0",
"version_full": "Debian clang version 21.0.0
(++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)"
Reference:
- https://lkft.validation.linaro.org/scheduler/job/8196258
- https://lkft.validation.linaro.org/scheduler/job/8196275
- Naresh
> For what it's
> worth, both of the arm64 boxes I have can boot a ThinLTO kernel compiled
> with a version of LLVM @ 749535ba2808e133682074f712ac6829335f8875, so it
> could be something that was broken for a little bit but Debian happened
> to sync before the fix was committed.
>
> [1]: https://github.com/ClangBuiltLinux/continuous-integration2/blob/fe4844afc1be91d469fc162c8a179f23fafb9384/scripts/parse-debian-clang.py
>
> Cheers,
> Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output
2025-04-03 10:47 ` Naresh Kamboju
@ 2025-04-05 0:27 ` Nathan Chancellor
2025-04-10 19:19 ` Naresh Kamboju
0 siblings, 1 reply; 6+ messages in thread
From: Nathan Chancellor @ 2025-04-05 0:27 UTC (permalink / raw)
To: Naresh Kamboju
Cc: open list, Linux ARM, lkft-triage, Linux Regressions,
clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter,
Anders Roxell
On Thu, Apr 03, 2025 at 04:17:44PM +0530, Naresh Kamboju wrote:
> I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board
> with different Clang nightly versions and the latest mainline kernel.
>
> The combination using clang-nightly:20250319 successfully booted the
> Rock Pi 4 board.
> However, the combination using clang-nightly:20250401 failed to boot
> the same board.
>
> "name": "clang",
> "version": "21.0.0",
> "version_full": "Debian clang version 21.0.0
> (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)"
>
> Reference:
> - https://lkft.validation.linaro.org/scheduler/job/8196258
> - https://lkft.validation.linaro.org/scheduler/job/8196275
Thank you for verifying. Do you test with clang-20 and if so, are tests
at that same revision passing? I want to make sure this is an LLVM
regression before trying to get into figuring out how to bisect with
your setup. Another way to do that is to see if clang-nightly boots are
failing on the stable branches, which would point to Linux not being at
fault.
Cheers,
Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output
2025-04-05 0:27 ` Nathan Chancellor
@ 2025-04-10 19:19 ` Naresh Kamboju
2025-04-12 1:45 ` Nathan Chancellor
0 siblings, 1 reply; 6+ messages in thread
From: Naresh Kamboju @ 2025-04-10 19:19 UTC (permalink / raw)
To: Nathan Chancellor
Cc: open list, Linux ARM, lkft-triage, Linux Regressions,
clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter,
Anders Roxell, Kees Cook
On Sat, 5 Apr 2025 at 05:58, Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Thu, Apr 03, 2025 at 04:17:44PM +0530, Naresh Kamboju wrote:
> > I’ve re-run the tests to validate the boot behavior on the Rock Pi 4 board
> > with different Clang nightly versions and the latest mainline kernel.
> >
> > The combination using clang-nightly:20250319 successfully booted the
> > Rock Pi 4 board.
> > However, the combination using clang-nightly:20250401 failed to boot
> > the same board.
> >
> > "name": "clang",
> > "version": "21.0.0",
> > "version_full": "Debian clang version 21.0.0
> > (++20250401112529+290d7b82cb5d-1~exp1~20250401112547.1360)"
> >
> > Reference:
> > - https://lkft.validation.linaro.org/scheduler/job/8196258
> > - https://lkft.validation.linaro.org/scheduler/job/8196275
>
> Thank you for verifying. Do you test with clang-20 and if so, are tests
> at that same revision passing? I want to make sure this is an LLVM
> regression before trying to get into figuring out how to bisect with
> your setup. Another way to do that is to see if clang-nightly boots are
> failing on the stable branches, which would point to Linux not being at
> fault.
clang-nightly boot failed with stable-rc 6.12, 6.13 and 6.14 and
mainline and next on following devices and virtual platforms.
- dragonboard-410c
- dragonboard-845c
- rk3399-rock-pi-4b
- Juno-r2
- e850-96
- FVP (Fast Models Fixed Virtual Platforms)
Boot Test links,
- https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076019/suite/boot/test/clang-nightly-lkftconfig-kselftest/history/
- https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076006/suite/boot/test/clang-nightly-lkftconfig-lto-thing/history/
- https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076009/suite/boot/test/clang-nightly-lkftconfig-hardening/history/
Lore link,
- https://lore.kernel.org/all/CA+G9fYve7+nXJNoV48TksXoMeVjgJuP8Gs=+1br+Qur1DPWV4A@mail.gmail.com/
>
> Cheers,
> Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output
2025-04-10 19:19 ` Naresh Kamboju
@ 2025-04-12 1:45 ` Nathan Chancellor
0 siblings, 0 replies; 6+ messages in thread
From: Nathan Chancellor @ 2025-04-12 1:45 UTC (permalink / raw)
To: Naresh Kamboju
Cc: open list, Linux ARM, lkft-triage, Linux Regressions,
clang-built-linux, Arnd Bergmann, Ard Biesheuvel, Dan Carpenter,
Anders Roxell, Kees Cook
Hi Naresh,
On Fri, Apr 11, 2025 at 12:49:17AM +0530, Naresh Kamboju wrote:
> clang-nightly boot failed with stable-rc 6.12, 6.13 and 6.14 and
> mainline and next on following devices and virtual platforms.
> - dragonboard-410c
> - dragonboard-845c
> - rk3399-rock-pi-4b
> - Juno-r2
> - e850-96
> - FVP (Fast Models Fixed Virtual Platforms)
>
> Boot Test links,
> - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076019/suite/boot/test/clang-nightly-lkftconfig-kselftest/history/
> - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076006/suite/boot/test/clang-nightly-lkftconfig-lto-thing/history/
> - https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-6.12.y/build/v6.12.23-2-g8e0839d16957/testrun/28076009/suite/boot/test/clang-nightly-lkftconfig-hardening/history/
Thanks again for confirming. As this is reproducible with the FVP, I was
able to cobble together how to boot it locally with tuxrun and your
configuration (which felt rather convoluted to get from the website
above) and use that to bisect LLVM:
https://github.com/ClangBuiltLinux/linux/issues/2082
I will continue to look into this next week.
Cheers,
Nathan
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2025-04-12 1:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-04-02 12:49 v6.14-12245-g91e5bfe317d8: Boot regression: rk3399-rock-pi-4b dragonboard-410c dragonboard-845c no console output Naresh Kamboju
2025-04-03 1:18 ` Nathan Chancellor
2025-04-03 10:47 ` Naresh Kamboju
2025-04-05 0:27 ` Nathan Chancellor
2025-04-10 19:19 ` Naresh Kamboju
2025-04-12 1:45 ` Nathan Chancellor
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).