Clang/LLVM 'main' regression testing and -Wthread-safety

Clang/LLVM 'main' regression testing and -Wthread-safety

[Marco Elver]

Hi Nathan,

See below - Clang's -Wthread-safety support was merged.

I'm not too familiar with ClangBuiltLinux testing and regression
testing setups; but I wanted to check if LLVM 'main' is being
continuously tested somewhere? I know the Intel test robot takes a
~1-2 week old snapshot of LLVM 'main' and tests that.

Basically I want to ensure that we're able to catch -Wthread-safety
regressions on the Clang side that affect the kernel. While I've been
adding more and more tests to the LLVM repo for the rather special
patterns we decided to use in the kernel to make Context Analysis
work, maybe some slipped through the cracks.

E.g. I've been reviewing
https://github.com/llvm/llvm-project/pull/178952 and tested it, too,
I'm only 99% certain that PR is ok -- being able to quickly catch a
newly introduced issue before waiting for the Intel test robot to
start spamming us would be good.

I recall there was a CI system that did this, but lost track of what's
the latest.

Thanks,
-- Marco

---------- Forwarded message ---------
From: <pr-tracker-bot@kernel.org>
Date: Tue, 10 Feb 2026 at 23:05
Subject: Re: [GIT PULL] locking updates for v6.20
To: Ingo Molnar <mingo@kernel.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
<linux-kernel@vger.kernel.org>, Peter Zijlstra <peterz@infradead.org>,
Thomas Gleixner <tglx@linutronix.de>, Will Deacon <will@kernel.org>,
Waiman Long <longman@redhat.com>, Boqun Feng <boqun.feng@gmail.com>,
Borislav Petkov <bp@alien8.de>, Uros Bizjak <ubizjak@gmail.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>, Gary Guo
<gary@garyguo.net>, Oleg Nesterov <oleg@redhat.com>, Marco Elver
<elver@google.com>


The pull request you sent on Sun, 8 Feb 2026 11:02:19 +0100:

> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git locking-core-2026-02-08

has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/0923fd0419a1a2c8846e15deacac11b619e996d9

Thank you!

--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Miguel Ojeda]

On Wed, Feb 11, 2026 at 11:59 AM Marco Elver <elver@google.com> wrote:
>
> I recall there was a CI system that did this, but lost track of what's
> the latest.

I think you may be referring to
https://github.com/ClangBuiltLinux/continuous-integration2.

By the way, some time ago I asked Aaron (Cc'd) whether we could get
the kernel into upstream LLVM's CI system, i.e. to at least build a
minimal kernel configuration of a fixed, tagged release for one or two
architectures. Would that help you?

(I also suggested Jose (Cc'd) the same, for GCC.)

The idea is that it helps both sides. It serves to catch issues very
early on (including with subtle things like diagnostics), and thus
avoids having to play catch up later to fix them (compiler) and/or
workaround them (kernel) in either side.

To be clear, this wouldn't replace the other CIs that run out of tree,
like Nathan's and the tests I do for Rust stuff etc., i.e. it is not
about catching every compiler issue or testing the kernel as a
product.

For context, this all happened because I asked the same to upstream
Rust and it was a success there. In that case, the goal was to catch
issues early with the unstable features we use, but it helps with
other things too, and thus I think it would help just the same for
Clang and GCC. So far, we only got pinged a few times, so it has not
been a lot of work, and it did catch a couple issues (which sometimes
may end with a preemptive kernel patch or fixing the upstream PR).

We also did it for upstream `bindgen`, and hopefully it will happen
for `rustc_codegen_gcc` (currently we have an out-of-tree one) and GCC
Rust eventually.

Anyway, this is a bit of a tangent, but I thought I could raise it up
again and see what comes out of it...

Cheers,
Miguel

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Nathan Chancellor]

On Wed, Feb 11, 2026 at 12:54:40PM +0100, Miguel Ojeda wrote:
> On Wed, Feb 11, 2026 at 11:59 AM Marco Elver <elver@google.com> wrote:
> >
> > I recall there was a CI system that did this, but lost track of what's
> > the latest.
> 
> I think you may be referring to
> https://github.com/ClangBuiltLinux/continuous-integration2.

Indeed.

> By the way, some time ago I asked Aaron (Cc'd) whether we could get
> the kernel into upstream LLVM's CI system, i.e. to at least build a
> minimal kernel configuration of a fixed, tagged release for one or two
> architectures. Would that help you?
> 
> (I also suggested Jose (Cc'd) the same, for GCC.)
> 
> The idea is that it helps both sides. It serves to catch issues very
> early on (including with subtle things like diagnostics), and thus
> avoids having to play catch up later to fix them (compiler) and/or
> workaround them (kernel) in either side.
> 
> To be clear, this wouldn't replace the other CIs that run out of tree,
> like Nathan's and the tests I do for Rust stuff etc., i.e. it is not
> about catching every compiler issue or testing the kernel as a
> product.
> 
> For context, this all happened because I asked the same to upstream
> Rust and it was a success there. In that case, the goal was to catch
> issues early with the unstable features we use, but it helps with
> other things too, and thus I think it would help just the same for
> Clang and GCC. So far, we only got pinged a few times, so it has not
> been a lot of work, and it did catch a couple issues (which sometimes
> may end with a preemptive kernel patch or fixing the upstream PR).
> 
> We also did it for upstream `bindgen`, and hopefully it will happen
> for `rustc_codegen_gcc` (currently we have an out-of-tree one) and GCC
> Rust eventually.
> 
> Anyway, this is a bit of a tangent, but I thought I could raise it up
> again and see what comes out of it...

It has always been on my backburner to try and get a build of the Linux
kernel as part of LLVM's continuous integration but I have never had the
time to sit down and interface with folks to see what exactly that
entails (I figured it would involve integration with LLVM's build bot
infrastructure but I have not dug in super hard to the documentation).
An arm64 and x86_64 defconfig build would probably catch quite a bit of
initial breakage and it would be easy to pin Linux's version and only
bump it when thoroughly tested.

Cheers,
Nathan

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Aaron Ballman]

On Wed, Feb 11, 2026 at 2:51 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> On Wed, Feb 11, 2026 at 12:54:40PM +0100, Miguel Ojeda wrote:
> > On Wed, Feb 11, 2026 at 11:59 AM Marco Elver <elver@google.com> wrote:
> > >
> > > I recall there was a CI system that did this, but lost track of what's
> > > the latest.
> >
> > I think you may be referring to
> > https://github.com/ClangBuiltLinux/continuous-integration2.
>
> Indeed.
>
> > By the way, some time ago I asked Aaron (Cc'd) whether we could get
> > the kernel into upstream LLVM's CI system, i.e. to at least build a
> > minimal kernel configuration of a fixed, tagged release for one or two
> > architectures. Would that help you?
> >
> > (I also suggested Jose (Cc'd) the same, for GCC.)
> >
> > The idea is that it helps both sides. It serves to catch issues very
> > early on (including with subtle things like diagnostics), and thus
> > avoids having to play catch up later to fix them (compiler) and/or
> > workaround them (kernel) in either side.
> >
> > To be clear, this wouldn't replace the other CIs that run out of tree,
> > like Nathan's and the tests I do for Rust stuff etc., i.e. it is not
> > about catching every compiler issue or testing the kernel as a
> > product.
> >
> > For context, this all happened because I asked the same to upstream
> > Rust and it was a success there. In that case, the goal was to catch
> > issues early with the unstable features we use, but it helps with
> > other things too, and thus I think it would help just the same for
> > Clang and GCC. So far, we only got pinged a few times, so it has not
> > been a lot of work, and it did catch a couple issues (which sometimes
> > may end with a preemptive kernel patch or fixing the upstream PR).
> >
> > We also did it for upstream `bindgen`, and hopefully it will happen
> > for `rustc_codegen_gcc` (currently we have an out-of-tree one) and GCC
> > Rust eventually.
> >
> > Anyway, this is a bit of a tangent, but I thought I could raise it up
> > again and see what comes out of it...
>
> It has always been on my backburner to try and get a build of the Linux
> kernel as part of LLVM's continuous integration but I have never had the
> time to sit down and interface with folks to see what exactly that
> entails (I figured it would involve integration with LLVM's build bot
> infrastructure but I have not dug in super hard to the documentation).
> An arm64 and x86_64 defconfig build would probably catch quite a bit of
> initial breakage and it would be easy to pin Linux's version and only
> bump it when thoroughly tested.

FWIW, I continue to be in full support of adding post-commit CI
resources for testing Linux kernel needs. I'm not super familiar with
the infrastructure side of things, but I'm happy to help make
connections with the folks who do know if it's of help.

~Aaron

>
> Cheers,
> Nathan

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Nathan Chancellor]

On Thu, Feb 12, 2026 at 07:22:26AM -0500, Aaron Ballman wrote:
> On Wed, Feb 11, 2026 at 2:51 PM Nathan Chancellor <nathan@kernel.org> wrote:
> > It has always been on my backburner to try and get a build of the Linux
> > kernel as part of LLVM's continuous integration but I have never had the
> > time to sit down and interface with folks to see what exactly that
> > entails (I figured it would involve integration with LLVM's build bot
> > infrastructure but I have not dug in super hard to the documentation).
> > An arm64 and x86_64 defconfig build would probably catch quite a bit of
> > initial breakage and it would be easy to pin Linux's version and only
> > bump it when thoroughly tested.
> 
> FWIW, I continue to be in full support of adding post-commit CI
> resources for testing Linux kernel needs. I'm not super familiar with
> the infrastructure side of things, but I'm happy to help make
> connections with the folks who do know if it's of help.

Yes, I would greatly appreciate that! I would like to see what it would
entail in full, as I could potentially carve out some time this quarter
or next to actually get it going since it would be beneficial for both
sides.

Cheers,
Nathan

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Miguel Ojeda]

On Thu, Feb 12, 2026 at 1:22 PM Aaron Ballman <aaron@aaronballman.com> wrote:
>
> FWIW, I continue to be in full support of adding post-commit CI
> resources for testing Linux kernel needs. I'm not super familiar with
> the infrastructure side of things, but I'm happy to help make
> connections with the folks who do know if it's of help.

Thanks Aaron. Like Nathan, if you could keep me in the loop, that
would be great!

By the way, the Rust and `bindgen` cases are a pre-merge CI, i.e. PRs
do not get merged if it fails to build. Post-commit is good too, but I
mention it because the Rust one shows it is possible to do even
pre-merge without being a big hassle to compiler developers.

Relatedly, by chance, just a few hours ago there has been a PR in
upstream Rust for a compiler bug fix that only the Linux kernel CI job
exercised -- apparently not a single one of 1832 open source crates
tested exercised the bug. So it shows the kernel is quite a good test
case ;)

Cheers,
Miguel

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Nathan Chancellor]

Hi Marco,

On Wed, Feb 11, 2026 at 11:58:55AM +0100, Marco Elver wrote:
> See below - Clang's -Wthread-safety support was merged.

Congratulations :) I am happy to hear that it has gotten across the
finish line and I look forward to more people trying it out.

> I'm not too familiar with ClangBuiltLinux testing and regression
> testing setups; but I wanted to check if LLVM 'main' is being
> continuously tested somewhere? I know the Intel test robot takes a
> ~1-2 week old snapshot of LLVM 'main' and tests that.

Yes, it is (at the repository that Miguel pointed out):

  https://github.com/ClangBuiltLinux/continuous-integration2/actions/workflows/mainline-clang-23.yml

You can see our full matrix on the ClangBuiltLinux homepage:

  https://ClangBuiltLinux.github.io

We rely on apt.llvm.org for the nightly builds, which are normally
updated every few days but it was down for a little over a month because
of the 22 to 23 transition (if I understand correctly). I actually need
to chase that transition in TuxMake.

I also have a testing framework I use for local validation of LLVM main,
so I should be able to flag issues fairly quickly:

  https://github.com/nathanchance/llvm-kernel-testing

> Basically I want to ensure that we're able to catch -Wthread-safety
> regressions on the Clang side that affect the kernel. While I've been
> adding more and more tests to the LLVM repo for the rather special
> patterns we decided to use in the kernel to make Context Analysis
> work, maybe some slipped through the cracks.
> 
> E.g. I've been reviewing
> https://github.com/llvm/llvm-project/pull/178952 and tested it, too,
> I'm only 99% certain that PR is ok -- being able to quickly catch a
> newly introduced issue before waiting for the Intel test robot to
> start spamming us would be good.

For what it's worth, I have a decent amount of local computing resources
so if you ever want a pull request explicitly tested, ping me on it and
I am happy to put it through my tests, which should flag any major
problems because the context analysis warnings are default on in
Kconfig.

> I recall there was a CI system that did this, but lost track of what's
> the latest.

Aside from our CI and Intel's, I know Linaro's toolchain group has some
testing of LLVM main that seems to be fairly up to date:

  https://lore.kernel.org/llvm/?q=f%3Aci_notify%40linaro.org

Cheers,
Nathan

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Marco Elver]

On Wed, 11 Feb 2026 at 20:40, Nathan Chancellor <nathan@kernel.org> wrote:
[...]
> Yes, it is (at the repository that Miguel pointed out):
>
>   https://github.com/ClangBuiltLinux/continuous-integration2/actions/workflows/mainline-clang-23.yml
>
> You can see our full matrix on the ClangBuiltLinux homepage:
>
>   https://ClangBuiltLinux.github.io

Thanks, that's what I was looking for.

> We rely on apt.llvm.org for the nightly builds, which are normally
> updated every few days but it was down for a little over a month because
> of the 22 to 23 transition (if I understand correctly). I actually need
> to chase that transition in TuxMake.
>
> I also have a testing framework I use for local validation of LLVM main,
> so I should be able to flag issues fairly quickly:
>
>   https://github.com/nathanchance/llvm-kernel-testing
[..]
> For what it's worth, I have a decent amount of local computing resources
> so if you ever want a pull request explicitly tested, ping me on it and
> I am happy to put it through my tests, which should flag any major
> problems because the context analysis warnings are default on in
> Kconfig.

Thanks. I'd have said try
https://github.com/llvm/llvm-project/pull/178952, but I think we don't
have enough CONTEXT_ANALYSIS:=y coverage yet to find anything (I
tested it). Bart found some unrelated crashes, but those have been
fixed I think. So yeah, while that's helpful, if 'main' crashes for
other reasons, it's hard to pinpoint what's wrong.

> > I recall there was a CI system that did this, but lost track of what's
> > the latest.
>
> Aside from our CI and Intel's, I know Linaro's toolchain group has some
> testing of LLVM main that seems to be fairly up to date:
>
>   https://lore.kernel.org/llvm/?q=f%3Aci_notify%40linaro.org

Glad to see. I hope that if there are breakages, those are bisected
and reported to LLVM PRs that introduced the issues?

But in general, I agree that getting some amount of Linux kernel
testing as part of LLVM CI would be helpful to catch regressions in
time and avoid the indirection.

Thanks,
-- Marco

Re: Clang/LLVM 'main' regression testing and -Wthread-safety

[Nathan Chancellor]

On Thu, Feb 12, 2026 at 12:53:24AM +0100, Marco Elver wrote:
> On Wed, 11 Feb 2026 at 20:40, Nathan Chancellor <nathan@kernel.org> wrote:
> > > I recall there was a CI system that did this, but lost track of what's
> > > the latest.
> >
> > Aside from our CI and Intel's, I know Linaro's toolchain group has some
> > testing of LLVM main that seems to be fairly up to date:
> >
> >   https://lore.kernel.org/llvm/?q=f%3Aci_notify%40linaro.org
> 
> Glad to see. I hope that if there are breakages, those are bisected
> and reported to LLVM PRs that introduced the issues?

Yes, at the very least by myself if not others since those reports are
set to the ClangBuiltLinux mailing list.

Cheers,
Nathan