Re: [PATCH v2 0/2] objtool: Add option to fail build on vmlinux warnings

Re: [PATCH v2 0/2] objtool: Add option to fail build on vmlinux warnings

[Nathan Chancellor]

Hi Josh,

On Wed, Dec 18, 2024 at 05:00:54PM -0800, Josh Poimboeuf wrote:
...
>   - obscure toolchain bugs in certain compiler versions which do weird
>     things with control flow.  some of these bugs break the kernel, some
>     don't.
> 
> Problem is, it usually falls on the objtool maintainers to figure out
> the root of the problem and the resolution, neither of which is
> necessarily straightforward, especially the latter.  There's only two of
> us maintainers at the moment, with limited bandwidth.
> 
> So yes, it *should* always fail the build.  But unless we get more
> maintainer bandwidth, I don't think we're ready for that.
> 
> We might end up being able to make CONFIG_OBJTOOL_WERROR=y the default,
> and then just require broken features to depend on
> CONFIG_OBJTOOL_WERROR=n.  And then print a big fat warning message at
> build and/or runtime in the case of warnings.
> 
> We also might need to add some features, like a way to mark certain
> compiler versions as bad, or a way to silence objtool warnings for
> certain known harmless cases, or improve the specificity and usefulness
> of certain vague warnings.
> 
> But as a first step I'll planning on just throwing these patches on a
> robot-monitored branch with CONFIG_OBJTOOL_WERROR=y over the holidays to
> see how bad the damage is.

For the record, I plan to monitor these reports for LLVM and try to
investigate and triage all other known objtool warnings for LLVM after
the holidays to try and prepare for this. I felt blind sided by the
compiler -Werror change so I'd rather not go through that again :) one
reason I would like to be objtool clean is to catch changed compiler
behavior quicker, as I tend to notice it is easier to get problems
addressed when the problem is reported as close as possible to the
original change.

I do agree with you that figuring our the root problem and resolution to
some of these warnings is not always the easiest, especially when they
are on the toolchain side, so I have often kicked the can down the road.
I know there is some documentation in objtool.txt around various
warnings, is that pretty up to date/accurate? Are there any other
resources I could look at to help with this work? I know Arnd just
recently fixed a set [1] that I saw in our builds as well due to a bare
unreachable(), which I think tend to hurt Clang more than GCC but maybe
I am imagining things there.

Some objtool reports get sent to only llvm@lists.linux.dev when clang is
involved (due to a historical filter IIRC, I cannot find the original
request), so you may want to glance at [2] to see if anything new pops
up.

[1]: https://git.kernel.org/netdev/net/c/cff865c700711ecc3824b2dfe181637f3ed23c80
[2]: https://lore.kernel.org/llvm/?q=objtool+f:lkp@intel.com

Cheers,
Nathan

Re: [PATCH v2 0/2] objtool: Add option to fail build on vmlinux warnings

[Josh Poimboeuf]

On Thu, Dec 19, 2024 at 03:19:13PM -0700, Nathan Chancellor wrote:
> I do agree with you that figuring our the root problem and resolution to
> some of these warnings is not always the easiest, especially when they
> are on the toolchain side, so I have often kicked the can down the road.
> I know there is some documentation in objtool.txt around various
> warnings, is that pretty up to date/accurate? Are there any other
> resources I could look at to help with this work?

I think the document is pretty up to date.  Some warnings are self
explanatory but others like "unreachable instruction" or "stack state
mismatch" do require digging.

One thing that can help is to "export OBJTOOL_VERBOSE=1", which will
tell objtool to disassemble any affected functions and show a backtrace
with all the taken branches leading up to the warning (if applicable).
Maybe that should be the default for --Werror.

I'd definitely like more people to be able to debug objtool warnings.
Any ideas on making that easier or educating people or improving
warnings are very welcome.  I'll be keeping that in mind when looking at
the build errors over the holidays.

> Some objtool reports get sent to only llvm@lists.linux.dev when clang is
> involved (due to a historical filter IIRC, I cannot find the original
> request), so you may want to glance at [2] to see if anything new pops
> up.

We need to figure out how to get that fixed, the commit author really
needs to know if their code causes a warning/error.

-- 
Josh

Re: [PATCH v2 0/2] objtool: Add option to fail build on vmlinux warnings

[Nick Desaulniers]

On Thu, Dec 19, 2024 at 2:19 PM Nathan Chancellor <nathan@kernel.org> wrote:
>
> Some objtool reports get sent to only llvm@lists.linux.dev when clang is
> involved (due to a historical filter IIRC, I cannot find the original
> request),

https://lore.kernel.org/all/CAKwvOdmJvWpmbP3GyzaZxyiuwooFXA8D7ui05QE7+f8Oaz+rXg@mail.gmail.com/
perhaps?
-- 
Thanks,
~Nick Desaulniers

Re: [PATCH v2 0/2] objtool: Add option to fail build on vmlinux warnings

[Brendan Jackman]

On Thu, 19 Dec 2024 at 22:56, Josh Poimboeuf <jpoimboe@kernel.org> wrote:
> One thing that can help is to "export OBJTOOL_VERBOSE=1", which will
> tell objtool to disassemble any affected functions and show a backtrace
> with all the taken branches leading up to the warning (if applicable).
> Maybe that should be the default for --Werror.

This sounds very useful, I will do this for v3 unless anyone objects.
If we're failing the build the risk of a mega-verbose splat seems like
a good tradeoff vs the risk of the user having to go and read the
objtool code.

Re: [PATCH v2 0/2] objtool: Add option to fail build on vmlinux warnings

[Nathan Chancellor]

On Thu, Dec 19, 2024 at 02:56:42PM -0800, Josh Poimboeuf wrote:
> On Thu, Dec 19, 2024 at 03:19:13PM -0700, Nathan Chancellor wrote:
> > I do agree with you that figuring our the root problem and resolution to
> > some of these warnings is not always the easiest, especially when they
> > are on the toolchain side, so I have often kicked the can down the road.
> > I know there is some documentation in objtool.txt around various
> > warnings, is that pretty up to date/accurate? Are there any other
> > resources I could look at to help with this work?
> 
> I think the document is pretty up to date.  Some warnings are self
> explanatory but others like "unreachable instruction" or "stack state
> mismatch" do require digging.
> 
> One thing that can help is to "export OBJTOOL_VERBOSE=1", which will
> tell objtool to disassemble any affected functions and show a backtrace
> with all the taken branches leading up to the warning (if applicable).
> Maybe that should be the default for --Werror.

Yeah, that does not sound like a bad idea. If the build is going to
break, it seems reasonable to give developers as much pertinent
information as possible so they can address the problem properly and
move on.

> I'd definitely like more people to be able to debug objtool warnings.
> Any ideas on making that easier or educating people or improving
> warnings are very welcome.  I'll be keeping that in mind when looking at
> the build errors over the holidays.

I will definitely ponder on that as well since I should have a good
perspective on that front.

> > Some objtool reports get sent to only llvm@lists.linux.dev when clang is
> > involved (due to a historical filter IIRC, I cannot find the original
> > request), so you may want to glance at [2] to see if anything new pops
> > up.
> 
> We need to figure out how to get that fixed, the commit author really
> needs to know if their code causes a warning/error.

I have started a new thread with the 0day folks to get that adjusted
with you and Peter on CC:

https://lore.kernel.org/20241220200617.GA936171@ax162/

Cheers,
Nathan