Re: [PATCH 2/3] ci: avoid bare "gcc" for osx-gcc job

[Patrick Steinhardt <ps@pks.im>]

[-- Attachment #1: Type: text/plain, Size: 2898 bytes --]

On Thu, May 16, 2024 at 03:19:30AM -0400, Jeff King wrote:
> On Sat, May 11, 2024 at 07:21:28PM +0200, Patrick Steinhardt wrote:
> 
> > On Fri, May 10, 2024 at 03:47:39PM -0700, Junio C Hamano wrote:
> > > Jeff King <peff@peff.net> writes:
> > [snip]
> > > > [1] Another quirk is that we run the whole test suite for both
> > > >     compilers, which is probably overkill. The main value in comparing
> > > >     gcc vs clang is that we don't use any constructs that the compiler
> > > >     complains about. It's _possible_ for there to be a construct that
> > > >     the compiler does not notice but which causes a runtime difference
> > > >     (say, undefined behavior which happens to work out on one compiler),
> > > >     but I think we're again hitting diminishing returns.
> > > 
> > > Yeah, that is a very good point.
> > 
> > On Linux, we have the "pedantic" job that runs Fedora and only compiles
> > the sources with DEVOPTS=pedantic without running any of the tests. We
> > could do the same on macOS.
> 
> Yeah, I think the infrastructure is there (looks like just resetting
> $run_tests). We probably could stand to use it in more places. E.g., is
> there even value in running the tests for linux-gcc and linux-clang?
> It's _possible_ for there to be a run-time difference in the compiler
> outputs, but we may be hitting diminishing returns. The main value I
> think is just seeing what the compilers complain about.

That's certainly the biggest part, yeah. But I have been hitting lots of
compiler-dependent behaviour. This is mostly in the area of bugs though,
where for example toolchain A may initialize variables on the stack to
all zeroes while toolchain B does not.

I guess this is mostly a question of defaults though, and I think it is
partially influenced by the overall toolchain environment as configured
by my distro. Especially hardening options are for example quite likely
to lead to different behaviour.

I'm not sure whether this is sufficient reason on its own to warrant
testing with several toolchains. But we can easily combine this with
additional tuning knobs. Two separate test jobs with GCC and Clang are
comparatively boring. But if we make it GCC+SHA1 and Clang+SHA256 then
it becomes more interesting.

So I think dropping the compiler coverage completely is rather pointless
because we already run multiple different jobs per platform anyway. But
we should investigate whether we can cleverly combine those so that we
do not need a separate jobs just to test a specific compiler.

Patrick

> But I dunno. This thread argues there is value in running the tests with
> the separate compiler:
> 
>   https://lore.kernel.org/git/pull.266.git.gitgitgadget@gmail.com/
> 
> which I guess would argue for doing the same for osx-clang and osx-gcc
> (if the latter continues to exist).
> 
> -Peff

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]