Re: [PATCH 2/3] ci: avoid bare "gcc" for osx-gcc job

Jeff King
2024-05-23T09:11:00Z

On Fri, May 17, 2024 at 09:59:35AM -0700, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
>
> > Yeah. Combining orthogonal properties into a single job lets us cover
> > both (for the common case of success on both) with less CPU. But:
> >
> > - it can sometimes be hard to figure out which of the properties was
> > responsible for a failure. That was the very subject of the thread I
> > referenced earlier, where "linux-gcc" was "use gcc" and also "set
> > lots of knobs".
> >
> > - they might not actually be orthogonal. If you care about checking
> > runtime behavior in the output of two compilers, then that _could_
> > manifest only in the sha256 code. Or as you get into more
> > properties, they may overlap in other ways. I think reftable+sha256
> > is an interesting (eventual) combo to test on top of reftable+sha1.
>
> We could consider permuting, then? If we (for the sake of
> simplicity) had two jobs available, one compiled with GCC and the
> other compiled with clang, we can enumerate other properties
> (e.g. <SHA-1 vs SHA-256>, <reftable vs reffiles>) into pairs, and in
> one run, GCC may be running SHA-1+reffiles while clang is running
> SHA-256+reftable, and in another run, GCC may be running
> SHA-256+reffiles, etc. --- eventually we cover all four combinations
> (admittedly for different commits).

That's a neat idea to get eventual coverage. I have a feeling it would
be a pain in practice, though, because now the CI results aren't quite
deterministic. So if commit X introduces a bug in some combination, we
might not find out until later, and seeing that X passed all tests
doesn't absolve it of responsibility anymore.

Likewise, I often have to re-run CI to get more data, or to see if a
failure is a flake. If it changed what it ran that would be confusing
(though I guess you could use the commit hash as the random "seed" for
deciding which permutation to run).

-Peff