Re: [Buildroot] [PATCH v2 RFC] support/scripts: don't build board defconfigs with Gitlab's pipelines trigged on tag

[Romain Naour <romain.naour@gmail.com>]

Hello Thomas, Yann, All,

Le 11/11/2021 à 11:15, Yann E. MORIN a écrit :
> Thomas, All,
> 
> On 2021-11-11 11:02 +0100, Thomas Petazzoni spake thusly:
>> On Thu, 11 Nov 2021 09:25:48 +0100
>> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
>>> On 2021-11-09 23:03 +0100, Romain Naour spake thusly:
>>>> In order to reduce the number of long jobs, don't build board
>>>> defconfigs with pipelines trigged on tag, keeping only the runtime
>>>> tests and the Qemu's defconfigs.
>>> Applied to master, thanks.
>> I hadn't had the chance to chime in on this one,
> 
> Sorry, I could have waited a bit more indeed...
> 
>> but I don't entirely
>> agree with the change (I also don't entirely disagree!). Indeed, I
>> believe we do want to have the results of the defconfigs builds on
>> tags. If we missed something, and one of our release has defconfigs
>> broken, we definitely want to know it and fix it.
> 
> On principle, I do agree with that, and that's what I initially coded in
> that script.
> 
>> This disabling of defconfig builds is only done because there is an
>> infrastructure issue. But really, from a proper QA/CI point of view, we
>> really want to build the defconfigs on tags I think.
> 
> Again, agreed on principle.

Indeed, Ideally defconfigs should be tested on release.

But each defconfig spend most of the time to build the internal toolchain maybe
another solution is to use an prebuilt toolchain. Still building a kernel takes
some time.

Since most of defconfigs jobs are executed before the testsuite jobs, it delay
too much the testsuite results.

> 
> But as Romain explained, that creates too much jobs when a tag is cut,
> so much so that a lot of them do not even stand a chance to even start.

See again what happened yesterday when Peter did two Buildroot releases at the
same time (2021.02.7 and 2021.08.2). The pipeline for the LTS 2021.02.7 didn't
even started yet and will probably reach the 24h timeout.

> 
> In that case, the CI is not usefull at all either.
> 
> On the other hand, the defconfigs are already built reasonably often,
> every week, with a scheduled pipeline. It is thus pretty quick to notice
> a build failure.
> 
> So, my opinion is that I prefer a partial CI that is stable and
> dependable, where each failure is an actual failure, rather than a
> full-range CI that has a very low SNR, where people eventually stop
> caring about failure reports because they are often spurious failures.

Since I care about the gitlab-ci, I restarted a lot of jobs that failed due to
timeout but this is an annoying work...

> 
> Of course, we can reverse course on that commit; after all, that's what
> git-revert is for! ;-)

I would be happy to revert this patch :)

> 
> The better course of action, obviously, would be to have an infra that
> is strong enough to absorb all that load when we push a tag, but that's
> an order of magnitude more complex to achieve.
> 
> One thing, though: maybe some of the runners could be configured to
> accept more jobs at once, so even if each job then takes longer to run,
> we start them faster and thus do not hit the 24-hour blockage limit.

I'm not sure if this solution can scale. We still add some new board defconfig :)

Best regards,
Romain


> 
> Regards,
> Yann E. MORIN.
> 

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot