[Buildroot] Failing tests on gitlab

[Buildroot] Failing tests on gitlab

[Arnout Vandecappelle]

  Hi all,

  The 2021.11-rc1 tag triggered a complete run of all the defconfigs and all the 
tests. 148 out of 725 jobs failed [0].


  There are two main causes for failures:


1. multiple definition of `yylloc' in u-boot or linux, e.g. [1]. This is 
triggered by the update of the docker image which has a new (host) gcc version 
10, which defaults to -fno-common. Since there are more than 100 defconfigs that 
suffer from this problem, I think it would be best to solve this in uboot.mk and 
linux.mk rather than patching all the defconfigs. The fix is rather simple: 
remove the definition "YYLTYPE yylloc" from scripts/dtc/dtc-lexer.l. There's a 
slight complication that older kernels still have a "shipped" version of the 
file, but that can be solved with wildcard matching. This will need a bit of 
testing though to see how it behaves with older kernel/uboot versions, because 
that code *has* changed a couple of times. Also, for the cases where there's a 
shipped version, we should probably take care with the timestamps, to avoid that 
the c file is regenerated.


2. Job timeout, e.g. [2]. I'm not entirely sure what this timeout is. It's 
definitely not the 6h job timeout, because the job hasn't even started. I 
suspect that there's some hidden overall 24h timeout for the entire pipeline, 
and that because we have so many jobs, we're not able to finish all of them 
within that time. The only solution for that is to add additional runners. So, 
if there's anybody out there with spare cycles, please volunteer to add a 
runner! You can contact me or Romain to learn how to set one up.


  Hopefully, we can get better results by -rc2!


  Regards,
  Arnout



[0] https://gitlab.com/buildroot.org/buildroot/-/pipelines/404249740/failures

[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/1758966028

[2] https://gitlab.com/buildroot.org/buildroot/-/jobs/1758966567


_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] Failing tests on gitlab

[Romain Naour]

Hello Arnout,

Le 09/11/2021 à 21:30, Arnout Vandecappelle a écrit :
>  Hi all,
> 
>  The 2021.11-rc1 tag triggered a complete run of all the defconfigs and all the
> tests. 148 out of 725 jobs failed [0].
> 
> 
>  There are two main causes for failures:
> 
> 
> 1. multiple definition of `yylloc' in u-boot or linux, e.g. [1]. This is
> triggered by the update of the docker image which has a new (host) gcc version
> 10, which defaults to -fno-common. Since there are more than 100 defconfigs that
> suffer from this problem, I think it would be best to solve this in uboot.mk and
> linux.mk rather than patching all the defconfigs. The fix is rather simple:
> remove the definition "YYLTYPE yylloc" from scripts/dtc/dtc-lexer.l. There's a
> slight complication that older kernels still have a "shipped" version of the
> file, but that can be solved with wildcard matching. This will need a bit of
> testing though to see how it behaves with older kernel/uboot versions, because
> that code *has* changed a couple of times. Also, for the cases where there's a
> shipped version, we should probably take care with the timestamps, to avoid that
> the c file is regenerated.

Actually this problem already affect users using a recent Linux distribution. We
used for too long the previous docker image based on Debian Stretch snapshot
(20171210).

> 
> 
> 2. Job timeout, e.g. [2]. I'm not entirely sure what this timeout is. It's
> definitely not the 6h job timeout, because the job hasn't even started. I
> suspect that there's some hidden overall 24h timeout for the entire pipeline,
> and that because we have so many jobs, we're not able to finish all of them
> within that time. The only solution for that is to add additional runners. So,
> if there's anybody out there with spare cycles, please volunteer to add a
> runner! You can contact me or Romain to learn how to set one up.

We can also reduce the number of long jobs:
http://patchwork.ozlabs.org/project/buildroot/patch/20211109220328.388872-1-romain.naour@gmail.com/

Building all boards defconfigs are time consuming for the CI and doesn't provide
much benefit. All defconfigs are build every weeks by a scheduled trigged pipeline.

> 
> 
>  Hopefully, we can get better results by -rc2!

Well, except for the gcc 10 issue, the results are already good.
Only one issue left in the runtime testing!

Best regards,
Romain


> 
> 
>  Regards,
>  Arnout
> 
> 
> 
> [0] https://gitlab.com/buildroot.org/buildroot/-/pipelines/404249740/failures
> 
> [1] https://gitlab.com/buildroot.org/buildroot/-/jobs/1758966028
> 
> [2] https://gitlab.com/buildroot.org/buildroot/-/jobs/1758966567
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot