Re: [PATCH] gitlab-ci: split clang-user to avoid timeout

[Richard Henderson <richard.henderson@linaro.org>]

On 11/4/22 09:32, Philippe Mathieu-Daudé wrote:
> +Richard
> 
> On 3/11/22 22:23, Stefan Hajnoczi wrote:
>> GitLab CI times out when the clang-user job takes over 1 hour. Split it
>> into parts that check various architectures.
>>
>> An alternative is to have one job per architecture but that clutters the
>> pipeline view and maybe there is some sharing when multiple targets are
>> built at once.
>>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> ---
>>   .gitlab-ci.d/buildtest-template.yml | 11 +++++++++++
>>   .gitlab-ci.d/buildtest.yml          | 18 +++++++++---------
>>   2 files changed, 20 insertions(+), 9 deletions(-)
>>
>> diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
>> index 73ecfabb8d..38b055e139 100644
>> --- a/.gitlab-ci.d/buildtest-template.yml
>> +++ b/.gitlab-ci.d/buildtest-template.yml
>> @@ -81,3 +81,14 @@
>>       - du -chs ${CI_PROJECT_DIR}/avocado-cache
>>     variables:
>>       QEMU_JOB_AVOCADO: 1
>> +
>> +.clang-user-template:
>> +  extends: .native_build_job_template
>> +  needs:
>> +    job: amd64-debian-user-cross-container
>> +  variables:
>> +    IMAGE: debian-all-test-cross
>> +    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
>> +      --extra-cflags=-fsanitize=undefined
>> +      --extra-cflags=-fno-sanitize-recover=undefined
>> +    MAKE_CHECK_ARGS: check-unit check-tcg
>> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
>> index 6c05c46397..116fce4e8f 100644
>> --- a/.gitlab-ci.d/buildtest.yml
>> +++ b/.gitlab-ci.d/buildtest.yml
>> @@ -323,16 +323,16 @@ clang-system:
>>         ppc-softmmu s390x-softmmu
>>       MAKE_CHECK_ARGS: check-qtest check-tcg
>> -clang-user:
>> -  extends: .native_build_job_template
>> -  needs:
>> -    job: amd64-debian-user-cross-container
>> +# clang-user takes too long so split it into parts
>> +clang-user-part1:
>> +  extends: .clang-user-template
>>     variables:
>> -    IMAGE: debian-all-test-cross
>> -    CONFIGURE_ARGS: --cc=clang --cxx=clang++ --disable-system
>> -      
>> --target-list-exclude=microblazeel-linux-user,aarch64_be-linux-user,i386-linux-user,m68k-linux-user,mipsn32el-linux-user,xtensaeb-linux-user
> 
> We can exclude these targets which are a bit redundant:
> 
> 
> armeb-linux-user,mips64-linux-user,mipsel-linux-user,mipsn32-linux-user,ppc64-linux-user,sh4-linux-user,sparc-linux-user,riscv32-linux-user
> 
> Alternatively, instead of using the exclude pattern, we can switch to including the 
> targets which do have tcg tests:
> 
> $ ls -1 tests/tcg/
> Makefile.target
> README
> aarch64
> alpha
> arm
> cris
> hexagon
> hppa
> i386
> loongarch64
> m68k
> minilib
> mips
> multiarch
> nios2
> openrisc
> ppc
> ppc64
> ppc64le
> riscv64
> s390x
> sh4
> sparc64
> tricore
> x86_64
> xtensa
> 
> Although we have 'multiarch' :/

If we're talking about tests, this runs on debian-all-test-cross, which has fewer 
cross-compilers than that.

However, the main purpose of clang-user is to make sure that stuff *builds* with clang, as 
opposed to gcc, which is where we've seen most problems in the past.  So we do want as 
much coverage across targets/*/ as possible, even if cross-compilers for tests are not 
available.

I agree that we can drop some redundancy, like aarch64_be, armbe, mips{set}, riscv32, 
which have no remarkable difference in linux-user/.  But be careful of e.g. ppc64 vs 
ppc64le and sparc vs sparc64 which have very different ABIs.

Perhaps an interesting split would be those guests supported by debian-all-test-cross, for 
which we build + test, and the others, for which we build only.


r~