Re: [PATCH v1] gitlab-ci.yml: Speed up CI by using "meson test --no-rebuild"

[Thomas Huth <thuth@redhat.com>]

On 25/01/2021 10.51, Paolo Bonzini wrote:
> On 25/01/21 10:03, Thomas Huth wrote:
>> Currently, our check-system-* jobs are recompiling the whole sources
>> again. This happens due to the fact that the jobs are checking out
>> the whole source tree and required submodules again, and only try
>> to use the "build" directory with the binaries and object files
>> as an artifact from the previous stage - which simply does not work
>> anymore (with the current version of meson). Due to some changed
>> time stamps, meson is always trying to rebuild the whole tree.
>>
>> To fix this problem, we can use "meson test --no-rebuild" instead of
>> make check" to avoid rebuilding all binaries every time. However, the
>> iotests ("make check-block") are not run by "meson test", so we have
>> to execute these manually now. But instead of adding them to the same
>> job as "meson test", it's better to put them into a separate new job
>> instead, to keep things nicely running in parallel in the CI.
>> This saves ca. 15 - 20 minutes of precious CI cycles in each run.
> 
> The reason why we're not using "meson test" is that the reporting is (still) 
> inferior to what you get from "make check", especially with respect to which 
> tests are failing.  This is being tracked at 
> https://github.com/mesonbuild/meson/issues/7830 and the last missing bits 
> are at https://github.com/mesonbuild/meson/issues/8200 (after which we'll 
> change the "meson test" command line to also include "meson test --verbose").
> 
> However, for CI this is a minor issue because we can let GitLab parse the 
> XML testing logs.  Can you add something like this to the test jobs for v2?
> 
> +  artifacts:
> +    when: always
> +    paths:
> +      - build/meson-logs/
> +    reports:
> +      junit: build/meson-logs/testlog.junit.xml

Ok, I've tried that but it also worked not quite as expected:

https://gitlab.com/huth/qemu/-/pipelines/246463068/test_report

The "check-*" jobs now show up in the test report page, but even though I've 
made some tests failing (e.g. the check-system-centos job), the failing jobs 
are showing up with "Failed: 0" there. Also the duration is always marked 
with 0.00ms.

> Another possibility could be to use "make check NINJA=:".  I am not sure if 
> that works, but if it does it would be the smallest possible workaround.

It works! So since all the other options seem to have other drawbacks right 
now, I guess that's the best way to go until we completely switch to "meson 
test". Thus I've just sent a patch for that.

  Thomas