[PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs

[PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs

[Thomas Huth]

Currently, our check-system-* jobs are recompiling the whole sources
again. This happens due to the fact that the jobs are checking out
the whole source tree and required submodules again, and only try
to use the "build" directory with the binaries and object files as an
artifact from the previous stage - which simply does not work right
anymore (with the current version of meson). Due to some changed
time stamps, meson/ninja are always trying to rebuild the whole tree.

In the long run, we could likely use "meson test --no-rebuild", but
there is still some work going on in that area to improve the user
experience. So until this has been done, simply avoid recompiling the
sources with a trick: pass NINJA=":" to the make process in the test
jobs. Also check out the submodules manually before updating the
timestamps in the build folder, so that the binaries are definitely
newer that all the source files.
This saves ca. 10 - 15 minutes of precious CI cycles in each run.

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 .gitlab-ci.yml | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index de3a3d25b5..16fea25ba9 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -38,9 +38,11 @@ include:
   stage: test
   image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
   script:
+    - scripts/git-submodule.sh update
+        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
     - cd build
     - find . -type f -exec touch {} +
-    - make $MAKE_CHECK_ARGS
+    - make NINJA=":" $MAKE_CHECK_ARGS
 
 .acceptance_template: &acceptance_definition
   cache:
-- 
2.27.0

Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs

[Philippe Mathieu-Daudé]

On 1/26/21 7:57 AM, Thomas Huth wrote:
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files as an
> artifact from the previous stage - which simply does not work right
> anymore (with the current version of meson). Due to some changed
> time stamps, meson/ninja are always trying to rebuild the whole tree.
> 
> In the long run, we could likely use "meson test --no-rebuild", but
> there is still some work going on in that area to improve the user
> experience. So until this has been done, simply avoid recompiling the
> sources with a trick: pass NINJA=":" to the make process in the test
> jobs. Also check out the submodules manually before updating the
> timestamps in the build folder, so that the binaries are definitely
> newer that all the source files.
> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
> 
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  .gitlab-ci.yml | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index de3a3d25b5..16fea25ba9 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -38,9 +38,11 @@ include:
>    stage: test
>    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>    script:
> +    - scripts/git-submodule.sh update
> +        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
>      - cd build
>      - find . -type f -exec touch {} +
> -    - make $MAKE_CHECK_ARGS
> +    - make NINJA=":" $MAKE_CHECK_ARGS

This ninja trick deserves a comment in the YAML file.

Otherwise:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

>  
>  .acceptance_template: &acceptance_definition
>    cache:
> 

Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs

[Thomas Huth]

On 26/01/2021 08.36, Philippe Mathieu-Daudé wrote:
> On 1/26/21 7:57 AM, Thomas Huth wrote:
>> Currently, our check-system-* jobs are recompiling the whole sources
>> again. This happens due to the fact that the jobs are checking out
>> the whole source tree and required submodules again, and only try
>> to use the "build" directory with the binaries and object files as an
>> artifact from the previous stage - which simply does not work right
>> anymore (with the current version of meson). Due to some changed
>> time stamps, meson/ninja are always trying to rebuild the whole tree.
>>
>> In the long run, we could likely use "meson test --no-rebuild", but
>> there is still some work going on in that area to improve the user
>> experience. So until this has been done, simply avoid recompiling the
>> sources with a trick: pass NINJA=":" to the make process in the test
>> jobs. Also check out the submodules manually before updating the
>> timestamps in the build folder, so that the binaries are definitely
>> newer that all the source files.
>> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
>>
>> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   .gitlab-ci.yml | 4 +++-
>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>> index de3a3d25b5..16fea25ba9 100644
>> --- a/.gitlab-ci.yml
>> +++ b/.gitlab-ci.yml
>> @@ -38,9 +38,11 @@ include:
>>     stage: test
>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>     script:
>> +    - scripts/git-submodule.sh update
>> +        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
>>       - cd build
>>       - find . -type f -exec touch {} +
>> -    - make $MAKE_CHECK_ARGS
>> +    - make NINJA=":" $MAKE_CHECK_ARGS
> 
> This ninja trick deserves a comment in the YAML file.

I'll add:

     # Avoid recompiling by hiding ninja with NINJA=":"

Ok?

  Thomas

Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs

[Philippe Mathieu-Daudé]

On 1/26/21 10:08 AM, Thomas Huth wrote:
> On 26/01/2021 08.36, Philippe Mathieu-Daudé wrote:
>> On 1/26/21 7:57 AM, Thomas Huth wrote:
>>> Currently, our check-system-* jobs are recompiling the whole sources
>>> again. This happens due to the fact that the jobs are checking out
>>> the whole source tree and required submodules again, and only try
>>> to use the "build" directory with the binaries and object files as an
>>> artifact from the previous stage - which simply does not work right
>>> anymore (with the current version of meson). Due to some changed
>>> time stamps, meson/ninja are always trying to rebuild the whole tree.
>>>
>>> In the long run, we could likely use "meson test --no-rebuild", but
>>> there is still some work going on in that area to improve the user
>>> experience. So until this has been done, simply avoid recompiling the
>>> sources with a trick: pass NINJA=":" to the make process in the test
>>> jobs. Also check out the submodules manually before updating the
>>> timestamps in the build folder, so that the binaries are definitely
>>> newer that all the source files.
>>> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
>>>
>>> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
>>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>>> ---
>>>   .gitlab-ci.yml | 4 +++-
>>>   1 file changed, 3 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
>>> index de3a3d25b5..16fea25ba9 100644
>>> --- a/.gitlab-ci.yml
>>> +++ b/.gitlab-ci.yml
>>> @@ -38,9 +38,11 @@ include:
>>>     stage: test
>>>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>>>     script:
>>> +    - scripts/git-submodule.sh update
>>> +        $(grep GIT_SUBMODULES build/config-host.mak | sed
>>> 's/GIT_SUBMODULES=//')
>>>       - cd build
>>>       - find . -type f -exec touch {} +
>>> -    - make $MAKE_CHECK_ARGS
>>> +    - make NINJA=":" $MAKE_CHECK_ARGS
>>
>> This ninja trick deserves a comment in the YAML file.
> 
> I'll add:
> 
>     # Avoid recompiling by hiding ninja with NINJA=":"
> 
> Ok?

Perfect, thanks!

Phil.

Re: [PATCH] gitlab-ci.yml: Avoid recompiling the sources in the test jobs

[Wainer dos Santos Moschetta]

On 1/26/21 3:57 AM, Thomas Huth wrote:
> Currently, our check-system-* jobs are recompiling the whole sources
> again. This happens due to the fact that the jobs are checking out
> the whole source tree and required submodules again, and only try
> to use the "build" directory with the binaries and object files as an
> artifact from the previous stage - which simply does not work right
> anymore (with the current version of meson). Due to some changed
> time stamps, meson/ninja are always trying to rebuild the whole tree.
>
> In the long run, we could likely use "meson test --no-rebuild", but
> there is still some work going on in that area to improve the user
> experience. So until this has been done, simply avoid recompiling the
> sources with a trick: pass NINJA=":" to the make process in the test
> jobs. Also check out the submodules manually before updating the
> timestamps in the build folder, so that the binaries are definitely
> newer that all the source files.
> This saves ca. 10 - 15 minutes of precious CI cycles in each run.
Thanks for working on this!
>
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   .gitlab-ci.yml | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)


Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>


>
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index de3a3d25b5..16fea25ba9 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -38,9 +38,11 @@ include:
>     stage: test
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     script:
> +    - scripts/git-submodule.sh update
> +        $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//')
>       - cd build
>       - find . -type f -exec touch {} +
> -    - make $MAKE_CHECK_ARGS
> +    - make NINJA=":" $MAKE_CHECK_ARGS
>   
>   .acceptance_template: &acceptance_definition
>     cache: