changing tests/qtest/meson.build causes unnecessary rebuilding

changing tests/qtest/meson.build causes unnecessary rebuilding

[Peter Maydell]

$ make -C build/arm-clang/ -j8
make: Entering directory
'/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
[1/23] Generating qemu-version.h with a meson_exe.py custom command
make: Leaving directory
'/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
$ touch tests/qtest/meson.build
$ make -C build/arm-clang/ -j8
make: Entering directory
'/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
/usr/bin/ninja  build.ninja && touch build.ninja.stamp
[0/1] Regenerating build files.
The Meson build system
[...]

It then goes on to rebuild hundreds or thousands of files, most of
which are not even in tests/. (Oddly, the exact set of files recompiled
seems to vary from run to run.)

Forcing a full rebuild of all of QEMU seems rather excessive when
the change was likely "add a new test case"...

thanks
-- PMM

Re: changing tests/qtest/meson.build causes unnecessary rebuilding

[Paolo Bonzini]

On 21/01/21 12:56, Peter Maydell wrote:
> $ make -C build/arm-clang/ -j8
> make: Entering directory
> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
> [1/23] Generating qemu-version.h with a meson_exe.py custom command
> make: Leaving directory
> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
> $ touch tests/qtest/meson.build
> $ make -C build/arm-clang/ -j8
> make: Entering directory
> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
> /usr/bin/ninja  build.ninja && touch build.ninja.stamp
> [0/1] Regenerating build files.
> The Meson build system
> [...]
> 
> It then goes on to rebuild hundreds or thousands of files, most of
> which are not even in tests/. (Oddly, the exact set of files recompiled
> seems to vary from run to run.)
> 
> Forcing a full rebuild of all of QEMU seems rather excessive when
> the change was likely "add a new test case"...

This is "avoid build.ninja changes due to order of hash table iteration" 
(https://github.com/mesonbuild/meson/pull/7900/).  I think Meson 0.57 
(with the fix) should be out soon, hopefully before 6.0.

Alternatively you can try to bug your distro to include the patches, 
they are pretty safe.

Paolo

Re: changing tests/qtest/meson.build causes unnecessary rebuilding

[Thomas Huth]

On 21/01/2021 14.12, Paolo Bonzini wrote:
> On 21/01/21 12:56, Peter Maydell wrote:
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> [1/23] Generating qemu-version.h with a meson_exe.py custom command
>> make: Leaving directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> $ touch tests/qtest/meson.build
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> /usr/bin/ninja  build.ninja && touch build.ninja.stamp
>> [0/1] Regenerating build files.
>> The Meson build system
>> [...]
>>
>> It then goes on to rebuild hundreds or thousands of files, most of
>> which are not even in tests/. (Oddly, the exact set of files recompiled
>> seems to vary from run to run.)
>>
>> Forcing a full rebuild of all of QEMU seems rather excessive when
>> the change was likely "add a new test case"...
> 
> This is "avoid build.ninja changes due to order of hash table iteration" 
> (https://github.com/mesonbuild/meson/pull/7900/).  I think Meson 0.57 (with 
> the fix) should be out soon, hopefully before 6.0.
> 
> Alternatively you can try to bug your distro to include the patches, they 
> are pretty safe.

Not sure if it is related, but I noticed that we are also rebuilding a lot 
of files in the gitlab-CI that we did not before the meson conversion, 
especially in the check-system-* jobs, e.g:

https://gitlab.com/qemu-project/qemu/-/jobs/977344949#L366

The check-system-* jobs should normally take the artifacts from the 
build-system-* jobs and thus hardly recompile anything at all.

A part of the problem seems to be that we check out the submodules again, I 
can get rid of the superfluous reconfiguration step by adding something like:

diff -u a/.gitlab-ci.yml b/.gitlab-ci.yml
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -39,6 +39,8 @@ include:
    image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
    script:
      - cd build
+    - touch *
+    - make git-submodule-update
      - find . -type f -exec touch {} +
      - make $MAKE_CHECK_ARGS

... but still, the jobs then recompile almost all files afterwards... could 
that be related to that meson problem, too?

  Thomas

Re: changing tests/qtest/meson.build causes unnecessary rebuilding

[Paolo Bonzini]

On 21/01/21 15:29, Thomas Huth wrote:
> 
> Not sure if it is related, but I noticed that we are also rebuilding a 
> lot of files in the gitlab-CI that we did not before the meson 
> conversion, especially in the check-system-* jobs, e.g:
> 
> https://gitlab.com/qemu-project/qemu/-/jobs/977344949#L366
> 
> The check-system-* jobs should normally take the artifacts from the 
> build-system-* jobs and thus hardly recompile anything at all.
> 
> A part of the problem seems to be that we check out the submodules 
> again, I can get rid of the superfluous reconfiguration step by adding 
> something like:

Yes, there's a

[0/1] Regenerating build files.
/usr/bin/python3: can't open file 
'/builds/qemu-project/qemu/meson/meson.py': [Errno 2] No such file or 
directory

because the submodules have not been checked out, and that causes meson 
to run again.

> diff -u a/.gitlab-ci.yml b/.gitlab-ci.yml
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -39,6 +39,8 @@ include:
>     image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
>     script:
>       - cd build
> +    - touch *
> +    - make git-submodule-update
>       - find . -type f -exec touch {} +
>       - make $MAKE_CHECK_ARGS
> 
> ... but still, the jobs then recompile almost all files afterwards... 
> could that be related to that meson problem, too?

No, I think it's just that, after the new checkout, the source files' 
timestamps should be quite new and cause everything to be rebuilt.

Paolo

Re: changing tests/qtest/meson.build causes unnecessary rebuilding

[Philippe Mathieu-Daudé]

Hi Peter,

On 1/21/21 2:12 PM, Paolo Bonzini wrote:
> On 21/01/21 12:56, Peter Maydell wrote:
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> [1/23] Generating qemu-version.h with a meson_exe.py custom command
>> make: Leaving directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> $ touch tests/qtest/meson.build
>> $ make -C build/arm-clang/ -j8
>> make: Entering directory
>> '/home/petmay01/linaro/qemu-from-laptop/qemu/build/arm-clang'
>> /usr/bin/ninja  build.ninja && touch build.ninja.stamp
>> [0/1] Regenerating build files.
>> The Meson build system
>> [...]
>>
>> It then goes on to rebuild hundreds or thousands of files, most of
>> which are not even in tests/. (Oddly, the exact set of files recompiled
>> seems to vary from run to run.)
>>
>> Forcing a full rebuild of all of QEMU seems rather excessive when
>> the change was likely "add a new test case"...
> 
> This is "avoid build.ninja changes due to order of hash table iteration"
> (https://github.com/mesonbuild/meson/pull/7900/).  I think Meson 0.57
> (with the fix) should be out soon, hopefully before 6.0.
> 
> Alternatively you can try to bug your distro to include the patches,
> they are pretty safe.

Possible kludge rebuilding meson locally:

Paolo's #7900 got merged in commit 1c582a9de:
("Merge pull request #7900 from bonzini/stabilize-hash"),
so a possible kludge is to build and install meson in your
$HOME/.local/bin/ directory with pip as:

  $ python3 -m pip install --user \
      git+https://github.com/mesonbuild/meson.git@1c582a9de

Then either update your $PATH or use:

  $ ./configure --meson=$HOME/.local/bin/meson ...

Regards,

Phil.