Re: [PATCH v3 22/24] gitlab: move the majority of artefact handling to a template

["Alex Bennée" <alex.bennee@linaro.org>]

Philippe Mathieu-Daudé <philmd@linaro.org> writes:

> On 1/3/23 11:00, Alex Bennée wrote:
>> Philippe Mathieu-Daudé <philmd@linaro.org> writes:
>> 
>>> On 1/3/23 07:51, Thomas Huth wrote:
>>>> On 28/02/2023 22.41, Philippe Mathieu-Daudé wrote:
>>>>> On 28/2/23 20:06, Alex Bennée wrote:
>>>>>> To avoid lots of copy and paste lets deal with artefacts in a
>>>>>> template. This way we can filter out most of the pre-binary object and
>>>>>> library files we no longer need as we have the final binaries.
>>>>>>
>>>>>> build-system-alpine also saved .git-submodule-status so for simplicity
>>>>>> we bring that into the template as well.
>>>>>>
>>>>>> As an example the build-system-ubuntu artefacts before this patch
>>>>>> where around 1.3 GB, after dropping the object files it comes to 970
>>>>>> MB.
>>>>>>
>>>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>>>> ---
>>>>>>    .gitlab-ci.d/buildtest-template.yml | 16 ++++++
>>>>>>    .gitlab-ci.d/buildtest.yml          | 81 +++++++++++------------------
>>>>>>    2 files changed, 46 insertions(+), 51 deletions(-)
>>>>>
>>>>> This is still kludging the fact that 'make check-qtest' rebuild
>>>>> the world even if QEMU binaries are present.
>>>> Did you try? ... if so, that would kill even more CI minutes, i.e.
>>>> it would IMHO be a show-stopper for this patch.
>>>
>>> I remember we had to pass all build/ otherwise calling 'make
>>> check-qtest check-tcg' would rebuild the same binaries, it it
>>> was pointless to split the jobs in 2 stages. I might have missed
>>> when that was fixed.
>>>
>>> I haven't tried, however I see in Alex job:
>>>
>>> $ scripts/git-submodule.sh update $(sed -n '/GIT_SUBMODULES=/ s/.*=//
>>> p' build/config-host.mak)
>>> sed: can't read build/config-host.mak: No such file or directory
>>> $ cd build
>>> /bin/bash: line 144: cd: build: No such file or directory
>>> ERROR: Job failed: exit code 1
>>>
>>> https://gitlab.com/stsquad/qemu/-/jobs/3847747769
>> That was an older run of the job where I thought the skip path would
>> merge from the template. In the end I've just include git-submodules in
>> all the build aretfacts instead of just for alpine.
>> It ran fine:
>>    https://gitlab.com/stsquad/qemu/-/jobs/3849512854
>
> Indeed:
> https://gitlab.com/stsquad/qemu/-/jobs/3849512800/artifacts/browse/build/
>
> So a good improvement, but I'd prefer having a stricter "only include
> what we need" policy rather than "exclude what we don't". The former
> will catch missing additions, while the later won't notice until someone
> spend time excluding pointless files manually again.

I'd argue the opposite because its hard to come up with patterns for all
the binaries we build but easy for all the object files.

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro