can we reduce the time taken for the msys2 jobs?

can we reduce the time taken for the msys2 jobs?

[Peter Maydell]

The msys2-32bit job currently seems to run into the 70 minute CI timeout
quite frequently. This successful pass took 61 minutes:
https://gitlab.com/qemu-project/qemu/-/jobs/3479763757
so I don't think it's a "time limit is fine but job has intermittent
hang" situation.

msys2-64bit also is quite close to the timeout, eg this job
https://gitlab.com/qemu-project/qemu/-/jobs/3482192864
took 61 minutes.

Can we cut down or split up these CI jobs?

thanks
-- PMM

Re: can we reduce the time taken for the msys2 jobs?

[Philippe Mathieu-Daudé]

On 16/12/22 15:55, Peter Maydell wrote:
> The msys2-32bit job currently seems to run into the 70 minute CI timeout
> quite frequently. This successful pass took 61 minutes:
> https://gitlab.com/qemu-project/qemu/-/jobs/3479763757
> so I don't think it's a "time limit is fine but job has intermittent
> hang" situation.
> 
> msys2-64bit also is quite close to the timeout, eg this job
> https://gitlab.com/qemu-project/qemu/-/jobs/3482192864
> took 61 minutes.

The 64-bit variant is building WHPX.

> Can we cut down or split up these CI jobs?

We can add --disable-tools to the slower.


What I don't understand is why the docker image is rebuilt, it should
be pulled from the registry.

Re: can we reduce the time taken for the msys2 jobs?

[Thomas Huth]

On 16/12/2022 17.23, Philippe Mathieu-Daudé wrote:
> On 16/12/22 15:55, Peter Maydell wrote:
>> The msys2-32bit job currently seems to run into the 70 minute CI timeout
>> quite frequently. This successful pass took 61 minutes:
>> https://gitlab.com/qemu-project/qemu/-/jobs/3479763757
>> so I don't think it's a "time limit is fine but job has intermittent
>> hang" situation.
>>
>> msys2-64bit also is quite close to the timeout, eg this job
>> https://gitlab.com/qemu-project/qemu/-/jobs/3482192864
>> took 61 minutes.
> 
> The 64-bit variant is building WHPX.
> 
>> Can we cut down or split up these CI jobs?

The Windows jobs are terribly slow, yes, and we've discussed that a couple 
of times before with no satisfying solution, e.g.:

 
https://lore.kernel.org/qemu-devel/e80726cc-633d-435c-7a2a-5cae43624282@redhat.com/

... so we currently settled on not running the qtests in the 64-bit job 
since that would take too long.

I also don't have a real good idea how to improve the situation ... we could 
switch to even simpler targets (e.g. s390x-softmmu should compile faster and 
run less tests than ppc64-softmmu, I think), or would it be still OK to bump 
the timeout to 80 minutes here? (90 minutes, like it has been discussed last 
year is already very borderline, but I think 80 minutes might still be OK, 
especially since those jobs don't wait for another job from the container 
stage?)

> We can add --disable-tools to the slower.

For the 32-bit job, this would further decrease the test scope ... should be 
OK as a last resort, I guess.

> What I don't understand is why the docker image is rebuilt, it should
> be pulled from the registry.

I think those Windows jobs are still quite a bit special ... e.g. until some 
months ago, the "timeout" setting was also not working at all IIRC.

  Thomas

Re: can we reduce the time taken for the msys2 jobs?

[Bin Meng]

On Tue, Jan 3, 2023 at 6:40 PM Thomas Huth <thuth@redhat.com> wrote:
>
> On 16/12/2022 17.23, Philippe Mathieu-Daudé wrote:
> > On 16/12/22 15:55, Peter Maydell wrote:
> >> The msys2-32bit job currently seems to run into the 70 minute CI timeout
> >> quite frequently. This successful pass took 61 minutes:
> >> https://gitlab.com/qemu-project/qemu/-/jobs/3479763757
> >> so I don't think it's a "time limit is fine but job has intermittent
> >> hang" situation.
> >>
> >> msys2-64bit also is quite close to the timeout, eg this job
> >> https://gitlab.com/qemu-project/qemu/-/jobs/3482192864
> >> took 61 minutes.
> >
> > The 64-bit variant is building WHPX.
> >
> >> Can we cut down or split up these CI jobs?
>
> The Windows jobs are terribly slow, yes, and we've discussed that a couple
> of times before with no satisfying solution, e.g.:
>
>
> https://lore.kernel.org/qemu-devel/e80726cc-633d-435c-7a2a-5cae43624282@redhat.com/
>
> ... so we currently settled on not running the qtests in the 64-bit job
> since that would take too long.
>
> I also don't have a real good idea how to improve the situation ... we could
> switch to even simpler targets (e.g. s390x-softmmu should compile faster and
> run less tests than ppc64-softmmu, I think), or would it be still OK to bump
> the timeout to 80 minutes here? (90 minutes, like it has been discussed last
> year is already very borderline, but I think 80 minutes might still be OK,
> especially since those jobs don't wait for another job from the container
> stage?)
>
> > We can add --disable-tools to the slower.
>
> For the 32-bit job, this would further decrease the test scope ... should be
> OK as a last resort, I guess.
>
> > What I don't understand is why the docker image is rebuilt, it should
> > be pulled from the registry.
>
> I think those Windows jobs are still quite a bit special ... e.g. until some
> months ago, the "timeout" setting was also not working at all IIRC.
>

I am not sure if:

1. Create a Windows docker container build job to include all MSYS2
build prerequisites
2. Change MSYS2 job to depend on the Windows docker job, and remove
all the build prerequisites preparation

would save us some time?

Regards,
Bin