Re: s390 private runner CI job timing out

[Thomas Huth <thuth@redhat.com>]

On 06/04/2023 14.13, Christian Borntraeger wrote:
> 
> 
> Am 06.04.23 um 14:05 schrieb Peter Maydell:
>> On Thu, 6 Apr 2023 at 12:17, Christian Borntraeger
>> <borntraeger@linux.ibm.com> wrote:
>>>
>>> Am 06.04.23 um 12:44 schrieb Peter Maydell:
>>>> On Thu, 6 Apr 2023 at 11:40, Christian Borntraeger
>>>> <borntraeger@linux.ibm.com> wrote:
>>>>> Am 06.04.23 um 11:21 schrieb Peter Maydell:
>>>>>> Christian, does our S390X machine get a guaranteed amount of CPU,
>>>>>> or does it depend on what else is running on the hardware?
>>>>>
>>>>> I think its a shared system with shared CPUs. Can you check the steal
>>>>> time in top or proc? If this is far too high we could ask to give you
>>>>> more weight for that VM.
>>>>
>>>> It's idle at the moment and steal time seems to be low (0.0 .. 0.3);
>>>> I'll try to remember to check next time it's running a job.
>>>>
>>>
>>> Do you have /proc/stat ?
>>
>> Yes; hopefully it means more to you than it does to me :-)
>>
>> linux1@qemu01:~$ cat /proc/stat
>> cpu  60904459 604975 15052194 1435958176 17128179 351949 758578 22218760 0 0
>> cpu0 15022535 146734 3786909 358774818 4283172 98313 237156 5894809 0 0
>> cpu1 15306890 151164 3746024 358968957 4378864 85629 172492 5434255 0 0
>> cpu2 15307709 157180 3762691 359141276 4138714 85736 176367 5474594 0 0
>> cpu3 15267324 149895 3756569 359073124 4327428 82269 172562 5415101 0 0
> 
> This is
> user,nice,system,idle,iowait,irq,softirq,steal,guest,guest_nice
> So overall there is some (20-30% since the last reboot) steal going on.
> Not sure if this is the real problem since it is only Ubuntu 20.04.
> Does a higher timeout make the problem go away?

The thing is: it shouldn't take that long to build QEMU and run the tests 
here, theoretically. Some days ago, the job was finishing in 39 minutes:

  https://gitlab.com/qemu-project/qemu/-/jobs/3973481571

The recent run took 74 minutes:

  https://gitlab.com/qemu-project/qemu/-/jobs/4066136770

That's almost a factor of two! So there is definitely something strange 
going on.

  Thomas