Re: s390 private runner CI job timing out

[Christian Borntraeger <borntraeger@linux.ibm.com>]

Am 06.04.23 um 11:21 schrieb Peter Maydell:
> On Thu, 6 Apr 2023 at 07:57, Thomas Huth <thuth@redhat.com> wrote:
>>
>> On 05/04/2023 17.15, Peter Maydell wrote:
>>> The s390 private runner CI job ubuntu-20.04-s390x-all seems to have
>>> started timing out a lot recently. Here's an example where it passed,
>>> but with only 53 seconds left on the clock before it would have been
>>> killed:
>>>
>>> https://gitlab.com/qemu-project/qemu/-/jobs/4066136770
>>>
>>> It looks like 'make check' was about 30 minutes of the 75 minutes
>>> total, and compilation was 45 minutes.
>>>
>>> Any suggestions for how we can trim this down? (Presumably we
>>> could also raise the time limit given that this is a private
>>> runner job...)
>>
>> I don't have access to that system, so I can only guess: Did you check
>> whether there are any other processes still running (leftovers from earlier
>> test runs)?
> 
> I did check for that, as it's been a problem in the past, but
> no, in this case no other jobs are running in the VM.
> 
>> If not, it's maybe because it is a VM that is running with other
>> VMs in parallel that hog the CPU? In that case, you could contact the owner
>> of the machine and ask whether there  is anything that could be done about
>> it. Or simply increase the timeout a little bit more... (our highest timeout
>> in another job is 90 minutes, so I think that would still be OK?).
> 
> Christian, does our S390X machine get a guaranteed amount of CPU,
> or does it depend on what else is running on the hardware?

I think its a shared system with shared CPUs. Can you check the steal
time in top or proc? If this is far too high we could ask to give you
more weight for that VM.