Re: powernv gitlab ci regression

[Daniel Henrique Barboza <danielhb413@gmail.com>]

Hey,

On 12/20/21 18:35, Richard Henderson wrote:
> Hi guys,
> 
> Somewhere within
> 
>> Merge tag 'pull-ppc-20211217' of https://github.com/legoater/qemu into staging
>> ppc 7.0 queue:
>>
>> * General cleanup for Mac machines (Peter)
>> * Fixes for FPU exceptions (Lucas)
>> * Support for new ISA31 instructions (Matheus)
>> * Fixes for ivshmem (Daniel)
>> * Cleanups for PowerNV PHB (Christophe and Cedric)
>> * Updates of PowerNV and pSeries documentation (Leonardo and Daniel)
>> * Fixes for PowerNV (Daniel)
>> * Large cleanup of FPU implementation (Richard)
>> * Removal of SoftTLBs support for PPC74x CPUs (Fabiano)
>> * Fixes for exception models in MPCx and 60x CPUs (Fabiano)
>> * Removal of 401/403 CPUs (Cedric)
>> * Deprecation of taihu machine (Thomas)
>> * Large rework of PPC405 machine (Cedric)
>> * Fixes for VSX instructions (Victor and Matheus)
>> * Fix for e6500 CPU (Fabiano)
>> * Initial support for PMU (Daniel)
> 
> is something that has caused a timeout regression in avocado-system-centos:
> 
>>  (047/171) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_ppc_powernv8:  INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n{'name': '047-tests/avocado/boot_linux_console.py:BootLinuxConsole.test_ppc_powernv8', 'logdir': '/builds/qemu-project/qemu/build/tests/results/job-2021-12-17T19.23-... (90.46 s)
>>  (048/171) tests/avocado/boot_linux_console.py:BootLinuxConsole.test_ppc_powernv9:  INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n{'name': '048-tests/avocado/boot_linux_console.py:BootLinuxConsole.test_ppc_powernv9', 'logdir': '/builds/qemu-project/qemu/build/tests/results/job-2021-12-17T19.23-... (90.55 s)
> 
> See e.g. https://gitlab.com/qemu-project/qemu/-/jobs/1898304074

Thanks for letting us know. I bisected it and the culprit is this patch:


commit 4db3907a40a087e2cc1839d19a3642539d36610b
Author: Daniel Henrique Barboza <danielhb413@gmail.com>
Date:   Fri Dec 17 17:57:18 2021 +0100

     target/ppc: enable PMU instruction count


This is a patch where I added instruction count in the ppc64 PMU. After this patch the
performance of these 2 tests are degraded to the point where we're hitting timeouts in
gitlab (didn't hit timeouts in my machine but the performance is noticeable worse).

I'll need to see the serial console of the VM booting up to evaluate if there's some kernel
module during boot time that is using the PMU and causing the delay. I'll also take a look
into improving the performance as well (e.g. using more TCG code and avoid calling helpers).

It might be the case that the performance gain is enough to make these tests happy again,
although my initial guess is that there's something during boot that is starting the PMU and
leaving it running.


Thanks,


Daniel









> 
> Timeouts are especially tedious with gitlab, because they're not usually consistent, and often go away with a retry.  If I don't see the same failure on my local machine, I often let it go.
> 
> But in this case, the gitlab ci regression has been consistent, not passing a single time since.  Which makes me think this is not just a ci artifact, but that there's a real slowdown.  Could you please have a look?
> 
> 
> r~