Re: Proposal for a regular upstream performance testing

["Lukáš Doktor" <ldoktor@redhat.com>]

Dne 26. 11. 20 v 11:17 Peter Maydell napsal(a):
> On Thu, 26 Nov 2020 at 08:13, Lukáš Doktor <ldoktor@redhat.com> wrote:
>> The goal of this initiative is to detect system-wide performance
>> regressions as well as improvements early, ideally pin-point the
>> individual commits and notify people that they should fix things.
>> All in upstream and ideally with least human interaction possible.
> 
> So, my general view on this is that automated testing of performance
> is nice, but unless it also comes with people willing to do the
> work of identifying and fixing the causes of performance regressions
> there's a risk that it degrades into another automated email or set
> of graphs that nobody pays much attention to (outside of the obvious
> "oops this commit dropped us by 50%" mistakes). As with fuzz testing,
> I'm a bit wary of setting up an automated system that just pushes
> work onto humans who already have full plates. Is RedHat also
> planning to set performance requirements and assign engineers to
> keep us from falling below them ?
> 
> thanks
> -- PMM
> 

In the proposed "solution 1" my role would be to maintain, judge and help analyzing the reports if needed. As for the fixing the code I can not serve, that would have to be on the individual contributors/maintainers, the best I can do is to bisect and CC the authors.

In "solution 2" that would be on the other volunteer with my assistance, if needed. Note that currently the pipeline is not that clever so it requires manual interaction for bisection but I do have plan on improving that soon.

Also note that the purpose of this email is also a call for ideas, because maybe there is a better tool out there and if it fitted our needs better I wouldn't mind to switching to it.

Regards,
Lukáš