Re: Performance improvement and regression with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]

On Thu, Jun 01, 2023 at 06:23:21AM +0200, Lukáš Doktor wrote:
> Dne 31. 05. 23 v 19:14 Stefan Hajnoczi napsal(a):
> > On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>
> >> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> >>> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>
> >>>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> >>>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
> >>>>>>
> >>>>>>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>>>>>
> >>>>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> >>>>>
> >>>>> The commit was not intended to change performance and I'm not sure why
> >>>>> it happens!
> >>>>>
> >>>>
> >>>> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
> >>>>
> >>>> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
> >>>
> >>
> >> Hello Stefan, folks,
> >>
> >> the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
> >>
> >>     https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
> >>
> >> The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.
> > 
> > Which commit caused the regression?
> 
> Hello Stefan,
> 
> the same one that caused the improvement on rotational disks: 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

Thanks. This commit definitely needs more investigation. I didn't expect
it to have a noticeable effect.

I will take a look in July.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]