Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Lukáš Doktor]

[-- Attachment #1.1.1: Type: text/plain, Size: 1909 bytes --]

Hello Stefan, folks,

the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).

    https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html

Based on the commit message I guess it's expected so take this just as a record of an improvement.

Bisect log:

git bisect start
# good: [aa222a8e4f975284b3f8f131653a4114b3d333b3] Merge tag 'for_upstream' of https://git.kernel.org/pub/scm/virt/kvm/mst/qemu into staging
git bisect good aa222a8e4f975284b3f8f131653a4114b3d333b3
# bad: [ad3387396a71416cacc0b394e5e440dd6e9ba19a] Merge tag 'for-upstream' of https://repo.or.cz/qemu/kevin into staging
git bisect bad ad3387396a71416cacc0b394e5e440dd6e9ba19a
# good: [71438d8dac07f28c01cf6d90fce14efe04c77824] graph-lock: Honour read locks even in the main thread
git bisect good 71438d8dac07f28c01cf6d90fce14efe04c77824
# good: [4b424c757188f7a47630a4d8edcf4ad9f19255bc] scripts: make sure scripts are invoked via $(PYTHON)
git bisect good 4b424c757188f7a47630a4d8edcf4ad9f19255bc
# bad: [80fc5d260002432628710f8b0c7cfc7d9b97bb9d] graph-lock: Disable locking for now
git bisect bad 80fc5d260002432628710f8b0c7cfc7d9b97bb9d
# bad: [6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee] aio-posix: do not nest poll handlers
git bisect bad 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee
# good: [78935fcd88ec2d26d50e45043b262f0326e6d410] iotests/245: Check if 'compress' driver is available
git bisect good 78935fcd88ec2d26d50e45043b262f0326e6d410
# first bad commit: [6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee] aio-posix: do not nest poll handlers

Regards,
Lukáš

PS: The list of perf-ci issues is available here: https://docs.google.com/spreadsheets/d/1HEXI5wDsNgAIgXl5MIhGond898Vz5A1Hrkl0lZmWEbg/edit#gid=0

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Stefan Hajnoczi]

On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>
>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>
> Based on the commit message I guess it's expected so take this just as a record of an improvement.

The commit was not intended to change performance and I'm not sure why
it happens!

Thanks for letting me know.

Stefan

Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Lukáš Doktor]

[-- Attachment #1.1.1: Type: text/plain, Size: 1304 bytes --]

Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>>
>>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>>
>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> 
> The commit was not intended to change performance and I'm not sure why
> it happens!
> 

It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.

From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.

Regards,
Lukáš

> Thanks for letting me know.
> 
> Stefan
> 

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Stefan Hajnoczi]

On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
>
> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> > On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
> >>
> >>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>
> >> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> >
> > The commit was not intended to change performance and I'm not sure why
> > it happens!
> >
>
> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
>
> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.

Thanks!

Stefan

Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Lukáš Doktor]

[-- Attachment #1.1.1: Type: text/plain, Size: 1814 bytes --]

Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>
>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>>>>
>>>>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>>>>
>>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
>>>
>>> The commit was not intended to change performance and I'm not sure why
>>> it happens!
>>>
>>
>> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
>>
>> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
> 

Hello Stefan, folks,

the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:

    https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html

The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.

Regards,
Lukáš

> Thanks!
> 
> Stefan
> 

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Performance improvement with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Stefan Hajnoczi]

On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
>
> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> > On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>
> >> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> >>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
> >>>>
> >>>>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>>>
> >>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> >>>
> >>> The commit was not intended to change performance and I'm not sure why
> >>> it happens!
> >>>
> >>
> >> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
> >>
> >> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
> >
>
> Hello Stefan, folks,
>
> the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
>
>     https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
>
> The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.

Which commit caused the regression?

Stefan

Re: Performance improvement and regression with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Lukáš Doktor]

[-- Attachment #1.1.1: Type: text/plain, Size: 2148 bytes --]

Dne 31. 05. 23 v 19:14 Stefan Hajnoczi napsal(a):
> On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>
>> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
>>> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>>>
>>>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
>>>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
>>>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
>>>>>>
>>>>>>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
>>>>>>
>>>>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
>>>>>
>>>>> The commit was not intended to change performance and I'm not sure why
>>>>> it happens!
>>>>>
>>>>
>>>> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
>>>>
>>>> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
>>>
>>
>> Hello Stefan, folks,
>>
>> the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
>>
>>     https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
>>
>> The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.
> 
> Which commit caused the regression?

Hello Stefan,

the same one that caused the improvement on rotational disks: 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

Lukáš

> 
> Stefan
> 

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 12925 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Performance improvement and regression with 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2406 bytes --]

On Thu, Jun 01, 2023 at 06:23:21AM +0200, Lukáš Doktor wrote:
> Dne 31. 05. 23 v 19:14 Stefan Hajnoczi napsal(a):
> > On Wed, 31 May 2023 at 12:50, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>
> >> Dne 26. 05. 23 v 12:56 Stefan Hajnoczi napsal(a):
> >>> On Fri, 26 May 2023 at 04:07, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>
> >>>> Dne 25. 05. 23 v 17:21 Stefan Hajnoczi napsal(a):
> >>>>> On Thu, 25 May 2023 at 06:18, Lukáš Doktor <ldoktor@redhat.com> wrote:
> >>>>>> the perf-ci detected and bisected the 6d740fb - aio-posix: do not nest poll handlers - as a performance improvement when using multiple concurrent jobs and 4k (22%) as well as 1024k (63%) blocks on aarch64 (on a slow rotational disk).
> >>>>>>
> >>>>>>     https://ldoktor.github.io/tmp/RedHat-virtlab-arm09/v8.0.0/150-improvement.html
> >>>>>>
> >>>>>> Based on the commit message I guess it's expected so take this just as a record of an improvement.
> >>>>>
> >>>>> The commit was not intended to change performance and I'm not sure why
> >>>>> it happens!
> >>>>>
> >>>>
> >>>> It had and today the x86_64 pipeline finished which shows similar improvement just not in read but rather in write instead and only for 4k blocks (~40%). For 1024k blocks I can see it scoring a bit better (~1.5%). Reads are too jittery to really tell anything on that machine. Anyway I have not done any thorough testing, just a bisection with the most significant setting.
> >>>>
> >>>> From around the same time I can see a NVMe regression in 4k writes, but first bisection job showed nothing. I'll increase the range and try again as each job since that day shows similar drop.
> >>>
> >>
> >> Hello Stefan, folks,
> >>
> >> the regression proved to be there and stably reproducible. With NVMe 4k writes with jobs=10 and iodepth=4 I can see a 50% regression on my machine:
> >>
> >>     https://ldoktor.github.io/tmp/RedHat-virtlab722/v8.0.0/150-regression.html
> >>
> >> The rest of the cases doesn't show any change at all. I can provide more data if someone is interested.
> > 
> > Which commit caused the regression?
> 
> Hello Stefan,
> 
> the same one that caused the improvement on rotational disks: 6d740fb01b9f0f5ea7a82f4d5e458d91940a19ee

Thanks. This commit definitely needs more investigation. I didn't expect
it to have a noticeable effect.

I will take a look in July.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]