Performance regression between V2_02_177 and V2_02_180

Performance regression between V2_02_177 and V2_02_180

[Ben Sims]

I'm not sure if anybody saw my original mail, there may have been a race condition with me actually joining the mailing list.


I can confirm the performance regression, is caused my async io, turning this off in the lvm.conf gets back my performance.

We work in a libaio intensive environment, so there is contention for libaio ring.


I'm wondering if libaio should be on by default?


Thanks,


Ben


________________________________
From: Ben Sims
Sent: 17 October 2019 15:59
To: lvm-devel@redhat.com
Subject: Performance regression between V2_02_177 and V2_02_180


Hi lvm-devel


I'm trying to analyze a performance regression in short lived commands such as lvchange.


V2_02_177


time /sbin/lvchange -ay /dev/abc/def


real    0m0.085s
user    0m0.000s
sys    0m0.013s


V2_02_180


time /sbin/lvchange -ay /dev/abc/def


real    0m0.115s
user    0m0.007s
sys    0m0.007s


I confirmed these measurements independently with perf over the course of a long test calling lvchange many times. The difference in individual calls is small but adds up.


The reg first occurred due to commit


commit db41fe6c5dab7ff66db9c0568f0e1e1b31657be3
Author: Alasdair G Kergon <agk@redhat.com>
Date:   Mon Jan 22 18:17:58 2018 +0000

    lvmcache: Use asynchronous I/O when scanning devices.


I notice that the new bcache is reading significantly more data 262144 bytes (two blocks of 131072 bytes from /dev/sda ) compared to 28672 bytes in 4k reads made in V2_02_177. Whats the rational for reading so much more data?


The extra overhead of this io estimated with strace does not appear to account for the slow down, but perhaps the processing of this data does?


I'm currently instrumenting the code, but thought i would ask on the mailing list.


Thanks,


Ben







-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20191029/902074bb/attachment.htm>

Performance regression between V2_02_177 and V2_02_180

[David Teigland]

On Tue, Oct 29, 2019 at 08:44:39AM +0000, Ben Sims wrote:
> I can confirm the performance regression, is caused my async io, turning
> this off in the lvm.conf gets back my performance.
> 
> We work in a libaio intensive environment, so there is contention for
> libaio ring.

That's interesting, I've never heard about aio from independent programs
running on the system interfering like that.  Apart from the observation,
have you read about this anywhere so I could understand it better?

> I'm wondering if libaio should be on by default?

It's possible that we should change the use_aio default to 0 if most cases
don't benefit (e.g. small number of PVs), or have contention as you've
seen.

> I notice that the new bcache is reading significantly more data 262144
> bytes (two blocks of 131072 bytes from /dev/sda ) compared to 28672
> bytes in 4k reads made in V2_02_177. Whats the rational for reading so
> much more data?

It seemed to be a reasonable balance between iops and size when testing
various combinations.

That said, the bcache.c io layer is being replaced and renamed soon since
there were some aspects of that code that weren't ideally suited for lvm.
That will be using smaller block sizes.

> The extra overhead of this io estimated with strace does not appear to
> account for the slow down, but perhaps the processing of this data does?

If you eliminate the aio contention you mentioned above, I wouldn't expect
the actual io to be a factor.

> I'm currently instrumenting the code, but thought i would ask on the
> mailing list.

There was a lot of transition churn that's best to avoid while doing test
comparisions.  I suggest testing with 2.02.176 as the old version, and
then using the latest releases from stable-2.02 and master branches.  I
think lvm with bcache was released too early, and there have been
significant fixes and improvements added in both branches.

Dave

Performance regression between V2_02_177 and V2_02_180

[Ben Sims]

Hi David,

Thanks for your detailed response.

I have not observed any contention directly but the comment in the commit

    "ca66d520326493311a3c7132b1bcee0807862301

    io: use sync io if aio fails
    
    io_setup() for aio may fail if a system has reached the
    aio request limit.  In this case, fall back to using sync io..." 

implies that aio is a resource with a limit and open for contention. I observed heavy background aio traffic with perf probes from our virtual disk backends, I don't believe the io_submit calls in LVM are blocking as they are submitted with O_DIRECT and strace time to return is low.

So that was my theory although it's only supported by circumstantial evidence.

If I have the test cycles today I could plot some graphs of performance against background aio activity, and if this shows contention jump in with some kernel probes.

This may off course be academic if lvm moves to io_uring. Have there been any experiments in this area?

Thanks,

Ben





________________________________________
From: David Teigland <teigland@redhat.com>
Sent: 29 October 2019 14:58
To: Ben Sims
Cc: lvm-devel at redhat.com
Subject: Re: [lvm-devel] Performance regression between V2_02_177 and V2_02_180

On Tue, Oct 29, 2019 at 08:44:39AM +0000, Ben Sims wrote:
> I can confirm the performance regression, is caused my async io, turning
> this off in the lvm.conf gets back my performance.
>
> We work in a libaio intensive environment, so there is contention for
> libaio ring.

That's interesting, I've never heard about aio from independent programs
running on the system interfering like that.  Apart from the observation,
have you read about this anywhere so I could understand it better?

> I'm wondering if libaio should be on by default?

It's possible that we should change the use_aio default to 0 if most cases
don't benefit (e.g. small number of PVs), or have contention as you've
seen.

> I notice that the new bcache is reading significantly more data 262144
> bytes (two blocks of 131072 bytes from /dev/sda ) compared to 28672
> bytes in 4k reads made in V2_02_177. Whats the rational for reading so
> much more data?

It seemed to be a reasonable balance between iops and size when testing
various combinations.

That said, the bcache.c io layer is being replaced and renamed soon since
there were some aspects of that code that weren't ideally suited for lvm.
That will be using smaller block sizes.

> The extra overhead of this io estimated with strace does not appear to
> account for the slow down, but perhaps the processing of this data does?

If you eliminate the aio contention you mentioned above, I wouldn't expect
the actual io to be a factor.

> I'm currently instrumenting the code, but thought i would ask on the
> mailing list.

There was a lot of transition churn that's best to avoid while doing test
comparisions.  I suggest testing with 2.02.176 as the old version, and
then using the latest releases from stable-2.02 and master branches.  I
think lvm with bcache was released too early, and there have been
significant fixes and improvements added in both branches.

Dave

Performance regression between V2_02_177 and V2_02_180

[David Teigland]

On Wed, Oct 30, 2019 at 09:34:51AM +0000, Ben Sims wrote:
> I have not observed any contention directly but the comment in the commit
> 
>     "ca66d520326493311a3c7132b1bcee0807862301
> 
>     io: use sync io if aio fails
>     
>     io_setup() for aio may fail if a system has reached the
>     aio request limit.  In this case, fall back to using sync io..." 
> 
> implies that aio is a resource with a limit and open for contention. I
> observed heavy background aio traffic with perf probes from our virtual
> disk backends, I don't believe the io_submit calls in LVM are blocking
> as they are submitted with O_DIRECT and strace time to return is low.

OK, I thought you were referring to contention that would cause lvm's
async io to suffer, as opposed to simply being unavailable.  If other
programs are consuming all of the available aio, then you can increase
/proc/sys/fs/aio-max-nr, which would hopefully make some available for
lvm.

> This may off course be academic if lvm moves to io_uring. Have there
> been any experiments in this area?

No, but it does sound interesting.  I should check if the userspace
library for that has matured.

Dave