[RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Petr Vorel]

Hi all,

LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed
down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These
tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while
16 other processes reads the buffer and check only zero is there.

Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
(Non default parameter creates much smaller file, thus the change is not that
obvious).

Because the slowdown has been here for few kernel releases I suppose nobody
complained and the test is somehow artificial (nobody uses this in a real world).
But still it'd be good to double check the problem. I can bisect a particular
commit.

Because 2 filesystems affected, could be "Improve asynchronous iomap DIO
performance" [4] block layer change somehow related?

Kind regards,
Petr

[1] https://github.com/linux-test-project/ltp/tree/master/testcases/kernel/io/ltp-aiodio/aiodio_sparse.c
[2] https://github.com/linux-test-project/ltp/tree/master/testcases/kernel/io/ltp-aiodio/dio_sparse.c
[3] https://github.com/linux-test-project/ltp/tree/master/testcases/kernel/io/ltp-aiodio/common.h
[4] https://kernelnewbies.org/Linux_6.6#Block_layer

Re: [RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Jens Axboe]

On 7/19/24 11:43 AM, Petr Vorel wrote:
> Hi all,
> 
> LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed
> down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These
> tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while
> 16 other processes reads the buffer and check only zero is there.
> 
> Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
> same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
> (Non default parameter creates much smaller file, thus the change is not that
> obvious).
> 
> Because the slowdown has been here for few kernel releases I suppose nobody
> complained and the test is somehow artificial (nobody uses this in a real world).
> But still it'd be good to double check the problem. I can bisect a particular
> commit.
> 
> Because 2 filesystems affected, could be "Improve asynchronous iomap DIO
> performance" [4] block layer change somehow related?

No, because that got disabled before release for unrelated reasons. Why
don't you just bisect it, since you have a simple test case?

-- 
Jens Axboe

Re: [RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Petr Vorel]

Hi Jens, all,

> On 7/19/24 11:43 AM, Petr Vorel wrote:
> > Hi all,

> > LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed
> > down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These
> > tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while
> > 16 other processes reads the buffer and check only zero is there.

> > Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
> > same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
> > (Non default parameter creates much smaller file, thus the change is not that
> > obvious).

> > Because the slowdown has been here for few kernel releases I suppose nobody
> > complained and the test is somehow artificial (nobody uses this in a real world).
> > But still it'd be good to double check the problem. I can bisect a particular
> > commit.

> > Because 2 filesystems affected, could be "Improve asynchronous iomap DIO
> > performance" [4] block layer change somehow related?

> No, because that got disabled before release for unrelated reasons. Why
> don't you just bisect it, since you have a simple test case?

Jens, thanks for info. Sure, I'll bisect next week and report.

The reason I reported before bisecting is because it wouldn't be the first time
the test was "artificial" and therefore reported problem was not fixed. If it's
a real problem I would expect it would be also caught by other people or even by
fstests.

Kind regards,
Petr

Re: [RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Jens Axboe]

On 7/19/24 2:13 PM, Petr Vorel wrote:
> Hi Jens, all,
> 
>> On 7/19/24 11:43 AM, Petr Vorel wrote:
>>> Hi all,
> 
>>> LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3]) slowed
>>> down on kernel 6.6 on Btrfs and XFS, when run with default parameters. These
>>> tests create 100 MB sparse file and write zeros (using libaio or O_DIRECT) while
>>> 16 other processes reads the buffer and check only zero is there.
> 
>>> Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
>>> same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
>>> (Non default parameter creates much smaller file, thus the change is not that
>>> obvious).
> 
>>> Because the slowdown has been here for few kernel releases I suppose nobody
>>> complained and the test is somehow artificial (nobody uses this in a real world).
>>> But still it'd be good to double check the problem. I can bisect a particular
>>> commit.
> 
>>> Because 2 filesystems affected, could be "Improve asynchronous iomap DIO
>>> performance" [4] block layer change somehow related?
> 
>> No, because that got disabled before release for unrelated reasons. Why
>> don't you just bisect it, since you have a simple test case?
> 
> Jens, thanks for info. Sure, I'll bisect next week and report.
> 
> The reason I reported before bisecting is because it wouldn't be the
> first time the test was "artificial" and therefore reported problem
> was not fixed. If it's a real problem I would expect it would be also
> caught by other people or even by fstests.

Didn't look at the test cases, so yeah may very well be bogus. But it
also may not... And a bisection may help shine some light on that too,
outside of just highlighting what commit made it slower.

-- 
Jens Axboe

Re: [RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Jan Kara]

Hi!

On Fri 19-07-24 19:43:25, Petr Vorel wrote:
> LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3])
> slowed down on kernel 6.6 on Btrfs and XFS, when run with default
> parameters. These tests create 100 MB sparse file and write zeros (using
> libaio or O_DIRECT) while 16 other processes reads the buffer and check
> only zero is there.

So the performance of this test is irrelevant because combining buffered
reads with direct IO writes was always in "better don't do it" territory.
Definitely not if you care about perfomance.

> Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
> same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
> (Non default parameter creates much smaller file, thus the change is not that
> obvious).

But still it's kind of curious what caused the 9x slow down. So I'd be
curious to know the result of the bisection :). Thanks for report!

								Honza
-- 
Jan Kara <jack@suse.com>
SUSE Labs, CR

Re: [RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Petr Vorel]

Hi Jan, all,

> Hi!

> On Fri 19-07-24 19:43:25, Petr Vorel wrote:
> > LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3])
> > slowed down on kernel 6.6 on Btrfs and XFS, when run with default
> > parameters. These tests create 100 MB sparse file and write zeros (using
> > libaio or O_DIRECT) while 16 other processes reads the buffer and check
> > only zero is there.

> So the performance of this test is irrelevant because combining buffered
> reads with direct IO writes was always in "better don't do it" territory.
> Definitely not if you care about perfomance.

Thanks a lot for having a look, Jan!

> > Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
> > same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
> > (Non default parameter creates much smaller file, thus the change is not that
> > obvious).

> But still it's kind of curious what caused the 9x slow down. So I'd be
> curious to know the result of the bisection :). Thanks for report!

I'm already working on it, report soon.

Kind regards,
Petr

> 								Honza

Re: [RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Petr Vorel]

Hi all,


[ Cc Peter and Mike ]
> Hi!

> On Fri 19-07-24 19:43:25, Petr Vorel wrote:
> > LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3])
> > slowed down on kernel 6.6 on Btrfs and XFS, when run with default
> > parameters. These tests create 100 MB sparse file and write zeros (using
> > libaio or O_DIRECT) while 16 other processes reads the buffer and check
> > only zero is there.

> So the performance of this test is irrelevant because combining buffered
> reads with direct IO writes was always in "better don't do it" territory.
> Definitely not if you care about perfomance.

> > Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
> > same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
> > (Non default parameter creates much smaller file, thus the change is not that
> > obvious).

> But still it's kind of curious what caused the 9x slow down. So I'd be
> curious to know the result of the bisection :). Thanks for report!

It looks to be the slowdown was introduced by commit 63304558ba5d
("sched/eevdf: Curb wakeup-preemption") [1] from v6.6-rc1.

I also compiled current next (next-20240724), it's also slow  and reverting
commit from it returns the original speed on both Btrfs and XFS.

Kind regards,
Petr

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=63304558ba5dcaaff9e052ee43cfdcc7f9c29e85

> 								Honza

Re: [RFC] Slow down of LTP tests aiodio_sparse.c and dio_sparse.c in kernel 6.6

[Mike Galbraith]

On Wed, 2024-07-24 at 15:18 +0200, Petr Vorel wrote:
> Hi all,
>
>
> [ Cc Peter and Mike ]
> > Hi!
>
> > On Fri 19-07-24 19:43:25, Petr Vorel wrote:
> > > LTP AIO DIO tests aiodio_sparse.c [1] and dio_sparse.c [2] (using [3])
> > > slowed down on kernel 6.6 on Btrfs and XFS, when run with default
> > > parameters. These tests create 100 MB sparse file and write zeros (using
> > > libaio or O_DIRECT) while 16 other processes reads the buffer and check
> > > only zero is there.
>
> > So the performance of this test is irrelevant because combining buffered
> > reads with direct IO writes was always in "better don't do it" territory.
> > Definitely not if you care about perfomance.
>
> > > Runtime of this particular setup (i.e. 100 MB file) on Btrfs and XFS on the
> > > same system slowed down 9x (6.5: ~1 min 6.6: ~9 min). Ext4 is not affected.
> > > (Non default parameter creates much smaller file, thus the change is not that
> > > obvious).
>
> > But still it's kind of curious what caused the 9x slow down. So I'd be
> > curious to know the result of the bisection :). Thanks for report!
>
> It looks to be the slowdown was introduced by commit 63304558ba5d
> ("sched/eevdf: Curb wakeup-preemption") [1] from v6.6-rc1.

That's not good, I'd rather not have over-zealous preemption back, my
box doesn't miss that one tiny bit.

Hrm... have you tried Peter's eevdf queue?  The DELAY_DEQUEUE feature
therein does good things for tbench, restoring pre-eevdf throughput.

	-Mike