[LTP] [PATCH v2] io: fix really slow dio_sparse on certain systems

[LTP] [PATCH v2] io: fix really slow dio_sparse on certain systems

[Andrea Cervesato]

From: Andrea Cervesato <andrea.cervesato@suse.com>

The reason why dio_sparse is happening to be slow on certain systems is
that, if data buffering is slow, we run more buffered read() for one
single dio write(). This slows down the whole test, because for each
read() we always need to move data from kernel space to user space.

Create a READ_BUFFER_SIZE variable inside the common.h header and use
it to define the write buffer size.

Signed-off-by: Andrea Cervesato <andrea.cervesato@suse.com>
---
Changes in v2:
- fix write option default value in the documentation
- Link to v1: https://lore.kernel.org/r/20260123-fix_dio_sparse_slow-v1-1-65bd0dc0d52f@suse.com
---
 testcases/kernel/io/ltp-aiodio/common.h     | 6 ++++--
 testcases/kernel/io/ltp-aiodio/dio_sparse.c | 4 ++--
 2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/testcases/kernel/io/ltp-aiodio/common.h b/testcases/kernel/io/ltp-aiodio/common.h
index 9a2d2716661651adf40c635ac516cc70068c9393..d3f05a3c1e75cd4baa629cbce1872729a0d641c6 100644
--- a/testcases/kernel/io/ltp-aiodio/common.h
+++ b/testcases/kernel/io/ltp-aiodio/common.h
@@ -9,6 +9,8 @@
 #include <stdlib.h>
 #include "tst_test.h"
 
+#define READ_BUFFER_SIZE 4096
+
 static inline char *check_zero(char *buf, int size)
 {
 	char *p;
@@ -57,7 +59,7 @@ static inline void io_append(const char *path, char pattern, int flags, size_t b
 
 static inline void io_read(const char *filename, int filesize, volatile int *run_child)
 {
-	char buff[4096];
+	char buff[READ_BUFFER_SIZE];
 	int fd;
 	int i;
 	int r;
@@ -102,7 +104,7 @@ exit:
 
 static inline void io_read_eof(const char *filename, volatile int *run_child)
 {
-	char buff[4096];
+	char buff[READ_BUFFER_SIZE];
 	int fd;
 	int r;
 
diff --git a/testcases/kernel/io/ltp-aiodio/dio_sparse.c b/testcases/kernel/io/ltp-aiodio/dio_sparse.c
index c87e5ab1aca7c8910f76ee87f86b561f548d4cdf..4edf286420831cef5f4280e620651554182aaa52 100644
--- a/testcases/kernel/io/ltp-aiodio/dio_sparse.c
+++ b/testcases/kernel/io/ltp-aiodio/dio_sparse.c
@@ -32,7 +32,7 @@ static char *str_filesize;
 static char *str_offset;
 
 static int numchildren = 16;
-static long long writesize = 1024;
+static long long writesize = READ_BUFFER_SIZE;
 static long long filesize = 100 * 1024 * 1024;
 static long long offset = 0;
 static long long alignment;
@@ -124,7 +124,7 @@ static struct tst_test test = {
 	.forks_child = 1,
 	.options = (struct tst_option[]) {
 		{"n:", &str_numchildren, "Number of threads (default 16)"},
-		{"w:", &str_writesize, "Size of writing blocks (default 1K)"},
+		{"w:", &str_writesize, "Size of writing blocks (default 4K)"},
 		{"s:", &str_filesize, "Size of file (default 100M)"},
 		{"o:", &str_offset, "File offset (default 0)"},
 		{}

---
base-commit: c47ab95d5fe077133e598ac7762387450a87cb2b
change-id: 20260123-fix_dio_sparse_slow-925958767d74

Best regards,
-- 
Andrea Cervesato <andrea.cervesato@suse.com>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] io: fix really slow dio_sparse on certain systems

[Petr Vorel]

Hi Andrea,

> Changes in v2:
> - fix write option default value in the documentation
> - Link to v1: https://lore.kernel.org/r/20260123-fix_dio_sparse_slow-v1-1-65bd0dc0d52f@suse.com

Reviewed-by: Petr Vorel <pvorel@suse.cz>

Thanks!

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] io: fix really slow dio_sparse on certain systems

[Cyril Hrubis]

Hi!
> The reason why dio_sparse is happening to be slow on certain systems is
> that, if data buffering is slow, we run more buffered read() for one
> single dio write(). This slows down the whole test, because for each
> read() we always need to move data from kernel space to user space.

I guess it's not about slow buffering. What I suppose happens is that
every time the writer thread writes with O_DIRECT it invalidates the
page cache and we have to re-read everything from disk. Which measn that
the data are often removed from the cache between the reads and the
reader processes are often forced to re-read the data from the disk. If
there was no O_DIRECT reader thread the first child that happens to read
a file block would cause kernel to put it into the page cache and all
other children would just copy that data without a need to reach the
disk at all.

However the test should finish as fast as the writer finishes writing
the file. So slow readers shouldn't matter unless there is some serious
contention on the disk I/O. That's probably the reason you are aligning
the writer as well.

What is the difference in runtime between test before and after this
patch on the slow hardware?

The only thing I wonder about is that if we aren't dropping some
coverage along with speeding up the test. For the reading part I guess
it doesn't matter that much how big the blocks are (if we speed up the
test we finish faster and do less operations, but that is something we
can live with). If we align the writer it may write directly whole
blocks instead of reading a block, modifying it and writing it back.
Looking at the runtest files, we do have dio_sparse there with a
different write block sizes, so the default shouldn't matter that much,
so why do we bother changing it?

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] io: fix really slow dio_sparse on certain systems

[Cyril Hrubis]

Hi!
> I guess it's not about slow buffering. What I suppose happens is that
> every time the writer thread writes with O_DIRECT it invalidates the
> page cache and we have to re-read everything from disk. Which measn that
> the data are often removed from the cache between the reads and the
> reader processes are often forced to re-read the data from the disk. If
> there was no O_DIRECT reader thread the first child that happens to read
                         ^
			 writer

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] io: fix really slow dio_sparse on certain systems

[Andrea Cervesato via ltp]

Hi!

On Mon Jan 26, 2026 at 12:42 PM CET, Cyril Hrubis wrote:
> Hi!
> > The reason why dio_sparse is happening to be slow on certain systems is
> > that, if data buffering is slow, we run more buffered read() for one
> > single dio write(). This slows down the whole test, because for each
> > read() we always need to move data from kernel space to user space.
>
> I guess it's not about slow buffering. What I suppose happens is that
> every time the writer thread writes with O_DIRECT it invalidates the
> page cache and we have to re-read everything from disk. Which measn that
> the data are often removed from the cache between the reads and the
> reader processes are often forced to re-read the data from the disk. If
> there was no O_DIRECT reader thread the first child that happens to read
> a file block would cause kernel to put it into the page cache and all
> other children would just copy that data without a need to reach the
> disk at all.

This is definetly a possible solution. I sent this patch by waiting for
some feebacks in order to have other opinions. What puzzles me is that
it's only happening in POWER10 on a random node during kernel tests.
Other architectures seem to work fine.

kernel 6.6+ seems to be the affected one.

>
> However the test should finish as fast as the writer finishes writing
> the file. So slow readers shouldn't matter unless there is some serious
> contention on the disk I/O. That's probably the reason you are aligning
> the writer as well.

Exactly, I would expect that.

>
> What is the difference in runtime between test before and after this
> patch on the slow hardware?

DS009 from 4 hours to 30 seconds. I also profiled the list of syscalls
with perf, obtaining a 63+ % of io_read() time consumption. Still, this
patch moves the execution from ~10 secs to ~3 secs on my laptop. There's
a big difference between 4h and 10 secs runtime, no matter the hard disk
which is running below.

>
> The only thing I wonder about is that if we aren't dropping some
> coverage along with speeding up the test. For the reading part I guess
> it doesn't matter that much how big the blocks are (if we speed up the
> test we finish faster and do less operations, but that is something we
> can live with). If we align the writer it may write directly whole
> blocks instead of reading a block, modifying it and writing it back.
> Looking at the runtest files, we do have dio_sparse there with a
> different write block sizes, so the default shouldn't matter that much,
> so why do we bother changing it?

My patch wants to be a default way to fix the problem for all the cases,
instead of adding parameters inside runtest file.


-- 
Andrea Cervesato
SUSE QE Automation Engineer Linux
andrea.cervesato@suse.com


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2] io: fix really slow dio_sparse on certain systems

[Cyril Hrubis]

Hi!
> This is definetly a possible solution. I sent this patch by waiting for
> some feebacks in order to have other opinions. What puzzles me is that
> it's only happening in POWER10 on a random node during kernel tests.
> Other architectures seem to work fine.
> 
> kernel 6.6+ seems to be the affected one.
> 
> >
> > However the test should finish as fast as the writer finishes writing
> > the file. So slow readers shouldn't matter unless there is some serious
> > contention on the disk I/O. That's probably the reason you are aligning
> > the writer as well.
> 
> Exactly, I would expect that.
> 
> >
> > What is the difference in runtime between test before and after this
> > patch on the slow hardware?
> 
> DS009 from 4 hours to 30 seconds. I also profiled the list of syscalls
> with perf, obtaining a 63+ % of io_read() time consumption. Still, this
> patch moves the execution from ~10 secs to ~3 secs on my laptop. There's
> a big difference between 4h and 10 secs runtime, no matter the hard disk
> which is running below.

Honestly that does sound like a kernel bug. You shouldn't get several
orders of magnitude slower just because your buffer is 4 times smaller.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp