* [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option
@ 2025-07-31 12:35 Ritesh Harjani (IBM)
2025-07-31 12:35 ` [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes Ritesh Harjani (IBM)
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Ritesh Harjani (IBM) @ 2025-07-31 12:35 UTC (permalink / raw)
To: fstests
Cc: linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry,
Ritesh Harjani (IBM)
This patch adds -D for O_DSYNC open flag to aio-dio-write-verify test.
We will use this in later patch for integrity verification test with
aio-dio.
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
---
src/aio-dio-regress/aio-dio-write-verify.c | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/src/aio-dio-regress/aio-dio-write-verify.c b/src/aio-dio-regress/aio-dio-write-verify.c
index 513a338b..0cf14a2a 100644
--- a/src/aio-dio-regress/aio-dio-write-verify.c
+++ b/src/aio-dio-regress/aio-dio-write-verify.c
@@ -40,6 +40,7 @@ void usage(char *progname)
"\t\tsize=N: AIO write size\n"
"\t\toff=M: AIO write startoff\n"
"\t-S: uses O_SYNC flag for open. By default O_SYNC is not used\n"
+ "\t-D: uses O_DSYNC flag for open. By default O_DSYNC is not used\n"
"\t-N: no_verify: means no write verification. By default noverify is false\n"
"e.g: %s -t 4608 -a size=4096,off=512 -a size=4096,off=4608 filename\n"
"e.g: %s -t 1048576 -a size=1048576 -S -N filename\n",
@@ -298,7 +299,7 @@ int main(int argc, char *argv[])
int o_sync = 0;
int no_verify = 0;
- while ((c = getopt(argc, argv, "a:t:SN")) != -1) {
+ while ((c = getopt(argc, argv, "a:t:SND")) != -1) {
char *endp;
switch (c) {
@@ -316,6 +317,9 @@ int main(int argc, char *argv[])
case 'S':
o_sync = O_SYNC;
break;
+ case 'D':
+ o_sync = O_DSYNC;
+ break;
case 'N':
no_verify = 1;
break;
--
2.49.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes
2025-07-31 12:35 [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option Ritesh Harjani (IBM)
@ 2025-07-31 12:35 ` Ritesh Harjani (IBM)
2025-08-01 20:18 ` Zorro Lang
2025-07-31 15:00 ` [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option Jan Kara
2025-08-02 8:18 ` Zorro Lang
2 siblings, 1 reply; 8+ messages in thread
From: Ritesh Harjani (IBM) @ 2025-07-31 12:35 UTC (permalink / raw)
To: fstests
Cc: linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry,
Ritesh Harjani (IBM)
This test verifies the data & required metadata (e.g. inode i_size for extending
writes) integrity when using O_DSYNC and RWF_DSYNC during writes operations,
across buffered-io, aio-dio and dio, in the event of a sudden filesystem
shutdown after write completion.
Man page of open says that -
O_DSYNC provides synchronized I/O data integrity completion, meaning
write operations will flush data to the underlying hardware, but will
only flush metadata updates that are required to allow a subsequent read
operation to complete successfully.
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
---
tests/generic/737 | 30 +++++++++++++++++++++++++++++-
tests/generic/737.out | 21 +++++++++++++++++++++
2 files changed, 50 insertions(+), 1 deletion(-)
diff --git a/tests/generic/737 b/tests/generic/737
index 99ca1f39..0f27c82b 100755
--- a/tests/generic/737
+++ b/tests/generic/737
@@ -4,7 +4,8 @@
#
# FS QA Test No. 737
#
-# Integrity test for O_SYNC with buff-io, dio, aio-dio with sudden shutdown.
+# Integrity test for O_[D]SYNC and/or RWF_DSYNC with buff-io, dio, aio-dio with
+# sudden shutdown.
# Based on a testcase reported by Gao Xiang <hsiangkao@linux.alibaba.com>
#
@@ -21,6 +22,15 @@ _require_aiodio aio-dio-write-verify
_scratch_mkfs > $seqres.full 2>&1
_scratch_mount
+echo "T-0: Create a 1M file using buff-io & RWF_DSYNC"
+$XFS_IO_PROG -f -c "pwrite -V 1 -D -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
+echo "T-0: Shutdown the fs suddenly"
+_scratch_shutdown
+echo "T-0: Cycle mount"
+_scratch_cycle_mount
+echo "T-0: File contents after cycle mount"
+_hexdump $SCRATCH_MNT/testfile.t1
+
echo "T-1: Create a 1M file using buff-io & O_SYNC"
$XFS_IO_PROG -fs -c "pwrite -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
echo "T-1: Shutdown the fs suddenly"
@@ -48,5 +58,23 @@ _scratch_cycle_mount
echo "T-3: File contents after cycle mount"
_hexdump $SCRATCH_MNT/testfile.t3
+echo "T-4: Create a 1M file using DIO & RWF_DSYNC"
+$XFS_IO_PROG -fdc "pwrite -V 1 -S 0x5a -D 0 1M" $SCRATCH_MNT/testfile.t4 > /dev/null 2>&1
+echo "T-4: Shutdown the fs suddenly"
+_scratch_shutdown
+echo "T-4: Cycle mount"
+_scratch_cycle_mount
+echo "T-4: File contents after cycle mount"
+_hexdump $SCRATCH_MNT/testfile.t4
+
+echo "T-5: Create a 1M file using AIO-DIO & O_DSYNC"
+$AIO_TEST -a size=1048576 -D -N $SCRATCH_MNT/testfile.t5 > /dev/null 2>&1
+echo "T-5: Shutdown the fs suddenly"
+_scratch_shutdown
+echo "T-5: Cycle mount"
+_scratch_cycle_mount
+echo "T-5: File contents after cycle mount"
+_hexdump $SCRATCH_MNT/testfile.t5
+
status=0
exit
diff --git a/tests/generic/737.out b/tests/generic/737.out
index efe6ff1f..2bafeefa 100644
--- a/tests/generic/737.out
+++ b/tests/generic/737.out
@@ -1,4 +1,11 @@
QA output created by 737
+T-0: Create a 1M file using buff-io & RWF_DSYNC
+T-0: Shutdown the fs suddenly
+T-0: Cycle mount
+T-0: File contents after cycle mount
+000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
+*
+100000
T-1: Create a 1M file using buff-io & O_SYNC
T-1: Shutdown the fs suddenly
T-1: Cycle mount
@@ -20,3 +27,17 @@ T-3: File contents after cycle mount
000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
*
100000
+T-4: Create a 1M file using DIO & RWF_DSYNC
+T-4: Shutdown the fs suddenly
+T-4: Cycle mount
+T-4: File contents after cycle mount
+000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
+*
+100000
+T-5: Create a 1M file using AIO-DIO & O_DSYNC
+T-5: Shutdown the fs suddenly
+T-5: Cycle mount
+T-5: File contents after cycle mount
+000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
+*
+100000
--
2.49.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option
2025-07-31 12:35 [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option Ritesh Harjani (IBM)
2025-07-31 12:35 ` [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes Ritesh Harjani (IBM)
@ 2025-07-31 15:00 ` Jan Kara
2025-08-02 8:18 ` Zorro Lang
2 siblings, 0 replies; 8+ messages in thread
From: Jan Kara @ 2025-07-31 15:00 UTC (permalink / raw)
To: Ritesh Harjani (IBM)
Cc: fstests, linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry
On Thu 31-07-25 18:05:54, Ritesh Harjani (IBM) wrote:
> This patch adds -D for O_DSYNC open flag to aio-dio-write-verify test.
> We will use this in later patch for integrity verification test with
> aio-dio.
>
> Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
Cool. Both patches look good to me and they fail without the iomap fix I've
submitted so feel free to add:
Reviewed-by: Jan Kara <jack@suse.cz>
and
Tested-by: Jan Kara <jack@suse.cz>
Honza
> ---
> src/aio-dio-regress/aio-dio-write-verify.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/src/aio-dio-regress/aio-dio-write-verify.c b/src/aio-dio-regress/aio-dio-write-verify.c
> index 513a338b..0cf14a2a 100644
> --- a/src/aio-dio-regress/aio-dio-write-verify.c
> +++ b/src/aio-dio-regress/aio-dio-write-verify.c
> @@ -40,6 +40,7 @@ void usage(char *progname)
> "\t\tsize=N: AIO write size\n"
> "\t\toff=M: AIO write startoff\n"
> "\t-S: uses O_SYNC flag for open. By default O_SYNC is not used\n"
> + "\t-D: uses O_DSYNC flag for open. By default O_DSYNC is not used\n"
> "\t-N: no_verify: means no write verification. By default noverify is false\n"
> "e.g: %s -t 4608 -a size=4096,off=512 -a size=4096,off=4608 filename\n"
> "e.g: %s -t 1048576 -a size=1048576 -S -N filename\n",
> @@ -298,7 +299,7 @@ int main(int argc, char *argv[])
> int o_sync = 0;
> int no_verify = 0;
>
> - while ((c = getopt(argc, argv, "a:t:SN")) != -1) {
> + while ((c = getopt(argc, argv, "a:t:SND")) != -1) {
> char *endp;
>
> switch (c) {
> @@ -316,6 +317,9 @@ int main(int argc, char *argv[])
> case 'S':
> o_sync = O_SYNC;
> break;
> + case 'D':
> + o_sync = O_DSYNC;
> + break;
> case 'N':
> no_verify = 1;
> break;
> --
> 2.49.0
>
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes
2025-07-31 12:35 ` [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes Ritesh Harjani (IBM)
@ 2025-08-01 20:18 ` Zorro Lang
2025-08-01 20:29 ` Ritesh Harjani
0 siblings, 1 reply; 8+ messages in thread
From: Zorro Lang @ 2025-08-01 20:18 UTC (permalink / raw)
To: Ritesh Harjani (IBM)
Cc: fstests, linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry
On Thu, Jul 31, 2025 at 06:05:55PM +0530, Ritesh Harjani (IBM) wrote:
> This test verifies the data & required metadata (e.g. inode i_size for extending
> writes) integrity when using O_DSYNC and RWF_DSYNC during writes operations,
> across buffered-io, aio-dio and dio, in the event of a sudden filesystem
> shutdown after write completion.
>
> Man page of open says that -
> O_DSYNC provides synchronized I/O data integrity completion, meaning
> write operations will flush data to the underlying hardware, but will
> only flush metadata updates that are required to allow a subsequent read
> operation to complete successfully.
>
> Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
> ---
> tests/generic/737 | 30 +++++++++++++++++++++++++++++-
> tests/generic/737.out | 21 +++++++++++++++++++++
> 2 files changed, 50 insertions(+), 1 deletion(-)
>
> diff --git a/tests/generic/737 b/tests/generic/737
> index 99ca1f39..0f27c82b 100755
> --- a/tests/generic/737
> +++ b/tests/generic/737
> @@ -4,7 +4,8 @@
> #
> # FS QA Test No. 737
> #
> -# Integrity test for O_SYNC with buff-io, dio, aio-dio with sudden shutdown.
> +# Integrity test for O_[D]SYNC and/or RWF_DSYNC with buff-io, dio, aio-dio with
> +# sudden shutdown.
> # Based on a testcase reported by Gao Xiang <hsiangkao@linux.alibaba.com>
> #
>
> @@ -21,6 +22,15 @@ _require_aiodio aio-dio-write-verify
> _scratch_mkfs > $seqres.full 2>&1
> _scratch_mount
>
> +echo "T-0: Create a 1M file using buff-io & RWF_DSYNC"
> +$XFS_IO_PROG -f -c "pwrite -V 1 -D -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
> +echo "T-0: Shutdown the fs suddenly"
> +_scratch_shutdown
> +echo "T-0: Cycle mount"
> +_scratch_cycle_mount
> +echo "T-0: File contents after cycle mount"
> +_hexdump $SCRATCH_MNT/testfile.t1
> +
> echo "T-1: Create a 1M file using buff-io & O_SYNC"
> $XFS_IO_PROG -fs -c "pwrite -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
> echo "T-1: Shutdown the fs suddenly"
> @@ -48,5 +58,23 @@ _scratch_cycle_mount
> echo "T-3: File contents after cycle mount"
> _hexdump $SCRATCH_MNT/testfile.t3
>
> +echo "T-4: Create a 1M file using DIO & RWF_DSYNC"
> +$XFS_IO_PROG -fdc "pwrite -V 1 -S 0x5a -D 0 1M" $SCRATCH_MNT/testfile.t4 > /dev/null 2>&1
> +echo "T-4: Shutdown the fs suddenly"
> +_scratch_shutdown
> +echo "T-4: Cycle mount"
> +_scratch_cycle_mount
> +echo "T-4: File contents after cycle mount"
> +_hexdump $SCRATCH_MNT/testfile.t4
> +
> +echo "T-5: Create a 1M file using AIO-DIO & O_DSYNC"
> +$AIO_TEST -a size=1048576 -D -N $SCRATCH_MNT/testfile.t5 > /dev/null 2>&1
> +echo "T-5: Shutdown the fs suddenly"
> +_scratch_shutdown
> +echo "T-5: Cycle mount"
> +_scratch_cycle_mount
> +echo "T-5: File contents after cycle mount"
> +_hexdump $SCRATCH_MNT/testfile.t5
I always hit "No such file or directory" [1], is this an expected test failure
which you hope to uncover?
[1]
# diff -u /root/git/xfstests/tests/generic/737.out /root/git/xfstests/results//generic/737.out.bad
--- /root/git/xfstests/tests/generic/737.out 2025-08-02 04:04:57.334489725 +0800
+++ /root/git/xfstests/results//generic/737.out.bad 2025-08-02 04:12:08.167934723 +0800
@@ -28,16 +28,14 @@
*
100000
T-4: Create a 1M file using DIO & RWF_DSYNC
T-4: Shutdown the fs suddenly
T-4: Cycle mount
T-4: File contents after cycle mount
-000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
-*
-100000
+od: /mnt/scratch/testfile.t4: No such file or directory
T-5: Create a 1M file using AIO-DIO & O_DSYNC
T-5: Shutdown the fs suddenly
T-5: Cycle mount
T-5: File contents after cycle mount
-000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
-*
-100000
+od: /mnt/scratch/testfile.t5: No such file or directory
> +
> status=0
> exit
> diff --git a/tests/generic/737.out b/tests/generic/737.out
> index efe6ff1f..2bafeefa 100644
> --- a/tests/generic/737.out
> +++ b/tests/generic/737.out
> @@ -1,4 +1,11 @@
> QA output created by 737
> +T-0: Create a 1M file using buff-io & RWF_DSYNC
> +T-0: Shutdown the fs suddenly
> +T-0: Cycle mount
> +T-0: File contents after cycle mount
> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> +*
> +100000
> T-1: Create a 1M file using buff-io & O_SYNC
> T-1: Shutdown the fs suddenly
> T-1: Cycle mount
> @@ -20,3 +27,17 @@ T-3: File contents after cycle mount
> 000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> *
> 100000
> +T-4: Create a 1M file using DIO & RWF_DSYNC
> +T-4: Shutdown the fs suddenly
> +T-4: Cycle mount
> +T-4: File contents after cycle mount
> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> +*
> +100000
> +T-5: Create a 1M file using AIO-DIO & O_DSYNC
> +T-5: Shutdown the fs suddenly
> +T-5: Cycle mount
> +T-5: File contents after cycle mount
> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> +*
> +100000
> --
> 2.49.0
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes
2025-08-01 20:18 ` Zorro Lang
@ 2025-08-01 20:29 ` Ritesh Harjani
2025-08-02 8:17 ` Zorro Lang
0 siblings, 1 reply; 8+ messages in thread
From: Ritesh Harjani @ 2025-08-01 20:29 UTC (permalink / raw)
To: Zorro Lang; +Cc: fstests, linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry
Zorro Lang <zlang@redhat.com> writes:
> On Thu, Jul 31, 2025 at 06:05:55PM +0530, Ritesh Harjani (IBM) wrote:
>> This test verifies the data & required metadata (e.g. inode i_size for extending
>> writes) integrity when using O_DSYNC and RWF_DSYNC during writes operations,
>> across buffered-io, aio-dio and dio, in the event of a sudden filesystem
>> shutdown after write completion.
>>
>> Man page of open says that -
>> O_DSYNC provides synchronized I/O data integrity completion, meaning
>> write operations will flush data to the underlying hardware, but will
>> only flush metadata updates that are required to allow a subsequent read
>> operation to complete successfully.
>>
>> Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
>> ---
>> tests/generic/737 | 30 +++++++++++++++++++++++++++++-
>> tests/generic/737.out | 21 +++++++++++++++++++++
>> 2 files changed, 50 insertions(+), 1 deletion(-)
>>
>> diff --git a/tests/generic/737 b/tests/generic/737
>> index 99ca1f39..0f27c82b 100755
>> --- a/tests/generic/737
>> +++ b/tests/generic/737
>> @@ -4,7 +4,8 @@
>> #
>> # FS QA Test No. 737
>> #
>> -# Integrity test for O_SYNC with buff-io, dio, aio-dio with sudden shutdown.
>> +# Integrity test for O_[D]SYNC and/or RWF_DSYNC with buff-io, dio, aio-dio with
>> +# sudden shutdown.
>> # Based on a testcase reported by Gao Xiang <hsiangkao@linux.alibaba.com>
>> #
>>
>> @@ -21,6 +22,15 @@ _require_aiodio aio-dio-write-verify
>> _scratch_mkfs > $seqres.full 2>&1
>> _scratch_mount
>>
>> +echo "T-0: Create a 1M file using buff-io & RWF_DSYNC"
>> +$XFS_IO_PROG -f -c "pwrite -V 1 -D -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
>> +echo "T-0: Shutdown the fs suddenly"
>> +_scratch_shutdown
>> +echo "T-0: Cycle mount"
>> +_scratch_cycle_mount
>> +echo "T-0: File contents after cycle mount"
>> +_hexdump $SCRATCH_MNT/testfile.t1
>> +
>> echo "T-1: Create a 1M file using buff-io & O_SYNC"
>> $XFS_IO_PROG -fs -c "pwrite -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
>> echo "T-1: Shutdown the fs suddenly"
>> @@ -48,5 +58,23 @@ _scratch_cycle_mount
>> echo "T-3: File contents after cycle mount"
>> _hexdump $SCRATCH_MNT/testfile.t3
>>
>> +echo "T-4: Create a 1M file using DIO & RWF_DSYNC"
>> +$XFS_IO_PROG -fdc "pwrite -V 1 -S 0x5a -D 0 1M" $SCRATCH_MNT/testfile.t4 > /dev/null 2>&1
>> +echo "T-4: Shutdown the fs suddenly"
>> +_scratch_shutdown
>> +echo "T-4: Cycle mount"
>> +_scratch_cycle_mount
>> +echo "T-4: File contents after cycle mount"
>> +_hexdump $SCRATCH_MNT/testfile.t4
>> +
>> +echo "T-5: Create a 1M file using AIO-DIO & O_DSYNC"
>> +$AIO_TEST -a size=1048576 -D -N $SCRATCH_MNT/testfile.t5 > /dev/null 2>&1
>> +echo "T-5: Shutdown the fs suddenly"
>> +_scratch_shutdown
>> +echo "T-5: Cycle mount"
>> +_scratch_cycle_mount
>> +echo "T-5: File contents after cycle mount"
>> +_hexdump $SCRATCH_MNT/testfile.t5
>
> I always hit "No such file or directory" [1], is this an expected test failure
> which you hope to uncover?
Yes, we will need this fix [1] from Jan. Sorry I missed to add that in the commit
message. Could you please give it a try with the fix maybe?
[1]:
https://lore.kernel.org/linux-fsdevel/20250730102840.20470-2-jack@suse.cz/
-ritesh
>
> [1]
> # diff -u /root/git/xfstests/tests/generic/737.out /root/git/xfstests/results//generic/737.out.bad
> --- /root/git/xfstests/tests/generic/737.out 2025-08-02 04:04:57.334489725 +0800
> +++ /root/git/xfstests/results//generic/737.out.bad 2025-08-02 04:12:08.167934723 +0800
> @@ -28,16 +28,14 @@
> *
> 100000
> T-4: Create a 1M file using DIO & RWF_DSYNC
> T-4: Shutdown the fs suddenly
> T-4: Cycle mount
> T-4: File contents after cycle mount
> -000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> -*
> -100000
> +od: /mnt/scratch/testfile.t4: No such file or directory
> T-5: Create a 1M file using AIO-DIO & O_DSYNC
> T-5: Shutdown the fs suddenly
> T-5: Cycle mount
> T-5: File contents after cycle mount
> -000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> -*
> -100000
> +od: /mnt/scratch/testfile.t5: No such file or directory
>
>> +
>> status=0
>> exit
>> diff --git a/tests/generic/737.out b/tests/generic/737.out
>> index efe6ff1f..2bafeefa 100644
>> --- a/tests/generic/737.out
>> +++ b/tests/generic/737.out
>> @@ -1,4 +1,11 @@
>> QA output created by 737
>> +T-0: Create a 1M file using buff-io & RWF_DSYNC
>> +T-0: Shutdown the fs suddenly
>> +T-0: Cycle mount
>> +T-0: File contents after cycle mount
>> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
>> +*
>> +100000
>> T-1: Create a 1M file using buff-io & O_SYNC
>> T-1: Shutdown the fs suddenly
>> T-1: Cycle mount
>> @@ -20,3 +27,17 @@ T-3: File contents after cycle mount
>> 000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
>> *
>> 100000
>> +T-4: Create a 1M file using DIO & RWF_DSYNC
>> +T-4: Shutdown the fs suddenly
>> +T-4: Cycle mount
>> +T-4: File contents after cycle mount
>> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
>> +*
>> +100000
>> +T-5: Create a 1M file using AIO-DIO & O_DSYNC
>> +T-5: Shutdown the fs suddenly
>> +T-5: Cycle mount
>> +T-5: File contents after cycle mount
>> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
>> +*
>> +100000
>> --
>> 2.49.0
>>
>>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes
2025-08-01 20:29 ` Ritesh Harjani
@ 2025-08-02 8:17 ` Zorro Lang
2025-08-03 5:23 ` Ritesh Harjani
0 siblings, 1 reply; 8+ messages in thread
From: Zorro Lang @ 2025-08-02 8:17 UTC (permalink / raw)
To: Ritesh Harjani
Cc: fstests, linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry
On Sat, Aug 02, 2025 at 01:59:18AM +0530, Ritesh Harjani wrote:
> Zorro Lang <zlang@redhat.com> writes:
>
> > On Thu, Jul 31, 2025 at 06:05:55PM +0530, Ritesh Harjani (IBM) wrote:
> >> This test verifies the data & required metadata (e.g. inode i_size for extending
> >> writes) integrity when using O_DSYNC and RWF_DSYNC during writes operations,
> >> across buffered-io, aio-dio and dio, in the event of a sudden filesystem
> >> shutdown after write completion.
> >>
> >> Man page of open says that -
> >> O_DSYNC provides synchronized I/O data integrity completion, meaning
> >> write operations will flush data to the underlying hardware, but will
> >> only flush metadata updates that are required to allow a subsequent read
> >> operation to complete successfully.
> >>
> >> Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
> >> ---
> >> tests/generic/737 | 30 +++++++++++++++++++++++++++++-
> >> tests/generic/737.out | 21 +++++++++++++++++++++
BTW, as you change a testcase with known case number, your subject can be
"generic/737: ..." (instead of "generic: ...").
> >> 2 files changed, 50 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/tests/generic/737 b/tests/generic/737
> >> index 99ca1f39..0f27c82b 100755
> >> --- a/tests/generic/737
> >> +++ b/tests/generic/737
> >> @@ -4,7 +4,8 @@
> >> #
> >> # FS QA Test No. 737
> >> #
> >> -# Integrity test for O_SYNC with buff-io, dio, aio-dio with sudden shutdown.
> >> +# Integrity test for O_[D]SYNC and/or RWF_DSYNC with buff-io, dio, aio-dio with
> >> +# sudden shutdown.
> >> # Based on a testcase reported by Gao Xiang <hsiangkao@linux.alibaba.com>
> >> #
> >>
> >> @@ -21,6 +22,15 @@ _require_aiodio aio-dio-write-verify
> >> _scratch_mkfs > $seqres.full 2>&1
> >> _scratch_mount
> >>
> >> +echo "T-0: Create a 1M file using buff-io & RWF_DSYNC"
> >> +$XFS_IO_PROG -f -c "pwrite -V 1 -D -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
> >> +echo "T-0: Shutdown the fs suddenly"
> >> +_scratch_shutdown
> >> +echo "T-0: Cycle mount"
> >> +_scratch_cycle_mount
> >> +echo "T-0: File contents after cycle mount"
> >> +_hexdump $SCRATCH_MNT/testfile.t1
> >> +
> >> echo "T-1: Create a 1M file using buff-io & O_SYNC"
> >> $XFS_IO_PROG -fs -c "pwrite -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
> >> echo "T-1: Shutdown the fs suddenly"
> >> @@ -48,5 +58,23 @@ _scratch_cycle_mount
> >> echo "T-3: File contents after cycle mount"
> >> _hexdump $SCRATCH_MNT/testfile.t3
> >>
> >> +echo "T-4: Create a 1M file using DIO & RWF_DSYNC"
> >> +$XFS_IO_PROG -fdc "pwrite -V 1 -S 0x5a -D 0 1M" $SCRATCH_MNT/testfile.t4 > /dev/null 2>&1
> >> +echo "T-4: Shutdown the fs suddenly"
> >> +_scratch_shutdown
> >> +echo "T-4: Cycle mount"
> >> +_scratch_cycle_mount
> >> +echo "T-4: File contents after cycle mount"
> >> +_hexdump $SCRATCH_MNT/testfile.t4
> >> +
> >> +echo "T-5: Create a 1M file using AIO-DIO & O_DSYNC"
> >> +$AIO_TEST -a size=1048576 -D -N $SCRATCH_MNT/testfile.t5 > /dev/null 2>&1
> >> +echo "T-5: Shutdown the fs suddenly"
> >> +_scratch_shutdown
> >> +echo "T-5: Cycle mount"
> >> +_scratch_cycle_mount
> >> +echo "T-5: File contents after cycle mount"
> >> +_hexdump $SCRATCH_MNT/testfile.t5
> >
> > I always hit "No such file or directory" [1], is this an expected test failure
> > which you hope to uncover?
>
> Yes, we will need this fix [1] from Jan. Sorry I missed to add that in the commit
> message. Could you please give it a try with the fix maybe?
Sure, with this patch, this test passed on my side:
FSTYP -- xfs (debug)
PLATFORM -- Linux/x86_64 dell-per750-41 6.16.0-mainline+ #9 SMP PREEMPT_DYNAMIC Sat Aug 2 16:01:22 CST 2025
MKFS_OPTIONS -- -f /dev/mapper/testvg-scratch--devA
MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/mapper/testvg-scratch--devA /mnt/scratch
generic/737 4s ... 9s
Ran: generic/737
Passed all 1 tests
So this "No such file or directory" failure is a known bug, and this patch trys to
uncover it. You didn't metion that, so I thought you just try to write a
new integrity test (rather than a regression test) :-D If this change uncover a known
fix, please feel free to add _fixed_by_kernel_commit.
BTW, I saw you mark this patchset with "RFC", do you need to change it more?
Generally I won't merge RFC patches directly, although this patch looks good to
me. So when you think you've gotten enough review points, please feel free to
remove the "RFC" label and resend it :) Then ...
Reviewed-by: Zorro Lang <zlang@redhat.com>
Thanks,
Zorro
>
> [1]:
> https://lore.kernel.org/linux-fsdevel/20250730102840.20470-2-jack@suse.cz/
>
> -ritesh
>
>
> >
> > [1]
> > # diff -u /root/git/xfstests/tests/generic/737.out /root/git/xfstests/results//generic/737.out.bad
> > --- /root/git/xfstests/tests/generic/737.out 2025-08-02 04:04:57.334489725 +0800
> > +++ /root/git/xfstests/results//generic/737.out.bad 2025-08-02 04:12:08.167934723 +0800
> > @@ -28,16 +28,14 @@
> > *
> > 100000
> > T-4: Create a 1M file using DIO & RWF_DSYNC
> > T-4: Shutdown the fs suddenly
> > T-4: Cycle mount
> > T-4: File contents after cycle mount
> > -000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> > -*
> > -100000
> > +od: /mnt/scratch/testfile.t4: No such file or directory
> > T-5: Create a 1M file using AIO-DIO & O_DSYNC
> > T-5: Shutdown the fs suddenly
> > T-5: Cycle mount
> > T-5: File contents after cycle mount
> > -000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> > -*
> > -100000
> > +od: /mnt/scratch/testfile.t5: No such file or directory
> >
> >> +
> >> status=0
> >> exit
> >> diff --git a/tests/generic/737.out b/tests/generic/737.out
> >> index efe6ff1f..2bafeefa 100644
> >> --- a/tests/generic/737.out
> >> +++ b/tests/generic/737.out
> >> @@ -1,4 +1,11 @@
> >> QA output created by 737
> >> +T-0: Create a 1M file using buff-io & RWF_DSYNC
> >> +T-0: Shutdown the fs suddenly
> >> +T-0: Cycle mount
> >> +T-0: File contents after cycle mount
> >> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> >> +*
> >> +100000
> >> T-1: Create a 1M file using buff-io & O_SYNC
> >> T-1: Shutdown the fs suddenly
> >> T-1: Cycle mount
> >> @@ -20,3 +27,17 @@ T-3: File contents after cycle mount
> >> 000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> >> *
> >> 100000
> >> +T-4: Create a 1M file using DIO & RWF_DSYNC
> >> +T-4: Shutdown the fs suddenly
> >> +T-4: Cycle mount
> >> +T-4: File contents after cycle mount
> >> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> >> +*
> >> +100000
> >> +T-5: Create a 1M file using AIO-DIO & O_DSYNC
> >> +T-5: Shutdown the fs suddenly
> >> +T-5: Cycle mount
> >> +T-5: File contents after cycle mount
> >> +000000 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a >ZZZZZZZZZZZZZZZZ<
> >> +*
> >> +100000
> >> --
> >> 2.49.0
> >>
> >>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option
2025-07-31 12:35 [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option Ritesh Harjani (IBM)
2025-07-31 12:35 ` [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes Ritesh Harjani (IBM)
2025-07-31 15:00 ` [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option Jan Kara
@ 2025-08-02 8:18 ` Zorro Lang
2 siblings, 0 replies; 8+ messages in thread
From: Zorro Lang @ 2025-08-02 8:18 UTC (permalink / raw)
To: Ritesh Harjani (IBM)
Cc: fstests, linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry
On Thu, Jul 31, 2025 at 06:05:54PM +0530, Ritesh Harjani (IBM) wrote:
> This patch adds -D for O_DSYNC open flag to aio-dio-write-verify test.
> We will use this in later patch for integrity verification test with
> aio-dio.
>
> Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
> ---
Reviewed-by: Zorro Lang <zlang@redhat.com>
> src/aio-dio-regress/aio-dio-write-verify.c | 6 +++++-
> 1 file changed, 5 insertions(+), 1 deletion(-)
>
> diff --git a/src/aio-dio-regress/aio-dio-write-verify.c b/src/aio-dio-regress/aio-dio-write-verify.c
> index 513a338b..0cf14a2a 100644
> --- a/src/aio-dio-regress/aio-dio-write-verify.c
> +++ b/src/aio-dio-regress/aio-dio-write-verify.c
> @@ -40,6 +40,7 @@ void usage(char *progname)
> "\t\tsize=N: AIO write size\n"
> "\t\toff=M: AIO write startoff\n"
> "\t-S: uses O_SYNC flag for open. By default O_SYNC is not used\n"
> + "\t-D: uses O_DSYNC flag for open. By default O_DSYNC is not used\n"
> "\t-N: no_verify: means no write verification. By default noverify is false\n"
> "e.g: %s -t 4608 -a size=4096,off=512 -a size=4096,off=4608 filename\n"
> "e.g: %s -t 1048576 -a size=1048576 -S -N filename\n",
> @@ -298,7 +299,7 @@ int main(int argc, char *argv[])
> int o_sync = 0;
> int no_verify = 0;
>
> - while ((c = getopt(argc, argv, "a:t:SN")) != -1) {
> + while ((c = getopt(argc, argv, "a:t:SND")) != -1) {
> char *endp;
>
> switch (c) {
> @@ -316,6 +317,9 @@ int main(int argc, char *argv[])
> case 'S':
> o_sync = O_SYNC;
> break;
> + case 'D':
> + o_sync = O_DSYNC;
> + break;
> case 'N':
> no_verify = 1;
> break;
> --
> 2.49.0
>
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes
2025-08-02 8:17 ` Zorro Lang
@ 2025-08-03 5:23 ` Ritesh Harjani
0 siblings, 0 replies; 8+ messages in thread
From: Ritesh Harjani @ 2025-08-03 5:23 UTC (permalink / raw)
To: Zorro Lang; +Cc: fstests, linux-fsdevel, Jan Kara, Darrick J . Wong, John Garry
Zorro Lang <zlang@redhat.com> writes:
> On Sat, Aug 02, 2025 at 01:59:18AM +0530, Ritesh Harjani wrote:
>> Zorro Lang <zlang@redhat.com> writes:
>>
>> > On Thu, Jul 31, 2025 at 06:05:55PM +0530, Ritesh Harjani (IBM) wrote:
>> >> This test verifies the data & required metadata (e.g. inode i_size for extending
>> >> writes) integrity when using O_DSYNC and RWF_DSYNC during writes operations,
>> >> across buffered-io, aio-dio and dio, in the event of a sudden filesystem
>> >> shutdown after write completion.
>> >>
>> >> Man page of open says that -
>> >> O_DSYNC provides synchronized I/O data integrity completion, meaning
>> >> write operations will flush data to the underlying hardware, but will
>> >> only flush metadata updates that are required to allow a subsequent read
>> >> operation to complete successfully.
>> >>
>> >> Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@gmail.com>
>> >> ---
>> >> tests/generic/737 | 30 +++++++++++++++++++++++++++++-
>> >> tests/generic/737.out | 21 +++++++++++++++++++++
>
> BTW, as you change a testcase with known case number, your subject can be
> "generic/737: ..." (instead of "generic: ...").
>
Make sense. I will fix in v2.
>> >> 2 files changed, 50 insertions(+), 1 deletion(-)
>> >>
>> >> diff --git a/tests/generic/737 b/tests/generic/737
>> >> index 99ca1f39..0f27c82b 100755
>> >> --- a/tests/generic/737
>> >> +++ b/tests/generic/737
>> >> @@ -4,7 +4,8 @@
>> >> #
>> >> # FS QA Test No. 737
>> >> #
>> >> -# Integrity test for O_SYNC with buff-io, dio, aio-dio with sudden shutdown.
>> >> +# Integrity test for O_[D]SYNC and/or RWF_DSYNC with buff-io, dio, aio-dio with
>> >> +# sudden shutdown.
>> >> # Based on a testcase reported by Gao Xiang <hsiangkao@linux.alibaba.com>
>> >> #
>> >>
>> >> @@ -21,6 +22,15 @@ _require_aiodio aio-dio-write-verify
>> >> _scratch_mkfs > $seqres.full 2>&1
>> >> _scratch_mount
>> >>
>> >> +echo "T-0: Create a 1M file using buff-io & RWF_DSYNC"
>> >> +$XFS_IO_PROG -f -c "pwrite -V 1 -D -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
>> >> +echo "T-0: Shutdown the fs suddenly"
>> >> +_scratch_shutdown
>> >> +echo "T-0: Cycle mount"
>> >> +_scratch_cycle_mount
>> >> +echo "T-0: File contents after cycle mount"
>> >> +_hexdump $SCRATCH_MNT/testfile.t1
>> >> +
>> >> echo "T-1: Create a 1M file using buff-io & O_SYNC"
>> >> $XFS_IO_PROG -fs -c "pwrite -S 0x5a 0 1M" $SCRATCH_MNT/testfile.t1 > /dev/null 2>&1
>> >> echo "T-1: Shutdown the fs suddenly"
>> >> @@ -48,5 +58,23 @@ _scratch_cycle_mount
>> >> echo "T-3: File contents after cycle mount"
>> >> _hexdump $SCRATCH_MNT/testfile.t3
>> >>
>> >> +echo "T-4: Create a 1M file using DIO & RWF_DSYNC"
>> >> +$XFS_IO_PROG -fdc "pwrite -V 1 -S 0x5a -D 0 1M" $SCRATCH_MNT/testfile.t4 > /dev/null 2>&1
>> >> +echo "T-4: Shutdown the fs suddenly"
>> >> +_scratch_shutdown
>> >> +echo "T-4: Cycle mount"
>> >> +_scratch_cycle_mount
>> >> +echo "T-4: File contents after cycle mount"
>> >> +_hexdump $SCRATCH_MNT/testfile.t4
>> >> +
>> >> +echo "T-5: Create a 1M file using AIO-DIO & O_DSYNC"
>> >> +$AIO_TEST -a size=1048576 -D -N $SCRATCH_MNT/testfile.t5 > /dev/null 2>&1
>> >> +echo "T-5: Shutdown the fs suddenly"
>> >> +_scratch_shutdown
>> >> +echo "T-5: Cycle mount"
>> >> +_scratch_cycle_mount
>> >> +echo "T-5: File contents after cycle mount"
>> >> +_hexdump $SCRATCH_MNT/testfile.t5
>> >
>> > I always hit "No such file or directory" [1], is this an expected test failure
>> > which you hope to uncover?
>>
>> Yes, we will need this fix [1] from Jan. Sorry I missed to add that in the commit
>> message. Could you please give it a try with the fix maybe?
>
> Sure, with this patch, this test passed on my side:
> FSTYP -- xfs (debug)
> PLATFORM -- Linux/x86_64 dell-per750-41 6.16.0-mainline+ #9 SMP PREEMPT_DYNAMIC Sat Aug 2 16:01:22 CST 2025
> MKFS_OPTIONS -- -f /dev/mapper/testvg-scratch--devA
> MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/mapper/testvg-scratch--devA /mnt/scratch
>
> generic/737 4s ... 9s
> Ran: generic/737
> Passed all 1 tests
>
> So this "No such file or directory" failure is a known bug, and this patch trys to
> uncover it. You didn't metion that, so I thought you just try to write a
> new integrity test (rather than a regression test) :-D
Your understanding is correct. Sorry, I missed to add the fix details.
I posted the test before the fix got merged. But yes, I will add the
necessary details in v2.
> If this change uncover a known fix, please feel free to add _fixed_by_kernel_commit.
The commit isn't yet showing up in the VFS tree. Once it gets in I will
share the v2 with the following changes.
diff --git a/tests/generic/737 b/tests/generic/737
index 0f27c82b..a51e9623 100755
--- a/tests/generic/737
+++ b/tests/generic/737
@@ -18,6 +18,9 @@ _require_aiodio aio-dio-write-verify
[[ "$FSTYP" =~ ext[0-9]+ ]] && _fixed_by_kernel_commit 91562895f803 \
"ext4: properly sync file size update after O_SYNC direct IO"
+# which is further fixed by
+_fixed_by_kernel_commit 16f206eebbf8 \
+ "iomap: Fix broken data integrity guarantees for O_SYNC writes"
_scratch_mkfs > $seqres.full 2>&1
_scratch_mount
> BTW, I saw you mark this patchset with "RFC", do you need to change it more?
> Generally I won't merge RFC patches directly, although this patch looks good to
> me. So when you think you've gotten enough review points, please feel free to
> remove the "RFC" label and resend it :) Then ...
>
Right. I will drop RFC and will resend it with above mentioned points
(once the fix appears in VFS tree).
> Reviewed-by: Zorro Lang <zlang@redhat.com>
>
Thanks!
-ritesh
> Thanks,
> Zorro
>
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-08-03 5:38 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-31 12:35 [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option Ritesh Harjani (IBM)
2025-07-31 12:35 ` [RFC 2/2] generic: Add integrity tests for O_DSYNC and RWF_DSYNC writes Ritesh Harjani (IBM)
2025-08-01 20:18 ` Zorro Lang
2025-08-01 20:29 ` Ritesh Harjani
2025-08-02 8:17 ` Zorro Lang
2025-08-03 5:23 ` Ritesh Harjani
2025-07-31 15:00 ` [RFC 1/2] aio-dio-write-verify: Add O_DSYNC option Jan Kara
2025-08-02 8:18 ` Zorro Lang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).