[LTP] [PATCH v3 1/2] nfs_lib.sh: Add support for extra NFS mount options

[LTP] [PATCH v3 1/2] nfs_lib.sh: Add support for extra NFS mount options

[Martin Doucha]

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---

Changes since v2: New patch

 testcases/network/nfs/nfs_stress/nfs_lib.sh | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/testcases/network/nfs/nfs_stress/nfs_lib.sh b/testcases/network/nfs/nfs_stress/nfs_lib.sh
index d3de3b7f1..f0da1fc99 100644
--- a/testcases/network/nfs/nfs_stress/nfs_lib.sh
+++ b/testcases/network/nfs/nfs_stress/nfs_lib.sh
@@ -168,6 +168,7 @@ nfs_setup()
 	local local_dir
 	local remote_dir
 	local mount_dir
+	local nfs_opts
 
 	if [ "$(stat -f . | grep "Type: nfs")" ]; then
 		tst_brk TCONF "Cannot run nfs-stress test on mounted NFS"
@@ -192,8 +193,14 @@ nfs_setup()
 		remote_dir="$(get_remote_dir $i $type)"
 		nfs_setup_server "$remote_dir" "$(($$ + n))"
 		local_dir="$(get_local_dir $i $n)"
+		nfs_opts="-o proto=$type,vers=$i"
+
+		if [ ! -z "$NFS_MOUNT_OPTS" ]; then
+			nfs_opts="$nfs_opts,$NFS_MOUNT_OPTS"
+		fi
+
 		tst_res TINFO "Mounting $local_dir"
-		nfs_mount "$local_dir" "$remote_dir" "-o proto=$type,vers=$i"
+		nfs_mount "$local_dir" "$remote_dir" "$nfs_opts"
 
 		n=$(( n + 1 ))
 	done
-- 
2.47.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v3 2/2] Add test for data integrity over NFS

[Martin Doucha]

Add NFS test which checks data integrity of random writes into a file,
with both buffered and direct I/O.

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---

Changes since v1: Added TST_TIMEOUT

Changes since v2: Force NFS blocksize to the underlying FS blocksize

By default, NFS uses very large blocksize (up to 1MB) to optimize network
communication. For the FS integrity test this means a significant slowdown
because the size of the test file and I/O operations increase with
blocksize. Autodetect the blocksize of the underlying filesystem and force
the same value during NFS mount to speed things up.

The TST_TIMEOUT value is calculated as worst-case scenario for 64KB
blocksize. On reasonably modern x86_64 machines, the new tests take about
6 minutes to complete in total (with Btrfs, Ext4 and XFS being tested).

 runtest/net.nfs                           | 11 ++++++
 testcases/network/nfs/nfs_stress/nfs10.sh | 47 +++++++++++++++++++++++
 2 files changed, 58 insertions(+)
 create mode 100755 testcases/network/nfs/nfs_stress/nfs10.sh

diff --git a/runtest/net.nfs b/runtest/net.nfs
index 7f84457bc..fef993da8 100644
--- a/runtest/net.nfs
+++ b/runtest/net.nfs
@@ -94,6 +94,17 @@ nfs09_v40_ip6t nfs09.sh -6 -v 4 -t tcp
 nfs09_v41_ip6t nfs09.sh -6 -v 4.1 -t tcp
 nfs09_v42_ip6t nfs09.sh -6 -v 4.2 -t tcp
 
+nfs10_v30_ip4u nfs10.sh -v 3 -t udp
+nfs10_v30_ip4t nfs10.sh -v 3 -t tcp
+nfs10_v40_ip4t nfs10.sh -v 4 -t tcp
+nfs10_v41_ip4t nfs10.sh -v 4.1 -t tcp
+nfs10_v42_ip4t nfs10.sh -v 4.2 -t tcp
+nfs10_v30_ip6u nfs10.sh -6 -v 3 -t udp
+nfs10_v30_ip6t nfs10.sh -6 -v 3 -t tcp
+nfs10_v40_ip6t nfs10.sh -6 -v 4 -t tcp
+nfs10_v41_ip6t nfs10.sh -6 -v 4.1 -t tcp
+nfs10_v42_ip6t nfs10.sh -6 -v 4.2 -t tcp
+
 nfslock01_v30_ip4u nfslock01.sh -v 3 -t udp
 nfslock01_v30_ip4t nfslock01.sh -v 3 -t tcp
 nfslock01_v40_ip4t nfslock01.sh -v 4 -t tcp
diff --git a/testcases/network/nfs/nfs_stress/nfs10.sh b/testcases/network/nfs/nfs_stress/nfs10.sh
new file mode 100755
index 000000000..2fca4f916
--- /dev/null
+++ b/testcases/network/nfs/nfs_stress/nfs10.sh
@@ -0,0 +1,47 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (C) 2024 SUSE LLC <mdoucha@suse.cz>
+#
+# DESCRIPTION: Verify data integrity over NFS, with and without O_DIRECT
+
+TST_CNT=4
+TST_SETUP="nfs10_setup"
+TST_TESTFUNC="do_test"
+TST_DEVICE_SIZE=1024
+TST_TIMEOUT=660
+
+nfs10_setup()
+{
+	local bsize=$(stat -f -c %s .)
+
+	bsize=$(( $bsize < 1024 ? 1024 : $bsize ))
+	NFS_MOUNT_OPTS="rsize=$bsize,wsize=$bsize"
+	nfs_setup
+}
+
+do_test1()
+{
+	tst_res TINFO "Testing buffered write, buffered read"
+	EXPECT_PASS fsplough -c 512 -d "$PWD"
+}
+
+do_test2()
+{
+	tst_res TINFO "Testing buffered write, direct read"
+	EXPECT_PASS fsplough -c 512 -R -d "$PWD"
+}
+
+do_test3()
+{
+	tst_res TINFO "Testing direct write, buffered read"
+	EXPECT_PASS fsplough -c 512 -W -d "$PWD"
+}
+
+do_test4()
+{
+	tst_res TINFO "Testing direct write, direct read"
+	EXPECT_PASS fsplough -c 512 -RW -d "$PWD"
+}
+
+. nfs_lib.sh
+tst_run
-- 
2.47.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3 1/2] nfs_lib.sh: Add support for extra NFS mount options

[Petr Vorel]

Hi Martin,

...
> +		nfs_opts="-o proto=$type,vers=$i"
> +
> +		if [ ! -z "$NFS_MOUNT_OPTS" ]; then

We have -n for non-null string. Even just [ "$NFS_MOUNT_OPTS" ] would work,
but -n is more obvious. I can change it before merge:

		if [ -n "$NFS_MOUNT_OPTS" ]; then

Reviewed-by: Petr Vorel <pvorel@suse.cz>

Kind regards,
Petr

> +			nfs_opts="$nfs_opts,$NFS_MOUNT_OPTS"
> +		fi
> +
>  		tst_res TINFO "Mounting $local_dir"
> -		nfs_mount "$local_dir" "$remote_dir" "-o proto=$type,vers=$i"
> +		nfs_mount "$local_dir" "$remote_dir" "$nfs_opts"

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3 2/2] Add test for data integrity over NFS

[Petr Vorel]

> Add NFS test which checks data integrity of random writes into a file,
> with both buffered and direct I/O.

> Signed-off-by: Martin Doucha <mdoucha@suse.cz>
> ---

> Changes since v1: Added TST_TIMEOUT

> Changes since v2: Force NFS blocksize to the underlying FS blocksize

> By default, NFS uses very large blocksize (up to 1MB) to optimize network
> communication. For the FS integrity test this means a significant slowdown
> because the size of the test file and I/O operations increase with
> blocksize. Autodetect the blocksize of the underlying filesystem and force
> the same value during NFS mount to speed things up.

> The TST_TIMEOUT value is calculated as worst-case scenario for 64KB
> blocksize. On reasonably modern x86_64 machines, the new tests take about
> 6 minutes to complete in total (with Btrfs, Ext4 and XFS being tested).

+1. I also did some testing. BTW TST_TIMEOUT is for single filesystem, thus 11
min is really safe enough even for slow machines like emulated risc-v.

> +++ b/testcases/network/nfs/nfs_stress/nfs10.sh
> @@ -0,0 +1,47 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (C) 2024 SUSE LLC <mdoucha@suse.cz>
> +#
> +# DESCRIPTION: Verify data integrity over NFS, with and without O_DIRECT
> +
> +TST_CNT=4
> +TST_SETUP="nfs10_setup"
> +TST_TESTFUNC="do_test"
> +TST_DEVICE_SIZE=1024
> +TST_TIMEOUT=660
> +
> +nfs10_setup()
> +{
> +	local bsize=$(stat -f -c %s .)
> +
> +	bsize=$(( $bsize < 1024 ? 1024 : $bsize ))

nit: this would work if stat is not installed (minimal systems) or fails for
some reason:

if [ -z "$bsize" ] || [ "$bsize" -lt 1024 ]; then
	bsize=1024
fi

If you agree, I can change it during merge.
Otherwise LGTM.

Thank you!
Reviewed-by: Petr Vorel <pvorel@suse.cz>

Kind regards,
Petr

> +	NFS_MOUNT_OPTS="rsize=$bsize,wsize=$bsize"
> +	nfs_setup

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3 1/2] nfs_lib.sh: Add support for extra NFS mount options

[Martin Doucha]

Hi,

On 05. 12. 24 11:34, Petr Vorel wrote:
> Hi Martin,
> 
> ...
>> +		nfs_opts="-o proto=$type,vers=$i"
>> +
>> +		if [ ! -z "$NFS_MOUNT_OPTS" ]; then
> 
> We have -n for non-null string. Even just [ "$NFS_MOUNT_OPTS" ] would work,
> but -n is more obvious. I can change it before merge:
> 
> 		if [ -n "$NFS_MOUNT_OPTS" ]; then

Please do, thanks.

> Reviewed-by: Petr Vorel <pvorel@suse.cz>
> 
> Kind regards,
> Petr
> 
>> +			nfs_opts="$nfs_opts,$NFS_MOUNT_OPTS"
>> +		fi
>> +
>>   		tst_res TINFO "Mounting $local_dir"
>> -		nfs_mount "$local_dir" "$remote_dir" "-o proto=$type,vers=$i"
>> +		nfs_mount "$local_dir" "$remote_dir" "$nfs_opts"


-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3 2/2] Add test for data integrity over NFS

[Martin Doucha]

Hi,

On 05. 12. 24 12:10, Petr Vorel wrote:
>> +
>> +nfs10_setup()
>> +{
>> +	local bsize=$(stat -f -c %s .)
>> +
>> +	bsize=$(( $bsize < 1024 ? 1024 : $bsize ))
> 
> nit: this would work if stat is not installed (minimal systems) or fails for
> some reason:
> 
> if [ -z "$bsize" ] || [ "$bsize" -lt 1024 ]; then
> 	bsize=1024
> fi
> 
> If you agree, I can change it during merge.
> Otherwise LGTM.

Please do, thank you.

-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3 2/2] Add test for data integrity over NFS

[Petr Vorel]

Hi,

> +++ b/testcases/network/nfs/nfs_stress/nfs10.sh
> @@ -0,0 +1,47 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (C) 2024 SUSE LLC <mdoucha@suse.cz>
> +#
> +# DESCRIPTION: Verify data integrity over NFS, with and without O_DIRECT
> +
> +TST_CNT=4
> +TST_SETUP="nfs10_setup"
> +TST_TESTFUNC="do_test"
> +TST_DEVICE_SIZE=1024

FYI requiring 1GB free space can break the test (even for UDP which should be
mostly skipped with TCONF due UDP being disabled on mainline kernel).
On UDP it runs quickly (< 10s), but on TCP it runs for whole timeout (~ 33 min).

This is not a new limitation in shell API, it always existed. But recent commit
8532eb2c61 ("Add support for setting loop device size in shell tests")
increased change of hitting the problem. I don't think it should block this
patchset, I'll send a patch fixing the API soon.

Kind regards,
Petr

# ./nfs10.sh -v 3 -t udp
nfs10 1 TINFO: Running: nfs10.sh -v 3 -t udp

nfs10 1 TINFO: 10.0.0.2/24 -- 10.0.0.1/24
nfs10 1 TINFO: fd00:1:1:1::2/64 -- fd00:1:1:1::1/64
nfs10 1 TINFO: Using /tmp/LTP_nfs10.XI7vLEjIkA as tmpdir (tmpfs filesystem)
tst_device.c:299: TWARN: Failed to create test_dev.img: ENOSPC (28)

Usage:
tst_device acquire [size [filename]]
tst_device release /path/to/device
tst_device clear /path/to/device

nfs10 1 TWARN: Failed to acquire device
tst_supported_fs_types.c:169: TINFO: Skipping ext2 as requested by the test
tst_supported_fs_types.c:169: TINFO: Skipping ext3 as requested by the test
tst_supported_fs_types.c:97: TINFO: Kernel supports ext4
tst_supported_fs_types.c:62: TINFO: mkfs.ext4 does exist
tst_supported_fs_types.c:97: TINFO: Kernel supports xfs
tst_supported_fs_types.c:62: TINFO: mkfs.xfs does exist
tst_supported_fs_types.c:97: TINFO: Kernel supports btrfs
tst_supported_fs_types.c:62: TINFO: mkfs.btrfs does exist
tst_supported_fs_types.c:97: TINFO: Kernel supports bcachefs
tst_supported_fs_types.c:62: TINFO: mkfs.bcachefs does exist
tst_supported_fs_types.c:169: TINFO: Skipping vfat as requested by the test
tst_supported_fs_types.c:169: TINFO: Skipping exfat as requested by the test
tst_supported_fs_types.c:169: TINFO: Skipping ntfs as requested by the test
tst_supported_fs_types.c:169: TINFO: Skipping tmpfs as requested by the test
nfs10 1 TINFO: === Testing on ext4 ===
tst_device.c:389: TWARN: Failed to clear 512k block on

Usage:
tst_device acquire [size [filename]]
tst_device release /path/to/device
tst_device clear /path/to/device

...

> +TST_TIMEOUT=660
...

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3 2/2] Add test for data integrity over NFS

[Petr Vorel]

Hi Martin,

> > > +	bsize=$(( $bsize < 1024 ? 1024 : $bsize ))

> > nit: this would work if stat is not installed (minimal systems) or fails for
> > some reason:

> > if [ -z "$bsize" ] || [ "$bsize" -lt 1024 ]; then
> > 	bsize=1024
> > fi

> > If you agree, I can change it during merge.
> > Otherwise LGTM.

> Please do, thank you.

The remaining two patches merged with agreed change. Thanks!
I suppose you send a patch to modify vectorize_buffer() fsplough.c.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp