[LTP] [PATCH 0/5] Add LVM and NFS tests for file data integrity

[LTP] [PATCH 0/5] Add LVM and NFS tests for file data integrity

[Martin Doucha]

Most of the existing read/write tests don't verify whether the data written
to the filesystem can also be correctly read back. Add a new reusable test
program that will check file data integrity. The test randomly switches
between multiple read/write syscalls and allows setting direct (unbuffered)
I/O mode independently for read or write operations.

Also enable the new test in LVM and NFS runfiles with all combinations
of direct and buffered I/O operations.

Martin Doucha (5):
  Move preadv()/pwritev() backup definitions to LAPI
  Add safe readv()/writev() functions
  Add test for file data integrity
  Add support for setting loop device size in shell tests
  Add test for data integrity over NFS

 doc/developers/writing_tests.rst              |   2 +-
 include/lapi/pwritev2.h                       |  26 --
 include/lapi/{preadv2.h => uio.h}             |  30 +-
 include/tst_safe_macros.h                     |  13 +
 include/tst_safe_prw.h                        |  58 ++++
 lib/tst_safe_macros.c                         |  45 +++
 runtest/net.nfs                               |  11 +
 testcases/kernel/fs/fsplough/.gitignore       |   1 +
 testcases/kernel/fs/fsplough/Makefile         |   8 +
 testcases/kernel/fs/fsplough/fsplough.c       | 279 ++++++++++++++++++
 testcases/kernel/syscalls/preadv/preadv.h     |  22 --
 testcases/kernel/syscalls/preadv/preadv01.c   |   2 +-
 testcases/kernel/syscalls/preadv/preadv02.c   |   2 +-
 testcases/kernel/syscalls/preadv/preadv03.c   |   2 +-
 testcases/kernel/syscalls/preadv2/preadv201.c |   2 +-
 testcases/kernel/syscalls/preadv2/preadv202.c |   2 +-
 testcases/kernel/syscalls/preadv2/preadv203.c |   2 +-
 testcases/kernel/syscalls/pwritev/pwritev.h   |  22 --
 testcases/kernel/syscalls/pwritev/pwritev01.c |   2 +-
 testcases/kernel/syscalls/pwritev/pwritev02.c |   2 +-
 testcases/kernel/syscalls/pwritev/pwritev03.c |   2 +-
 .../kernel/syscalls/pwritev2/pwritev201.c     |   2 +-
 .../kernel/syscalls/pwritev2/pwritev202.c     |   2 +-
 testcases/lib/tst_test.sh                     |   3 +-
 testcases/misc/lvm/datafiles/runfile.tpl      |   4 +
 testcases/network/nfs/nfs_stress/nfs10.sh     |  36 +++
 26 files changed, 498 insertions(+), 84 deletions(-)
 delete mode 100644 include/lapi/pwritev2.h
 rename include/lapi/{preadv2.h => uio.h} (50%)
 create mode 100644 testcases/kernel/fs/fsplough/.gitignore
 create mode 100644 testcases/kernel/fs/fsplough/Makefile
 create mode 100644 testcases/kernel/fs/fsplough/fsplough.c
 delete mode 100644 testcases/kernel/syscalls/preadv/preadv.h
 delete mode 100644 testcases/kernel/syscalls/pwritev/pwritev.h
 create mode 100755 testcases/network/nfs/nfs_stress/nfs10.sh

-- 
2.46.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 1/5] Move preadv()/pwritev() backup definitions to LAPI

[Martin Doucha]

Merge existing LAPI definitions of preadv2()/pwritev2() to lapi/uio.h
and add preadv()/pwritev() definitions from test code. Also fix minor
bugs in return value type and offset handling.

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---
 include/lapi/pwritev2.h                       | 26 -----------------
 include/lapi/{preadv2.h => uio.h}             | 29 ++++++++++++++++++-
 testcases/kernel/syscalls/preadv/preadv.h     | 22 --------------
 testcases/kernel/syscalls/preadv/preadv01.c   |  2 +-
 testcases/kernel/syscalls/preadv/preadv02.c   |  2 +-
 testcases/kernel/syscalls/preadv/preadv03.c   |  2 +-
 testcases/kernel/syscalls/preadv2/preadv201.c |  2 +-
 testcases/kernel/syscalls/preadv2/preadv202.c |  2 +-
 testcases/kernel/syscalls/preadv2/preadv203.c |  2 +-
 testcases/kernel/syscalls/pwritev/pwritev.h   | 22 --------------
 testcases/kernel/syscalls/pwritev/pwritev01.c |  2 +-
 testcases/kernel/syscalls/pwritev/pwritev02.c |  2 +-
 testcases/kernel/syscalls/pwritev/pwritev03.c |  2 +-
 .../kernel/syscalls/pwritev2/pwritev201.c     |  2 +-
 .../kernel/syscalls/pwritev2/pwritev202.c     |  2 +-
 15 files changed, 39 insertions(+), 82 deletions(-)
 delete mode 100644 include/lapi/pwritev2.h
 rename include/lapi/{preadv2.h => uio.h} (51%)
 delete mode 100644 testcases/kernel/syscalls/preadv/preadv.h
 delete mode 100644 testcases/kernel/syscalls/pwritev/pwritev.h

diff --git a/include/lapi/pwritev2.h b/include/lapi/pwritev2.h
deleted file mode 100644
index 48b53f463..000000000
--- a/include/lapi/pwritev2.h
+++ /dev/null
@@ -1,26 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Copyright (c) 2019 FUJITSU LIMITED. All rights reserved.
- * Author: Jinhui Huang <huangjh.jy@cn.fujitsu.com>
- */
-
-#ifndef LAPI_PWRITEV2_H__
-#define LAPI_PWRITEV2_H__
-
-#include "config.h"
-#include "lapi/syscalls.h"
-
-#if !defined(HAVE_PWRITEV2)
-
-/* LO_HI_LONG taken from glibc */
-# define LO_HI_LONG(val) (long) (val), (long) (((uint64_t) (val)) >> 32)
-
-static inline ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt,
-                               off_t offset, int flags)
-{
-	return tst_syscall(__NR_pwritev2, fd, iov, iovcnt,
-			   LO_HI_LONG(offset), flags);
-}
-#endif
-
-#endif /* LAPI_PWRITEV2_H__ */
diff --git a/include/lapi/preadv2.h b/include/lapi/uio.h
similarity index 51%
rename from include/lapi/preadv2.h
rename to include/lapi/uio.h
index db89547e2..a78103d99 100644
--- a/include/lapi/preadv2.h
+++ b/include/lapi/uio.h
@@ -14,11 +14,27 @@
 # define RWF_NOWAIT 0x00000008
 #endif
 
-#if !defined(HAVE_PREADV2)
 
 /* LO_HI_LONG taken from glibc */
 # define LO_HI_LONG(val) (long) (val), (long) (((uint64_t) (val)) >> 32)
 
+#if !defined(HAVE_PREADV)
+static inline ssize_t preadv(int fd, const struct iovec *iov, int iovcnt,
+	off_t offset)
+{
+	return tst_syscall(__NR_preadv, fd, iov, iovcnt, LO_HI_LONG(offset));
+}
+#endif
+
+#if !defined(HAVE_PWRITEV)
+static inline ssize_t pwritev(int fd, const struct iovec *iov, int iovcnt,
+	off_t offset)
+{
+	return tst_syscall(__NR_pwritev, fd, iov, iovcnt, LO_HI_LONG(offset));
+}
+#endif
+
+#if !defined(HAVE_PREADV2)
 static inline ssize_t preadv2(int fd, const struct iovec *iov, int iovcnt,
                               off_t offset, int flags)
 {
@@ -27,4 +43,15 @@ static inline ssize_t preadv2(int fd, const struct iovec *iov, int iovcnt,
 }
 #endif
 
+#if !defined(HAVE_PWRITEV2)
+static inline ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt,
+                               off_t offset, int flags)
+{
+	return tst_syscall(__NR_pwritev2, fd, iov, iovcnt,
+			   LO_HI_LONG(offset), flags);
+}
+#endif
+
+#undef LO_HI_LONG
+
 #endif /* LAPI_PREADV2_H__ */
diff --git a/testcases/kernel/syscalls/preadv/preadv.h b/testcases/kernel/syscalls/preadv/preadv.h
deleted file mode 100644
index c715715b1..000000000
--- a/testcases/kernel/syscalls/preadv/preadv.h
+++ /dev/null
@@ -1,22 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later
- *
- * Copyright (c) 2015 Fujitsu Ltd.
- * Author: Xiao Yang <yangx.jy@cn.fujitsu.com>
- * Copyright (c) Linux Test Project, 2016-2023
- */
-
-#ifndef PREADV_H
-#define PREADV_H
-
-#include <sys/types.h>
-#include "config.h"
-#include "lapi/syscalls.h"
-
-#if !defined(HAVE_PREADV)
-int preadv(int fd, const struct iovec *iov, int iovcnt, off_t offset)
-{
-	return tst_syscall(__NR_preadv, fd, iov, iovcnt, offset);
-}
-#endif
-
-#endif /* RREADV_H */
diff --git a/testcases/kernel/syscalls/preadv/preadv01.c b/testcases/kernel/syscalls/preadv/preadv01.c
index 871b3ab42..d5e5bd95f 100644
--- a/testcases/kernel/syscalls/preadv/preadv01.c
+++ b/testcases/kernel/syscalls/preadv/preadv01.c
@@ -20,7 +20,7 @@
 #include <sys/uio.h>
 
 #include "tst_test.h"
-#include "preadv.h"
+#include "lapi/uio.h"
 
 #define CHUNK           64
 
diff --git a/testcases/kernel/syscalls/preadv/preadv02.c b/testcases/kernel/syscalls/preadv/preadv02.c
index 9977a4f48..19b194314 100644
--- a/testcases/kernel/syscalls/preadv/preadv02.c
+++ b/testcases/kernel/syscalls/preadv/preadv02.c
@@ -23,7 +23,7 @@
 #include <sys/uio.h>
 #include <unistd.h>
 #include "tst_test.h"
-#include "preadv.h"
+#include "lapi/uio.h"
 
 #define CHUNK           64
 
diff --git a/testcases/kernel/syscalls/preadv/preadv03.c b/testcases/kernel/syscalls/preadv/preadv03.c
index 558d85050..997e7a07c 100644
--- a/testcases/kernel/syscalls/preadv/preadv03.c
+++ b/testcases/kernel/syscalls/preadv/preadv03.c
@@ -22,7 +22,7 @@
 #include <sys/ioctl.h>
 #include <sys/mount.h>
 #include "tst_test.h"
-#include "preadv.h"
+#include "lapi/uio.h"
 
 #define MNTPOINT	"mntpoint"
 #define FNAME	MNTPOINT"/file"
diff --git a/testcases/kernel/syscalls/preadv2/preadv201.c b/testcases/kernel/syscalls/preadv2/preadv201.c
index 110977290..50f5a2e01 100644
--- a/testcases/kernel/syscalls/preadv2/preadv201.c
+++ b/testcases/kernel/syscalls/preadv2/preadv201.c
@@ -20,7 +20,7 @@
 #include <sys/uio.h>
 
 #include "tst_test.h"
-#include "lapi/preadv2.h"
+#include "lapi/uio.h"
 
 #define CHUNK           64
 
diff --git a/testcases/kernel/syscalls/preadv2/preadv202.c b/testcases/kernel/syscalls/preadv2/preadv202.c
index 4e1e82ebd..fd3b7c9d7 100644
--- a/testcases/kernel/syscalls/preadv2/preadv202.c
+++ b/testcases/kernel/syscalls/preadv2/preadv202.c
@@ -25,7 +25,7 @@
 #include <unistd.h>
 
 #include "tst_test.h"
-#include "lapi/preadv2.h"
+#include "lapi/uio.h"
 
 #define CHUNK           64
 
diff --git a/testcases/kernel/syscalls/preadv2/preadv203.c b/testcases/kernel/syscalls/preadv2/preadv203.c
index c87deb674..42b265680 100644
--- a/testcases/kernel/syscalls/preadv2/preadv203.c
+++ b/testcases/kernel/syscalls/preadv2/preadv203.c
@@ -45,7 +45,7 @@
 
 #include "tst_test.h"
 #include "tst_safe_pthread.h"
-#include "lapi/preadv2.h"
+#include "lapi/uio.h"
 
 #define CHUNK_SZ 4123
 #define CHUNKS 60
diff --git a/testcases/kernel/syscalls/pwritev/pwritev.h b/testcases/kernel/syscalls/pwritev/pwritev.h
deleted file mode 100644
index e657dc7e6..000000000
--- a/testcases/kernel/syscalls/pwritev/pwritev.h
+++ /dev/null
@@ -1,22 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later
- *
- * Copyright (c) 2015 Fujitsu Ltd.
- * Author: Xiao Yang <yangx.jy@cn.fujitsu.com>
- * Copyright (c) Linux Test Project, 2016-2023
- */
-
-#ifndef PWRITEV_H
-#define PWRITEV_H
-
-#include <sys/types.h>
-#include "config.h"
-#include "lapi/syscalls.h"
-
-#if !defined(HAVE_PWRITEV)
-int pwritev(int fd, const struct iovec *iov, int iovcnt, off_t offset)
-{
-	return tst_syscall(__NR_pwritev, fd, iov, iovcnt, offset);
-}
-#endif
-
-#endif /* PWRITEV_H */
diff --git a/testcases/kernel/syscalls/pwritev/pwritev01.c b/testcases/kernel/syscalls/pwritev/pwritev01.c
index f5fce81f2..1a4f672bf 100644
--- a/testcases/kernel/syscalls/pwritev/pwritev01.c
+++ b/testcases/kernel/syscalls/pwritev/pwritev01.c
@@ -18,7 +18,7 @@
 #include <string.h>
 #include <sys/uio.h>
 #include "tst_test.h"
-#include "pwritev.h"
+#include "lapi/uio.h"
 #include "tst_safe_prw.h"
 
 #define	CHUNK		64
diff --git a/testcases/kernel/syscalls/pwritev/pwritev02.c b/testcases/kernel/syscalls/pwritev/pwritev02.c
index 59a286847..25eb242f5 100644
--- a/testcases/kernel/syscalls/pwritev/pwritev02.c
+++ b/testcases/kernel/syscalls/pwritev/pwritev02.c
@@ -22,7 +22,7 @@
 #include <sys/uio.h>
 #include <unistd.h>
 #include "tst_test.h"
-#include "pwritev.h"
+#include "lapi/uio.h"
 
 #define CHUNK           64
 
diff --git a/testcases/kernel/syscalls/pwritev/pwritev03.c b/testcases/kernel/syscalls/pwritev/pwritev03.c
index 1bf9d5731..81ccee9be 100644
--- a/testcases/kernel/syscalls/pwritev/pwritev03.c
+++ b/testcases/kernel/syscalls/pwritev/pwritev03.c
@@ -22,7 +22,7 @@
 #include <sys/ioctl.h>
 #include <sys/mount.h>
 #include "tst_test.h"
-#include "pwritev.h"
+#include "lapi/uio.h"
 #include "tst_safe_prw.h"
 
 #define MNTPOINT	"mntpoint"
diff --git a/testcases/kernel/syscalls/pwritev2/pwritev201.c b/testcases/kernel/syscalls/pwritev2/pwritev201.c
index 987412ba8..ae9047cdd 100644
--- a/testcases/kernel/syscalls/pwritev2/pwritev201.c
+++ b/testcases/kernel/syscalls/pwritev2/pwritev201.c
@@ -22,7 +22,7 @@
 #include <sys/uio.h>
 
 #include "tst_test.h"
-#include "lapi/pwritev2.h"
+#include "lapi/uio.h"
 #include "tst_safe_prw.h"
 
 #define CHUNK	64
diff --git a/testcases/kernel/syscalls/pwritev2/pwritev202.c b/testcases/kernel/syscalls/pwritev2/pwritev202.c
index 5a1c4c6b6..e4d80070d 100644
--- a/testcases/kernel/syscalls/pwritev2/pwritev202.c
+++ b/testcases/kernel/syscalls/pwritev2/pwritev202.c
@@ -26,7 +26,7 @@
 #include <unistd.h>
 
 #include "tst_test.h"
-#include "lapi/pwritev2.h"
+#include "lapi/uio.h"
 
 #define CHUNK	64
 
-- 
2.46.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 2/5] Add safe readv()/writev() functions

[Martin Doucha]

Add safe macros for vectorized I/O functions:
- SAFE_READV()
- SAFE_PREADV()
- SAFE_WRITEV()
- SAFE_PWRITEV()

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---
 include/lapi/uio.h        |  1 +
 include/tst_safe_macros.h | 13 +++++++++
 include/tst_safe_prw.h    | 58 +++++++++++++++++++++++++++++++++++++++
 lib/tst_safe_macros.c     | 45 ++++++++++++++++++++++++++++++
 4 files changed, 117 insertions(+)

diff --git a/include/lapi/uio.h b/include/lapi/uio.h
index a78103d99..0ad2faacf 100644
--- a/include/lapi/uio.h
+++ b/include/lapi/uio.h
@@ -7,6 +7,7 @@
 #ifndef LAPI_PREADV2_H__
 #define LAPI_PREADV2_H__
 
+#include <sys/uio.h>
 #include "config.h"
 #include "lapi/syscalls.h"
 
diff --git a/include/tst_safe_macros.h b/include/tst_safe_macros.h
index 4df23e602..a9fa7dbe1 100644
--- a/include/tst_safe_macros.h
+++ b/include/tst_safe_macros.h
@@ -14,6 +14,7 @@
 #include <sys/stat.h>
 #include <sys/vfs.h>
 #include <sys/sysinfo.h>
+#include <sys/uio.h>
 #include <fcntl.h>
 #include <libgen.h>
 #include <signal.h>
@@ -515,4 +516,16 @@ int safe_symlinkat(const char *file, const int lineno,
 #define SAFE_SYMLINKAT(oldpath, newdirfd, newpath) \
 	safe_symlinkat(__FILE__, __LINE__, (oldpath), (newdirfd), (newpath))
 
+ssize_t safe_readv(const char *file, const int lineno, char len_strict,
+	int fildes, const struct iovec *iov, int iovcnt);
+#define SAFE_READV(len_strict, fildes, iov, iovcnt) \
+	safe_readv(__FILE__, __LINE__, (len_strict), (fildes), \
+		(iov), (iovcnt))
+
+ssize_t safe_writev(const char *file, const int lineno, char len_strict,
+	int fildes, const struct iovec *iov, int iovcnt);
+#define SAFE_WRITEV(len_strict, fildes, iov, iovcnt) \
+	safe_writev(__FILE__, __LINE__, (len_strict), (fildes), \
+		(iov), (iovcnt))
+
 #endif /* TST_SAFE_MACROS_H__ */
diff --git a/include/tst_safe_prw.h b/include/tst_safe_prw.h
index 2e506cb41..349fb46b4 100644
--- a/include/tst_safe_prw.h
+++ b/include/tst_safe_prw.h
@@ -5,6 +5,8 @@
 #ifndef TST_SAFE_PRW_H__
 #define TST_SAFE_PRW_H__
 
+#include "lapi/uio.h"
+
 static inline ssize_t safe_pread(const char *file, const int lineno,
 		char len_strict, int fildes, void *buf, size_t nbyte,
 		off_t offset)
@@ -52,4 +54,60 @@ static inline ssize_t safe_pwrite(const char *file, const int lineno,
 	safe_pwrite(__FILE__, __LINE__, (len_strict), (fildes), \
 	            (buf), (nbyte), (offset))
 
+static inline ssize_t safe_preadv(const char *file, const int lineno,
+	char len_strict, int fildes, const struct iovec *iov, int iovcnt,
+	off_t offset)
+{
+	ssize_t rval, nbyte;
+	int i;
+
+	for (i = 0, nbyte = 0; i < iovcnt; i++)
+		nbyte += iov[i].iov_len;
+
+	rval = preadv(fildes, iov, iovcnt, offset);
+
+	if (rval == -1 || (len_strict && rval != nbyte)) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"preadv(%d,%p,%d,%lld) failed",
+			fildes, iov, iovcnt, (long long)offset);
+	} else if (rval < 0) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"Invalid preadv(%d,%p,%d,%lld) return value %zd",
+			fildes, iov, iovcnt, (long long)offset, rval);
+	}
+
+	return rval;
+}
+#define SAFE_PREADV(len_strict, fildes, iov, iovcnt, offset) \
+	safe_preadv(__FILE__, __LINE__, (len_strict), (fildes), \
+		(iov), (iovcnt), (offset))
+
+static inline ssize_t safe_pwritev(const char *file, const int lineno,
+	char len_strict, int fildes, const struct iovec *iov, int iovcnt,
+	off_t offset)
+{
+	ssize_t rval, nbyte;
+	int i;
+
+	for (i = 0, nbyte = 0; i < iovcnt; i++)
+		nbyte += iov[i].iov_len;
+
+	rval = pwritev(fildes, iov, iovcnt, offset);
+
+	if (rval == -1 || (len_strict && rval != nbyte)) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"pwritev(%d,%p,%d,%lld) failed",
+			fildes, iov, iovcnt, (long long)offset);
+	} else if (rval < 0) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"Invalid pwritev(%d,%p,%d,%lld) return value %zd",
+			fildes, iov, iovcnt, (long long)offset, rval);
+	}
+
+	return rval;
+}
+#define SAFE_PWRITEV(len_strict, fildes, iov, iovcnt, offset) \
+	safe_pwritev(__FILE__, __LINE__, (len_strict), (fildes), \
+		(iov), (iovcnt), (offset))
+
 #endif /* SAFE_PRW_H__ */
diff --git a/lib/tst_safe_macros.c b/lib/tst_safe_macros.c
index 868ebc08e..633b00404 100644
--- a/lib/tst_safe_macros.c
+++ b/lib/tst_safe_macros.c
@@ -732,6 +732,29 @@ int safe_prctl(const char *file, const int lineno,
 	return rval;
 }
 
+ssize_t safe_readv(const char *file, const int lineno, char len_strict,
+	int fildes, const struct iovec *iov, int iovcnt)
+{
+	ssize_t rval, nbyte;
+	int i;
+
+	for (i = 0, nbyte = 0; i < iovcnt; i++)
+		nbyte += iov[i].iov_len;
+
+	rval = readv(fildes, iov, iovcnt);
+
+	if (rval == -1 || (len_strict && rval != nbyte)) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"readv(%d,%p,%d) failed", fildes, iov, iovcnt);
+	} else if (rval < 0) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"Invalid readv(%d,%p,%d) return value %zd",
+			fildes, iov, iovcnt, rval);
+	}
+
+	return rval;
+}
+
 int safe_symlinkat(const char *file, const int lineno,
                  const char *oldpath, const int newdirfd, const char *newpath)
 {
@@ -751,3 +774,25 @@ int safe_symlinkat(const char *file, const int lineno,
 	return rval;
 }
 
+ssize_t safe_writev(const char *file, const int lineno, char len_strict,
+	int fildes, const struct iovec *iov, int iovcnt)
+{
+	ssize_t rval, nbyte;
+	int i;
+
+	for (i = 0, nbyte = 0; i < iovcnt; i++)
+		nbyte += iov[i].iov_len;
+
+	rval = writev(fildes, iov, iovcnt);
+
+	if (rval == -1 || (len_strict && rval != nbyte)) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"writev(%d,%p,%d) failed", fildes, iov, iovcnt);
+	} else if (rval < 0) {
+		tst_brk_(file, lineno, TBROK | TERRNO,
+			"Invalid writev(%d,%p,%d) return value %zd",
+			fildes, iov, iovcnt, rval);
+	}
+
+	return rval;
+}
-- 
2.46.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 3/5] Add test for file data integrity

[Martin Doucha]

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---
 testcases/kernel/fs/fsplough/.gitignore  |   1 +
 testcases/kernel/fs/fsplough/Makefile    |   8 +
 testcases/kernel/fs/fsplough/fsplough.c  | 279 +++++++++++++++++++++++
 testcases/misc/lvm/datafiles/runfile.tpl |   4 +
 4 files changed, 292 insertions(+)
 create mode 100644 testcases/kernel/fs/fsplough/.gitignore
 create mode 100644 testcases/kernel/fs/fsplough/Makefile
 create mode 100644 testcases/kernel/fs/fsplough/fsplough.c

diff --git a/testcases/kernel/fs/fsplough/.gitignore b/testcases/kernel/fs/fsplough/.gitignore
new file mode 100644
index 000000000..34fee75c7
--- /dev/null
+++ b/testcases/kernel/fs/fsplough/.gitignore
@@ -0,0 +1 @@
+/fsplough
diff --git a/testcases/kernel/fs/fsplough/Makefile b/testcases/kernel/fs/fsplough/Makefile
new file mode 100644
index 000000000..887756d04
--- /dev/null
+++ b/testcases/kernel/fs/fsplough/Makefile
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (c) 2017 Linux Test Project
+
+top_srcdir		?= ../../../..
+
+include $(top_srcdir)/include/mk/testcases.mk
+
+include $(top_srcdir)/include/mk/generic_leaf_target.mk
diff --git a/testcases/kernel/fs/fsplough/fsplough.c b/testcases/kernel/fs/fsplough/fsplough.c
new file mode 100644
index 000000000..900aaacab
--- /dev/null
+++ b/testcases/kernel/fs/fsplough/fsplough.c
@@ -0,0 +1,279 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2024 SUSE LLC <mdoucha@suse.cz>
+ */
+
+/*\
+ * [Description]
+ *
+ * Write data into a test file using various methods and verify that file
+ * contents match what was written.
+ */
+
+#define _GNU_SOURCE
+#include <fcntl.h>
+#include <sys/statvfs.h>
+#include "tst_test.h"
+#include "tst_safe_prw.h"
+
+#define MAX_VEC 8
+#define TEST_FILENAME "fsplough.dat"
+
+typedef void (*io_func)(void *buf, size_t offset, size_t size);
+
+static char *workdir_arg;
+static char *directwr_flag;
+static char *directrd_flag;
+static char *loop_arg;
+static int loop_count = 4096;
+
+static int read_fd = -1, write_fd = -1;
+static char *writebuf, *filedata;
+static size_t blocksize, bufsize, filesize;
+
+static void do_write(void *buf, size_t offset, size_t size);
+static void do_pwrite(void *buf, size_t offset, size_t size);
+static void do_writev(void *buf, size_t offset, size_t size);
+static void do_pwritev(void *buf, size_t offset, size_t size);
+static void do_read(void *buf, size_t offset, size_t size);
+static void do_pread(void *buf, size_t offset, size_t size);
+static void do_readv(void *buf, size_t offset, size_t size);
+static void do_preadv(void *buf, size_t offset, size_t size);
+
+static const io_func write_funcs[] = {
+	do_write,
+	do_pwrite,
+	do_writev,
+	do_pwritev
+};
+
+static const io_func read_funcs[] = {
+	do_read,
+	do_pread,
+	do_readv,
+	do_preadv
+};
+
+static size_t fill_buffer(char *buf, size_t size)
+{
+	size_t i, ret = MAX_VEC + 1 + rand() % (size - MAX_VEC);
+
+	/* Align buffer size to block size */
+	if (directwr_flag || directrd_flag)
+		ret = MAX(LTP_ALIGN(ret, blocksize), MAX_VEC * blocksize);
+
+	for (i = 0; i < ret; i++)
+		buf[i] = rand();
+
+	return ret;
+}
+
+static void vectorize_buffer(struct iovec *vec, size_t vec_size, char *buf,
+	size_t buf_size, int align)
+{
+	size_t i, len, chunk = align ? blocksize : 1;
+
+	memset(vec, 0, vec_size * sizeof(struct iovec));
+	buf_size /= chunk;
+
+	for (i = 0; buf_size && i < vec_size; i++) {
+		len = 1 + rand() % (buf_size + i + 1 - vec_size);
+		vec[i].iov_base = buf;
+		vec[i].iov_len = len * chunk;
+		buf += vec[i].iov_len;
+		buf_size -= len;
+	}
+
+	vec[vec_size - 1].iov_len += buf_size * chunk;
+}
+
+static void update_filedata(const void *buf, size_t offset, size_t size)
+{
+	memcpy(filedata + offset, buf, size * sizeof(char));
+}
+
+static void do_write(void *buf, size_t offset, size_t size)
+{
+	SAFE_LSEEK(write_fd, offset, SEEK_SET);
+	SAFE_WRITE(1, write_fd, buf, size);
+}
+
+static void do_pwrite(void *buf, size_t offset, size_t size)
+{
+	SAFE_PWRITE(1, write_fd, buf, size, offset);
+}
+
+static void do_writev(void *buf, size_t offset, size_t size)
+{
+	struct iovec vec[MAX_VEC] = {};
+
+	vectorize_buffer(vec, MAX_VEC, buf, size, !!directwr_flag);
+	SAFE_LSEEK(write_fd, offset, SEEK_SET);
+	SAFE_WRITEV(1, write_fd, vec, MAX_VEC);
+}
+
+static void do_pwritev(void *buf, size_t offset, size_t size)
+{
+	struct iovec vec[MAX_VEC] = {};
+
+	vectorize_buffer(vec, MAX_VEC, buf, size, !!directwr_flag);
+	SAFE_PWRITEV(1, write_fd, vec, MAX_VEC, offset);
+}
+
+static void do_read(void *buf, size_t offset, size_t size)
+{
+	SAFE_LSEEK(read_fd, offset, SEEK_SET);
+	SAFE_READ(1, read_fd, buf, size);
+}
+
+static void do_pread(void *buf, size_t offset, size_t size)
+{
+	SAFE_PREAD(1, read_fd, buf, size, offset);
+}
+
+static void do_readv(void *buf, size_t offset, size_t size)
+{
+	struct iovec vec[MAX_VEC] = {};
+
+	vectorize_buffer(vec, MAX_VEC, buf, size, !!directrd_flag);
+	SAFE_LSEEK(read_fd, offset, SEEK_SET);
+	SAFE_READV(1, read_fd, vec, MAX_VEC);
+}
+
+static void do_preadv(void *buf, size_t offset, size_t size)
+{
+	struct iovec vec[MAX_VEC] = {};
+
+	vectorize_buffer(vec, MAX_VEC, buf, size, !!directrd_flag);
+	SAFE_PREADV(1, read_fd, vec, MAX_VEC, offset);
+}
+
+static int open_testfile(int flags)
+{
+	if ((flags & O_WRONLY) && directwr_flag)
+		flags |= O_DIRECT;
+
+	if ((flags & O_RDONLY) && directrd_flag)
+		flags |= O_DIRECT;
+
+	return SAFE_OPEN(TEST_FILENAME, flags, 0644);
+}
+
+static void setup(void)
+{
+	struct statvfs statbuf;
+	size_t pagesize;
+
+	srand(time(0));
+	pagesize = SAFE_SYSCONF(_SC_PAGESIZE);
+
+	if (workdir_arg)
+		SAFE_CHDIR(workdir_arg);
+
+	if (tst_parse_int(loop_arg, &loop_count, 0, INT_MAX))
+		tst_brk(TBROK, "Invalid write loop count: %s", loop_arg);
+
+	write_fd = open_testfile(O_WRONLY | O_CREAT | O_TRUNC);
+	read_fd = open_testfile(O_RDONLY);
+	TEST(fstatvfs(write_fd, &statbuf));
+
+	if (TST_RET == -1)
+		tst_brk(TBROK | TTERRNO, "fstatvfs() failed");
+	else if (TST_RET)
+		tst_brk(TBROK | TTERRNO, "Invalid fstatvfs() return value");
+
+	blocksize = statbuf.f_bsize;
+	tst_res(TINFO, "Block size: %zu", blocksize);
+	bufsize = 4 * MAX_VEC * MAX(pagesize, blocksize);
+	filesize = 1024 * MAX(pagesize, blocksize);
+	writebuf = SAFE_MMAP(NULL, bufsize, PROT_READ | PROT_WRITE,
+		MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+	filedata = SAFE_MALLOC(filesize);
+
+	tst_set_max_runtime(bufsize * loop_count / (8 * 1024 * 1024));
+}
+
+static void run(void)
+{
+	size_t start, length;
+	int i, f, fails = 0;
+
+	/* Test data consistency between random writes */
+	for (i = 0; i < loop_count; i++) {
+		length = fill_buffer(writebuf, bufsize);
+		start = rand() % (filesize + 1 - length);
+
+		/* Align offset to blocksize if needed */
+		if (directrd_flag || directwr_flag)
+			start = (start + blocksize / 2) & ~(blocksize - 1);
+
+		update_filedata(writebuf, start, length);
+		f = rand() % ARRAY_SIZE(write_funcs);
+		write_funcs[f](writebuf, start, length);
+
+		memset(writebuf, 0, length);
+		f = rand() % ARRAY_SIZE(read_funcs);
+		read_funcs[f](writebuf, start, length);
+
+		if (memcmp(writebuf, filedata + start, length)) {
+			tst_res(TFAIL, "Partial data mismatch at [%zu:%zu]",
+				start, start + length);
+			fails++;
+		}
+	}
+
+	if (!fails)
+		tst_res(TPASS, "Partial data are consistent");
+
+	/* Ensure that the testfile has the expected size */
+	do_write(writebuf, filesize - blocksize, blocksize);
+	update_filedata(writebuf, filesize - blocksize, blocksize);
+
+	/* Sync the testfile and clear cache */
+	SAFE_CLOSE(read_fd);
+	SAFE_FSYNC(write_fd);
+	SAFE_FILE_PRINTF("/proc/sys/vm/drop_caches", "1");
+	read_fd = open_testfile(O_RDONLY);
+
+	/* Check final file contents */
+	for (start = 0; start < filesize; start += bufsize) {
+		length = MIN(bufsize, filesize - start);
+		SAFE_READ(1, read_fd, writebuf, length);
+
+		if (memcmp(writebuf, filedata + start, length)) {
+			tst_res(TFAIL, "Final data mismatch at [%zu:%zu]",
+				start, start + length);
+			return;
+		}
+	}
+
+	tst_res(TPASS, "Final data are consistent");
+}
+
+static void cleanup(void)
+{
+	SAFE_MUNMAP(writebuf, bufsize);
+	free(filedata);
+
+	if (read_fd >= 0)
+		SAFE_CLOSE(read_fd);
+
+	if (write_fd >= 0)
+		SAFE_CLOSE(write_fd);
+
+	SAFE_UNLINK(TEST_FILENAME);
+}
+
+static struct tst_test test = {
+	.test_all = run,
+	.setup = setup,
+	.cleanup = cleanup,
+	.needs_tmpdir = 1,
+	.options = (struct tst_option[]) {
+		{"c:", &loop_arg, "Number of write loops (default: 4096)"},
+		{"d:", &workdir_arg, "Path to working directory"},
+		{"W", &directwr_flag, "Use direct I/O for writing"},
+		{"R", &directrd_flag, "Use direct I/O for reading"},
+		{}
+	}
+};
diff --git a/testcases/misc/lvm/datafiles/runfile.tpl b/testcases/misc/lvm/datafiles/runfile.tpl
index 4c80e9391..ee733626f 100644
--- a/testcases/misc/lvm/datafiles/runfile.tpl
+++ b/testcases/misc/lvm/datafiles/runfile.tpl
@@ -29,6 +29,10 @@
 {fsname}_gf28 growfiles -W {fsname}_gf28 -b -D 0 -w -g 16b -C 1 -b -i 1000 -u {tempdir}/{fsname}/gfsparse-2-$$
 {fsname}_gf29 growfiles -W {fsname}_gf29 -b -D 0 -r 1-4096 -R 0-33554432 -i 0 -L 60 -B 805306368 -C 1 -u {tempdir}/{fsname}/gfsparse-3-$$
 {fsname}_gf30 growfiles -W {fsname}_gf30 -D 0 -b -i 0 -L 60 -u -B 1000b -e 1 -o O_RDWR,O_CREAT,O_SYNC -g 20480 -T 10 -t 20480 {tempdir}/{fsname}/gf-sync-$$
+{fsname}_plough01 fsplough -d {tempdir}/{fsname}
+{fsname}_plough02 fsplough -R -d {tempdir}/{fsname}
+{fsname}_plough03 fsplough -W -d {tempdir}/{fsname}
+{fsname}_plough04 fsplough -RW -d {tempdir}/{fsname}
 {fsname}_rwtest01 rwtest -N {fsname}_rwtest01 -c -q -i 60s  -f sync 10%25000:rw-sync-$$ 500b:{tempdir}/{fsname}/rwtest01%f
 {fsname}_rwtest02 rwtest -N {fsname}_rwtest02 -c -q -i 60s  -f buffered 10%25000:rw-buffered-$$ 500b:{tempdir}/{fsname}/rwtest02%f
 {fsname}_rwtest03 rwtest -N {fsname}_rwtest03 -c -q -i 60s -n 2  -f buffered -s mmread,mmwrite -m random -Dv 10%25000:mm-buff-$$ 500b:{tempdir}/{fsname}/rwtest03%f
-- 
2.46.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 4/5] Add support for setting loop device size in shell tests

[Martin Doucha]

Add TST_DEVICE_SIZE variable to set loop device size.

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---
 doc/developers/writing_tests.rst | 2 +-
 testcases/lib/tst_test.sh        | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/doc/developers/writing_tests.rst b/doc/developers/writing_tests.rst
index 2293800a1..ab633121e 100644
--- a/doc/developers/writing_tests.rst
+++ b/doc/developers/writing_tests.rst
@@ -390,7 +390,7 @@ LTP C And Shell Test API Comparison
       - TST_FS_TYPE
 
     * - .dev_min_size
-      - not applicable
+      - TST_DEVICE_SIZE
 
     * - .format_device
       - TST_FORMAT_DEVICE
diff --git a/testcases/lib/tst_test.sh b/testcases/lib/tst_test.sh
index c19c30b76..cfdae0230 100644
--- a/testcases/lib/tst_test.sh
+++ b/testcases/lib/tst_test.sh
@@ -687,6 +687,7 @@ tst_run()
 			CHECKPOINT_WAKE2|CHECKPOINT_WAKE_AND_WAIT);;
 			DEV_EXTRA_OPTS|DEV_FS_OPTS|FORMAT_DEVICE|MOUNT_DEVICE);;
 			SKIP_FILESYSTEMS|SKIP_IN_LOCKDOWN|SKIP_IN_SECUREBOOT);;
+			DEVICE_SIZE);;
 			*) tst_res TWARN "Reserved variable TST_$_tst_i used!";;
 			esac
 		done
@@ -750,7 +751,7 @@ tst_run()
 
 	# needs to be after cd $TST_TMPDIR to keep test_dev.img under $TST_TMPDIR
 	if [ "$TST_NEEDS_DEVICE" = 1 ]; then
-		TST_DEVICE=$(tst_device acquire)
+		TST_DEVICE=$(tst_device acquire $TST_DEVICE_SIZE)
 
 		if [ ! -b "$TST_DEVICE" -o $? -ne 0 ]; then
 			unset TST_DEVICE
-- 
2.46.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH 5/5] Add test for data integrity over NFS

[Martin Doucha]

Add NFS test which checks data integrity of random writes into a file,
with both buffered and direct I/O.

Signed-off-by: Martin Doucha <mdoucha@suse.cz>
---

The lower loop count is necessary because NFS has very large block size,
up to 256KB on x86_64. The new tests take ~50 minutes to complete in total
on my laptop. With the default loop count, the TCP tests would all time out.

 runtest/net.nfs                           | 11 +++++++
 testcases/network/nfs/nfs_stress/nfs10.sh | 36 +++++++++++++++++++++++
 2 files changed, 47 insertions(+)
 create mode 100755 testcases/network/nfs/nfs_stress/nfs10.sh

diff --git a/runtest/net.nfs b/runtest/net.nfs
index 7f84457bc..fef993da8 100644
--- a/runtest/net.nfs
+++ b/runtest/net.nfs
@@ -94,6 +94,17 @@ nfs09_v40_ip6t nfs09.sh -6 -v 4 -t tcp
 nfs09_v41_ip6t nfs09.sh -6 -v 4.1 -t tcp
 nfs09_v42_ip6t nfs09.sh -6 -v 4.2 -t tcp
 
+nfs10_v30_ip4u nfs10.sh -v 3 -t udp
+nfs10_v30_ip4t nfs10.sh -v 3 -t tcp
+nfs10_v40_ip4t nfs10.sh -v 4 -t tcp
+nfs10_v41_ip4t nfs10.sh -v 4.1 -t tcp
+nfs10_v42_ip4t nfs10.sh -v 4.2 -t tcp
+nfs10_v30_ip6u nfs10.sh -6 -v 3 -t udp
+nfs10_v30_ip6t nfs10.sh -6 -v 3 -t tcp
+nfs10_v40_ip6t nfs10.sh -6 -v 4 -t tcp
+nfs10_v41_ip6t nfs10.sh -6 -v 4.1 -t tcp
+nfs10_v42_ip6t nfs10.sh -6 -v 4.2 -t tcp
+
 nfslock01_v30_ip4u nfslock01.sh -v 3 -t udp
 nfslock01_v30_ip4t nfslock01.sh -v 3 -t tcp
 nfslock01_v40_ip4t nfslock01.sh -v 4 -t tcp
diff --git a/testcases/network/nfs/nfs_stress/nfs10.sh b/testcases/network/nfs/nfs_stress/nfs10.sh
new file mode 100755
index 000000000..b4e777285
--- /dev/null
+++ b/testcases/network/nfs/nfs_stress/nfs10.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0-or-later
+# Copyright (C) 2024 SUSE LLC <mdoucha@suse.cz>
+#
+# DESCRIPTION: Verify data integrity over NFS, with and without O_DIRECT
+
+TST_CNT=4
+TST_TESTFUNC="do_test"
+TST_DEVICE_SIZE=1024
+
+do_test1()
+{
+	tst_res TINFO "Testing buffered write, buffered read"
+	EXPECT_PASS fsplough -c 512 -d "$PWD"
+}
+
+do_test2()
+{
+	tst_res TINFO "Testing buffered write, direct read"
+	EXPECT_PASS fsplough -c 512 -R -d "$PWD"
+}
+
+do_test3()
+{
+	tst_res TINFO "Testing direct write, buffered read"
+	EXPECT_PASS fsplough -c 512 -W -d "$PWD"
+}
+
+do_test4()
+{
+	tst_res TINFO "Testing direct write, direct read"
+	EXPECT_PASS fsplough -c 512 -RW -d "$PWD"
+}
+
+. nfs_lib.sh
+tst_run
-- 
2.46.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 1/5] Move preadv()/pwritev() backup definitions to LAPI

[Petr Vorel]

Hi Martin,

> Merge existing LAPI definitions of preadv2()/pwritev2() to lapi/uio.h
> and add preadv()/pwritev() definitions from test code. Also fix minor
> bugs in return value type and offset handling.

Good catch int => ssize_t.

Obviously correct, merged with second commit.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 4/5] Add support for setting loop device size in shell tests

[Petr Vorel]

Hi Martin,

> Add TST_DEVICE_SIZE variable to set loop device size.

obviously correct, thanks!
Reviewed-by: Petr Vorel <pvorel@suse.cz>

Maybe we should (later) add trivial shell API test into lib/newlib_tests/shell,
which creates the device and tests if the size is expected.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Petr Vorel]

Hi Martin,

[ Cc Li and Cyril due tst_set_timeout vs. tst_loader.sh and tst_run_shell.c. ]

> Add NFS test which checks data integrity of random writes into a file,
> with both buffered and direct I/O.

LGTM.
Reviewed-by: Petr Vorel <pvorel@suse.cz>

I'll try to have look on the core work (fsplough.c) on Monday.
Hopefully Li or Cyril would have look into this as well.

> The lower loop count is necessary because NFS has very large block size,
> up to 256KB on x86_64. The new tests take ~50 minutes to complete in total
> on my laptop. With the default loop count, the TCP tests would all time out.

BTW I got timeout when testing nfs10.sh -v 4 -t tcp on Tumbleweed VM (2 CPU,
1.4 GB RAM, 2 GB SWAP) on my laptop:
...
nfs10 1 TINFO: === Testing on ext4 ===
nfs10 1 TINFO: Formatting ext4 with opts='/dev/loop0'
nfs10 1 TINFO: Mounting device: mount -t ext4 /dev/loop0 /var/tmp/LTP_nfs10.zWBiahjI48/mntpoint
nfs10 1 TINFO: timeout per run is 0h 5m 0s

=> nfs10.sh runs only for 5 min.

nfs10 1 TINFO: mount.nfs: (linux nfs-utils 2.6.3)
nfs10 1 TINFO: setup NFSv4, socket type tcp
nfs10 1 TINFO: Mounting /var/tmp/LTP_nfs10.zWBiahjI48/4/0
nfs10 1 TINFO: Mounting NFS: mount -v -t nfs -o proto=tcp,vers=4 10.0.0.2:/var/tmp/LTP_nfs10.zWBiahjI48/mntpoint/4/tcp /var/tmp/LTP_nfs10.zWBiahjI48/4/0
nfs10 1 TINFO: Testing buffered write, buffered read
tst_tmpdir.c:316: TINFO: Using /var/tmp//LTP_fspL4p41F as tmpdir (btrfs filesystem)
tst_test.c:1890: TINFO: LTP version: 20240930-49-g3facdd035
tst_test.c:1894: TINFO: Tested kernel: 6.12.0-rc4-1.gf83465d-default #1 SMP PREEMPT_DYNAMIC Sun Oct 20 22:44:57 UTC 2024 (f83465d) x86_64
tst_test.c:1725: TINFO: Timeout per run is 0h 00m 30s
fsplough.c:186: TINFO: Block size: 262144
tst_test.c:1733: TINFO: Updating max runtime to 0h 08m 32s
tst_test.c:1725: TINFO: Timeout per run is 0h 09m 02s

=> fsplough.c is updated to 9 min, but that does not help due nfs10.sh not
having updated.

Test timed out, sending SIGTERM!
If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1
Sending SIGKILL to test process...
Test is still running... 10
Test is still running... 9
Test is still running... 8
Test is still running... 7
Test is still running... 6
Test is still running... 5
Test is still running... 4
Test is still running... 3
Test is still running... 2
Test is still running... 1
Test is still running, sending SIGKILL
Killed

The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
time) in nfs10.sh and increase tst_set_timeout with.

Or, I wonder if using tst_loader.sh and tst_run_shell.c would help to integrate
these. But I'm not sure how easily could be nfs_lib.sh integrated, e.g.
* isn't it too late, when it uses tst_net.sh (maybe this file would need to be
  integrated)
* should use keep using TST_ALL_FILESYSTEMS=1 from tst_test.sh or configure
  via all_filesystems=1 via json?

OT: Other tests which are considerable are these which use netstress.c (via
tst_netload_compare).

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Cyril Hrubis]

Hi!
> Test timed out, sending SIGTERM!
> If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1
> Sending SIGKILL to test process...
> Test is still running... 10
> Test is still running... 9
> Test is still running... 8
> Test is still running... 7
> Test is still running... 6
> Test is still running... 5
> Test is still running... 4
> Test is still running... 3
> Test is still running... 2
> Test is still running... 1
> Test is still running, sending SIGKILL
> Killed

Isn't the propblem here that the fsplough.c itself does not have
.max_runtime set?

I think that main problem here is that we have a LTP test that is being
executed from another LTP test which means that we have two layers of
everything including timeouts.

> The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
> time) in nfs10.sh and increase tst_set_timeout with.
> 
> Or, I wonder if using tst_loader.sh and tst_run_shell.c would help to integrate
> these. But I'm not sure how easily could be nfs_lib.sh integrated, e.g.
> * isn't it too late, when it uses tst_net.sh (maybe this file would need to be
>   integrated)
> * should use keep using TST_ALL_FILESYSTEMS=1 from tst_test.sh or configure
>   via all_filesystems=1 via json?

I guess that the best solution would be to add NFS support into the
tst_test.c as another filesystem. That way we would get much more
coverate than we do now. I guess that it would be doable as long as the
configuration on how to do that is passed to the test library somehow.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Cyril Hrubis]

Hi!
> > The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
> > time) in nfs10.sh and increase tst_set_timeout with.
> > 
> > Or, I wonder if using tst_loader.sh and tst_run_shell.c would help to integrate
> > these. But I'm not sure how easily could be nfs_lib.sh integrated, e.g.
> > * isn't it too late, when it uses tst_net.sh (maybe this file would need to be
> >   integrated)
> > * should use keep using TST_ALL_FILESYSTEMS=1 from tst_test.sh or configure
> >   via all_filesystems=1 via json?
> 
> I guess that the best solution would be to add NFS support into the
> tst_test.c as another filesystem. That way we would get much more
> coverate than we do now. I guess that it would be doable as long as the
> configuration on how to do that is passed to the test library somehow.

I'm looking at the nfs_lib.sh, we do have a lot there actually since it
runs for all filesystems and we run the nfs tests for different nfs
versions as well. This does not seem that would integrate well into the
tst_test all_filesystems testing.

I will think a bit how to integrate things better, because having LTP
test inside LTP test will never work well.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Cyril Hrubis]

Hi!
> I will think a bit how to integrate things better, because having LTP
> test inside LTP test will never work well.

So one posibility is to use the new shell test library that integrates
cleanly with C, then we can have a shell nfs test that runs for all
filesystems, mounts the nfs and then runs the C fsplough itself.

Minimal example how to do that is:

From b48184d4fa1918afe6c84d0e691f8d46c986e92c Mon Sep 17 00:00:00 2001
From: Cyril Hrubis <chrubis@suse.cz>
Date: Mon, 4 Nov 2024 14:29:24 +0100
Subject: [PATCH] shell lib: Add example how to run C child

Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
---
 testcases/lib/run_tests.sh                  |  2 +-
 testcases/lib/tests/.gitignore              |  1 +
 testcases/lib/tests/shell_c_child.c         | 16 ++++++++++++++
 testcases/lib/tests/shell_loader_c_child.sh | 24 +++++++++++++++++++++
 4 files changed, 42 insertions(+), 1 deletion(-)
 create mode 100644 testcases/lib/tests/shell_c_child.c
 create mode 100755 testcases/lib/tests/shell_loader_c_child.sh

diff --git a/testcases/lib/run_tests.sh b/testcases/lib/run_tests.sh
index 8ea615f96..40d415e6c 100755
--- a/testcases/lib/run_tests.sh
+++ b/testcases/lib/run_tests.sh
@@ -14,7 +14,7 @@ for i in shell_loader.sh shell_loader_all_filesystems.sh shell_loader_no_metadat
 	 shell_loader_wrong_metadata.sh shell_loader_invalid_metadata.sh\
 	 shell_loader_supported_archs.sh shell_loader_filesystems.sh\
 	 shell_loader_tcnt.sh shell_loader_kconfigs.sh shell_loader_tags.sh \
-	 shell_loader_invalid_block.sh; do
+	 shell_loader_invalid_block.sh shell_loader_c_child.sh; do
 	echo
 	echo "*** Running $i ***"
 	echo
diff --git a/testcases/lib/tests/.gitignore b/testcases/lib/tests/.gitignore
index da967c4d6..e9e163d13 100644
--- a/testcases/lib/tests/.gitignore
+++ b/testcases/lib/tests/.gitignore
@@ -4,3 +4,4 @@ shell_test03
 shell_test04
 shell_test05
 shell_test06
+shell_c_child
diff --git a/testcases/lib/tests/shell_c_child.c b/testcases/lib/tests/shell_c_child.c
new file mode 100644
index 000000000..fda5133a6
--- /dev/null
+++ b/testcases/lib/tests/shell_c_child.c
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Shell test C child example.
+ */
+
+#define TST_NO_DEFAULT_MAIN
+#include "tst_test.h"
+
+int main(void)
+{
+	tst_reinit();
+
+	tst_res(TPASS, "C child works fine!");
+
+	return 0;
+}
diff --git a/testcases/lib/tests/shell_loader_c_child.sh b/testcases/lib/tests/shell_loader_c_child.sh
new file mode 100755
index 000000000..d190be6e2
--- /dev/null
+++ b/testcases/lib/tests/shell_loader_c_child.sh
@@ -0,0 +1,24 @@
+#!/bin/sh
+#
+# ---
+# doc
+#
+# [Description]
+#
+# This is an example how to run C child from shell.
+# ---
+#
+# ---
+# env
+# {
+# }
+# ---
+
+. tst_loader.sh
+
+if [ -n "LTP_IPC_PATH" ]; then
+	tst_res TPASS "LTP_IPC_PATH=$LTP_IPC_PATH!"
+fi
+
+tst_res TINFO "Running C child"
+shell_c_child

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Petr Vorel]

> Hi!
> > Test timed out, sending SIGTERM!
> > If you are running on slow machine, try exporting LTP_TIMEOUT_MUL > 1
> > Sending SIGKILL to test process...
> > Test is still running... 10
> > Test is still running... 9
> > Test is still running... 8
> > Test is still running... 7
> > Test is still running... 6
> > Test is still running... 5
> > Test is still running... 4
> > Test is still running... 3
> > Test is still running... 2
> > Test is still running... 1
> > Test is still running, sending SIGKILL
> > Killed

> Isn't the propblem here that the fsplough.c itself does not have
> .max_runtime set?

It has timeout adjusted in the setup function:
tst_set_max_runtime(bufsize * loop_count / (8 * 1024 * 1024));

> I think that main problem here is that we have a LTP test that is being
> executed from another LTP test which means that we have two layers of
> everything including timeouts.

Similarly netstress.c, which is also called from tst_net.sh. But these tests
does not run on all filesystems.

> > The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
> > time) in nfs10.sh and increase tst_set_timeout with.

> > Or, I wonder if using tst_loader.sh and tst_run_shell.c would help to integrate
> > these. But I'm not sure how easily could be nfs_lib.sh integrated, e.g.
> > * isn't it too late, when it uses tst_net.sh (maybe this file would need to be
> >   integrated)
> > * should use keep using TST_ALL_FILESYSTEMS=1 from tst_test.sh or configure
> >   via all_filesystems=1 via json?

> I guess that the best solution would be to add NFS support into the
> tst_test.c as another filesystem. That way we would get much more
> coverate than we do now. I guess that it would be doable as long as the
> configuration on how to do that is passed to the test library somehow.

Test coverage would really increase a lot. Even there are more NFS testsuites,
IMHO it'd be useful to bring this to LTP - we test a different things then
fstests, pynfs, cthon, ....

But OTOH it does not have much sense to run NFS tests on a loop device which
backing file is on tmpfs (the default setup on distros which use tmpfs on /tmp).
It's a question, whether we should start using /var/tmp as a default for TMPDIR
(if exists). LTP does cleanup of the functions, unless it fails. Specially NFS
tests when they timeout, the cleanup is not done due mounted filesystem being
used. Then /var/tmp gets filled up quickly and reboot does not do automatic
cleanup.

Adding NFS support would require setup code from nfs_lib.sh (e.g. exportfs, run
mount binary or start using mount(2), doing various checks etc) to be run by
tst_test.c, via another process via fork() and execvp(), e.g. with LTP's
tst_cmd().  I'm not sure if minimal required subset should be put in some shell
script, which would be executed by tst_test.c (maybe using tst_loader.sh to join
shared memory to unify the counter) or run individual commands one by one. WDYT?

This simpler version would allow to add various NFS to all_filesystems. I wonder
if we want to support just single NFS version e.g. the newest NFSv4.2 or more.
I suppose we would test only on TCP (UDP is deprecated, not enabled by default
on newer kernels). But even the core of the NFS setup in nfs_mount() uses
tst_rhost_run() from tst_net.sh. E.g. some parts of tst_net.sh would need to be
rewritten into C code (or it would have to call tst_net.sh, if gluing with
tst_loader.sh would work).

My original idea to convert all NFS tests would basically mean to rewrite quite
to use even more nfs_lib.sh and tst_net.sh. code (there is a support for
setting NFS version and protocol and other things).

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Petr Vorel]

> Hi!

> > I'm looking at the nfs_lib.sh, we do have a lot there actually since it
> > runs for all filesystems and we run the nfs tests for different nfs
> > versions as well. This does not seem that would integrate well into the
> > tst_test all_filesystems testing.

Yes (found that as well when replied to your first email).
I thought you want to integrate NFS just for a shake of other tests, which use
.all_filesystems (e.g. some NFS to be added into fs_type_whitelist[], e.g.
"nfsv4), which would of course require u specific setup.

That would, of course did not cover all range of NFS versions + also using UDP,
which is used for current NFS tests (including this one).

> > I will think a bit how to integrate things better, because having LTP
> > test inside LTP test will never work well.

Reviewed-by: Petr Vorel <pvorel@suse.cz>
Tested-by: Petr Vorel <pvorel@suse.cz>

Looks usable + later on tests which use netstress.c could be converted to use it
as well.

Kind regards,
Petr

> So one posibility is to use the new shell test library that integrates
> cleanly with C, then we can have a shell nfs test that runs for all
> filesystems, mounts the nfs and then runs the C fsplough itself.

> Minimal example how to do that is:

> From b48184d4fa1918afe6c84d0e691f8d46c986e92c Mon Sep 17 00:00:00 2001
> From: Cyril Hrubis <chrubis@suse.cz>
> Date: Mon, 4 Nov 2024 14:29:24 +0100
> Subject: [PATCH] shell lib: Add example how to run C child

> Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> ---
>  testcases/lib/run_tests.sh                  |  2 +-
>  testcases/lib/tests/.gitignore              |  1 +
>  testcases/lib/tests/shell_c_child.c         | 16 ++++++++++++++
>  testcases/lib/tests/shell_loader_c_child.sh | 24 +++++++++++++++++++++
>  4 files changed, 42 insertions(+), 1 deletion(-)
>  create mode 100644 testcases/lib/tests/shell_c_child.c
>  create mode 100755 testcases/lib/tests/shell_loader_c_child.sh

> diff --git a/testcases/lib/run_tests.sh b/testcases/lib/run_tests.sh
> index 8ea615f96..40d415e6c 100755
> --- a/testcases/lib/run_tests.sh
> +++ b/testcases/lib/run_tests.sh
> @@ -14,7 +14,7 @@ for i in shell_loader.sh shell_loader_all_filesystems.sh shell_loader_no_metadat
>  	 shell_loader_wrong_metadata.sh shell_loader_invalid_metadata.sh\
>  	 shell_loader_supported_archs.sh shell_loader_filesystems.sh\
>  	 shell_loader_tcnt.sh shell_loader_kconfigs.sh shell_loader_tags.sh \
> -	 shell_loader_invalid_block.sh; do
> +	 shell_loader_invalid_block.sh shell_loader_c_child.sh; do
>  	echo
>  	echo "*** Running $i ***"
>  	echo
> diff --git a/testcases/lib/tests/.gitignore b/testcases/lib/tests/.gitignore
> index da967c4d6..e9e163d13 100644
> --- a/testcases/lib/tests/.gitignore
> +++ b/testcases/lib/tests/.gitignore
> @@ -4,3 +4,4 @@ shell_test03
>  shell_test04
>  shell_test05
>  shell_test06
> +shell_c_child
> diff --git a/testcases/lib/tests/shell_c_child.c b/testcases/lib/tests/shell_c_child.c
> new file mode 100644
> index 000000000..fda5133a6
> --- /dev/null
> +++ b/testcases/lib/tests/shell_c_child.c
> @@ -0,0 +1,16 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +/*
> + * Shell test C child example.
> + */
> +
> +#define TST_NO_DEFAULT_MAIN
> +#include "tst_test.h"
> +
> +int main(void)
> +{
> +	tst_reinit();
> +
> +	tst_res(TPASS, "C child works fine!");
> +
> +	return 0;
> +}
> diff --git a/testcases/lib/tests/shell_loader_c_child.sh b/testcases/lib/tests/shell_loader_c_child.sh
> new file mode 100755
> index 000000000..d190be6e2
> --- /dev/null
> +++ b/testcases/lib/tests/shell_loader_c_child.sh
> @@ -0,0 +1,24 @@
> +#!/bin/sh
> +#
> +# ---
> +# doc
> +#
> +# [Description]
> +#
> +# This is an example how to run C child from shell.
> +# ---
> +#
> +# ---
> +# env
> +# {
> +# }
> +# ---
> +
> +. tst_loader.sh
> +
> +if [ -n "LTP_IPC_PATH" ]; then
> +	tst_res TPASS "LTP_IPC_PATH=$LTP_IPC_PATH!"
> +fi
> +
> +tst_res TINFO "Running C child"
> +shell_c_child

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Martin Doucha]

On 02. 11. 24 0:32, Petr Vorel wrote:
> Hi Martin,
> 
> BTW I got timeout when testing nfs10.sh -v 4 -t tcp on Tumbleweed VM (2 CPU,
> 1.4 GB RAM, 2 GB SWAP) on my laptop:
> ...
> 
> The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
> time) in nfs10.sh and increase tst_set_timeout with.

Yes, that is expected for now, we need to add separate timeouts for each 
subtest, including per-filesystem.

> Or, I wonder if using tst_loader.sh and tst_run_shell.c would help to integrate
> these. But I'm not sure how easily could be nfs_lib.sh integrated, e.g.
> * isn't it too late, when it uses tst_net.sh (maybe this file would need to be
>    integrated)
> * should use keep using TST_ALL_FILESYSTEMS=1 from tst_test.sh or configure
>    via all_filesystems=1 via json?

I'm not sure I understand your idea.

-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Petr Vorel]

> > Hi!

> > > I'm looking at the nfs_lib.sh, we do have a lot there actually since it
> > > runs for all filesystems and we run the nfs tests for different nfs
> > > versions as well. This does not seem that would integrate well into the
> > > tst_test all_filesystems testing.

> Yes (found that as well when replied to your first email).
> I thought you want to integrate NFS just for a shake of other tests, which use
> .all_filesystems (e.g. some NFS to be added into fs_type_whitelist[], e.g.
> "nfsv4), which would of course require u specific setup.

> That would, of course did not cover all range of NFS versions + also using UDP,
> which is used for current NFS tests (including this one).

> > > I will think a bit how to integrate things better, because having LTP
> > > test inside LTP test will never work well.

> Reviewed-by: Petr Vorel <pvorel@suse.cz>
> Tested-by: Petr Vorel <pvorel@suse.cz>

> Looks usable + later on tests which use netstress.c could be converted to use it
> as well.

@Cyril: gently ping, could you please merge the patch you posted below?
Obviously usable test + example which can be used for network tests.

Kind regards,
Petr

> Kind regards,
> Petr

> > So one posibility is to use the new shell test library that integrates
> > cleanly with C, then we can have a shell nfs test that runs for all
> > filesystems, mounts the nfs and then runs the C fsplough itself.

> > Minimal example how to do that is:

> > From b48184d4fa1918afe6c84d0e691f8d46c986e92c Mon Sep 17 00:00:00 2001
> > From: Cyril Hrubis <chrubis@suse.cz>
> > Date: Mon, 4 Nov 2024 14:29:24 +0100
> > Subject: [PATCH] shell lib: Add example how to run C child

> > Signed-off-by: Cyril Hrubis <chrubis@suse.cz>
> > ---
> >  testcases/lib/run_tests.sh                  |  2 +-
> >  testcases/lib/tests/.gitignore              |  1 +
> >  testcases/lib/tests/shell_c_child.c         | 16 ++++++++++++++
> >  testcases/lib/tests/shell_loader_c_child.sh | 24 +++++++++++++++++++++
> >  4 files changed, 42 insertions(+), 1 deletion(-)
> >  create mode 100644 testcases/lib/tests/shell_c_child.c
> >  create mode 100755 testcases/lib/tests/shell_loader_c_child.sh

> > diff --git a/testcases/lib/run_tests.sh b/testcases/lib/run_tests.sh
> > index 8ea615f96..40d415e6c 100755
> > --- a/testcases/lib/run_tests.sh
> > +++ b/testcases/lib/run_tests.sh
> > @@ -14,7 +14,7 @@ for i in shell_loader.sh shell_loader_all_filesystems.sh shell_loader_no_metadat
> >  	 shell_loader_wrong_metadata.sh shell_loader_invalid_metadata.sh\
> >  	 shell_loader_supported_archs.sh shell_loader_filesystems.sh\
> >  	 shell_loader_tcnt.sh shell_loader_kconfigs.sh shell_loader_tags.sh \
> > -	 shell_loader_invalid_block.sh; do
> > +	 shell_loader_invalid_block.sh shell_loader_c_child.sh; do
> >  	echo
> >  	echo "*** Running $i ***"
> >  	echo
> > diff --git a/testcases/lib/tests/.gitignore b/testcases/lib/tests/.gitignore
> > index da967c4d6..e9e163d13 100644
> > --- a/testcases/lib/tests/.gitignore
> > +++ b/testcases/lib/tests/.gitignore
> > @@ -4,3 +4,4 @@ shell_test03
> >  shell_test04
> >  shell_test05
> >  shell_test06
> > +shell_c_child
> > diff --git a/testcases/lib/tests/shell_c_child.c b/testcases/lib/tests/shell_c_child.c
> > new file mode 100644
> > index 000000000..fda5133a6
> > --- /dev/null
> > +++ b/testcases/lib/tests/shell_c_child.c
> > @@ -0,0 +1,16 @@
> > +// SPDX-License-Identifier: GPL-2.0-or-later
> > +/*
> > + * Shell test C child example.
> > + */
> > +
> > +#define TST_NO_DEFAULT_MAIN
> > +#include "tst_test.h"
> > +
> > +int main(void)
> > +{
> > +	tst_reinit();
> > +
> > +	tst_res(TPASS, "C child works fine!");
> > +
> > +	return 0;
> > +}
> > diff --git a/testcases/lib/tests/shell_loader_c_child.sh b/testcases/lib/tests/shell_loader_c_child.sh
> > new file mode 100755
> > index 000000000..d190be6e2
> > --- /dev/null
> > +++ b/testcases/lib/tests/shell_loader_c_child.sh
> > @@ -0,0 +1,24 @@
> > +#!/bin/sh
> > +#
> > +# ---
> > +# doc
> > +#
> > +# [Description]
> > +#
> > +# This is an example how to run C child from shell.
> > +# ---
> > +#
> > +# ---
> > +# env
> > +# {
> > +# }
> > +# ---
> > +
> > +. tst_loader.sh
> > +
> > +if [ -n "LTP_IPC_PATH" ]; then
> > +	tst_res TPASS "LTP_IPC_PATH=$LTP_IPC_PATH!"
> > +fi
> > +
> > +tst_res TINFO "Running C child"
> > +shell_c_child

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Petr Vorel]

> On 02. 11. 24 0:32, Petr Vorel wrote:
> > Hi Martin,

> > BTW I got timeout when testing nfs10.sh -v 4 -t tcp on Tumbleweed VM (2 CPU,
> > 1.4 GB RAM, 2 GB SWAP) on my laptop:
> > ...

> > The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
> > time) in nfs10.sh and increase tst_set_timeout with.

> Yes, that is expected for now, we need to add separate timeouts for each
> subtest, including per-filesystem.

Right, I suppose you post v2.

I wonder why you did not get timeouts on your laptop. I suppose calculation for
fsplough was below 5 min, right?

> > Or, I wonder if using tst_loader.sh and tst_run_shell.c would help to integrate
> > these. But I'm not sure how easily could be nfs_lib.sh integrated, e.g.
> > * isn't it too late, when it uses tst_net.sh (maybe this file would need to be
> >    integrated)
> > * should use keep using TST_ALL_FILESYSTEMS=1 from tst_test.sh or configure
> >    via all_filesystems=1 via json?

> I'm not sure I understand your idea.

Second idea (TST_ALL_FILESYSTEMS=1 => .all_filesystems=1) is wrong.

I was thinking how to run fsplough in nfs10.sh similarly to
testcases/lib/tests/shell_loader_c_child.sh to use $LTP_IPC_PATH.  That would
help better integration (e.g. common results counter, maybe even timeout on
single place). This should be quite easy.

I was also thinking whether rewrite nfs_lib.sh into C (much more work,
there would probably  need to be tst_net.sh equivalent in C questionable if
worth of the effort).

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Cyril Hrubis]

Hi!
> @Cyril: gently ping, could you please merge the patch you posted below?
> Obviously usable test + example which can be used for network tests.

Done.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Cyril Hrubis]

Hi!
> I'm not sure I understand your idea.

See:

https://github.com/linux-test-project/ltp/commit/762ee52cfe9b1e3ab8733f711f2a2f044ede334d

What we can do now is to start the test from shell and then run the C
child with a single instance of the test library. The end result looks
like this:

C shell loader process
  |
  Shell child process
    |
    C child process

And all of them increments counters in the same piece of shared memory,
there is only single timeout enforced by the top level process, etc.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Martin Doucha]

On 12. 11. 24 22:40, Petr Vorel wrote:
>> On 02. 11. 24 0:32, Petr Vorel wrote:
>>> Hi Martin,
> 
>>> BTW I got timeout when testing nfs10.sh -v 4 -t tcp on Tumbleweed VM (2 CPU,
>>> 1.4 GB RAM, 2 GB SWAP) on my laptop:
>>> ...
> 
>>> The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
>>> time) in nfs10.sh and increase tst_set_timeout with.
> 
>> Yes, that is expected for now, we need to add separate timeouts for each
>> subtest, including per-filesystem.
> 
> Right, I suppose you post v2.
> 
> I wonder why you did not get timeouts on your laptop. I suppose calculation for
> fsplough was below 5 min, right?

I've tested nfs10.sh with LTP_TIMEOUT_MUL=2 which was enough to make it 
pass. The per-testcase timeout implementation needs to be a separate 
patch anyway so sending a v2 doesn't make sense at the moment. I can 
write the timeout patch but I'll need to figure out how the shell 
timeout works.

-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Petr Vorel]

> On 12. 11. 24 22:40, Petr Vorel wrote:
> > > On 02. 11. 24 0:32, Petr Vorel wrote:
> > > > Hi Martin,

> > > > BTW I got timeout when testing nfs10.sh -v 4 -t tcp on Tumbleweed VM (2 CPU,
> > > > 1.4 GB RAM, 2 GB SWAP) on my laptop:
> > > > ...

> > > > The quickest way would be to use the same calculation (* $TST_CNT + 5% for spare
> > > > time) in nfs10.sh and increase tst_set_timeout with.

> > > Yes, that is expected for now, we need to add separate timeouts for each
> > > subtest, including per-filesystem.

> > Right, I suppose you post v2.

> > I wonder why you did not get timeouts on your laptop. I suppose calculation for
> > fsplough was below 5 min, right?

> I've tested nfs7.sh with LTP_TIMEOUT_MUL=2 which was enough to make it
> pass. The per-testcase timeout implementation needs to be a separate patch
> anyway so sending a v2 doesn't make sense at the moment. I can write the
> timeout patch but I'll need to figure out how the shell timeout works.

IMHO using TST_TIMEOUT in nfs10.sh should be enough (hence asking for v2).
Because TST_TIMEOUT is for each test run. It's visible from the test output I
posted, also _tst_setup_timer, which sets the timeout is called only in
_tst_run_iterations. And _tst_run_iterations is called for each filesystem in
_tst_run_tcases_per_fs (or directly otherwise).

_tst_run_iterations()
{
	local _tst_i=$TST_ITERATIONS
	local _tst_j

	[ "$TST_NEEDS_TMPDIR" = 1 ] && cd "$TST_TMPDIR"

	_prepare_device

	_tst_setup_timer

_tst_run_tcases_per_fs()
{
	local fs
	local filesystems

	filesystems="$(tst_supported_fs -s "$TST_SKIP_FILESYSTEMS")"
	if [ $? -ne 0 ]; then
		tst_brk TCONF "There are no supported filesystems or all skipped"
	fi

	for fs in $filesystems; do
		tst_res TINFO "=== Testing on $fs ==="
		TST_FS_TYPE="$fs"
		_tst_run_iterations
	done
}

tst_run()
{
	...

	if [ "$TST_ALL_FILESYSTEMS" = 1 ]; then
		_tst_run_tcases_per_fs
	else
		_tst_run_iterations
	fi

	_tst_do_exit
}

Using the same calculation you use in fsplough:
tst_set_max_runtime(bufsize * loop_count / (8 * 1024 * 1024))
In TST_TIMEOUT the number of the above would be 4* (4 tests), loop_count == 512.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 4/5] Add support for setting loop device size in shell tests

[Petr Vorel]

Hi Martin,

merged previous commit (fsplough) and this one. Thanks!

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 5/5] Add test for data integrity over NFS

[Petr Vorel]

Hi all,

we had a chat with Martin. I now understand that test speed really depends on
the speed of host disc. Instead of trying to calculate precise value for shell,
let's just be pragmatic and set high enough value for shell, e.g 30 min
or even 1 hour. Because timeout is driven by fsplough.c
(C app) anyway.

@Martin, please send v2 with TST_TIMEOUT=1800 (or whatever value you consider
high enough. I would just try to avoid -1 (unlimited timeout).

In the future we should migrate NFS tests to using Cyril's shell loader
but that might take time.

Kind regards,
Petr

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 3/5] Add test for file data integrity

[Cyril Hrubis]

Hi!
> +static void vectorize_buffer(struct iovec *vec, size_t vec_size, char *buf,
> +	size_t buf_size, int align)
> +{
> +	size_t i, len, chunk = align ? blocksize : 1;
> +
> +	memset(vec, 0, vec_size * sizeof(struct iovec));
> +	buf_size /= chunk;
> +
> +	for (i = 0; buf_size && i < vec_size; i++) {
> +		len = 1 + rand() % (buf_size + i + 1 - vec_size);

You have 1 + ... before the rand should it be buf_size:

i - 1 - vec_size

Also since 0 sized iov_len is actually correct, it means skip the entry,
we do not have to make sure that the length is > 0 here and neither we
need to make sure that we have enough buffer saved for the rest of the
vector.


> +		vec[i].iov_base = buf;
> +		vec[i].iov_len = len * chunk;
> +		buf += vec[i].iov_len;
> +		buf_size -= len;
> +	}
> +
> +	vec[vec_size - 1].iov_len += buf_size * chunk;
> +}
> +
> +static void update_filedata(const void *buf, size_t offset, size_t size)
> +{
> +	memcpy(filedata + offset, buf, size * sizeof(char));
                                                ^
						This is 1 by definition.
> +}
> +
> +static void do_write(void *buf, size_t offset, size_t size)
> +{
> +	SAFE_LSEEK(write_fd, offset, SEEK_SET);
> +	SAFE_WRITE(1, write_fd, buf, size);
> +}
> +
> +static void do_pwrite(void *buf, size_t offset, size_t size)
> +{
> +	SAFE_PWRITE(1, write_fd, buf, size, offset);
> +}
> +
> +static void do_writev(void *buf, size_t offset, size_t size)
> +{
> +	struct iovec vec[MAX_VEC] = {};
> +
> +	vectorize_buffer(vec, MAX_VEC, buf, size, !!directwr_flag);
> +	SAFE_LSEEK(write_fd, offset, SEEK_SET);
> +	SAFE_WRITEV(1, write_fd, vec, MAX_VEC);
> +}
> +
> +static void do_pwritev(void *buf, size_t offset, size_t size)
> +{
> +	struct iovec vec[MAX_VEC] = {};
> +
> +	vectorize_buffer(vec, MAX_VEC, buf, size, !!directwr_flag);
> +	SAFE_PWRITEV(1, write_fd, vec, MAX_VEC, offset);
> +}
> +
> +static void do_read(void *buf, size_t offset, size_t size)
> +{
> +	SAFE_LSEEK(read_fd, offset, SEEK_SET);
> +	SAFE_READ(1, read_fd, buf, size);
> +}
> +
> +static void do_pread(void *buf, size_t offset, size_t size)
> +{
> +	SAFE_PREAD(1, read_fd, buf, size, offset);
> +}
> +
> +static void do_readv(void *buf, size_t offset, size_t size)
> +{
> +	struct iovec vec[MAX_VEC] = {};
> +
> +	vectorize_buffer(vec, MAX_VEC, buf, size, !!directrd_flag);
> +	SAFE_LSEEK(read_fd, offset, SEEK_SET);
> +	SAFE_READV(1, read_fd, vec, MAX_VEC);
> +}
> +
> +static void do_preadv(void *buf, size_t offset, size_t size)
> +{
> +	struct iovec vec[MAX_VEC] = {};
> +
> +	vectorize_buffer(vec, MAX_VEC, buf, size, !!directrd_flag);
> +	SAFE_PREADV(1, read_fd, vec, MAX_VEC, offset);
> +}
> +
> +static int open_testfile(int flags)
> +{
> +	if ((flags & O_WRONLY) && directwr_flag)
> +		flags |= O_DIRECT;
> +
> +	if ((flags & O_RDONLY) && directrd_flag)
> +		flags |= O_DIRECT;
> +
> +	return SAFE_OPEN(TEST_FILENAME, flags, 0644);
> +}
> +
> +static void setup(void)
> +{
> +	struct statvfs statbuf;
> +	size_t pagesize;
> +
> +	srand(time(0));
> +	pagesize = SAFE_SYSCONF(_SC_PAGESIZE);
> +
> +	if (workdir_arg)
> +		SAFE_CHDIR(workdir_arg);
> +
> +	if (tst_parse_int(loop_arg, &loop_count, 0, INT_MAX))
> +		tst_brk(TBROK, "Invalid write loop count: %s", loop_arg);
> +
> +	write_fd = open_testfile(O_WRONLY | O_CREAT | O_TRUNC);
> +	read_fd = open_testfile(O_RDONLY);
> +	TEST(fstatvfs(write_fd, &statbuf));
> +
> +	if (TST_RET == -1)
> +		tst_brk(TBROK | TTERRNO, "fstatvfs() failed");
> +	else if (TST_RET)
> +		tst_brk(TBROK | TTERRNO, "Invalid fstatvfs() return value");
> +
> +	blocksize = statbuf.f_bsize;
> +	tst_res(TINFO, "Block size: %zu", blocksize);
> +	bufsize = 4 * MAX_VEC * MAX(pagesize, blocksize);
> +	filesize = 1024 * MAX(pagesize, blocksize);
> +	writebuf = SAFE_MMAP(NULL, bufsize, PROT_READ | PROT_WRITE,
> +		MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
> +	filedata = SAFE_MALLOC(filesize);
> +
> +	tst_set_max_runtime(bufsize * loop_count / (8 * 1024 * 1024));

I think that it may be better to define the test to be time based rather
than iteration based. If we decide that the default runtime would be 5
minutes the test would do as much iterations during that time as
possible and report how many it did at the end of the test. Possibly it
may warn if minimal amount of iterations wasn't reached. That would make
the test runtime much more predictable

> +}
> +
> +static void run(void)
> +{
> +	size_t start, length;
> +	int i, f, fails = 0;
> +
> +	/* Test data consistency between random writes */
> +	for (i = 0; i < loop_count; i++) {
> +		length = fill_buffer(writebuf, bufsize);
> +		start = rand() % (filesize + 1 - length);
> +
> +		/* Align offset to blocksize if needed */
> +		if (directrd_flag || directwr_flag)
> +			start = (start + blocksize / 2) & ~(blocksize - 1);
> +
> +		update_filedata(writebuf, start, length);
> +		f = rand() % ARRAY_SIZE(write_funcs);
> +		write_funcs[f](writebuf, start, length);
> +
> +		memset(writebuf, 0, length);
> +		f = rand() % ARRAY_SIZE(read_funcs);
> +		read_funcs[f](writebuf, start, length);
> +
> +		if (memcmp(writebuf, filedata + start, length)) {
> +			tst_res(TFAIL, "Partial data mismatch at [%zu:%zu]",
> +				start, start + length);
> +			fails++;
> +		}
> +	}
> +
> +	if (!fails)
> +		tst_res(TPASS, "Partial data are consistent");
> +
> +	/* Ensure that the testfile has the expected size */
> +	do_write(writebuf, filesize - blocksize, blocksize);
> +	update_filedata(writebuf, filesize - blocksize, blocksize);
> +
> +	/* Sync the testfile and clear cache */
> +	SAFE_CLOSE(read_fd);
> +	SAFE_FSYNC(write_fd);
> +	SAFE_FILE_PRINTF("/proc/sys/vm/drop_caches", "1");
> +	read_fd = open_testfile(O_RDONLY);
> +
> +	/* Check final file contents */
> +	for (start = 0; start < filesize; start += bufsize) {
> +		length = MIN(bufsize, filesize - start);
> +		SAFE_READ(1, read_fd, writebuf, length);
> +
> +		if (memcmp(writebuf, filedata + start, length)) {
> +			tst_res(TFAIL, "Final data mismatch at [%zu:%zu]",
> +				start, start + length);
> +			return;
> +		}
> +	}
> +
> +	tst_res(TPASS, "Final data are consistent");
> +}
> +
> +static void cleanup(void)
> +{
> +	SAFE_MUNMAP(writebuf, bufsize);
> +	free(filedata);
> +
> +	if (read_fd >= 0)
> +		SAFE_CLOSE(read_fd);
> +
> +	if (write_fd >= 0)
> +		SAFE_CLOSE(write_fd);
> +
> +	SAFE_UNLINK(TEST_FILENAME);
> +}
> +
> +static struct tst_test test = {
> +	.test_all = run,
> +	.setup = setup,
> +	.cleanup = cleanup,
> +	.needs_tmpdir = 1,
> +	.options = (struct tst_option[]) {
> +		{"c:", &loop_arg, "Number of write loops (default: 4096)"},
> +		{"d:", &workdir_arg, "Path to working directory"},
> +		{"W", &directwr_flag, "Use direct I/O for writing"},
> +		{"R", &directrd_flag, "Use direct I/O for reading"},
> +		{}
> +	}
> +};
> diff --git a/testcases/misc/lvm/datafiles/runfile.tpl b/testcases/misc/lvm/datafiles/runfile.tpl
> index 4c80e9391..ee733626f 100644
> --- a/testcases/misc/lvm/datafiles/runfile.tpl
> +++ b/testcases/misc/lvm/datafiles/runfile.tpl
> @@ -29,6 +29,10 @@
>  {fsname}_gf28 growfiles -W {fsname}_gf28 -b -D 0 -w -g 16b -C 1 -b -i 1000 -u {tempdir}/{fsname}/gfsparse-2-$$
>  {fsname}_gf29 growfiles -W {fsname}_gf29 -b -D 0 -r 1-4096 -R 0-33554432 -i 0 -L 60 -B 805306368 -C 1 -u {tempdir}/{fsname}/gfsparse-3-$$
>  {fsname}_gf30 growfiles -W {fsname}_gf30 -D 0 -b -i 0 -L 60 -u -B 1000b -e 1 -o O_RDWR,O_CREAT,O_SYNC -g 20480 -T 10 -t 20480 {tempdir}/{fsname}/gf-sync-$$
> +{fsname}_plough01 fsplough -d {tempdir}/{fsname}
> +{fsname}_plough02 fsplough -R -d {tempdir}/{fsname}
> +{fsname}_plough03 fsplough -W -d {tempdir}/{fsname}
> +{fsname}_plough04 fsplough -RW -d {tempdir}/{fsname}
>  {fsname}_rwtest01 rwtest -N {fsname}_rwtest01 -c -q -i 60s  -f sync 10%25000:rw-sync-$$ 500b:{tempdir}/{fsname}/rwtest01%f
>  {fsname}_rwtest02 rwtest -N {fsname}_rwtest02 -c -q -i 60s  -f buffered 10%25000:rw-buffered-$$ 500b:{tempdir}/{fsname}/rwtest02%f
>  {fsname}_rwtest03 rwtest -N {fsname}_rwtest03 -c -q -i 60s -n 2  -f buffered -s mmread,mmwrite -m random -Dv 10%25000:mm-buff-$$ 500b:{tempdir}/{fsname}/rwtest03%f
> -- 
> 2.46.0
> 
> 
> -- 
> Mailing list info: https://lists.linux.it/listinfo/ltp

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH 3/5] Add test for file data integrity

[Martin Doucha]

On 04. 12. 24 14:45, Cyril Hrubis wrote:
> Hi!
>> +static void vectorize_buffer(struct iovec *vec, size_t vec_size, char *buf,
>> +	size_t buf_size, int align)
>> +{
>> +	size_t i, len, chunk = align ? blocksize : 1;
>> +
>> +	memset(vec, 0, vec_size * sizeof(struct iovec));
>> +	buf_size /= chunk;
>> +
>> +	for (i = 0; buf_size && i < vec_size; i++) {
>> +		len = 1 + rand() % (buf_size + i + 1 - vec_size);
> 
> You have 1 + ... before the rand should it be buf_size:
> 
> i - 1 - vec_size
> 
> Also since 0 sized iov_len is actually correct, it means skip the entry,
> we do not have to make sure that the length is > 0 here and neither we
> need to make sure that we have enough buffer saved for the rest of the
> vector.

If we dropped the 1+ from the calculation, it should look like this:
len = rand() % (bufsize + 1);

Because rand() % x <= x-1.

> 
> 
>> +		vec[i].iov_base = buf;
>> +		vec[i].iov_len = len * chunk;
>> +		buf += vec[i].iov_len;
>> +		buf_size -= len;
>> +	}
>> +
>> +	vec[vec_size - 1].iov_len += buf_size * chunk;
>> +}
>> +
>> +static void update_filedata(const void *buf, size_t offset, size_t size)
>> +{
>> +	memcpy(filedata + offset, buf, size * sizeof(char));
>                                                  ^
> 						This is 1 by definition.

I know, but I prefer to explicitly annotate the array type in 
allocations and copy operations for future review and reference. The 
compiler will optimize it out anyway.

>> +	blocksize = statbuf.f_bsize;
>> +	tst_res(TINFO, "Block size: %zu", blocksize);
>> +	bufsize = 4 * MAX_VEC * MAX(pagesize, blocksize);
>> +	filesize = 1024 * MAX(pagesize, blocksize);
>> +	writebuf = SAFE_MMAP(NULL, bufsize, PROT_READ | PROT_WRITE,
>> +		MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
>> +	filedata = SAFE_MALLOC(filesize);
>> +
>> +	tst_set_max_runtime(bufsize * loop_count / (8 * 1024 * 1024));
> 
> I think that it may be better to define the test to be time based rather
> than iteration based. If we decide that the default runtime would be 5
> minutes the test would do as much iterations during that time as
> possible and report how many it did at the end of the test. Possibly it
> may warn if minimal amount of iterations wasn't reached. That would make
> the test runtime much more predictable

That would be a useful command line argument for the NFS test variants. 
But this patch is already merged so a separate new patch is needed 
either way.

-- 
Martin Doucha   mdoucha@suse.cz
SW Quality Engineer
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp