[PATCH 00/13] More integck improvements

[PATCH 00/13] More integck improvements

[Artem Bityutskiy]

Now integck power cut mode works, mostly. It fails sometimes, though.
I know the reason of some of the failures and this needs work in UBIFS.

The test still does not verify integrity of clean files after an emulated
power cut - it only makes sure the file-system is mountable and usable
after power cuts. But this is already a lot and I see several issues.
Once I've fixed them - I'll implement better integrity checking.

Remind - the long term goal is to solve the unstable bits problem, but
to even start doing this we need:
1. Rock solid user-space test which can stress-test the FS well, and
   be aware of the power cut emulation.
2. UBIFS and integck should survive the current UBIFS power cut emulation
   which does not emulate unstable bits.

Only after this we can move further. I do not know when I reach that
point, but I'm trying.

Anyway, these tests fix several integck problems I encountered, make
it nicer, and improve it.

Artem.

[PATCH 01/13] fs-tests: integck: improve help message

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

Fix typos, make it a bit more descriptive.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |   35 +++++++++++++++++++----------------
 1 files changed, 19 insertions(+), 16 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 511739f..522c6de 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -2572,25 +2572,28 @@ static void get_tested_fs_info(void)
 
 static const char doc[] = PROGRAM_NAME " version " PROGRAM_VERSION
 " - a stress test which checks the file-system integrity.\n"
-"It creates a directory named \"integck_test_dir_pid\", where where pid is the\n"
-"process id. Then it randomly creates and deletes files, directories, symlinks\n"
-"and hardlinks, randomly writes and truncate files, sometimes makes holes in\n"
-"files, sometimes fsync()'s them. Then it un-mounts and re-mounts the test file\n"
-"system and checks the contents - everything (files, dirs, etc) should be there\n"
-"and the contents of the files should be correct. This is repeated a number of\n"
-"times (set with -n, default 1).\n\n"
-"This test is also able to perform powe cut testing. The underlying file-system\n"
-"or the device driver should be able to emulate power-cuts, e.g., but switching\n"
-"to R/O mode at random points of time. And the file-system should return EROFS\n"
-"(read-only file-system error) for all operations which modify it. In this case\n"
-"this test program re-mounts the file-system and checks that all files and\n"
-"directories which have been successfully synchronized before the power cut are\n"
-"there and contains correct data. Then the test continues.\n";
+"\n"
+"The test creates a directory named \"integck_test_dir_<pid>\", where where\n"
+"<pid> is the process id. Then it randomly creates and deletes files,\n"
+"directories, symlinks, and hardlinks, randomly writes and truncate files,\n"
+"sometimes makes holes in files, sometimes fsync()'s them. Then it un-mounts and\n"
+"re-mounts the tested file-system and checks the contents - everything (files,\n"
+"directories, etc) should be there and the contents of the files should be\n"
+"correct. This is repeated a number of times (set with -n, default 1).\n"
+"\n"
+"This test is also able to perform power cut testing. The underlying file-system\n"
+"or the device driver should be able to emulate power-cuts, by switching to R/O\n"
+"mode at random moments. And the file-system should return EROFS (read-only\n"
+"file-system error) for all operations which modify it. In this case this test\n"
+"program re-mounts the file-system and checks that all files and directories\n"
+"which have been successfully synchronized before the power cut. And the test\n"
+"continues forever.\n";
 
 static const char optionsstr[] =
 "-n, --repeat=<count> repeat count, default is 1; zero value - repeat forever\n"
-"-p, --power-cut      power cut testing mode\n"
-"-v, --verbose        be verbose about failure during power cut testing\n"
+"-p, --power-cut      power cut testing mode (-n parameter is ignored and the\n"
+"                     test continues forever)\n"
+"-v, --verbose        be verbose about failures during power cut testing\n"
 "-h, -?, --help       print help message\n"
 "-V, --version        print program version\n";
 
-- 
1.7.2.3

[PATCH 02/13] fs-tests: integck: do not inlcude lib directory

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

The integck test does not depend on the "common" directory any more,
so we do not have to have -I ../lib for compilation.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/Makefile |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tests/fs-tests/integrity/Makefile b/tests/fs-tests/integrity/Makefile
index 03652a2..976c3ce 100644
--- a/tests/fs-tests/integrity/Makefile
+++ b/tests/fs-tests/integrity/Makefile
@@ -5,7 +5,7 @@ endif
 
 COMMON_HEADERS_DIR := ../../../include
 
-CFLAGS := $(CFLAGS) -Wall -g -O2 -I../lib -I$(COMMON_HEADERS_DIR)
+CFLAGS := $(CFLAGS) -Wall -g -O2 -I$(COMMON_HEADERS_DIR)
 
 LDFLAGS := $(LDFLAGS)
 
-- 
1.7.2.3

[PATCH 03/13] fs-tests: integck: print backtrace in case of failure

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

This patch teaches the 'CHECK()' macro to print the back-trace in
case of failure which is sometimes very useful for debugging. This
patch also adds a helper function for the 'CHECK()' macro because
otherwise it becomes too large.

Note, since all functions in the test are static, the stack trace
does not really show symbols.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |   39 ++++++++++++++++++++++++++---------
 1 files changed, 29 insertions(+), 10 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 522c6de..9822b3b 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -17,7 +17,6 @@
  *
  * Author: Adrian Hunter
  */
-
 #include <unistd.h>
 #include <stdlib.h>
 #include <stdio.h>
@@ -32,6 +31,7 @@
 #include <getopt.h>
 #include <assert.h>
 #include <mntent.h>
+#include <execinfo.h>
 #include <sys/mman.h>
 #include <sys/vfs.h>
 #include <sys/mount.h>
@@ -54,15 +54,9 @@
  */
 #define stringify1(x) #x
 #define stringify(x) stringify1(x)
-#define CHECK(cond) do {                                       \
-	if (!(cond)) {                                         \
-		int _err = errno;                              \
-		fflush(stdout);                                \
-		errmsg("condition '%s' failed at %s:%d",       \
-		       stringify(cond), __FILE__, __LINE__);   \
-		errmsg("error %d (%s)", _err, strerror(_err)); \
-		exit(EXIT_FAILURE);                            \
-	}                                                      \
+#define CHECK(cond) do {                                                     \
+	if (!(cond))                                                         \
+		check_failed(stringify(cond), __func__, __FILE__, __LINE__); \
 } while(0)
 
 #define pcv(fmt, ...) do {                                         \
@@ -215,6 +209,31 @@ static unsigned int check_run_no;
 static char *random_name_buf;
 
 /*
+ * This is a helper for the 'CHECK()' macro - prints a scary error message and
+ * terminates the program.
+ */
+static void check_failed(const char *cond, const char *func, const char *file,
+			 int line)
+{
+	int error = errno, count;
+	void *addresses[128];
+
+	fflush(stdout);
+	errmsg("condition '%s' failed in %s() at %s:%d",
+	       cond, func, __FILE__, __LINE__);
+	normsg("error %d (%s)", error, strerror(error));
+	/*
+	 * Note, to make this work well you need:
+	 * 1. Make all functions non-static - add "#define static'
+	 * 2. Compile with -rdynamic and -g gcc options
+	 * 3. Preferrably compile with -O0 to avoid inlining
+	 */
+	count = backtrace(addresses, 128);
+	backtrace_symbols_fd(addresses, count, fileno(stdout));
+	exit(EXIT_FAILURE);
+}
+
+/*
  * Is this 'struct write_info' actually holds information about a truncation?
  */
 static int is_truncation(struct write_info *w)
-- 
1.7.2.3

[PATCH 04/13] fs-tests: integck: add -e option to verify operations

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

Improve the test and add -e option which forces integck to verify all
operation it performs. This patch does not implement file writes and
holes operations so far - will be done in subsequent patches.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |  138 +++++++++++++++++++++++++++++++++---
 1 files changed, 127 insertions(+), 11 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 9822b3b..f39745f 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -68,6 +68,7 @@
 static struct {
 	long repeat_cnt;
 	int power_cut_mode;
+	int verify_ops;
 	int verbose;
 	const char *mount_point;
 } args = {
@@ -529,6 +530,13 @@ static int dir_new(struct dir_info *parent, const char *name)
 		free(path);
 		return -1;
 	}
+
+	if (args.verify_ops) {
+		struct stat st;
+
+		CHECK(lstat(path, &st) == 0);
+		CHECK(S_ISDIR(st.st_mode));
+	}
 	free(path);
 
 	add_dir_entry(parent, 'd', name, NULL);
@@ -569,8 +577,16 @@ static int dir_remove(struct dir_info *dir)
 		return -1;
 	}
 
+	if (args.verify_ops) {
+		struct stat st;
+
+		CHECK(lstat(path, &st) == -1);
+		CHECK(errno == ENOENT);
+	}
+
 	/* Remove entry from parent directory */
 	remove_dir_entry(dir->entry, 1);
+
 	free(path);
 	return 0;
 }
@@ -597,10 +613,17 @@ static int file_new(struct dir_info *parent, const char *name)
 		free(path);
 		return -1;
 	}
-	free(path);
 
+	if (args.verify_ops) {
+		struct stat st;
+
+		CHECK(lstat(path, &st) == 0);
+		CHECK(S_ISREG(st.st_mode));
+	}
 	file = add_dir_entry(parent, 'f', name, NULL);
 	add_fd(file, fd);
+
+	free(path);
 	return 0;
 }
 
@@ -610,6 +633,7 @@ static int link_new(struct dir_info *parent, const char *name,
 	struct dir_entry_info *entry;
 	char *path, *target;
 	int ret;
+	struct stat st1, st2;
 
 	entry = file->links;
 	if (!entry)
@@ -617,6 +641,10 @@ static int link_new(struct dir_info *parent, const char *name,
 
 	path = dir_path(parent, name);
 	target = dir_path(entry->parent, entry->name);
+
+	if (args.verify_ops)
+		CHECK(lstat(target, &st1) == 0);
+
 	ret = link(target, path);
 	if (ret != 0) {
 		if (errno == ENOSPC) {
@@ -629,9 +657,19 @@ static int link_new(struct dir_info *parent, const char *name,
 		free(path);
 		return ret;
 	}
+
+	if (args.verify_ops) {
+		CHECK(lstat(path, &st2) == 0);
+		CHECK(S_ISREG(st2.st_mode));
+		CHECK(st1.st_ino == st2.st_ino);
+		CHECK(st2.st_nlink > 1);
+		CHECK(st2.st_nlink == st1.st_nlink + 1);
+	}
+
+	add_dir_entry(parent, 'f', name, file);
+
 	free(target);
 	free(path);
-	add_dir_entry(parent, 'f', name, file);
 	return 0;
 }
 
@@ -665,10 +703,18 @@ static int file_unlink(struct dir_entry_info *entry)
 		free(path);
 		return -1;
 	}
-	free(path);
+
+	if (args.verify_ops) {
+		struct stat st;
+
+		CHECK(lstat(path, &st) == -1);
+		CHECK(errno == ENOENT);
+	}
 
 	/* Remove file entry from parent directory */
 	remove_dir_entry(entry, 1);
+
+	free(path);
 	return 0;
 }
 
@@ -984,6 +1030,9 @@ static int file_ftruncate(struct file_info *file, int fd, off_t new_length)
 		return -1;
 	}
 
+	if (args.verify_ops)
+		CHECK(lseek(fd, 0, SEEK_END) == new_length);
+
 	return 0;
 }
 
@@ -1724,6 +1773,7 @@ static int rename_entry(struct dir_entry_info *entry)
 	struct dir_info *parent;
 	char *path, *to, *name;
 	int ret, isdir, retry;
+	struct stat st1, st2;
 
 	if (!entry->parent)
 		return 0;
@@ -1757,6 +1807,9 @@ static int rename_entry(struct dir_entry_info *entry)
 	if (!path)
 		return 0;
 
+	if (args.verify_ops)
+		CHECK(lstat(path, &st1) == 0);
+
 	ret = rename(path, to);
 	if (ret != 0) {
 		ret = 0;
@@ -1772,6 +1825,11 @@ static int rename_entry(struct dir_entry_info *entry)
 		return ret;
 	}
 
+	if (args.verify_ops) {
+		CHECK(lstat(to, &st2) == 0);
+		CHECK(st1.st_ino == st2.st_ino);
+	}
+
 	free(path);
 	free(to);
 
@@ -1870,6 +1928,18 @@ static char *pick_symlink_target(const char *symlink_path)
 	return rel_path;
 }
 
+static void verify_symlink(const char *target, const char *path)
+{
+	int bytes;
+	char buf[PATH_MAX + 1];
+
+	bytes = readlink(path, buf, PATH_MAX);
+	CHECK(bytes >= 0);
+	CHECK(bytes < PATH_MAX);
+	buf[bytes] = '\0';
+	CHECK(!strcmp(buf, target));
+}
+
 static int symlink_new(struct dir_info *dir, const char *nm)
 {
 	struct symlink_info *s;
@@ -1896,10 +1966,14 @@ static int symlink_new(struct dir_info *dir, const char *nm)
 		free(path);
 		return ret;
 	}
-	free(path);
+
+	if (args.verify_ops)
+		verify_symlink(target, path);
 
 	s = add_dir_entry(dir, 's', name, NULL);
 	s->target_pathname = target;
+
+	free(path);
 	free(name);
 	return 0;
 }
@@ -1915,7 +1989,15 @@ static int symlink_remove(struct symlink_info *symlink)
 		return -1;
 	}
 
+	if (args.verify_ops) {
+		struct stat st;
+
+		CHECK(lstat(path, &st) == -1);
+		CHECK(errno == ENOENT);
+	}
+
 	remove_dir_entry(symlink->entry, 1);
+
 	free(path);
 	return 0;
 }
@@ -2247,11 +2329,34 @@ static int rm_minus_rf_dir(const char *dir_name)
 
 	CHECK(chdir(buf) == 0);
 	CHECK(closedir(dir) == 0);
+
+	if (args.verify_ops) {
+		dir = opendir(dir_name);
+		CHECK(dir != NULL);
+		do {
+			errno = 0;
+			dent = readdir(dir);
+			if (dent)
+				CHECK(!strcmp(dent->d_name, ".") ||
+				      !strcmp(dent->d_name, ".."));
+		} while (dent);
+		CHECK(errno == 0);
+		CHECK(closedir(dir) == 0);
+	}
+
 	ret = rmdir(dir_name);
 	if (ret) {
 		pcv("cannot remove directory %s", dir_name);
 		return -1;
 	}
+
+	if (args.verify_ops) {
+		struct stat st;
+
+		CHECK(lstat(dir_name, &st) == -1);
+		CHECK(errno == ENOENT);
+	}
+
 	return 0;
 }
 
@@ -2600,6 +2705,11 @@ static const char doc[] = PROGRAM_NAME " version " PROGRAM_VERSION
 "directories, etc) should be there and the contents of the files should be\n"
 "correct. This is repeated a number of times (set with -n, default 1).\n"
 "\n"
+"By default the test does not verify file-system modifications and assumes they\n"
+"are done correctly if the file-system returns success. However, the -e flag\n"
+"enables additional verifications and the test verifies all the file-system\n"
+"operations it performs.\n"
+"\n"
 "This test is also able to perform power cut testing. The underlying file-system\n"
 "or the device driver should be able to emulate power-cuts, by switching to R/O\n"
 "mode at random moments. And the file-system should return EROFS (read-only\n"
@@ -2612,16 +2722,20 @@ static const char optionsstr[] =
 "-n, --repeat=<count> repeat count, default is 1; zero value - repeat forever\n"
 "-p, --power-cut      power cut testing mode (-n parameter is ignored and the\n"
 "                     test continues forever)\n"
+"-e, --verify-ops     verify all operations, e.g., every time a file is written\n"
+"                     to, read the data back and verify it, every time a\n"
+"                     directory is created, check that it exists, etc\n"
 "-v, --verbose        be verbose about failures during power cut testing\n"
 "-h, -?, --help       print help message\n"
 "-V, --version        print program version\n";
 
 static const struct option long_options[] = {
-	{ .name = "repeat",    .has_arg = 1, .flag = NULL, .val = 'n' },
-	{ .name = "power-cut", .has_arg = 0, .flag = NULL, .val = 'p' },
-	{ .name = "verbose",   .has_arg = 0, .flag = NULL, .val = 'v' },
-	{ .name = "help",      .has_arg = 0, .flag = NULL, .val = 'h' },
-	{ .name = "version",   .has_arg = 0, .flag = NULL, .val = 'V' },
+	{ .name = "repeat",     .has_arg = 1, .flag = NULL, .val = 'n' },
+	{ .name = "power-cut",  .has_arg = 0, .flag = NULL, .val = 'p' },
+	{ .name = "verify-ops", .has_arg = 0, .flag = NULL, .val = 'e' },
+	{ .name = "verbose",    .has_arg = 0, .flag = NULL, .val = 'v' },
+	{ .name = "help",       .has_arg = 0, .flag = NULL, .val = 'h' },
+	{ .name = "version",    .has_arg = 0, .flag = NULL, .val = 'V' },
 	{ NULL, 0, NULL, 0},
 };
 
@@ -2636,7 +2750,7 @@ static int parse_opts(int argc, char * const argv[])
 	while (1) {
 		int key, error = 0;
 
-		key = getopt_long(argc, argv, "n:pvVh?", long_options, NULL);
+		key = getopt_long(argc, argv, "n:pevVh?", long_options, NULL);
 		if (key == -1)
 			break;
 
@@ -2649,6 +2763,9 @@ static int parse_opts(int argc, char * const argv[])
 		case 'p':
 			args.power_cut_mode = 1;
 			break;
+		case 'e':
+			args.verify_ops = 1;
+			break;
 		case 'v':
 			args.verbose = 1;
 			break;
@@ -2867,7 +2984,6 @@ int main(int argc, char *argv[])
 		} while (ret);
 	}
 
-
 	free_test_data();
 
 out_free:
-- 
1.7.2.3

[PATCH 05/13] fs-tests: integck: implement data verification

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

When -e options is used - verify written data. Re-use the existing
file_check_data() function to implement this.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |   36 ++++++++++++++++++++++++------------
 1 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index f39745f..e93e141 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -885,8 +885,15 @@ static ssize_t file_write_data(struct file_info *file, int fd, off_t offset,
 	return actual;
 }
 
-static void file_write_info(struct file_info *file, off_t offset, size_t size,
-			    unsigned int seed)
+static void file_check_data(struct file_info *file, int fd,
+			    struct write_info *w);
+
+/*
+ * Save the information about a file write operation and verify the write if
+ * necessary.
+ */
+static void file_write_info(struct file_info *file, int fd, off_t offset,
+			    size_t size, unsigned int seed)
 {
 	struct write_info *new_write, *w, **prev, *tmp;
 	int inserted;
@@ -905,6 +912,9 @@ static void file_write_info(struct file_info *file, off_t offset, size_t size,
 	w->random_seed = seed;
 	file->raw_writes = w;
 
+	if (args.verify_ops)
+		file_check_data(file, fd, new_write);
+
 	/* Insert it into file->writes */
 	inserted = 0;
 	end = offset + size;
@@ -1077,17 +1087,14 @@ static int file_mmap_write(struct file_info *file)
 	fd = open(path, O_RDWR);
 	if (fd == -1) {
 		pcv("cannot open file %s to do mmap", path);
-		free(path);
-		return -1;
+		goto out_error;
 	}
 
 	/* mmap it */
 	addr = mmap(NULL, len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, offs);
-	CHECK(close(fd) == 0);
 	if (addr == MAP_FAILED) {
 		pcv("cannot mmap file %s", path);
-		free(path);
-		return -1;
+		goto out_error;
 	}
 
 	/* Randomly select a part of the mmapped area to write */
@@ -1108,14 +1115,19 @@ static int file_mmap_write(struct file_info *file)
 	/* Unmap it */
 	if (munmap(addr, len)) {
 		pcv("cannot unmap file %s", path);
-		free(path);
-		return -1;
+		goto out_error;
 	}
 
 	/* Record what was written */
-	file_write_info(file, offset, size, seed);
+	file_write_info(file, fd, offset, size, seed);
 	free(path);
+	CHECK(close(fd) == 0);
 	return 0;
+
+out_error:
+	free(path);
+	CHECK(close(fd) == 0);
+	return -1;
 }
 
 /*
@@ -1146,7 +1158,7 @@ static int file_write(struct file_info *file, int fd)
 			truncate = 1;
 
 	if (actual != 0)
-		file_write_info(file, offset, actual, seed);
+		file_write_info(file, fd, offset, actual, seed);
 
 	/* Delete errored files */
 	if (!fsinfo.nospc_size_ok && file->no_space_error)
@@ -1175,7 +1187,7 @@ static int file_write_file(struct file_info *file)
 	char *path;
 
 	path = dir_path(file->links->parent, file->links->name);
-	fd = open(path, O_WRONLY);
+	fd = open(path, O_RDWR);
 	if (fd == -1) {
 		pcv("cannot open file %s for writing", path);
 		free(path);
-- 
1.7.2.3

[PATCH 06/13] fs-tests: integck: implement holes verification

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

When -e option is used verify file holes after the truncate
operation. Re-use the existing 'file_check_hole()' function for
this.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |   77 +++++++++++++++++++----------------
 1 files changed, 42 insertions(+), 35 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index e93e141..f09cfaf 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -1017,7 +1017,46 @@ static void get_offset_and_size(struct file_info *file,
 		*size = 1;
 }
 
-static void file_truncate_info(struct file_info *file, size_t new_length);
+static void file_check_hole(struct file_info *file, int fd, off_t offset,
+			    size_t size);
+
+static void file_truncate_info(struct file_info *file, int fd,
+			       size_t new_length)
+{
+	struct write_info *w, **prev, *tmp;
+
+	/* Remove / truncate file->writes */
+	w = file->writes;
+	prev = &file->writes;
+	while (w) {
+		if (w->offset >= new_length) {
+			/* w comes after eof, so remove it */
+			*prev = w->next;
+			tmp = w;
+			w = w->next;
+			free(tmp);
+			continue;
+		}
+		if (w->offset + w->size > new_length)
+			w->size = new_length - w->offset;
+		prev = &w->next;
+		w = w->next;
+	}
+	/* Add an entry in raw_writes for the truncation */
+	w = zalloc(sizeof(struct write_info));
+	w->next = file->raw_writes;
+	w->offset = file->length;
+	w->new_length = new_length;
+	w->random_seed = MAX_RANDOM_SEED + 1;
+	file->raw_writes = w;
+
+	if (new_length > file->length)
+		file_check_hole(file, fd, file->length,
+				new_length - file->length);
+
+	/* Update file length */
+	file->length = new_length;
+}
 
 /*
  * Truncate a file to length 'new_length'. If there is no enough space to
@@ -1171,7 +1210,7 @@ static int file_write(struct file_info *file, int fd)
 		if (ret == -1)
 			return -1;
 		if (!ret)
-			file_truncate_info(file, new_length);
+			file_truncate_info(file, fd, new_length);
 	}
 
 	return 0;
@@ -1199,38 +1238,6 @@ static int file_write_file(struct file_info *file)
 	return ret;
 }
 
-static void file_truncate_info(struct file_info *file, size_t new_length)
-{
-	struct write_info *w, **prev, *tmp;
-
-	/* Remove / truncate file->writes */
-	w = file->writes;
-	prev = &file->writes;
-	while (w) {
-		if (w->offset >= new_length) {
-			/* w comes after eof, so remove it */
-			*prev = w->next;
-			tmp = w;
-			w = w->next;
-			free(tmp);
-			continue;
-		}
-		if (w->offset + w->size > new_length)
-			w->size = new_length - w->offset;
-		prev = &w->next;
-		w = w->next;
-	}
-	/* Add an entry in raw_writes for the truncation */
-	w = zalloc(sizeof(struct write_info));
-	w->next = file->raw_writes;
-	w->offset = file->length;
-	w->new_length = new_length;
-	w->random_seed = MAX_RANDOM_SEED + 1;
-	file->raw_writes = w;
-	/* Update file length */
-	file->length = new_length;
-}
-
 /*
  * Truncate an open file randomly.
  */
@@ -1244,7 +1251,7 @@ static int file_truncate(struct file_info *file, int fd)
 	if (ret == -1)
 		return -1;
 	if (!ret)
-		file_truncate_info(file, new_length);
+		file_truncate_info(file, fd, new_length);
 	return 0;
 }
 
-- 
1.7.2.3

[PATCH 07/13] fs-tests: integck: add debug compilation target

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

I often need to use gdb to debug integck, or sometimes even attach
already running integck. But when it is compiled with optimizations, it
is nearly impossible to use gdb. Introduce 'debug' target to compile
integck without optimizations.

Additionally, to make the stack backtraces work, add a dirty hack to
integck.c to remove all the static keywords - this turns static
functions to non-static and makes them appear in stack backtraces.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/Makefile  |    5 +++++
 tests/fs-tests/integrity/integck.c |   10 ++++++++++
 2 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/tests/fs-tests/integrity/Makefile b/tests/fs-tests/integrity/Makefile
index 976c3ce..2cdd470 100644
--- a/tests/fs-tests/integrity/Makefile
+++ b/tests/fs-tests/integrity/Makefile
@@ -13,5 +13,10 @@ TARGETS = integck
 
 all: $(TARGETS)
 
+# Disable optimizations to make it possible to use gdb comfortably
+# Use -rdynamic to have stack backtraces
+debug:
+	gcc $(CFLAGS) -O0 -D INTEGCK_DEBUG -rdynamic integck.c -o integck
+
 clean:
 	rm -f *.o $(TARGETS)
diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index f09cfaf..fbd6cc5 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -41,6 +41,16 @@
 #define PROGRAM_NAME "integck"
 #include "common.h"
 
+/*
+ * WARNING! This is a dirty hack! The symbols for static functions are not
+ * printed in the stack backtrace. So we remove ths 'static' keyword using the
+ * pre-processor. This is really error-prone because this won't work if, e.g.,
+ * local static variables were used.
+ */
+#ifdef INTEGCK_DEBUG
+#define static
+#endif
+
 #define MAX_RANDOM_SEED 10000000
 
 /* The pattern for the top directory where we run the test */
-- 
1.7.2.3

[PATCH 08/13] fs-tests: use independent random generators for ops and data

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

Currently integck uses the same global random generator for everything -
for choosing the operation, generating the data, and for checking. This
makes integck to become stuck sometimes. My guess this is because of
we somehow re-set it back with srand() when checking files.

This patch makes integck use different generators for data and for
choosing operations by using rand_r() with own seed for operations.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index fbd6cc5..5fb990e 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -214,6 +214,8 @@ static uint64_t operation_count = 0; /* Number of operations used to fill
                                         up the file system */
 static unsigned int check_run_no;
 
+static unsigned int random_seed;
+
 /*
  * A buffer which is used by 'make_name()' to return the generated random name.
  */
@@ -260,7 +262,7 @@ static unsigned int random_no(unsigned int max)
 	assert(max < RAND_MAX);
 	if (max == 0)
 		return 0;
-	return rand() % max;
+	return rand_r(&random_seed) % max;
 }
 
 /*
@@ -2952,6 +2954,7 @@ int main(int argc, char *argv[])
 {
 	int ret;
 	long rpt;
+	unsigned int pid = getpid();
 
 	ret = parse_opts(argc, argv);
 	if (ret)
@@ -2960,7 +2963,8 @@ int main(int argc, char *argv[])
 	get_tested_fs_info();
 
 	/* Seed the random generator with out PID */
-	srand(getpid());
+	srand(pid);
+	random_seed = pid;
 
 	random_name_buf = malloc(fsinfo.max_name_len + 1);
 	CHECK(random_name_buf != NULL);
-- 
1.7.2.3

[PATCH 09/13] fs-tests: integck: use rand_r everywhere

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

Instead of using rand_r with own seed for operations and rand() with
global state for data it is cleaner to use rand_r with own different
states for these things. This patch eliminates 'srand()' and 'rand()'
usage.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |   44 ++++++++++++++++-------------------
 1 files changed, 20 insertions(+), 24 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 5fb990e..0f48377 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -131,11 +131,11 @@ struct write_info /* Record of random data written into a file */
 	struct write_info *next;
 	off_t offset; /* Where in the file the data was written */
 	union {
-		off_t random_offset; /* Call rand() this number of times first */
+		off_t random_offset; /* Call rand_r() this number of times first */
 		off_t new_length; /* For truncation records new file length */
 	};
 	size_t size; /* Number of bytes written */
-	unsigned int random_seed; /* Seed for rand() to create random data. If
+	unsigned int random_seed; /* Seed for rand_r() to create random data. If
 				     greater than MAX_RANDOM_SEED then this is
 				     a truncation record (raw_writes only) */
 };
@@ -860,7 +860,6 @@ static ssize_t file_write_data(struct file_info *file, int fd, off_t offset,
 	ssize_t written;
 	char buf[IO_BUFFER_SIZE];
 
-	srand(seed);
 	CHECK(lseek(fd, offset, SEEK_SET) != (off_t)-1);
 	remains = size;
 	actual = 0;
@@ -873,7 +872,7 @@ static ssize_t file_write_data(struct file_info *file, int fd, off_t offset,
 		} else
 			written = 0;
 		for (; written < IO_BUFFER_SIZE; ++written)
-			buf[written] = rand();
+			buf[written] = rand_r(&seed);
 		/* Write a block of data */
 		if (remains > IO_BUFFER_SIZE)
 			block = IO_BUFFER_SIZE;
@@ -1107,7 +1106,7 @@ static int file_mmap_write(struct file_info *file)
 	void *addr;
 	char *waddr, *path;
 	off_t offs, offset;
-	unsigned int seed;
+	unsigned int seed, seed_tmp;
 	uint64_t free_space;
 	int fd;
 
@@ -1157,11 +1156,10 @@ static int file_mmap_write(struct file_info *file)
 	offset = w->offset + random_no(w->size - size);
 
 	/* Write it */
-	seed = random_no(MAX_RANDOM_SEED);
-	srand(seed);
+	seed_tmp = seed = random_no(MAX_RANDOM_SEED);
 	waddr = addr + (offset - offs);
 	for (i = 0; i < size; i++)
-		waddr[i] = rand();
+		waddr[i] = rand_r(&seed_tmp);
 
 	/* Unmap it */
 	if (munmap(addr, len)) {
@@ -1318,10 +1316,10 @@ static void file_rewrite_data(int fd, struct write_info *w, char *buf)
 	size_t remains, block;
 	ssize_t written;
 	off_t r;
+	unsigned int seed = w->random_seed;
 
-	srand(w->random_seed);
 	for (r = 0; r < w->random_offset; ++r)
-		rand();
+		rand_r(&seed);
 	CHECK(lseek(fd, w->offset, SEEK_SET) != (off_t)-1);
 	remains = w->size;
 	written = IO_BUFFER_SIZE;
@@ -1332,7 +1330,7 @@ static void file_rewrite_data(int fd, struct write_info *w, char *buf)
 		else
 			written = 0;
 		for (; written < IO_BUFFER_SIZE; ++written)
-			buf[written] = rand();
+			buf[written] = rand_r(&seed);
 		/* Write a block of data */
 		if (remains > IO_BUFFER_SIZE)
 			block = IO_BUFFER_SIZE;
@@ -1419,10 +1417,10 @@ static void file_check_data(struct file_info *file, int fd,
 	size_t remains, block, i;
 	off_t r;
 	char buf[IO_BUFFER_SIZE];
+	unsigned int seed = w->random_seed;
 
-	srand(w->random_seed);
 	for (r = 0; r < w->random_offset; ++r)
-		rand();
+		rand_r(&seed);
 	CHECK(lseek(fd, w->offset, SEEK_SET) != (off_t)-1);
 	remains = w->size;
 	while (remains) {
@@ -1432,7 +1430,7 @@ static void file_check_data(struct file_info *file, int fd,
 			block = remains;
 		CHECK(read(fd, buf, block) == block);
 		for (i = 0; i < block; ++i) {
-			char c = (char)rand();
+			char c = (char)rand_r(&seed);
 			if (buf[i] != c) {
 				errmsg("file_check_data failed at %zu checking "
 				       "data at %llu size %zu", w->size - remains + i,
@@ -2404,11 +2402,11 @@ static int remount_tested_fs(void)
 	CHECK(chdir("/") == 0);
 
 	/* Choose what to do */
-	rorw1 = rand() & 1;
-	um = rand() & 1;
-	um_ro = rand() & 1;
-	um_rorw = rand() & 1;
-	rorw2 = rand() & 1;
+	rorw1 = random_no(2);
+	um = random_no(2);
+	um_ro = random_no(2);
+	um_rorw = random_no(2);
+	rorw2 = random_no(2);
 
 	if (rorw1 + um + rorw2 == 0)
 		um = 1;
@@ -2876,8 +2874,8 @@ static int recover_tested_fs(void)
 	CHECK(chdir("/") == 0);
 
 	/* Choose what to do */
-	um_rorw = rand() & 1;
-	rorw2 = rand() & 1;
+	um_rorw = random_no(2);
+	rorw2 = random_no(2);
 
 	/*
 	 * At this point we do not know for sure whether the tested FS is
@@ -2954,7 +2952,6 @@ int main(int argc, char *argv[])
 {
 	int ret;
 	long rpt;
-	unsigned int pid = getpid();
 
 	ret = parse_opts(argc, argv);
 	if (ret)
@@ -2963,8 +2960,7 @@ int main(int argc, char *argv[])
 	get_tested_fs_info();
 
 	/* Seed the random generator with out PID */
-	srand(pid);
-	random_seed = pid;
+	random_seed = getpid();
 
 	random_name_buf = malloc(fsinfo.max_name_len + 1);
 	CHECK(random_name_buf != NULL);
-- 
1.7.2.3

[PATCH 10/13] fs-tests: integck: make -v switch work

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

-v (verbose) switch is supposed to turn power cut testing error
messages on/off. However, it does not work because the messages
are printed even if -v was not specified. This patch fixes it.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 0f48377..372a348 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -70,7 +70,7 @@
 } while(0)
 
 #define pcv(fmt, ...) do {                                         \
-	if (args.power_cut_mode)                                   \
+	if (args.power_cut_mode && args.verbose)                   \
 		normsg(fmt " (line %d)", ##__VA_ARGS__, __LINE__); \
 } while(0)
 
-- 
1.7.2.3

[PATCH 11/13] fs-tests: integck: re-mount always if not root fs

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

Commit 08cfd9d7cacc3c0b97530ad7e13fa8db75e99947 introduced a bug:
we do not re-mount the FS before checking at the first iteration.
This patch fixes the issue.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 372a348..9a2f3b4 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -2534,7 +2534,7 @@ static int integck(void)
 	if (ret)
 		return -1;
 
-	if (fsinfo.is_rootfs) {
+	if (!fsinfo.is_rootfs) {
 		close_open_files();
 		ret = remount_tested_fs();
 		if (ret)
-- 
1.7.2.3

[PATCH 12/13] fs-tests: integck: improve remountability logic

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

The integck test assumes that if the FS is rootfs then it cannot
re-mount it, otherwise it can. However, this is not true because the
tested FS can be not remountable even if it is not rootfs, e.g., if
'integck' lives on this FS and is executed from it.

This patch improves the integck logic and actually checks wheter it
is possible to re-mount the FS before starting the test.

Also, it makes sure that power cut testing is run only on re-mountable
FS, because we have to be able to clean the "corrupted" state of the
file-system.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |   46 +++++++++++++++++++++++++++--------
 1 files changed, 35 insertions(+), 11 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 9a2f3b4..55fe8be 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -95,7 +95,7 @@ static struct {
  * nospc_size_ok: file size is updated even if the write operation failed with
  *                ENOSPC error
  * can_mmap: file-system supports share writable 'mmap()' operation
- * is_rootfs: the tested file-system the root file-system
+ * can_remount: is it possible to re-mount the tested file-system?
  * fstype: file-system type (e.g., "ubifs")
  * fsdev: the underlying device mounted by the tested file-system
  * mount_opts: non-standard mount options of the tested file-system (non-standard
@@ -111,7 +111,7 @@ static struct {
 	unsigned int log10_initial_free;
 	unsigned int nospc_size_ok:1;
 	unsigned int can_mmap:1;
-	unsigned int is_rootfs:1;
+	unsigned int can_remount:1;
 	char *fstype;
 	char *fsdev;
 	char *mount_opts;
@@ -2534,12 +2534,13 @@ static int integck(void)
 	if (ret)
 		return -1;
 
-	if (!fsinfo.is_rootfs) {
+	if (fsinfo.can_remount) {
 		close_open_files();
 		ret = remount_tested_fs();
 		if (ret)
 			return -1;
-	}
+	} else
+		assert(!args.power_cut_mode);
 
 	/* Check everything */
 	check_run_no += 1;
@@ -2551,7 +2552,7 @@ static int integck(void)
 		if (ret)
 			return -1;
 
-		if (!fsinfo.is_rootfs) {
+		if (fsinfo.can_remount) {
 			close_open_files();
 			ret = remount_tested_fs();
 			if (ret)
@@ -2668,7 +2669,6 @@ static void get_tested_fs_info(void)
 	uint64_t z;
 	char *p;
 	unsigned int pid;
-	struct stat st1, st2;
 
 	/* Remove trailing '/' symbols from the mount point */
 	p = dup_string(args.mount_point);
@@ -2716,11 +2716,6 @@ static void get_tested_fs_info(void)
 
 	normsg("pid %u, testing \"%s\" at \"%s\"",
 	       pid, fsinfo.fstype, fsinfo.mount_point);
-
-	CHECK(stat(fsinfo.mount_point, &st1) == 0);
-	CHECK(stat("/", &st2) != -1);
-	if (st1.st_dev == st2.st_dev)
-		fsinfo.is_rootfs = 1;
 }
 
 static const char doc[] = PROGRAM_NAME " version " PROGRAM_VERSION
@@ -2972,6 +2967,35 @@ int main(int argc, char *argv[])
 		goto out_free;
 	}
 
+	/* Check whether we can re-mount the tested FS  */
+	do {
+		ret = recover_tested_fs();
+	} while (ret && args.power_cut_mode && errno == EROFS);
+
+	if (!ret) {
+		fsinfo.can_remount = 1;
+	} else {
+		warnmsg("file-system %s cannot be unmounted (%s)",
+			fsinfo.mount_point, strerror(errno));
+		if (args.power_cut_mode) {
+			/*
+			 * When testing emulated power cuts we have to be able
+			 * to re-mount the file-system to clean the EROFS
+			 * state.
+			 *
+			 * But there is also another reason. Imaging the test
+			 * writes many files successfully, and decides to check
+			 * them. But the test has done many modifications, so
+			 * there will be write-back. And when write-back fails,
+			 * Linux discards the dirty pages. So, if meanwhile FS
+			 * write-back encounters emulated power cut error, the
+			 * file checking will fail as well.
+			 */
+			errmsg("power cut mode requers unmountable FS");
+			goto out_free;
+		}
+	}
+
 	/* Do the actual test */
 	for (rpt = 0; ; rpt++) {
 		top_dir = zalloc(sizeof(struct dir_info));
-- 
1.7.2.3

[PATCH 13/13] fs-tests: integck: print a bit less if in power cut mode

[Artem Bityutskiy]

From: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

Do not print extra messages when unable to re-mount - they are
already printed by the recover function.

Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
---
 tests/fs-tests/integrity/integck.c |   25 ++++++++++++-------------
 1 files changed, 12 insertions(+), 13 deletions(-)

diff --git a/tests/fs-tests/integrity/integck.c b/tests/fs-tests/integrity/integck.c
index 55fe8be..63f6956 100644
--- a/tests/fs-tests/integrity/integck.c
+++ b/tests/fs-tests/integrity/integck.c
@@ -3014,6 +3014,18 @@ int main(int argc, char *argv[])
 
 		free_test_data();
 
+		do {
+			ret = recover_tested_fs();
+			if (ret) {
+				CHECK(errno == EROFS);
+				rpt += 1;
+			}
+			/*
+			 * Mount may also fail due to an emulated power cut
+			 * while mounting - keep re-starting.
+			 */
+		} while (ret);
+
 		/*
 		 * The file-system became read-only and we are in power cut
 		 * testing mode. Re-mount the file-system and re-start the
@@ -3022,19 +3034,6 @@ int main(int argc, char *argv[])
 		if (args.verbose)
 			normsg("re-mount the FS and re-start - count %ld", rpt);
 
-		do {
-			ret = recover_tested_fs();
-			if (ret) {
-				CHECK(errno == EROFS);
-				/*
-				 * Mount may also fail due to an emulated power
-				 * cut while mounting - keep re-starting.
-				 */
-				if (args.verbose)
-					normsg("could not mount, try again - count %ld",
-					       ++rpt);
-			}
-		} while (ret);
 	}
 
 	free_test_data();
-- 
1.7.2.3