[PATCH v4 0/8] Crash consistency xfstest using dm-log-writes

[PATCH v4 0/8] Crash consistency xfstest using dm-log-writes

[Amir Goldstein]

Eryu,

These are the rest of the patches following the fsx patches
v4 posting.

There are no changes to the replay-log patches since v3.

The test patches address your v3 review comments.

The status of tests in current upstream:
- 500 fails sometimes on btrfs/ext4/xfs
- 501 fails always on ext4
- 502 fails sometimes on xfs
- 503 fails always on xfs

The data loss issue found by test 500 on xfs is fixed by
commit 47c7d0b19502 ("xfs: fix incorrect log_flushed on fsync")
now in master.

Amir.

Amir Goldstein (8):
  log-writes: add replay-log program to replay dm-log-writes target
  replay-log: add validations for corrupt log entries
  replay-log: add support for replaying ops in target device sector
  fstests: add support for working with dm-log-writes target
  fstests: crash consistency fsx test using dm-log-writes
  fstests: regression test for ext4 crash consistency bug
  fstests: crash consistency fsx test for cloned files
  fstests: regression test for xfs leftover CoW extent error

 .gitignore                   |   1 +
 README                       |   2 +
 common/dmlogwrites           |  90 +++++++++
 doc/auxiliary-programs.txt   |   8 +
 doc/requirement-checking.txt |  20 ++
 src/Makefile                 |   2 +-
 src/log-writes/Makefile      |  23 +++
 src/log-writes/SOURCE        |   6 +
 src/log-writes/log-writes.c  | 458 +++++++++++++++++++++++++++++++++++++++++++
 src/log-writes/log-writes.h  |  81 ++++++++
 src/log-writes/replay-log.c  | 389 ++++++++++++++++++++++++++++++++++++
 tests/generic/500            | 138 +++++++++++++
 tests/generic/500.out        |   2 +
 tests/generic/501            |  80 ++++++++
 tests/generic/501.out        |   2 +
 tests/generic/502            | 141 +++++++++++++
 tests/generic/502.out        |   2 +
 tests/generic/503            |  74 +++++++
 tests/generic/503.out        |   2 +
 tests/generic/group          |   4 +
 20 files changed, 1524 insertions(+), 1 deletion(-)
 create mode 100644 common/dmlogwrites
 create mode 100644 src/log-writes/Makefile
 create mode 100644 src/log-writes/SOURCE
 create mode 100644 src/log-writes/log-writes.c
 create mode 100644 src/log-writes/log-writes.h
 create mode 100644 src/log-writes/replay-log.c
 create mode 100755 tests/generic/500
 create mode 100644 tests/generic/500.out
 create mode 100755 tests/generic/501
 create mode 100644 tests/generic/501.out
 create mode 100755 tests/generic/502
 create mode 100644 tests/generic/502.out
 create mode 100755 tests/generic/503
 create mode 100644 tests/generic/503.out

-- 
2.7.4

[PATCH v4 1/8] log-writes: add replay-log program to replay dm-log-writes target

[Amir Goldstein]

Imported Josef Bacik's code from:
https://github.com/josefbacik/log-writes.git

Specialized program for replaying a write log that was recorded by
device mapper log-writes target.  The tools is used to perform
crash consistency tests, allowing to run an arbitrary check tool
(fsck) at specified checkpoints in the write log.

[Amir:]
- Add project Makefile and SOURCE files
- Document the replay-log auxiliary program
- Address review comments by Eryu Guan

Cc: Josef Bacik <jbacik@fb.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 .gitignore                  |   1 +
 doc/auxiliary-programs.txt  |   8 +
 src/Makefile                |   2 +-
 src/log-writes/Makefile     |  23 +++
 src/log-writes/SOURCE       |   6 +
 src/log-writes/log-writes.c | 381 ++++++++++++++++++++++++++++++++++++++++++++
 src/log-writes/log-writes.h |  77 +++++++++
 src/log-writes/replay-log.c | 357 +++++++++++++++++++++++++++++++++++++++++
 8 files changed, 854 insertions(+), 1 deletion(-)
 create mode 100644 src/log-writes/Makefile
 create mode 100644 src/log-writes/SOURCE
 create mode 100644 src/log-writes/log-writes.c
 create mode 100644 src/log-writes/log-writes.h
 create mode 100644 src/log-writes/replay-log.c

diff --git a/.gitignore b/.gitignore
index 28fe84d..beb9504 100644
--- a/.gitignore
+++ b/.gitignore
@@ -154,6 +154,7 @@
 /src/t_mmap_stale_pmd
 /src/t_mmap_cow_race
 /src/t_mmap_fallocate
+/src/log-writes/replay-log
 
 # dmapi/ binaries
 /dmapi/src/common/cmd/read_invis
diff --git a/doc/auxiliary-programs.txt b/doc/auxiliary-programs.txt
index bcab453..de15832 100644
--- a/doc/auxiliary-programs.txt
+++ b/doc/auxiliary-programs.txt
@@ -18,6 +18,7 @@ Contents:
  - af_unix		-- Create an AF_UNIX socket
  - dmerror		-- fault injection block device control
  - fsync-err		-- tests fsync error reporting after failed writeback
+ - log-writes/replay-log -- Replay log from device mapper log-writes target
  - open_by_handle	-- open_by_handle_at syscall exercise
  - stat_test		-- statx syscall exercise
  - t_dir_type		-- print directory entries and their file type
@@ -46,6 +47,13 @@ fsync-err
 	writeback and test that errors are reported during fsync and cleared
 	afterward.
 
+log-writes/replay-log
+
+	Specialized program for replaying a write log that was recorded by
+	device mapper log-writes target.  The tools is used to perform crash
+	consistency tests, allowing to run an arbitrary check tool (fsck) at
+	specified checkpoints in the write log.
+
 open_by_handle
 
 	The open_by_handle program exercises the open_by_handle_at() system
diff --git a/src/Makefile b/src/Makefile
index b8aff49..7d1306b 100644
--- a/src/Makefile
+++ b/src/Makefile
@@ -25,7 +25,7 @@ LINUX_TARGETS = xfsctl bstat t_mtab getdevicesize preallo_rw_pattern_reader \
 	attr-list-by-handle-cursor-test listxattr dio-interleaved t_dir_type \
 	dio-invalidate-cache stat_test t_encrypted_d_revalidate
 
-SUBDIRS =
+SUBDIRS = log-writes
 
 LLDLIBS = $(LIBATTR) $(LIBHANDLE) $(LIBACL) -lpthread
 
diff --git a/src/log-writes/Makefile b/src/log-writes/Makefile
new file mode 100644
index 0000000..d114177
--- /dev/null
+++ b/src/log-writes/Makefile
@@ -0,0 +1,23 @@
+TOPDIR = ../..
+include $(TOPDIR)/include/builddefs
+
+TARGETS = replay-log
+
+CFILES = replay-log.c log-writes.c
+LDIRT = $(TARGETS)
+
+default: depend $(TARGETS)
+
+depend: .dep
+
+include $(BUILDRULES)
+
+$(TARGETS): $(CFILES)
+	@echo "    [CC]    $@"
+	$(Q)$(LTLINK) $(CFILES) -o $@ $(CFLAGS) $(LDFLAGS) $(LDLIBS)
+
+install:
+	$(INSTALL) -m 755 -d $(PKG_LIB_DIR)/src/log-writes
+	$(INSTALL) -m 755 $(TARGETS) $(PKG_LIB_DIR)/src/log-writes
+
+-include .dep
diff --git a/src/log-writes/SOURCE b/src/log-writes/SOURCE
new file mode 100644
index 0000000..d6d143c
--- /dev/null
+++ b/src/log-writes/SOURCE
@@ -0,0 +1,6 @@
+From:
+https://github.com/josefbacik/log-writes.git
+
+description	Helper code for dm-log-writes target
+owner	Josef Bacik <jbacik@fb.com>
+URL	https://github.com/josefbacik/log-writes.git
diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
new file mode 100644
index 0000000..a215fef
--- /dev/null
+++ b/src/log-writes/log-writes.c
@@ -0,0 +1,381 @@
+#include <linux/fs.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/ioctl.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <unistd.h>
+#include <string.h>
+#include "log-writes.h"
+
+int log_writes_verbose = 0;
+
+/*
+ * @log: the log to free.
+ *
+ * This will close any open fd's the log has and free up its memory.
+ */
+void log_free(struct log *log)
+{
+	if (log->replayfd >= 0)
+		close(log->replayfd);
+	if (log->logfd >= 0)
+		close(log->logfd);
+	free(log);
+}
+
+static int discard_range(struct log *log, u64 start, u64 len)
+{
+	u64 range[2] = { start, len };
+
+	if (ioctl(log->replayfd, BLKDISCARD, &range) < 0) {
+		if (log_writes_verbose)
+			printf("replay device doesn't support discard, "
+			       "switching to writing zeros\n");
+		log->flags |= LOG_DISCARD_NOT_SUPP;
+	}
+	return 0;
+}
+
+static int zero_range(struct log *log, u64 start, u64 len)
+{
+	u64 bufsize = len;
+	ssize_t ret;
+	char *buf = NULL;
+
+	if (log->max_zero_size < len) {
+		if (log_writes_verbose)
+			printf("discard len %llu larger than max %llu\n",
+			       (unsigned long long)len,
+			       (unsigned long long)log->max_zero_size);
+		return 0;
+	}
+
+	while (!buf) {
+		buf = malloc(bufsize);
+		if (!buf)
+			bufsize >>= 1;
+		if (!bufsize) {
+			fprintf(stderr, "Couldn't allocate zero buffer");
+			return -1;
+		}
+	}
+
+	memset(buf, 0, bufsize);
+	while (len) {
+		ret = pwrite(log->replayfd, buf, bufsize, start);
+		if (ret != bufsize) {
+			fprintf(stderr, "Error zeroing file: %d\n", errno);
+			free(buf);
+			return -1;
+		}
+		len -= ret;
+		start += ret;
+	}
+	free(buf);
+	return 0;
+}
+
+/*
+ * @log: the log we are replaying.
+ * @entry: the discard entry.
+ *
+ * Discard the given length.  If the device supports discard we will call that
+ * ioctl, otherwise we will write 0's to emulate discard.  If the discard size
+ * is larger than log->max_zero_size then we will simply skip the zero'ing if
+ * the drive doesn't support discard.
+ */
+int log_discard(struct log *log, struct log_write_entry *entry)
+{
+	u64 start = le64_to_cpu(entry->sector) * log->sectorsize;
+	u64 size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
+	u64 max_chunk = 1 * 1024 * 1024 * 1024;
+
+	if (log->flags & LOG_IGNORE_DISCARD)
+		return 0;
+
+	while (size) {
+		u64 len = size > max_chunk ? max_chunk : size;
+		int ret;
+
+		/*
+		 * Do this check first in case it is our first discard, that way
+		 * if we return EOPNOTSUPP we will fall back to the 0 method
+		 * automatically.
+		 */
+		if (!(log->flags & LOG_DISCARD_NOT_SUPP))
+			ret = discard_range(log, start, len);
+		if (log->flags & LOG_DISCARD_NOT_SUPP)
+			ret = zero_range(log, start, len);
+		if (ret)
+			return -1;
+		size -= len;
+		start += len;
+	}
+	return 0;
+}
+
+/*
+ * @log: the log we are replaying.
+ * @entry: where we put the entry.
+ * @read_data: read the entry data as well, entry must be log->sectorsize sized
+ * if this is set.
+ *
+ * @return: 0 if we replayed, 1 if we are at the end, -1 if there was an error.
+ *
+ * Replay the next entry in our log onto the replay device.
+ */
+int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
+			  int read_data)
+{
+	u64 size;
+	u64 flags;
+	size_t read_size = read_data ? log->sectorsize :
+		sizeof(struct log_write_entry);
+	char *buf;
+	ssize_t ret;
+	off_t offset;
+
+	if (log->cur_entry >= log->nr_entries)
+		return 1;
+
+	ret = read(log->logfd, entry, read_size);
+	if (ret != read_size) {
+		fprintf(stderr, "Error reading entry: %d\n", errno);
+		return -1;
+	}
+	log->cur_entry++;
+
+	size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
+	if (read_size < log->sectorsize) {
+		if (lseek(log->logfd,
+			  log->sectorsize - sizeof(struct log_write_entry),
+			  SEEK_CUR) == (off_t)-1) {
+			fprintf(stderr, "Error seeking in log: %d\n", errno);
+			return -1;
+		}
+	}
+
+	if (log_writes_verbose)
+		printf("replaying %d: sector %llu, size %llu, flags %llu\n",
+		       (int)log->cur_entry - 1,
+		       (unsigned long long)le64_to_cpu(entry->sector),
+		       (unsigned long long)size,
+		       (unsigned long long)le64_to_cpu(entry->flags));
+	if (!size)
+		return 0;
+
+	flags = le64_to_cpu(entry->flags);
+	if (flags & LOG_DISCARD_FLAG)
+		return log_discard(log, entry);
+
+	buf = malloc(size);
+	if (!buf) {
+		fprintf(stderr, "Error allocating buffer %llu entry %llu\n", (unsigned long long)size, (unsigned long long)log->cur_entry - 1);
+		return -1;
+	}
+
+	ret = read(log->logfd, buf, size);
+	if (ret != size) {
+		fprintf(stderr, "Error reading data: %d\n", errno);
+		free(buf);
+		return -1;
+	}
+
+	offset = le64_to_cpu(entry->sector) * log->sectorsize;
+	ret = pwrite(log->replayfd, buf, size, offset);
+	free(buf);
+	if (ret != size) {
+		fprintf(stderr, "Error writing data: %d\n", errno);
+		return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * @log: the log we are manipulating.
+ * @entry_num: the entry we want.
+ *
+ * Seek to the given entry in the log, starting at 0 and ending at
+ * log->nr_entries - 1.
+ */
+int log_seek_entry(struct log *log, u64 entry_num)
+{
+	u64 i = 0;
+
+	if (entry_num >= log->nr_entries) {
+		fprintf(stderr, "Invalid entry number\n");
+		return -1;
+	}
+
+	/* Skip the first sector containing the log super block */
+	if (lseek(log->logfd, log->sectorsize, SEEK_SET) == (off_t)-1) {
+		fprintf(stderr, "Error seeking in file: %d\n", errno);
+		return -1;
+	}
+
+	log->cur_entry = 0;
+	for (i = 0; i < entry_num; i++) {
+		struct log_write_entry entry;
+		ssize_t ret;
+		off_t seek_size;
+		u64 flags;
+
+		ret = read(log->logfd, &entry, sizeof(entry));
+		if (ret != sizeof(entry)) {
+			fprintf(stderr, "Error reading entry: %d\n", errno);
+			return -1;
+		}
+		if (log_writes_verbose > 1)
+			printf("seek entry %d: %llu, size %llu, flags %llu\n",
+			       (int)i,
+			       (unsigned long long)le64_to_cpu(entry.sector),
+			       (unsigned long long)le64_to_cpu(entry.nr_sectors),
+			       (unsigned long long)le64_to_cpu(entry.flags));
+		flags = le64_to_cpu(entry.flags);
+		seek_size = log->sectorsize - sizeof(entry);
+		if (!(flags & LOG_DISCARD_FLAG))
+			seek_size += le64_to_cpu(entry.nr_sectors) *
+				log->sectorsize;
+		if (lseek(log->logfd, seek_size, SEEK_CUR) == (off_t)-1) {
+			fprintf(stderr, "Error seeking in file: %d\n", errno);
+			return -1;
+		}
+		log->cur_entry++;
+	}
+
+	return 0;
+}
+
+/*
+ * @log: the log we are manipulating.
+ * @entry: the entry we read.
+ * @read_data: read the extra data for the entry, your entry must be
+ * log->sectorsize large.
+ *
+ * @return: 1 if we hit the end of the log, 0 we got the next entry, < 0 if
+ * there was an error.
+ *
+ * Seek to the next entry in the log.
+ */
+int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
+			int read_data)
+{
+	size_t read_size = read_data ? log->sectorsize :
+		sizeof(struct log_write_entry);
+	u64 flags;
+	ssize_t ret;
+
+	if (log->cur_entry >= log->nr_entries)
+		return 1;
+
+	ret = read(log->logfd, entry, read_size);
+	if (ret != read_size) {
+		fprintf(stderr, "Error reading entry: %d\n", errno);
+		return -1;
+	}
+	log->cur_entry++;
+
+	if (read_size < log->sectorsize) {
+		if (lseek(log->logfd,
+			  log->sectorsize - sizeof(struct log_write_entry),
+			  SEEK_CUR) == (off_t)-1) {
+			fprintf(stderr, "Error seeking in log: %d\n", errno);
+			return -1;
+		}
+	}
+	if (log_writes_verbose > 1)
+		printf("seek entry %d: %llu, size %llu, flags %llu\n",
+		       (int)log->cur_entry - 1,
+		       (unsigned long long)le64_to_cpu(entry->sector),
+		       (unsigned long long)le64_to_cpu(entry->nr_sectors),
+		       (unsigned long long)le64_to_cpu(entry->flags));
+
+	flags = le32_to_cpu(entry->flags);
+	read_size = le32_to_cpu(entry->nr_sectors) * log->sectorsize;
+	if (!read_size || (flags & LOG_DISCARD_FLAG))
+		return 0;
+
+	if (lseek(log->logfd, read_size, SEEK_CUR) == (off_t)-1) {
+		fprintf(stderr, "Error seeking in log: %d\n", errno);
+		return -1;
+	}
+
+	return 0;
+}
+
+/*
+ * @logfile: the file that contains the write log.
+ * @replayfile: the file/device to replay onto, can be NULL.
+ *
+ * Opens a logfile and makes sure it is valid and returns a struct log.
+ */
+struct log *log_open(char *logfile, char *replayfile)
+{
+	struct log *log;
+	struct log_write_super super;
+	ssize_t ret;
+
+	log = malloc(sizeof(struct log));
+	if (!log) {
+		fprintf(stderr, "Couldn't alloc log\n");
+		return NULL;
+	}
+
+	log->replayfd = -1;
+
+	log->logfd = open(logfile, O_RDONLY);
+	if (log->logfd < 0) {
+		fprintf(stderr, "Couldn't open log %s: %d\n", logfile,
+			errno);
+		log_free(log);
+		return NULL;
+	}
+
+	if (replayfile) {
+		log->replayfd = open(replayfile, O_WRONLY);
+		if (log->replayfd < 0) {
+			fprintf(stderr, "Couldn't open replay file %s: %d\n",
+				replayfile, errno);
+			log_free(log);
+			return NULL;
+		}
+	}
+
+	ret = read(log->logfd, &super, sizeof(struct log_write_super));
+	if (ret < sizeof(struct log_write_super)) {
+		fprintf(stderr, "Error reading super: %d\n", errno);
+		log_free(log);
+		return NULL;
+	}
+
+	if (le64_to_cpu(super.magic) != WRITE_LOG_MAGIC) {
+		fprintf(stderr, "Magic doesn't match\n");
+		log_free(log);
+		return NULL;
+	}
+
+	if (le64_to_cpu(super.version) != WRITE_LOG_VERSION) {
+		fprintf(stderr, "Version mismatch, wanted %d, have %d\n",
+			WRITE_LOG_VERSION, (int)le64_to_cpu(super.version));
+		log_free(log);
+		return NULL;
+	}
+
+	log->sectorsize = le32_to_cpu(super.sectorsize);
+	log->nr_entries = le64_to_cpu(super.nr_entries);
+	log->max_zero_size = 128 * 1024 * 1024;
+
+	if (lseek(log->logfd, log->sectorsize - sizeof(super), SEEK_CUR) ==
+	    (off_t) -1) {
+		fprintf(stderr, "Error seeking to first entry: %d\n", errno);
+		log_free(log);
+		return NULL;
+	}
+	log->cur_entry = 0;
+
+	return log;
+}
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
new file mode 100644
index 0000000..6cadb66
--- /dev/null
+++ b/src/log-writes/log-writes.h
@@ -0,0 +1,77 @@
+#ifndef _LOG_WRITES_H_
+#define _LOG_WRITES_H_
+
+#include <linux/types.h>
+#include <linux/byteorder/little_endian.h>
+
+extern int log_writes_verbose;
+
+#define le64_to_cpu __le64_to_cpu
+#define le32_to_cpu __le32_to_cpu
+
+typedef __u64 u64;
+typedef __u32 u32;
+
+/*
+ * Constants copied from kernel file drivers/md/dm-log-writes.c
+ */
+#define LOG_FLUSH_FLAG (1 << 0)
+#define LOG_FUA_FLAG (1 << 1)
+#define LOG_DISCARD_FLAG (1 << 2)
+#define LOG_MARK_FLAG (1 << 3)
+
+#define WRITE_LOG_VERSION 1
+#define WRITE_LOG_MAGIC 0x6a736677736872
+
+
+/*
+ * Basic info about the log for userspace.
+ *
+ * Copied from kernel file drivers/md/dm-log-writes.c
+ */
+struct log_write_super {
+	__le64 magic;
+	__le64 version;
+	__le64 nr_entries;
+	__le32 sectorsize;
+};
+
+/*
+ * sector - the sector we wrote.
+ * nr_sectors - the number of sectors we wrote.
+ * flags - flags for this log entry.
+ * data_len - the size of the data in this log entry, this is for private log
+ * entry stuff, the MARK data provided by userspace for example.
+ *
+ * Copied from kernel file drivers/md/dm-log-writes.c
+ */
+struct log_write_entry {
+	__le64 sector;
+	__le64 nr_sectors;
+	__le64 flags;
+	__le64 data_len;
+};
+
+#define LOG_IGNORE_DISCARD (1 << 0)
+#define LOG_DISCARD_NOT_SUPP (1 << 1)
+
+struct log {
+	int logfd;
+	int replayfd;
+	unsigned long flags;
+	u64 sectorsize;
+	u64 nr_entries;
+	u64 cur_entry;
+	u64 max_zero_size;
+	off_t cur_pos;
+};
+
+struct log *log_open(char *logfile, char *replayfile);
+int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
+			  int read_data);
+int log_seek_entry(struct log *log, u64 entry_num);
+int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
+			int read_data);
+void log_free(struct log *log);
+
+#endif
diff --git a/src/log-writes/replay-log.c b/src/log-writes/replay-log.c
new file mode 100644
index 0000000..c3de9c4
--- /dev/null
+++ b/src/log-writes/replay-log.c
@@ -0,0 +1,357 @@
+#include <stdio.h>
+#include <unistd.h>
+#include <getopt.h>
+#include <stdlib.h>
+#include <string.h>
+#include "log-writes.h"
+
+enum option_indexes {
+	NEXT_FLUSH,
+	NEXT_FUA,
+	START_ENTRY,
+	END_MARK,
+	LOG,
+	REPLAY,
+	LIMIT,
+	VERBOSE,
+	FIND,
+	NUM_ENTRIES,
+	NO_DISCARD,
+	FSCK,
+	CHECK,
+	START_MARK,
+};
+
+static struct option long_options[] = {
+	{"next-flush", no_argument, NULL, 0},
+	{"next-fua", no_argument, NULL, 0},
+	{"start-entry", required_argument, NULL, 0},
+	{"end-mark", required_argument, NULL, 0},
+	{"log", required_argument, NULL, 0},
+	{"replay", required_argument, NULL, 0},
+	{"limit", required_argument, NULL, 0},
+	{"verbose", no_argument, NULL, 'v'},
+	{"find", no_argument, NULL, 0},
+	{"num-entries", no_argument, NULL, 0},
+	{"no-discard", no_argument, NULL, 0},
+	{"fsck", required_argument, NULL, 0},
+	{"check", required_argument, NULL, 0},
+	{"start-mark", required_argument, NULL, 0},
+	{ NULL, 0, NULL, 0 },
+};
+
+static void usage(void)
+{
+	fprintf(stderr, "Usage: replay-log --log <logfile> [options]\n");
+	fprintf(stderr, "\t--replay <device> - replay onto a specific "
+		"device\n");
+	fprintf(stderr, "\t--limit <number> - number of entries to replay\n");
+	fprintf(stderr, "\t--next-flush - replay to/find the next flush\n");
+	fprintf(stderr, "\t--next-fua - replay to/find the next fua\n");
+	fprintf(stderr, "\t--start-entry <entry> - start at the given "
+		"entry #\n");
+	fprintf(stderr, "\t--start-mark <mark> - mark to start from\n");
+	fprintf(stderr, "\t--end-mark <mark> - replay to/find the given mark\n");
+	fprintf(stderr, "\t--find - put replay-log in find mode, will search "
+		"based on the other options\n");
+	fprintf(stderr, "\t--number-entries - print the number of entries in "
+		"the log\n");
+	fprintf(stderr, "\t--no-discard - don't process discard entries\n");
+	fprintf(stderr, "\t--fsck - the fsck command to run, must specify "
+		"--check\n");
+	fprintf(stderr, "\t--check [<number>|flush|fua] when to check the "
+		"file system, mush specify --fsck\n");
+	exit(1);
+}
+
+/*
+ * Check if the log entry flag matches one of the stop_flags.
+ * If stop_flag has LOG_MARK, then looking also for match of
+ * the mark label.
+ */
+static int should_stop(struct log_write_entry *entry, u64 stop_flags,
+		       char *mark)
+{
+	u64 flags = le64_to_cpu(entry->flags);
+	int check_mark = (stop_flags & LOG_MARK_FLAG);
+	/* mark data begins after entry header */
+	char *buf = (char *)(entry + 1);
+	/* entry buffer is padded with at least 1 zero after data_len */
+	u64 buflen = le64_to_cpu(entry->data_len) + 1;
+
+	if (flags & stop_flags) {
+		if (!check_mark)
+			return 1;
+		if ((flags & LOG_MARK_FLAG) &&
+		    !strncmp(mark, buf, buflen))
+			return 1;
+	}
+	return 0;
+}
+
+static int run_fsck(struct log *log, char *fsck_command)
+{
+	int ret = fsync(log->replayfd);
+	if (ret)
+		return ret;
+	ret = system(fsck_command);
+	if (ret >= 0)
+		ret = WEXITSTATUS(ret);
+	return ret ? -1 : 0;
+}
+
+enum log_replay_check_mode {
+	CHECK_NUMBER = 1,
+	CHECK_FUA = 2,
+	CHECK_FLUSH = 3,
+};
+
+static int seek_to_mark(struct log *log, struct log_write_entry *entry,
+			char *mark)
+{
+	int ret;
+
+	while ((ret = log_seek_next_entry(log, entry, 1)) == 0) {
+		if (should_stop(entry, LOG_MARK_FLAG, mark))
+			break;
+	}
+	if (ret == 1) {
+		fprintf(stderr, "Couldn't find starting mark\n");
+		ret = -1;
+	}
+
+	return ret;
+}
+
+int main(int argc, char **argv)
+{
+	char *logfile = NULL, *replayfile = NULL, *fsck_command = NULL;
+	struct log_write_entry *entry;
+	u64 stop_flags = 0;
+	u64 start_entry = 0;
+	u64 run_limit = 0;
+	u64 num_entries = 0;
+	u64 check_number = 0;
+	char *end_mark = NULL, *start_mark = NULL;
+	char *tmp = NULL;
+	struct log *log;
+	int find_mode = 0;
+	int c;
+	int opt_index;
+	int ret;
+	int print_num_entries = 0;
+	int discard = 1;
+	enum log_replay_check_mode check_mode = 0;
+
+	while ((c = getopt_long(argc, argv, "v", long_options,
+				&opt_index)) >= 0) {
+		switch(c) {
+		case 'v':
+			log_writes_verbose++;
+			continue;
+		default:
+			break;
+		}
+
+		switch(opt_index) {
+		case NEXT_FLUSH:
+			stop_flags |= LOG_FLUSH_FLAG;
+			break;
+		case NEXT_FUA:
+			stop_flags |= LOG_FUA_FLAG;
+			break;
+		case START_ENTRY:
+			start_entry = strtoull(optarg, &tmp, 0);
+			if (tmp && *tmp != '\0') {
+				fprintf(stderr, "Invalid entry number\n");
+				exit(1);
+			}
+			tmp = NULL;
+			break;
+		case START_MARK:
+			/*
+			 * Biggest sectorsize is 4k atm, so limit the mark to 4k
+			 * minus the size of the entry.  Say 4097 since we want
+			 * an extra slot for \0.
+			 */
+			start_mark = strndup(optarg, 4097 -
+					     sizeof(struct log_write_entry));
+			if (!start_mark) {
+				fprintf(stderr, "Couldn't allocate memory\n");
+				exit(1);
+			}
+			break;
+		case END_MARK:
+			/*
+			 * Biggest sectorsize is 4k atm, so limit the mark to 4k
+			 * minus the size of the entry.  Say 4097 since we want
+			 * an extra slot for \0.
+			 */
+			end_mark = strndup(optarg, 4097 -
+					   sizeof(struct log_write_entry));
+			if (!end_mark) {
+				fprintf(stderr, "Couldn't allocate memory\n");
+				exit(1);
+			}
+			stop_flags |= LOG_MARK_FLAG;
+			break;
+		case LOG:
+			logfile = strdup(optarg);
+			if (!logfile) {
+				fprintf(stderr, "Couldn't allocate memory\n");
+				exit(1);
+			}
+			break;
+		case REPLAY:
+			replayfile = strdup(optarg);
+			if (!replayfile) {
+				fprintf(stderr, "Couldn't allocate memory\n");
+				exit(1);
+			}
+			break;
+		case LIMIT:
+			run_limit = strtoull(optarg, &tmp, 0);
+			if (tmp && *tmp != '\0') {
+				fprintf(stderr, "Invalid entry number\n");
+				exit(1);
+			}
+			tmp = NULL;
+			break;
+		case FIND:
+			find_mode = 1;
+			break;
+		case NUM_ENTRIES:
+			print_num_entries = 1;
+			break;
+		case NO_DISCARD:
+			discard = 0;
+			break;
+		case FSCK:
+			fsck_command = strdup(optarg);
+			if (!fsck_command) {
+				fprintf(stderr, "Couldn't allocate memory\n");
+				exit(1);
+			}
+			break;
+		case CHECK:
+			if (!strcmp(optarg, "flush")) {
+				check_mode = CHECK_FLUSH;
+			} else if (!strcmp(optarg, "fua")) {
+				check_mode = CHECK_FUA;
+			} else {
+				check_mode = CHECK_NUMBER;
+				check_number = strtoull(optarg, &tmp, 0);
+				if (!check_number || (tmp && *tmp != '\0')) {
+					fprintf(stderr,
+						"Invalid entry number\n");
+					exit(1);
+				}
+				tmp = NULL;
+			}
+			break;
+		default:
+			usage();
+		}
+	}
+
+	if (!logfile)
+		usage();
+
+	log = log_open(logfile, replayfile);
+	if (!log)
+		exit(1);
+	free(logfile);
+	free(replayfile);
+
+	if (!discard)
+		log->flags |= LOG_IGNORE_DISCARD;
+
+	entry = malloc(log->sectorsize);
+	if (!entry) {
+		fprintf(stderr, "Couldn't allocate buffer\n");
+		log_free(log);
+		exit(1);
+	}
+
+	if (start_mark) {
+		ret = seek_to_mark(log, entry, start_mark);
+		if (ret)
+			exit(1);
+		free(start_mark);
+	} else {
+		ret = log_seek_entry(log, start_entry);
+		if (ret)
+			exit(1);
+	}
+
+	if ((fsck_command && !check_mode) || (!fsck_command && check_mode))
+		usage();
+
+	/* We just want to find a given entry */
+	if (find_mode) {
+		while ((ret = log_seek_next_entry(log, entry, 1)) == 0) {
+			num_entries++;
+			if ((run_limit && num_entries == run_limit) ||
+			    should_stop(entry, stop_flags, end_mark)) {
+				printf("%llu\n",
+				       (unsigned long long)log->cur_entry - 1);
+				log_free(log);
+				return 0;
+			}
+		}
+		log_free(log);
+		if (ret < 0)
+			return ret;
+		fprintf(stderr, "Couldn't find entry\n");
+		return 1;
+	}
+
+	/* Used for scripts, just print the number of entries in the log */
+	if (print_num_entries) {
+		printf("%llu\n", (unsigned long long)log->nr_entries);
+		log_free(log);
+		return 0;
+	}
+
+	/* No replay, just spit out the log info. */
+	if (!replayfile) {
+		printf("Log version=%d, sectorsize=%lu, entries=%llu\n",
+		       WRITE_LOG_VERSION, (unsigned long)log->sectorsize,
+		       (unsigned long long)log->nr_entries);
+		log_free(log);
+		return 0;
+	}
+
+	while ((ret = log_replay_next_entry(log, entry, 1)) == 0) {
+		num_entries++;
+		if (fsck_command) {
+			if ((check_mode == CHECK_NUMBER) &&
+			    !(num_entries % check_number))
+				ret = run_fsck(log, fsck_command);
+			else if ((check_mode == CHECK_FUA) &&
+				 should_stop(entry, LOG_FUA_FLAG, NULL))
+				ret = run_fsck(log, fsck_command);
+			else if ((check_mode == CHECK_FLUSH) &&
+				 should_stop(entry, LOG_FLUSH_FLAG, NULL))
+				ret = run_fsck(log, fsck_command);
+			else
+				ret = 0;
+			if (ret) {
+				fprintf(stderr, "Fsck errored out on entry "
+					"%llu\n",
+					(unsigned long long)log->cur_entry - 1);
+				break;
+			}
+		}
+
+		if ((run_limit && num_entries == run_limit) ||
+		    should_stop(entry, stop_flags, end_mark))
+			break;
+	}
+	fsync(log->replayfd);
+	log_free(log);
+	free(end_mark);
+	if (ret < 0)
+		exit(1);
+	return 0;
+}
-- 
2.7.4

[PATCH v4 2/8] replay-log: add validations for corrupt log entries

[Amir Goldstein]

Check for all zeros entry and for non zero padded entry
and report log offset of corrupted log entry.

Also report log offsets with -v and -vv debug prints.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 src/log-writes/log-writes.c | 84 ++++++++++++++++++++++++++++++++++-----------
 src/log-writes/log-writes.h |  2 ++
 src/log-writes/replay-log.c |  7 ++--
 3 files changed, 70 insertions(+), 23 deletions(-)

diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
index a215fef..dbfeef7 100644
--- a/src/log-writes/log-writes.c
+++ b/src/log-writes/log-writes.c
@@ -118,6 +118,26 @@ int log_discard(struct log *log, struct log_write_entry *entry)
 }
 
 /*
+ * @entry: entry to be replayed.
+ *
+ * @return: 1 if the entry is sane, 0 if it is invalid.
+ *
+ * Check if this is a sane log entry.
+ */
+int log_entry_valid(struct log_write_entry *entry)
+{
+	u64 flags = le64_to_cpu(entry->flags);
+
+	/* Suspect all zeroes entry */
+	if (!flags && !entry->nr_sectors)
+		return 0;
+	/* Suspect non zero padded entry */
+	if (flags != LOG_MARK_FLAG && entry->data[0] != 0)
+		return 0;
+	return 1;
+}
+
+/*
  * @log: the log we are replaying.
  * @entry: where we put the entry.
  * @read_data: read the entry data as well, entry must be log->sectorsize sized
@@ -146,24 +166,32 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
 		fprintf(stderr, "Error reading entry: %d\n", errno);
 		return -1;
 	}
+	if (!log_entry_valid(entry)) {
+		fprintf(stderr, "Malformed entry @%llu\n",
+				log->cur_pos / log->sectorsize);
+		return -1;
+	}
 	log->cur_entry++;
 
 	size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
 	if (read_size < log->sectorsize) {
-		if (lseek(log->logfd,
-			  log->sectorsize - sizeof(struct log_write_entry),
-			  SEEK_CUR) == (off_t)-1) {
+		log->cur_pos = lseek(log->logfd,
+			log->sectorsize - sizeof(struct log_write_entry), SEEK_CUR);
+		if (log->cur_pos == (off_t)-1) {
 			fprintf(stderr, "Error seeking in log: %d\n", errno);
 			return -1;
 		}
+	} else {
+		log->cur_pos += read_size;
 	}
 
-	if (log_writes_verbose)
-		printf("replaying %d: sector %llu, size %llu, flags %llu\n",
-		       (int)log->cur_entry - 1,
+	if (log_writes_verbose) {
+		printf("replaying %d@%llu: sector %llu, size %llu, flags %llu\n",
+		       (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
 		       (unsigned long long)le64_to_cpu(entry->sector),
 		       (unsigned long long)size,
 		       (unsigned long long)le64_to_cpu(entry->flags));
+	}
 	if (!size)
 		return 0;
 
@@ -183,6 +211,7 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
 		free(buf);
 		return -1;
 	}
+	log->cur_pos += size;
 
 	offset = le64_to_cpu(entry->sector) * log->sectorsize;
 	ret = pwrite(log->replayfd, buf, size, offset);
@@ -212,7 +241,8 @@ int log_seek_entry(struct log *log, u64 entry_num)
 	}
 
 	/* Skip the first sector containing the log super block */
-	if (lseek(log->logfd, log->sectorsize, SEEK_SET) == (off_t)-1) {
+	log->cur_pos = lseek(log->logfd, log->sectorsize, SEEK_SET);
+	if (log->cur_pos == (off_t)-1) {
 		fprintf(stderr, "Error seeking in file: %d\n", errno);
 		return -1;
 	}
@@ -229,9 +259,14 @@ int log_seek_entry(struct log *log, u64 entry_num)
 			fprintf(stderr, "Error reading entry: %d\n", errno);
 			return -1;
 		}
+		if (!log_entry_valid(&entry)) {
+			fprintf(stderr, "Malformed entry @%llu\n",
+					log->cur_pos / log->sectorsize);
+			return -1;
+		}
 		if (log_writes_verbose > 1)
-			printf("seek entry %d: %llu, size %llu, flags %llu\n",
-			       (int)i,
+			printf("seek entry %d@%llu: %llu, size %llu, flags %llu\n",
+			       (int)i, log->cur_pos / log->sectorsize,
 			       (unsigned long long)le64_to_cpu(entry.sector),
 			       (unsigned long long)le64_to_cpu(entry.nr_sectors),
 			       (unsigned long long)le64_to_cpu(entry.flags));
@@ -240,7 +275,8 @@ int log_seek_entry(struct log *log, u64 entry_num)
 		if (!(flags & LOG_DISCARD_FLAG))
 			seek_size += le64_to_cpu(entry.nr_sectors) *
 				log->sectorsize;
-		if (lseek(log->logfd, seek_size, SEEK_CUR) == (off_t)-1) {
+		log->cur_pos = lseek(log->logfd, seek_size, SEEK_CUR);
+		if (log->cur_pos == (off_t)-1) {
 			fprintf(stderr, "Error seeking in file: %d\n", errno);
 			return -1;
 		}
@@ -277,29 +313,37 @@ int log_seek_next_entry(struct log *log, struct log_write_entry *entry,
 		fprintf(stderr, "Error reading entry: %d\n", errno);
 		return -1;
 	}
+	if (!log_entry_valid(entry)) {
+		fprintf(stderr, "Malformed entry @%llu\n",
+				log->cur_pos / log->sectorsize);
+		return -1;
+	}
 	log->cur_entry++;
 
 	if (read_size < log->sectorsize) {
-		if (lseek(log->logfd,
-			  log->sectorsize - sizeof(struct log_write_entry),
-			  SEEK_CUR) == (off_t)-1) {
+		log->cur_pos = lseek(log->logfd,
+			log->sectorsize - sizeof(struct log_write_entry), SEEK_CUR);
+		if (log->cur_pos == (off_t)-1) {
 			fprintf(stderr, "Error seeking in log: %d\n", errno);
 			return -1;
 		}
+	} else {
+		log->cur_pos += read_size;
 	}
 	if (log_writes_verbose > 1)
-		printf("seek entry %d: %llu, size %llu, flags %llu\n",
-		       (int)log->cur_entry - 1,
+		printf("seek entry %d@%llu: %llu, size %llu, flags %llu\n",
+		       (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
 		       (unsigned long long)le64_to_cpu(entry->sector),
 		       (unsigned long long)le64_to_cpu(entry->nr_sectors),
 		       (unsigned long long)le64_to_cpu(entry->flags));
 
-	flags = le32_to_cpu(entry->flags);
-	read_size = le32_to_cpu(entry->nr_sectors) * log->sectorsize;
+	flags = le64_to_cpu(entry->flags);
+	read_size = le64_to_cpu(entry->nr_sectors) * log->sectorsize;
 	if (!read_size || (flags & LOG_DISCARD_FLAG))
 		return 0;
 
-	if (lseek(log->logfd, read_size, SEEK_CUR) == (off_t)-1) {
+	log->cur_pos = lseek(log->logfd, read_size, SEEK_CUR);
+	if (log->cur_pos == (off_t)-1) {
 		fprintf(stderr, "Error seeking in log: %d\n", errno);
 		return -1;
 	}
@@ -369,8 +413,8 @@ struct log *log_open(char *logfile, char *replayfile)
 	log->nr_entries = le64_to_cpu(super.nr_entries);
 	log->max_zero_size = 128 * 1024 * 1024;
 
-	if (lseek(log->logfd, log->sectorsize - sizeof(super), SEEK_CUR) ==
-	    (off_t) -1) {
+	log->cur_pos = lseek(log->logfd, log->sectorsize - sizeof(super), SEEK_CUR);
+	if (log->cur_pos == (off_t) -1) {
 		fprintf(stderr, "Error seeking to first entry: %d\n", errno);
 		log_free(log);
 		return NULL;
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
index 6cadb66..c89b119 100644
--- a/src/log-writes/log-writes.h
+++ b/src/log-writes/log-writes.h
@@ -50,6 +50,8 @@ struct log_write_entry {
 	__le64 nr_sectors;
 	__le64 flags;
 	__le64 data_len;
+	/* Read extra byte when seeking to verify that header is zero padded */
+	char data[1];
 };
 
 #define LOG_IGNORE_DISCARD (1 << 0)
diff --git a/src/log-writes/replay-log.c b/src/log-writes/replay-log.c
index c3de9c4..cf67931 100644
--- a/src/log-writes/replay-log.c
+++ b/src/log-writes/replay-log.c
@@ -75,7 +75,7 @@ static int should_stop(struct log_write_entry *entry, u64 stop_flags,
 	u64 flags = le64_to_cpu(entry->flags);
 	int check_mark = (stop_flags & LOG_MARK_FLAG);
 	/* mark data begins after entry header */
-	char *buf = (char *)(entry + 1);
+	char *buf = entry->data;
 	/* entry buffer is padded with at least 1 zero after data_len */
 	u64 buflen = le64_to_cpu(entry->data_len) + 1;
 
@@ -293,8 +293,9 @@ int main(int argc, char **argv)
 			num_entries++;
 			if ((run_limit && num_entries == run_limit) ||
 			    should_stop(entry, stop_flags, end_mark)) {
-				printf("%llu\n",
-				       (unsigned long long)log->cur_entry - 1);
+				printf("%llu@%llu\n",
+				       (unsigned long long)log->cur_entry - 1,
+				       log->cur_pos / log->sectorsize);
 				log_free(log);
 				return 0;
 			}
-- 
2.7.4

[PATCH v4 3/8] replay-log: add support for replaying ops in target device sector range

[Amir Goldstein]

Using command line options --start-sector and --end-sector, only
operations acting on the specified target device range will be
replayed.

Single vebbose mode (-v) prints out only replayed operations.
Double verbose mode (-vv) prints out also skipped operations.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 src/log-writes/log-writes.c | 37 +++++++++++++++++++++++++++++++++++--
 src/log-writes/log-writes.h |  2 ++
 src/log-writes/replay-log.c | 31 +++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+), 2 deletions(-)

diff --git a/src/log-writes/log-writes.c b/src/log-writes/log-writes.c
index dbfeef7..fab447a 100644
--- a/src/log-writes/log-writes.c
+++ b/src/log-writes/log-writes.c
@@ -118,6 +118,27 @@ int log_discard(struct log *log, struct log_write_entry *entry)
 }
 
 /*
+ * @log: the log we are replaying.
+ * @entry: entry to be replayed.
+ *
+ * @return: 0 if we should replay the entry, > 0 if we should skip it.
+ *
+ * Should we skip the entry in our log or replay onto the replay device.
+ */
+int log_should_skip(struct log *log, struct log_write_entry *entry)
+{
+	u64 sector = le64_to_cpu(entry->sector);
+	u64 nr_sectors = le64_to_cpu(entry->nr_sectors);
+
+	if (!nr_sectors)
+		return 0;
+	if (sector + nr_sectors <= log->start_sector ||
+	    sector > log->end_sector)
+		return 1;
+	return 0;
+}
+
+/*
  * @entry: entry to be replayed.
  *
  * @return: 1 if the entry is sane, 0 if it is invalid.
@@ -157,6 +178,7 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
 	char *buf;
 	ssize_t ret;
 	off_t offset;
+	int skip = 0;
 
 	if (log->cur_entry >= log->nr_entries)
 		return 1;
@@ -185,8 +207,10 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
 		log->cur_pos += read_size;
 	}
 
-	if (log_writes_verbose) {
-		printf("replaying %d@%llu: sector %llu, size %llu, flags %llu\n",
+	skip = log_should_skip(log, entry);
+	if (log_writes_verbose > 1 || (log_writes_verbose && !skip)) {
+		printf("%s %d@%llu: sector %llu, size %llu, flags %llu\n",
+		       skip ? "skipping" : "replaying",
 		       (int)log->cur_entry - 1, log->cur_pos / log->sectorsize,
 		       (unsigned long long)le64_to_cpu(entry->sector),
 		       (unsigned long long)size,
@@ -199,6 +223,15 @@ int log_replay_next_entry(struct log *log, struct log_write_entry *entry,
 	if (flags & LOG_DISCARD_FLAG)
 		return log_discard(log, entry);
 
+	if (skip) {
+		log->cur_pos = lseek(log->logfd, size, SEEK_CUR);
+		if (log->cur_pos == (off_t)-1) {
+			fprintf(stderr, "Error seeking in log: %d\n", errno);
+			return -1;
+		}
+		return 0;
+	}
+
 	buf = malloc(size);
 	if (!buf) {
 		fprintf(stderr, "Error allocating buffer %llu entry %llu\n", (unsigned long long)size, (unsigned long long)log->cur_entry - 1);
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
index c89b119..14242ee 100644
--- a/src/log-writes/log-writes.h
+++ b/src/log-writes/log-writes.h
@@ -62,6 +62,8 @@ struct log {
 	int replayfd;
 	unsigned long flags;
 	u64 sectorsize;
+	u64 start_sector;
+	u64 end_sector;
 	u64 nr_entries;
 	u64 cur_entry;
 	u64 max_zero_size;
diff --git a/src/log-writes/replay-log.c b/src/log-writes/replay-log.c
index cf67931..8457937 100644
--- a/src/log-writes/replay-log.c
+++ b/src/log-writes/replay-log.c
@@ -20,6 +20,8 @@ enum option_indexes {
 	FSCK,
 	CHECK,
 	START_MARK,
+	START_SECTOR,
+	END_SECTOR,
 };
 
 static struct option long_options[] = {
@@ -37,6 +39,8 @@ static struct option long_options[] = {
 	{"fsck", required_argument, NULL, 0},
 	{"check", required_argument, NULL, 0},
 	{"start-mark", required_argument, NULL, 0},
+	{"start-sector", required_argument, NULL, 0},
+	{"end-sector", required_argument, NULL, 0},
 	{ NULL, 0, NULL, 0 },
 };
 
@@ -61,6 +65,12 @@ static void usage(void)
 		"--check\n");
 	fprintf(stderr, "\t--check [<number>|flush|fua] when to check the "
 		"file system, mush specify --fsck\n");
+	fprintf(stderr, "\t--start-sector <sector> - replay ops on region "
+		"from <sector> onto <device>\n");
+	fprintf(stderr, "\t--end-sector <sector> - replay ops on region "
+		"to <sector> onto <device>\n");
+	fprintf(stderr, "\t-v or --verbose - print replayed ops\n");
+	fprintf(stderr, "\t-vv - print also skipped ops\n");
 	exit(1);
 }
 
@@ -129,6 +139,8 @@ int main(int argc, char **argv)
 	struct log_write_entry *entry;
 	u64 stop_flags = 0;
 	u64 start_entry = 0;
+	u64 start_sector = 0;
+	u64 end_sector = -1ULL;
 	u64 run_limit = 0;
 	u64 num_entries = 0;
 	u64 check_number = 0;
@@ -249,6 +261,22 @@ int main(int argc, char **argv)
 				tmp = NULL;
 			}
 			break;
+		case START_SECTOR:
+			start_sector = strtoull(optarg, &tmp, 0);
+			if (tmp && *tmp != '\0') {
+				fprintf(stderr, "Invalid sector number\n");
+				exit(1);
+			}
+			tmp = NULL;
+			break;
+		case END_SECTOR:
+			end_sector = strtoull(optarg, &tmp, 0);
+			if (tmp && *tmp != '\0') {
+				fprintf(stderr, "Invalid sector number\n");
+				exit(1);
+			}
+			tmp = NULL;
+			break;
 		default:
 			usage();
 		}
@@ -266,6 +294,9 @@ int main(int argc, char **argv)
 	if (!discard)
 		log->flags |= LOG_IGNORE_DISCARD;
 
+	log->start_sector = start_sector;
+	log->end_sector = end_sector;
+
 	entry = malloc(log->sectorsize);
 	if (!entry) {
 		fprintf(stderr, "Couldn't allocate buffer\n");
-- 
2.7.4

[PATCH v4 4/8] fstests: add support for working with dm-log-writes target

[Amir Goldstein]

Cherry-picked the relevant common bits from commit 70d41e17164b
in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
Quoting from Josef's commit message:

  This patch adds the supporting code for using the dm-log-writes
  target.  The dmlogwrites code is similar to the dmflakey code, it just
  gives us functions to build and tear down a dm-log-writes target.  We
  add a new LOGWRITES_DEV variable to take in the device we will use as
  the log and add checks for that.

[Amir:]
- Removed unneeded _test_falloc_support
- Moved _require_log_writes to dmlogwrites
- Document _require_log_writes
- Address review comments by Eryu Guan

Cc: Josef Bacik <jbacik@fb.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 README                       |  2 +
 common/dmlogwrites           | 90 ++++++++++++++++++++++++++++++++++++++++++++
 doc/requirement-checking.txt | 20 ++++++++++
 3 files changed, 112 insertions(+)
 create mode 100644 common/dmlogwrites

diff --git a/README b/README
index 9456fa7..4963d28 100644
--- a/README
+++ b/README
@@ -91,6 +91,8 @@ Preparing system for tests:
              - set TEST_XFS_SCRUB=1 to have _check_xfs_filesystem run
                xfs_scrub -vd to scrub the filesystem metadata online before
                unmounting to run the offline check.
+             - setenv LOGWRITES_DEV to a block device to use for power fail
+               testing.
 
         - or add a case to the switch in common/config assigning
           these variables based on the hostname of your test
diff --git a/common/dmlogwrites b/common/dmlogwrites
new file mode 100644
index 0000000..247c744
--- /dev/null
+++ b/common/dmlogwrites
@@ -0,0 +1,90 @@
+##/bin/bash
+#
+# Copyright (c) 2015 Facebook, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#
+#
+# common functions for setting up and tearing down a dm log-writes device
+
+_require_log_writes()
+{
+	[ -z "$LOGWRITES_DEV" -o ! -b "$LOGWRITES_DEV" ] && \
+		_notrun "This test requires a valid \$LOGWRITES_DEV"
+
+	_exclude_scratch_mount_option dax
+	_require_dm_target log-writes
+	_require_test_program "log-writes/replay-log"
+}
+
+_log_writes_init()
+{
+	local BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+	LOGWRITES_NAME=logwrites-test
+	LOGWRITES_DMDEV=/dev/mapper/$LOGWRITES_NAME
+	LOGWRITES_TABLE="0 $BLK_DEV_SIZE log-writes $SCRATCH_DEV $LOGWRITES_DEV"
+	$DMSETUP_PROG create $LOGWRITES_NAME --table "$LOGWRITES_TABLE" || \
+		_fail "failed to create log-writes device"
+	$DMSETUP_PROG mknodes > /dev/null 2>&1
+}
+
+_log_writes_mark()
+{
+	[ $# -ne 1 ] && _fail "_log_writes_mark takes one argument"
+	$DMSETUP_PROG message $LOGWRITES_NAME 0 mark $1
+}
+
+_log_writes_mkfs()
+{
+	_scratch_options mkfs
+	_mkfs_dev $SCRATCH_OPTIONS $LOGWRITES_DMDEV
+	_log_writes_mark mkfs
+}
+
+_log_writes_mount()
+{
+	_scratch_options mount
+	$MOUNT_PROG -t $FSTYP `_common_dev_mount_options $*` $SCRATCH_OPTIONS \
+		$LOGWRITES_DMDEV $SCRATCH_MNT
+}
+
+_log_writes_unmount()
+{
+	$UMOUNT_PROG $SCRATCH_MNT
+}
+
+# _log_writes_replay_log <mark>
+#
+# This replays the log contained on $LOGWRITES_DEV onto $SCRATCH_DEV upto the
+# mark passed in.
+_log_writes_replay_log()
+{
+	_mark=$1
+
+	$here/src/log-writes/replay-log --log $LOGWRITES_DEV --replay $SCRATCH_DEV \
+		--end-mark $_mark >> $seqres.full 2>&1
+	[ $? -ne 0 ] && _fail "replay failed"
+}
+
+_log_writes_remove()
+{
+	$DMSETUP_PROG remove $LOGWRITES_NAME > /dev/null 2>&1
+	$DMSETUP_PROG mknodes > /dev/null 2>&1
+}
+
+_log_writes_cleanup()
+{
+	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_log_writes_remove
+}
diff --git a/doc/requirement-checking.txt b/doc/requirement-checking.txt
index 95d10e6..4e01b1f 100644
--- a/doc/requirement-checking.txt
+++ b/doc/requirement-checking.txt
@@ -21,6 +21,10 @@ they have.  This is done with _require_<xxx> macros, which may take parameters.
 
 	_require_statx
 
+ (4) Device mapper requirement.
+
+	_require_dm_target
+	_require_log_writes
 
 ====================
 GENERAL REQUIREMENTS
@@ -102,3 +106,19 @@ _require_statx
 
      The test requires the use of the statx() system call and will be skipped
      if it isn't available in the kernel.
+
+
+==========================
+DEVICE MAPPER REQUIREMENTS
+==========================
+
+_require_dm_target <name>
+
+     The test requires the use of the device mapper target and will be skipped
+     if it isn't available in the kernel.
+
+_require_log_writes
+
+     The test requires the use of the device mapper target log-writes.
+     The test also requires the test program log-writes/replay-log is built
+     and will be skipped if either isn't available.
-- 
2.7.4

[PATCH v4 5/8] fstests: crash consistency fsx test using dm-log-writes

[Amir Goldstein]

Cherry-picked the test from commit 70d41e17164b
in Josef Bacik's fstests tree (https://github.com/josefbacik/fstests).
Quoting from Josef's commit message:

  The test just runs some ops and exits, then finds all of the good buffers
  in the directory we provided and:
  - replays up to the mark given
  - mounts the file system and compares the md5sum
  - unmounts and fsck's to check for metadata integrity

  dm-log-writes will pretend to do discard and the replay-log tool will
  replay it properly depending on the underlying device, either by writing
  0's or actually calling the discard ioctl, so I've enabled discard in the
  test for maximum fun.

[Amir:]
- Removed unneeded _test_falloc_support dynamic FSX_OPTS
- Fold repetitions into for loops
- Added place holders for using constant random seeds
- Add pre umount checkpint
- Add test to new 'replay' group
- Address review comments by Eryu Guan

Cc: Josef Bacik <jbacik@fb.com>
Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/500     | 138 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/500.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 141 insertions(+)
 create mode 100755 tests/generic/500
 create mode 100644 tests/generic/500.out

diff --git a/tests/generic/500 b/tests/generic/500
new file mode 100755
index 0000000..fd4d6ec
--- /dev/null
+++ b/tests/generic/500
@@ -0,0 +1,138 @@
+#! /bin/bash
+# FS QA Test No. 500
+#
+# Run fsx with log writes to verify power fail safeness.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2015 Facebook. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_log_writes_cleanup
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmlogwrites
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch_nocheck
+_require_log_writes
+
+rm -f $seqres.full
+
+check_files()
+{
+	local name=$1
+
+	# Now look for our files
+	for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
+	do
+		local filename=$(basename $i)
+		local mark="${filename##*.}"
+		echo "checking $filename" >> $seqres.full
+		_log_writes_replay_log $filename
+		_scratch_mount
+		local expected_md5=$(_md5_checksum $i)
+		local md5=$(_md5_checksum $SCRATCH_MNT/$name)
+		[ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
+		_scratch_unmount
+		_check_scratch_fs
+	done
+}
+
+SANITY_DIR=$TEST_DIR/fsxtests
+rm -rf $SANITY_DIR
+mkdir $SANITY_DIR
+
+# Create the log
+_log_writes_init
+
+_log_writes_mkfs >> $seqres.full 2>&1
+
+# Log writes emulates discard support, turn it on for maximum crying.
+_log_writes_mount -o discard
+
+NUM_FILES=4
+NUM_OPS=200
+FSX_OPTS="-N $NUM_OPS -d -P $SANITY_DIR -i $LOGWRITES_DMDEV"
+# Set random seeds for fsx runs (0 for timestamp + pid)
+# When test failure is detected, check the seed values printed
+# by fsx processes to $seqres.full and set them in this array
+# to repeat the same fsx runs
+seeds=(0 0 0 0)
+# Run fsx for a while
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	run_check $here/ltp/fsx $FSX_OPTS -S ${seeds[$j]} -j $j $SCRATCH_MNT/testfile$j &
+done
+wait
+
+test_md5=()
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	test_md5[$j]=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+done
+
+# Unmount the scratch dir and tear down the log writes target
+_log_writes_mark last
+_log_writes_unmount
+_log_writes_mark end
+_log_writes_remove
+_check_scratch_fs
+
+# check pre umount
+echo "checking pre umount" >> $seqres.full
+_log_writes_replay_log last
+_scratch_mount
+_scratch_unmount
+_check_scratch_fs
+
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	check_files testfile$j
+done
+
+# Check the end
+echo "checking post umount" >> $seqres.full
+_log_writes_replay_log end
+_scratch_mount
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	md5=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+	[ "${md5}" != "${test_md5[$j]}" ] && _fail "testfile$j end md5sum mismatched"
+done
+_scratch_unmount
+_check_scratch_fs
+
+echo "Silence is golden"
+status=0
+exit
+
diff --git a/tests/generic/500.out b/tests/generic/500.out
new file mode 100644
index 0000000..883b2ca
--- /dev/null
+++ b/tests/generic/500.out
@@ -0,0 +1,2 @@
+QA output created by 500
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 1894de0..c857153 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -457,3 +457,4 @@
 452 auto quick
 453 auto quick dir
 454 auto quick attr
+500 auto log replay
-- 
2.7.4

[PATCH v4 6/8] fstests: regression test for ext4 crash consistency bug

[Amir Goldstein]

This test is motivated by this inconsistency found in ext4 during random
crash consistency tests:

 *** fsck.ext4 output ***
 fsck from util-linux 2.27.1
 e2fsck 1.42.13 (17-May-2015)
 Pass 1: Checking inodes, blocks, and sizes
 Inode 12, end of extent exceeds allowed value
         (logical block 33, physical block 33817, len 7)
 Clear? no

 Inode 12, i_blocks is 240, should be 184.  Fix? no

This test uses device mapper flakey target to demonstrate the bug
found using device mapper log-writes target.

Signed-off-by: Amir Goldstein <amir73il@gmail.com>

fix 501
---
 tests/generic/501     | 80 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/501.out |  2 ++
 tests/generic/group   |  1 +
 3 files changed, 83 insertions(+)
 create mode 100755 tests/generic/501
 create mode 100644 tests/generic/501.out

diff --git a/tests/generic/501 b/tests/generic/501
new file mode 100755
index 0000000..d0ac49d
--- /dev/null
+++ b/tests/generic/501
@@ -0,0 +1,80 @@
+#! /bin/bash
+# FS QA Test No. 501
+#
+# This test is motivated by a bug found in ext4 during random crash
+# consistency tests.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch
+_require_dm_target flakey
+
+rm -f $seqres.full
+
+_scratch_mkfs >> $seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+
+_init_flakey
+_mount_flakey
+
+fsxops=$tmp.fsxops
+cat <<EOF > $fsxops
+write 0x137dd 0xdc69 0x0
+fallocate 0xb531 0xb5ad 0x21446
+collapse_range 0x1c000 0x4000 0x21446
+write 0x3e5ec 0x1a14 0x21446
+zero_range 0x20fac 0x6d9c 0x40000 keep_size
+mapwrite 0x216ad 0x274f 0x40000
+EOF
+run_check $here/ltp/fsx -d --replay-ops $fsxops $SCRATCH_MNT/testfile
+
+_flakey_drop_and_remount
+_unmount_flakey
+_cleanup_flakey
+_check_scratch_fs
+
+echo "Silence is golden"
+
+status=0
+exit
diff --git a/tests/generic/501.out b/tests/generic/501.out
new file mode 100644
index 0000000..00133b6
--- /dev/null
+++ b/tests/generic/501.out
@@ -0,0 +1,2 @@
+QA output created by 501
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index c857153..1e1b524 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -458,3 +458,4 @@
 453 auto quick dir
 454 auto quick attr
 500 auto log replay
+501 auto quick metadata
-- 
2.7.4

[PATCH v4 7/8] fstests: crash consistency fsx test for cloned files

[Amir Goldstein]

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/502     | 141 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/502.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 144 insertions(+)
 create mode 100755 tests/generic/502
 create mode 100644 tests/generic/502.out

diff --git a/tests/generic/502 b/tests/generic/502
new file mode 100755
index 0000000..96908f5
--- /dev/null
+++ b/tests/generic/502
@@ -0,0 +1,141 @@
+#! /bin/bash
+# FS QA Test No. 502
+#
+# Run fsx with log writes on cloned files to verify power fail safeness.
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+status=1	# failure is the default!
+
+_cleanup()
+{
+	_log_writes_cleanup
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+. ./common/dmlogwrites
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_test
+_require_scratch_reflink
+_require_cp_reflink
+_require_log_writes
+
+rm -f $seqres.full
+
+check_files()
+{
+	local name=$1
+
+	# Now look for our files
+	for i in $(find $SANITY_DIR -type f | grep $name | grep mark)
+	do
+		local filename=$(basename $i)
+		local mark="${filename##*.}"
+		echo "checking $filename" >> $seqres.full
+		_log_writes_replay_log $filename
+		_scratch_mount
+		local expected_md5=$(_md5_checksum $i)
+		local md5=$(_md5_checksum $SCRATCH_MNT/$name)
+		[ "${md5}" != "${expected_md5}" ] && _fail "$filename md5sum mismatched"
+		_scratch_unmount
+		_check_scratch_fs
+	done
+}
+
+SANITY_DIR=$TEST_DIR/fsxtests
+rm -rf $SANITY_DIR
+mkdir $SANITY_DIR
+
+# Create the log
+_log_writes_init
+
+_log_writes_mkfs >> $seqres.full 2>&1
+
+# Log writes emulates discard support, turn it on for maximum crying.
+_log_writes_mount -o discard
+
+# write testfile index -1 to be cloned to testfile0
+$XFS_IO_PROG -f -c "pwrite -S 0xff 0 256k" -c "fsync" \
+	$SCRATCH_MNT/testfile-1 > /dev/null 2>&1
+
+NUM_FILES=10
+NUM_OPS=10
+FSX_OPTS="-N $NUM_OPS -d -k -P $SANITY_DIR -i $LOGWRITES_DMDEV"
+# Run fsx for a while
+# Set random seeds for fsx runs (0 for timestamp + pid)
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	# clone the clone from prev iteration which may have already mutated
+	_cp_reflink $SCRATCH_MNT/testfile$((j-1)) $SCRATCH_MNT/testfile$j
+	run_check $here/ltp/fsx $FSX_OPTS -S 0 -j $j $SCRATCH_MNT/testfile$j &
+done
+wait
+
+test_md5=()
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	test_md5[$j]=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+done
+
+# Unmount the scratch dir and tear down the log writes target
+_log_writes_mark last
+_log_writes_unmount
+_log_writes_mark end
+_log_writes_remove
+_check_scratch_fs
+
+# check pre umount
+echo "checking pre umount" >> $seqres.full
+_log_writes_replay_log last
+_scratch_mount
+_scratch_unmount
+_check_scratch_fs
+
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	check_files testfile$j
+done
+
+# Check the end
+echo "checking post umount" >> $seqres.full
+_log_writes_replay_log end
+_scratch_mount
+for j in `seq 0 $((NUM_FILES-1))`
+do
+	md5=$(_md5_checksum $SCRATCH_MNT/testfile$j)
+	[ "${md5}" != "${test_md5[$j]}" ] && _fail "testfile$j end md5sum mismatched"
+done
+
+echo "Silence is golden"
+status=0
+exit
+
diff --git a/tests/generic/502.out b/tests/generic/502.out
new file mode 100644
index 0000000..930f6d4
--- /dev/null
+++ b/tests/generic/502.out
@@ -0,0 +1,2 @@
+QA output created by 502
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 1e1b524..4324775 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -459,3 +459,4 @@
 454 auto quick attr
 500 auto log replay
 501 auto quick metadata
+502 auto log replay clone
-- 
2.7.4

[PATCH v4 8/8] fstests: regression test for xfs leftover CoW extent error

[Amir Goldstein]

The following error are reported after running this test:

*** xfs_check output ***
leftover CoW extent (0/2147483736) len 1
block 0/2147483736 out of range
blocks 0/2147483736..2147483736 claimed by block 0/6
leftover CoW extent (0/2147483738) len 2
blocks 0/2147483738..2147483739 out of range
blocks 0/2147483738..2147483739 claimed by block 0/6
leftover CoW extent (0/2147483741) len 3
blocks 0/2147483741..2147483743 out of range
blocks 0/2147483741..2147483743 claimed by block 0/6
block 0/88 type unknown not expected
block 0/90 type unknown not expected
block 0/91 type unknown not expected
block 0/93 type unknown not expected
block 0/94 type unknown not expected
block 0/95 type unknown not expected

*** xfs_repair -n output ***
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
leftover CoW extent (0/88) len 1
leftover CoW extent (0/90) len 2
leftover CoW extent (0/93) len 3
        - found root inode chunk

Signed-off-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/503     | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/503.out |  2 ++
 tests/generic/group   |  1 +
 3 files changed, 77 insertions(+)
 create mode 100755 tests/generic/503
 create mode 100644 tests/generic/503.out

diff --git a/tests/generic/503 b/tests/generic/503
new file mode 100755
index 0000000..5662eb6
--- /dev/null
+++ b/tests/generic/503
@@ -0,0 +1,74 @@
+#! /bin/bash
+# FS QA Test No. 503
+#
+# Regression test for xfs leftover CoW extents after truncate
+# and umount
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2017 CTERA Networks. All Rights Reserved.
+# Author: Amir Goldstein <amir73il@gmail.com>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch_reflink
+_require_cp_reflink
+_require_xfs_io_command "fzero"
+_require_xfs_io_command "fcollapse"
+_require_xfs_io_command "finsert"
+_require_xfs_io_command "truncate"
+
+rm -f $seqres.full
+
+_scratch_mkfs >/dev/null 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -c "pwrite 0 0x40000" \
+	$SCRATCH_MNT/foo > /dev/null 2>&1
+
+_cp_reflink $SCRATCH_MNT/foo $SCRATCH_MNT/bar
+
+$XFS_IO_PROG -f -c "fzero -k 0x169f 0x387c" \
+		-c "fcollapse 0x29000 0xd000" \
+		-c "finsert 0 0x8000" \
+		-c "truncate 0x8000" \
+	$SCRATCH_MNT/foo > /dev/null 2>&1
+
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/generic/503.out b/tests/generic/503.out
new file mode 100644
index 0000000..6c1400a
--- /dev/null
+++ b/tests/generic/503.out
@@ -0,0 +1,2 @@
+QA output created by 503
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index 4324775..7a9cd78 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -460,3 +460,4 @@
 500 auto log replay
 501 auto quick metadata
 502 auto log replay clone
+503 auto quick clone
-- 
2.7.4

Re: [PATCH v4 1/8] log-writes: add replay-log program to replay dm-log-writes target

[Eryu Guan]

On Thu, Sep 07, 2017 at 01:03:06PM +0300, Amir Goldstein wrote:
> Imported Josef Bacik's code from:
> https://github.com/josefbacik/log-writes.git
> 
> Specialized program for replaying a write log that was recorded by
> device mapper log-writes target.  The tools is used to perform
> crash consistency tests, allowing to run an arbitrary check tool
> (fsck) at specified checkpoints in the write log.
> 
> [Amir:]
> - Add project Makefile and SOURCE files
> - Document the replay-log auxiliary program
> - Address review comments by Eryu Guan
> 
> Cc: Josef Bacik <jbacik@fb.com>
> Signed-off-by: Amir Goldstein <amir73il@gmail.com>
> ---
...
> diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
> new file mode 100644
> index 0000000..6cadb66
> --- /dev/null
> +++ b/src/log-writes/log-writes.h
> @@ -0,0 +1,77 @@
> +#ifndef _LOG_WRITES_H_
> +#define _LOG_WRITES_H_
> +
> +#include <linux/types.h>
> +#include <linux/byteorder/little_endian.h>

This only works on little endian hosts, big endian hosts like ppc64 fail
the tests with "Magic doesn't match" error, because le64_to_cpu is an
no-op there.

I did the following changes and it worked for me. If this looks fine to
you, I can fold the changes into the original patch.

--- 8< ---
diff --git a/src/log-writes/log-writes.h b/src/log-writes/log-writes.h
index 14242ee13c6b..0fb324a57c4d 100644
--- a/src/log-writes/log-writes.h
+++ b/src/log-writes/log-writes.h
@@ -2,7 +2,12 @@
 #define _LOG_WRITES_H_

 #include <linux/types.h>
-#include <linux/byteorder/little_endian.h>
+#include <endian.h>
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+#include <linux/byteorder/little_endian.h>
+#else
+#include <linux/byteorder/big_endian.h>
+#endif

 extern int log_writes_verbose;

--- >8 ---
Thanks,
Eryu