[PATCH 0/4] ioengine: File based ioengines

[PATCH 0/4] ioengine: File based ioengines

[Dmitry Monakhov]

Fio is very cool tool for generating various io patterns.
And I use it as a base io-loader for xfsstress framework

I've prepared two file-based ioengines which allow to generate
interesting io patterns. 'falloc' is more generic and may be useful
for many cases, other 'e4defrag' is more specific but still
usfull. It would be nice to have this engines in-tree if possible.

TOC:
 # fixes for external engines
 export file_lookup_open
 ioengine: allow several external ioengines
 # file based ioengines
 ioengine: Add fallocate ioengine
 ioengine: Add e4defrag ioengine

[PATCH 1/4] export file_lookup_open

[Dmitry Monakhov]

generic_open_file is not always appropriate method, but other modules
may want to implement similar callback and use hash optimization.

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 file.h      |    1 +
 filesetup.c |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/file.h b/file.h
index 68f9a6e..42fd58c 100644
--- a/file.h
+++ b/file.h
@@ -153,6 +153,7 @@ extern int __must_check file_invalidate_cache(struct thread_data *, struct fio_f
 extern int __must_check generic_open_file(struct thread_data *, struct fio_file *);
 extern int __must_check generic_close_file(struct thread_data *, struct fio_file *);
 extern int __must_check generic_get_file_size(struct thread_data *, struct fio_file *);
+extern int __must_check file_lookup_open(struct fio_file *f, int flags);
 extern int __must_check pre_read_files(struct thread_data *);
 extern int add_file(struct thread_data *, const char *);
 extern int add_file_exclusive(struct thread_data *, const char *);
diff --git a/filesetup.c b/filesetup.c
index 64da8bb..06b7d7f 100644
--- a/filesetup.c
+++ b/filesetup.c
@@ -435,7 +435,7 @@ int generic_close_file(struct thread_data fio_unused *td, struct fio_file *f)
 	return ret;
 }
 
-static int file_lookup_open(struct fio_file *f, int flags)
+int file_lookup_open(struct fio_file *f, int flags)
 {
 	struct fio_file *__f;
 	int from_hash;
-- 
1.7.7.6

[PATCH 2/4] ioengine: allow several external ioengines

[Dmitry Monakhov]

Currently only one external ioengine can be exported because
it use hardcoded 'ioengine' symbol, but if we allow external
modules to have ops symbol similar to it's name then several
extrnal engines become possible.

NOTE: Old linking layout preserved

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 ioengines.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/ioengines.c b/ioengines.c
index b43374e..c2a64cb 100644
--- a/ioengines.c
+++ b/ioengines.c
@@ -104,7 +104,9 @@ static struct ioengine_ops *dlopen_ioengine(struct thread_data *td,
 	 * Unlike the included modules, external engines should have a
 	 * non-static ioengine structure that we can reference.
 	 */
-	ops = dlsym(dlhandle, "ioengine");
+	ops = dlsym(dlhandle, engine_lib);
+	if (!ops)
+		ops = dlsym(dlhandle, "ioengine");
 	if (!ops) {
 		td_vmsg(td, -1, dlerror(), "dlsym");
 		dlclose(dlhandle);
-- 
1.7.7.6

[PATCH 3/4] ioengine: Add fallocate ioengine

[Dmitry Monakhov]

IO engine that does regular fallocate to simulate data transfer
as fio ioengine.
 DDIR_READ  does fallocate(,mode=FALLOC_FL_KEEP_SIZE,)
 DDIR_WRITE does fallocate(,mode=0) : fallocate with file extention
 DDIR_TRIM  does fallocate(FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)

This engine may be useful to perform various tests:
1) Generation highly fragmentated files
2) Various fs stress testing in parallel with others io activity

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 HOWTO            |    2 +
 Makefile         |    2 +-
 engines/falloc.c |  121 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 124 insertions(+), 1 deletions(-)
 create mode 100644 engines/falloc.c

diff --git a/HOWTO b/HOWTO
index 025443b..3eb5510 100644
--- a/HOWTO
+++ b/HOWTO
@@ -602,6 +602,8 @@ ioengine=str	Defines how the job issues io to the file. The following
 				channel semantics (Send/Recv) for the
 				InfiniBand, RoCE and iWARP protocols.
 
+			falloc  Perform fallocate/punch_hole to files
+	
 			external Prefix to specify loading an external
 				IO engine object file. Append the engine
 				filename, eg ioengine=external:/tmp/foo.o
diff --git a/Makefile b/Makefile
index 83fbd59..a047079 100644
--- a/Makefile
+++ b/Makefile
@@ -21,7 +21,7 @@ ifeq ($(UNAME), Linux)
 		engines/libaio.c engines/posixaio.c engines/sg.c \
 		engines/splice.c engines/syslet-rw.c engines/guasi.c \
 		engines/binject.c engines/rdma.c profiles/tiobench.c \
-		engines/fusion-aw.c
+		engines/fusion-aw.c engines/falloc.c
   LIBS += -lpthread -ldl -lrt -laio
   LDFLAGS += -rdynamic
 endif
diff --git a/engines/falloc.c b/engines/falloc.c
new file mode 100644
index 0000000..cbb30cb
--- /dev/null
+++ b/engines/falloc.c
@@ -0,0 +1,121 @@
+/*
+ * falloc: ioengine for git://git.kernel.dk/fio.git
+ *
+ * IO engine that does regular fallocate to simulate data transfer 
+ * as fio ioengine.
+ * DDIR_READ  does fallocate(,mode = FALLOC_FL_KEEP_SIZE,)
+ * DDIR_WRITE does fallocate(,mode = 0) : fallocate with size extention 
+ * DDIR_TRIM  does fallocate(,mode = FALLOC_FL_KEEP_SIZE | FALLOC_FL_PUNCH_HOLE)
+ *
+ */
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/uio.h>
+#include <errno.h>
+#include <assert.h>
+#include <fcntl.h>
+
+#include "../fio.h"
+#include "../filehash.h"
+
+/*
+ * generic_open_file is not appropriate because does not allow to perform
+ * TRIM in to file
+ */
+int open_file(struct thread_data *td, struct fio_file *f)
+{
+	int from_hash = 0;
+
+	dprint(FD_FILE, "fd open %s\n", f->file_name);
+
+	if (f->filetype != FIO_TYPE_FILE) {
+		log_err("fio: only files are supported fallocate \n");
+		return 1;
+	}
+	if (!strcmp(f->file_name, "-")) {
+		log_err("fio: can't read/write to stdin/out\n");
+		return 1;
+	}
+
+open_again:
+	from_hash = file_lookup_open(f, O_CREAT|O_RDWR);
+
+	if (f->fd == -1) {
+		char buf[FIO_VERROR_SIZE];
+		int __e = errno;
+		snprintf(buf, sizeof(buf) - 1, "open(%s)", f->file_name);
+		td_verror(td, __e, buf);
+	}
+
+	if (!from_hash && f->fd != -1) {
+		if (add_file_hash(f)) {
+			int fio_unused ret;
+
+			/*
+			 * OK to ignore, we haven't done anything with it
+			 */
+			ret = generic_close_file(td, f);
+			goto open_again;
+		}
+	}
+
+	return 0;
+}
+
+#ifndef FALLOC_FL_KEEP_SIZE
+#define FALLOC_FL_KEEP_SIZE     0x01 /* default is extend size */
+#endif
+#ifndef FALLOC_FL_PUNCH_HOLE
+#define FALLOC_FL_PUNCH_HOLE    0x02 /* de-allocates range */
+#endif 
+static int fio_fallocate_queue(struct thread_data *td, struct io_u *io_u)
+{
+	struct fio_file *f = io_u->file;
+	int ret;
+	int flags = 0;
+
+	fio_ro_check(td, io_u);
+
+	if (io_u->ddir == DDIR_READ)
+		flags = FALLOC_FL_KEEP_SIZE;
+	else if (io_u->ddir == DDIR_WRITE)
+		flags = 0;
+	else if (io_u->ddir == DDIR_TRIM)
+		flags = FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE;
+
+	ret = fallocate(f->fd, flags, io_u->offset, io_u->xfer_buflen);
+
+	if (ret) {
+		io_u->error = errno;
+		if (io_u->error)
+			td_verror(td, io_u->error, "xfer");
+	}
+	if (io_u->ddir == DDIR_TRIM && !ret)
+		return io_u->xfer_buflen;
+
+	if (io_u->file && ret == 0 && ddir_rw(io_u->ddir))
+		io_u->file->file_pos = io_u->offset + ret;
+
+	return FIO_Q_COMPLETED;
+}
+
+static struct ioengine_ops ioengine = {
+	.name		= "falloc",
+	.version	= FIO_IOOPS_VERSION,
+	.queue		= fio_fallocate_queue,
+	.open_file	= open_file,
+	.close_file	= generic_close_file,
+	.get_file_size	= generic_get_file_size,
+	.flags		= FIO_SYNCIO
+};
+
+static void fio_init fio_syncio_register(void)
+{
+	register_ioengine(&ioengine);
+}
+
+static void fio_exit fio_syncio_unregister(void)
+{
+	unregister_ioengine(&ioengine);
+}
-- 
1.7.7.6

[PATCH 4/4] engine: add e4defrag engine

[Dmitry Monakhov]

IO engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate
defragment activity

Signed-off-by: Dmitry Monakhov <dmonakhov@openvz.org>
---
 Makefile           |    2 +-
 engines/e4defrag.c |  215 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 examples/e4defrag  |   32 ++++++++
 3 files changed, 248 insertions(+), 1 deletions(-)
 create mode 100644 engines/e4defrag.c
 create mode 100644 examples/e4defrag

diff --git a/Makefile b/Makefile
index a047079..7972508 100644
--- a/Makefile
+++ b/Makefile
@@ -21,7 +21,7 @@ ifeq ($(UNAME), Linux)
 		engines/libaio.c engines/posixaio.c engines/sg.c \
 		engines/splice.c engines/syslet-rw.c engines/guasi.c \
 		engines/binject.c engines/rdma.c profiles/tiobench.c \
-		engines/fusion-aw.c engines/falloc.c
+		engines/fusion-aw.c engines/falloc.c engines/e4defrag.c
   LIBS += -lpthread -ldl -lrt -laio
   LDFLAGS += -rdynamic
 endif
diff --git a/engines/e4defrag.c b/engines/e4defrag.c
new file mode 100644
index 0000000..5affaa0
--- /dev/null
+++ b/engines/e4defrag.c
@@ -0,0 +1,215 @@
+/*
+ * ioe_e4defrag:  ioengine for git://git.kernel.dk/fio.git
+ *
+ * IO engine that does regular EXT4_IOC_MOVE_EXT ioctls to simulate
+ * defragment activity
+ *
+ */
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/uio.h>
+#include <errno.h>
+#include <assert.h>
+#include <fcntl.h>
+
+#include "../fio.h"
+
+#ifndef EXT4_IOC_MOVE_EXT
+#define EXT4_IOC_MOVE_EXT               _IOWR('f', 15, struct move_extent)
+struct move_extent {
+	__u32 reserved;         /* should be zero */
+	__u32 donor_fd;         /* donor file descriptor */
+	__u64 orig_start;       /* logical start offset in block for orig */
+	__u64 donor_start;      /* logical start offset in block for donor */
+	__u64 len;              /* block length to be moved */
+	__u64 moved_len;        /* moved block length */
+};
+#endif
+
+struct e4defrag_data {
+	int donor_fd;
+	int bsz;
+};
+
+struct e4defrag_options {
+	struct thread_data *td;
+	unsigned int inplace;
+	char * donor_name;
+};
+
+static struct fio_option options[] = {
+	{
+		.name	= "donorname",
+		.type	= FIO_OPT_STR_STORE,
+		.off1	= offsetof(struct e4defrag_options, donor_name),
+		.help	= "File used as a block donor",
+	},
+	{
+		.name	= "inplace",
+		.type	= FIO_OPT_INT,
+		.off1	= offsetof(struct e4defrag_options, inplace),
+		.minval	= 0,
+		.maxval	= 1,
+		.help	= "Alloc and free space inside defrag event",
+	},
+	{
+		.name	= NULL,
+	},
+};
+
+static int fio_e4defrag_init(struct thread_data *td)
+{
+	int r, len = 0;
+	struct e4defrag_options *o = td->eo;
+	struct e4defrag_data *ed;
+	struct stat stub;
+	char donor_name[PATH_MAX];
+
+	if (!strlen(o->donor_name)) {
+		log_err("'donorname' options required\n");
+		return 1;
+	}
+
+	ed = malloc(sizeof(*ed));
+	if (!ed) {
+		td_verror(td, -ENOMEM, "io_queue_init");
+		return 1;
+	}
+	memset(ed, 0 ,sizeof(*ed));
+
+	if (td->o.directory)
+		len = sprintf(donor_name, "%s/", td->o.directory);
+	sprintf(donor_name + len, "%s", o->donor_name);
+
+	ed->donor_fd = open(donor_name, O_CREAT|O_WRONLY, 0644);
+	if (ed->donor_fd < 0) {
+		td_verror(td, ed->donor_fd, "io_queue_init");
+		log_err("Can't open donor file %s err:%d", ed->donor_fd);
+		free(ed);
+		return 1;
+	}
+
+	if (!o->inplace) {
+		long long len = td->o.file_size_high - td->o.start_offset;
+		r = fallocate(ed->donor_fd, 0, td->o.start_offset, len);
+		if (r)
+			goto err;
+	}
+	r = fstat(ed->donor_fd, &stub);
+	if (r)
+		goto err;
+
+	ed->bsz = stub.st_blksize;
+	td->io_ops->data = ed;
+	return 0;
+err:
+	td_verror(td, errno, "io_queue_init");
+	close(ed->donor_fd);
+	free(ed);
+	return 1;
+}
+
+static void fio_e4defrag_cleanup(struct thread_data *td)
+{
+	struct e4defrag_data *ed = td->io_ops->data;
+	if (ed) {
+		if (ed->donor_fd >= 0)
+			close(ed->donor_fd);
+		free(ed);
+	}
+}
+
+
+static int fio_e4defrag_queue(struct thread_data *td, struct io_u *io_u)
+{
+
+	int ret;
+	unsigned long long len;
+	struct move_extent me;
+	struct fio_file *f = io_u->file;
+	struct e4defrag_data *ed = td->io_ops->data;
+	struct e4defrag_options *o = td->eo;
+
+	fio_ro_check(td, io_u);
+
+	/* Theoretically defragmentation should not change data, but it
+	 * changes data layout. So this function handle only DDIR_WRITE
+	 * in order to satisfy strict read only access pattern
+	 */
+	if (io_u->ddir != DDIR_WRITE) {
+		io_u->error = errno;
+		return FIO_Q_COMPLETED;
+	}
+
+	if (o->inplace) {
+		ret = fallocate(ed->donor_fd, 0, io_u->offset, io_u->xfer_buflen);
+		if (ret) {
+			io_u->error = errno;
+			goto out;
+		}
+	}
+
+	memset(&me, 0, sizeof(me));
+	me.donor_fd = ed->donor_fd;
+	me.orig_start = io_u->offset / ed->bsz;
+	me.donor_start = me.orig_start;
+	len = (io_u->offset + io_u->xfer_buflen + ed->bsz -1);
+	me.len = len / ed->bsz - me.orig_start;
+
+	ret = ioctl(f->fd, EXT4_IOC_MOVE_EXT, &me);
+	len = me.moved_len * ed->bsz;
+
+	if (io_u->file && len >= 0 && ddir_rw(io_u->ddir))
+		io_u->file->file_pos = io_u->offset + len;
+
+	if (len > io_u->xfer_buflen)
+		len = io_u->xfer_buflen;
+
+	if (len != io_u->xfer_buflen) {
+		io_u->resid = io_u->xfer_buflen - len;
+		io_u->error = 0;
+	}
+	if (ret)
+		io_u->error = errno;
+	
+	if (o->inplace) {
+		ret = ftruncate(ed->donor_fd, 0);
+		if (ret)
+			io_u->error = errno;
+	}
+out:
+	if (io_u->error)
+		td_verror(td, errno, "xfer");
+
+
+	return FIO_Q_COMPLETED;
+}
+
+static struct ioengine_ops ioengine = {
+	.name			= "e4defrag",
+	.version		= FIO_IOOPS_VERSION,
+	.init			= fio_e4defrag_init,
+	.queue			= fio_e4defrag_queue,
+	.open_file		= generic_open_file,
+	.close_file		= generic_close_file,
+	.get_file_size		= generic_get_file_size,
+	.flags			= FIO_SYNCIO,
+	.cleanup		= fio_e4defrag_cleanup,
+	.options		= options,
+	.option_struct_size	= sizeof(struct e4defrag_options),
+
+};
+
+static void fio_init fio_syncio_register(void)
+{
+	register_ioengine(&ioengine);
+}
+
+static void fio_exit fio_syncio_unregister(void)
+{
+	unregister_ioengine(&ioengine);
+}
diff --git a/examples/e4defrag b/examples/e4defrag
new file mode 100644
index 0000000..d392149
--- /dev/null
+++ b/examples/e4defrag
@@ -0,0 +1,32 @@
+[global]
+
+direct=0
+buffered=0
+directory=/scratch
+
+nrfiles=1
+
+filesize=4G
+fadvise_hint=0
+
+group_reporting
+
+[defrag-fuzzer-8k]
+ioengine=e4defrag
+iodepth=1
+size=1G
+bs=8k
+donorname=file.def
+filename=file
+inplace=0
+rw=randwrite
+numjobs=1
+
+[random-aio-32k]
+ioengine=libaio
+iodepth=128
+bs=32k
+size=4G
+filename=file
+rw=randwrite
+numjobs=1
-- 
1.7.7.6

Re: [PATCH 0/4] ioengine: File based ioengines

[Jens Axboe]

On 2012-09-19 21:22, Dmitry Monakhov wrote:
> Fio is very cool tool for generating various io patterns.
> And I use it as a base io-loader for xfsstress framework
> 
> I've prepared two file-based ioengines which allow to generate
> interesting io patterns. 'falloc' is more generic and may be useful
> for many cases, other 'e4defrag' is more specific but still
> usfull. It would be nice to have this engines in-tree if possible.
> 
> TOC:
>  # fixes for external engines
>  export file_lookup_open
>  ioengine: allow several external ioengines
>  # file based ioengines
>  ioengine: Add fallocate ioengine
>  ioengine: Add e4defrag ioengine

This looks nifty, especially the fallocate engine. I've added the
patchset, thanks a lot.

Care to send a bit more careful documentation on both of them? The
commit message for fallocate is good, something like that in the HOWTO
would be useful. Bonus points for updating the man page, too! And
e4defrag seems largely undocumented. Some comments in the sample files
(which are great to have, btw) would be nice too.

Will also want wiring up on options.c, the ioengine part, now that they
are in tree.

-- 
Jens Axboe