* [PATCH V2 2/6] virtio/console: Add a failback for unstealable pipe buffer
From: Yoshihiro YUNOMAE @ 2012-08-09 12:30 UTC (permalink / raw)
To: linux-kernel
Cc: Herbert Xu, Arnd Bergmann, Frederic Weisbecker, qemu-devel,
Borislav Petkov, virtualization, Masami Hiramatsu,
Franch Ch. Eigler, Ingo Molnar, Mathieu Desnoyers, Steven Rostedt,
Anthony Liguori, Greg Kroah-Hartman, Amit Shah, yrl.pp-manager.tt
In-Reply-To: <20120809123029.8542.38311.stgit@ltc189.sdl.hitachi.co.jp>
From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Add a failback memcpy path for unstealable pipe buffer.
If buf->ops->steal() fails, virtio-serial tries to
copy the page contents to an allocated page, instead
of just failing splice().
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
drivers/char/virtio_console.c | 28 +++++++++++++++++++++++++---
1 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 730816c..22b7373 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -794,7 +794,7 @@ static int pipe_to_sg(struct pipe_inode_info *pipe, struct pipe_buffer *buf,
struct splice_desc *sd)
{
struct sg_list *sgl = sd->u.data;
- unsigned int len = 0;
+ unsigned int offset, len;
if (sgl->n == MAX_SPLICE_PAGES)
return 0;
@@ -807,9 +807,31 @@ static int pipe_to_sg(struct pipe_inode_info *pipe, struct pipe_buffer *buf,
len = min(buf->len, sd->len);
sg_set_page(&(sgl->sg[sgl->n]), buf->page, len, buf->offset);
- sgl->n++;
- sgl->len += len;
+ } else {
+ /* Failback to copying a page */
+ struct page *page = alloc_page(GFP_KERNEL);
+ char *src = buf->ops->map(pipe, buf, 1);
+ char *dst;
+
+ if (!page)
+ return -ENOMEM;
+ dst = kmap(page);
+
+ offset = sd->pos & ~PAGE_MASK;
+
+ len = sd->len;
+ if (len + offset > PAGE_SIZE)
+ len = PAGE_SIZE - offset;
+
+ memcpy(dst + offset, src + buf->offset, len);
+
+ kunmap(page);
+ buf->ops->unmap(pipe, buf, src);
+
+ sg_set_page(&(sgl->sg[sgl->n]), page, len, offset);
}
+ sgl->n++;
+ sgl->len += len;
return len;
}
^ permalink raw reply related
* [PATCH V2 3/6] virtio/console: Wait until the port is ready on splice
From: Yoshihiro YUNOMAE @ 2012-08-09 12:31 UTC (permalink / raw)
To: linux-kernel
Cc: Herbert Xu, Arnd Bergmann, Frederic Weisbecker, qemu-devel,
Borislav Petkov, virtualization, Masami Hiramatsu,
Franch Ch. Eigler, Ingo Molnar, Mathieu Desnoyers, Steven Rostedt,
Anthony Liguori, Greg Kroah-Hartman, Amit Shah, yrl.pp-manager.tt
In-Reply-To: <20120809123029.8542.38311.stgit@ltc189.sdl.hitachi.co.jp>
From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Wait if the port is not connected or full on splice
like as write is doing.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
drivers/char/virtio_console.c | 39 +++++++++++++++++++++++++++------------
1 files changed, 27 insertions(+), 12 deletions(-)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 22b7373..b2fc2ab 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -724,6 +724,26 @@ static ssize_t port_fops_read(struct file *filp, char __user *ubuf,
return fill_readbuf(port, ubuf, count, true);
}
+static int wait_port_writable(struct port *port, bool nonblock)
+{
+ int ret;
+
+ if (will_write_block(port)) {
+ if (nonblock)
+ return -EAGAIN;
+
+ ret = wait_event_freezable(port->waitqueue,
+ !will_write_block(port));
+ if (ret < 0)
+ return ret;
+ }
+ /* Port got hot-unplugged. */
+ if (!port->guest_connected)
+ return -ENODEV;
+
+ return 0;
+}
+
static ssize_t port_fops_write(struct file *filp, const char __user *ubuf,
size_t count, loff_t *offp)
{
@@ -740,18 +760,9 @@ static ssize_t port_fops_write(struct file *filp, const char __user *ubuf,
nonblock = filp->f_flags & O_NONBLOCK;
- if (will_write_block(port)) {
- if (nonblock)
- return -EAGAIN;
-
- ret = wait_event_freezable(port->waitqueue,
- !will_write_block(port));
- if (ret < 0)
- return ret;
- }
- /* Port got hot-unplugged. */
- if (!port->guest_connected)
- return -ENODEV;
+ ret = wait_port_writable(port, nonblock);
+ if (ret < 0)
+ return ret;
count = min((size_t)(32 * 1024), count);
@@ -851,6 +862,10 @@ static ssize_t port_fops_splice_write(struct pipe_inode_info *pipe,
.u.data = &sgl,
};
+ ret = wait_port_writable(port, filp->f_flags & O_NONBLOCK);
+ if (ret < 0)
+ return ret;
+
sgl.n = 0;
sgl.len = 0;
sgl.sg = kmalloc(sizeof(struct scatterlist) * MAX_SPLICE_PAGES,
^ permalink raw reply related
* [PATCH V2 4/6] ftrace: Allow stealing pages from pipe buffer
From: Yoshihiro YUNOMAE @ 2012-08-09 12:31 UTC (permalink / raw)
To: linux-kernel
Cc: Herbert Xu, Arnd Bergmann, Frederic Weisbecker, qemu-devel,
Borislav Petkov, virtualization, Masami Hiramatsu,
Franch Ch. Eigler, Ingo Molnar, Mathieu Desnoyers, Steven Rostedt,
Anthony Liguori, Greg Kroah-Hartman, Amit Shah, yrl.pp-manager.tt
In-Reply-To: <20120809123029.8542.38311.stgit@ltc189.sdl.hitachi.co.jp>
From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Use generic steal operation on pipe buffer to allow stealing
ring buffer's read page from pipe buffer.
Note that this could reduce the performance of splice on the
splice_write side operation without affinity setting.
Since the ring buffer's read pages are allocated on the
tracing-node, but the splice user does not always execute
splice write side operation on the same node. In this case,
the page will be accessed from the another node.
Thus, it is strongly recommended to assign the splicing
thread to corresponding node.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Acked-by: Steven Rostedt <rostedt@goodmis.org>
---
kernel/trace/trace.c | 8 +-------
1 files changed, 1 insertions(+), 7 deletions(-)
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index a120f98..ae01930 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -4194,12 +4194,6 @@ static void buffer_pipe_buf_release(struct pipe_inode_info *pipe,
buf->private = 0;
}
-static int buffer_pipe_buf_steal(struct pipe_inode_info *pipe,
- struct pipe_buffer *buf)
-{
- return 1;
-}
-
static void buffer_pipe_buf_get(struct pipe_inode_info *pipe,
struct pipe_buffer *buf)
{
@@ -4215,7 +4209,7 @@ static const struct pipe_buf_operations buffer_pipe_buf_ops = {
.unmap = generic_pipe_buf_unmap,
.confirm = generic_pipe_buf_confirm,
.release = buffer_pipe_buf_release,
- .steal = buffer_pipe_buf_steal,
+ .steal = generic_pipe_buf_steal,
.get = buffer_pipe_buf_get,
};
^ permalink raw reply related
* [PATCH V2 5/6] virtio/console: Allocate scatterlist according to the current pipe size
From: Yoshihiro YUNOMAE @ 2012-08-09 12:31 UTC (permalink / raw)
To: linux-kernel
Cc: Herbert Xu, Arnd Bergmann, Frederic Weisbecker, qemu-devel,
Borislav Petkov, virtualization, Masami Hiramatsu,
Franch Ch. Eigler, Ingo Molnar, Mathieu Desnoyers, Steven Rostedt,
Anthony Liguori, Greg Kroah-Hartman, Amit Shah, yrl.pp-manager.tt
In-Reply-To: <20120809123029.8542.38311.stgit@ltc189.sdl.hitachi.co.jp>
From: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Allocate scatterlist according to the current pipe size.
This allows splicing bigger buffer if the pipe size has
been changed by fcntl.
Changes in v2:
- Just a minor fix for avoiding a confliction with previous patch.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
---
drivers/char/virtio_console.c | 23 ++++++++++++-----------
1 files changed, 12 insertions(+), 11 deletions(-)
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index b2fc2ab..e88f843 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -229,7 +229,6 @@ struct port {
bool guest_connected;
};
-#define MAX_SPLICE_PAGES 32
/* This is the very early arch-specified put chars function. */
static int (*early_put_chars)(u32, const char *, int);
@@ -482,15 +481,16 @@ struct buffer_token {
void *buf;
struct scatterlist *sg;
} u;
- bool sgpages;
+ /* If sgpages == 0 then buf is used, else sg is used */
+ unsigned int sgpages;
};
-static void reclaim_sg_pages(struct scatterlist *sg)
+static void reclaim_sg_pages(struct scatterlist *sg, unsigned int nrpages)
{
int i;
struct page *page;
- for (i = 0; i < MAX_SPLICE_PAGES; i++) {
+ for (i = 0; i < nrpages; i++) {
page = sg_page(&sg[i]);
if (!page)
break;
@@ -511,7 +511,7 @@ static void reclaim_consumed_buffers(struct port *port)
}
while ((tok = virtqueue_get_buf(port->out_vq, &len))) {
if (tok->sgpages)
- reclaim_sg_pages(tok->u.sg);
+ reclaim_sg_pages(tok->u.sg, tok->sgpages);
else
kfree(tok->u.buf);
kfree(tok);
@@ -581,7 +581,7 @@ static ssize_t send_buf(struct port *port, void *in_buf, size_t in_count,
tok = kmalloc(sizeof(*tok), GFP_ATOMIC);
if (!tok)
return -ENOMEM;
- tok->sgpages = false;
+ tok->sgpages = 0;
tok->u.buf = in_buf;
sg_init_one(sg, in_buf, in_count);
@@ -597,7 +597,7 @@ static ssize_t send_pages(struct port *port, struct scatterlist *sg, int nents,
tok = kmalloc(sizeof(*tok), GFP_ATOMIC);
if (!tok)
return -ENOMEM;
- tok->sgpages = true;
+ tok->sgpages = nents;
tok->u.sg = sg;
return __send_to_port(port, sg, nents, in_count, tok, nonblock);
@@ -797,6 +797,7 @@ out:
struct sg_list {
unsigned int n;
+ unsigned int size;
size_t len;
struct scatterlist *sg;
};
@@ -807,7 +808,7 @@ static int pipe_to_sg(struct pipe_inode_info *pipe, struct pipe_buffer *buf,
struct sg_list *sgl = sd->u.data;
unsigned int offset, len;
- if (sgl->n == MAX_SPLICE_PAGES)
+ if (sgl->n == sgl->size)
return 0;
/* Try lock this page */
@@ -868,12 +869,12 @@ static ssize_t port_fops_splice_write(struct pipe_inode_info *pipe,
sgl.n = 0;
sgl.len = 0;
- sgl.sg = kmalloc(sizeof(struct scatterlist) * MAX_SPLICE_PAGES,
- GFP_KERNEL);
+ sgl.size = pipe->nrbufs;
+ sgl.sg = kmalloc(sizeof(struct scatterlist) * sgl.size, GFP_KERNEL);
if (unlikely(!sgl.sg))
return -ENOMEM;
- sg_init_table(sgl.sg, MAX_SPLICE_PAGES);
+ sg_init_table(sgl.sg, sgl.size);
ret = __splice_from_pipe(pipe, &sd, pipe_to_sg);
if (likely(ret > 0))
ret = send_pages(port, sgl.sg, sgl.n, sgl.len, true);
^ permalink raw reply related
* [PATCH V2 6/6] tools: Add guest trace agent as a user tool
From: Yoshihiro YUNOMAE @ 2012-08-09 12:31 UTC (permalink / raw)
To: linux-kernel
Cc: Herbert Xu, Arnd Bergmann, Frederic Weisbecker, qemu-devel,
Borislav Petkov, virtualization, Masami Hiramatsu,
Franch Ch. Eigler, Ingo Molnar, Mathieu Desnoyers, Steven Rostedt,
Anthony Liguori, Greg Kroah-Hartman, Amit Shah, yrl.pp-manager.tt
In-Reply-To: <20120809123029.8542.38311.stgit@ltc189.sdl.hitachi.co.jp>
This patch adds a user tool, "trace agent" for sending trace data of a guest to
a Host in low overhead. This agent has the following functions:
- splice a page of ring-buffer to read_pipe without memory copying
- splice the page from write_pipe to virtio-console without memory copying
- write trace data to stdout by using -o option
- controlled by start/stop orders from a Host
Changes in v2:
- Cleanup (change fprintf() to pr_err() and an include guard)
Signed-off-by: Yoshihiro YUNOMAE <yoshihiro.yunomae.ez@hitachi.com>
---
tools/virtio/virtio-trace/Makefile | 14 +
tools/virtio/virtio-trace/README | 118 ++++++++++++
tools/virtio/virtio-trace/trace-agent-ctl.c | 137 ++++++++++++++
tools/virtio/virtio-trace/trace-agent-rw.c | 192 +++++++++++++++++++
tools/virtio/virtio-trace/trace-agent.c | 270 +++++++++++++++++++++++++++
tools/virtio/virtio-trace/trace-agent.h | 75 ++++++++
6 files changed, 806 insertions(+), 0 deletions(-)
create mode 100644 tools/virtio/virtio-trace/Makefile
create mode 100644 tools/virtio/virtio-trace/README
create mode 100644 tools/virtio/virtio-trace/trace-agent-ctl.c
create mode 100644 tools/virtio/virtio-trace/trace-agent-rw.c
create mode 100644 tools/virtio/virtio-trace/trace-agent.c
create mode 100644 tools/virtio/virtio-trace/trace-agent.h
diff --git a/tools/virtio/virtio-trace/Makefile b/tools/virtio/virtio-trace/Makefile
new file mode 100644
index 0000000..ef3adfc
--- /dev/null
+++ b/tools/virtio/virtio-trace/Makefile
@@ -0,0 +1,14 @@
+CC = gcc
+CFLAGS = -O2 -Wall
+LFLAG = -lpthread
+
+all: trace-agent
+
+.c.o:
+ $(CC) $(CFLAGS) $(LFLAG) -c $^ -o $@
+
+trace-agent: trace-agent.o trace-agent-ctl.o trace-agent-rw.o
+ $(CC) $(CFLAGS) $(LFLAG) -o $@ $^
+
+clean:
+ rm -f *.o trace-agent
diff --git a/tools/virtio/virtio-trace/README b/tools/virtio/virtio-trace/README
new file mode 100644
index 0000000..b64845b
--- /dev/null
+++ b/tools/virtio/virtio-trace/README
@@ -0,0 +1,118 @@
+Trace Agent for virtio-trace
+============================
+
+Trace agent is a user tool for sending trace data of a guest to a Host in low
+overhead. Trace agent has the following functions:
+ - splice a page of ring-buffer to read_pipe without memory copying
+ - splice the page from write_pipe to virtio-console without memory copying
+ - write trace data to stdout by using -o option
+ - controlled by start/stop orders from a Host
+
+The trace agent operates as follows:
+ 1) Initialize all structures.
+ 2) Create a read/write thread per CPU. Each thread is bound to a CPU.
+ The read/write threads hold it.
+ 3) A controller thread does poll() for a start order of a host.
+ 4) After the controller of the trace agent receives a start order from a host,
+ the controller wake read/write threads.
+ 5) The read/write threads start to read trace data from ring-buffers and
+ write the data to virtio-serial.
+ 6) If the controller receives a stop order from a host, the read/write threads
+ stop to read trace data.
+
+
+Files
+=====
+
+README: this file
+Makefile: Makefile of trace agent for virtio-trace
+trace-agent.c: includes main function, sets up for operating trace agent
+trace-agent.h: includes all structures and some macros
+trace-agent-ctl.c: includes controller function for read/write threads
+trace-agent-rw.c: includes read/write threads function
+
+
+Setup
+=====
+
+To use this trace agent for virtio-trace, we need to prepare some virtio-serial
+I/Fs.
+
+1) Make FIFO in a host
+ virtio-trace uses virtio-serial pipe as trace data paths as to the number
+of CPUs and a control path, so FIFO (named pipe) should be created as follows:
+ # mkdir /tmp/virtio-trace/
+ # mkfifo /tmp/virtio-trace/trace-path-cpu{0,1,2,...,X}.{in,out}
+ # mkfifo /tmp/virtio-trace/agent-ctl-path.{in,out}
+
+For example, if a guest use three CPUs, the names are
+ trace-path-cpu{0,1,2}.{in.out}
+and
+ agent-ctl-path.{in,out}.
+
+2) Set up of virtio-serial pipe in a host
+ Add qemu option to use virtio-serial pipe.
+
+ ##virtio-serial device##
+ -device virtio-serial-pci,id=virtio-serial0\
+ ##control path##
+ -chardev pipe,id=charchannel0,path=/tmp/virtio-trace/agent-ctl-path\
+ -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,\
+ id=channel0,name=agent-ctl-path\
+ ##data path##
+ -chardev pipe,id=charchannel1,path=/tmp/virtio-trace/trace-path-cpu0\
+ -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel0,\
+ id=channel1,name=trace-path-cpu0\
+ ...
+
+If you manage guests with libvirt, add the following tags to domain XML files.
+Then, libvirt passes the same command option to qemu.
+
+ <channel type='pipe'>
+ <source path='/tmp/virtio-trace/agent-ctl-path'/>
+ <target type='virtio' name='agent-ctl-path'/>
+ <address type='virtio-serial' controller='0' bus='0' port='0'/>
+ </channel>
+ <channel type='pipe'>
+ <source path='/tmp/virtio-trace/trace-path-cpu0'/>
+ <target type='virtio' name='trace-path-cpu0'/>
+ <address type='virtio-serial' controller='0' bus='0' port='1'/>
+ </channel>
+ ...
+Here, chardev names are restricted to trace-path-cpuX and agent-ctl-path. For
+example, if a guest use three CPUs, chardev names should be trace-path-cpu0,
+trace-path-cpu1, trace-path-cpu2, and agent-ctl-path.
+
+3) Boot the guest
+ You can find some chardev in /dev/virtio-ports/ in the guest.
+
+
+Run
+===
+
+0) Build trace agent in a guest
+ $ make
+
+1) Enable ftrace in the guest
+ <Example>
+ # echo 1 > /sys/kernel/debug/tracing/events/sched/enable
+
+2) Run trace agent in the guest
+ This agent must be operated as root.
+ # ./trace-agent
+read/write threads in the agent wait for start order from host. If you add -o
+option, trace data are output via stdout in the guest.
+
+3) Open FIFO in a host
+ # cat /tmp/virtio-trace/trace-path-cpu0.out
+If a host does not open these, trace data get stuck in buffers of virtio. Then,
+the guest will stop by specification of chardev in QEMU. This blocking mode may
+be solved in the future.
+
+4) Start to read trace data by ordering from a host
+ A host injects read start order to the guest via virtio-serial.
+ # echo 1 > /tmp/virtio-trace/agent-ctl-path.in
+
+5) Stop to read trace data by ordering from a host
+ A host injects read stop order to the guest via virtio-serial.
+ # echo 0 > /tmp/virtio-trace/agent-ctl-path.in
diff --git a/tools/virtio/virtio-trace/trace-agent-ctl.c b/tools/virtio/virtio-trace/trace-agent-ctl.c
new file mode 100644
index 0000000..a2d0403
--- /dev/null
+++ b/tools/virtio/virtio-trace/trace-agent-ctl.c
@@ -0,0 +1,137 @@
+/*
+ * Controller of read/write threads for virtio-trace
+ *
+ * Copyright (C) 2012 Hitachi, Ltd.
+ * Created by Yoshihiro Yunomae <yoshihiro.yunomae.ez@hitachi.com>
+ * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
+ *
+ * Licensed under GPL version 2 only.
+ *
+ */
+
+#define _GNU_SOURCE
+#include <fcntl.h>
+#include <poll.h>
+#include <signal.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include "trace-agent.h"
+
+#define HOST_MSG_SIZE 256
+#define EVENT_WAIT_MSEC 100
+
+static volatile sig_atomic_t global_signal_val;
+bool global_sig_receive; /* default false */
+bool global_run_operation; /* default false*/
+
+/* Handle SIGTERM/SIGINT/SIGQUIT to exit */
+static void signal_handler(int sig)
+{
+ global_signal_val = sig;
+}
+
+int rw_ctl_init(const char *ctl_path)
+{
+ int ctl_fd;
+
+ ctl_fd = open(ctl_path, O_RDONLY);
+ if (ctl_fd == -1) {
+ pr_err("Cannot open ctl_fd\n");
+ goto error;
+ }
+
+ return ctl_fd;
+
+error:
+ exit(EXIT_FAILURE);
+}
+
+static int wait_order(int ctl_fd)
+{
+ struct pollfd poll_fd;
+ int ret = 0;
+
+ while (!global_sig_receive) {
+ poll_fd.fd = ctl_fd;
+ poll_fd.events = POLLIN;
+
+ ret = poll(&poll_fd, 1, EVENT_WAIT_MSEC);
+
+ if (global_signal_val) {
+ global_sig_receive = true;
+ pr_info("Receive interrupt %d\n", global_signal_val);
+
+ /* Wakes rw-threads when they are sleeping */
+ if (!global_run_operation)
+ pthread_cond_broadcast(&cond_wakeup);
+
+ ret = -1;
+ break;
+ }
+
+ if (ret < 0) {
+ pr_err("Polling error\n");
+ goto error;
+ }
+
+ if (ret)
+ break;
+ };
+
+ return ret;
+
+error:
+ exit(EXIT_FAILURE);
+}
+
+/*
+ * contol read/write threads by handling global_run_operation
+ */
+void *rw_ctl_loop(int ctl_fd)
+{
+ ssize_t rlen;
+ char buf[HOST_MSG_SIZE];
+ int ret;
+
+ /* Setup signal handlers */
+ signal(SIGTERM, signal_handler);
+ signal(SIGINT, signal_handler);
+ signal(SIGQUIT, signal_handler);
+
+ while (!global_sig_receive) {
+
+ ret = wait_order(ctl_fd);
+ if (ret < 0)
+ break;
+
+ rlen = read(ctl_fd, buf, sizeof(buf));
+ if (rlen < 0) {
+ pr_err("read data error in ctl thread\n");
+ goto error;
+ }
+
+ if (rlen == 2 && buf[0] == '1') {
+ /*
+ * If host writes '1' to a control path,
+ * this controller wakes all read/write threads.
+ */
+ global_run_operation = true;
+ pthread_cond_broadcast(&cond_wakeup);
+ pr_debug("Wake up all read/write threads\n");
+ } else if (rlen == 2 && buf[0] == '0') {
+ /*
+ * If host writes '0' to a control path, read/write
+ * threads will wait for notification from Host.
+ */
+ global_run_operation = false;
+ pr_debug("Stop all read/write threads\n");
+ } else
+ pr_info("Invalid host notification: %s\n", buf);
+ }
+
+ return NULL;
+
+error:
+ exit(EXIT_FAILURE);
+}
diff --git a/tools/virtio/virtio-trace/trace-agent-rw.c b/tools/virtio/virtio-trace/trace-agent-rw.c
new file mode 100644
index 0000000..3aace5e
--- /dev/null
+++ b/tools/virtio/virtio-trace/trace-agent-rw.c
@@ -0,0 +1,192 @@
+/*
+ * Read/write thread of a guest agent for virtio-trace
+ *
+ * Copyright (C) 2012 Hitachi, Ltd.
+ * Created by Yoshihiro Yunomae <yoshihiro.yunomae.ez@hitachi.com>
+ * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
+ *
+ * Licensed under GPL version 2 only.
+ *
+ */
+
+#define _GNU_SOURCE
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/syscall.h>
+#include "trace-agent.h"
+
+#define READ_WAIT_USEC 100000
+
+void *rw_thread_info_new(void)
+{
+ struct rw_thread_info *rw_ti;
+
+ rw_ti = zalloc(sizeof(struct rw_thread_info));
+ if (rw_ti == NULL) {
+ pr_err("rw_thread_info zalloc error\n");
+ exit(EXIT_FAILURE);
+ }
+
+ rw_ti->cpu_num = -1;
+ rw_ti->in_fd = -1;
+ rw_ti->out_fd = -1;
+ rw_ti->read_pipe = -1;
+ rw_ti->write_pipe = -1;
+ rw_ti->pipe_size = PIPE_INIT;
+
+ return rw_ti;
+}
+
+void *rw_thread_init(int cpu, const char *in_path, const char *out_path,
+ bool stdout_flag, unsigned long pipe_size,
+ struct rw_thread_info *rw_ti)
+{
+ int data_pipe[2];
+
+ rw_ti->cpu_num = cpu;
+
+ /* set read(input) fd */
+ rw_ti->in_fd = open(in_path, O_RDONLY);
+ if (rw_ti->in_fd == -1) {
+ pr_err("Could not open in_fd (CPU:%d)\n", cpu);
+ goto error;
+ }
+
+ /* set write(output) fd */
+ if (!stdout_flag) {
+ /* virtio-serial output mode */
+ rw_ti->out_fd = open(out_path, O_WRONLY);
+ if (rw_ti->out_fd == -1) {
+ pr_err("Could not open out_fd (CPU:%d)\n", cpu);
+ goto error;
+ }
+ } else
+ /* stdout mode */
+ rw_ti->out_fd = STDOUT_FILENO;
+
+ if (pipe2(data_pipe, O_NONBLOCK) < 0) {
+ pr_err("Could not create pipe in rw-thread(%d)\n", cpu);
+ goto error;
+ }
+
+ /*
+ * Size of pipe is 64kB in default based on fs/pipe.c.
+ * To read/write trace data speedy, pipe size is changed.
+ */
+ if (fcntl(*data_pipe, F_SETPIPE_SZ, pipe_size) < 0) {
+ pr_err("Could not change pipe size in rw-thread(%d)\n", cpu);
+ goto error;
+ }
+
+ rw_ti->read_pipe = data_pipe[1];
+ rw_ti->write_pipe = data_pipe[0];
+ rw_ti->pipe_size = pipe_size;
+
+ return NULL;
+
+error:
+ exit(EXIT_FAILURE);
+}
+
+/* Bind a thread to a cpu */
+static void bind_cpu(int cpu_num)
+{
+ cpu_set_t mask;
+
+ CPU_ZERO(&mask);
+ CPU_SET(cpu_num, &mask);
+
+ /* bind my thread to cpu_num by assigning zero to the first argument */
+ if (sched_setaffinity(0, sizeof(mask), &mask) == -1)
+ pr_err("Could not set CPU#%d affinity\n", (int)cpu_num);
+}
+
+static void *rw_thread_main(void *thread_info)
+{
+ ssize_t rlen, wlen;
+ ssize_t ret;
+ struct rw_thread_info *ts = (struct rw_thread_info *)thread_info;
+
+ bind_cpu(ts->cpu_num);
+
+ while (1) {
+ /* Wait for a read order of trace data by Host OS */
+ if (!global_run_operation) {
+ pthread_mutex_lock(&mutex_notify);
+ pthread_cond_wait(&cond_wakeup, &mutex_notify);
+ pthread_mutex_unlock(&mutex_notify);
+ }
+
+ if (global_sig_receive)
+ break;
+
+ /*
+ * Each thread read trace_pipe_raw of each cpu bounding the
+ * thread, so contention of multi-threads does not occur.
+ */
+ rlen = splice(ts->in_fd, NULL, ts->read_pipe, NULL,
+ ts->pipe_size, SPLICE_F_MOVE | SPLICE_F_MORE);
+
+ if (rlen < 0) {
+ pr_err("Splice_read in rw-thread(%d)\n", ts->cpu_num);
+ goto error;
+ } else if (rlen == 0) {
+ /*
+ * If trace data do not exist or are unreadable not
+ * for exceeding the page size, splice_read returns
+ * NULL. Then, this waits for being filled the data in a
+ * ring-buffer.
+ */
+ usleep(READ_WAIT_USEC);
+ pr_debug("Read retry(cpu:%d)\n", ts->cpu_num);
+ continue;
+ }
+
+ wlen = 0;
+
+ do {
+ ret = splice(ts->write_pipe, NULL, ts->out_fd, NULL,
+ rlen - wlen,
+ SPLICE_F_MOVE | SPLICE_F_MORE);
+
+ if (ret < 0) {
+ pr_err("Splice_write in rw-thread(%d)\n",
+ ts->cpu_num);
+ goto error;
+ } else if (ret == 0)
+ /*
+ * When host reader is not in time for reading
+ * trace data, guest will be stopped. This is
+ * because char dev in QEMU is not supported
+ * non-blocking mode. Then, writer might be
+ * sleep in that case.
+ * This sleep will be removed by supporting
+ * non-blocking mode.
+ */
+ sleep(1);
+ wlen += ret;
+ } while (wlen < rlen);
+ }
+
+ return NULL;
+
+error:
+ exit(EXIT_FAILURE);
+}
+
+
+pthread_t rw_thread_run(struct rw_thread_info *rw_ti)
+{
+ int ret;
+ pthread_t rw_thread_per_cpu;
+
+ ret = pthread_create(&rw_thread_per_cpu, NULL, rw_thread_main, rw_ti);
+ if (ret != 0) {
+ pr_err("Could not create a rw thread(%d)\n", rw_ti->cpu_num);
+ exit(EXIT_FAILURE);
+ }
+
+ return rw_thread_per_cpu;
+}
diff --git a/tools/virtio/virtio-trace/trace-agent.c b/tools/virtio/virtio-trace/trace-agent.c
new file mode 100644
index 0000000..0a0a7dd
--- /dev/null
+++ b/tools/virtio/virtio-trace/trace-agent.c
@@ -0,0 +1,270 @@
+/*
+ * Guest agent for virtio-trace
+ *
+ * Copyright (C) 2012 Hitachi, Ltd.
+ * Created by Yoshihiro Yunomae <yoshihiro.yunomae.ez@hitachi.com>
+ * Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
+ *
+ * Licensed under GPL version 2 only.
+ *
+ */
+
+#define _GNU_SOURCE
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include "trace-agent.h"
+
+#define PAGE_SIZE (sysconf(_SC_PAGE_SIZE))
+#define PIPE_DEF_BUFS 16
+#define PIPE_MIN_SIZE (PAGE_SIZE*PIPE_DEF_BUFS)
+#define PIPE_MAX_SIZE (1024*1024)
+#define READ_PATH_FMT \
+ "/sys/kernel/debug/tracing/per_cpu/cpu%d/trace_pipe_raw"
+#define WRITE_PATH_FMT "/dev/virtio-ports/trace-path-cpu%d"
+#define CTL_PATH "/dev/virtio-ports/agent-ctl-path"
+
+pthread_mutex_t mutex_notify = PTHREAD_MUTEX_INITIALIZER;
+pthread_cond_t cond_wakeup = PTHREAD_COND_INITIALIZER;
+
+static int get_total_cpus(void)
+{
+ int nr_cpus = (int)sysconf(_SC_NPROCESSORS_CONF);
+
+ if (nr_cpus <= 0) {
+ pr_err("Could not read cpus\n");
+ goto error;
+ } else if (nr_cpus > MAX_CPUS) {
+ pr_err("Exceed max cpus(%d)\n", (int)MAX_CPUS);
+ goto error;
+ }
+
+ return nr_cpus;
+
+error:
+ exit(EXIT_FAILURE);
+}
+
+static void *agent_info_new(void)
+{
+ struct agent_info *s;
+ int i;
+
+ s = zalloc(sizeof(struct agent_info));
+ if (s == NULL) {
+ pr_err("agent_info zalloc error\n");
+ exit(EXIT_FAILURE);
+ }
+
+ s->pipe_size = PIPE_INIT;
+ s->use_stdout = false;
+ s->cpus = get_total_cpus();
+ s->ctl_fd = -1;
+
+ /* read/write threads init */
+ for (i = 0; i < s->cpus; i++)
+ s->rw_ti[i] = rw_thread_info_new();
+
+ return s;
+}
+
+static unsigned long parse_size(const char *arg)
+{
+ unsigned long value, round;
+ char *ptr;
+
+ value = strtoul(arg, &ptr, 10);
+ switch (*ptr) {
+ case 'K': case 'k':
+ value <<= 10;
+ break;
+ case 'M': case 'm':
+ value <<= 20;
+ break;
+ default:
+ break;
+ }
+
+ if (value > PIPE_MAX_SIZE) {
+ pr_err("Pipe size must be less than 1MB\n");
+ goto error;
+ } else if (value < PIPE_MIN_SIZE) {
+ pr_err("Pipe size must be over 64KB\n");
+ goto error;
+ }
+
+ /* Align buffer size with page unit */
+ round = value & (PAGE_SIZE - 1);
+ value = value - round;
+
+ return value;
+error:
+ return 0;
+}
+
+static void usage(char const *prg)
+{
+ pr_err("usage: %s [-h] [-o] [-s <size of pipe>]\n", prg);
+}
+
+static const char *make_path(int cpu_num, bool this_is_write_path)
+{
+ int ret;
+ char *buf;
+
+ buf = zalloc(PATH_MAX);
+ if (buf == NULL) {
+ pr_err("Could not allocate buffer\n");
+ goto error;
+ }
+
+ if (this_is_write_path)
+ /* write(output) path */
+ ret = snprintf(buf, PATH_MAX, WRITE_PATH_FMT, cpu_num);
+ else
+ /* read(input) path */
+ ret = snprintf(buf, PATH_MAX, READ_PATH_FMT, cpu_num);
+
+ if (ret <= 0) {
+ pr_err("Failed to generate %s path(CPU#%d):%d\n",
+ this_is_write_path ? "read" : "write", cpu_num, ret);
+ goto error;
+ }
+
+ return buf;
+
+error:
+ free(buf);
+ return NULL;
+}
+
+static const char *make_input_path(int cpu_num)
+{
+ return make_path(cpu_num, false);
+}
+
+static const char *make_output_path(int cpu_num)
+{
+ return make_path(cpu_num, true);
+}
+
+static void *agent_info_init(struct agent_info *s)
+{
+ int cpu;
+ const char *in_path = NULL;
+ const char *out_path = NULL;
+
+ /* init read/write threads */
+ for (cpu = 0; cpu < s->cpus; cpu++) {
+ /* set read(input) path per read/write thread */
+ in_path = make_input_path(cpu);
+ if (in_path == NULL)
+ goto error;
+
+ /* set write(output) path per read/write thread*/
+ if (!s->use_stdout) {
+ out_path = make_output_path(cpu);
+ if (out_path == NULL)
+ goto error;
+ } else
+ /* stdout mode */
+ pr_debug("stdout mode\n");
+
+ rw_thread_init(cpu, in_path, out_path, s->use_stdout,
+ s->pipe_size, s->rw_ti[cpu]);
+ }
+
+ /* init controller of read/write threads */
+ s->ctl_fd = rw_ctl_init((const char *)CTL_PATH);
+
+ return NULL;
+
+error:
+ exit(EXIT_FAILURE);
+}
+
+static void *parse_args(int argc, char *argv[], struct agent_info *s)
+{
+ int cmd;
+ unsigned long size;
+
+ while ((cmd = getopt(argc, argv, "hos:")) != -1) {
+ switch (cmd) {
+ /* stdout mode */
+ case 'o':
+ s->use_stdout = true;
+ break;
+ /* size of pipe */
+ case 's':
+ size = parse_size(optarg);
+ if (size == 0)
+ goto error;
+ s->pipe_size = size;
+ break;
+ case 'h':
+ default:
+ usage(argv[0]);
+ goto error;
+ }
+ }
+
+ agent_info_init(s);
+
+ return NULL;
+
+error:
+ exit(EXIT_FAILURE);
+}
+
+static void agent_main_loop(struct agent_info *s)
+{
+ int cpu;
+ pthread_t rw_thread_per_cpu[MAX_CPUS];
+
+ /* Start all read/write threads */
+ for (cpu = 0; cpu < s->cpus; cpu++)
+ rw_thread_per_cpu[cpu] = rw_thread_run(s->rw_ti[cpu]);
+
+ rw_ctl_loop(s->ctl_fd);
+
+ /* Finish all read/write threads */
+ for (cpu = 0; cpu < s->cpus; cpu++) {
+ int ret;
+
+ ret = pthread_join(rw_thread_per_cpu[cpu], NULL);
+ if (ret != 0) {
+ pr_err("pthread_join() error:%d (cpu %d)\n", ret, cpu);
+ exit(EXIT_FAILURE);
+ }
+ }
+}
+
+static void agent_info_free(struct agent_info *s)
+{
+ int i;
+
+ close(s->ctl_fd);
+ for (i = 0; i < s->cpus; i++) {
+ close(s->rw_ti[i]->in_fd);
+ close(s->rw_ti[i]->out_fd);
+ close(s->rw_ti[i]->read_pipe);
+ close(s->rw_ti[i]->write_pipe);
+ free(s->rw_ti[i]);
+ }
+ free(s);
+}
+
+int main(int argc, char *argv[])
+{
+ struct agent_info *s = NULL;
+
+ s = agent_info_new();
+ parse_args(argc, argv, s);
+
+ agent_main_loop(s);
+
+ agent_info_free(s);
+
+ return 0;
+}
diff --git a/tools/virtio/virtio-trace/trace-agent.h b/tools/virtio/virtio-trace/trace-agent.h
new file mode 100644
index 0000000..8de79bf
--- /dev/null
+++ b/tools/virtio/virtio-trace/trace-agent.h
@@ -0,0 +1,75 @@
+#ifndef __TRACE_AGENT_H__
+#define __TRACE_AGENT_H__
+#include <pthread.h>
+#include <stdbool.h>
+
+#define MAX_CPUS 256
+#define PIPE_INIT (1024*1024)
+
+/*
+ * agent_info - structure managing total information of guest agent
+ * @pipe_size: size of pipe (default 1MB)
+ * @use_stdout: set to true when o option is added (default false)
+ * @cpus: total number of CPUs
+ * @ctl_fd: fd of control path, /dev/virtio-ports/agent-ctl-path
+ * @rw_ti: structure managing information of read/write threads
+ */
+struct agent_info {
+ unsigned long pipe_size;
+ bool use_stdout;
+ int cpus;
+ int ctl_fd;
+ struct rw_thread_info *rw_ti[MAX_CPUS];
+};
+
+/*
+ * rw_thread_info - structure managing a read/write thread a cpu
+ * @cpu_num: cpu number operating this read/write thread
+ * @in_fd: fd of reading trace data path in cpu_num
+ * @out_fd: fd of writing trace data path in cpu_num
+ * @read_pipe: fd of read pipe
+ * @write_pipe: fd of write pipe
+ * @pipe_size: size of pipe (default 1MB)
+ */
+struct rw_thread_info {
+ int cpu_num;
+ int in_fd;
+ int out_fd;
+ int read_pipe;
+ int write_pipe;
+ unsigned long pipe_size;
+};
+
+/* use for stopping rw threads */
+extern bool global_sig_receive;
+
+/* use for notification */
+extern bool global_run_operation;
+extern pthread_mutex_t mutex_notify;
+extern pthread_cond_t cond_wakeup;
+
+/* for controller of read/write threads */
+extern int rw_ctl_init(const char *ctl_path);
+extern void *rw_ctl_loop(int ctl_fd);
+
+/* for trace read/write thread */
+extern void *rw_thread_info_new(void);
+extern void *rw_thread_init(int cpu, const char *in_path, const char *out_path,
+ bool stdout_flag, unsigned long pipe_size,
+ struct rw_thread_info *rw_ti);
+extern pthread_t rw_thread_run(struct rw_thread_info *rw_ti);
+
+static inline void *zalloc(size_t size)
+{
+ return calloc(1, size);
+}
+
+#define pr_err(format, ...) fprintf(stderr, format, ## __VA_ARGS__)
+#define pr_info(format, ...) fprintf(stdout, format, ## __VA_ARGS__)
+#ifdef DEBUG
+#define pr_debug(format, ...) fprintf(stderr, format, ## __VA_ARGS__)
+#else
+#define pr_debug(format, ...) do {} while (0)
+#endif
+
+#endif /*__TRACE_AGENT_H__*/
^ permalink raw reply related
* Re: [RFC PATCH 2/6] virtio/console: Add a failback for unstealable pipe buffer
From: Steven Rostedt @ 2012-08-09 12:35 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: Arnd Bergmann, qemu-devel, Frederic Weisbecker, yrl.pp-manager.tt,
linux-kernel, Borislav Petkov, virtualization, Herbert Xu,
Franch Ch. Eigler, Ingo Molnar, Mathieu Desnoyers,
Anthony Liguori, Greg Kroah-Hartman, Amit Shah
In-Reply-To: <502381EA.80805@hitachi.com>
On Thu, 2012-08-09 at 18:24 +0900, Masami Hiramatsu wrote:
> Yeah, it is really easy to fix that.
> But out of curiosity, would that be really a problem?
> I guess that host can access any guest page if need. If that
> is right, is that really insecure to leak randomly allocated
> unused page to the host?
Yeah, it's like protecting userspace pages from the kernel ;-)
-- Steve
^ permalink raw reply
* Re: [PATCH v6 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-09 14:48 UTC (permalink / raw)
To: Mel Gorman
Cc: Rik van Riel, Konrad Rzeszutek Wilk, Michael S. Tsirkin,
linux-kernel, virtualization, linux-mm, Andi Kleen, Minchan Kim,
Andrew Morton
In-Reply-To: <20120809090019.GB10288@csn.ul.ie>
On Thu, Aug 09, 2012 at 10:00:19AM +0100, Mel Gorman wrote:
> On Wed, Aug 08, 2012 at 07:53:19PM -0300, Rafael Aquini wrote:
> > Memory fragmentation introduced by ballooning might reduce significantly
> > the number of 2MB contiguous memory blocks that can be used within a guest,
> > thus imposing performance penalties associated with the reduced number of
> > transparent huge pages that could be used by the guest workload.
> >
> > This patch introduces the helper functions as well as the necessary changes
> > to teach compaction and migration bits how to cope with pages which are
> > part of a guest memory balloon, in order to make them movable by memory
> > compaction procedures.
> >
> > Signed-off-by: Rafael Aquini <aquini@redhat.com>
>
> Mostly looks ok but I have one question;
>
> > <SNIP>
> >
> > +/* putback_lru_page() counterpart for a ballooned page */
> > +bool putback_balloon_page(struct page *page)
> > +{
> > + if (WARN_ON(!movable_balloon_page(page)))
> > + return false;
> > +
> > + if (likely(trylock_page(page))) {
> > + if (movable_balloon_page(page)) {
> > + __putback_balloon_page(page);
> > + put_page(page);
> > + unlock_page(page);
> > + return true;
> > + }
> > + unlock_page(page);
> > + }
>
> You might have answered this already as I skipped over a few revisions
> and if you have, sorry about that and please add a comment :)
>
> This trylock_page looks risky as it looks like it can fail if another
> process running compaction tries to isolate this page. It locks the page,
> finds it cant and releases the lock but in the meantime this trylock can
> fail. It triggers a WARN_ON so we'll get a bug report but it leaves the
> reference count elevated and this page has now leaked.
>
Good catch!
I had those bits changed to follow the same logics you had suggested for
isolate_balloon_page(), but I ended up completely missing this potential page
leak case you spotted. Thanks a lot!
> Why not just lock_page(page)? As you have already isolated this page you
> know that the lock is only going to be held by a parallel compacting
> process checking the reference count and the delay will be short. As a
> bonus you can drop the WARN_ON check in the caller and make this void as
> the WARN_ON check in the caller becomes redundant.
>
Sure!
what do you think of:
+/* putback_lru_page() counterpart for a ballooned page */
+void putback_balloon_page(struct page *page)
+{
+ lock_page(page);
+ if (!WARN_ON(!movable_balloon_page(page))) {
+ __putback_balloon_page(page);
+ put_page(page);
+ }
+ unlock_page(page);
+}
^ permalink raw reply
* Re: [PATCH v6 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-09 15:12 UTC (permalink / raw)
To: Mel Gorman
Cc: Rik van Riel, Konrad Rzeszutek Wilk, Michael S. Tsirkin,
linux-kernel, virtualization, linux-mm, Andi Kleen, Minchan Kim,
Andrew Morton
In-Reply-To: <20120809144835.GA2719@t510.redhat.com>
On Thu, Aug 09, 2012 at 11:48:36AM -0300, Rafael Aquini wrote:
> Sure!
> what do you think of:
>
> +/* putback_lru_page() counterpart for a ballooned page */
> +void putback_balloon_page(struct page *page)
> +{
> + lock_page(page);
> + if (!WARN_ON(!movable_balloon_page(page))) {
> + __putback_balloon_page(page);
> + put_page(page);
> + }
> + unlock_page(page);
> +}
>
Or perhaps
+/* putback_lru_page() counterpart for a ballooned page */
+void putback_balloon_page(struct page *page)
+{
+ if (!WARN_ON(!movable_balloon_page(page))) {
+ lock_page(page);
+ __putback_balloon_page(page);
+ put_page(page);
+ unlock_page(page);
+ }
+}
^ permalink raw reply
* Re: [net-next RFC V5 3/5] virtio: intorduce an API to set affinity for a virtqueue
From: Paolo Bonzini @ 2012-08-09 15:13 UTC (permalink / raw)
To: Jason Wang
Cc: krkumar2, habanero, mashirle, kvm, mst, netdev, linux-kernel,
virtualization, edumazet, tahm, jwhan, davem, sri
In-Reply-To: <1341484194-8108-4-git-send-email-jasowang@redhat.com>
Il 05/07/2012 12:29, Jason Wang ha scritto:
> Sometimes, virtio device need to configure irq affiniry hint to maximize the
> performance. Instead of just exposing the irq of a virtqueue, this patch
> introduce an API to set the affinity for a virtqueue.
>
> The api is best-effort, the affinity hint may not be set as expected due to
> platform support, irq sharing or irq type. Currently, only pci method were
> implemented and we set the affinity according to:
>
> - if device uses INTX, we just ignore the request
> - if device has per vq vector, we force the affinity hint
> - if the virtqueues share MSI, make the affinity OR over all affinities
> requested
>
> Signed-off-by: Jason Wang <jasowang@redhat.com>
It looks like both I and Jason will need these patches during the 3.7
merge window, and from different trees (net-next vs. scsi). How do we
synchronize?
Paolo
^ permalink raw reply
* Re: [net-next RFC V5 3/5] virtio: intorduce an API to set affinity for a virtqueue
From: Paolo Bonzini @ 2012-08-09 15:14 UTC (permalink / raw)
Cc: krkumar2, habanero, kvm, Michael S. Tsirkin, netdev, mashirle,
linux-kernel, virtualization, edumazet, tahm, jwhan, davem, sri
In-Reply-To: <5016296A.5070809@redhat.com>
Il 30/07/2012 08:27, Paolo Bonzini ha scritto:
>>>> >> > Did you set the affinity manually in your experiments, or perhaps there
>>>> >> > is a difference between scsi and networking... (interrupt mitigation?)
>> >
>> > You need to run irqbalancer in guest to make it actually work. Do you?
> Yes, of course, now on to debugging that one. I just wanted to ask
> before the weekend if I was missing something as obvious as that.
It was indeed an irqbalance bug, it is fixed now upstream.
Paolo
^ permalink raw reply
* Re: [net-next RFC V5 3/5] virtio: intorduce an API to set affinity for a virtqueue
From: Avi Kivity @ 2012-08-09 15:35 UTC (permalink / raw)
To: Paolo Bonzini
Cc: krkumar2, habanero, kvm, mst, netdev, mashirle, linux-kernel,
virtualization, edumazet, tahm, jwhan, davem, sri
In-Reply-To: <5023D3A1.8040102@redhat.com>
On 08/09/2012 06:13 PM, Paolo Bonzini wrote:
> Il 05/07/2012 12:29, Jason Wang ha scritto:
>> Sometimes, virtio device need to configure irq affiniry hint to maximize the
>> performance. Instead of just exposing the irq of a virtqueue, this patch
>> introduce an API to set the affinity for a virtqueue.
>>
>> The api is best-effort, the affinity hint may not be set as expected due to
>> platform support, irq sharing or irq type. Currently, only pci method were
>> implemented and we set the affinity according to:
>>
>> - if device uses INTX, we just ignore the request
>> - if device has per vq vector, we force the affinity hint
>> - if the virtqueues share MSI, make the affinity OR over all affinities
>> requested
>>
>> Signed-off-by: Jason Wang <jasowang@redhat.com>
>
> It looks like both I and Jason will need these patches during the 3.7
> merge window, and from different trees (net-next vs. scsi). How do we
> synchronize?
Get one of them to promise not to rebase, merge it, and base your
patches on top of the merge.
--
error compiling committee.c: too many arguments to function
^ permalink raw reply
* [Question]About KVM network zero-copy feature!
From: Peter Huang(Peng) @ 2012-08-10 3:34 UTC (permalink / raw)
To: kvm, avi, mtosatti, mst, virtualization, netdev
Hi,All
I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
about RX zero-copy?
For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?
Thanks!
^ permalink raw reply
* Re: [PATCH v6 1/3] mm: introduce compaction and migration for virtio ballooned pages
From: Mel Gorman @ 2012-08-10 12:46 UTC (permalink / raw)
To: Rafael Aquini
Cc: Rik van Riel, Konrad Rzeszutek Wilk, Michael S. Tsirkin,
linux-kernel, virtualization, linux-mm, Andi Kleen, Minchan Kim,
Andrew Morton
In-Reply-To: <20120809151218.GB2719@t510.redhat.com>
On Thu, Aug 09, 2012 at 12:12:19PM -0300, Rafael Aquini wrote:
> On Thu, Aug 09, 2012 at 11:48:36AM -0300, Rafael Aquini wrote:
> > Sure!
> > what do you think of:
> >
> > +/* putback_lru_page() counterpart for a ballooned page */
> > +void putback_balloon_page(struct page *page)
> > +{
> > + lock_page(page);
> > + if (!WARN_ON(!movable_balloon_page(page))) {
> > + __putback_balloon_page(page);
> > + put_page(page);
> > + }
> > + unlock_page(page);
> > +}
> >
> Or perhaps
>
> +/* putback_lru_page() counterpart for a ballooned page */
> +void putback_balloon_page(struct page *page)
> +{
> + if (!WARN_ON(!movable_balloon_page(page))) {
> + lock_page(page);
> + __putback_balloon_page(page);
> + put_page(page);
> + unlock_page(page);
> + }
> +}
That should be fine. I find the WARN_ON construct odd to read but only
because it's unusual. It is more typical to see something like
if (WARN_ON(!movable_balooon_page(page)))
return;
lock_page(page);
__putback_balloon_page(page);
put_page(page);
unlock_page(page);
but either works. Do not forget to update the caller of course.
Thanks.
--
Mel Gorman
SUSE Labs
^ permalink raw reply
* [PATCH v7 0/4] make balloon pages movable by compaction
From: Rafael Aquini @ 2012-08-10 17:55 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
Memory fragmentation introduced by ballooning might reduce significantly
the number of 2MB contiguous memory blocks that can be used within a guest,
thus imposing performance penalties associated with the reduced number of
transparent huge pages that could be used by the guest workload.
This patch-set follows the main idea discussed at 2012 LSFMMS session:
"Ballooning for transparent huge pages" -- http://lwn.net/Articles/490114/
to introduce the required changes to the virtio_balloon driver, as well as
the changes to the core compaction & migration bits, in order to make those
subsystems aware of ballooned pages and allow memory balloon pages become
movable within a guest, thus avoiding the aforementioned fragmentation issue
Rafael Aquini (4):
mm: introduce compaction and migration for virtio ballooned pages
virtio_balloon: introduce migration primitives to balloon pages
mm: introduce putback_movable_pages()
mm: add vm event counters for balloon pages compaction
drivers/virtio/virtio_balloon.c | 139 +++++++++++++++++++++++++++++++++++++---
include/linux/migrate.h | 2 +
include/linux/mm.h | 17 +++++
include/linux/virtio_balloon.h | 4 ++
include/linux/vm_event_item.h | 8 ++-
mm/compaction.c | 131 +++++++++++++++++++++++++++++++------
mm/migrate.c | 51 ++++++++++++++-
mm/page_alloc.c | 2 +-
mm/vmstat.c | 10 ++-
9 files changed, 331 insertions(+), 33 deletions(-)
Change log:
v7:
* fix a potential page leak case at 'putback_balloon_page' (Mel);
* adjust vm-events-counter patch and remove its drop-on-merge message (Rik);
* add 'putback_movable_pages' to avoid hacks on 'putback_lru_pages' (Minchan);
v6:
* rename 'is_balloon_page()' to 'movable_balloon_page()' (Rik);
v5:
* address Andrew Morton's review comments on the patch series;
* address a couple extra nitpick suggestions on PATCH 01 (Minchan);
v4:
* address Rusty Russel's review comments on PATCH 02;
* re-base virtio_balloon patch on 9c378abc5c0c6fc8e3acf5968924d274503819b3;
V3:
* address reviewers nitpick suggestions on PATCH 01 (Mel, Minchan);
V2:
* address Mel Gorman's review comments on PATCH 01;
Preliminary test results:
(2 VCPU 2048mB RAM KVM guest running 3.6.0_rc1+ -- after a reboot)
* 64mB balloon:
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 0
compact_pages_moved 0
compact_pagemigrate_failed 0
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_isolated 0
compact_balloon_migrated 0
compact_balloon_returned 0
compact_balloon_released 0
[root@localhost ~]#
[root@localhost ~]# for i in $(seq 1 6); do echo 1 > /proc/sys/vm/compact_memory & done &>/dev/null
[1] Done echo 1 > /proc/sys/vm/compact_memory
[2] Done echo 1 > /proc/sys/vm/compact_memory
[3] Done echo 1 > /proc/sys/vm/compact_memory
[4] Done echo 1 > /proc/sys/vm/compact_memory
[5]- Done echo 1 > /proc/sys/vm/compact_memory
[6]+ Done echo 1 > /proc/sys/vm/compact_memory
[root@localhost ~]#
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 6579
compact_pages_moved 50114
compact_pagemigrate_failed 111
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_isolated 18361
compact_balloon_migrated 18306
compact_balloon_returned 55
compact_balloon_released 18306
* 128 mB balloon:
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 0
compact_pages_moved 0
compact_pagemigrate_failed 0
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_isolated 0
compact_balloon_migrated 0
compact_balloon_returned 0
compact_balloon_released 0
[root@localhost ~]#
[root@localhost ~]# for i in $(seq 1 6); do echo 1 > /proc/sys/vm/compact_memory & done &>/dev/null
[1] Done echo 1 > /proc/sys/vm/compact_memory
[2] Done echo 1 > /proc/sys/vm/compact_memory
[3] Done echo 1 > /proc/sys/vm/compact_memory
[4] Done echo 1 > /proc/sys/vm/compact_memory
[5]- Done echo 1 > /proc/sys/vm/compact_memory
[6]+ Done echo 1 > /proc/sys/vm/compact_memory
[root@localhost ~]#
[root@localhost ~]# awk '/compact/ {print}' /proc/vmstat
compact_blocks_moved 6789
compact_pages_moved 64479
compact_pagemigrate_failed 127
compact_stall 0
compact_fail 0
compact_success 0
compact_balloon_isolated 33937
compact_balloon_migrated 33869
compact_balloon_returned 68
compact_balloon_released 33869
--
1.7.11.2
^ permalink raw reply
* [PATCH v7 1/4] mm: introduce compaction and migration for virtio ballooned pages
From: Rafael Aquini @ 2012-08-10 17:55 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
In-Reply-To: <cover.1344619987.git.aquini@redhat.com>
Memory fragmentation introduced by ballooning might reduce significantly
the number of 2MB contiguous memory blocks that can be used within a guest,
thus imposing performance penalties associated with the reduced number of
transparent huge pages that could be used by the guest workload.
This patch introduces the helper functions as well as the necessary changes
to teach compaction and migration bits how to cope with pages which are
part of a guest memory balloon, in order to make them movable by memory
compaction procedures.
Signed-off-by: Rafael Aquini <aquini@redhat.com>
---
include/linux/mm.h | 17 ++++++++
mm/compaction.c | 125 +++++++++++++++++++++++++++++++++++++++++++++--------
mm/migrate.c | 30 ++++++++++++-
3 files changed, 152 insertions(+), 20 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 311be90..56cc553 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1662,5 +1662,22 @@ static inline unsigned int debug_guardpage_minorder(void) { return 0; }
static inline bool page_is_guard(struct page *page) { return false; }
#endif /* CONFIG_DEBUG_PAGEALLOC */
+#if (defined(CONFIG_VIRTIO_BALLOON) || \
+ defined(CONFIG_VIRTIO_BALLOON_MODULE)) && defined(CONFIG_COMPACTION)
+extern bool isolate_balloon_page(struct page *);
+extern void putback_balloon_page(struct page *);
+extern struct address_space *balloon_mapping;
+
+static inline bool movable_balloon_page(struct page *page)
+{
+ return (page->mapping && page->mapping == balloon_mapping);
+}
+
+#else
+static inline bool isolate_balloon_page(struct page *page) { return false; }
+static inline void putback_balloon_page(struct page *page) { return false; }
+static inline bool movable_balloon_page(struct page *page) { return false; }
+#endif /* (VIRTIO_BALLOON || VIRTIO_BALLOON_MODULE) && CONFIG_COMPACTION */
+
#endif /* __KERNEL__ */
#endif /* _LINUX_MM_H */
diff --git a/mm/compaction.c b/mm/compaction.c
index e78cb96..e4e871b 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -14,6 +14,7 @@
#include <linux/backing-dev.h>
#include <linux/sysctl.h>
#include <linux/sysfs.h>
+#include <linux/export.h>
#include "internal.h"
#if defined CONFIG_COMPACTION || defined CONFIG_CMA
@@ -21,6 +22,84 @@
#define CREATE_TRACE_POINTS
#include <trace/events/compaction.h>
+#if defined(CONFIG_VIRTIO_BALLOON) || defined(CONFIG_VIRTIO_BALLOON_MODULE)
+/*
+ * Balloon pages special page->mapping.
+ * Users must properly allocate and initialize an instance of balloon_mapping,
+ * and set it as the page->mapping for balloon enlisted page instances.
+ * There is no need on utilizing struct address_space locking schemes for
+ * balloon_mapping as, once it gets initialized at balloon driver, it will
+ * remain just like a static reference that helps us on identifying a guest
+ * ballooned page by its mapping, as well as it will keep the 'a_ops' callback
+ * pointers to the functions that will execute the balloon page mobility tasks.
+ *
+ * address_space_operations necessary methods for ballooned pages:
+ * .migratepage - used to perform balloon's page migration (as is)
+ * .invalidatepage - used to isolate a page from balloon's page list
+ * .freepage - used to reinsert an isolated page to balloon's page list
+ */
+struct address_space *balloon_mapping;
+EXPORT_SYMBOL_GPL(balloon_mapping);
+
+static inline void __isolate_balloon_page(struct page *page)
+{
+ page->mapping->a_ops->invalidatepage(page, 0);
+}
+
+static inline void __putback_balloon_page(struct page *page)
+{
+ page->mapping->a_ops->freepage(page);
+}
+
+/* __isolate_lru_page() counterpart for a ballooned page */
+bool isolate_balloon_page(struct page *page)
+{
+ if (WARN_ON(!movable_balloon_page(page)))
+ return false;
+
+ if (likely(get_page_unless_zero(page))) {
+ /*
+ * As balloon pages are not isolated from LRU lists, concurrent
+ * compaction threads can race against page migration functions
+ * move_to_new_page() & __unmap_and_move().
+ * In order to avoid having an already isolated balloon page
+ * being (wrongly) re-isolated while it is under migration,
+ * lets be sure we have the page lock before proceeding with
+ * the balloon page isolation steps.
+ */
+ if (likely(trylock_page(page))) {
+ /*
+ * A ballooned page, by default, has just one refcount.
+ * Prevent concurrent compaction threads from isolating
+ * an already isolated balloon page.
+ */
+ if (movable_balloon_page(page) &&
+ (page_count(page) == 2)) {
+ __isolate_balloon_page(page);
+ unlock_page(page);
+ return true;
+ }
+ unlock_page(page);
+ }
+ /* Drop refcount taken for this already isolated page */
+ put_page(page);
+ }
+ return false;
+}
+
+/* putback_lru_page() counterpart for a ballooned page */
+void putback_balloon_page(struct page *page)
+{
+ if (WARN_ON(!movable_balloon_page(page)))
+ return;
+
+ lock_page(page);
+ __putback_balloon_page(page);
+ put_page(page);
+ unlock_page(page);
+}
+#endif /* CONFIG_VIRTIO_BALLOON || CONFIG_VIRTIO_BALLOON_MODULE */
+
static unsigned long release_freepages(struct list_head *freelist)
{
struct page *page, *next;
@@ -312,32 +391,40 @@ isolate_migratepages_range(struct zone *zone, struct compact_control *cc,
continue;
}
- if (!PageLRU(page))
- continue;
-
/*
- * PageLRU is set, and lru_lock excludes isolation,
- * splitting and collapsing (collapsing has already
- * happened if PageLRU is set).
+ * It is possible to migrate LRU pages and balloon pages.
+ * Skip any other type of page.
*/
- if (PageTransHuge(page)) {
- low_pfn += (1 << compound_order(page)) - 1;
- continue;
- }
+ if (PageLRU(page)) {
+ /*
+ * PageLRU is set, and lru_lock excludes isolation,
+ * splitting and collapsing (collapsing has already
+ * happened if PageLRU is set).
+ */
+ if (PageTransHuge(page)) {
+ low_pfn += (1 << compound_order(page)) - 1;
+ continue;
+ }
- if (!cc->sync)
- mode |= ISOLATE_ASYNC_MIGRATE;
+ if (!cc->sync)
+ mode |= ISOLATE_ASYNC_MIGRATE;
- lruvec = mem_cgroup_page_lruvec(page, zone);
+ lruvec = mem_cgroup_page_lruvec(page, zone);
- /* Try isolate the page */
- if (__isolate_lru_page(page, mode) != 0)
- continue;
+ /* Try isolate the page */
+ if (__isolate_lru_page(page, mode) != 0)
+ continue;
+
+ VM_BUG_ON(PageTransCompound(page));
- VM_BUG_ON(PageTransCompound(page));
+ /* Successfully isolated */
+ del_page_from_lru_list(page, lruvec, page_lru(page));
+ } else if (unlikely(movable_balloon_page(page))) {
+ if (!isolate_balloon_page(page))
+ continue;
+ } else
+ continue;
- /* Successfully isolated */
- del_page_from_lru_list(page, lruvec, page_lru(page));
list_add(&page->lru, migratelist);
cc->nr_migratepages++;
nr_isolated++;
diff --git a/mm/migrate.c b/mm/migrate.c
index 77ed2d7..80f22bb 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -79,7 +79,10 @@ void putback_lru_pages(struct list_head *l)
list_del(&page->lru);
dec_zone_page_state(page, NR_ISOLATED_ANON +
page_is_file_cache(page));
- putback_lru_page(page);
+ if (unlikely(movable_balloon_page(page)))
+ putback_balloon_page(page);
+ else
+ putback_lru_page(page);
}
}
@@ -778,6 +781,17 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
}
}
+ if (unlikely(movable_balloon_page(page))) {
+ /*
+ * A ballooned page does not need any special attention from
+ * physical to virtual reverse mapping procedures.
+ * Skip any attempt to unmap PTEs or to remap swap cache,
+ * in order to avoid burning cycles at rmap level.
+ */
+ remap_swapcache = 0;
+ goto skip_unmap;
+ }
+
/*
* Corner case handling:
* 1. When a new swap-cache page is read into, it is added to the LRU
@@ -846,6 +860,20 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
goto out;
rc = __unmap_and_move(page, newpage, force, offlining, mode);
+
+ if (unlikely(movable_balloon_page(newpage))) {
+ /*
+ * A ballooned page has been migrated already. Now, it is the
+ * time to wrap-up counters, handle the old page back to Buddy
+ * and return.
+ */
+ list_del(&page->lru);
+ dec_zone_page_state(page, NR_ISOLATED_ANON +
+ page_is_file_cache(page));
+ put_page(page);
+ __free_page(page);
+ return rc;
+ }
out:
if (rc != -EAGAIN) {
/*
--
1.7.11.2
^ permalink raw reply related
* [PATCH v7 2/4] virtio_balloon: introduce migration primitives to balloon pages
From: Rafael Aquini @ 2012-08-10 17:55 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
In-Reply-To: <cover.1344619987.git.aquini@redhat.com>
Memory fragmentation introduced by ballooning might reduce significantly
the number of 2MB contiguous memory blocks that can be used within a guest,
thus imposing performance penalties associated with the reduced number of
transparent huge pages that could be used by the guest workload.
Besides making balloon pages movable at allocation time and introducing
the necessary primitives to perform balloon page migration/compaction,
this patch also introduces the following locking scheme to provide the
proper synchronization and protection for struct virtio_balloon elements
against concurrent accesses due to parallel operations introduced by
memory compaction / page migration.
- balloon_lock (mutex) : synchronizes the access demand to elements of
struct virtio_balloon and its queue operations;
- pages_lock (spinlock): special protection to balloon pages list against
concurrent list handling operations;
Signed-off-by: Rafael Aquini <aquini@redhat.com>
---
drivers/virtio/virtio_balloon.c | 138 +++++++++++++++++++++++++++++++++++++---
include/linux/virtio_balloon.h | 4 ++
2 files changed, 134 insertions(+), 8 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 0908e60..7c937a0 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -27,6 +27,7 @@
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/module.h>
+#include <linux/fs.h>
/*
* Balloon device works in 4K page units. So each page is pointed to by
@@ -35,6 +36,12 @@
*/
#define VIRTIO_BALLOON_PAGES_PER_PAGE (PAGE_SIZE >> VIRTIO_BALLOON_PFN_SHIFT)
+/* Synchronizes accesses/updates to the struct virtio_balloon elements */
+DEFINE_MUTEX(balloon_lock);
+
+/* Protects 'virtio_balloon->pages' list against concurrent handling */
+DEFINE_SPINLOCK(pages_lock);
+
struct virtio_balloon
{
struct virtio_device *vdev;
@@ -51,6 +58,7 @@ struct virtio_balloon
/* Number of balloon pages we've told the Host we're not using. */
unsigned int num_pages;
+
/*
* The pages we've told the Host we're not using.
* Each page on this list adds VIRTIO_BALLOON_PAGES_PER_PAGE
@@ -125,10 +133,12 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
/* We can only do one array worth at a time. */
num = min(num, ARRAY_SIZE(vb->pfns));
+ mutex_lock(&balloon_lock);
for (vb->num_pfns = 0; vb->num_pfns < num;
vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
- struct page *page = alloc_page(GFP_HIGHUSER | __GFP_NORETRY |
- __GFP_NOMEMALLOC | __GFP_NOWARN);
+ struct page *page = alloc_page(GFP_HIGHUSER_MOVABLE |
+ __GFP_NORETRY | __GFP_NOWARN |
+ __GFP_NOMEMALLOC);
if (!page) {
if (printk_ratelimit())
dev_printk(KERN_INFO, &vb->vdev->dev,
@@ -141,7 +151,10 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
set_page_pfns(vb->pfns + vb->num_pfns, page);
vb->num_pages += VIRTIO_BALLOON_PAGES_PER_PAGE;
totalram_pages--;
+ spin_lock(&pages_lock);
list_add(&page->lru, &vb->pages);
+ page->mapping = balloon_mapping;
+ spin_unlock(&pages_lock);
}
/* Didn't get any? Oh well. */
@@ -149,6 +162,7 @@ static void fill_balloon(struct virtio_balloon *vb, size_t num)
return;
tell_host(vb, vb->inflate_vq);
+ mutex_unlock(&balloon_lock);
}
static void release_pages_by_pfn(const u32 pfns[], unsigned int num)
@@ -169,10 +183,22 @@ static void leak_balloon(struct virtio_balloon *vb, size_t num)
/* We can only do one array worth at a time. */
num = min(num, ARRAY_SIZE(vb->pfns));
+ mutex_lock(&balloon_lock);
for (vb->num_pfns = 0; vb->num_pfns < num;
vb->num_pfns += VIRTIO_BALLOON_PAGES_PER_PAGE) {
+ /*
+ * We can race against virtballoon_isolatepage() and end up
+ * stumbling across a _temporarily_ empty 'pages' list.
+ */
+ spin_lock(&pages_lock);
+ if (unlikely(list_empty(&vb->pages))) {
+ spin_unlock(&pages_lock);
+ break;
+ }
page = list_first_entry(&vb->pages, struct page, lru);
+ page->mapping = NULL;
list_del(&page->lru);
+ spin_unlock(&pages_lock);
set_page_pfns(vb->pfns + vb->num_pfns, page);
vb->num_pages -= VIRTIO_BALLOON_PAGES_PER_PAGE;
}
@@ -182,8 +208,11 @@ static void leak_balloon(struct virtio_balloon *vb, size_t num)
* virtio_has_feature(vdev, VIRTIO_BALLOON_F_MUST_TELL_HOST);
* is true, we *have* to do it in this order
*/
- tell_host(vb, vb->deflate_vq);
- release_pages_by_pfn(vb->pfns, vb->num_pfns);
+ if (vb->num_pfns > 0) {
+ tell_host(vb, vb->deflate_vq);
+ release_pages_by_pfn(vb->pfns, vb->num_pfns);
+ }
+ mutex_unlock(&balloon_lock);
}
static inline void update_stat(struct virtio_balloon *vb, int idx,
@@ -239,6 +268,7 @@ static void stats_handle_request(struct virtio_balloon *vb)
struct scatterlist sg;
unsigned int len;
+ mutex_lock(&balloon_lock);
vb->need_stats_update = 0;
update_balloon_stats(vb);
@@ -249,6 +279,7 @@ static void stats_handle_request(struct virtio_balloon *vb)
if (virtqueue_add_buf(vq, &sg, 1, 0, vb, GFP_KERNEL) < 0)
BUG();
virtqueue_kick(vq);
+ mutex_unlock(&balloon_lock);
}
static void virtballoon_changed(struct virtio_device *vdev)
@@ -261,22 +292,27 @@ static void virtballoon_changed(struct virtio_device *vdev)
static inline s64 towards_target(struct virtio_balloon *vb)
{
__le32 v;
- s64 target;
+ s64 target, actual;
+ mutex_lock(&balloon_lock);
+ actual = vb->num_pages;
vb->vdev->config->get(vb->vdev,
offsetof(struct virtio_balloon_config, num_pages),
&v, sizeof(v));
target = le32_to_cpu(v);
- return target - vb->num_pages;
+ mutex_unlock(&balloon_lock);
+ return target - actual;
}
static void update_balloon_size(struct virtio_balloon *vb)
{
- __le32 actual = cpu_to_le32(vb->num_pages);
-
+ __le32 actual;
+ mutex_lock(&balloon_lock);
+ actual = cpu_to_le32(vb->num_pages);
vb->vdev->config->set(vb->vdev,
offsetof(struct virtio_balloon_config, actual),
&actual, sizeof(actual));
+ mutex_unlock(&balloon_lock);
}
static int balloon(void *_vballoon)
@@ -339,6 +375,76 @@ static int init_vqs(struct virtio_balloon *vb)
return 0;
}
+/*
+ * '*vb_ptr' allows virtballoon_migratepage() & virtballoon_putbackpage() to
+ * access pertinent elements from struct virtio_balloon
+ */
+struct virtio_balloon *vb_ptr;
+
+/*
+ * Populate balloon_mapping->a_ops->migratepage method to perform the balloon
+ * page migration task.
+ *
+ * After a ballooned page gets isolated by compaction procedures, this is the
+ * function that performs the page migration on behalf of move_to_new_page(),
+ * when the last calls (page)->mapping->a_ops->migratepage.
+ *
+ * Page migration for virtio balloon is done in a simple swap fashion which
+ * follows these two steps:
+ * 1) insert newpage into vb->pages list and update the host about it;
+ * 2) update the host about the removed old page from vb->pages list;
+ */
+int virtballoon_migratepage(struct address_space *mapping,
+ struct page *newpage, struct page *page, enum migrate_mode mode)
+{
+ mutex_lock(&balloon_lock);
+
+ /* balloon's page migration 1st step */
+ vb_ptr->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
+ spin_lock(&pages_lock);
+ list_add(&newpage->lru, &vb_ptr->pages);
+ spin_unlock(&pages_lock);
+ set_page_pfns(vb_ptr->pfns, newpage);
+ tell_host(vb_ptr, vb_ptr->inflate_vq);
+
+ /* balloon's page migration 2nd step */
+ vb_ptr->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
+ set_page_pfns(vb_ptr->pfns, page);
+ tell_host(vb_ptr, vb_ptr->deflate_vq);
+
+ mutex_unlock(&balloon_lock);
+
+ return 0;
+}
+
+/*
+ * Populate balloon_mapping->a_ops->invalidatepage method to help compaction on
+ * isolating a page from the balloon page list.
+ */
+void virtballoon_isolatepage(struct page *page, unsigned long mode)
+{
+ spin_lock(&pages_lock);
+ list_del(&page->lru);
+ spin_unlock(&pages_lock);
+}
+
+/*
+ * Populate balloon_mapping->a_ops->freepage method to help compaction on
+ * re-inserting an isolated page into the balloon page list.
+ */
+void virtballoon_putbackpage(struct page *page)
+{
+ spin_lock(&pages_lock);
+ list_add(&page->lru, &vb_ptr->pages);
+ spin_unlock(&pages_lock);
+}
+
+static const struct address_space_operations virtio_balloon_aops = {
+ .migratepage = virtballoon_migratepage,
+ .invalidatepage = virtballoon_isolatepage,
+ .freepage = virtballoon_putbackpage,
+};
+
static int virtballoon_probe(struct virtio_device *vdev)
{
struct virtio_balloon *vb;
@@ -351,11 +457,25 @@ static int virtballoon_probe(struct virtio_device *vdev)
}
INIT_LIST_HEAD(&vb->pages);
+
vb->num_pages = 0;
init_waitqueue_head(&vb->config_change);
init_waitqueue_head(&vb->acked);
vb->vdev = vdev;
vb->need_stats_update = 0;
+ vb_ptr = vb;
+
+ /* Init the ballooned page->mapping special balloon_mapping */
+ balloon_mapping = kmalloc(sizeof(*balloon_mapping), GFP_KERNEL);
+ if (!balloon_mapping) {
+ err = -ENOMEM;
+ goto out_free_vb;
+ }
+
+ INIT_RADIX_TREE(&balloon_mapping->page_tree, GFP_ATOMIC | __GFP_NOWARN);
+ INIT_LIST_HEAD(&balloon_mapping->i_mmap_nonlinear);
+ spin_lock_init(&balloon_mapping->tree_lock);
+ balloon_mapping->a_ops = &virtio_balloon_aops;
err = init_vqs(vb);
if (err)
@@ -373,6 +493,7 @@ out_del_vqs:
vdev->config->del_vqs(vdev);
out_free_vb:
kfree(vb);
+ kfree(balloon_mapping);
out:
return err;
}
@@ -397,6 +518,7 @@ static void __devexit virtballoon_remove(struct virtio_device *vdev)
kthread_stop(vb->thread);
remove_common(vb);
kfree(vb);
+ kfree(balloon_mapping);
}
#ifdef CONFIG_PM
diff --git a/include/linux/virtio_balloon.h b/include/linux/virtio_balloon.h
index 652dc8b..930f1b7 100644
--- a/include/linux/virtio_balloon.h
+++ b/include/linux/virtio_balloon.h
@@ -56,4 +56,8 @@ struct virtio_balloon_stat {
u64 val;
} __attribute__((packed));
+#if !defined(CONFIG_COMPACTION)
+struct address_space *balloon_mapping;
+#endif
+
#endif /* _LINUX_VIRTIO_BALLOON_H */
--
1.7.11.2
^ permalink raw reply related
* [PATCH v7 3/4] mm: introduce putback_movable_pages()
From: Rafael Aquini @ 2012-08-10 17:55 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
In-Reply-To: <cover.1344619987.git.aquini@redhat.com>
The PATCH "mm: introduce compaction and migration for virtio ballooned pages"
hacks around putback_lru_pages() in order to allow ballooned pages to be
re-inserted on balloon page list as if a ballooned page was like a LRU page.
As ballooned pages are not legitimate LRU pages, this patch introduces
putback_movable_pages() to properly cope with cases where the isolated
pageset contains ballooned pages and LRU pages, thus fixing the mentioned
inelegant hack around putback_lru_pages().
Signed-off-by: Rafael Aquini <aquini@redhat.com>
---
include/linux/migrate.h | 2 ++
mm/compaction.c | 4 ++--
mm/migrate.c | 20 ++++++++++++++++++++
mm/page_alloc.c | 2 +-
4 files changed, 25 insertions(+), 3 deletions(-)
diff --git a/include/linux/migrate.h b/include/linux/migrate.h
index ce7e667..ff103a1 100644
--- a/include/linux/migrate.h
+++ b/include/linux/migrate.h
@@ -10,6 +10,7 @@ typedef struct page *new_page_t(struct page *, unsigned long private, int **);
#ifdef CONFIG_MIGRATION
extern void putback_lru_pages(struct list_head *l);
+extern void putback_movable_pages(struct list_head *l);
extern int migrate_page(struct address_space *,
struct page *, struct page *, enum migrate_mode);
extern int migrate_pages(struct list_head *l, new_page_t x,
@@ -33,6 +34,7 @@ extern int migrate_huge_page_move_mapping(struct address_space *mapping,
#else
static inline void putback_lru_pages(struct list_head *l) {}
+static inline void putback_movable_pages(struct list_head *l) {}
static inline int migrate_pages(struct list_head *l, new_page_t x,
unsigned long private, bool offlining,
enum migrate_mode mode) { return -ENOSYS; }
diff --git a/mm/compaction.c b/mm/compaction.c
index e4e871b..8567bb8 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -837,9 +837,9 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
trace_mm_compaction_migratepages(nr_migrate - nr_remaining,
nr_remaining);
- /* Release LRU pages not migrated */
+ /* Release isolated pages not migrated */
if (err) {
- putback_lru_pages(&cc->migratepages);
+ putback_movable_pages(&cc->migratepages);
cc->nr_migratepages = 0;
if (err == -ENOMEM) {
ret = COMPACT_PARTIAL;
diff --git a/mm/migrate.c b/mm/migrate.c
index 80f22bb..1165134 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -79,6 +79,26 @@ void putback_lru_pages(struct list_head *l)
list_del(&page->lru);
dec_zone_page_state(page, NR_ISOLATED_ANON +
page_is_file_cache(page));
+ putback_lru_page(page);
+ }
+}
+
+/*
+ * Put previously isolated pages back onto the appropriated lists
+ * from where they were once taken off for compaction/migration.
+ *
+ * This function shall be used instead of putback_lru_pages(),
+ * whenever the isolated pageset has been built by isolate_migratepages_range()
+ */
+void putback_movable_pages(struct list_head *l)
+{
+ struct page *page;
+ struct page *page2;
+
+ list_for_each_entry_safe(page, page2, l, lru) {
+ list_del(&page->lru);
+ dec_zone_page_state(page, NR_ISOLATED_ANON +
+ page_is_file_cache(page));
if (unlikely(movable_balloon_page(page)))
putback_balloon_page(page);
else
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 009ac28..78b7663 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -5669,7 +5669,7 @@ static int __alloc_contig_migrate_range(unsigned long start, unsigned long end)
0, false, MIGRATE_SYNC);
}
- putback_lru_pages(&cc.migratepages);
+ putback_movable_pages(&cc.migratepages);
return ret > 0 ? 0 : ret;
}
--
1.7.11.2
^ permalink raw reply related
* [PATCH v7 4/4] mm: add vm event counters for balloon pages compaction
From: Rafael Aquini @ 2012-08-10 17:55 UTC (permalink / raw)
To: linux-mm
Cc: Rik van Riel, Rafael Aquini, Konrad Rzeszutek Wilk,
Michael S. Tsirkin, linux-kernel, virtualization, Minchan Kim,
Andi Kleen, Andrew Morton
In-Reply-To: <cover.1344619987.git.aquini@redhat.com>
This patch introduces a new set of vm event counters to keep track of
ballooned pages compaction activity.
Signed-off-by: Rafael Aquini <aquini@redhat.com>
---
drivers/virtio/virtio_balloon.c | 1 +
include/linux/vm_event_item.h | 8 +++++++-
mm/compaction.c | 2 ++
mm/migrate.c | 1 +
mm/vmstat.c | 10 +++++++++-
5 files changed, 20 insertions(+), 2 deletions(-)
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index 7c937a0..b8f7ea5 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -414,6 +414,7 @@ int virtballoon_migratepage(struct address_space *mapping,
mutex_unlock(&balloon_lock);
+ count_vm_event(COMPACTBALLOONMIGRATED);
return 0;
}
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index 57f7b10..b1841a2 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -41,7 +41,13 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
#ifdef CONFIG_COMPACTION
COMPACTBLOCKS, COMPACTPAGES, COMPACTPAGEFAILED,
COMPACTSTALL, COMPACTFAIL, COMPACTSUCCESS,
-#endif
+#if defined(CONFIG_VIRTIO_BALLOON) || defined(CONFIG_VIRTIO_BALLOON_MODULE)
+ COMPACTBALLOONISOLATED, /* isolated from balloon pagelist */
+ COMPACTBALLOONMIGRATED, /* balloon page sucessfully migrated */
+ COMPACTBALLOONRETURNED, /* putback to pagelist, not-migrated */
+ COMPACTBALLOONRELEASED, /* old-page released after migration */
+#endif /* CONFIG_VIRTIO_BALLOON || CONFIG_VIRTIO_BALLOON_MODULE */
+#endif /* CONFIG_COMPACTION */
#ifdef CONFIG_HUGETLB_PAGE
HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,
#endif
diff --git a/mm/compaction.c b/mm/compaction.c
index 8567bb8..ff0f9ac 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -77,6 +77,7 @@ bool isolate_balloon_page(struct page *page)
(page_count(page) == 2)) {
__isolate_balloon_page(page);
unlock_page(page);
+ count_vm_event(COMPACTBALLOONISOLATED);
return true;
}
unlock_page(page);
@@ -97,6 +98,7 @@ void putback_balloon_page(struct page *page)
__putback_balloon_page(page);
put_page(page);
unlock_page(page);
+ count_vm_event(COMPACTBALLOONRETURNED);
}
#endif /* CONFIG_VIRTIO_BALLOON || CONFIG_VIRTIO_BALLOON_MODULE */
diff --git a/mm/migrate.c b/mm/migrate.c
index 1165134..024566f 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -892,6 +892,7 @@ static int unmap_and_move(new_page_t get_new_page, unsigned long private,
page_is_file_cache(page));
put_page(page);
__free_page(page);
+ count_vm_event(COMPACTBALLOONRELEASED);
return rc;
}
out:
diff --git a/mm/vmstat.c b/mm/vmstat.c
index df7a674..ad5c4f1 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -768,7 +768,15 @@ const char * const vmstat_text[] = {
"compact_stall",
"compact_fail",
"compact_success",
-#endif
+
+#if defined(CONFIG_VIRTIO_BALLOON) || defined(CONFIG_VIRTIO_BALLOON_MODULE)
+ "compact_balloon_isolated",
+ "compact_balloon_migrated",
+ "compact_balloon_returned",
+ "compact_balloon_released",
+#endif /* CONFIG_VIRTIO_BALLOON || CONFIG_VIRTIO_BALLOON_MODULE */
+
+#endif /* CONFIG_COMPACTION */
#ifdef CONFIG_HUGETLB_PAGE
"htlb_buddy_alloc_success",
--
1.7.11.2
^ permalink raw reply related
* virtio-scsi <-> vhost multi lun/adapter performance results with 3.6-rc0
From: Nicholas A. Bellinger @ 2012-08-10 23:23 UTC (permalink / raw)
To: target-devel, linux-scsi, LKML
Cc: kvm-devel, Michael S. Tsirkin, qemu-devel, lf-virt,
James Bottomley, Anthony Liguori, Jens Axboe, Paolo Bonzini,
Linus Torvalds, Christoph Hellwig
[-- Attachment #1: Type: text/plain, Size: 15362 bytes --]
Hi folks,
The following are initial virtio-scsi + target vhost benchmark results
using multiple target LUNs per vhost and multiple virtio PCI adapters to
scale the total number of virtio-scsi LUNs into a single KVM guest.
The test setup is currently using 4x SCSI LUNs per vhost WWPN, with 8x
virtio PCI adapters for a total of 32x 500MB ramdisk LUNs into a single
guest, along with each backend setting emulate_write_cache=1 to expose
WCE=1 via virtio-scsi to SCSI core.
Using a KVM guest with 32x vCPUs and 4G memory, the results for 4x
random I/O now look like:
workload | jobs | 25% write / 75% read | 75% write / 25% read
-----------------|------|----------------------|---------------------
1x rd_mcp LUN | 8 | ~155K IOPs | ~145K IOPs
16x rd_mcp LUNs | 16 | ~315K IOPs | ~305K IOPs
32x rd_mcp LUNs | 16 | ~425K IOPs | ~410K IOPs
The full fio randrw results for the six test cases are attached below.
Also, using a workload of fio numjobs > 16 currently makes performance
start to fall off pretty sharply regardless of the number of vCPUs..
So running a similar workload with loopback SCSI ports on bare-metal
produces ~1M random IOPs with 12x LUNs + numjobs=32. At numjobs=16 here
with vhost the 16x LUN configuration ends up being in the range of ~310K
IOPs for the current sweet spot..
Here is a more detailed breakdown of the test setup:
- host hardware:
*) Dual Xeon-E5-2687W (Romley-EP) 3.10 Ghz w/ 32x threads +
32 GB of DDR3 1600Mhz memory
- host kernel:
*) Using 3.6-rc0 from target-pending/for-linus
*) qemu vhost-scsi from nab's qemu-kvm.git/vhost-scsi on k.o
*) Set QEMU vCPU process affinity to dedicated cpus based on
'info cpus' (as recommended by Stefan)
target backstores + vhost configuration from rtsadmin/targetcli shell:
/> ls backstores/rd_mcp/
o- rd_mcp ................................................. [32 Storage Objects]
o- ramdisk0 .............................................. [ramdisk activated]
o- ramdisk1 .............................................. [ramdisk activated]
o- ramdisk10 ............................................. [ramdisk activated]
o- ramdisk11 ............................................. [ramdisk activated]
o- ramdisk12 ............................................. [ramdisk activated]
o- ramdisk13 ............................................. [ramdisk activated]
o- ramdisk14 ............................................. [ramdisk activated]
o- ramdisk15 ............................................. [ramdisk activated]
o- ramdisk16 ............................................. [ramdisk activated]
o- ramdisk17 ............................................. [ramdisk activated]
o- ramdisk18 ............................................. [ramdisk activated]
o- ramdisk19 ............................................. [ramdisk activated]
o- ramdisk2 .............................................. [ramdisk activated]
o- ramdisk20 ............................................. [ramdisk activated]
o- ramdisk21 ............................................. [ramdisk activated]
o- ramdisk22 ............................................. [ramdisk activated]
o- ramdisk23 ............................................. [ramdisk activated]
o- ramdisk24 ............................................. [ramdisk activated]
o- ramdisk25 ............................................. [ramdisk activated]
o- ramdisk26 ............................................. [ramdisk activated]
o- ramdisk27 ............................................. [ramdisk activated]
o- ramdisk28 ............................................. [ramdisk activated]
o- ramdisk29 ............................................. [ramdisk activated]
o- ramdisk3 .............................................. [ramdisk activated]
o- ramdisk30 ............................................. [ramdisk activated]
o- ramdisk31 ............................................. [ramdisk activated]
o- ramdisk4 .............................................. [ramdisk activated]
o- ramdisk5 .............................................. [ramdisk activated]
o- ramdisk6 .............................................. [ramdisk activated]
o- ramdisk7 .............................................. [ramdisk activated]
o- ramdisk8 .............................................. [ramdisk activated]
o- ramdisk9 .............................................. [ramdisk activated]
/> ls vhost/
o- vhost ........................................................... [8 Targets]
o- naa.60014053fd613910 ............................... [naa.600140539e23ee71]
| o- luns ........................................................... [4 LUNs]
| o- lun0 ...................................... [rd_mcp/ramdisk5 (ramdisk)]
| o- lun1 ..................................... [rd_mcp/ramdisk23 (ramdisk)]
| o- lun2 ..................................... [rd_mcp/ramdisk24 (ramdisk)]
| o- lun3 ..................................... [rd_mcp/ramdisk25 (ramdisk)]
o- naa.60014058fd33725f ............................... [naa.6001405b0dcc8c8f]
| o- luns ........................................................... [4 LUNs]
| o- lun0 ...................................... [rd_mcp/ramdisk2 (ramdisk)]
| o- lun1 ..................................... [rd_mcp/ramdisk14 (ramdisk)]
| o- lun2 ..................................... [rd_mcp/ramdisk15 (ramdisk)]
| o- lun3 ..................................... [rd_mcp/ramdisk16 (ramdisk)]
o- naa.60014059af47b6a0 ............................... [naa.600140567c5ac7f1]
| o- luns ........................................................... [4 LUNs]
| o- lun0 ...................................... [rd_mcp/ramdisk0 (ramdisk)]
| o- lun1 ...................................... [rd_mcp/ramdisk8 (ramdisk)]
| o- lun2 ...................................... [rd_mcp/ramdisk9 (ramdisk)]
| o- lun3 ..................................... [rd_mcp/ramdisk10 (ramdisk)]
o- naa.6001405dfae0c05b ............................... [naa.6001405ce8ccfc96]
| o- luns ........................................................... [4 LUNs]
| o- lun0 ...................................... [rd_mcp/ramdisk4 (ramdisk)]
| o- lun1 ..................................... [rd_mcp/ramdisk20 (ramdisk)]
| o- lun2 ..................................... [rd_mcp/ramdisk21 (ramdisk)]
| o- lun3 ..................................... [rd_mcp/ramdisk22 (ramdisk)]
o- naa.6001405e0c55744e ............................... [naa.600140569bdc4c76]
| o- luns ........................................................... [4 LUNs]
| o- lun0 ...................................... [rd_mcp/ramdisk3 (ramdisk)]
| o- lun1 ..................................... [rd_mcp/ramdisk17 (ramdisk)]
| o- lun2 ..................................... [rd_mcp/ramdisk18 (ramdisk)]
| o- lun3 ..................................... [rd_mcp/ramdisk19 (ramdisk)]
o- naa.6001405e6b23dd27 ............................... [naa.600140503c996b52]
| o- luns ........................................................... [4 LUNs]
| o- lun0 ...................................... [rd_mcp/ramdisk7 (ramdisk)]
| o- lun1 ..................................... [rd_mcp/ramdisk29 (ramdisk)]
| o- lun2 ..................................... [rd_mcp/ramdisk30 (ramdisk)]
| o- lun3 ..................................... [rd_mcp/ramdisk31 (ramdisk)]
o- naa.6001405e86ec6fbc ............................... [naa.6001405948737af9]
| o- luns ........................................................... [4 LUNs]
| o- lun0 ...................................... [rd_mcp/ramdisk6 (ramdisk)]
| o- lun1 ..................................... [rd_mcp/ramdisk26 (ramdisk)]
| o- lun2 ..................................... [rd_mcp/ramdisk27 (ramdisk)]
| o- lun3 ..................................... [rd_mcp/ramdisk28 (ramdisk)]
o- naa.6001405f2a1036cf ............................... [naa.6001405a44e2740d]
o- luns ........................................................... [4 LUNs]
o- lun0 ...................................... [rd_mcp/ramdisk1 (ramdisk)]
o- lun1 ..................................... [rd_mcp/ramdisk11 (ramdisk)]
o- lun2 ..................................... [rd_mcp/ramdisk12 (ramdisk)]
o- lun3 ..................................... [rd_mcp/ramdisk13 (ramdisk)]
- guest kernel:
*) 3.5.0-rc2 from target-pending/for-next-merge w/ virtio-scsi LUN scan
bugfix applied
*) Use nop scheduler for all virtio-scsi LUNs
*) Set virtio-queue IRQ affinity to dedicated vCPUs
QEMU cli opts:
./x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 32 -m 4096 -serial
file:/tmp/vhost-serial.txt
-hda /usr/src/debian_squeeze_amd64_standard-old.qcow2 -vhost-scsi
id=vhost-scsi0,wwpn=naa.60014059af47b6a0,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi0,event_idx=off -vhost-scsi
id=vhost-scsi1,wwpn=naa.6001405f2a1036cf,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi1,event_idx=off -vhost-scsi
id=vhost-scsi2,wwpn=naa.60014058fd33725f,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi2,event_idx=off -vhost-scsi
id=vhost-scsi3,wwpn=naa.6001405e0c55744e,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi3,event_idx=off -vhost-scsi
id=vhost-scsi4,wwpn=naa.6001405dfae0c05b,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi4,event_idx=off -vhost-scsi
id=vhost-scsi5,wwpn=naa.60014053fd613910,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi5,event_idx=off -vhost-scsi
id=vhost-scsi6,wwpn=naa.6001405e86ec6fbc,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi6,event_idx=off -vhost-scsi
id=vhost-scsi7,wwpn=naa.6001405e6b23dd27,tpgt=1 -device
virtio-scsi-pci,vhost-scsi=vhost-scsi7,event_idx=off
Using the following fio base options:
[randrw]
rw=randrw
rwmixwrite=25
rwmixread=75
ioengine=libaio
direct=1
size=100G
iodepth=64
iodepth_batch=4
iodepth_batch_complete=32
numjobs=32
blocksize=4k
filename=/dev/sdb
filename=/dev/sdc
filename=/dev/sdd
filename=/dev/sde
filename=/dev/sdf
filename=/dev/sdg
filename=/dev/sdh
filename=/dev/sdi
filename=/dev/sdj
filename=/dev/sdk
filename=/dev/sdl
filename=/dev/sdm
filename=/dev/sdn
filename=/dev/sdo
filename=/dev/sdp
filename=/dev/sdq
filename=/dev/sdr
filename=/dev/sds
filename=/dev/sdt
filename=/dev/sdu
filename=/dev/sdv
filename=/dev/sdw
filename=/dev/sdx
filename=/dev/sdy
filename=/dev/sdz
filename=/dev/sdaa
filename=/dev/sdab
filename=/dev/sdac
filename=/dev/sdad
filename=/dev/sdae
filename=/dev/sdaf
filename=/dev/sdag
Guest lsscsi output:
[0:0:0:0] disk ATA QEMU HARDDISK 1.1. /dev/sda
[1:0:0:0] cd/dvd QEMU QEMU DVD-ROM 1.1. /dev/sr0
[2:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdb
[2:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdc
[2:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdd
[2:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sde
[3:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdf
[3:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdg
[3:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdh
[3:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdi
[4:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdj
[4:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdk
[4:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdl
[4:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdm
[5:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdn
[5:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdo
[5:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdp
[5:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdq
[6:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdr
[6:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sds
[6:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdt
[6:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdu
[7:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdv
[7:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdw
[7:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdx
[7:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdy
[8:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdz
[8:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdaa
[8:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdab
[8:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdac
[9:0:0:0] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdad
[9:0:0:1] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdae
[9:0:0:2] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdaf
[9:0:0:3] disk LIO-ORG RAMDISK-MCP 4.0 /dev/sdag
and the relevant virtio PCI layout:
00:04.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 11
I/O ports at c040 [size=64]
Memory at febf1000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
00:05.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 10
I/O ports at c080 [size=64]
Memory at febf2000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
00:06.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 10
I/O ports at c0c0 [size=64]
Memory at febf3000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
00:07.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 11
I/O ports at c100 [size=64]
Memory at febf4000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
00:08.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 11
I/O ports at c140 [size=64]
Memory at febf5000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
00:09.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 10
I/O ports at c180 [size=64]
Memory at febf6000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
00:0a.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 10
I/O ports at c1c0 [size=64]
Memory at febf7000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
00:0b.0 SCSI storage controller: Red Hat, Inc Device 1004
Subsystem: Red Hat, Inc Device 0008
Flags: bus master, fast devsel, latency 0, IRQ 11
I/O ports at c200 [size=64]
Memory at febf8000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
Kernel driver in use: virtio-pci
[-- Attachment #2: fio-1x-rd_mcp-25-75-4k.txt --]
[-- Type: text/plain, Size: 9343 bytes --]
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
...
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
Starting 8 processes
randrw: (groupid=0, jobs=1): err= 0: pid=7155
read : io=384MB, bw=62,934KB/s, iops=15,733, runt= 6243msec
slat (usec): min=4, max=11,681, avg=167.00, stdev=448.35
clat (usec): min=3, max=19,552, avg=2217.48, stdev=1367.27
bw (KB/s) : min= 0, max=70143, per=1.27%, avg=6362.85, stdev=19236.22
write: io=128MB, bw=21,046KB/s, iops=5,261, runt= 6243msec
slat (usec): min=4, max=11,678, avg=164.73, stdev=442.99
clat (usec): min=11, max=19,552, avg=2229.92, stdev=1367.33
bw (KB/s) : min= 0, max=24448, per=2.03%, avg=3402.15, stdev=7878.23
cpu : usr=0.35%, sys=18.66%, ctx=6023, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98224/32848, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%, 100=0.04%
lat (usec): 250=0.42%, 500=2.84%, 750=3.37%, 1000=6.66%
lat (msec): 2=37.59%, 4=39.83%, 10=8.97%, 20=0.24%
randrw: (groupid=0, jobs=1): err= 0: pid=7156
read : io=385MB, bw=62,926KB/s, iops=15,731, runt= 6260msec
slat (usec): min=4, max=16,646, avg=166.27, stdev=453.56
clat (usec): min=3, max=32,886, avg=2226.99, stdev=1434.22
bw (KB/s) : min= 0, max=69432, per=0.96%, avg=4817.50, stdev=16888.25
write: io=127MB, bw=20,826KB/s, iops=5,206, runt= 6260msec
slat (usec): min=4, max=10,487, avg=170.66, stdev=467.69
clat (usec): min=23, max=32,886, avg=2227.84, stdev=1460.15
bw (KB/s) : min= 0, max=22411, per=1.93%, avg=3227.51, stdev=7631.44
cpu : usr=1.10%, sys=17.53%, ctx=6050, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98479/32593, short=0/0
lat (usec): 4=0.01%, 20=0.01%, 50=0.01%, 100=0.02%, 250=0.39%
lat (usec): 500=2.78%, 750=3.53%, 1000=6.63%
lat (msec): 2=36.59%, 4=41.12%, 10=8.62%, 20=0.28%, 50=0.03%
randrw: (groupid=0, jobs=1): err= 0: pid=7158
read : io=383MB, bw=63,262KB/s, iops=15,815, runt= 6197msec
slat (usec): min=4, max=15,504, avg=164.81, stdev=444.28
clat (usec): min=4, max=16,908, avg=2195.10, stdev=1334.18
bw (KB/s) : min= 0, max=69306, per=0.98%, avg=4912.46, stdev=17117.32
write: io=129MB, bw=21,342KB/s, iops=5,335, runt= 6197msec
slat (usec): min=4, max=15,504, avg=162.70, stdev=447.56
clat (usec): min=8, max=16,908, avg=2199.40, stdev=1360.04
bw (KB/s) : min= 0, max=22996, per=2.09%, avg=3492.01, stdev=8008.36
cpu : usr=1.03%, sys=17.72%, ctx=6039, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98008/33064, short=0/0
lat (usec): 10=0.01%, 20=0.01%, 50=0.01%, 100=0.04%, 250=0.51%
lat (usec): 500=2.81%, 750=3.79%, 1000=6.46%
lat (msec): 2=38.05%, 4=39.50%, 10=8.63%, 20=0.19%
randrw: (groupid=0, jobs=1): err= 0: pid=7159
read : io=385MB, bw=63,135KB/s, iops=15,783, runt= 6240msec
slat (usec): min=4, max=13,685, avg=164.46, stdev=440.87
clat (usec): min=3, max=23,286, avg=2209.34, stdev=1366.29
bw (KB/s) : min= 0, max=66936, per=1.80%, avg=9013.58, stdev=22400.18
write: io=127MB, bw=20,886KB/s, iops=5,221, runt= 6240msec
slat (usec): min=4, max=13,685, avg=170.27, stdev=458.26
clat (usec): min=3, max=22,950, avg=2205.22, stdev=1358.38
bw (KB/s) : min= 0, max=22568, per=2.30%, avg=3842.18, stdev=8236.50
cpu : usr=0.74%, sys=17.57%, ctx=6029, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98490/32582, short=0/0
lat (usec): 4=0.01%, 20=0.01%, 50=0.01%, 100=0.03%, 250=0.35%
lat (usec): 500=2.55%, 750=3.49%, 1000=6.43%
lat (msec): 2=38.10%, 4=39.86%, 10=9.01%, 20=0.12%, 50=0.04%
randrw: (groupid=0, jobs=1): err= 0: pid=7161
read : io=384MB, bw=62,776KB/s, iops=15,694, runt= 6266msec
slat (usec): min=4, max=17,937, avg=164.90, stdev=456.16
clat (usec): min=3, max=21,231, avg=2229.46, stdev=1450.21
bw (KB/s) : min= 0, max=68712, per=1.15%, avg=5749.50, stdev=18303.03
write: io=128MB, bw=20,896KB/s, iops=5,223, runt= 6266msec
slat (usec): min=4, max=9,211, avg=167.28, stdev=446.50
clat (usec): min=4, max=21,191, avg=2236.08, stdev=1467.63
bw (KB/s) : min= 0, max=22952, per=2.19%, avg=3658.33, stdev=8053.15
cpu : usr=1.40%, sys=17.57%, ctx=6073, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98339/32733, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.47%, 500=2.74%, 750=3.48%, 1000=6.63%
lat (msec): 2=37.63%, 4=39.57%, 10=9.06%, 20=0.35%, 50=0.02%
randrw: (groupid=0, jobs=1): err= 0: pid=7162
read : io=384MB, bw=63,143KB/s, iops=15,785, runt= 6234msec
slat (usec): min=4, max=15,722, avg=165.04, stdev=449.99
clat (usec): min=3, max=23,118, avg=2207.15, stdev=1373.05
bw (KB/s) : min= 0, max=68984, per=0.77%, avg=3881.52, stdev=15316.72
write: io=128MB, bw=20,958KB/s, iops=5,239, runt= 6234msec
slat (usec): min=4, max=15,722, avg=164.39, stdev=456.09
clat (usec): min=3, max=23,104, avg=2214.88, stdev=1384.18
bw (KB/s) : min= 0, max=23712, per=2.14%, avg=3572.75, stdev=8004.58
cpu : usr=0.00%, sys=18.53%, ctx=6103, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98409/32663, short=0/0
lat (usec): 4=0.01%, 20=0.01%, 50=0.01%, 100=0.02%, 250=0.52%
lat (usec): 500=2.81%, 750=3.54%, 1000=6.39%
lat (msec): 2=37.81%, 4=39.58%, 10=9.07%, 20=0.22%, 50=0.02%
randrw: (groupid=0, jobs=1): err= 0: pid=7163
read : io=384MB, bw=65,080KB/s, iops=16,269, runt= 6042msec
slat (usec): min=4, max=15,459, avg=158.67, stdev=435.37
clat (usec): min=19, max=17,897, avg=2140.08, stdev=1286.23
bw (KB/s) : min= 0, max=70986, per=0.87%, avg=4344.29, stdev=16333.50
write: io=128MB, bw=21,694KB/s, iops=5,423, runt= 6042msec
slat (usec): min=4, max=15,458, avg=158.76, stdev=431.68
clat (usec): min=25, max=17,850, avg=2146.92, stdev=1303.91
bw (KB/s) : min= 0, max=24440, per=2.00%, avg=3340.78, stdev=7905.87
cpu : usr=0.00%, sys=16.74%, ctx=6031, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98303/32769, short=0/0
lat (usec): 20=0.01%, 50=0.01%, 100=0.02%, 250=0.42%, 500=2.66%
lat (usec): 750=3.83%, 1000=6.97%
lat (msec): 2=38.80%, 4=38.74%, 10=8.42%, 20=0.13%
randrw: (groupid=0, jobs=1): err= 0: pid=7164
read : io=384MB, bw=62,752KB/s, iops=15,687, runt= 6267msec
slat (usec): min=4, max=9,502, avg=166.92, stdev=452.29
clat (usec): min=4, max=19,765, avg=2208.51, stdev=1382.61
bw (KB/s) : min= 0, max=68439, per=1.14%, avg=5740.39, stdev=18294.01
write: io=128MB, bw=20,907KB/s, iops=5,226, runt= 6267msec
slat (usec): min=4, max=9,502, avg=164.06, stdev=452.04
clat (usec): min=3, max=19,765, avg=2211.70, stdev=1381.06
bw (KB/s) : min= 0, max=22072, per=2.22%, avg=3714.18, stdev=8116.61
cpu : usr=0.34%, sys=18.77%, ctx=6035, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=98316/32756, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.40%, 500=2.43%, 750=3.51%, 1000=6.98%
lat (msec): 2=38.68%, 4=38.79%, 10=8.85%, 20=0.30%
Run status group 0 (all jobs):
READ: io=3,073MB, aggrb=490MB/s, minb=64,257KB/s, maxb=66,641KB/s, mint=6042msec, maxt=6267msec
WRITE: io=1,023MB, aggrb=163MB/s, minb=21,326KB/s, maxb=22,214KB/s, mint=6042msec, maxt=6267msec
Disk stats (read/write):
sdb: ios=784970/261501, merge=117/13, ticks=662700/221522, in_queue=884812, util=98.44%
[-- Attachment #3: fio-1x-rd_mcp-75-25-4k.txt --]
[-- Type: text/plain, Size: 9383 bytes --]
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
...
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
Starting 8 processes
randrw: (groupid=0, jobs=1): err= 0: pid=7215
read : io=128MB, bw=19,811KB/s, iops=4,952, runt= 6624msec
slat (usec): min=4, max=14,525, avg=181.82, stdev=463.25
clat (usec): min=12, max=20,029, avg=2386.98, stdev=1442.11
bw (KB/s) : min= 0, max=23152, per=2.02%, avg=3188.60, stdev=7446.08
write: io=384MB, bw=59,339KB/s, iops=14,834, runt= 6624msec
slat (usec): min=4, max=14,525, avg=180.59, stdev=457.75
clat (usec): min=3, max=20,029, avg=2379.34, stdev=1440.41
bw (KB/s) : min= 0, max=69792, per=0.98%, avg=4636.81, stdev=16212.49
cpu : usr=0.32%, sys=21.41%, ctx=6125, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=32807/98265, short=0/0
lat (usec): 4=0.01%, 20=0.01%, 50=0.02%, 100=0.03%, 250=0.33%
lat (usec): 500=1.94%, 750=2.67%, 1000=5.30%
lat (msec): 2=35.56%, 4=42.91%, 10=11.03%, 20=0.19%, 50=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=7216
read : io=128MB, bw=19,780KB/s, iops=4,944, runt= 6605msec
slat (usec): min=4, max=9,005, avg=179.88, stdev=439.38
clat (usec): min=4, max=14,494, avg=2389.54, stdev=1406.06
bw (KB/s) : min= 0, max=22056, per=1.51%, avg=2386.43, stdev=6538.84
write: io=384MB, bw=59,598KB/s, iops=14,899, runt= 6605msec
slat (usec): min=4, max=10,318, avg=181.25, stdev=449.03
clat (usec): min=3, max=14,494, avg=2380.70, stdev=1413.22
bw (KB/s) : min= 0, max=67169, per=0.90%, avg=4273.31, stdev=15608.83
cpu : usr=0.00%, sys=21.94%, ctx=6125, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=32661/98411, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%, 100=0.02%
lat (usec): 250=0.28%, 500=1.91%, 750=3.12%, 1000=6.06%
lat (msec): 2=34.23%, 4=42.57%, 10=11.69%, 20=0.09%
randrw: (groupid=0, jobs=1): err= 0: pid=7218
read : io=128MB, bw=20,424KB/s, iops=5,106, runt= 6394msec
slat (usec): min=4, max=13,178, avg=174.12, stdev=471.54
clat (usec): min=27, max=22,506, avg=2305.59, stdev=1434.24
bw (KB/s) : min= 0, max=24391, per=3.16%, avg=4995.04, stdev=9037.65
write: io=384MB, bw=61,573KB/s, iops=15,393, runt= 6394msec
slat (usec): min=4, max=13,177, avg=173.35, stdev=440.90
clat (usec): min=3, max=22,506, avg=2301.23, stdev=1427.33
bw (KB/s) : min= 0, max=74483, per=1.14%, avg=5410.33, stdev=17718.59
cpu : usr=0.41%, sys=18.71%, ctx=6157, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=32648/98424, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%, 100=0.01%
lat (usec): 250=0.31%, 500=1.94%, 750=2.99%, 1000=5.83%
lat (msec): 2=37.17%, 4=41.34%, 10=10.15%, 20=0.21%, 50=0.02%
randrw: (groupid=0, jobs=1): err= 0: pid=7219
read : io=128MB, bw=20,380KB/s, iops=5,095, runt= 6421msec
slat (usec): min=4, max=10,035, avg=176.28, stdev=440.97
clat (usec): min=3, max=13,759, avg=2307.50, stdev=1352.78
bw (KB/s) : min= 0, max=22920, per=2.48%, avg=3923.44, stdev=8180.44
write: io=384MB, bw=61,272KB/s, iops=15,318, runt= 6421msec
slat (usec): min=4, max=10,034, avg=174.73, stdev=443.56
clat (usec): min=3, max=13,759, avg=2312.77, stdev=1364.91
bw (KB/s) : min= 0, max=68536, per=0.78%, avg=3680.16, stdev=14716.61
cpu : usr=0.79%, sys=18.27%, ctx=6140, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=32715/98357, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.31%, 500=1.96%, 750=2.86%, 1000=5.61%
lat (msec): 2=36.58%, 4=42.33%, 10=10.15%, 20=0.16%
randrw: (groupid=0, jobs=1): err= 0: pid=7220
read : io=129MB, bw=19,864KB/s, iops=4,966, runt= 6635msec
slat (usec): min=4, max=23,513, avg=179.83, stdev=462.13
clat (usec): min=24, max=27,037, avg=2360.42, stdev=1496.60
bw (KB/s) : min= 0, max=23457, per=2.52%, avg=3979.62, stdev=8156.09
write: io=383MB, bw=59,154KB/s, iops=14,788, runt= 6635msec
slat (usec): min=4, max=23,513, avg=182.37, stdev=477.27
clat (usec): min=4, max=27,037, avg=2364.14, stdev=1497.99
bw (KB/s) : min= 0, max=68279, per=1.07%, avg=5064.11, stdev=16852.94
cpu : usr=0.81%, sys=20.43%, ctx=6140, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=32950/98122, short=0/0
lat (usec): 10=0.01%, 20=0.01%, 50=0.03%, 100=0.02%, 250=0.30%
lat (usec): 500=2.24%, 750=2.83%, 1000=5.31%
lat (msec): 2=35.94%, 4=42.00%, 10=11.09%, 20=0.22%, 50=0.02%
randrw: (groupid=0, jobs=1): err= 0: pid=7221
read : io=128MB, bw=19,720KB/s, iops=4,930, runt= 6639msec
slat (usec): min=4, max=12,265, avg=182.40, stdev=454.67
clat (usec): min=19, max=20,011, avg=2350.63, stdev=1455.94
bw (KB/s) : min= 0, max=22842, per=2.00%, avg=3165.91, stdev=7376.05
write: io=384MB, bw=59,250KB/s, iops=14,812, runt= 6639msec
slat (usec): min=4, max=12,266, avg=183.31, stdev=463.27
clat (usec): min=3, max=20,048, avg=2362.69, stdev=1469.85
bw (KB/s) : min= 0, max=68303, per=0.94%, avg=4434.98, stdev=15835.65
cpu : usr=1.48%, sys=19.76%, ctx=6129, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=32731/98341, short=0/0
lat (usec): 4=0.01%, 20=0.01%, 50=0.01%, 100=0.01%, 250=0.32%
lat (usec): 500=2.10%, 750=2.84%, 1000=6.15%
lat (msec): 2=35.43%, 4=41.82%, 10=11.10%, 20=0.21%, 50=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=7222
read : io=128MB, bw=20,213KB/s, iops=5,053, runt= 6496msec
slat (usec): min=4, max=9,123, avg=176.22, stdev=446.73
clat (usec): min=3, max=19,319, avg=2340.56, stdev=1457.01
bw (KB/s) : min= 0, max=23105, per=2.13%, avg=3358.21, stdev=7671.12
write: io=384MB, bw=60,496KB/s, iops=15,124, runt= 6496msec
slat (usec): min=4, max=11,905, avg=178.51, stdev=459.69
clat (usec): min=3, max=19,319, avg=2342.55, stdev=1446.37
bw (KB/s) : min= 0, max=69125, per=2.69%, avg=12738.23, stdev=25281.56
cpu : usr=0.43%, sys=19.41%, ctx=6130, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=32826/98246, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%, 100=0.02%
lat (usec): 250=0.25%, 500=1.99%, 750=2.90%, 1000=5.44%
lat (msec): 2=36.38%, 4=42.45%, 10=10.32%, 20=0.20%
randrw: (groupid=0, jobs=1): err= 0: pid=7224
read : io=129MB, bw=19,885KB/s, iops=4,971, runt= 6643msec
slat (usec): min=4, max=12,139, avg=181.12, stdev=463.72
clat (usec): min=3, max=24,060, avg=2366.15, stdev=1580.29
bw (KB/s) : min= 0, max=22568, per=2.01%, avg=3176.32, stdev=7468.64
write: io=383MB, bw=59,038KB/s, iops=14,759, runt= 6643msec
slat (usec): min=4, max=12,140, avg=181.58, stdev=472.47
clat (usec): min=11, max=24,060, avg=2370.99, stdev=1574.54
bw (KB/s) : min= 0, max=66688, per=0.75%, avg=3573.29, stdev=14348.03
cpu : usr=0.29%, sys=21.18%, ctx=6200, majf=0, minf=25
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=33024/98048, short=0/0
lat (usec): 4=0.01%, 20=0.01%, 50=0.02%, 100=0.02%, 250=0.33%
lat (usec): 500=2.27%, 750=2.97%, 1000=5.59%
lat (msec): 2=36.38%, 4=41.16%, 10=10.79%, 20=0.43%, 50=0.02%
Run status group 0 (all jobs):
READ: io=1,025MB, aggrb=154MB/s, minb=20,193KB/s, maxb=20,914KB/s, mint=6394msec, maxt=6643msec
WRITE: io=3,071MB, aggrb=462MB/s, minb=60,455KB/s, maxb=63,050KB/s, mint=6394msec, maxt=6643msec
Disk stats (read/write):
sdb: ios=260356/780128, merge=12/127, ticks=229906/690447, in_queue=920955, util=98.57%
[-- Attachment #4: fio-16x-rd_mcp-25-75-4k.txt --]
[-- Type: text/plain, Size: 20703 bytes --]
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
...
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
Starting 16 processes
randrw: (groupid=0, jobs=1): err= 0: pid=2547
read : io=6,124MB, bw=66,100KB/s, iops=16,524, runt= 94870msec
slat (usec): min=12, max=27,971, avg=174.21, stdev=266.86
clat (usec): min=3, max=864K, avg=2106.47, stdev=1738.44
bw (KB/s) : min= 0, max=107584, per=0.61%, avg=5409.05, stdev=19273.52
write: io=2,044MB, bw=22,063KB/s, iops=5,515, runt= 94870msec
slat (usec): min=13, max=27,970, avg=176.52, stdev=272.77
clat (usec): min=4, max=53,769, avg=2105.06, stdev=1560.23
bw (KB/s) : min= 0, max=35984, per=1.15%, avg=3379.14, stdev=8489.10
cpu : usr=2.53%, sys=94.42%, ctx=6501, majf=0, minf=356
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1567718/523290, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.07%, 750=0.44%, 1000=4.52%
lat (msec): 2=60.94%, 4=27.23%, 10=6.10%, 20=0.62%, 50=0.05%
lat (msec): 100=0.01%, 500=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2550
read : io=6,128MB, bw=58,320KB/s, iops=14,580, runt=107600msec
slat (usec): min=12, max=24,164, avg=196.17, stdev=275.12
clat (usec): min=3, max=935K, avg=2393.34, stdev=1837.28
bw (KB/s) : min= 0, max=170576, per=0.42%, avg=3737.76, stdev=15458.85
write: io=2,040MB, bw=19,413KB/s, iops=4,853, runt=107600msec
slat (usec): min=11, max=24,162, avg=197.17, stdev=256.00
clat (usec): min=4, max=312K, avg=2391.86, stdev=1708.51
bw (KB/s) : min= 0, max=55960, per=0.99%, avg=2897.19, stdev=7540.20
cpu : usr=2.92%, sys=94.04%, ctx=11358, majf=0, minf=367
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568810/522198, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.03%, 500=0.35%, 750=1.72%, 1000=4.74%
lat (msec): 2=45.40%, 4=37.23%, 10=9.87%, 20=0.63%, 50=0.04%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2552
read : io=6,126MB, bw=57,739KB/s, iops=14,434, runt=108647msec
slat (usec): min=11, max=22,024, avg=183.39, stdev=275.04
clat (usec): min=2, max=8,027K, avg=2495.05, stdev=45792.61
bw (KB/s) : min= 0, max=183488, per=0.49%, avg=4288.32, stdev=17503.79
write: io=2,042MB, bw=19,245KB/s, iops=4,811, runt=108647msec
slat (usec): min=12, max=22,023, avg=185.99, stdev=291.58
clat (usec): min=3, max=8,027K, avg=2431.55, stdev=40055.79
bw (KB/s) : min= 0, max=61248, per=1.08%, avg=3162.73, stdev=8362.27
cpu : usr=2.37%, sys=87.04%, ctx=14078, majf=0, minf=354
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568288/522720, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.03%, 500=0.61%, 750=2.93%, 1000=6.95%
lat (msec): 2=52.36%, 4=26.64%, 10=9.74%, 20=0.68%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2553
read : io=6,128MB, bw=68,578KB/s, iops=17,144, runt= 91509msec
slat (usec): min=12, max=21,715, avg=167.06, stdev=193.91
clat (usec): min=3, max=987K, avg=2033.80, stdev=3616.51
bw (KB/s) : min= 0, max=99856, per=0.61%, avg=5392.37, stdev=19330.77
write: io=2,040MB, bw=22,824KB/s, iops=5,705, runt= 91509msec
slat (usec): min=12, max=15,149, avg=168.73, stdev=188.41
clat (usec): min=3, max=610K, avg=2040.64, stdev=4036.89
bw (KB/s) : min= 0, max=33720, per=1.21%, avg=3563.95, stdev=8712.34
cpu : usr=2.40%, sys=94.98%, ctx=6749, majf=0, minf=294
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568867/522141, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.03%, 750=0.26%, 1000=3.95%
lat (msec): 2=62.25%, 4=28.61%, 10=4.59%, 20=0.29%, 50=0.01%
lat (msec): 100=0.01%, 750=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2554
read : io=6,129MB, bw=57,566KB/s, iops=14,391, runt=109021msec
slat (usec): min=11, max=868K, avg=198.80, stdev=1023.29
clat (usec): min=4, max=947K, avg=2426.93, stdev=5200.69
bw (KB/s) : min= 0, max=149128, per=0.50%, avg=4440.55, stdev=17256.87
write: io=2,039MB, bw=19,153KB/s, iops=4,788, runt=109021msec
slat (usec): min=11, max=868K, avg=203.19, stdev=1725.25
clat (usec): min=3, max=870K, avg=2417.48, stdev=4278.32
bw (KB/s) : min= 0, max=49472, per=1.01%, avg=2973.29, stdev=7867.38
cpu : usr=2.87%, sys=93.23%, ctx=12934, majf=0, minf=363
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568981/522027, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.02%
lat (usec): 500=0.44%, 750=2.30%, 1000=5.96%
lat (msec): 2=52.55%, 4=24.84%, 10=12.87%, 20=0.96%, 50=0.06%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2555
read : io=6,126MB, bw=59,703KB/s, iops=14,925, runt=105069msec
slat (usec): min=11, max=27,921, avg=186.30, stdev=275.04
clat (usec): min=2, max=1,007K, avg=2352.14, stdev=9607.09
bw (KB/s) : min= 0, max=111976, per=0.46%, avg=4078.26, stdev=16419.93
write: io=2,042MB, bw=19,902KB/s, iops=4,975, runt=105069msec
slat (usec): min=12, max=24,662, avg=188.34, stdev=273.10
clat (usec): min=3, max=1,007K, avg=2349.94, stdev=9357.34
bw (KB/s) : min= 0, max=37288, per=1.03%, avg=3016.35, stdev=7836.94
cpu : usr=2.41%, sys=91.31%, ctx=8846, majf=0, minf=362
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568232/522776, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.02%, 500=0.13%, 750=0.64%, 1000=4.32%
lat (msec): 2=55.52%, 4=29.73%, 10=9.11%, 20=0.47%, 50=0.05%
lat (msec): 100=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2559
read : io=6,124MB, bw=62,863KB/s, iops=15,715, runt= 99763msec
slat (usec): min=9, max=23,329, avg=182.81, stdev=262.64
clat (usec): min=3, max=334K, avg=2216.18, stdev=1519.34
bw (KB/s) : min= 0, max=109664, per=0.47%, avg=4137.62, stdev=16528.11
write: io=2,044MB, bw=20,976KB/s, iops=5,244, runt= 99763msec
slat (usec): min=12, max=17,396, avg=184.15, stdev=257.56
clat (usec): min=2, max=73,810, avg=2212.76, stdev=1479.10
bw (KB/s) : min= 0, max=36272, per=1.13%, avg=3318.54, stdev=8172.56
cpu : usr=2.74%, sys=93.90%, ctx=8540, majf=0, minf=368
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1567850/523158, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.09%, 750=0.50%, 1000=3.96%
lat (msec): 2=54.35%, 4=33.40%, 10=7.21%, 20=0.43%, 50=0.04%
lat (msec): 100=0.01%, 500=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2561
read : io=6,128MB, bw=58,758KB/s, iops=14,689, runt=106787msec
slat (usec): min=11, max=23,133, avg=173.71, stdev=257.70
clat (usec): min=3, max=11,841K, avg=2470.54, stdev=65500.47
bw (KB/s) : min= 0, max=140056, per=0.59%, avg=5205.73, stdev=18953.32
write: io=2,040MB, bw=19,566KB/s, iops=4,891, runt=106787msec
slat (usec): min=12, max=20,316, avg=175.69, stdev=262.82
clat (usec): min=4, max=11,841K, avg=2468.04, stdev=65543.80
bw (KB/s) : min= 0, max=47184, per=1.16%, avg=3408.06, stdev=8525.59
cpu : usr=2.60%, sys=83.36%, ctx=10825, majf=0, minf=358
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568647/522361, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.20%, 750=1.35%, 1000=5.61%
lat (msec): 2=56.43%, 4=29.65%, 10=6.27%, 20=0.42%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2566
read : io=6,125MB, bw=64,424KB/s, iops=16,105, runt= 97360msec
slat (usec): min=11, max=28,794, avg=174.49, stdev=259.93
clat (usec): min=3, max=1,067K, avg=2188.75, stdev=8685.78
bw (KB/s) : min= 0, max=104240, per=0.50%, avg=4371.86, stdev=17166.21
write: io=2,043MB, bw=21,484KB/s, iops=5,371, runt= 97360msec
slat (usec): min=12, max=25,628, avg=175.58, stdev=254.95
clat (usec): min=3, max=1,065K, avg=2159.64, stdev=6795.08
bw (KB/s) : min= 0, max=34528, per=1.09%, avg=3201.77, stdev=8151.24
cpu : usr=2.46%, sys=92.14%, ctx=6923, majf=0, minf=358
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568076/522932, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.01%, 500=0.08%, 750=0.42%, 1000=3.97%
lat (msec): 2=57.99%, 4=31.63%, 10=5.43%, 20=0.40%, 50=0.04%
lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2568
read : io=6,127MB, bw=62,184KB/s, iops=15,545, runt=100892msec
slat (usec): min=12, max=24,702, avg=184.77, stdev=262.08
clat (usec): min=3, max=835K, avg=2243.55, stdev=1722.47
bw (KB/s) : min= 0, max=99480, per=0.48%, avg=4246.35, stdev=16608.96
write: io=2,041MB, bw=20,717KB/s, iops=5,179, runt=100892msec
slat (usec): min=12, max=24,701, avg=186.54, stdev=267.69
clat (usec): min=8, max=50,242, avg=2235.46, stdev=1442.06
bw (KB/s) : min= 0, max=33400, per=1.02%, avg=2994.72, stdev=7757.57
cpu : usr=2.68%, sys=94.22%, ctx=8180, majf=0, minf=368
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568463/522545, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.01%, 500=0.10%, 750=0.46%, 1000=3.70%
lat (msec): 2=53.45%, 4=34.21%, 10=7.63%, 20=0.40%, 50=0.04%
lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2571
read : io=6,127MB, bw=58,672KB/s, iops=14,668, runt=106933msec
slat (usec): min=11, max=26,233, avg=173.92, stdev=270.82
clat (usec): min=3, max=10,788K, avg=2433.15, stdev=56084.02
bw (KB/s) : min= 0, max=153856, per=0.52%, avg=4580.43, stdev=18033.52
write: io=2,041MB, bw=19,545KB/s, iops=4,886, runt=106933msec
slat (usec): min=12, max=26,232, avg=176.49, stdev=282.94
clat (usec): min=2, max=10,788K, avg=2603.59, stdev=70235.11
bw (KB/s) : min= 0, max=50616, per=1.11%, avg=3251.19, stdev=8439.40
cpu : usr=2.41%, sys=83.42%, ctx=10540, majf=0, minf=366
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568497/522511, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.02%
lat (usec): 500=0.29%, 750=1.67%, 1000=5.98%
lat (msec): 2=57.91%, 4=27.24%, 10=5.97%, 20=0.85%, 50=0.06%
lat (msec): 100=0.01%, 250=0.01%, 1000=0.01%, 2000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2575
read : io=6,123MB, bw=59,671KB/s, iops=14,917, runt=105073msec
slat (usec): min=12, max=22,307, avg=190.15, stdev=281.91
clat (usec): min=3, max=985K, avg=2325.21, stdev=4263.24
bw (KB/s) : min= 0, max=134784, per=0.51%, avg=4465.73, stdev=17179.67
write: io=2,045MB, bw=19,931KB/s, iops=4,982, runt=105073msec
slat (usec): min=13, max=22,308, avg=192.36, stdev=286.43
clat (usec): min=5, max=984K, avg=2325.95, stdev=4453.78
bw (KB/s) : min= 0, max=45200, per=1.02%, avg=2986.76, stdev=7837.42
cpu : usr=2.78%, sys=93.13%, ctx=8545, majf=0, minf=359
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1567451/523557, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.02%, 500=0.14%, 750=0.95%, 1000=5.31%
lat (msec): 2=53.83%, 4=29.84%, 10=8.86%, 20=0.98%, 50=0.06%
lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2578
read : io=6,127MB, bw=60,637KB/s, iops=15,159, runt=103462msec
slat (usec): min=11, max=24,010, avg=187.54, stdev=267.45
clat (usec): min=3, max=614K, avg=2306.54, stdev=3870.17
bw (KB/s) : min= 0, max=128480, per=0.43%, avg=3826.70, stdev=15713.78
write: io=2,041MB, bw=20,204KB/s, iops=5,051, runt=103462msec
slat (usec): min=12, max=24,008, avg=189.02, stdev=265.42
clat (usec): min=4, max=613K, avg=2304.43, stdev=3913.98
bw (KB/s) : min= 0, max=43568, per=1.01%, avg=2958.06, stdev=7664.00
cpu : usr=2.79%, sys=93.40%, ctx=8121, majf=0, minf=365
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568410/522598, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.02%, 500=0.12%, 750=0.90%, 1000=4.44%
lat (msec): 2=50.07%, 4=36.48%, 10=7.33%, 20=0.56%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, 750=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2580
read : io=6,126MB, bw=55,126KB/s, iops=13,781, runt=113790msec
slat (usec): min=12, max=21,659, avg=186.51, stdev=294.57
clat (usec): min=3, max=11,301K, avg=2622.98, stdev=63184.53
bw (KB/s) : min= 0, max=255064, per=0.54%, avg=4798.37, stdev=20558.38
write: io=2,042MB, bw=18,378KB/s, iops=4,594, runt=113790msec
slat (usec): min=12, max=21,557, avg=188.72, stdev=286.56
clat (usec): min=22, max=11,301K, avg=2596.75, stdev=60542.75
bw (KB/s) : min= 0, max=83896, per=1.04%, avg=3066.39, stdev=9231.50
cpu : usr=2.64%, sys=84.00%, ctx=35733, majf=0, minf=356
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1568203/522805, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.02%, 500=7.40%, 750=6.99%, 1000=7.36%
lat (msec): 2=38.21%, 4=25.60%, 10=13.67%, 20=0.68%, 50=0.05%
lat (msec): 100=0.01%, 500=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2588
read : io=6,131MB, bw=67,903KB/s, iops=16,975, runt= 92456msec
slat (usec): min=11, max=26,772, avg=168.33, stdev=229.38
clat (usec): min=2, max=977K, avg=2059.06, stdev=5459.86
bw (KB/s) : min= 0, max=111328, per=0.64%, avg=5634.86, stdev=19828.90
write: io=2,037MB, bw=22,562KB/s, iops=5,640, runt= 92456msec
slat (usec): min=13, max=22,712, avg=169.62, stdev=222.60
clat (usec): min=3, max=977K, avg=2065.39, stdev=6119.42
bw (KB/s) : min= 0, max=36864, per=1.25%, avg=3676.28, stdev=8868.76
cpu : usr=2.82%, sys=93.79%, ctx=6430, majf=0, minf=349
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1569515/521493, short=0/0
lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.04%
lat (usec): 750=0.45%, 1000=4.85%
lat (msec): 2=61.24%, 4=27.98%, 10=5.07%, 20=0.32%, 50=0.03%
lat (msec): 100=0.01%, 500=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=2599
read : io=6,123MB, bw=61,463KB/s, iops=15,365, runt=102016msec
slat (usec): min=11, max=24,381, avg=184.89, stdev=262.77
clat (usec): min=3, max=973K, avg=2275.52, stdev=5303.92
bw (KB/s) : min= 0, max=114752, per=0.48%, avg=4224.03, stdev=16680.52
write: io=2,045MB, bw=20,524KB/s, iops=5,131, runt=102016msec
slat (usec): min=12, max=24,380, avg=186.70, stdev=267.43
clat (usec): min=4, max=972K, avg=2279.80, stdev=5680.26
bw (KB/s) : min= 0, max=38352, per=1.16%, avg=3415.46, stdev=8256.67
cpu : usr=2.97%, sys=93.01%, ctx=8536, majf=0, minf=359
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1567555/523453, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.02%, 500=0.13%, 750=0.72%, 1000=4.79%
lat (msec): 2=52.14%, 4=33.91%, 10=7.75%, 20=0.50%, 50=0.04%
lat (msec): 100=0.01%, 1000=0.01%
Run status group 0 (all jobs):
READ: io=98,022MB, aggrb=861MB/s, minb=56,449KB/s, maxb=70,223KB/s, mint=91509msec, maxt=113790msec
WRITE: io=32,666MB, aggrb=287MB/s, minb=18,818KB/s, maxb=23,371KB/s, mint=91509msec, maxt=113790msec
Disk stats (read/write):
sdb: ios=1570992/526078, merge=2/0, ticks=351692/144852, in_queue=496272, util=91.13%
sdc: ios=1572517/524594, merge=3/0, ticks=510340/132744, in_queue=645383, util=94.00%
sdf: ios=1573298/523843, merge=1/1, ticks=423040/120090, in_queue=543710, util=92.36%
sdg: ios=1573447/523777, merge=1/1, ticks=517825/171550, in_queue=690113, util=94.20%
sdj: ios=1572489/524739, merge=1/0, ticks=380283/131216, in_queue=513255, util=91.21%
sdk: ios=1574152/523069, merge=0/0, ticks=415897/145661, in_queue=561750, util=92.40%
sdn: ios=1573508/523738, merge=0/0, ticks=482623/146468, in_queue=629581, util=93.21%
sdo: ios=1572543/524674, merge=0/0, ticks=518038/178062, in_queue=699814, util=93.85%
sdr: ios=1573160/524087, merge=0/0, ticks=472194/156134, in_queue=628538, util=94.07%
sds: ios=1572887/524358, merge=0/0, ticks=512695/160155, in_queue=673744, util=94.59%
sdv: ios=1574006/523222, merge=0/0, ticks=412086/140019, in_queue=553846, util=91.52%
sdw: ios=1574190/522957, merge=13/6, ticks=1028215/319336, in_queue=1338369, util=96.72%
sdz: ios=1571843/525402, merge=0/0, ticks=531606/189539, in_queue=721936, util=94.44%
sdaa: ios=1535462/511435, merge=1/0, ticks=396725/142481, in_queue=539675, util=90.01%
sdad: ios=1573613/523652, merge=0/0, ticks=559885/197060, in_queue=758248, util=94.65%
sdae: ios=1535093/511783, merge=0/0, ticks=396290/179216, in_queue=576360, util=91.36%
[-- Attachment #5: fio-16x-rd_mcp-75-25-4k.txt --]
[-- Type: text/plain, Size: 20676 bytes --]
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
...
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
Starting 16 processes
randrw: (groupid=0, jobs=1): err= 0: pid=5400
read : io=2,042MB, bw=20,460KB/s, iops=5,114, runt=102203msec
slat (usec): min=12, max=22,671, avg=175.37, stdev=231.46
clat (usec): min=3, max=6,003K, avg=2380.80, stdev=38172.74
bw (KB/s) : min= 0, max=69632, per=1.01%, avg=3319.37, stdev=8556.85
write: io=6,126MB, bw=61,378KB/s, iops=15,344, runt=102203msec
slat (usec): min=11, max=23,406, avg=175.63, stdev=225.77
clat (usec): min=3, max=6,003K, avg=2316.07, stdev=31731.75
bw (KB/s) : min= 0, max=209680, per=0.49%, avg=4772.01, stdev=18459.42
cpu : usr=2.54%, sys=88.49%, ctx=18173, majf=0, minf=292
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522760/1568248, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.01%, 500=1.07%, 750=1.56%, 1000=5.44%
lat (msec): 2=57.00%, 4=26.19%, 10=8.19%, 20=0.51%, 50=0.02%
lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5403
read : io=2,042MB, bw=21,219KB/s, iops=5,304, runt= 98540msec
slat (usec): min=12, max=1,037K, avg=175.41, stdev=1452.13
clat (usec): min=12, max=2,265K, avg=2222.05, stdev=14539.38
bw (KB/s) : min= 0, max=40920, per=1.04%, avg=3399.56, stdev=8589.12
write: io=6,126MB, bw=63,660KB/s, iops=15,915, runt= 98540msec
slat (usec): min=12, max=1,037K, avg=175.82, stdev=1452.46
clat (usec): min=3, max=2,265K, avg=2210.09, stdev=14167.54
bw (KB/s) : min= 0, max=123904, per=0.55%, avg=5396.55, stdev=19414.89
cpu : usr=2.24%, sys=91.12%, ctx=10329, majf=0, minf=279
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522738/1568270, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.01%, 500=0.08%, 750=0.93%, 1000=5.74%
lat (msec): 2=59.91%, 4=25.46%, 10=7.34%, 20=0.48%, 50=0.03%
lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec): >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5405
read : io=2,041MB, bw=23,166KB/s, iops=5,791, runt= 90229msec
slat (usec): min=12, max=26,837, avg=164.06, stdev=192.74
clat (usec): min=88, max=1,061K, avg=2002.88, stdev=4616.16
bw (KB/s) : min= 0, max=35328, per=1.18%, avg=3855.86, stdev=9105.04
write: io=6,127MB, bw=69,532KB/s, iops=17,383, runt= 90229msec
slat (usec): min=12, max=26,838, avg=165.07, stdev=197.00
clat (usec): min=5, max=1,063K, avg=2022.56, stdev=6095.61
bw (KB/s) : min= 0, max=106512, per=0.52%, avg=5084.68, stdev=18985.13
cpu : usr=2.47%, sys=94.18%, ctx=9418, majf=0, minf=224
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522557/1568451, short=0/0
lat (usec): 10=0.01%, 20=0.01%, 100=0.01%, 250=0.01%, 500=0.03%
lat (usec): 750=0.32%, 1000=4.12%
lat (msec): 2=64.10%, 4=26.07%, 10=5.15%, 20=0.19%, 50=0.01%
lat (msec): 500=0.01%, 1000=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5406
read : io=2,040MB, bw=20,823KB/s, iops=5,205, runt=100313msec
slat (usec): min=10, max=21,390, avg=182.87, stdev=265.77
clat (usec): min=2, max=166K, avg=2225.22, stdev=2174.95
bw (KB/s) : min= 0, max=47000, per=0.98%, avg=3196.89, stdev=8344.89
write: io=6,128MB, bw=62,557KB/s, iops=15,639, runt=100313msec
slat (usec): min=10, max=24,630, avg=183.91, stdev=266.89
clat (usec): min=42, max=1,137K, avg=2231.98, stdev=2514.15
bw (KB/s) : min= 0, max=140096, per=0.48%, avg=4694.76, stdev=18152.16
cpu : usr=2.61%, sys=94.53%, ctx=12908, majf=0, minf=286
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522197/1568811, short=0/0
lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.18%
lat (usec): 750=1.40%, 1000=5.72%
lat (msec): 2=57.76%, 4=26.26%, 10=7.41%, 20=1.16%, 50=0.10%
lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5408
read : io=2,042MB, bw=21,579KB/s, iops=5,394, runt= 96893msec
slat (usec): min=12, max=24,417, avg=177.97, stdev=245.24
clat (usec): min=4, max=606K, avg=2160.41, stdev=2076.76
bw (KB/s) : min= 0, max=39824, per=0.95%, avg=3118.12, stdev=8213.33
write: io=6,126MB, bw=64,744KB/s, iops=16,185, runt= 96893msec
slat (usec): min=11, max=24,418, avg=178.20, stdev=239.99
clat (usec): min=2, max=794K, avg=2157.34, stdev=2056.01
bw (KB/s) : min= 0, max=119616, per=0.48%, avg=4760.83, stdev=18210.70
cpu : usr=2.89%, sys=94.53%, ctx=9526, majf=0, minf=290
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522706/1568302, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.08%, 750=0.66%, 1000=5.12%
lat (msec): 2=60.12%, 4=25.23%, 10=8.23%, 20=0.51%, 50=0.04%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5411
read : io=2,045MB, bw=23,417KB/s, iops=5,854, runt= 89409msec
slat (usec): min=12, max=24,150, avg=163.38, stdev=188.59
clat (usec): min=2, max=961K, avg=1997.13, stdev=5359.34
bw (KB/s) : min= 0, max=33176, per=1.20%, avg=3921.81, stdev=9249.26
write: io=6,123MB, bw=70,131KB/s, iops=17,532, runt= 89409msec
slat (usec): min=11, max=24,151, avg=163.63, stdev=187.56
clat (usec): min=2, max=963K, avg=1997.59, stdev=5377.38
bw (KB/s) : min= 0, max=98808, per=0.59%, avg=5801.44, stdev=20325.41
cpu : usr=2.50%, sys=94.71%, ctx=8925, majf=0, minf=225
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=523433/1567575, short=0/0
lat (usec): 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.03%
lat (usec): 750=0.26%, 1000=4.09%
lat (msec): 2=66.49%, 4=23.59%, 10=5.29%, 20=0.23%, 50=0.01%
lat (msec): 500=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5415
read : io=2,047MB, bw=21,052KB/s, iops=5,263, runt= 99562msec
slat (usec): min=12, max=23,013, avg=179.49, stdev=231.02
clat (usec): min=5, max=990K, avg=2232.38, stdev=7309.81
bw (KB/s) : min= 0, max=40664, per=1.05%, avg=3450.76, stdev=8443.34
write: io=6,121MB, bw=62,956KB/s, iops=15,738, runt= 99562msec
slat (usec): min=11, max=23,017, avg=179.92, stdev=228.20
clat (usec): min=2, max=991K, avg=2232.22, stdev=7402.44
bw (KB/s) : min= 0, max=120560, per=0.48%, avg=4713.87, stdev=17808.57
cpu : usr=2.55%, sys=93.26%, ctx=12195, majf=0, minf=293
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=524006/1567002, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.01%, 500=0.10%, 750=0.81%, 1000=4.50%
lat (msec): 2=56.69%, 4=29.72%, 10=7.60%, 20=0.53%, 50=0.03%
lat (msec): 100=0.01%, 500=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5418
read : io=2,043MB, bw=20,860KB/s, iops=5,215, runt=100302msec
slat (usec): min=12, max=427K, avg=181.53, stdev=869.43
clat (usec): min=58, max=974K, avg=2238.65, stdev=6606.73
bw (KB/s) : min= 0, max=72088, per=1.04%, avg=3394.31, stdev=8585.12
write: io=6,125MB, bw=62,528KB/s, iops=15,632, runt=100302msec
slat (usec): min=12, max=427K, avg=181.22, stdev=540.35
clat (usec): min=3, max=974K, avg=2251.61, stdev=6893.62
bw (KB/s) : min= 0, max=213368, per=0.51%, avg=4994.41, stdev=18717.14
cpu : usr=2.41%, sys=92.93%, ctx=12565, majf=0, minf=293
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=523081/1567927, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.01%, 500=0.76%, 750=2.30%, 1000=4.84%
lat (msec): 2=56.04%, 4=26.96%, 10=8.49%, 20=0.55%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5419
read : io=2,041MB, bw=21,657KB/s, iops=5,414, runt= 96514msec
slat (usec): min=12, max=19,996, avg=177.09, stdev=231.33
clat (usec): min=3, max=112K, avg=2147.31, stdev=1724.55
bw (KB/s) : min= 0, max=40432, per=1.07%, avg=3499.30, stdev=8651.00
write: io=6,127MB, bw=65,005KB/s, iops=16,251, runt= 96514msec
slat (usec): min=11, max=20,001, avg=177.60, stdev=228.69
clat (usec): min=3, max=983K, avg=2147.52, stdev=1869.10
bw (KB/s) : min= 0, max=119984, per=0.49%, avg=4768.17, stdev=18249.18
cpu : usr=2.80%, sys=94.82%, ctx=9438, majf=0, minf=289
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522545/1568463, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.04%, 750=0.45%, 1000=4.45%
lat (msec): 2=61.29%, 4=25.70%, 10=7.42%, 20=0.59%, 50=0.06%
lat (msec): 100=0.01%, 250=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5421
read : io=2,043MB, bw=21,508KB/s, iops=5,376, runt= 97253msec
slat (usec): min=11, max=24,839, avg=176.88, stdev=233.19
clat (usec): min=2, max=1,028K, avg=2179.00, stdev=6253.87
bw (KB/s) : min= 0, max=35480, per=1.02%, avg=3334.78, stdev=8350.75
write: io=6,125MB, bw=64,495KB/s, iops=16,123, runt= 97253msec
slat (usec): min=10, max=24,838, avg=177.39, stdev=229.57
clat (usec): min=3, max=1,029K, avg=2172.26, stdev=5624.04
bw (KB/s) : min= 0, max=107000, per=0.47%, avg=4657.42, stdev=17796.07
cpu : usr=2.58%, sys=93.92%, ctx=9420, majf=0, minf=293
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522919/1568089, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec): 250=0.01%, 500=0.07%, 750=0.41%, 1000=4.05%
lat (msec): 2=59.96%, 4=27.26%, 10=7.84%, 20=0.37%, 50=0.03%
lat (msec): 100=0.01%, 250=0.01%, 750=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5422
read : io=2,046MB, bw=20,865KB/s, iops=5,216, runt=100431msec
slat (usec): min=12, max=18,936, avg=178.24, stdev=220.92
clat (usec): min=3, max=1,085K, avg=2277.03, stdev=10543.06
bw (KB/s) : min= 0, max=64048, per=0.98%, avg=3214.49, stdev=8315.93
write: io=6,122MB, bw=62,416KB/s, iops=15,604, runt=100431msec
slat (usec): min=11, max=18,936, avg=179.54, stdev=238.08
clat (usec): min=3, max=1,218K, avg=2252.51, stdev=9355.08
bw (KB/s) : min= 0, max=188896, per=0.47%, avg=4623.42, stdev=17907.56
cpu : usr=2.40%, sys=92.13%, ctx=14663, majf=0, minf=289
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=523881/1567127, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.60%, 750=2.00%, 1000=5.03%
lat (msec): 2=55.68%, 4=28.08%, 10=7.99%, 20=0.54%, 50=0.04%
lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5423
read : io=2,043MB, bw=21,866KB/s, iops=5,466, runt= 95692msec
slat (usec): min=10, max=25,313, avg=172.80, stdev=218.96
clat (usec): min=32, max=1,748K, avg=2109.43, stdev=4990.48
bw (KB/s) : min= 0, max=42856, per=1.14%, avg=3741.56, stdev=8800.77
write: io=6,125MB, bw=65,539KB/s, iops=16,384, runt= 95692msec
slat (usec): min=10, max=25,312, avg=173.74, stdev=225.83
clat (usec): min=3, max=1,750K, avg=2122.99, stdev=7473.72
bw (KB/s) : min= 0, max=124768, per=0.57%, avg=5562.20, stdev=19413.97
cpu : usr=2.46%, sys=93.24%, ctx=9598, majf=0, minf=296
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=523110/1567898, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.06%, 750=0.49%, 1000=4.28%
lat (msec): 2=59.46%, 4=28.76%, 10=6.62%, 20=0.30%, 50=0.02%
lat (msec): 500=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5426
read : io=2,042MB, bw=21,241KB/s, iops=5,310, runt= 98463msec
slat (usec): min=12, max=328K, avg=181.08, stdev=512.92
clat (usec): min=2, max=329K, avg=2194.43, stdev=2135.67
bw (KB/s) : min= 0, max=41096, per=1.01%, avg=3321.97, stdev=8418.27
write: io=6,126MB, bw=63,704KB/s, iops=15,926, runt= 98463msec
slat (usec): min=12, max=328K, avg=181.45, stdev=511.49
clat (usec): min=3, max=806K, avg=2190.69, stdev=2246.42
bw (KB/s) : min= 0, max=123336, per=0.42%, avg=4136.03, stdev=16998.12
cpu : usr=2.38%, sys=95.15%, ctx=10125, majf=0, minf=301
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522875/1568133, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.08%, 750=1.02%, 1000=5.56%
lat (msec): 2=58.21%, 4=25.73%, 10=8.73%, 20=0.61%, 50=0.03%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5430
read : io=2,044MB, bw=23,139KB/s, iops=5,784, runt= 90434msec
slat (usec): min=9, max=17,771, avg=166.73, stdev=205.87
clat (usec): min=3, max=28,581, avg=2011.96, stdev=1287.60
bw (KB/s) : min= 0, max=33024, per=1.21%, avg=3977.11, stdev=9220.77
write: io=6,124MB, bw=69,349KB/s, iops=17,337, runt= 90434msec
slat (usec): min=8, max=17,772, avg=167.18, stdev=200.49
clat (usec): min=2, max=322K, avg=2012.38, stdev=1318.95
bw (KB/s) : min= 0, max=101904, per=0.56%, avg=5468.81, stdev=19634.52
cpu : usr=2.67%, sys=95.33%, ctx=9582, majf=0, minf=223
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=523142/1567866, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.04%
lat (usec): 750=0.30%, 1000=4.09%
lat (msec): 2=64.07%, 4=25.85%, 10=5.29%, 20=0.33%, 50=0.01%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5433
read : io=2,041MB, bw=21,631KB/s, iops=5,407, runt= 96622msec
slat (usec): min=12, max=24,818, avg=177.45, stdev=239.57
clat (usec): min=2, max=54,795, avg=2148.21, stdev=1634.62
bw (KB/s) : min= 0, max=44208, per=1.15%, avg=3781.51, stdev=9009.51
write: io=6,127MB, bw=64,934KB/s, iops=16,233, runt= 96622msec
slat (usec): min=11, max=24,820, avg=178.45, stdev=235.05
clat (usec): min=3, max=827K, avg=2153.44, stdev=1880.85
bw (KB/s) : min= 0, max=131424, per=0.51%, avg=5039.91, stdev=18868.19
cpu : usr=2.46%, sys=95.35%, ctx=9448, majf=0, minf=292
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=522502/1568506, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
lat (usec): 500=0.07%, 750=0.91%, 1000=5.80%
lat (msec): 2=59.72%, 4=24.46%, 10=8.39%, 20=0.59%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=5435
read : io=2,043MB, bw=21,489KB/s, iops=5,372, runt= 97369msec
slat (usec): min=12, max=19,281, avg=177.99, stdev=228.71
clat (usec): min=52, max=51,356, avg=2165.59, stdev=1640.70
bw (KB/s) : min= 0, max=40880, per=1.17%, avg=3832.59, stdev=8911.65
write: io=6,125MB, bw=64,412KB/s, iops=16,102, runt= 97369msec
slat (usec): min=11, max=21,294, avg=179.67, stdev=243.44
clat (usec): min=23, max=917K, avg=2169.18, stdev=1955.65
bw (KB/s) : min= 0, max=117336, per=0.47%, avg=4608.21, stdev=17816.78
cpu : usr=2.60%, sys=94.74%, ctx=10485, majf=0, minf=295
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=523084/1567924, short=0/0
lat (usec): 50=0.01%, 100=0.01%, 250=0.01%, 500=0.07%, 750=0.51%
lat (usec): 1000=4.43%
lat (msec): 2=60.54%, 4=26.37%, 10=7.22%, 20=0.81%, 50=0.03%
lat (msec): 100=0.01%, 500=0.01%, 1000=0.01%
Run status group 0 (all jobs):
READ: io=32,686MB, aggrb=320MB/s, minb=20,950KB/s, maxb=23,979KB/s, mint=89409msec, maxt=102203msec
WRITE: io=98,002MB, aggrb=959MB/s, minb=62,850KB/s, maxb=71,813KB/s, mint=89409msec, maxt=102203msec
Disk stats (read/write):
sdb: ios=524732/1572507, merge=0/0, ticks=103623/284056, in_queue=386938, util=90.65%
sdc: ios=524007/1573232, merge=0/0, ticks=136650/387572, in_queue=526039, util=95.12%
sdf: ios=524172/1573069, merge=1/0, ticks=103816/328857, in_queue=432885, util=92.30%
sdg: ios=524563/1572702, merge=0/0, ticks=143133/430861, in_queue=574566, util=95.14%
sdj: ios=524697/1572620, merge=0/0, ticks=105559/287185, in_queue=393271, util=91.25%
sdk: ios=524189/1573176, merge=0/1, ticks=108667/340430, in_queue=449542, util=92.97%
sdn: ios=524898/1572471, merge=0/0, ticks=117013/362832, in_queue=480406, util=94.40%
sdo: ios=525975/1571417, merge=0/0, ticks=127692/400875, in_queue=530214, util=95.18%
sdr: ios=525067/1572325, merge=0/1, ticks=123347/364557, in_queue=487667, util=95.68%
sds: ios=523876/1573487, merge=0/1, ticks=126661/401779, in_queue=529278, util=96.14%
sdv: ios=525253/1572170, merge=0/0, ticks=104001/300690, in_queue=406165, util=92.12%
sdw: ios=525082/1572239, merge=1/16, ticks=281593/787233, in_queue=1059862, util=97.55%
sdz: ios=525599/1571785, merge=0/1, ticks=155029/453394, in_queue=608861, util=95.61%
sdaa: ios=511876/1536335, merge=0/0, ticks=113820/306599, in_queue=420507, util=89.85%
sdad: ios=524086/1573304, merge=0/2, ticks=164113/491454, in_queue=656369, util=95.99%
sdae: ios=512678/1535528, merge=0/0, ticks=117502/347213, in_queue=465071, util=91.67%
[-- Attachment #6: fio-32x-rd_mcp-25-75-4k.txt --]
[-- Type: text/plain, Size: 22525 bytes --]
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
...
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
Starting 16 processes
randrw: (groupid=0, jobs=1): err= 0: pid=22395
read : io=12,216MB, bw=80,966KB/s, iops=20,241, runt=154494msec
slat (usec): min=8, max=4,569K, avg=136.78, stdev=2597.58
clat (usec): min=3, max=4,578K, avg=1732.56, stdev=14917.19
bw (KB/s) : min= 0, max=146768, per=0.53%, avg=6065.16, stdev=23015.44
write: io=4,072MB, bw=26,992KB/s, iops=6,747, runt=154494msec
slat (usec): min=8, max=4,569K, avg=149.37, stdev=7755.07
clat (usec): min=2, max=4,580K, avg=1752.79, stdev=17405.37
bw (KB/s) : min= 0, max=49040, per=1.13%, avg=4271.80, stdev=10712.10
cpu : usr=3.24%, sys=89.85%, ctx=18901, majf=0, minf=819
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3127203/1042525, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.21%, 500=1.55%, 750=8.01%, 1000=16.99%
lat (msec): 2=53.61%, 4=14.01%, 10=5.11%, 20=0.45%, 50=0.02%
lat (msec): 100=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22397
read : io=12,210MB, bw=84,717KB/s, iops=21,179, runt=147590msec
slat (usec): min=8, max=21,641, avg=129.13, stdev=256.01
clat (usec): min=2, max=4,581K, avg=1683.74, stdev=18882.06
bw (KB/s) : min= 0, max=143920, per=0.59%, avg=6705.74, stdev=24352.29
write: io=4,078MB, bw=28,291KB/s, iops=7,072, runt=147590msec
slat (usec): min=9, max=21,638, avg=131.20, stdev=265.13
clat (usec): min=3, max=4,579K, avg=1657.09, stdev=14909.86
bw (KB/s) : min= 0, max=49160, per=1.21%, avg=4600.54, stdev=11174.60
cpu : usr=3.16%, sys=89.92%, ctx=18316, majf=0, minf=684
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3125853/1043875, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.22%, 500=1.59%, 750=8.27%, 1000=17.39%
lat (msec): 2=55.34%, 4=12.71%, 10=4.01%, 20=0.40%, 50=0.02%
lat (msec): 100=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22399
read : io=12,212MB, bw=77,117KB/s, iops=19,279, runt=162159msec
slat (usec): min=8, max=27,289, avg=132.79, stdev=265.15
clat (usec): min=2, max=9,276K, avg=1883.60, stdev=41160.52
bw (KB/s) : min= 0, max=151536, per=0.55%, avg=6197.07, stdev=23377.98
write: io=4,076MB, bw=25,738KB/s, iops=6,434, runt=162159msec
slat (usec): min=8, max=27,290, avg=133.88, stdev=258.92
clat (usec): min=3, max=9,276K, avg=1873.74, stdev=39353.99
bw (KB/s) : min= 0, max=50720, per=1.20%, avg=4559.71, stdev=11083.83
cpu : usr=3.02%, sys=84.72%, ctx=19976, majf=0, minf=837
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3126308/1043420, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.22%, 500=1.55%, 750=8.05%, 1000=17.20%
lat (msec): 2=54.76%, 4=12.94%, 10=4.77%, 20=0.44%, 50=0.03%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, 2000=0.01%
lat (msec): >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22402
read : io=12,214MB, bw=74,694KB/s, iops=18,673, runt=167442msec
slat (usec): min=8, max=24,386, avg=134.85, stdev=269.15
clat (usec): min=3, max=10,418K, avg=1966.31, stdev=48498.01
bw (KB/s) : min= 0, max=187928, per=0.53%, avg=6061.80, stdev=23005.81
write: io=4,074MB, bw=24,916KB/s, iops=6,229, runt=167442msec
slat (usec): min=9, max=24,387, avg=136.34, stdev=266.66
clat (usec): min=3, max=10,418K, avg=1920.65, stdev=43939.65
bw (KB/s) : min= 0, max=62456, per=1.09%, avg=4143.37, stdev=10587.60
cpu : usr=3.07%, sys=82.84%, ctx=24048, majf=0, minf=822
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3126720/1043008, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.26%, 500=1.95%, 750=8.52%, 1000=16.50%
lat (msec): 2=53.13%, 4=13.81%, 10=5.31%, 20=0.46%, 50=0.02%
lat (msec): 100=0.01%, 250=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22403
read : io=12,215MB, bw=71,041KB/s, iops=17,760, runt=176068msec
slat (usec): min=8, max=29,865, avg=152.04, stdev=319.74
clat (usec): min=3, max=4,588K, avg=2000.83, stdev=18770.88
bw (KB/s) : min= 0, max=280880, per=0.45%, avg=5152.73, stdev=21989.85
write: io=4,073MB, bw=23,689KB/s, iops=5,922, runt=176068msec
slat (usec): min=8, max=29,866, avg=153.63, stdev=317.61
clat (usec): min=3, max=4,589K, avg=1993.23, stdev=18038.05
bw (KB/s) : min= 0, max=92584, per=0.95%, avg=3597.80, stdev=10280.93
cpu : usr=3.55%, sys=89.19%, ctx=35018, majf=0, minf=808
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3127029/1042699, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.34%, 500=5.55%, 750=10.81%, 1000=15.08%
lat (msec): 2=42.67%, 4=15.95%, 10=8.42%, 20=1.09%, 50=0.05%
lat (msec): 100=0.01%, 750=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22404
read : io=12,222MB, bw=78,710KB/s, iops=19,677, runt=159009msec
slat (usec): min=8, max=4,567K, avg=144.08, stdev=5171.90
clat (usec): min=3, max=4,580K, avg=1791.25, stdev=15586.55
bw (KB/s) : min= 0, max=149477, per=0.56%, avg=6403.28, stdev=23478.33
write: io=4,066MB, bw=26,183KB/s, iops=6,545, runt=159009msec
slat (usec): min=8, max=27,577, avg=139.01, stdev=276.79
clat (usec): min=3, max=4,578K, avg=1806.79, stdev=17360.04
bw (KB/s) : min= 0, max=50514, per=1.11%, avg=4193.45, stdev=10554.51
cpu : usr=3.16%, sys=89.71%, ctx=19195, majf=0, minf=832
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3128880/1040848, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.22%, 500=1.62%, 750=7.68%, 1000=16.51%
lat (msec): 2=54.73%, 4=13.06%, 10=5.40%, 20=0.70%, 50=0.03%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec): >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22408
read : io=12,218MB, bw=73,776KB/s, iops=18,443, runt=169590msec
slat (usec): min=8, max=21,677, avg=147.93, stdev=288.61
clat (usec): min=2, max=4,580K, avg=1925.74, stdev=18121.79
bw (KB/s) : min= 0, max=265544, per=0.49%, avg=5538.14, stdev=21748.68
write: io=4,070MB, bw=24,573KB/s, iops=6,143, runt=169590msec
slat (usec): min=9, max=21,676, avg=149.84, stdev=289.61
clat (usec): min=3, max=4,581K, avg=1926.44, stdev=18119.12
bw (KB/s) : min= 0, max=87992, per=1.00%, avg=3796.22, stdev=10025.61
cpu : usr=3.05%, sys=90.13%, ctx=26624, majf=0, minf=835
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3127908/1041820, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.24%, 500=2.71%, 750=8.41%, 1000=14.39%
lat (msec): 2=48.88%, 4=17.77%, 10=6.90%, 20=0.62%, 50=0.04%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22409
read : io=12,220MB, bw=76,303KB/s, iops=19,075, runt=163991msec
slat (usec): min=7, max=28,325, avg=143.55, stdev=277.69
clat (usec): min=2, max=4,583K, avg=1857.00, stdev=17230.21
bw (KB/s) : min= 0, max=149944, per=0.49%, avg=5618.48, stdev=21750.53
write: io=4,068MB, bw=25,403KB/s, iops=6,350, runt=163991msec
slat (usec): min=8, max=28,326, avg=144.78, stdev=275.74
clat (usec): min=3, max=4,584K, avg=1884.12, stdev=20111.38
bw (KB/s) : min= 0, max=50264, per=1.03%, avg=3919.65, stdev=10094.00
cpu : usr=3.28%, sys=90.13%, ctx=22085, majf=0, minf=842
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3128250/1041478, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.27%, 500=1.81%, 750=7.83%, 1000=15.74%
lat (msec): 2=50.39%, 4=17.16%, 10=6.24%, 20=0.49%, 50=0.02%
lat (msec): 100=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22410
read : io=12,224MB, bw=77,799KB/s, iops=19,449, runt=160898msec
slat (usec): min=8, max=26,825, avg=134.63, stdev=267.61
clat (usec): min=2, max=6,469K, avg=1858.52, stdev=31406.14
bw (KB/s) : min= 0, max=143512, per=0.57%, avg=6478.70, stdev=23685.21
write: io=4,064MB, bw=25,862KB/s, iops=6,465, runt=160898msec
slat (usec): min=9, max=32,224, avg=136.51, stdev=273.92
clat (usec): min=3, max=6,469K, avg=1846.93, stdev=30145.26
bw (KB/s) : min= 0, max=47816, per=1.06%, avg=4002.19, stdev=10380.51
cpu : usr=3.25%, sys=85.92%, ctx=20839, majf=0, minf=826
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3129436/1040292, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.26%, 500=1.70%, 750=7.91%, 1000=16.86%
lat (msec): 2=54.50%, 4=12.95%, 10=5.30%, 20=0.45%, 50=0.02%
lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22411
read : io=12,214MB, bw=73,153KB/s, iops=18,288, runt=170973msec
slat (usec): min=7, max=23,204, avg=149.32, stdev=292.93
clat (usec): min=2, max=4,580K, avg=1941.94, stdev=18542.04
bw (KB/s) : min= 0, max=221648, per=0.49%, avg=5527.81, stdev=22157.62
write: io=4,074MB, bw=24,400KB/s, iops=6,099, runt=170973msec
slat (usec): min=9, max=23,911, avg=151.42, stdev=297.10
clat (usec): min=3, max=4,581K, avg=1928.40, stdev=16230.37
bw (KB/s) : min= 0, max=73792, per=1.03%, avg=3896.15, stdev=10359.09
cpu : usr=3.16%, sys=90.52%, ctx=27202, majf=0, minf=821
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3126798/1042930, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.22%, 500=2.77%, 750=9.88%, 1000=15.46%
lat (msec): 2=46.07%, 4=16.83%, 10=8.07%, 20=0.64%, 50=0.02%
lat (msec): 100=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22415
read : io=12,209MB, bw=84,621KB/s, iops=21,155, runt=147744msec
slat (usec): min=8, max=18,538, avg=128.89, stdev=251.57
clat (usec): min=2, max=4,580K, avg=1671.90, stdev=17015.20
bw (KB/s) : min= 0, max=147800, per=0.63%, avg=7208.46, stdev=25218.41
write: io=4,079MB, bw=28,270KB/s, iops=7,067, runt=147744msec
slat (usec): min=8, max=16,877, avg=130.57, stdev=254.44
clat (usec): min=3, max=4,581K, avg=1701.15, stdev=20552.12
bw (KB/s) : min= 0, max=49320, per=1.21%, avg=4598.80, stdev=11206.49
cpu : usr=3.18%, sys=89.68%, ctx=18201, majf=0, minf=688
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3125565/1044163, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.23%, 500=1.61%, 750=8.22%, 1000=17.55%
lat (msec): 2=55.56%, 4=12.04%, 10=4.34%, 20=0.38%, 50=0.01%
lat (msec): 100=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22416
read : io=12,220MB, bw=77,554KB/s, iops=19,388, runt=161349msec
slat (usec): min=8, max=4,568K, avg=146.66, stdev=5171.61
clat (usec): min=2, max=4,577K, avg=1819.06, stdev=16623.13
bw (KB/s) : min= 0, max=153872, per=0.52%, avg=5954.67, stdev=22402.40
write: io=4,068MB, bw=25,818KB/s, iops=6,454, runt=161349msec
slat (usec): min=8, max=24,489, avg=143.50, stdev=274.33
clat (usec): min=3, max=4,577K, avg=1790.75, stdev=11943.83
bw (KB/s) : min= 0, max=51456, per=1.10%, avg=4178.47, stdev=10400.76
cpu : usr=2.97%, sys=90.56%, ctx=19828, majf=0, minf=815
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3128305/1041423, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.23%, 500=1.54%, 750=7.48%, 1000=15.73%
lat (msec): 2=52.17%, 4=16.62%, 10=5.74%, 20=0.44%, 50=0.02%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22417
read : io=12,214MB, bw=71,726KB/s, iops=17,931, runt=174381msec
slat (usec): min=8, max=4,569K, avg=155.66, stdev=4485.24
clat (usec): min=2, max=4,580K, avg=1961.31, stdev=16268.06
bw (KB/s) : min= 0, max=271928, per=0.45%, avg=5089.22, stdev=21511.23
write: io=4,074MB, bw=23,920KB/s, iops=5,980, runt=174381msec
slat (usec): min=9, max=4,569K, avg=157.96, stdev=4484.44
clat (usec): min=2, max=4,577K, avg=1951.27, stdev=15352.89
bw (KB/s) : min= 0, max=91808, per=0.96%, avg=3639.59, stdev=10176.32
cpu : usr=3.58%, sys=89.65%, ctx=32535, majf=0, minf=789
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3126909/1042819, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.24%, 500=4.60%, 750=10.31%, 1000=14.22%
lat (msec): 2=42.70%, 4=18.94%, 10=8.28%, 20=0.64%, 50=0.03%
lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22418
read : io=12,214MB, bw=75,810KB/s, iops=18,952, runt=164987msec
slat (usec): min=7, max=24,871, avg=140.88, stdev=271.37
clat (usec): min=2, max=4,586K, avg=1892.04, stdev=20020.35
bw (KB/s) : min= 0, max=192272, per=0.46%, avg=5245.63, stdev=21513.63
write: io=4,074MB, bw=25,283KB/s, iops=6,320, runt=164987msec
slat (usec): min=8, max=21,580, avg=143.11, stdev=280.82
clat (usec): min=2, max=4,594K, avg=1870.45, stdev=17160.68
bw (KB/s) : min= 0, max=63040, per=1.03%, avg=3904.71, stdev=10319.94
cpu : usr=3.28%, sys=88.37%, ctx=20950, majf=0, minf=837
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3126901/1042827, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.21%, 500=1.66%, 750=8.54%, 1000=16.80%
lat (msec): 2=50.22%, 4=16.03%, 10=5.92%, 20=0.53%, 50=0.02%
lat (msec): 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec): >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22420
read : io=12,220MB, bw=80,362KB/s, iops=20,090, runt=155706msec
slat (usec): min=7, max=24,355, avg=134.22, stdev=266.68
clat (usec): min=3, max=4,585K, avg=1778.65, stdev=19420.92
bw (KB/s) : min= 0, max=149488, per=0.53%, avg=5978.29, stdev=22997.10
write: io=4,068MB, bw=26,756KB/s, iops=6,689, runt=155706msec
slat (usec): min=8, max=22,745, avg=135.75, stdev=269.45
clat (usec): min=3, max=4,584K, avg=1769.07, stdev=19025.29
bw (KB/s) : min= 0, max=48416, per=1.12%, avg=4245.39, stdev=10743.45
cpu : usr=2.95%, sys=89.12%, ctx=18774, majf=0, minf=821
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3128207/1041521, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.19%, 500=1.49%, 750=8.28%, 1000=17.32%
lat (msec): 2=53.73%, 4=13.56%, 10=4.90%, 20=0.46%, 50=0.03%
lat (msec): 100=0.01%, 1000=0.01%, 2000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=22421
read : io=12,215MB, bw=77,001KB/s, iops=19,250, runt=162437msec
slat (usec): min=8, max=75,403, avg=139.99, stdev=284.59
clat (usec): min=3, max=4,581K, avg=1856.61, stdev=19203.62
bw (KB/s) : min= 0, max=132620, per=0.53%, avg=6009.79, stdev=22532.96
write: io=4,073MB, bw=25,679KB/s, iops=6,419, runt=162437msec
slat (usec): min=9, max=30,397, avg=141.63, stdev=280.34
clat (usec): min=2, max=4,580K, avg=1839.33, stdev=17043.77
bw (KB/s) : min= 0, max=44344, per=1.12%, avg=4259.86, stdev=10513.95
cpu : usr=3.07%, sys=89.02%, ctx=19530, majf=0, minf=830
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=3126935/1042793, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.23%, 500=1.50%, 750=7.08%, 1000=15.79%
lat (msec): 2=54.69%, 4=14.43%, 10=5.68%, 20=0.52%, 50=0.03%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec): 2000=0.01%, >=2000=0.01%
Run status group 0 (all jobs):
READ: io=191GB, aggrb=1,110MB/s, minb=72,746KB/s, maxb=86,750KB/s, mint=147590msec, maxt=176068msec
WRITE: io=65,150MB, aggrb=370MB/s, minb=24,257KB/s, maxb=28,970KB/s, mint=147590msec, maxt=176068msec
Disk stats (read/write):
sdb: ios=1572921/523974, merge=2/0, ticks=537483/175078, in_queue=712861, util=85.40%
sdc: ios=1572381/524466, merge=3/0, ticks=622035/219281, in_queue=843725, util=89.54%
sdd: ios=1571775/525069, merge=3/0, ticks=674714/213212, in_queue=888358, util=90.31%
sde: ios=1572525/524347, merge=1/0, ticks=573611/188605, in_queue=762312, util=86.67%
sdf: ios=1572598/524268, merge=5/0, ticks=542790/175335, in_queue=718138, util=85.61%
sdg: ios=1573381/523533, merge=4/1, ticks=586667/201351, in_queue=788474, util=88.03%
sdh: ios=1572095/524937, merge=1/0, ticks=642402/198541, in_queue=841663, util=89.24%
sdi: ios=1572401/524617, merge=3/1, ticks=673903/212615, in_queue=890034, util=89.97%
sdj: ios=1572545/524471, merge=7/1, ticks=546496/194615, in_queue=741977, util=86.54%
sdk: ios=1573740/523283, merge=1/1, ticks=611331/203176, in_queue=815062, util=88.85%
sdl: ios=1575187/521833, merge=5/1, ticks=668675/210578, in_queue=879803, util=89.98%
sdm: ios=1573163/523828, merge=2/1, ticks=691317/236949, in_queue=928990, util=90.67%
sdn: ios=1572525/524491, merge=5/0, ticks=642577/212384, in_queue=855473, util=89.73%
sdo: ios=1573043/523931, merge=3/0, ticks=676825/231416, in_queue=911619, util=90.71%
sdp: ios=1572846/524148, merge=4/0, ticks=545988/198146, in_queue=746201, util=85.28%
sdq: ios=1572211/524806, merge=4/0, ticks=700928/245643, in_queue=947201, util=90.80%
sdr: ios=1572668/524330, merge=2/1, ticks=1007038/312676, in_queue=1319680, util=86.56%
sds: ios=1571890/525076, merge=1/0, ticks=1049626/341068, in_queue=1391286, util=88.81%
sdt: ios=1572811/524164, merge=1/0, ticks=1062760/381623, in_queue=1444902, util=89.99%
sdu: ios=1573310/523634, merge=4/0, ticks=1107180/385919, in_queue=1495581, util=90.78%
sdv: ios=1572449/524536, merge=1/0, ticks=603415/191740, in_queue=797280, util=88.87%
sdw: ios=1572722/524199, merge=10/1, ticks=1209922/387813, in_queue=1594951, util=92.88%
sdx: ios=1535506/512159, merge=4/0, ticks=721691/235736, in_queue=958247, util=92.23%
sdy: ios=1535155/512484, merge=2/0, ticks=772392/248061, in_queue=1021180, util=92.78%
sdz: ios=1572242/524733, merge=3/0, ticks=716009/238652, in_queue=955588, util=91.40%
sdaa: ios=1536555/511103, merge=3/1, ticks=550871/172118, in_queue=723485, util=83.79%
sdab: ios=1535608/512067, merge=3/0, ticks=597814/197613, in_queue=797447, util=86.61%
sdac: ios=1535442/512209, merge=2/2, ticks=620642/227371, in_queue=848562, util=87.96%
sdad: ios=1571846/525120, merge=3/0, ticks=786443/251214, in_queue=1038692, util=92.43%
sdae: ios=1536538/511125, merge=5/0, ticks=614019/183964, in_queue=798341, util=85.99%
sdaf: ios=1536427/511230, merge=0/0, ticks=672041/202135, in_queue=874828, util=88.23%
sdag: ios=1536701/510949, merge=4/0, ticks=706537/223669, in_queue=930778, util=89.31%
[-- Attachment #7: fio-32x-rd_mcp-75-25-4k.txt --]
[-- Type: text/plain, Size: 22067 bytes --]
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
...
randrw: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=64
Starting 16 processes
randrw: (groupid=0, jobs=1): err= 0: pid=26389
read : io=4,072MB, bw=26,257KB/s, iops=6,564, runt=158809msec
slat (usec): min=8, max=21,714, avg=142.35, stdev=303.00
clat (usec): min=3, max=524K, avg=1775.66, stdev=2514.83
bw (KB/s) : min= 0, max=66808, per=1.02%, avg=4069.12, stdev=10513.33
write: io=12,216MB, bw=78,768KB/s, iops=19,692, runt=158809msec
slat (usec): min=8, max=23,481, avg=143.28, stdev=308.46
clat (usec): min=3, max=525K, avg=1779.98, stdev=2516.54
bw (KB/s) : min= 0, max=202776, per=0.46%, avg=5490.65, stdev=21989.41
cpu : usr=3.50%, sys=91.64%, ctx=22229, majf=0, minf=1157
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1042460/3127268, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.22%, 500=2.15%, 750=8.96%, 1000=16.28%
lat (msec): 2=49.38%, 4=16.35%, 10=6.02%, 20=0.56%, 50=0.05%
lat (msec): 100=0.01%, 500=0.01%, 750=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26391
read : io=4,067MB, bw=27,476KB/s, iops=6,869, runt=151562msec
slat (usec): min=8, max=21,141, avg=136.43, stdev=298.96
clat (usec): min=2, max=507K, avg=1699.46, stdev=2481.95
bw (KB/s) : min= 0, max=49680, per=1.11%, avg=4432.78, stdev=10904.19
write: io=12,221MB, bw=82,571KB/s, iops=20,642, runt=151562msec
slat (usec): min=8, max=31,711, avg=136.96, stdev=298.44
clat (usec): min=2, max=505K, avg=1698.63, stdev=2254.11
bw (KB/s) : min= 0, max=149704, per=0.50%, avg=6018.27, stdev=22991.82
cpu : usr=3.28%, sys=91.95%, ctx=19454, majf=0, minf=1215
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1041086/3128642, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.18%, 500=1.37%, 750=7.47%, 1000=16.83%
lat (msec): 2=55.37%, 4=13.29%, 10=4.86%, 20=0.55%, 50=0.04%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26392
read : io=4,069MB, bw=25,625KB/s, iops=6,406, runt=162584msec
slat (usec): min=8, max=114K, avg=139.80, stdev=331.06
clat (usec): min=2, max=5,816K, avg=1866.57, stdev=25145.87
bw (KB/s) : min= 0, max=69976, per=1.01%, avg=4024.77, stdev=10464.38
write: io=12,219MB, bw=76,961KB/s, iops=19,240, runt=162584msec
slat (usec): min=8, max=114K, avg=140.73, stdev=332.26
clat (usec): min=3, max=5,816K, avg=1843.61, stdev=22415.52
bw (KB/s) : min= 0, max=212664, per=0.51%, avg=6105.10, stdev=23134.68
cpu : usr=3.29%, sys=87.82%, ctx=24561, majf=0, minf=1227
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1041574/3128154, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.21%, 500=1.89%, 750=8.25%, 1000=16.66%
lat (msec): 2=52.42%, 4=14.36%, 10=5.45%, 20=0.66%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%, 1000=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26393
read : io=4,077MB, bw=29,171KB/s, iops=7,292, runt=143110msec
slat (usec): min=8, max=24,024, avg=129.42, stdev=287.12
clat (usec): min=3, max=76,375, avg=1602.96, stdev=1485.79
bw (KB/s) : min= 0, max=46000, per=1.15%, avg=4567.84, stdev=11275.85
write: io=12,211MB, bw=87,375KB/s, iops=21,843, runt=143110msec
slat (usec): min=7, max=28,954, avg=129.66, stdev=289.20
clat (usec): min=3, max=86,283, avg=1604.15, stdev=1486.02
bw (KB/s) : min= 0, max=141648, per=0.49%, avg=5842.31, stdev=23085.59
cpu : usr=3.47%, sys=91.80%, ctx=18402, majf=0, minf=1269
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1043661/3126067, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.18%, 500=1.41%, 750=7.94%, 1000=17.92%
lat (msec): 2=56.92%, 4=10.87%, 10=4.23%, 20=0.45%, 50=0.04%
lat (msec): 100=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26394
read : io=4,069MB, bw=25,230KB/s, iops=6,307, runt=165159msec
slat (usec): min=7, max=37,025, avg=148.65, stdev=330.70
clat (usec): min=3, max=258K, avg=1855.58, stdev=2150.01
bw (KB/s) : min= 0, max=87216, per=0.94%, avg=3740.50, stdev=10288.91
write: io=12,219MB, bw=75,757KB/s, iops=18,939, runt=165159msec
slat (usec): min=7, max=37,025, avg=148.70, stdev=323.55
clat (usec): min=2, max=258K, avg=1856.63, stdev=2106.39
bw (KB/s) : min= 0, max=260432, per=0.43%, avg=5113.10, stdev=21569.01
cpu : usr=3.43%, sys=91.52%, ctx=29755, majf=0, minf=1195
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1041750/3127978, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.27%, 500=3.88%, 750=10.66%, 1000=15.64%
lat (msec): 2=44.36%, 4=16.80%, 10=7.48%, 20=0.81%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, 500=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26395
read : io=4,074MB, bw=26,755KB/s, iops=6,688, runt=155915msec
slat (usec): min=8, max=30,012, avg=140.76, stdev=313.75
clat (usec): min=3, max=102K, avg=1756.64, stdev=1718.71
bw (KB/s) : min= 0, max=47073, per=1.06%, avg=4228.39, stdev=10600.94
write: io=12,214MB, bw=80,220KB/s, iops=20,054, runt=155915msec
slat (usec): min=8, max=30,013, avg=140.90, stdev=308.76
clat (usec): min=3, max=99,616, avg=1754.12, stdev=1707.37
bw (KB/s) : min= 0, max=139960, per=0.46%, avg=5502.34, stdev=21871.11
cpu : usr=3.29%, sys=91.78%, ctx=19704, majf=0, minf=1110
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1042872/3126856, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.23%, 500=1.53%, 750=7.69%, 1000=16.78%
lat (msec): 2=53.07%, 4=14.16%, 10=5.83%, 20=0.63%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26397
read : io=4,076MB, bw=24,896KB/s, iops=6,223, runt=167668msec
slat (usec): min=7, max=23,669, avg=144.32, stdev=324.33
clat (usec): min=3, max=7,095K, avg=1951.10, stdev=32621.56
bw (KB/s) : min= 0, max=106296, per=1.00%, avg=3968.05, stdev=10896.76
write: io=12,212MB, bw=74,580KB/s, iops=18,644, runt=167668msec
slat (usec): min=8, max=23,670, avg=144.54, stdev=321.38
clat (usec): min=3, max=7,095K, avg=1893.69, stdev=26063.31
bw (KB/s) : min= 0, max=319192, per=0.48%, avg=5726.40, stdev=23434.41
cpu : usr=3.28%, sys=87.65%, ctx=35641, majf=0, minf=1157
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1043562/3126166, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.30%, 500=4.96%, 750=10.55%, 1000=16.74%
lat (msec): 2=44.30%, 4=14.63%, 10=7.69%, 20=0.73%, 50=0.06%
lat (msec): 100=0.01%, 500=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26398
read : io=4,074MB, bw=26,382KB/s, iops=6,595, runt=158130msec
slat (usec): min=8, max=26,524, avg=142.40, stdev=307.29
clat (usec): min=3, max=115K, avg=1770.74, stdev=1656.08
bw (KB/s) : min= 0, max=51328, per=1.02%, avg=4078.83, stdev=10402.39
write: io=12,214MB, bw=79,094KB/s, iops=19,773, runt=158130msec
slat (usec): min=7, max=26,523, avg=142.96, stdev=310.90
clat (usec): min=2, max=114K, avg=1773.17, stdev=1681.89
bw (KB/s) : min= 0, max=151600, per=0.50%, avg=5960.00, stdev=22602.53
cpu : usr=3.42%, sys=91.96%, ctx=21402, majf=0, minf=1181
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1042930/3126798, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.22%, 500=1.51%, 750=7.43%, 1000=16.31%
lat (msec): 2=53.13%, 4=14.74%, 10=5.96%, 20=0.61%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26399
read : io=4,074MB, bw=26,949KB/s, iops=6,737, runt=154811msec
slat (usec): min=7, max=20,613, avg=138.20, stdev=291.93
clat (usec): min=3, max=1,559K, avg=1738.90, stdev=6465.14
bw (KB/s) : min= 0, max=56768, per=1.04%, avg=4154.38, stdev=10392.33
write: io=12,214MB, bw=80,788KB/s, iops=20,197, runt=154811msec
slat (usec): min=8, max=28,931, avg=138.65, stdev=295.06
clat (usec): min=2, max=1,559K, avg=1738.92, stdev=6225.32
bw (KB/s) : min= 0, max=166040, per=0.45%, avg=5376.50, stdev=21372.89
cpu : usr=3.34%, sys=90.87%, ctx=21127, majf=0, minf=1253
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1043004/3126724, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.22%, 500=1.49%, 750=6.76%, 1000=15.54%
lat (msec): 2=55.58%, 4=15.04%, 10=4.84%, 20=0.45%, 50=0.04%
lat (msec): 100=0.01%, 2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26400
read : io=4,076MB, bw=30,022KB/s, iops=7,505, runt=139038msec
slat (usec): min=8, max=28,097, avg=125.15, stdev=283.11
clat (usec): min=3, max=947K, avg=1562.21, stdev=3596.95
bw (KB/s) : min= 0, max=46864, per=1.25%, avg=4971.09, stdev=11911.15
write: io=12,212MB, bw=89,937KB/s, iops=22,484, runt=139038msec
slat (usec): min=7, max=28,101, avg=125.38, stdev=287.52
clat (usec): min=2, max=953K, avg=1564.01, stdev=4053.41
bw (KB/s) : min= 0, max=140096, per=0.54%, avg=6501.52, stdev=24703.32
cpu : usr=2.91%, sys=91.42%, ctx=17524, majf=0, minf=1217
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1043565/3126163, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.20%, 500=1.52%, 750=8.91%, 1000=19.37%
lat (msec): 2=56.11%, 4=9.57%, 10=3.78%, 20=0.47%, 50=0.04%
lat (msec): 100=0.01%, 250=0.01%, 1000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26402
read : io=4,073MB, bw=25,890KB/s, iops=6,472, runt=161111msec
slat (usec): min=7, max=27,305, avg=139.62, stdev=318.44
clat (usec): min=2, max=5,900K, avg=1862.48, stdev=26524.95
bw (KB/s) : min= 0, max=84736, per=1.02%, avg=4060.42, stdev=10760.98
write: io=12,215MB, bw=77,634KB/s, iops=19,408, runt=161111msec
slat (usec): min=7, max=27,304, avg=140.00, stdev=314.32
clat (usec): min=3, max=5,900K, avg=1824.57, stdev=21942.02
bw (KB/s) : min= 0, max=254176, per=0.50%, avg=5947.74, stdev=23346.07
cpu : usr=3.21%, sys=88.74%, ctx=26564, majf=0, minf=1211
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1042798/3126930, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.25%, 500=3.20%, 750=10.57%, 1000=16.23%
lat (msec): 2=49.86%, 4=12.94%, 10=6.07%, 20=0.81%, 50=0.05%
lat (msec): 100=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26404
read : io=4,073MB, bw=27,944KB/s, iops=6,986, runt=149244msec
slat (usec): min=8, max=28,453, avg=129.65, stdev=287.74
clat (usec): min=3, max=5,791K, avg=1688.61, stdev=21262.25
bw (KB/s) : min= 0, max=46312, per=1.14%, avg=4531.55, stdev=11079.47
write: io=12,215MB, bw=83,812KB/s, iops=20,952, runt=149244msec
slat (usec): min=8, max=28,453, avg=129.91, stdev=289.11
clat (usec): min=2, max=5,791K, avg=1705.47, stdev=23195.36
bw (KB/s) : min= 0, max=137752, per=0.49%, avg=5818.54, stdev=22757.79
cpu : usr=3.25%, sys=88.14%, ctx=19889, majf=0, minf=1209
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1042637/3127091, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.20%, 500=1.40%, 750=7.48%, 1000=17.11%
lat (msec): 2=57.57%, 4=11.92%, 10=3.77%, 20=0.48%, 50=0.04%
lat (msec): 100=0.01%, >=2000=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26406
read : io=4,075MB, bw=28,159KB/s, iops=7,039, runt=148181msec
slat (usec): min=8, max=31,522, avg=134.04, stdev=299.49
clat (usec): min=2, max=116K, avg=1661.66, stdev=1604.23
bw (KB/s) : min= 0, max=47120, per=1.09%, avg=4355.45, stdev=10927.55
write: io=12,213MB, bw=84,399KB/s, iops=21,099, runt=148181msec
slat (usec): min=8, max=31,523, avg=134.27, stdev=301.46
clat (usec): min=2, max=114K, avg=1661.16, stdev=1602.61
bw (KB/s) : min= 0, max=139544, per=0.49%, avg=5789.11, stdev=22745.02
cpu : usr=3.27%, sys=92.08%, ctx=18294, majf=0, minf=1250
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1043160/3126568, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.18%, 500=1.39%, 750=8.03%, 1000=17.68%
lat (msec): 2=54.94%, 4=12.61%, 10=4.52%, 20=0.59%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26408
read : io=4,070MB, bw=27,300KB/s, iops=6,825, runt=152645msec
slat (usec): min=7, max=24,869, avg=137.85, stdev=300.47
clat (usec): min=3, max=117K, avg=1711.33, stdev=1596.21
bw (KB/s) : min= 0, max=46984, per=1.08%, avg=4281.09, stdev=10787.26
write: io=12,218MB, bw=81,966KB/s, iops=20,491, runt=152645msec
slat (usec): min=7, max=28,302, avg=138.59, stdev=303.73
clat (usec): min=3, max=120K, avg=1712.23, stdev=1616.33
bw (KB/s) : min= 0, max=140408, per=0.53%, avg=6355.15, stdev=23655.92
cpu : usr=3.23%, sys=92.26%, ctx=18064, majf=0, minf=1262
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1041807/3127921, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.02%
lat (usec): 250=0.17%, 500=1.36%, 750=8.15%, 1000=17.41%
lat (msec): 2=52.64%, 4=14.22%, 10=5.51%, 20=0.47%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26411
read : io=4,075MB, bw=26,861KB/s, iops=6,715, runt=155350msec
slat (usec): min=7, max=26,509, avg=140.09, stdev=309.83
clat (usec): min=3, max=211K, avg=1747.01, stdev=1825.79
bw (KB/s) : min= 0, max=47392, per=1.05%, avg=4193.70, stdev=10657.56
write: io=12,213MB, bw=80,503KB/s, iops=20,125, runt=155350msec
slat (usec): min=7, max=26,509, avg=140.56, stdev=308.29
clat (usec): min=2, max=210K, avg=1745.35, stdev=1806.65
bw (KB/s) : min= 0, max=142480, per=0.47%, avg=5610.50, stdev=22231.49
cpu : usr=3.34%, sys=91.70%, ctx=19922, majf=0, minf=1234
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1043211/3126517, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.21%, 500=1.59%, 750=8.35%, 1000=17.44%
lat (msec): 2=50.94%, 4=15.06%, 10=5.78%, 20=0.54%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%
randrw: (groupid=0, jobs=1): err= 0: pid=26412
read : io=4,074MB, bw=26,382KB/s, iops=6,595, runt=158129msec
slat (usec): min=8, max=21,946, avg=140.47, stdev=301.38
clat (usec): min=3, max=1,713K, avg=1765.08, stdev=5577.94
bw (KB/s) : min= 0, max=48400, per=1.00%, avg=3978.68, stdev=10390.00
write: io=12,214MB, bw=79,094KB/s, iops=19,773, runt=158129msec
slat (usec): min=7, max=1,711K, avg=143.93, stdev=1960.83
clat (usec): min=2, max=1,713K, avg=1763.04, stdev=5231.15
bw (KB/s) : min= 0, max=145896, per=0.48%, avg=5754.93, stdev=22437.88
cpu : usr=3.35%, sys=90.99%, ctx=20147, majf=0, minf=1254
IO depths : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.1%, 16=0.1%, 32=87.5%, >=64=12.5%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=100.0%, 64=0.0%, >=64=0.0%
issued r/w: total=1042952/3126776, short=0/0
lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.03%
lat (usec): 250=0.24%, 500=1.80%, 750=8.69%, 1000=17.58%
lat (msec): 2=50.98%, 4=14.00%, 10=5.93%, 20=0.68%, 50=0.05%
lat (msec): 100=0.01%, 250=0.01%, 2000=0.01%
Run status group 0 (all jobs):
READ: io=65,168MB, aggrb=389MB/s, minb=25,493KB/s, maxb=30,742KB/s, mint=139038msec, maxt=167668msec
WRITE: io=191GB, aggrb=1,166MB/s, minb=76,369KB/s, maxb=92,095KB/s, mint=139038msec, maxt=167668msec
Disk stats (read/write):
sdb: ios=525124/1571776, merge=0/9, ticks=167454/525304, in_queue=692778, util=85.09%
sdc: ios=525064/1571786, merge=0/5, ticks=216926/605471, in_queue=824913, util=89.19%
sdd: ios=524042/1572890, merge=1/3, ticks=224349/650607, in_queue=875179, util=89.87%
sde: ios=523330/1573592, merge=0/6, ticks=186353/557050, in_queue=743680, util=86.38%
sdf: ios=524456/1572392, merge=0/2, ticks=167653/534467, in_queue=702611, util=85.26%
sdg: ios=524724/1572127, merge=0/2, ticks=193275/580095, in_queue=773745, util=87.41%
sdh: ios=524332/1572522, merge=0/2, ticks=198071/629634, in_queue=828185, util=88.50%
sdi: ios=524648/1572207, merge=0/3, ticks=217532/658179, in_queue=880456, util=89.23%
sdj: ios=525142/1571731, merge=0/3, ticks=173963/498673, in_queue=674092, util=84.72%
sdk: ios=524743/1572101, merge=0/3, ticks=196944/544797, in_queue=742591, util=87.07%
sdl: ios=524709/1572164, merge=0/2, ticks=198160/595749, in_queue=794145, util=88.25%
sdm: ios=524180/1572669, merge=1/2, ticks=201995/637815, in_queue=840439, util=88.98%
sdn: ios=522790/1574058, merge=0/5, ticks=194279/572101, in_queue=766629, util=88.04%
sdo: ios=524223/1572631, merge=1/2, ticks=223614/603127, in_queue=829504, util=89.10%
sdp: ios=525027/1571835, merge=1/6, ticks=163208/501028, in_queue=666764, util=83.48%
sdq: ios=523815/1573112, merge=1/3, ticks=214284/650445, in_queue=865288, util=89.34%
sdr: ios=524245/1572815, merge=0/5, ticks=182308/518986, in_queue=701251, util=85.35%
sds: ios=524671/1572372, merge=0/1, ticks=196275/569763, in_queue=766303, util=87.44%
sdt: ios=525275/1571749, merge=1/1, ticks=197651/617356, in_queue=815363, util=88.60%
sdu: ios=524187/1572848, merge=0/2, ticks=220648/644335, in_queue=868320, util=89.26%
sdv: ios=524712/1572326, merge=0/2, ticks=183421/559122, in_queue=744933, util=87.66%
sdw: ios=524056/1572911, merge=1/17, ticks=413381/1212283, in_queue=1623938, util=92.74%
sdx: ios=513201/1534943, merge=1/6, ticks=220390/669249, in_queue=890566, util=90.58%
sdy: ios=513000/1535168, merge=0/4, ticks=233635/717875, in_queue=952996, util=91.21%
sdz: ios=524205/1572814, merge=1/2, ticks=221065/648404, in_queue=870744, util=90.39%
sdaa: ios=511667/1536508, merge=0/3, ticks=157582/494725, in_queue=652308, util=82.51%
sdab: ios=511913/1536239, merge=0/0, ticks=166986/551124, in_queue=720456, util=85.29%
sdac: ios=513297/1534856, merge=0/1, ticks=198131/567311, in_queue=765924, util=86.60%
sdad: ios=524519/1572536, merge=1/0, ticks=231644/703731, in_queue=936830, util=91.69%
sdae: ios=511199/1536992, merge=0/3, ticks=184996/528796, in_queue=714042, util=85.03%
sdaf: ios=513087/1535093, merge=0/1, ticks=192418/587021, in_queue=780104, util=87.35%
sdag: ios=513544/1534626, merge=0/1, ticks=207118/621327, in_queue=828903, util=88.26%
[-- Attachment #8: Type: text/plain, Size: 183 bytes --]
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
^ permalink raw reply
* Re: virtio-scsi <-> vhost multi lun/adapter performance results with 3.6-rc0
From: Stefan Hajnoczi @ 2012-08-11 12:52 UTC (permalink / raw)
To: Nicholas A. Bellinger
Cc: linux-scsi, kvm-devel, Michael S. Tsirkin, qemu-devel, Jens Axboe,
LKML, lf-virt, James Bottomley, target-devel, Anthony Liguori,
Paolo Bonzini, Linus Torvalds, Christoph Hellwig
In-Reply-To: <1344641012.22564.294.camel@haakon2.linux-iscsi.org>
On Sat, Aug 11, 2012 at 12:23 AM, Nicholas A. Bellinger
<nab@linux-iscsi.org> wrote:
> Using a KVM guest with 32x vCPUs and 4G memory, the results for 4x
> random I/O now look like:
>
> workload | jobs | 25% write / 75% read | 75% write / 25% read
> -----------------|------|----------------------|---------------------
> 1x rd_mcp LUN | 8 | ~155K IOPs | ~145K IOPs
> 16x rd_mcp LUNs | 16 | ~315K IOPs | ~305K IOPs
> 32x rd_mcp LUNs | 16 | ~425K IOPs | ~410K IOPs
>
> The full fio randrw results for the six test cases are attached below.
> Also, using a workload of fio numjobs > 16 currently makes performance
> start to fall off pretty sharply regardless of the number of vCPUs..
>
> So running a similar workload with loopback SCSI ports on bare-metal
> produces ~1M random IOPs with 12x LUNs + numjobs=32. At numjobs=16 here
> with vhost the 16x LUN configuration ends up being in the range of ~310K
> IOPs for the current sweet spot..
This makes me wonder what a comparison against baremetal looks like
and the perf top, mpstat, and kvm_stat results on the host.
Stefan
^ permalink raw reply
* Re: [Question]About KVM network zero-copy feature!
From: Stephen Hemminger @ 2012-08-11 20:55 UTC (permalink / raw)
To: Peter Huang(Peng); +Cc: kvm, mst, netdev, virtualization, avi
In-Reply-To: <50248148.4070204@huawei.com>
On Fri, 10 Aug 2012 11:34:32 +0800
"Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:
> Hi,All
>
> I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
> about RX zero-copy?
>
> For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?
>
There is no guarantee that packet will ever be read by receiver. This means zero-copy could
create memory back pressure stalls.
^ permalink raw reply
* Re: [Question]About KVM network zero-copy feature!
From: Robert Vineyard @ 2012-08-11 21:54 UTC (permalink / raw)
To: Stephen Hemminger
Cc: kvm, mst, netdev, Peter Huang(Peng), virtualization, avi
In-Reply-To: <20120811135556.58ddf48d@nehalam.linuxnetplumber.net>
> On Fri, 10 Aug 2012 11:34:32 +0800
> "Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:
>
>> I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
>> about RX zero-copy?
On 08/11/2012 04:55 PM, Stephen Hemminger wrote:
> There is no guarantee that packet will ever be read by receiver. This means zero-copy could
> create memory back pressure stalls.
It would be handy if this could be an optional feature, perhaps not
enabled by default due to the problem with stalls you mentioned. I would
love to see RX zero-copy implemented natively in KVM, as it might
alleviate the need for custom solutions like vPF_RING:
http://www.ntop.org/products/pf_ring/vpf_ring/
Every time a packet is copied, especially from kernel space to user
space, there is an opportunity for it to be dropped on its way to the
receiving application - which is unacceptable when monitoring high-speed
networks for security or bandwidth accounting purposes.
I am attempting to find a highly-efficient way to deploy virtualized
network monitoring sensors (Snort, for example). Ideally I want to
exploit symmetric hardware-based RSS and SR-IOV functionality for
load-balancing and packet distribution completely in ASIC. I've found
other existing work in this area (also using custom drivers) indicating
significant performance gains in the non-virtualized case:
http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf
Is there any interest in exploring native RX zero-copy within the
mainline KVM networking code?
Thanks,
Robert Vineyard
^ permalink raw reply
* Re: [Question]About KVM network zero-copy feature!
From: Robert Vineyard @ 2012-08-12 0:42 UTC (permalink / raw)
To: Stephen Hemminger
Cc: xiaohui.xin, kvm, mst, netdev, Peter Huang(Peng), virtualization,
avi
In-Reply-To: <5026D49B.10305@tuffmail.com>
(adding Xin Xiaohui to the conversation for comment)
According to the NetworkingTodo page on the KVM wiki, zero-copy RX for
macvtap is in fact on the roadmap, assigned to Xin:
http://www.linux-kvm.org/page/NetworkingTodo
The Release Notes for RHEL 6.2 (originally published on 12/06/2011) also
specifically mention macvtap/vhost zero-copy capabilities as being
included as a Technology Preview:
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html
Since I've been unable to find much information on the current status of
this development, can anyone confirm if this functionality is still in
the works? If so, is there any planned ETA?
Thanks,
Robert Vineyard
On 08/11/2012 05:54 PM, Robert Vineyard wrote:
>> On Fri, 10 Aug 2012 11:34:32 +0800
>> "Peter Huang(Peng)" <peter.huangpeng@huawei.com> wrote:
>>
>>> I searched from git-log, and found that until now we have vhost TX
>>> zero-copy experiment feature, how
>>> about RX zero-copy?
>
> On 08/11/2012 04:55 PM, Stephen Hemminger wrote:
>> There is no guarantee that packet will ever be read by receiver. This
>> means zero-copy could
>> create memory back pressure stalls.
>
> It would be handy if this could be an optional feature, perhaps not
> enabled by default due to the problem with stalls you mentioned. I would
> love to see RX zero-copy implemented natively in KVM, as it might
> alleviate the need for custom solutions like vPF_RING:
>
> http://www.ntop.org/products/pf_ring/vpf_ring/
>
> Every time a packet is copied, especially from kernel space to user
> space, there is an opportunity for it to be dropped on its way to the
> receiving application - which is unacceptable when monitoring high-speed
> networks for security or bandwidth accounting purposes.
>
> I am attempting to find a highly-efficient way to deploy virtualized
> network monitoring sensors (Snort, for example). Ideally I want to
> exploit symmetric hardware-based RSS and SR-IOV functionality for
> load-balancing and packet distribution completely in ASIC. I've found
> other existing work in this area (also using custom drivers) indicating
> significant performance gains in the non-virtualized case:
>
> http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf
>
> Is there any interest in exploring native RX zero-copy within the
> mainline KVM networking code?
>
> Thanks,
> Robert Vineyard
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: [Question]About KVM network zero-copy feature!
From: Michael S. Tsirkin @ 2012-08-12 9:37 UTC (permalink / raw)
To: Robert Vineyard
Cc: xiaohui.xin, kvm, netdev, Peter Huang(Peng), virtualization, avi,
Stephen Hemminger
In-Reply-To: <5026FC04.6080108@tuffmail.com>
On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:
> (adding Xin Xiaohui to the conversation for comment)
>
> According to the NetworkingTodo page on the KVM wiki, zero-copy RX
> for macvtap is in fact on the roadmap, assigned to Xin:
>
> http://www.linux-kvm.org/page/NetworkingTodo
AFAIK Xin left Intel and is not working on it.
Contributions are welcome.
> The Release Notes for RHEL 6.2 (originally published on 12/06/2011)
> also specifically mention macvtap/vhost zero-copy capabilities as
> being included as a Technology Preview:
>
> http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html
>
> Since I've been unable to find much information on the current
> status of this development, can anyone confirm if this functionality
> is still in the works? If so, is there any planned ETA?
>
> Thanks,
> Robert Vineyard
I think this means TX.
--
MST
^ permalink raw reply
* Re: [PATCH v7 1/4] mm: introduce compaction and migration for virtio ballooned pages
From: Minchan Kim @ 2012-08-12 23:14 UTC (permalink / raw)
To: Rafael Aquini
Cc: Rik van Riel, Konrad Rzeszutek Wilk, Michael S. Tsirkin,
linux-kernel, virtualization, linux-mm, Andi Kleen, Andrew Morton
In-Reply-To: <292b1b52e863a05b299f94bda69a61371011ac19.1344619987.git.aquini@redhat.com>
On Fri, Aug 10, 2012 at 02:55:14PM -0300, Rafael Aquini wrote:
> Memory fragmentation introduced by ballooning might reduce significantly
> the number of 2MB contiguous memory blocks that can be used within a guest,
> thus imposing performance penalties associated with the reduced number of
> transparent huge pages that could be used by the guest workload.
>
> This patch introduces the helper functions as well as the necessary changes
> to teach compaction and migration bits how to cope with pages which are
> part of a guest memory balloon, in order to make them movable by memory
> compaction procedures.
>
> Signed-off-by: Rafael Aquini <aquini@redhat.com>
Reviewed-by: Minchan Kim <minchan@kernel.org>
--
Kind regards,
Minchan Kim
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox