* [Qemu-devel] [RFC PATCH v2 01/23] QEMUSizedBuffer/QEMUFile
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 02/23] configure: add CONFIG_COLO to switch COLO support Yang Hongyang
` (22 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: Stefan Berger, GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert,
mrhines, Joel Schopp, yanghy
From: "Dr. David Alan Gilbert" <dgilbert@redhat.com>
** Intended to merge separately; see the separate qemu-devel thread for
latest version
**
COLO buffer: Use QEMUSizedBuffer/QEMUFile to store migration data
We need a buffer to store migration data.
On save side:
all saved data is written into colo buffer first, so that we can know
the total size of the migration data. this can also separate the data
transmission from colo control data, we use colo control data over
socket fd to synchronous both side's stat.
On restore side:
all migration data was read into colo buffer first, then load data
from the buffer: If network error happens during data transmission,
the slave can still function because the migration data is not yet
loaded.
This is based on Stefan Berger's patch that creates a QEMUFile that goes
to a memory buffer; from:
http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html
Using the QEMUFile interface, this patch adds support functions for
operating on in-memory sized buffers that can be written to or read from.
Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>
Signed-off-by: Joel Schopp <jschopp@linux.vnet.ibm.com>
For minor tweeks/rebase I've done to it:
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
For commit messages about colo buffer:
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
include/migration/qemu-file.h | 28 +++
include/qemu/typedefs.h | 1 +
qemu-file.c | 410 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 439 insertions(+)
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index c90f529..80af3ff 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -25,6 +25,8 @@
#define QEMU_FILE_H 1
#include "exec/cpu-common.h"
+#include <stdint.h>
+
/* This function writes a chunk of data to a file at the given position.
* The pos argument can be ignored if the file is only being used for
* streaming. The handler should try to write all of the data it can.
@@ -94,11 +96,21 @@ typedef struct QEMUFileOps {
QEMURamSaveFunc *save_page;
} QEMUFileOps;
+struct QEMUSizedBuffer {
+ struct iovec *iov;
+ size_t n_iov;
+ size_t size; /* total allocated size in all iov's */
+ size_t used; /* number of used bytes */
+};
+
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;
+
QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops);
QEMUFile *qemu_fopen(const char *filename, const char *mode);
QEMUFile *qemu_fdopen(int fd, const char *mode);
QEMUFile *qemu_fopen_socket(int fd, const char *mode);
QEMUFile *qemu_popen_cmd(const char *command, const char *mode);
+QEMUFile *qemu_bufopen(const char *mode, QEMUSizedBuffer *input);
int qemu_get_fd(QEMUFile *f);
int qemu_fclose(QEMUFile *f);
int64_t qemu_ftell(QEMUFile *f);
@@ -111,6 +123,22 @@ void qemu_put_byte(QEMUFile *f, int v);
void qemu_put_buffer_async(QEMUFile *f, const uint8_t *buf, int size);
bool qemu_file_mode_is_not_valid(const char *mode);
+QEMUSizedBuffer *qsb_create(const uint8_t *buffer, size_t len);
+QEMUSizedBuffer *qsb_clone(const QEMUSizedBuffer *);
+void qsb_free(QEMUSizedBuffer *);
+size_t qsb_set_length(QEMUSizedBuffer *qsb, size_t length);
+size_t qsb_get_length(const QEMUSizedBuffer *qsb);
+ssize_t qsb_get_buffer(const QEMUSizedBuffer *, off_t start, size_t count,
+ uint8_t **buf);
+ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *buf,
+ off_t pos, size_t count);
+
+
+/*
+ * For use on files opened with qemu_bufopen
+ */
+const QEMUSizedBuffer *qemu_buf_get(QEMUFile *f);
+
static inline void qemu_put_ubyte(QEMUFile *f, unsigned int v)
{
qemu_put_byte(f, (int)v);
diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
index 5f20b0e..db1153a 100644
--- a/include/qemu/typedefs.h
+++ b/include/qemu/typedefs.h
@@ -60,6 +60,7 @@ typedef struct PCIEAERLog PCIEAERLog;
typedef struct PCIEAERErr PCIEAERErr;
typedef struct PCIEPort PCIEPort;
typedef struct PCIESlot PCIESlot;
+typedef struct QEMUSizedBuffer QEMUSizedBuffer;
typedef struct MSIMessage MSIMessage;
typedef struct SerialState SerialState;
typedef struct PCMCIACardState PCMCIACardState;
diff --git a/qemu-file.c b/qemu-file.c
index a8e3912..d64bee2 100644
--- a/qemu-file.c
+++ b/qemu-file.c
@@ -878,3 +878,413 @@ uint64_t qemu_get_be64(QEMUFile *f)
v |= qemu_get_be32(f);
return v;
}
+
+#define QSB_CHUNK_SIZE (1 << 10)
+#define QSB_MAX_CHUNK_SIZE (10 * QSB_CHUNK_SIZE)
+
+/**
+ * Create a QEMUSizedBuffer
+ * This type of buffer uses scatter-gather lists internally and
+ * can grow to any size. Any data array in the scatter-gather list
+ * can hold different amount of bytes.
+ *
+ * @buffer: Optional buffer to copy into the QSB
+ * @len: size of initial buffer; if @buffer is given, buffer must
+ * hold at least len bytes
+ *
+ * Returns a pointer to a QEMUSizedBuffer
+ */
+QEMUSizedBuffer *qsb_create(const uint8_t *buffer, size_t len)
+{
+ QEMUSizedBuffer *qsb;
+ size_t alloc_len, num_chunks, i, to_copy;
+ size_t chunk_size = (len > QSB_MAX_CHUNK_SIZE)
+ ? QSB_MAX_CHUNK_SIZE
+ : QSB_CHUNK_SIZE;
+
+ if (len == 0) {
+ /* we want to allocate at least one chunk */
+ len = QSB_CHUNK_SIZE;
+ }
+
+ num_chunks = DIV_ROUND_UP(len, chunk_size);
+ alloc_len = num_chunks * chunk_size;
+
+ qsb = g_new0(QEMUSizedBuffer, 1);
+ qsb->iov = g_new0(struct iovec, num_chunks);
+ qsb->n_iov = num_chunks;
+
+ for (i = 0; i < num_chunks; i++) {
+ qsb->iov[i].iov_base = g_malloc0(chunk_size);
+ qsb->iov[i].iov_len = chunk_size;
+ if (buffer) {
+ to_copy = (len - qsb->used) > chunk_size
+ ? chunk_size : (len - qsb->used);
+ memcpy(qsb->iov[i].iov_base, &buffer[qsb->used], to_copy);
+ qsb->used += to_copy;
+ }
+ }
+
+ qsb->size = alloc_len;
+
+ return qsb;
+}
+
+/**
+ * Free the QEMUSizedBuffer
+ *
+ * @qsb: The QEMUSizedBuffer to free
+ */
+void qsb_free(QEMUSizedBuffer *qsb)
+{
+ size_t i;
+
+ if (!qsb) {
+ return;
+ }
+
+ for (i = 0; i < qsb->n_iov; i++) {
+ g_free(qsb->iov[i].iov_base);
+ }
+ g_free(qsb->iov);
+ g_free(qsb);
+}
+
+/**
+ * Get the number of of used bytes in the QEMUSizedBuffer
+ *
+ * @qsb: A QEMUSizedBuffer
+ *
+ * Returns the number of bytes currently used in this buffer
+ */
+size_t qsb_get_length(const QEMUSizedBuffer *qsb)
+{
+ return qsb->used;
+}
+
+/**
+ * Set the length of the buffer; the primary usage of this
+ * function is to truncate the number of used bytes in the buffer.
+ * The size will not be extended beyond the current number of
+ * allocated bytes in the QEMUSizedBuffer.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @new_len : The new length of bytes in the buffer
+ *
+ * Returns the number of bytes the buffer was truncated or extended
+ * to.
+ */
+size_t qsb_set_length(QEMUSizedBuffer *qsb, size_t new_len)
+{
+ if (new_len <= qsb->size) {
+ qsb->used = new_len;
+ } else {
+ qsb->used = qsb->size;
+ }
+ return qsb->used;
+}
+
+/**
+ * Get the iovec that holds the data for a given position @pos.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @pos: The index of a byte in the buffer
+ * @d_off: Pointer to an offset that this function will indicate
+ * at what position within the returned iovec the byte
+ * is to be found
+ *
+ * Returns the index of the iovec that holds the byte at the given
+ * index @pos in the byte stream; a negative number if the iovec
+ * for the given position @pos does not exist.
+ */
+static ssize_t qsb_get_iovec(const QEMUSizedBuffer *qsb,
+ off_t pos, off_t *d_off)
+{
+ ssize_t i;
+ off_t curr = 0;
+
+ if (pos > qsb->used) {
+ return -1;
+ }
+
+ for (i = 0; i < qsb->n_iov; i++) {
+ if (curr + qsb->iov[i].iov_len > pos) {
+ *d_off = pos - curr;
+ return i;
+ }
+ curr += qsb->iov[i].iov_len;
+ }
+ return -1;
+}
+
+/*
+ * Convert the QEMUSizedBuffer into a flat buffer.
+ *
+ * Note: If at all possible, try to avoid this function since it
+ * may unnecessarily copy memory around.
+ *
+ * @qsb: pointer to QEMUSizedBuffer
+ * @start : offset to start at
+ * @count: number of bytes to copy
+ * @buf: a pointer to an optional buffer to write into; the pointer may
+ * point to NULL in which case the buffer will be allocated;
+ * if buffer is provided, it must be large enough to hold @count bytes
+ *
+ * Returns the number of bytes copied into the output buffer
+ */
+ssize_t qsb_get_buffer(const QEMUSizedBuffer *qsb, off_t start,
+ size_t count, uint8_t **buf)
+{
+ uint8_t *buffer;
+ const struct iovec *iov;
+ size_t to_copy, all_copy;
+ ssize_t index;
+ off_t s_off;
+ off_t d_off = 0;
+ char *s;
+
+ if (start > qsb->used) {
+ return 0;
+ }
+
+ all_copy = qsb->used - start;
+ if (all_copy > count) {
+ all_copy = count;
+ } else {
+ count = all_copy;
+ }
+
+ if (*buf == NULL) {
+ *buf = g_malloc(all_copy);
+ }
+ buffer = *buf;
+
+ index = qsb_get_iovec(qsb, start, &s_off);
+ if (index < 0) {
+ return 0;
+ }
+
+ while (all_copy > 0) {
+ iov = &qsb->iov[index];
+
+ s = iov->iov_base;
+
+ to_copy = iov->iov_len - s_off;
+ if (to_copy > all_copy) {
+ to_copy = all_copy;
+ }
+ memcpy(&buffer[d_off], &s[s_off], to_copy);
+
+ d_off += to_copy;
+ all_copy -= to_copy;
+
+ s_off = 0;
+ index++;
+ }
+
+ return count;
+}
+
+/**
+ * Grow the QEMUSizedBuffer to the given size and allocated
+ * memory for it.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @new_size: The new size of the buffer
+ *
+ * Returns an error code in case of memory allocation failure
+ * or the new size of the buffer otherwise. The returned size
+ * may be greater or equal to @new_size.
+ */
+static ssize_t qsb_grow(QEMUSizedBuffer *qsb, size_t new_size)
+{
+ size_t needed_chunks, i;
+ size_t chunk_size = QSB_CHUNK_SIZE;
+
+ if (qsb->size < new_size) {
+ needed_chunks = DIV_ROUND_UP(new_size - qsb->size,
+ chunk_size);
+
+ qsb->iov = g_realloc_n(qsb->iov, qsb->n_iov + needed_chunks,
+ sizeof(struct iovec));
+ if (qsb->iov == NULL) {
+ return -ENOMEM;
+ }
+
+ for (i = qsb->n_iov; i < qsb->n_iov + needed_chunks; i++) {
+ qsb->iov[i].iov_base = g_malloc0(chunk_size);
+ qsb->iov[i].iov_len = chunk_size;
+ }
+
+ qsb->n_iov += needed_chunks;
+ qsb->size += (needed_chunks * chunk_size);
+ }
+
+ return qsb->size;
+}
+
+/**
+ * Write into the QEMUSizedBuffer at a given position and a given
+ * number of bytes. This function will automatically grow the
+ * QEMUSizedBuffer.
+ *
+ * @qsb: A QEMUSizedBuffer
+ * @source: A byte array to copy data from
+ * @pos: The position withing the @qsb to write data to
+ * @size: The number of bytes to copy into the @qsb
+ *
+ * Returns an error code in case of memory allocation failure,
+ * @size otherwise.
+ */
+ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *source,
+ off_t pos, size_t count)
+{
+ ssize_t rc = qsb_grow(qsb, pos + count);
+ size_t to_copy;
+ size_t all_copy = count;
+ const struct iovec *iov;
+ ssize_t index;
+ char *dest;
+ off_t d_off, s_off = 0;
+
+ if (rc < 0) {
+ return rc;
+ }
+
+ if (pos + count > qsb->used) {
+ qsb->used = pos + count;
+ }
+
+ index = qsb_get_iovec(qsb, pos, &d_off);
+ if (index < 0) {
+ return 0;
+ }
+
+ while (all_copy > 0) {
+ iov = &qsb->iov[index];
+
+ dest = iov->iov_base;
+
+ to_copy = iov->iov_len - d_off;
+ if (to_copy > all_copy) {
+ to_copy = all_copy;
+ }
+
+ memcpy(&dest[d_off], &source[s_off], to_copy);
+
+ s_off += to_copy;
+ all_copy -= to_copy;
+
+ d_off = 0;
+ index++;
+ }
+
+ return count;
+}
+
+/**
+ * Create an exact copy of the given QEMUSizedBuffer.
+ *
+ * @qsb : A QEMUSizedBuffer
+ *
+ * Returns a clone of @qsb
+ */
+QEMUSizedBuffer *qsb_clone(const QEMUSizedBuffer *qsb)
+{
+ QEMUSizedBuffer *out = qsb_create(NULL, qsb_get_length(qsb));
+ size_t i;
+ off_t pos = 0;
+
+ for (i = 0; i < qsb->n_iov; i++) {
+ pos += qsb_write_at(out, qsb->iov[i].iov_base,
+ pos, qsb->iov[i].iov_len);
+ }
+
+ return out;
+}
+
+typedef struct QEMUBuffer {
+ QEMUSizedBuffer *qsb;
+ QEMUFile *file;
+} QEMUBuffer;
+
+static int buf_get_buffer(void *opaque, uint8_t *buf, int64_t pos, int size)
+{
+ QEMUBuffer *s = opaque;
+ ssize_t len = qsb_get_length(s->qsb) - pos;
+
+ if (len <= 0) {
+ return 0;
+ }
+
+ if (len > size) {
+ len = size;
+ }
+ return qsb_get_buffer(s->qsb, pos, len, &buf);
+}
+
+static int buf_put_buffer(void *opaque, const uint8_t *buf,
+ int64_t pos, int size)
+{
+ QEMUBuffer *s = opaque;
+
+ return qsb_write_at(s->qsb, buf, pos, size);
+}
+
+static int buf_close(void *opaque)
+{
+ QEMUBuffer *s = opaque;
+
+ qsb_free(s->qsb);
+
+ g_free(s);
+
+ return 0;
+}
+
+const QEMUSizedBuffer *qemu_buf_get(QEMUFile *f)
+{
+ QEMUBuffer *p;
+
+ qemu_fflush(f);
+
+ p = (QEMUBuffer *)f->opaque;
+
+ return p->qsb;
+}
+
+static const QEMUFileOps buf_read_ops = {
+ .get_buffer = buf_get_buffer,
+ .close = buf_close
+};
+
+static const QEMUFileOps buf_write_ops = {
+ .put_buffer = buf_put_buffer,
+ .close = buf_close
+};
+
+QEMUFile *qemu_bufopen(const char *mode, QEMUSizedBuffer *input)
+{
+ QEMUBuffer *s;
+
+ if (mode == NULL || (mode[0] != 'r' && mode[0] != 'w') || mode[1] != 0) {
+ fprintf(stderr, "qemu_bufopen: Argument validity check failed\n");
+ return NULL;
+ }
+
+ s = g_malloc0(sizeof(QEMUBuffer));
+ if (mode[0] == 'r') {
+ s->qsb = input;
+ }
+
+ if (s->qsb == NULL) {
+ s->qsb = qsb_create(NULL, 0);
+ }
+
+ if (mode[0] == 'r') {
+ s->file = qemu_fopen_ops(s, &buf_read_ops);
+ } else {
+ s->file = qemu_fopen_ops(s, &buf_write_ops);
+ }
+ return s->file;
+}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 02/23] configure: add CONFIG_COLO to switch COLO support
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 01/23] QEMUSizedBuffer/QEMUFile Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 03/23] COLO: introduce an api colo_supported() to indicate " Yang Hongyang
` (21 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
./configure --enable-colo/--disable-colo to switch COLO
support on/off.
COLO support is on by default.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
configure | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/configure b/configure
index 6c3d6cd..12635cd 100755
--- a/configure
+++ b/configure
@@ -258,6 +258,7 @@ xfs=""
vhost_net="no"
vhost_scsi="no"
kvm="no"
+colo="yes"
rdma=""
gprof="no"
debug_tcg="no"
@@ -923,6 +924,10 @@ for opt do
;;
--enable-kvm) kvm="yes"
;;
+ --disable-colo) colo="no"
+ ;;
+ --enable-colo) colo="yes"
+ ;;
--disable-tcg-interpreter) tcg_interpreter="no"
;;
--enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1320,6 +1325,10 @@ Advanced options (experts only):
--disable-slirp disable SLIRP userspace network connectivity
--disable-kvm disable KVM acceleration support
--enable-kvm enable KVM acceleration support
+ --disable-colo disable COarse-grain LOck-stepping Virtual
+ Machines for Non-stop Service
+ --enable-colo enable COarse-grain LOck-stepping Virtual
+ Machines for Non-stop Service (default)
--disable-rdma disable RDMA-based migration support
--enable-rdma enable RDMA-based migration support
--enable-tcg-interpreter enable TCG with bytecode interpreter (TCI)
@@ -4291,6 +4300,7 @@ echo "Linux AIO support $linux_aio"
echo "ATTR/XATTR support $attr"
echo "Install blobs $blobs"
echo "KVM support $kvm"
+echo "COLO support $colo"
echo "RDMA support $rdma"
echo "TCG interpreter $tcg_interpreter"
echo "fdt support $fdt"
@@ -4842,6 +4852,10 @@ if have_backend "ftrace"; then
fi
echo "CONFIG_TRACE_FILE=$trace_file" >> $config_host_mak
+if test "$colo" = "yes"; then
+ echo "CONFIG_COLO=y" >> $config_host_mak
+fi
+
if test "$rdma" = "yes" ; then
echo "CONFIG_RDMA=y" >> $config_host_mak
fi
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 03/23] COLO: introduce an api colo_supported() to indicate COLO support
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 01/23] QEMUSizedBuffer/QEMUFile Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 02/23] configure: add CONFIG_COLO to switch COLO support Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-10-08 15:02 ` Eric Blake
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 04/23] COLO migration: add a migration capability 'colo' Yang Hongyang
` (20 subsequent siblings)
23 siblings, 1 reply; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
introduce an api colo_supported() to indicate COLO support, returns
true if colo supported (configured with --enable-colo).
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
Makefile.objs | 1 +
include/migration/migration-colo.h | 18 ++++++++++++++++++
migration-colo.c | 16 ++++++++++++++++
stubs/Makefile.objs | 1 +
stubs/migration-colo.c | 16 ++++++++++++++++
5 files changed, 52 insertions(+)
create mode 100644 include/migration/migration-colo.h
create mode 100644 migration-colo.c
create mode 100644 stubs/migration-colo.c
diff --git a/Makefile.objs b/Makefile.objs
index 97db978..9654c04 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -49,6 +49,7 @@ common-obj-$(CONFIG_POSIX) += os-posix.o
common-obj-$(CONFIG_LINUX) += fsdev/
common-obj-y += migration.o migration-tcp.o
+common-obj-$(CONFIG_COLO) += migration-colo.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o
common-obj-$(CONFIG_RDMA) += migration-rdma.o
diff --git a/include/migration/migration-colo.h b/include/migration/migration-colo.h
new file mode 100644
index 0000000..35b384c
--- /dev/null
+++ b/include/migration/migration-colo.h
@@ -0,0 +1,18 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_MIGRATION_COLO_H
+#define QEMU_MIGRATION_COLO_H
+
+#include "qemu-common.h"
+
+bool colo_supported(void);
+
+#endif
diff --git a/migration-colo.c b/migration-colo.c
new file mode 100644
index 0000000..1d3bef8
--- /dev/null
+++ b/migration-colo.c
@@ -0,0 +1,16 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include "migration/migration-colo.h"
+
+bool colo_supported(void)
+{
+ return true;
+}
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 5e347d0..9fe6b4c 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -40,3 +40,4 @@ stub-obj-$(CONFIG_WIN32) += fd-register.o
stub-obj-y += cpus.o
stub-obj-y += kvm.o
stub-obj-y += qmp_pc_dimm_device_list.o
+stub-obj-y += migration-colo.o
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
new file mode 100644
index 0000000..b9ee6a0
--- /dev/null
+++ b/stubs/migration-colo.c
@@ -0,0 +1,16 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ */
+
+#include "migration/migration-colo.h"
+
+bool colo_supported(void)
+{
+ return false;
+}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* Re: [Qemu-devel] [RFC PATCH v2 03/23] COLO: introduce an api colo_supported() to indicate COLO support
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 03/23] COLO: introduce an api colo_supported() to indicate " Yang Hongyang
@ 2014-10-08 15:02 ` Eric Blake
2014-10-09 1:06 ` Wen Congyang
0 siblings, 1 reply; 36+ messages in thread
From: Eric Blake @ 2014-10-08 15:02 UTC (permalink / raw)
To: Yang Hongyang, qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines
[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]
On 09/23/2014 03:23 AM, Yang Hongyang wrote:
> introduce an api colo_supported() to indicate COLO support, returns
> true if colo supported (configured with --enable-colo).
>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> ---
> +++ b/include/migration/migration-colo.h
> @@ -0,0 +1,18 @@
> +/*
> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
> + * (a.k.a. Fault Tolerance or Continuous Replication)
> + *
> + * Copyright (C) 2014 FUJITSU LIMITED
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2. See
> + * the COPYING file in the top-level directory.
Is there any reason you are forbidding the use of this file in a GPLv3
project? We prefer new files to be GPLv2+, not GPLv2-only (by the use
of the "or later" clause), unless there is strong reason why it is not
possible.
> +++ b/migration-colo.c
> @@ -0,0 +1,16 @@
> +/*
> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
> + * (a.k.a. Fault Tolerance or Continuous Replication)
> + *
> + * Copyright (C) 2014 FUJITSU LIMITED
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2. See
> + * the COPYING file in the top-level directory.
Same comment for all new files.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 03/23] COLO: introduce an api colo_supported() to indicate COLO support
2014-10-08 15:02 ` Eric Blake
@ 2014-10-09 1:06 ` Wen Congyang
0 siblings, 0 replies; 36+ messages in thread
From: Wen Congyang @ 2014-10-09 1:06 UTC (permalink / raw)
To: Eric Blake, Yang Hongyang, qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines
On 10/08/2014 11:02 PM, Eric Blake wrote:
> On 09/23/2014 03:23 AM, Yang Hongyang wrote:
>> introduce an api colo_supported() to indicate COLO support, returns
>> true if colo supported (configured with --enable-colo).
>>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>> ---
>
>> +++ b/include/migration/migration-colo.h
>> @@ -0,0 +1,18 @@
>> +/*
>> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
>> + * (a.k.a. Fault Tolerance or Continuous Replication)
>> + *
>> + * Copyright (C) 2014 FUJITSU LIMITED
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2. See
>> + * the COPYING file in the top-level directory.
>
> Is there any reason you are forbidding the use of this file in a GPLv3
> project? We prefer new files to be GPLv2+, not GPLv2-only (by the use
> of the "or later" clause), unless there is strong reason why it is not
> possible.
I don't know the reason, and I think it is OK to be GPLv2+
Thanks
Wen Congyang
>
>
>> +++ b/migration-colo.c
>> @@ -0,0 +1,16 @@
>> +/*
>> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
>> + * (a.k.a. Fault Tolerance or Continuous Replication)
>> + *
>> + * Copyright (C) 2014 FUJITSU LIMITED
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2. See
>> + * the COPYING file in the top-level directory.
>
> Same comment for all new files.
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* [Qemu-devel] [RFC PATCH v2 04/23] COLO migration: add a migration capability 'colo'
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (2 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 03/23] COLO: introduce an api colo_supported() to indicate " Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 05/23] COLO info: use colo info to tell migration target colo is enabled Yang Hongyang
` (19 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
Add a migration capability 'colo'. If this capability is on,
The migration will never end, and the VM will be continuously
checkpointed.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration.c | 8 ++++++++
qapi-schema.json | 5 ++++-
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/migration.c b/migration.c
index 8d675b3..5d28237 100644
--- a/migration.c
+++ b/migration.c
@@ -25,6 +25,7 @@
#include "qemu/thread.h"
#include "qmp-commands.h"
#include "trace.h"
+#include "migration/migration-colo.h"
enum {
MIG_STATE_ERROR = -1,
@@ -277,6 +278,13 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
}
for (cap = params; cap; cap = cap->next) {
+ if (cap->value->capability == MIGRATION_CAPABILITY_COLO &&
+ cap->value->state && !colo_supported()) {
+ error_setg(errp, "COLO is not currently supported, please rerun"
+ " configure with --enable-colo option in order to"
+ " support COLO feature");
+ continue;
+ }
s->enabled_capabilities[cap->value->capability] = cap->value->state;
}
}
diff --git a/qapi-schema.json b/qapi-schema.json
index 689b548..8d58ef7 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -491,10 +491,13 @@
# @auto-converge: If enabled, QEMU will automatically throttle down the guest
# to speed up convergence of RAM migration. (since 1.6)
#
+# @colo: If enabled, the migration will never end, and the VM will instead be
+# continuously checkpointed. (since 2.2)
+#
# Since: 1.2
##
{ 'enum': 'MigrationCapability',
- 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] }
+ 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', 'colo'] }
##
# @MigrationCapabilityStatus
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 05/23] COLO info: use colo info to tell migration target colo is enabled
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (3 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 04/23] COLO migration: add a migration capability 'colo' Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 06/23] COLO save: integrate COLO checkpointed save into qemu migration Yang Hongyang
` (18 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
migrate colo info to migration target to tell the target colo is
enabled.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
Makefile.objs | 1 +
include/migration/migration-colo.h | 3 ++
migration-colo-comm.c | 68 ++++++++++++++++++++++++++++++++++++++
vl.c | 4 +++
4 files changed, 76 insertions(+)
create mode 100644 migration-colo-comm.c
diff --git a/Makefile.objs b/Makefile.objs
index 9654c04..e305444 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -49,6 +49,7 @@ common-obj-$(CONFIG_POSIX) += os-posix.o
common-obj-$(CONFIG_LINUX) += fsdev/
common-obj-y += migration.o migration-tcp.o
+common-obj-y += migration-colo-comm.o
common-obj-$(CONFIG_COLO) += migration-colo.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o
diff --git a/include/migration/migration-colo.h b/include/migration/migration-colo.h
index 35b384c..e3735d8 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -12,6 +12,9 @@
#define QEMU_MIGRATION_COLO_H
#include "qemu-common.h"
+#include "migration/migration.h"
+
+void colo_info_mig_init(void);
bool colo_supported(void);
diff --git a/migration-colo-comm.c b/migration-colo-comm.c
new file mode 100644
index 0000000..ccbc246
--- /dev/null
+++ b/migration-colo-comm.c
@@ -0,0 +1,68 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <migration/migration-colo.h>
+
+#define DEBUG_COLO
+
+#ifdef DEBUG_COLO
+#define DPRINTF(fmt, ...) \
+ do { fprintf(stdout, "COLO: " fmt, ## __VA_ARGS__); } while (0)
+#else
+#define DPRINTF(fmt, ...) \
+ do { } while (0)
+#endif
+
+static bool colo_requested;
+
+/* save */
+
+static bool migrate_use_colo(void)
+{
+ MigrationState *s = migrate_get_current();
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_COLO];
+}
+
+static void colo_info_save(QEMUFile *f, void *opaque)
+{
+ qemu_put_byte(f, migrate_use_colo());
+}
+
+/* restore */
+
+static int colo_info_load(QEMUFile *f, void *opaque, int version_id)
+{
+ int value = qemu_get_byte(f);
+
+ if (value && !colo_supported()) {
+ fprintf(stderr, "COLO is not supported\n");
+ return -EINVAL;
+ }
+
+ if (value && !colo_requested) {
+ DPRINTF("COLO requested!\n");
+ }
+
+ colo_requested = value;
+
+ return 0;
+}
+
+static SaveVMHandlers savevm_colo_info_handlers = {
+ .save_state = colo_info_save,
+ .load_state = colo_info_load,
+};
+
+void colo_info_mig_init(void)
+{
+ register_savevm_live(NULL, "colo info", -1, 1,
+ &savevm_colo_info_handlers, NULL);
+}
diff --git a/vl.c b/vl.c
index dc792fe..09d9851 100644
--- a/vl.c
+++ b/vl.c
@@ -89,6 +89,7 @@ int main(int argc, char **argv)
#include "sysemu/dma.h"
#include "audio/audio.h"
#include "migration/migration.h"
+#include "migration/migration-colo.h"
#include "sysemu/kvm.h"
#include "qapi/qmp/qjson.h"
#include "qemu/option.h"
@@ -4371,6 +4372,9 @@ int main(int argc, char **argv, char **envp)
blk_mig_init();
ram_mig_init();
+ if (colo_supported()) {
+ colo_info_mig_init();
+ }
/* open the virtual block devices */
if (snapshot)
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 06/23] COLO save: integrate COLO checkpointed save into qemu migration
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (4 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 05/23] COLO info: use colo info to tell migration target colo is enabled Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 07/23] COLO restore: integrate COLO checkpointed restore into qemu restore Yang Hongyang
` (17 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
Integrate COLO checkpointed save flow into qemu migration.
Add a migrate state: MIG_STATE_COLO, enter this migrate state
after the first live migration successfully finished.
Create a colo thread to do the checkpointed save.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
include/migration/migration-colo.h | 4 ++++
include/migration/migration.h | 13 +++++++++++
migration-colo-comm.c | 2 +-
migration-colo.c | 46 ++++++++++++++++++++++++++++++++++++++
migration.c | 36 ++++++++++++++++-------------
stubs/migration-colo.c | 4 ++++
6 files changed, 89 insertions(+), 16 deletions(-)
diff --git a/include/migration/migration-colo.h b/include/migration/migration-colo.h
index e3735d8..24589c0 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -18,4 +18,8 @@ void colo_info_mig_init(void);
bool colo_supported(void);
+/* save */
+bool migrate_use_colo(void);
+void colo_init_checkpointer(MigrationState *s);
+
#endif
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 3cb5ba8..3e81a27 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -64,6 +64,19 @@ struct MigrationState
int64_t dirty_sync_count;
};
+enum {
+ MIG_STATE_ERROR = -1,
+ MIG_STATE_NONE,
+ MIG_STATE_SETUP,
+ MIG_STATE_CANCELLING,
+ MIG_STATE_CANCELLED,
+ MIG_STATE_ACTIVE,
+ MIG_STATE_COLO,
+ MIG_STATE_COMPLETED,
+};
+
+void migrate_set_state(MigrationState *s, int old_state, int new_state);
+
void process_incoming_migration(QEMUFile *f);
void qemu_start_incoming_migration(const char *uri, Error **errp);
diff --git a/migration-colo-comm.c b/migration-colo-comm.c
index ccbc246..4504ceb 100644
--- a/migration-colo-comm.c
+++ b/migration-colo-comm.c
@@ -25,7 +25,7 @@ static bool colo_requested;
/* save */
-static bool migrate_use_colo(void)
+bool migrate_use_colo(void)
{
MigrationState *s = migrate_get_current();
return s->enabled_capabilities[MIGRATION_CAPABILITY_COLO];
diff --git a/migration-colo.c b/migration-colo.c
index 1d3bef8..c3fc313 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -8,9 +8,55 @@
* the COPYING file in the top-level directory.
*/
+#include "qemu/main-loop.h"
+#include "qemu/thread.h"
#include "migration/migration-colo.h"
+static QEMUBH *colo_bh;
+
bool colo_supported(void)
{
return true;
}
+
+/* save */
+
+static void *colo_thread(void *opaque)
+{
+ MigrationState *s = opaque;
+
+ /*TODO: COLO checkpointed save loop*/
+
+ migrate_set_state(s, MIG_STATE_COLO, MIG_STATE_COMPLETED);
+
+ qemu_mutex_lock_iothread();
+ qemu_bh_schedule(s->cleanup_bh);
+ qemu_mutex_unlock_iothread();
+
+ return NULL;
+}
+
+static void colo_start_checkpointer(void *opaque)
+{
+ MigrationState *s = opaque;
+
+ if (colo_bh) {
+ qemu_bh_delete(colo_bh);
+ colo_bh = NULL;
+ }
+
+ qemu_mutex_unlock_iothread();
+ qemu_thread_join(&s->thread);
+ qemu_mutex_lock_iothread();
+
+ migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_COLO);
+
+ qemu_thread_create(&s->thread, "colo", colo_thread, s,
+ QEMU_THREAD_JOINABLE);
+}
+
+void colo_init_checkpointer(MigrationState *s)
+{
+ colo_bh = qemu_bh_new(colo_start_checkpointer, s);
+ qemu_bh_schedule(colo_bh);
+}
diff --git a/migration.c b/migration.c
index 5d28237..6a6dce5 100644
--- a/migration.c
+++ b/migration.c
@@ -27,16 +27,6 @@
#include "trace.h"
#include "migration/migration-colo.h"
-enum {
- MIG_STATE_ERROR = -1,
- MIG_STATE_NONE,
- MIG_STATE_SETUP,
- MIG_STATE_CANCELLING,
- MIG_STATE_CANCELLED,
- MIG_STATE_ACTIVE,
- MIG_STATE_COMPLETED,
-};
-
#define MAX_THROTTLE (32 << 20) /* Migration speed throttling */
/* Amount of time to allocate to each "chunk" of bandwidth-throttled
@@ -229,6 +219,11 @@ MigrationInfo *qmp_query_migrate(Error **errp)
get_xbzrle_cache_stats(info);
break;
+ case MIG_STATE_COLO:
+ info->has_status = true;
+ info->status = g_strdup("colo");
+ /* TODO: display COLO specific informations(checkpoint info etc.),*/
+ break;
case MIG_STATE_COMPLETED:
get_xbzrle_cache_stats(info);
@@ -272,7 +267,8 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
MigrationState *s = migrate_get_current();
MigrationCapabilityStatusList *cap;
- if (s->state == MIG_STATE_ACTIVE || s->state == MIG_STATE_SETUP) {
+ if (s->state == MIG_STATE_ACTIVE || s->state == MIG_STATE_SETUP ||
+ s->state == MIG_STATE_COLO) {
error_set(errp, QERR_MIGRATION_ACTIVE);
return;
}
@@ -291,7 +287,7 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
/* shared migration helpers */
-static void migrate_set_state(MigrationState *s, int old_state, int new_state)
+void migrate_set_state(MigrationState *s, int old_state, int new_state)
{
if (atomic_cmpxchg(&s->state, old_state, new_state) == new_state) {
trace_migrate_set_state(new_state);
@@ -425,7 +421,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
params.shared = has_inc && inc;
if (s->state == MIG_STATE_ACTIVE || s->state == MIG_STATE_SETUP ||
- s->state == MIG_STATE_CANCELLING) {
+ s->state == MIG_STATE_CANCELLING || s->state == MIG_STATE_COLO) {
error_set(errp, QERR_MIGRATION_ACTIVE);
return;
}
@@ -593,6 +589,7 @@ static void *migration_thread(void *opaque)
int64_t max_size = 0;
int64_t start_time = initial_time;
bool old_vm_running = false;
+ bool use_colo = migrate_use_colo();
qemu_savevm_state_begin(s->file, &s->params);
@@ -629,7 +626,10 @@ static void *migration_thread(void *opaque)
}
if (!qemu_file_get_error(s->file)) {
- migrate_set_state(s, MIG_STATE_ACTIVE, MIG_STATE_COMPLETED);
+ if (!use_colo) {
+ migrate_set_state(s, MIG_STATE_ACTIVE,
+ MIG_STATE_COMPLETED);
+ }
break;
}
}
@@ -679,11 +679,17 @@ static void *migration_thread(void *opaque)
}
runstate_set(RUN_STATE_POSTMIGRATE);
} else {
+ if (s->state == MIG_STATE_ACTIVE && use_colo) {
+ colo_init_checkpointer(s);
+ }
if (old_vm_running) {
vm_start();
}
}
- qemu_bh_schedule(s->cleanup_bh);
+
+ if (!use_colo) {
+ qemu_bh_schedule(s->cleanup_bh);
+ }
qemu_mutex_unlock_iothread();
return NULL;
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index b9ee6a0..9013c40 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -14,3 +14,7 @@ bool colo_supported(void)
{
return false;
}
+
+void colo_init_checkpointer(MigrationState *s)
+{
+}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 07/23] COLO restore: integrate COLO checkpointed restore into qemu restore
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (5 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 06/23] COLO save: integrate COLO checkpointed save into qemu migration Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 08/23] COLO: disable qdev hotplug Yang Hongyang
` (16 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
enter colo checkpointed restore loop after live migration.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
include/migration/migration-colo.h | 6 ++++++
migration-colo-comm.c | 10 ++++++++++
migration-colo.c | 22 ++++++++++++++++++++++
migration.c | 3 +++
stubs/migration-colo.c | 4 ++++
5 files changed, 45 insertions(+)
diff --git a/include/migration/migration-colo.h b/include/migration/migration-colo.h
index 24589c0..861fa27 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -22,4 +22,10 @@ bool colo_supported(void);
bool migrate_use_colo(void);
void colo_init_checkpointer(MigrationState *s);
+/* restore */
+bool restore_use_colo(void);
+void restore_exit_colo(void);
+
+void colo_process_incoming_checkpoints(QEMUFile *f);
+
#endif
diff --git a/migration-colo-comm.c b/migration-colo-comm.c
index 4504ceb..b12a57a 100644
--- a/migration-colo-comm.c
+++ b/migration-colo-comm.c
@@ -38,6 +38,16 @@ static void colo_info_save(QEMUFile *f, void *opaque)
/* restore */
+bool restore_use_colo(void)
+{
+ return colo_requested;
+}
+
+void restore_exit_colo(void)
+{
+ colo_requested = false;
+}
+
static int colo_info_load(QEMUFile *f, void *opaque, int version_id)
{
int value = qemu_get_byte(f);
diff --git a/migration-colo.c b/migration-colo.c
index c3fc313..391e0b7 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -10,6 +10,7 @@
#include "qemu/main-loop.h"
#include "qemu/thread.h"
+#include "block/coroutine.h"
#include "migration/migration-colo.h"
static QEMUBH *colo_bh;
@@ -60,3 +61,24 @@ void colo_init_checkpointer(MigrationState *s)
colo_bh = qemu_bh_new(colo_start_checkpointer, s);
qemu_bh_schedule(colo_bh);
}
+
+/* restore */
+
+static Coroutine *colo;
+
+void colo_process_incoming_checkpoints(QEMUFile *f)
+{
+ if (!restore_use_colo()) {
+ return;
+ }
+
+ colo = qemu_coroutine_self();
+ assert(colo != NULL);
+
+ /* TODO: COLO checkpointed restore loop */
+
+ colo = NULL;
+ restore_exit_colo();
+
+ return;
+}
diff --git a/migration.c b/migration.c
index 6a6dce5..06165a7 100644
--- a/migration.c
+++ b/migration.c
@@ -86,6 +86,9 @@ static void process_incoming_migration_co(void *opaque)
int ret;
ret = qemu_loadvm_state(f);
+ if (!ret) {
+ colo_process_incoming_checkpoints(f);
+ }
qemu_fclose(f);
free_xbzrle_decoded_buf();
if (ret < 0) {
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 9013c40..55f0d37 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -18,3 +18,7 @@ bool colo_supported(void)
void colo_init_checkpointer(MigrationState *s)
{
}
+
+void colo_process_incoming_checkpoints(QEMUFile *f)
+{
+}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 08/23] COLO: disable qdev hotplug
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (6 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 07/23] COLO restore: integrate COLO checkpointed restore into qemu restore Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 09/23] COLO ctl: implement API's that communicate with colo agent Yang Hongyang
` (15 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
COLO do not support qdev hotplug migration, disable it.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/migration-colo.c b/migration-colo.c
index 391e0b7..1e6cd70 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -11,6 +11,7 @@
#include "qemu/main-loop.h"
#include "qemu/thread.h"
#include "block/coroutine.h"
+#include "hw/qdev-core.h"
#include "migration/migration-colo.h"
static QEMUBH *colo_bh;
@@ -25,6 +26,9 @@ bool colo_supported(void)
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
+ int dev_hotplug = qdev_hotplug;
+
+ qdev_hotplug = 0;
/*TODO: COLO checkpointed save loop*/
@@ -34,6 +38,8 @@ static void *colo_thread(void *opaque)
qemu_bh_schedule(s->cleanup_bh);
qemu_mutex_unlock_iothread();
+ qdev_hotplug = dev_hotplug;
+
return NULL;
}
@@ -68,10 +74,14 @@ static Coroutine *colo;
void colo_process_incoming_checkpoints(QEMUFile *f)
{
+ int dev_hotplug = qdev_hotplug;
+
if (!restore_use_colo()) {
return;
}
+ qdev_hotplug = 0;
+
colo = qemu_coroutine_self();
assert(colo != NULL);
@@ -80,5 +90,7 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
colo = NULL;
restore_exit_colo();
+ qdev_hotplug = dev_hotplug;
+
return;
}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 09/23] COLO ctl: implement API's that communicate with colo agent
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (7 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 08/23] COLO: disable qdev hotplug Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 10/23] COLO ctl: introduce is_slave() and is_master() Yang Hongyang
` (14 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
We use COLO agent to compare the packets returned by
Primary VM and Secondary VM, and decide whether to start a
checkpoint according to some rules. It is a linux kernel
module for host.
COLO controller communicate with the agent through ioctl().
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 117 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 114 insertions(+), 3 deletions(-)
diff --git a/migration-colo.c b/migration-colo.c
index 1e6cd70..8610e15 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -12,7 +12,17 @@
#include "qemu/thread.h"
#include "block/coroutine.h"
#include "hw/qdev-core.h"
+#include "qemu/timer.h"
#include "migration/migration-colo.h"
+#include <sys/ioctl.h>
+#include "qemu/error-report.h"
+
+/*
+ * checkpoint timer: unit ms
+ * this is large because COLO checkpoint will mostly depend on
+ * COLO compare module.
+ */
+#define CHKPOINT_TIMER 10000
static QEMUBH *colo_bh;
@@ -21,16 +31,100 @@ bool colo_supported(void)
return true;
}
+/* colo compare */
+#define COMP_IOC_MAGIC 'k'
+#define COMP_IOCTWAIT _IO(COMP_IOC_MAGIC, 0)
+#define COMP_IOCTFLUSH _IO(COMP_IOC_MAGIC, 1)
+#define COMP_IOCTRESUME _IO(COMP_IOC_MAGIC, 2)
+
+#define COMPARE_DEV "/dev/HA_compare"
+/* COLO compare module FD */
+static int comp_fd = -1;
+
+static int colo_compare_init(void)
+{
+ comp_fd = open(COMPARE_DEV, O_RDONLY);
+ if (comp_fd < 0) {
+ return -1;
+ }
+
+ return 0;
+}
+
+static void colo_compare_destroy(void)
+{
+ if (comp_fd >= 0) {
+ close(comp_fd);
+ comp_fd = -1;
+ }
+}
+
+/*
+ * Communicate with COLO Agent through ioctl.
+ * return:
+ * 0: start a checkpoint
+ * other: errno == ETIME or ERESTART, try again
+ * errno == other, error, quit colo save
+ */
+static int colo_compare(void)
+{
+ return ioctl(comp_fd, COMP_IOCTWAIT, 250);
+}
+
+static __attribute__((unused)) int colo_compare_flush(void)
+{
+ return ioctl(comp_fd, COMP_IOCTFLUSH, 1);
+}
+
+static __attribute__((unused)) int colo_compare_resume(void)
+{
+ return ioctl(comp_fd, COMP_IOCTRESUME, 1);
+}
+
/* save */
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
- int dev_hotplug = qdev_hotplug;
+ int dev_hotplug = qdev_hotplug, wait_cp = 0;
+ int64_t start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ int64_t current_time;
+
+ if (colo_compare_init() < 0) {
+ error_report("Init colo compare error");
+ goto out;
+ }
qdev_hotplug = 0;
- /*TODO: COLO checkpointed save loop*/
+ while (s->state == MIG_STATE_COLO) {
+ /* wait for a colo checkpoint */
+ wait_cp = colo_compare();
+ if (wait_cp) {
+ if (errno != ETIME && errno != ERESTART) {
+ error_report("compare module failed(%s)", strerror(errno));
+ goto out;
+ }
+ /*
+ * no checkpoint is needed, wait for 1ms and then
+ * check if we need checkpoint
+ */
+ current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ if (current_time - start_time < CHKPOINT_TIMER) {
+ usleep(1000);
+ continue;
+ }
+ }
+
+ /* start a colo checkpoint */
+
+ /*TODO: COLO save */
+
+ start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ }
+
+out:
+ colo_compare_destroy();
migrate_set_state(s, MIG_STATE_COLO, MIG_STATE_COMPLETED);
@@ -72,6 +166,17 @@ void colo_init_checkpointer(MigrationState *s)
static Coroutine *colo;
+/*
+ * return:
+ * 0: start a checkpoint
+ * 1: some error happend, exit colo restore
+ */
+static int slave_wait_new_checkpoint(QEMUFile *f)
+{
+ /* TODO: wait checkpoint start command from master */
+ return 1;
+}
+
void colo_process_incoming_checkpoints(QEMUFile *f)
{
int dev_hotplug = qdev_hotplug;
@@ -85,7 +190,13 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
colo = qemu_coroutine_self();
assert(colo != NULL);
- /* TODO: COLO checkpointed restore loop */
+ while (true) {
+ if (slave_wait_new_checkpoint(f)) {
+ break;
+ }
+
+ /* TODO: COLO restore */
+ }
colo = NULL;
restore_exit_colo();
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 10/23] COLO ctl: introduce is_slave() and is_master()
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (8 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 09/23] COLO ctl: implement API's that communicate with colo agent Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 11/23] COLO ctl: implement colo checkpoint protocol Yang Hongyang
` (13 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
is_slaver is to determine whether the QEMU instance is a
slaver(migration target) at runtime.
is_master is to determine whether the QEMU instance is a
master(migration starter) at runtime.
This 2 APIs will be used later.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/migration-colo.c b/migration-colo.c
index 8610e15..37003f5 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -83,6 +83,12 @@ static __attribute__((unused)) int colo_compare_resume(void)
/* save */
+static __attribute__((unused)) bool colo_is_master(void)
+{
+ MigrationState *s = migrate_get_current();
+ return (s->state == MIG_STATE_COLO);
+}
+
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
@@ -166,6 +172,11 @@ void colo_init_checkpointer(MigrationState *s)
static Coroutine *colo;
+static __attribute__((unused)) bool colo_is_slave(void)
+{
+ return colo != NULL;
+}
+
/*
* return:
* 0: start a checkpoint
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 11/23] COLO ctl: implement colo checkpoint protocol
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (9 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 10/23] COLO ctl: introduce is_slave() and is_master() Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 12/23] COLO ctl: add a RunState RUN_STATE_COLO Yang Hongyang
` (12 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
implement colo checkpoint protocol.
Checkpoint synchronzing points.
Primary Secondary
NEW @
Suspend
SUSPENDED @
Suspend&Save state
SEND @
Send state Receive state
RECEIVED @
Flush network Load state
LOADED @
Resume Resume
Start Comparing
NOTE:
1) '@' who sends the message
2) Every sync-point is synchronized by two sides with only
one handshake(single direction) for low-latency.
If more strict synchronization is required, a opposite direction
sync-point should be added.
3) Since sync-points are single direction, the remote side may
go forward a lot when this side just receives the sync-point.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 250 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 244 insertions(+), 6 deletions(-)
diff --git a/migration-colo.c b/migration-colo.c
index 37003f5..2e478e9 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -24,6 +24,41 @@
*/
#define CHKPOINT_TIMER 10000
+enum {
+ COLO_READY = 0x46,
+
+ /*
+ * Checkpoint synchronzing points.
+ *
+ * Primary Secondary
+ * NEW @
+ * Suspend
+ * SUSPENDED @
+ * Suspend&Save state
+ * SEND @
+ * Send state Receive state
+ * RECEIVED @
+ * Flush network Load state
+ * LOADED @
+ * Resume Resume
+ *
+ * Start Comparing
+ * NOTE:
+ * 1) '@' who sends the message
+ * 2) Every sync-point is synchronized by two sides with only
+ * one handshake(single direction) for low-latency.
+ * If more strict synchronization is required, a opposite direction
+ * sync-point should be added.
+ * 3) Since sync-points are single direction, the remote side may
+ * go forward a lot when this side just receives the sync-point.
+ */
+ COLO_CHECKPOINT_NEW,
+ COLO_CHECKPOINT_SUSPENDED,
+ COLO_CHECKPOINT_SEND,
+ COLO_CHECKPOINT_RECEIVED,
+ COLO_CHECKPOINT_LOADED,
+};
+
static QEMUBH *colo_bh;
bool colo_supported(void)
@@ -81,28 +116,159 @@ static __attribute__((unused)) int colo_compare_resume(void)
return ioctl(comp_fd, COMP_IOCTRESUME, 1);
}
+/* colo checkpoint control helper */
+static bool colo_is_master(void);
+static bool colo_is_slave(void);
+
+static void ctl_error_handler(void *opaque, int err)
+{
+ if (colo_is_slave()) {
+ /* TODO: determine whether we need to failover */
+ /* FIXME: we will not failover currently, just kill slave */
+ error_report("error: colo transmission failed!");
+ exit(1);
+ } else if (colo_is_master()) {
+ /* Master still alive, do not failover */
+ error_report("error: colo transmission failed!");
+ return;
+ } else {
+ error_report("COLO: Unexpected error happend!");
+ exit(EXIT_FAILURE);
+ }
+}
+
+static int colo_ctl_put(QEMUFile *f, uint64_t request)
+{
+ int ret = 0;
+
+ qemu_put_be64(f, request);
+ qemu_fflush(f);
+
+ ret = qemu_file_get_error(f);
+ if (ret < 0) {
+ ctl_error_handler(f, ret);
+ return 1;
+ }
+
+ return ret;
+}
+
+static int colo_ctl_get_value(QEMUFile *f, uint64_t *value)
+{
+ int ret = 0;
+ uint64_t temp;
+
+ temp = qemu_get_be64(f);
+
+ ret = qemu_file_get_error(f);
+ if (ret < 0) {
+ ctl_error_handler(f, ret);
+ return 1;
+ }
+
+ *value = temp;
+ return 0;
+}
+
+static int colo_ctl_get(QEMUFile *f, uint64_t require)
+{
+ int ret;
+ uint64_t value;
+
+ ret = colo_ctl_get_value(f, &value);
+ if (ret) {
+ return ret;
+ }
+
+ if (value != require) {
+ error_report("unexpected state! expected: %"PRIu64
+ ", received: %"PRIu64, require, value);
+ exit(1);
+ }
+
+ return ret;
+}
+
/* save */
-static __attribute__((unused)) bool colo_is_master(void)
+static bool colo_is_master(void)
{
MigrationState *s = migrate_get_current();
return (s->state == MIG_STATE_COLO);
}
+static int do_colo_transaction(MigrationState *s, QEMUFile *control)
+{
+ int ret;
+
+ ret = colo_ctl_put(s->file, COLO_CHECKPOINT_NEW);
+ if (ret) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(control, COLO_CHECKPOINT_SUSPENDED);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: suspend and save vm state to colo buffer */
+
+ ret = colo_ctl_put(s->file, COLO_CHECKPOINT_SEND);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: send vmstate to slave */
+
+ ret = colo_ctl_get(control, COLO_CHECKPOINT_RECEIVED);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: Flush network etc. */
+
+ ret = colo_ctl_get(control, COLO_CHECKPOINT_LOADED);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: resume master */
+
+out:
+ return ret;
+}
+
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
int dev_hotplug = qdev_hotplug, wait_cp = 0;
int64_t start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
int64_t current_time;
+ QEMUFile *colo_control = NULL;
+ int ret;
if (colo_compare_init() < 0) {
error_report("Init colo compare error");
goto out;
}
+ colo_control = qemu_fopen_socket(qemu_get_fd(s->file), "rb");
+ if (!colo_control) {
+ error_report("Open colo_control failed!");
+ goto out;
+ }
+
qdev_hotplug = 0;
+ /*
+ * Wait for slave finish loading vm states and enter COLO
+ * restore.
+ */
+ ret = colo_ctl_get(colo_control, COLO_READY);
+ if (ret) {
+ goto out;
+ }
+
while (s->state == MIG_STATE_COLO) {
/* wait for a colo checkpoint */
wait_cp = colo_compare();
@@ -124,12 +290,18 @@ static void *colo_thread(void *opaque)
/* start a colo checkpoint */
- /*TODO: COLO save */
+ if (do_colo_transaction(s, colo_control)) {
+ goto out;
+ }
start_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
}
out:
+ if (colo_control) {
+ qemu_fclose(colo_control);
+ }
+
colo_compare_destroy();
migrate_set_state(s, MIG_STATE_COLO, MIG_STATE_COMPLETED);
@@ -172,7 +344,7 @@ void colo_init_checkpointer(MigrationState *s)
static Coroutine *colo;
-static __attribute__((unused)) bool colo_is_slave(void)
+static bool colo_is_slave(void)
{
return colo != NULL;
}
@@ -184,13 +356,32 @@ static __attribute__((unused)) bool colo_is_slave(void)
*/
static int slave_wait_new_checkpoint(QEMUFile *f)
{
- /* TODO: wait checkpoint start command from master */
- return 1;
+ int fd = qemu_get_fd(f);
+ int ret;
+ uint64_t cmd;
+
+ yield_until_fd_readable(fd);
+
+ ret = colo_ctl_get_value(f, &cmd);
+ if (ret) {
+ return 1;
+ }
+
+ if (cmd == COLO_CHECKPOINT_NEW) {
+ return 0;
+ } else {
+ /* Unexpected data received */
+ ctl_error_handler(f, ret);
+ return 1;
+ }
}
void colo_process_incoming_checkpoints(QEMUFile *f)
{
+ int fd = qemu_get_fd(f);
int dev_hotplug = qdev_hotplug;
+ QEMUFile *ctl = NULL;
+ int ret;
if (!restore_use_colo()) {
return;
@@ -201,15 +392,62 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
colo = qemu_coroutine_self();
assert(colo != NULL);
+ ctl = qemu_fopen_socket(fd, "wb");
+ if (!ctl) {
+ error_report("Can't open incoming channel!");
+ goto out;
+ }
+
+ ret = colo_ctl_put(ctl, COLO_READY);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: in COLO mode, slave is runing, so start the vm */
+
while (true) {
if (slave_wait_new_checkpoint(f)) {
break;
}
- /* TODO: COLO restore */
+ /* start colo checkpoint */
+
+ /* TODO: suspend guest */
+
+ ret = colo_ctl_put(ctl, COLO_CHECKPOINT_SUSPENDED);
+ if (ret) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(f, COLO_CHECKPOINT_SEND);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: read migration data into colo buffer */
+
+ ret = colo_ctl_put(ctl, COLO_CHECKPOINT_RECEIVED);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: load vm state */
+
+ ret = colo_ctl_put(ctl, COLO_CHECKPOINT_LOADED);
+ if (ret) {
+ goto out;
+ }
+
+ /* TODO: resume guest */
}
+out:
colo = NULL;
+
+ if (ctl) {
+ qemu_fclose(ctl);
+ }
+
restore_exit_colo();
qdev_hotplug = dev_hotplug;
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 12/23] COLO ctl: add a RunState RUN_STATE_COLO
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (10 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 11/23] COLO ctl: implement colo checkpoint protocol Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 13/23] COLO ctl: implement colo save Yang Hongyang
` (11 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
Guest will enter this state when paused to save/restore VM state
under colo checkpoint.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
qapi-schema.json | 5 ++++-
vl.c | 8 ++++++++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/qapi-schema.json b/qapi-schema.json
index 8d58ef7..7dcfb90 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -145,12 +145,15 @@
# @watchdog: the watchdog action is configured to pause and has been triggered
#
# @guest-panicked: guest has been panicked as a result of guest OS panic
+#
+# @colo: guest is paused to save/restore VM state under colo checkpoint (since
+# 2.2)
##
{ 'enum': 'RunState',
'data': [ 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
- 'guest-panicked' ] }
+ 'guest-panicked', 'colo' ] }
##
# @StatusInfo:
diff --git a/vl.c b/vl.c
index 09d9851..b0de38e 100644
--- a/vl.c
+++ b/vl.c
@@ -619,6 +619,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_INMIGRATE, RUN_STATE_RUNNING },
{ RUN_STATE_INMIGRATE, RUN_STATE_PAUSED },
+ { RUN_STATE_INMIGRATE, RUN_STATE_COLO },
{ RUN_STATE_INTERNAL_ERROR, RUN_STATE_PAUSED },
{ RUN_STATE_INTERNAL_ERROR, RUN_STATE_FINISH_MIGRATE },
@@ -628,6 +629,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_PAUSED, RUN_STATE_RUNNING },
{ RUN_STATE_PAUSED, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_PAUSED, RUN_STATE_COLO},
{ RUN_STATE_POSTMIGRATE, RUN_STATE_RUNNING },
{ RUN_STATE_POSTMIGRATE, RUN_STATE_FINISH_MIGRATE },
@@ -638,9 +640,12 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_RUNNING },
{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_POSTMIGRATE },
+ { RUN_STATE_FINISH_MIGRATE, RUN_STATE_COLO},
{ RUN_STATE_RESTORE_VM, RUN_STATE_RUNNING },
+ { RUN_STATE_COLO, RUN_STATE_RUNNING },
+
{ RUN_STATE_RUNNING, RUN_STATE_DEBUG },
{ RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR },
{ RUN_STATE_RUNNING, RUN_STATE_IO_ERROR },
@@ -651,6 +656,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_RUNNING, RUN_STATE_SHUTDOWN },
{ RUN_STATE_RUNNING, RUN_STATE_WATCHDOG },
{ RUN_STATE_RUNNING, RUN_STATE_GUEST_PANICKED },
+ { RUN_STATE_RUNNING, RUN_STATE_COLO},
{ RUN_STATE_SAVE_VM, RUN_STATE_RUNNING },
@@ -661,9 +667,11 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_RUNNING, RUN_STATE_SUSPENDED },
{ RUN_STATE_SUSPENDED, RUN_STATE_RUNNING },
{ RUN_STATE_SUSPENDED, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_SUSPENDED, RUN_STATE_COLO},
{ RUN_STATE_WATCHDOG, RUN_STATE_RUNNING },
{ RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_WATCHDOG, RUN_STATE_COLO},
{ RUN_STATE_GUEST_PANICKED, RUN_STATE_RUNNING },
{ RUN_STATE_GUEST_PANICKED, RUN_STATE_FINISH_MIGRATE },
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 13/23] COLO ctl: implement colo save
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (11 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 12/23] COLO ctl: add a RunState RUN_STATE_COLO Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-10-08 10:23 ` Shunsuke Kurumatani
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 14/23] COLO ctl: implement colo restore Yang Hongyang
` (10 subsequent siblings)
23 siblings, 1 reply; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
implement colo save
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++-------
1 file changed, 53 insertions(+), 7 deletions(-)
diff --git a/migration-colo.c b/migration-colo.c
index 2e478e9..d99342a 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -13,6 +13,7 @@
#include "block/coroutine.h"
#include "hw/qdev-core.h"
#include "qemu/timer.h"
+#include "sysemu/sysemu.h"
#include "migration/migration-colo.h"
#include <sys/ioctl.h>
#include "qemu/error-report.h"
@@ -106,12 +107,12 @@ static int colo_compare(void)
return ioctl(comp_fd, COMP_IOCTWAIT, 250);
}
-static __attribute__((unused)) int colo_compare_flush(void)
+static int colo_compare_flush(void)
{
return ioctl(comp_fd, COMP_IOCTFLUSH, 1);
}
-static __attribute__((unused)) int colo_compare_resume(void)
+static int colo_compare_resume(void)
{
return ioctl(comp_fd, COMP_IOCTRESUME, 1);
}
@@ -200,6 +201,9 @@ static bool colo_is_master(void)
static int do_colo_transaction(MigrationState *s, QEMUFile *control)
{
int ret;
+ uint8_t *buf;
+ size_t size;
+ QEMUFile *trans = NULL;
ret = colo_ctl_put(s->file, COLO_CHECKPOINT_NEW);
if (ret) {
@@ -211,30 +215,73 @@ static int do_colo_transaction(MigrationState *s, QEMUFile *control)
goto out;
}
- /* TODO: suspend and save vm state to colo buffer */
+ /* open colo buffer for write */
+ trans = qemu_bufopen("w", NULL);
+ if (!trans) {
+ error_report("Open colo buffer for write failed");
+ goto out;
+ }
+
+ /* suspend and save vm state to colo buffer */
+ qemu_mutex_lock_iothread();
+ vm_stop_force_state(RUN_STATE_COLO);
+ qemu_mutex_unlock_iothread();
+ /* Disable block migration */
+ s->params.blk = 0;
+ s->params.shared = 0;
+ qemu_savevm_state_begin(trans, &s->params);
+ qemu_savevm_state_complete(trans);
+
+ qemu_fflush(trans);
ret = colo_ctl_put(s->file, COLO_CHECKPOINT_SEND);
if (ret) {
goto out;
}
- /* TODO: send vmstate to slave */
+ /* send vmstate to slave */
+
+ /* we send the total size of the vmstate first */
+ size = qsb_get_length(qemu_buf_get(trans));
+ ret = colo_ctl_put(s->file, size);
+ if (ret) {
+ goto out;
+ }
+
+ buf = g_malloc(size);
+ qsb_get_buffer(qemu_buf_get(trans), 0, size, &buf);
+ qemu_put_buffer(s->file, buf, size);
+ g_free(buf);
+ ret = qemu_file_get_error(s->file);
+ if (ret < 0) {
+ goto out;
+ }
+ qemu_fflush(s->file);
ret = colo_ctl_get(control, COLO_CHECKPOINT_RECEIVED);
if (ret) {
goto out;
}
- /* TODO: Flush network etc. */
+ /* Flush network etc. */
+ colo_compare_flush();
ret = colo_ctl_get(control, COLO_CHECKPOINT_LOADED);
if (ret) {
goto out;
}
- /* TODO: resume master */
+ colo_compare_resume();
+ ret = 0;
out:
+ if (trans)
+ qemu_fclose(trans);
+ /* resume master */
+ qemu_mutex_lock_iothread();
+ vm_start();
+ qemu_mutex_unlock_iothread();
+
return ret;
}
@@ -289,7 +336,6 @@ static void *colo_thread(void *opaque)
}
/* start a colo checkpoint */
-
if (do_colo_transaction(s, colo_control)) {
goto out;
}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* Re: [Qemu-devel] [RFC PATCH v2 13/23] COLO ctl: implement colo save
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 13/23] COLO ctl: implement colo save Yang Hongyang
@ 2014-10-08 10:23 ` Shunsuke Kurumatani
0 siblings, 0 replies; 36+ messages in thread
From: Shunsuke Kurumatani @ 2014-10-08 10:23 UTC (permalink / raw)
To: Yang Hongyang, qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines
Hi,
I tried and executed this exciting patches named colo. However this
patch causes abnormal termination in my environment. Although I
think it's a known issue, the details and a presumed origin is
described below:
On 2014/09/23 18:23, Yang Hongyang wrote:
> implement colo save
>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> ---
> migration-colo.c | 60 +++++++++++++++++++++++++++++++++++++++++++++++++-------
> 1 file changed, 53 insertions(+), 7 deletions(-)
>
> diff --git a/migration-colo.c b/migration-colo.c
> index 2e478e9..d99342a 100644
> --- a/migration-colo.c
> +++ b/migration-colo.c
> @@ -13,6 +13,7 @@
> #include "block/coroutine.h"
> #include "hw/qdev-core.h"
> #include "qemu/timer.h"
> +#include "sysemu/sysemu.h"
> #include "migration/migration-colo.h"
> #include <sys/ioctl.h>
> #include "qemu/error-report.h"
> @@ -106,12 +107,12 @@ static int colo_compare(void)
> return ioctl(comp_fd, COMP_IOCTWAIT, 250);
> }
>
> -static __attribute__((unused)) int colo_compare_flush(void)
> +static int colo_compare_flush(void)
> {
> return ioctl(comp_fd, COMP_IOCTFLUSH, 1);
> }
>
> -static __attribute__((unused)) int colo_compare_resume(void)
> +static int colo_compare_resume(void)
> {
> return ioctl(comp_fd, COMP_IOCTRESUME, 1);
> }
> @@ -200,6 +201,9 @@ static bool colo_is_master(void)
> static int do_colo_transaction(MigrationState *s, QEMUFile *control)
> {
> int ret;
> + uint8_t *buf;
> + size_t size;
> + QEMUFile *trans = NULL;
>
> ret = colo_ctl_put(s->file, COLO_CHECKPOINT_NEW);
> if (ret) {
> @@ -211,30 +215,73 @@ static int do_colo_transaction(MigrationState *s, QEMUFile *control)
> goto out;
> }
>
> - /* TODO: suspend and save vm state to colo buffer */
> + /* open colo buffer for write */
> + trans = qemu_bufopen("w", NULL);
> + if (!trans) {
> + error_report("Open colo buffer for write failed");
> + goto out;
> + }
> +
> + /* suspend and save vm state to colo buffer */
> + qemu_mutex_lock_iothread();
> + vm_stop_force_state(RUN_STATE_COLO);
> + qemu_mutex_unlock_iothread();
> + /* Disable block migration */
> + s->params.blk = 0;
> + s->params.shared = 0;
> + qemu_savevm_state_begin(trans, &s->params);
> + qemu_savevm_state_complete(trans);
This line causes aborting Qemu immediately after starting a colo's
migration process. If I'm not mistaken, the cause of aborting is not
getting mutex lock when calling qemu_savevm_state_complete(). The
aborting was resolved by getting mutex lock chen calling
qemu_save_state_complete().
Thanks,
Shunsuke
> +
> + qemu_fflush(trans);
>
> ret = colo_ctl_put(s->file, COLO_CHECKPOINT_SEND);
> if (ret) {
> goto out;
> }
>
> - /* TODO: send vmstate to slave */
> + /* send vmstate to slave */
> +
> + /* we send the total size of the vmstate first */
> + size = qsb_get_length(qemu_buf_get(trans));
> + ret = colo_ctl_put(s->file, size);
> + if (ret) {
> + goto out;
> + }
> +
> + buf = g_malloc(size);
> + qsb_get_buffer(qemu_buf_get(trans), 0, size, &buf);
> + qemu_put_buffer(s->file, buf, size);
> + g_free(buf);
> + ret = qemu_file_get_error(s->file);
> + if (ret < 0) {
> + goto out;
> + }
> + qemu_fflush(s->file);
>
> ret = colo_ctl_get(control, COLO_CHECKPOINT_RECEIVED);
> if (ret) {
> goto out;
> }
>
> - /* TODO: Flush network etc. */
> + /* Flush network etc. */
> + colo_compare_flush();
>
> ret = colo_ctl_get(control, COLO_CHECKPOINT_LOADED);
> if (ret) {
> goto out;
> }
>
> - /* TODO: resume master */
> + colo_compare_resume();
> + ret = 0;
>
> out:
> + if (trans)
> + qemu_fclose(trans);
> + /* resume master */
> + qemu_mutex_lock_iothread();
> + vm_start();
> + qemu_mutex_unlock_iothread();
> +
> return ret;
> }
>
> @@ -289,7 +336,6 @@ static void *colo_thread(void *opaque)
> }
>
> /* start a colo checkpoint */
> -
> if (do_colo_transaction(s, colo_control)) {
> goto out;
> }
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* [Qemu-devel] [RFC PATCH v2 14/23] COLO ctl: implement colo restore
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (12 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 13/23] COLO ctl: implement colo save Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 15/23] COLO save: reuse migration bitmap under colo checkpoint Yang Hongyang
` (9 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
implement colo restore
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 46 ++++++++++++++++++++++++++++++++++++++++------
1 file changed, 40 insertions(+), 6 deletions(-)
diff --git a/migration-colo.c b/migration-colo.c
index d99342a..91634d2 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -426,8 +426,11 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
{
int fd = qemu_get_fd(f);
int dev_hotplug = qdev_hotplug;
- QEMUFile *ctl = NULL;
+ QEMUFile *ctl = NULL, *fb = NULL;
int ret;
+ uint64_t total_size;
+ uint8_t *buf;
+ QEMUSizedBuffer *colo_buffer;
if (!restore_use_colo()) {
return;
@@ -449,7 +452,8 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
goto out;
}
- /* TODO: in COLO mode, slave is runing, so start the vm */
+ /* in COLO mode, slave is runing, so start the vm */
+ vm_start();
while (true) {
if (slave_wait_new_checkpoint(f)) {
@@ -458,7 +462,8 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
/* start colo checkpoint */
- /* TODO: suspend guest */
+ /* suspend guest */
+ vm_stop_force_state(RUN_STATE_COLO);
ret = colo_ctl_put(ctl, COLO_CHECKPOINT_SUSPENDED);
if (ret) {
@@ -470,26 +475,55 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
goto out;
}
- /* TODO: read migration data into colo buffer */
+ /* read migration data into colo buffer */
+
+ /* read the vmstate total size first */
+ ret = colo_ctl_get_value(f, &total_size);
+ if (ret) {
+ goto out;
+ }
+ buf = g_malloc(total_size);
+ qemu_get_buffer(f, buf, total_size);
+ colo_buffer = qsb_create(buf, total_size);
+ g_free(buf);
ret = colo_ctl_put(ctl, COLO_CHECKPOINT_RECEIVED);
if (ret) {
goto out;
}
- /* TODO: load vm state */
+ /* open colo buffer for read */
+ fb = qemu_bufopen("r", colo_buffer);
+ if (!fb) {
+ error_report("can't open colo buffer for read");
+ goto out;
+ }
+
+ /* load vm state */
+ if (qemu_loadvm_state(fb) < 0) {
+ error_report("COLO: loadvm failed\n");
+ goto out;
+ }
ret = colo_ctl_put(ctl, COLO_CHECKPOINT_LOADED);
if (ret) {
goto out;
}
- /* TODO: resume guest */
+ /* resume guest */
+ vm_start();
+
+ qemu_fclose(fb);
+ fb = NULL;
}
out:
colo = NULL;
+ if (fb) {
+ qemu_fclose(fb);
+ }
+
if (ctl) {
qemu_fclose(ctl);
}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 15/23] COLO save: reuse migration bitmap under colo checkpoint
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (13 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 14/23] COLO ctl: implement colo restore Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 16/23] COLO ram cache: implement colo ram cache on slave Yang Hongyang
` (8 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
reuse migration bitmap under colo checkpoint, only send dirty pages
per-checkpoint.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
arch_init.c | 20 +++++++++++++++++++-
include/migration/migration-colo.h | 2 ++
migration-colo.c | 6 ++----
stubs/migration-colo.c | 10 ++++++++++
4 files changed, 33 insertions(+), 5 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index c974f3f..00d5793 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -52,6 +52,7 @@
#include "exec/ram_addr.h"
#include "hw/acpi/acpi.h"
#include "qemu/host-utils.h"
+#include "migration/migration-colo.h"
#ifdef DEBUG_ARCH_INIT
#define DPRINTF(fmt, ...) \
@@ -771,6 +772,15 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
RAMBlock *block;
int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */
+ /*
+ * migration has already setup the bitmap, reuse it.
+ */
+ if (colo_is_master()) {
+ qemu_mutex_lock_ramlist();
+ reset_ram_globals();
+ goto out_setup;
+ }
+
mig_throttle_on = false;
dirty_rate_high_cnt = 0;
bitmap_sync_count = 0;
@@ -830,6 +840,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
migration_bitmap_sync();
qemu_mutex_unlock_iothread();
+out_setup:
qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
QTAILQ_FOREACH(block, &ram_list.blocks, next) {
@@ -939,7 +950,14 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
}
ram_control_after_iterate(f, RAM_CONTROL_FINISH);
- migration_end();
+
+ /*
+ * Since we need to reuse dirty bitmap in colo,
+ * don't cleanup the bitmap.
+ */
+ if (!migrate_use_colo() || migration_has_failed(migrate_get_current())) {
+ migration_end();
+ }
qemu_mutex_unlock_ramlist();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
diff --git a/include/migration/migration-colo.h b/include/migration/migration-colo.h
index 861fa27..9b82b71 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -21,10 +21,12 @@ bool colo_supported(void);
/* save */
bool migrate_use_colo(void);
void colo_init_checkpointer(MigrationState *s);
+bool colo_is_master(void);
/* restore */
bool restore_use_colo(void);
void restore_exit_colo(void);
+bool colo_is_slave(void);
void colo_process_incoming_checkpoints(QEMUFile *f);
diff --git a/migration-colo.c b/migration-colo.c
index 91634d2..8477fbd 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -118,8 +118,6 @@ static int colo_compare_resume(void)
}
/* colo checkpoint control helper */
-static bool colo_is_master(void);
-static bool colo_is_slave(void);
static void ctl_error_handler(void *opaque, int err)
{
@@ -192,7 +190,7 @@ static int colo_ctl_get(QEMUFile *f, uint64_t require)
/* save */
-static bool colo_is_master(void)
+bool colo_is_master(void)
{
MigrationState *s = migrate_get_current();
return (s->state == MIG_STATE_COLO);
@@ -390,7 +388,7 @@ void colo_init_checkpointer(MigrationState *s)
static Coroutine *colo;
-static bool colo_is_slave(void)
+bool colo_is_slave(void)
{
return colo != NULL;
}
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 55f0d37..4828316 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -22,3 +22,13 @@ void colo_init_checkpointer(MigrationState *s)
void colo_process_incoming_checkpoints(QEMUFile *f)
{
}
+
+bool colo_is_master(void)
+{
+ return false;
+}
+
+bool colo_is_slave(void)
+{
+ return false;
+}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 16/23] COLO ram cache: implement colo ram cache on slave
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (14 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 15/23] COLO save: reuse migration bitmap under colo checkpoint Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 17/23] HACK: trigger checkpoint every 500ms Yang Hongyang
` (7 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
The ram cache was initially the same as PVM's memory. At
checkpoint, we cache the dirty memory of PVM into ram cache
(so that ram cache always the same as PVM's memory at every
checkpoint), flush cached memory to SVM after we received
all PVM dirty memory (only needed to flush memory that was
both dirty on PVM and SVM since last checkpoint).
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
arch_init.c | 154 ++++++++++++++++++++++++++++++++++++-
include/exec/cpu-all.h | 1 +
include/migration/migration-colo.h | 3 +
migration-colo.c | 4 +
4 files changed, 159 insertions(+), 3 deletions(-)
diff --git a/arch_init.c b/arch_init.c
index 00d5793..108cca0 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -1015,6 +1015,7 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
return 0;
}
+static void *memory_region_get_ram_cache_ptr(MemoryRegion *mr, RAMBlock *block);
static inline void *host_from_stream_offset(QEMUFile *f,
ram_addr_t offset,
int flags)
@@ -1029,7 +1030,12 @@ static inline void *host_from_stream_offset(QEMUFile *f,
return NULL;
}
- return memory_region_get_ram_ptr(block->mr) + offset;
+ if (colo_is_slave()) {
+ migration_bitmap_set_dirty(block->mr->ram_addr + offset);
+ return memory_region_get_ram_cache_ptr(block->mr, block) + offset;
+ } else {
+ return memory_region_get_ram_ptr(block->mr) + offset;
+ }
}
len = qemu_get_byte(f);
@@ -1037,8 +1043,15 @@ static inline void *host_from_stream_offset(QEMUFile *f,
id[len] = 0;
QTAILQ_FOREACH(block, &ram_list.blocks, next) {
- if (!strncmp(id, block->idstr, sizeof(id)))
- return memory_region_get_ram_ptr(block->mr) + offset;
+ if (!strncmp(id, block->idstr, sizeof(id))) {
+ if (colo_is_slave()) {
+ migration_bitmap_set_dirty(block->mr->ram_addr + offset);
+ return memory_region_get_ram_cache_ptr(block->mr, block)
+ + offset;
+ } else {
+ return memory_region_get_ram_ptr(block->mr) + offset;
+ }
+ }
}
error_report("Can't find block %s!", id);
@@ -1056,11 +1069,13 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)
}
}
+static void ram_flush_cache(void);
static int ram_load(QEMUFile *f, void *opaque, int version_id)
{
ram_addr_t addr;
int flags, ret = 0;
static uint64_t seq_iter;
+ bool need_flush = false;
seq_iter++;
@@ -1123,6 +1138,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
break;
}
+ need_flush = true;
ch = qemu_get_byte(f);
ram_handle_compressed(host, ch, TARGET_PAGE_SIZE);
} else if (flags & RAM_SAVE_FLAG_PAGE) {
@@ -1135,6 +1151,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
break;
}
+ need_flush = true;
qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
} else if (flags & RAM_SAVE_FLAG_XBZRLE) {
void *host = host_from_stream_offset(f, addr, flags);
@@ -1150,6 +1167,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
ret = -EINVAL;
break;
}
+ need_flush = true;
} else if (flags & RAM_SAVE_FLAG_HOOK) {
ram_control_load_hook(f, flags);
} else if (flags & RAM_SAVE_FLAG_EOS) {
@@ -1163,11 +1181,141 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
ret = qemu_file_get_error(f);
}
+ if (!ret && colo_is_slave() && need_flush) {
+ ram_flush_cache();
+ }
+
DPRINTF("Completed load of VM with exit code %d seq iteration "
"%" PRIu64 "\n", ret, seq_iter);
return ret;
}
+/*
+ * colo cache: this is for secondary VM, we cache the whole
+ * memory of the secondary VM.
+ */
+void create_and_init_ram_cache(void)
+{
+ /*
+ * called after first migration
+ */
+ RAMBlock *block;
+ int64_t ram_cache_pages = last_ram_offset() >> TARGET_PAGE_BITS;
+
+ QTAILQ_FOREACH(block, &ram_list.blocks, next) {
+ block->host_cache = g_malloc(block->length);
+ memcpy(block->host_cache, block->host, block->length);
+ }
+
+ migration_bitmap = bitmap_new(ram_cache_pages);
+ migration_dirty_pages = 0;
+ memory_global_dirty_log_start();
+}
+
+void release_ram_cache(void)
+{
+ RAMBlock *block;
+
+ if (migration_bitmap) {
+ memory_global_dirty_log_stop();
+ g_free(migration_bitmap);
+ migration_bitmap = NULL;
+ }
+
+ QTAILQ_FOREACH(block, &ram_list.blocks, next) {
+ g_free(block->host_cache);
+ }
+}
+
+static void *memory_region_get_ram_cache_ptr(MemoryRegion *mr, RAMBlock *block)
+{
+ if (mr->alias) {
+ return memory_region_get_ram_cache_ptr(mr->alias, block) +
+ mr->alias_offset;
+ }
+
+ assert(mr->terminates);
+
+ ram_addr_t addr = mr->ram_addr & TARGET_PAGE_MASK;
+
+ assert(addr - block->offset < block->length);
+
+ return block->host_cache + (addr - block->offset);
+}
+
+static inline
+ram_addr_t host_bitmap_find_and_reset_dirty(MemoryRegion *mr,
+ ram_addr_t start)
+{
+ unsigned long base = mr->ram_addr >> TARGET_PAGE_BITS;
+ unsigned long nr = base + (start >> TARGET_PAGE_BITS);
+ unsigned long size = base + (int128_get64(mr->size) >> TARGET_PAGE_BITS);
+
+ unsigned long next;
+
+ next = find_next_bit(ram_list.dirty_memory[DIRTY_MEMORY_MIGRATION],
+ size, nr);
+ if (next < size) {
+ clear_bit(next, ram_list.dirty_memory[DIRTY_MEMORY_MIGRATION]);
+ }
+ return (next - base) << TARGET_PAGE_BITS;
+}
+
+static void ram_flush_cache(void)
+{
+ RAMBlock *block = NULL;
+ void *dst_host;
+ void *src_host;
+ ram_addr_t ca = 0, ha = 0;
+ bool got_ca = 0, got_ha = 0;
+ int64_t host_dirty = 0, both_dirty = 0;
+
+ address_space_sync_dirty_bitmap(&address_space_memory);
+
+ block = QTAILQ_FIRST(&ram_list.blocks);
+ while (true) {
+ if (ca < block->length && ca <= ha) {
+ ca = migration_bitmap_find_and_reset_dirty(block->mr, ca);
+ if (ca < block->length) {
+ got_ca = 1;
+ }
+ }
+ if (ha < block->length && ha <= ca) {
+ ha = host_bitmap_find_and_reset_dirty(block->mr, ha);
+ if (ha < block->length && ha != ca) {
+ got_ha = 1;
+ }
+ host_dirty += (ha < block->length ? 1 : 0);
+ both_dirty += (ha < block->length && ha == ca ? 1 : 0);
+ }
+ if (ca >= block->length && ha >= block->length) {
+ ca = 0;
+ ha = 0;
+ block = QTAILQ_NEXT(block, next);
+ if (!block) {
+ break;
+ }
+ } else {
+ if (got_ha) {
+ got_ha = 0;
+ dst_host = memory_region_get_ram_ptr(block->mr) + ha;
+ src_host = memory_region_get_ram_cache_ptr(block->mr, block)
+ + ha;
+ memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
+ }
+ if (got_ca) {
+ got_ca = 0;
+ dst_host = memory_region_get_ram_ptr(block->mr) + ca;
+ src_host = memory_region_get_ram_cache_ptr(block->mr, block)
+ + ca;
+ memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
+ }
+ }
+ }
+
+ assert(migration_dirty_pages == 0);
+}
+
static SaveVMHandlers savevm_ram_handlers = {
.save_live_setup = ram_save_setup,
.save_live_iterate = ram_save_iterate,
diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h
index f9d132f..4783cf9 100644
--- a/include/exec/cpu-all.h
+++ b/include/exec/cpu-all.h
@@ -302,6 +302,7 @@ CPUArchState *cpu_copy(CPUArchState *env);
typedef struct RAMBlock {
struct MemoryRegion *mr;
uint8_t *host;
+ uint8_t *host_cache;
ram_addr_t offset;
ram_addr_t length;
uint32_t flags;
diff --git a/include/migration/migration-colo.h b/include/migration/migration-colo.h
index 9b82b71..0f4e2bc 100644
--- a/include/migration/migration-colo.h
+++ b/include/migration/migration-colo.h
@@ -29,5 +29,8 @@ void restore_exit_colo(void);
bool colo_is_slave(void);
void colo_process_incoming_checkpoints(QEMUFile *f);
+/* ram cache */
+void create_and_init_ram_cache(void);
+void release_ram_cache(void);
#endif
diff --git a/migration-colo.c b/migration-colo.c
index 8477fbd..71a500e 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -445,6 +445,8 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
goto out;
}
+ create_and_init_ram_cache();
+
ret = colo_ctl_put(ctl, COLO_READY);
if (ret) {
goto out;
@@ -522,6 +524,8 @@ out:
qemu_fclose(fb);
}
+ release_ram_cache();
+
if (ctl) {
qemu_fclose(ctl);
}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 17/23] HACK: trigger checkpoint every 500ms
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (15 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 16/23] COLO ram cache: implement colo ram cache on slave Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 18/23] COLO nic: add command line switch Yang Hongyang
` (6 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
Because COLO Agent is under development. We add this hack for
test purpose. Trigger checkpoint every 500ms so that we can
test the process of COLO save/restore.
NOTE:
This is only a hack, and will be removed at last.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 14 +++++---------
1 file changed, 5 insertions(+), 9 deletions(-)
diff --git a/migration-colo.c b/migration-colo.c
index 71a500e..1da5629 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -23,7 +23,7 @@
* this is large because COLO checkpoint will mostly depend on
* COLO compare module.
*/
-#define CHKPOINT_TIMER 10000
+#define CHKPOINT_TIMER 500
enum {
COLO_READY = 0x46,
@@ -79,11 +79,6 @@ static int comp_fd = -1;
static int colo_compare_init(void)
{
- comp_fd = open(COMPARE_DEV, O_RDONLY);
- if (comp_fd < 0) {
- return -1;
- }
-
return 0;
}
@@ -104,17 +99,18 @@ static void colo_compare_destroy(void)
*/
static int colo_compare(void)
{
- return ioctl(comp_fd, COMP_IOCTWAIT, 250);
+ errno = ERESTART;
+ return 1;
}
static int colo_compare_flush(void)
{
- return ioctl(comp_fd, COMP_IOCTFLUSH, 1);
+ return 0;
}
static int colo_compare_resume(void)
{
- return ioctl(comp_fd, COMP_IOCTRESUME, 1);
+ return 0;
}
/* colo checkpoint control helper */
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 18/23] COLO nic: add command line switch
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (16 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 17/23] HACK: trigger checkpoint every 500ms Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 17:04 ` Eric Blake
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 19/23] COLO nic: init/remove colo nic devices when add/cleanup tap devices Yang Hongyang
` (5 subsequent siblings)
23 siblings, 1 reply; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: Stefan Hajnoczi, GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert,
mrhines, Anthony Liguori, yanghy
add command line switch
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Cc: Anthony Liguori <aliguori@amazon.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
---
qapi-schema.json | 8 +++++++-
qemu-options.hx | 10 +++++++++-
2 files changed, 16 insertions(+), 2 deletions(-)
diff --git a/qapi-schema.json b/qapi-schema.json
index 7dcfb90..e53a571 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -2008,6 +2008,10 @@
#
# @queues: #optional number of queues to be created for multiqueue capable tap
#
+# @colo_nicname: #optional the host physical nic for QEMU (Since 2.2)
+#
+# @colo_script: #optional the script file which used by COLO (Since 2.2)
+#
# Since 1.2
##
{ 'type': 'NetdevTapOptions',
@@ -2024,7 +2028,9 @@
'*vhostfd': 'str',
'*vhostfds': 'str',
'*vhostforce': 'bool',
- '*queues': 'uint32'} }
+ '*queues': 'uint32',
+ '*colo_nicname': 'str',
+ '*colo_script': 'str'} }
##
# @NetdevSocketOptions
diff --git a/qemu-options.hx b/qemu-options.hx
index 365b56c..9f31dcc 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -1412,7 +1412,11 @@ DEF("net", HAS_ARG, QEMU_OPTION_net,
"-net tap[,vlan=n][,name=str],ifname=name\n"
" connect the host TAP network interface to VLAN 'n'\n"
#else
- "-net tap[,vlan=n][,name=str][,fd=h][,fds=x:y:...:z][,ifname=name][,script=file][,downscript=dfile][,helper=helper][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off][,vhostfd=h][,vhostfds=x:y:...:z][,vhostforce=on|off][,queues=n]\n"
+ "-net tap[,vlan=n][,name=str][,fd=h][,fds=x:y:...:z][,ifname=name][,script=file][,downscript=dfile][,helper=helper][,sndbuf=nbytes][,vnet_hdr=on|off][,vhost=on|off][,vhostfd=h][,vhostfds=x:y:...:z][,vhostforce=on|off][,queues=n]"
+#ifdef CONFIG_COLO
+ "[,colo_nicname=nicname][,colo_script=scriptfile]"
+#endif
+ "\n"
" connect the host TAP network interface to VLAN 'n'\n"
" use network scripts 'file' (default=" DEFAULT_NETWORK_SCRIPT ")\n"
" to configure it and 'dfile' (default=" DEFAULT_NETWORK_DOWN_SCRIPT ")\n"
@@ -1432,6 +1436,10 @@ DEF("net", HAS_ARG, QEMU_OPTION_net,
" use 'vhostfd=h' to connect to an already opened vhost net device\n"
" use 'vhostfds=x:y:...:z to connect to multiple already opened vhost net devices\n"
" use 'queues=n' to specify the number of queues to be created for multiqueue TAP\n"
+#ifdef CONFIG_COLO
+ " use 'colo_nicname=nicname' to specify the host physical nic for QEMU\n"
+ " use 'colo_script=scriptfile' to specify script file when colo is enabled\n"
+#endif
"-net bridge[,vlan=n][,name=str][,br=bridge][,helper=helper]\n"
" connects a host TAP network interface to a host bridge device 'br'\n"
" (default=" DEFAULT_BRIDGE_INTERFACE ") using the program 'helper'\n"
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* Re: [Qemu-devel] [RFC PATCH v2 18/23] COLO nic: add command line switch
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 18/23] COLO nic: add command line switch Yang Hongyang
@ 2014-09-23 17:04 ` Eric Blake
0 siblings, 0 replies; 36+ messages in thread
From: Eric Blake @ 2014-09-23 17:04 UTC (permalink / raw)
To: Yang Hongyang, qemu-devel
Cc: Stefan Hajnoczi, GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert,
mrhines, Anthony Liguori
[-- Attachment #1: Type: text/plain, Size: 947 bytes --]
On 09/23/2014 03:23 AM, Yang Hongyang wrote:
> add command line switch
>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> Cc: Anthony Liguori <aliguori@amazon.com>
> Cc: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> qapi-schema.json | 8 +++++++-
> qemu-options.hx | 10 +++++++++-
> 2 files changed, 16 insertions(+), 2 deletions(-)
>
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 7dcfb90..e53a571 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -2008,6 +2008,10 @@
> #
> # @queues: #optional number of queues to be created for multiqueue capable tap
> #
> +# @colo_nicname: #optional the host physical nic for QEMU (Since 2.2)
> +#
> +# @colo_script: #optional the script file which used by COLO (Since 2.2)
Please use dash, not underscore, for these two new names.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]
^ permalink raw reply [flat|nested] 36+ messages in thread
* [Qemu-devel] [RFC PATCH v2 19/23] COLO nic: init/remove colo nic devices when add/cleanup tap devices
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (17 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 18/23] COLO nic: add command line switch Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 20/23] COLO nic: implement colo nic device interface support_colo() Yang Hongyang
` (4 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines,
Stefan Hajnoczi, yanghy
init/remove colo nic devices when add/cleanup tap devices
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>
---
include/net/colo-nic.h | 18 +++++++++++++++++
include/net/net.h | 3 +++
net/Makefile.objs | 1 +
net/colo-nic.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++
net/tap.c | 45 +++++++++++++++++++++++++++++++++--------
5 files changed, 113 insertions(+), 8 deletions(-)
create mode 100644 include/net/colo-nic.h
create mode 100644 net/colo-nic.c
diff --git a/include/net/colo-nic.h b/include/net/colo-nic.h
new file mode 100644
index 0000000..fe6874b
--- /dev/null
+++ b/include/net/colo-nic.h
@@ -0,0 +1,18 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef COLO_NIC_H
+#define COLO_NIC_H
+
+void colo_add_nic_devices(NetClientState *nc);
+void colo_remove_nic_devices(NetClientState *nc);
+
+#endif
diff --git a/include/net/net.h b/include/net/net.h
index ed594f9..62050c5 100644
--- a/include/net/net.h
+++ b/include/net/net.h
@@ -85,6 +85,9 @@ struct NetClientState {
char *model;
char *name;
char info_str[256];
+ char colo_script[1024];
+ char colo_nicname[128];
+ char ifname[128];
unsigned receive_disabled : 1;
NetClientDestructor *destructor;
unsigned int queue_index;
diff --git a/net/Makefile.objs b/net/Makefile.objs
index ec19cb3..73f4a81 100644
--- a/net/Makefile.objs
+++ b/net/Makefile.objs
@@ -13,3 +13,4 @@ common-obj-$(CONFIG_HAIKU) += tap-haiku.o
common-obj-$(CONFIG_SLIRP) += slirp.o
common-obj-$(CONFIG_VDE) += vde.o
common-obj-$(CONFIG_NETMAP) += netmap.o
+common-obj-$(CONFIG_COLO) += colo-nic.o
diff --git a/net/colo-nic.c b/net/colo-nic.c
new file mode 100644
index 0000000..2f7ca23
--- /dev/null
+++ b/net/colo-nic.c
@@ -0,0 +1,54 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (C) 2014 FUJITSU LIMITED
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ *
+ */
+#include "net/net.h"
+#include "net/colo-nic.h"
+
+typedef struct nic_device {
+ NetClientState *nc;
+ bool (*support_colo)(NetClientState *nc);
+ int (*configure)(NetClientState *nc, bool up, bool is_slave);
+ QTAILQ_ENTRY(nic_device) next;
+ bool is_up;
+} nic_device;
+
+QTAILQ_HEAD(, nic_device) nic_devices = QTAILQ_HEAD_INITIALIZER(nic_devices);
+
+void colo_add_nic_devices(NetClientState *nc)
+{
+ struct nic_device *nic = g_malloc0(sizeof(*nic));
+
+ /* TODO: init colo function pointers */
+ /*
+ * TODO
+ * only support "-netdev tap,colo_scripte..." options
+ * "-net nic -net tap..." options is not supported
+ */
+ nic->nc = nc;
+
+ QTAILQ_INSERT_TAIL(&nic_devices, nic, next);
+}
+
+void colo_remove_nic_devices(NetClientState *nc)
+{
+ struct nic_device *nic, *next_nic;
+
+ if (!nc) {
+ return;
+ }
+
+ QTAILQ_FOREACH_SAFE(nic, &nic_devices, next, next_nic) {
+ if (nic->nc == nc) {
+ /* TODO: teardown colo nic */
+ QTAILQ_REMOVE(&nic_devices, nic, next);
+ g_free(nic);
+ }
+ }
+}
diff --git a/net/tap.c b/net/tap.c
index a40f7f0..d1a1dab 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -41,6 +41,7 @@
#include "qemu/error-report.h"
#include "net/tap.h"
+#include "net/colo-nic.h"
#include "net/vhost_net.h"
@@ -284,6 +285,8 @@ static void tap_cleanup(NetClientState *nc)
qemu_purge_queued_packets(nc);
+ colo_remove_nic_devices(nc);
+
if (s->down_script[0])
launch_script(s->down_script, s->down_script_arg, s->fd);
@@ -591,10 +594,11 @@ static int net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
const char *model, const char *name,
const char *ifname, const char *script,
const char *downscript, const char *vhostfdname,
- int vnet_hdr, int fd)
+ int vnet_hdr, int fd, bool setup_colo)
{
TAPState *s;
int vhostfd;
+ NetClientState *nc = NULL;
s = net_tap_fd_init(peer, model, name, fd, vnet_hdr);
if (!s) {
@@ -623,6 +627,21 @@ static int net_init_tap_one(const NetdevTapOptions *tap, NetClientState *peer,
}
}
+ nc = &(s->nc);
+ snprintf(nc->ifname, sizeof(nc->ifname), "%s", ifname);
+ if (tap->has_colo_script) {
+ snprintf(nc->colo_script, sizeof(nc->colo_script), "%s",
+ tap->colo_script);
+ }
+ if (tap->has_colo_nicname) {
+ snprintf(nc->colo_nicname, sizeof(nc->colo_nicname), "%s",
+ tap->colo_nicname);
+ }
+
+ if (setup_colo) {
+ colo_add_nic_devices(nc);
+ }
+
if (tap->has_vhost ? tap->vhost :
vhostfdname || (tap->has_vhostforce && tap->vhostforce)) {
VhostNetOptions options;
@@ -729,7 +748,7 @@ int net_init_tap(const NetClientOptions *opts, const char *name,
if (net_init_tap_one(tap, peer, "tap", name, NULL,
script, downscript,
- vhostfdname, vnet_hdr, fd)) {
+ vhostfdname, vnet_hdr, fd, true)) {
return -1;
}
} else if (tap->has_fds) {
@@ -739,9 +758,10 @@ int net_init_tap(const NetClientOptions *opts, const char *name,
if (tap->has_ifname || tap->has_script || tap->has_downscript ||
tap->has_vnet_hdr || tap->has_helper || tap->has_queues ||
- tap->has_vhostfd) {
+ tap->has_vhostfd || tap->has_colo_script || tap->has_colo_nicname) {
error_report("ifname=, script=, downscript=, vnet_hdr=, "
"helper=, queues=, and vhostfd= "
+ "colo_script=, and colo_nicname= "
"are invalid with fds=");
return -1;
}
@@ -774,15 +794,17 @@ int net_init_tap(const NetClientOptions *opts, const char *name,
if (net_init_tap_one(tap, peer, "tap", name, ifname,
script, downscript,
tap->has_vhostfds ? vhost_fds[i] : NULL,
- vnet_hdr, fd)) {
+ vnet_hdr, fd, false)) {
return -1;
}
}
} else if (tap->has_helper) {
if (tap->has_ifname || tap->has_script || tap->has_downscript ||
- tap->has_vnet_hdr || tap->has_queues || tap->has_vhostfds) {
+ tap->has_vnet_hdr || tap->has_queues || tap->has_vhostfds ||
+ tap->has_colo_script || tap->has_colo_nicname) {
error_report("ifname=, script=, downscript=, and vnet_hdr= "
- "queues=, and vhostfds= are invalid with helper=");
+ "queues=, vhostfds=, colo_script=, and "
+ "colo_nicname= are invalid with helper=");
return -1;
}
@@ -796,10 +818,16 @@ int net_init_tap(const NetClientOptions *opts, const char *name,
if (net_init_tap_one(tap, peer, "bridge", name, ifname,
script, downscript, vhostfdname,
- vnet_hdr, fd)) {
+ vnet_hdr, fd, false)) {
return -1;
}
} else {
+ if (queues > 1 && (tap->has_colo_script || tap->has_colo_nicname)) {
+ error_report("queues > 1 is invalid if colo_script or "
+ "colo_nicname is specified");
+ return -1;
+ }
+
if (tap->has_vhostfds) {
error_report("vhostfds= is invalid if fds= wasn't specified");
return -1;
@@ -831,7 +859,8 @@ int net_init_tap(const NetClientOptions *opts, const char *name,
if (net_init_tap_one(tap, peer, "tap", name, ifname,
i >= 1 ? "no" : script,
i >= 1 ? "no" : downscript,
- vhostfdname, vnet_hdr, fd)) {
+ vhostfdname, vnet_hdr, fd,
+ i == 0)) {
return -1;
}
}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 20/23] COLO nic: implement colo nic device interface support_colo()
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (18 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 19/23] COLO nic: init/remove colo nic devices when add/cleanup tap devices Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 21/23] COLO nic: implement colo nic device interface configure() Yang Hongyang
` (3 subsequent siblings)
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
implement colo nic device interface support_colo()
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
net/colo-nic.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/net/colo-nic.c b/net/colo-nic.c
index 2f7ca23..7255a48 100644
--- a/net/colo-nic.c
+++ b/net/colo-nic.c
@@ -21,11 +21,17 @@ typedef struct nic_device {
QTAILQ_HEAD(, nic_device) nic_devices = QTAILQ_HEAD_INITIALIZER(nic_devices);
+static bool nic_support_colo(NetClientState *nc)
+{
+ return nc && nc->colo_script[0] && nc->colo_nicname[0];
+}
+
void colo_add_nic_devices(NetClientState *nc)
{
struct nic_device *nic = g_malloc0(sizeof(*nic));
- /* TODO: init colo function pointers */
+ nic->support_colo = nic_support_colo;
+
/*
* TODO
* only support "-netdev tap,colo_scripte..." options
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 21/23] COLO nic: implement colo nic device interface configure()
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (19 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 20/23] COLO nic: implement colo nic device interface support_colo() Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-10-27 17:49 ` Dr. David Alan Gilbert
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 22/23] COLO nic: export colo nic APIs Yang Hongyang
` (2 subsequent siblings)
23 siblings, 1 reply; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
implement colo nic device interface configure()
add a script to configure nic devices:
${QEMU_SCRIPT_DIR}/network-colo
Script for configuring the network of Master & Slaver.
Usage:
network-colo (master|slaver) (install|uninstall) vif pif [ifb1 ifb2]
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
include/net/net.h | 1 +
net/colo-nic.c | 106 +++++++++++++++++++++++++++++
network-colo | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 301 insertions(+)
create mode 100755 network-colo
diff --git a/include/net/net.h b/include/net/net.h
index 62050c5..9cc9b5c 100644
--- a/include/net/net.h
+++ b/include/net/net.h
@@ -88,6 +88,7 @@ struct NetClientState {
char colo_script[1024];
char colo_nicname[128];
char ifname[128];
+ char ifb[2][128];
unsigned receive_disabled : 1;
NetClientDestructor *destructor;
unsigned int queue_index;
diff --git a/net/colo-nic.c b/net/colo-nic.c
index 7255a48..d661d8b 100644
--- a/net/colo-nic.c
+++ b/net/colo-nic.c
@@ -10,6 +10,7 @@
*/
#include "net/net.h"
#include "net/colo-nic.h"
+#include "qemu/error-report.h"
typedef struct nic_device {
NetClientState *nc;
@@ -26,11 +27,116 @@ static bool nic_support_colo(NetClientState *nc)
return nc && nc->colo_script[0] && nc->colo_nicname[0];
}
+#define STDOUT_BUF_LEN 1024
+static char stdout_buf[STDOUT_BUF_LEN];
+
+static int launch_colo_script(char *argv[])
+{
+ int pid, status;
+ char *script = argv[0];
+ int fds[2];
+
+ bzero(stdout_buf, sizeof(stdout_buf));
+
+ if (pipe(fds) < 0) {
+ return -1;
+ }
+ /* try to launch network script */
+ pid = fork();
+ if (pid == 0) {
+ close(fds[0]);
+ dup2(fds[1], STDOUT_FILENO);
+ execv(script, argv);
+ _exit(1);
+ } else if (pid > 0) {
+ FILE *stream;
+ int n;
+ close(fds[1]);
+ stream = fdopen(fds[0], "r");
+ n = fread(stdout_buf, 1, STDOUT_BUF_LEN - 1, stream);
+ stdout_buf[n] = '\0';
+ close(fds[0]);
+
+ while (waitpid(pid, &status, 0) != pid) {
+ /* loop */
+ }
+
+ if (WIFEXITED(status) && WEXITSTATUS(status) == 0) {
+ return 0;
+ }
+ }
+ fprintf(stderr, "%s\n", stdout_buf);
+ fprintf(stderr, "%s: could not launch network script\n", script);
+ return -1;
+}
+
+static void store_ifbname(NetClientState *nc)
+{
+ char *str_b = NULL, *str_e = NULL;
+
+ str_b = strstr(stdout_buf, "ifb0=");
+ if (str_b) {
+ str_e = strstr(str_b, "\n");
+ }
+ if (str_e) {
+ snprintf(nc->ifb[0], str_e - str_b - 5 + 1, "%s", str_b + 5);
+ }
+
+ str_b = str_e = NULL;
+ str_b = strstr(stdout_buf, "ifb1=");
+ if (str_b) {
+ str_e = strstr(str_b, "\n");
+ }
+ if (str_e) {
+ snprintf(nc->ifb[1], str_e - str_b - 5 + 1, "%s", str_b + 5);
+ }
+}
+
+static int nic_configure(NetClientState *nc, bool up, bool is_slave)
+{
+ char *argv[8];
+ char **parg;
+ int ret = -1, i;
+ int argc = (!is_slave && !up) ? 7 : 5;
+
+ if (!nc) {
+ error_report("Can not parse colo_script or colo_nicname");
+ return ret;
+ }
+
+ parg = argv;
+ *parg++ = nc->colo_script;
+ *parg++ = (char *)(is_slave ? "slaver" : "master");
+ *parg++ = (char *)(up ? "install" : "uninstall");
+ *parg++ = nc->ifname;
+ *parg++ = nc->colo_nicname;
+ if (!is_slave && !up) {
+ *parg++ = nc->ifb[0];
+ *parg++ = nc->ifb[1];
+ }
+ *parg = NULL;
+
+ for (i = 0; i < argc; i++) {
+ if (!argv[i][0]) {
+ error_report("Can not get colo_script argument");
+ return ret;
+ }
+ }
+
+ ret = launch_colo_script(argv);
+ if (!is_slave && up && ret == 0) {
+ store_ifbname(nc);
+ }
+
+ return ret;
+}
+
void colo_add_nic_devices(NetClientState *nc)
{
struct nic_device *nic = g_malloc0(sizeof(*nic));
nic->support_colo = nic_support_colo;
+ nic->configure = nic_configure;
/*
* TODO
diff --git a/network-colo b/network-colo
new file mode 100755
index 0000000..9112888
--- /dev/null
+++ b/network-colo
@@ -0,0 +1,194 @@
+#! /bin/bash
+#============================================================================
+# ${QEMU_SCRIPT_DIR}/network-colo
+#
+# Script for configuring the network of Master & Slaver.
+#
+# Usage:
+# network-colo (master|slaver) (install|uninstall) vif pif [ifb1 ifb2]
+#============================================================================
+
+sides=$1
+op=$2
+vif=$3
+pif=$4
+ifb1=$5
+ifb2=$6
+BR=br1
+
+qlen=40960
+module="HA_compare"
+device="HA_compare"
+
+# start_master ifbx
+function start_master() {
+
+ # In colo mode, we don't use gso, gro...
+ ip link set dev $vif qlen $qlen
+
+ # copy and foward input packets to $pif
+ tc qdisc add dev $vif root handle 1: prio
+ tc filter add dev $vif parent 1: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $pif
+ tc filter add dev $vif parent 1: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $pif
+
+ # foward output packets to ifbx
+ tc qdisc add dev $vif ingress
+ tc filter add dev $vif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $1
+ tc filter add dev $vif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $1
+}
+
+function stop_master() {
+ # don't copy and foward input packets to $pif
+ tc filter del dev $vif parent 1: protocol ip prio 10 u32
+ tc filter del dev $vif parent 1: protocol arp prio 11 u32
+ tc qdisc del dev $vif root handle 1: prio
+
+ # don't foward output packets to ifbx
+ tc filter del dev $vif parent ffff: protocol ip prio 10 u32
+ tc filter del dev $vif parent ffff: protocol arp prio 11 u32
+ tc qdisc del dev $vif ingress
+}
+
+# load_module module parameter
+function load_module()
+{
+ local module=$1
+ shift
+
+ lsmod | grep -q "$module"
+ if [[ $? -eq 0 ]]; then
+ # The module has been loaded
+ return
+ fi
+
+ modprobe $module "$@"
+}
+
+function select_ifb()
+{
+ local -i index
+
+ for (( index = 0; index < 100; index++)); do
+ state=$(ip link show dev ifb$index | sed -n -e 's/.*state \([a-zA-Z]*\) .*/\1/p')
+ if [[ $state == "DOWN" ]]; then
+ return $index
+ fi
+ done
+
+ return 100
+}
+
+
+function install_master() {
+ load_module sch_colo
+ load_module HA_compare
+ load_module HA_compare_icmp
+
+ if [[ ! -e "/dev/$device" ]]; then
+ major=$(awk "\$2==\"$module\" {print \$1}" /proc/devices)
+ mknod /dev/$device c $major 0
+ fi
+
+ load_module ifb numifbs=100
+
+ select_ifb
+ index1=$?
+ if [[ $index1 -eq 100 ]]; then
+ echo "index1 $index1 overflow" >>/root/network-colo.log
+ exit 1
+ fi
+ ip link set ifb$index1 up
+ ip link set ifb$index1 qlen $qlen
+
+ select_ifb
+ index2=$?
+ if [[ $index2 -eq 100 ]]; then
+ echo "index1 $index1 overflow" >>/root/network-colo.log
+ exit 1
+ fi
+ ip link set ifb$index2 up
+ ip link set ifb$index2 qlen $qlen
+ colo_tc qdisc add dev ifb$index1 root handle 1: colo dev ifb$index2 master
+ colo_tc qdisc add dev ifb$index2 root handle 1: colo dev ifb$index1 slaver
+
+ ifconfig $pif promisc
+ ip link set $pif qlen $qlen
+
+ # forward packets from $pif to ifb$index2
+ tc qdisc add dev $pif ingress
+ tc filter add dev $pif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev ifb$index2
+ tc filter add dev $pif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev ifb$index2
+
+ start_master ifb$index1
+}
+
+function uninstall_master() {
+ stop_master
+
+ # shutdown $ifb1
+ tc qdisc del dev $ifb1 root handle 1: colo
+ ip link set $ifb1 down
+
+ # don't forward packets from $pif to $ifb2
+ tc filter del dev $pif parent ffff: protocol ip prio 10 u32
+ tc qdisc del dev $pif ingress
+
+ # shutdown $ifb2
+ tc qdisc del dev $ifb2 root handle 1: colo
+ ip link set $ifb2 down
+
+ ifconfig $pif -promisc
+}
+
+function install_slaver()
+{
+ ifconfig $pif promisc
+ ip link set $pif qlen $qlen
+
+ # forward packets from $pif to $vif
+ tc qdisc add dev $pif ingress
+ tc filter add dev $pif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $vif
+ tc filter add dev $pif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $vif
+
+ ip link set $vif qlen $qlen
+ # forward packets from $vif to $pif
+ tc qdisc add dev $vif ingress
+ tc filter add dev $vif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $pif
+ tc filter add dev $vif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $pif
+
+ brctl delif $BR $vif
+}
+
+function uninstall_slaver()
+{
+ # don't forward packets from $pif to $vif
+ tc filter del dev $pif parent ffff: protocol ip prio 10 u32
+ tc filter del dev $pif parent ffff: protocol arp prio 11 u32
+ tc qdisc del dev $pif ingress
+
+ # don't forward packets from $vif to $pif
+ tc filter del dev $vif parent ffff: protocol ip prio 10 u32
+ tc filter del dev $vif parent ffff: protocol arp prio 11 u32
+ tc qdisc del dev $vif ingress
+
+ ifconfig $pif -promisc
+
+ brctl addif $BR $vif
+}
+
+echo "$@" >/root/network-colo.log
+if [[ $1 != "master" && $1 != "slaver" ]]; then
+ echo "$1 != master/slaver" >>/root/network-colo.log
+ exit 1
+fi
+
+if [[ $2 != "install" && $2 != "uninstall" ]]; then
+ echo "$2 != install/uninstall" >>/root/network-colo.log
+ exit 1
+fi
+
+${op}_$sides 1>>/root/network-colo.log 2>&1
+if [[ $1 == "master" && $2 == "install" ]]; then
+ echo ifb0=ifb$index1
+ echo ifb1=ifb$index2
+fi
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* Re: [Qemu-devel] [RFC PATCH v2 21/23] COLO nic: implement colo nic device interface configure()
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 21/23] COLO nic: implement colo nic device interface configure() Yang Hongyang
@ 2014-10-27 17:49 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 36+ messages in thread
From: Dr. David Alan Gilbert @ 2014-10-27 17:49 UTC (permalink / raw)
To: Yang Hongyang; +Cc: GuiJianfeng, yunhong.jiang, eddie.dong, qemu-devel, mrhines
* Yang Hongyang (yanghy@cn.fujitsu.com) wrote:
> implement colo nic device interface configure()
> add a script to configure nic devices:
> ${QEMU_SCRIPT_DIR}/network-colo
>
> Script for configuring the network of Master & Slaver.
You might also like to think about making this possible via libvirt;
it does a lot of network setup for VMs already.
> Usage:
> network-colo (master|slaver) (install|uninstall) vif pif [ifb1 ifb2]
Can you just describe what the four interfaces are; I'm assuming
'vif' is the virtual interface for the VM, and 'pif' is a physical interface
the packets from that VM are going to go to/come from.
Then I was expecting one more interface where you copy the outgoing packets
from the slave so that they go to the primary VM for comparison; why are there
two?
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> ---
> include/net/net.h | 1 +
> net/colo-nic.c | 106 +++++++++++++++++++++++++++++
> network-colo | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 3 files changed, 301 insertions(+)
> create mode 100755 network-colo
>
> diff --git a/include/net/net.h b/include/net/net.h
> index 62050c5..9cc9b5c 100644
> --- a/include/net/net.h
> +++ b/include/net/net.h
> @@ -88,6 +88,7 @@ struct NetClientState {
> char colo_script[1024];
> char colo_nicname[128];
> char ifname[128];
> + char ifb[2][128];
It's a shame that these are all magic 128 character lengths.
> unsigned receive_disabled : 1;
> NetClientDestructor *destructor;
> unsigned int queue_index;
> diff --git a/net/colo-nic.c b/net/colo-nic.c
> index 7255a48..d661d8b 100644
> --- a/net/colo-nic.c
> +++ b/net/colo-nic.c
> @@ -10,6 +10,7 @@
> */
> #include "net/net.h"
> #include "net/colo-nic.h"
> +#include "qemu/error-report.h"
>
> typedef struct nic_device {
> NetClientState *nc;
> @@ -26,11 +27,116 @@ static bool nic_support_colo(NetClientState *nc)
> return nc && nc->colo_script[0] && nc->colo_nicname[0];
> }
>
> +#define STDOUT_BUF_LEN 1024
> +static char stdout_buf[STDOUT_BUF_LEN];
> +
> +static int launch_colo_script(char *argv[])
> +{
> + int pid, status;
> + char *script = argv[0];
> + int fds[2];
> +
> + bzero(stdout_buf, sizeof(stdout_buf));
> +
> + if (pipe(fds) < 0) {
> + return -1;
> + }
> + /* try to launch network script */
> + pid = fork();
> + if (pid == 0) {
> + close(fds[0]);
> + dup2(fds[1], STDOUT_FILENO);
> + execv(script, argv);
> + _exit(1);
> + } else if (pid > 0) {
> + FILE *stream;
> + int n;
> + close(fds[1]);
> + stream = fdopen(fds[0], "r");
> + n = fread(stdout_buf, 1, STDOUT_BUF_LEN - 1, stream);
> + stdout_buf[n] = '\0';
> + close(fds[0]);
> +
> + while (waitpid(pid, &status, 0) != pid) {
> + /* loop */
> + }
> +
> + if (WIFEXITED(status) && WEXITSTATUS(status) == 0) {
> + return 0;
> + }
> + }
> + fprintf(stderr, "%s\n", stdout_buf);
> + fprintf(stderr, "%s: could not launch network script\n", script);
> + return -1;
> +}
> +
> +static void store_ifbname(NetClientState *nc)
> +{
> + char *str_b = NULL, *str_e = NULL;
> +
> + str_b = strstr(stdout_buf, "ifb0=");
> + if (str_b) {
> + str_e = strstr(str_b, "\n");
> + }
> + if (str_e) {
> + snprintf(nc->ifb[0], str_e - str_b - 5 + 1, "%s", str_b + 5);
> + }
> +
> + str_b = str_e = NULL;
> + str_b = strstr(stdout_buf, "ifb1=");
> + if (str_b) {
> + str_e = strstr(str_b, "\n");
> + }
> + if (str_e) {
> + snprintf(nc->ifb[1], str_e - str_b - 5 + 1, "%s", str_b + 5);
> + }
> +}
I don't think these protect against a bad string that's longer
than the 128 character ifb[] length.
I think it would also be good to return sometype of status from this
function to know if it worked.
> +static int nic_configure(NetClientState *nc, bool up, bool is_slave)
> +{
> + char *argv[8];
> + char **parg;
> + int ret = -1, i;
> + int argc = (!is_slave && !up) ? 7 : 5;
> +
> + if (!nc) {
> + error_report("Can not parse colo_script or colo_nicname");
> + return ret;
> + }
> +
> + parg = argv;
> + *parg++ = nc->colo_script;
> + *parg++ = (char *)(is_slave ? "slaver" : "master");
> + *parg++ = (char *)(up ? "install" : "uninstall");
> + *parg++ = nc->ifname;
> + *parg++ = nc->colo_nicname;
> + if (!is_slave && !up) {
> + *parg++ = nc->ifb[0];
> + *parg++ = nc->ifb[1];
> + }
> + *parg = NULL;
> +
> + for (i = 0; i < argc; i++) {
> + if (!argv[i][0]) {
> + error_report("Can not get colo_script argument");
> + return ret;
> + }
> + }
> +
> + ret = launch_colo_script(argv);
> + if (!is_slave && up && ret == 0) {
> + store_ifbname(nc);
> + }
> +
> + return ret;
> +}
> +
> void colo_add_nic_devices(NetClientState *nc)
> {
> struct nic_device *nic = g_malloc0(sizeof(*nic));
>
> nic->support_colo = nic_support_colo;
> + nic->configure = nic_configure;
>
> /*
> * TODO
> diff --git a/network-colo b/network-colo
> new file mode 100755
> index 0000000..9112888
> --- /dev/null
> +++ b/network-colo
> @@ -0,0 +1,194 @@
> +#! /bin/bash
> +#============================================================================
> +# ${QEMU_SCRIPT_DIR}/network-colo
> +#
> +# Script for configuring the network of Master & Slaver.
> +#
> +# Usage:
> +# network-colo (master|slaver) (install|uninstall) vif pif [ifb1 ifb2]
> +#============================================================================
> +
> +sides=$1
> +op=$2
> +vif=$3
> +pif=$4
> +ifb1=$5
> +ifb2=$6
> +BR=br1
> +
> +qlen=40960
> +module="HA_compare"
> +device="HA_compare"
> +
> +# start_master ifbx
> +function start_master() {
> +
> + # In colo mode, we don't use gso, gro...
> + ip link set dev $vif qlen $qlen
> +
> + # copy and foward input packets to $pif
> + tc qdisc add dev $vif root handle 1: prio
> + tc filter add dev $vif parent 1: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $pif
> + tc filter add dev $vif parent 1: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $pif
> +
> + # foward output packets to ifbx
> + tc qdisc add dev $vif ingress
> + tc filter add dev $vif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $1
> + tc filter add dev $vif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $1
> +}
> +
> +function stop_master() {
> + # don't copy and foward input packets to $pif
> + tc filter del dev $vif parent 1: protocol ip prio 10 u32
> + tc filter del dev $vif parent 1: protocol arp prio 11 u32
> + tc qdisc del dev $vif root handle 1: prio
> +
> + # don't foward output packets to ifbx
> + tc filter del dev $vif parent ffff: protocol ip prio 10 u32
> + tc filter del dev $vif parent ffff: protocol arp prio 11 u32
> + tc qdisc del dev $vif ingress
> +}
> +
> +# load_module module parameter
> +function load_module()
> +{
> + local module=$1
> + shift
> +
> + lsmod | grep -q "$module"
That's perhaps not selective enough, especially since 'HA_compare' would
match in the 'HA_compare_icmp' line.
> + if [[ $? -eq 0 ]]; then
> + # The module has been loaded
> + return
> + fi
> +
> + modprobe $module "$@"
> +}
> +
> +function select_ifb()
> +{
> + local -i index
> +
> + for (( index = 0; index < 100; index++)); do
> + state=$(ip link show dev ifb$index | sed -n -e 's/.*state \([a-zA-Z]*\) .*/\1/p')
> + if [[ $state == "DOWN" ]]; then
> + return $index
> + fi
> + done
The fixed sized loop seems weird; maybe something like:
ip link show |awk -F' ' ' { if ( $9 == "DOWN" ) { print $2; }; }'
or even :
grep down /sys/class/net/ifb*/operstate
> +
> + return 100
> +}
> +
> +
> +function install_master() {
> + load_module sch_colo
> + load_module HA_compare
> + load_module HA_compare_icmp
> +
> + if [[ ! -e "/dev/$device" ]]; then
> + major=$(awk "\$2==\"$module\" {print \$1}" /proc/devices)
> + mknod /dev/$device c $major 0
> + fi
So I think module at that point is HA_compare ?
> + load_module ifb numifbs=100
> +
> + select_ifb
> + index1=$?
> + if [[ $index1 -eq 100 ]]; then
> + echo "index1 $index1 overflow" >>/root/network-colo.log
> + exit 1
> + fi
> + ip link set ifb$index1 up
> + ip link set ifb$index1 qlen $qlen
> +
> + select_ifb
> + index2=$?
> + if [[ $index2 -eq 100 ]]; then
> + echo "index1 $index1 overflow" >>/root/network-colo.log
> + exit 1
> + fi
> + ip link set ifb$index2 up
> + ip link set ifb$index2 qlen $qlen
> + colo_tc qdisc add dev ifb$index1 root handle 1: colo dev ifb$index2 master
> + colo_tc qdisc add dev ifb$index2 root handle 1: colo dev ifb$index1 slaver
> +
> + ifconfig $pif promisc
> + ip link set $pif qlen $qlen
> +
> + # forward packets from $pif to ifb$index2
> + tc qdisc add dev $pif ingress
> + tc filter add dev $pif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev ifb$index2
> + tc filter add dev $pif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev ifb$index2
> +
> + start_master ifb$index1
> +}
> +
> +function uninstall_master() {
> + stop_master
> +
> + # shutdown $ifb1
> + tc qdisc del dev $ifb1 root handle 1: colo
> + ip link set $ifb1 down
> +
> + # don't forward packets from $pif to $ifb2
> + tc filter del dev $pif parent ffff: protocol ip prio 10 u32
> + tc qdisc del dev $pif ingress
> +
> + # shutdown $ifb2
> + tc qdisc del dev $ifb2 root handle 1: colo
> + ip link set $ifb2 down
> +
> + ifconfig $pif -promisc
> +}
> +
> +function install_slaver()
> +{
> + ifconfig $pif promisc
> + ip link set $pif qlen $qlen
> +
> + # forward packets from $pif to $vif
> + tc qdisc add dev $pif ingress
> + tc filter add dev $pif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $vif
> + tc filter add dev $pif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $vif
> +
> + ip link set $vif qlen $qlen
> + # forward packets from $vif to $pif
> + tc qdisc add dev $vif ingress
> + tc filter add dev $vif parent ffff: protocol ip prio 10 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $pif
> + tc filter add dev $vif parent ffff: protocol arp prio 11 u32 match u32 0 0 flowid 1:2 action mirred egress redirect dev $pif
> +
> + brctl delif $BR $vif
> +}
> +
> +function uninstall_slaver()
> +{
> + # don't forward packets from $pif to $vif
> + tc filter del dev $pif parent ffff: protocol ip prio 10 u32
> + tc filter del dev $pif parent ffff: protocol arp prio 11 u32
> + tc qdisc del dev $pif ingress
> +
> + # don't forward packets from $vif to $pif
> + tc filter del dev $vif parent ffff: protocol ip prio 10 u32
> + tc filter del dev $vif parent ffff: protocol arp prio 11 u32
> + tc qdisc del dev $vif ingress
> +
> + ifconfig $pif -promisc
> +
> + brctl addif $BR $vif
> +}
> +
> +echo "$@" >/root/network-colo.log
> +if [[ $1 != "master" && $1 != "slaver" ]]; then
> + echo "$1 != master/slaver" >>/root/network-colo.log
> + exit 1
> +fi
> +
> +if [[ $2 != "install" && $2 != "uninstall" ]]; then
> + echo "$2 != install/uninstall" >>/root/network-colo.log
> + exit 1
> +fi
> +
> +${op}_$sides 1>>/root/network-colo.log 2>&1
> +if [[ $1 == "master" && $2 == "install" ]]; then
> + echo ifb0=ifb$index1
> + echo ifb1=ifb$index2
> +fi
> --
> 1.9.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 36+ messages in thread
* [Qemu-devel] [RFC PATCH v2 22/23] COLO nic: export colo nic APIs
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (20 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 21/23] COLO nic: implement colo nic device interface configure() Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 23/23] COLO nic: setup/teardown colo nic devices Yang Hongyang
2014-10-29 6:53 ` [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Wen Congyang
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
export colo nic APIs:
colo_configure_nic()
colo_teardown_nic()
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
include/net/colo-nic.h | 2 ++
net/colo-nic.c | 63 +++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 64 insertions(+), 1 deletion(-)
diff --git a/include/net/colo-nic.h b/include/net/colo-nic.h
index fe6874b..66d2dcc 100644
--- a/include/net/colo-nic.h
+++ b/include/net/colo-nic.h
@@ -14,5 +14,7 @@
void colo_add_nic_devices(NetClientState *nc);
void colo_remove_nic_devices(NetClientState *nc);
+int colo_configure_nic(bool is_slave);
+void colo_teardown_nic(bool is_slave);
#endif
diff --git a/net/colo-nic.c b/net/colo-nic.c
index d661d8b..5d6d352 100644
--- a/net/colo-nic.c
+++ b/net/colo-nic.c
@@ -11,6 +11,7 @@
#include "net/net.h"
#include "net/colo-nic.h"
#include "qemu/error-report.h"
+#include "migration/migration-colo.h"
typedef struct nic_device {
NetClientState *nc;
@@ -131,6 +132,35 @@ static int nic_configure(NetClientState *nc, bool up, bool is_slave)
return ret;
}
+static int configure_one_nic(NetClientState *nc, bool up, bool is_slave)
+{
+ struct nic_device *nic;
+
+ if (!nc) {
+ return -1;
+ }
+
+ QTAILQ_FOREACH(nic, &nic_devices, next) {
+ if (nic->nc == nc) {
+ if (!nic->support_colo || !nic->support_colo(nic->nc)
+ || !nic->configure) {
+ return -1;
+ }
+ if (up == nic->is_up) {
+ return 0;
+ }
+
+ if (nic->configure(nic->nc, up, is_slave) && up) {
+ return -1;
+ }
+ nic->is_up = up;
+ return 0;
+ }
+ }
+
+ return -1;
+}
+
void colo_add_nic_devices(NetClientState *nc)
{
struct nic_device *nic = g_malloc0(sizeof(*nic));
@@ -158,9 +188,40 @@ void colo_remove_nic_devices(NetClientState *nc)
QTAILQ_FOREACH_SAFE(nic, &nic_devices, next, next_nic) {
if (nic->nc == nc) {
- /* TODO: teardown colo nic */
+ if (colo_is_slave()) {
+ configure_one_nic(nc, 0, 1);
+ }
+ if (colo_is_master()) {
+ configure_one_nic(nc, 0, 0);
+ }
QTAILQ_REMOVE(&nic_devices, nic, next);
g_free(nic);
}
}
}
+
+int colo_configure_nic(bool is_slave)
+{
+ struct nic_device *nic;
+
+ if (QTAILQ_EMPTY(&nic_devices)) {
+ return -1;
+ }
+
+ QTAILQ_FOREACH(nic, &nic_devices, next) {
+ if (configure_one_nic(nic->nc, 1, is_slave)) {
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+void colo_teardown_nic(bool is_slave)
+{
+ struct nic_device *nic;
+
+ QTAILQ_FOREACH(nic, &nic_devices, next) {
+ configure_one_nic(nic->nc, 0, is_slave);
+ }
+}
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* [Qemu-devel] [RFC PATCH v2 23/23] COLO nic: setup/teardown colo nic devices
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (21 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 22/23] COLO nic: export colo nic APIs Yang Hongyang
@ 2014-09-23 9:23 ` Yang Hongyang
2014-10-29 6:53 ` [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Wen Congyang
23 siblings, 0 replies; 36+ messages in thread
From: Yang Hongyang @ 2014-09-23 9:23 UTC (permalink / raw)
To: qemu-devel
Cc: GuiJianfeng, yunhong.jiang, eddie.dong, dgilbert, mrhines, yanghy
setup/teardown colo nic devices when enter/leave
colo process.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration-colo.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/migration-colo.c b/migration-colo.c
index 1da5629..13f55d9 100644
--- a/migration-colo.c
+++ b/migration-colo.c
@@ -17,6 +17,7 @@
#include "migration/migration-colo.h"
#include <sys/ioctl.h>
#include "qemu/error-report.h"
+#include "net/colo-nic.h"
/*
* checkpoint timer: unit ms
@@ -121,6 +122,7 @@ static void ctl_error_handler(void *opaque, int err)
/* TODO: determine whether we need to failover */
/* FIXME: we will not failover currently, just kill slave */
error_report("error: colo transmission failed!");
+ colo_teardown_nic(true);
exit(1);
} else if (colo_is_master()) {
/* Master still alive, do not failover */
@@ -301,6 +303,7 @@ static void *colo_thread(void *opaque)
qdev_hotplug = 0;
+ colo_configure_nic(false);
/*
* Wait for slave finish loading vm states and enter COLO
* restore.
@@ -343,6 +346,7 @@ out:
}
colo_compare_destroy();
+ colo_teardown_nic(false);
migrate_set_state(s, MIG_STATE_COLO, MIG_STATE_COMPLETED);
@@ -442,6 +446,7 @@ void colo_process_incoming_checkpoints(QEMUFile *f)
}
create_and_init_ram_cache();
+ colo_configure_nic(true);
ret = colo_ctl_put(ctl, COLO_READY);
if (ret) {
@@ -521,6 +526,7 @@ out:
}
release_ram_cache();
+ colo_teardown_nic(true);
if (ctl) {
qemu_fclose(ctl);
--
1.9.1
^ permalink raw reply related [flat|nested] 36+ messages in thread* Re: [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service
2014-09-23 9:23 [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Yang Hongyang
` (22 preceding siblings ...)
2014-09-23 9:23 ` [Qemu-devel] [RFC PATCH v2 23/23] COLO nic: setup/teardown colo nic devices Yang Hongyang
@ 2014-10-29 6:53 ` Wen Congyang
2014-10-29 9:34 ` Dr. David Alan Gilbert
23 siblings, 1 reply; 36+ messages in thread
From: Wen Congyang @ 2014-10-29 6:53 UTC (permalink / raw)
To: Yang Hongyang, qemu-devel
Cc: Gui Jianfeng, Jiang Yunhong, Dong Eddie, Dr. David Alan Gilbert,
Michael R. Hines, Paolo Bonzini, Walid Nouri
On 09/23/2014 05:23 PM, Yang Hongyang wrote:
> Virtual machine (VM) replication is a well known technique for
> providing application-agnostic software-implemented hardware fault
> tolerance "non-stop service". COLO is a high availability solution.
> Both primary VM (PVM) and secondary VM (SVM) run in parallel. They
> receive the same request from client, and generate response in parallel
> too. If the response packets from PVM and SVM are identical, they are
> released immediately. Otherwise, a VM checkpoint (on demand) is
> conducted. The idea is presented in Xen summit 2012, and 2013,
> and academia paper in SOCC 2013. It's also presented in KVM forum
> 2013:
> http://www.linux-kvm.org/wiki/images/1/1d/Kvm-forum-2013-COLO.pdf
> Please refer to above document for detailed information.
> Please also refer to previous posted RFC proposal:
> http://lists.nongnu.org/archive/html/qemu-devel/2014-06/msg05567.html
>
> The patchset is also hosted on github:
> https://github.com/macrosheep/qemu/tree/colo_v0.5
>
> v2:
> use QEMUSizedBuffer/QEMUFile as COLO buffer
> colo support is enabled by default
> add nic replication support
> addressed comments from Eric Blake and Dr. David Alan Gilbert
>
> v1:
> implement the frame of colo
>
> This patchset is RFC, But it is ready for demo the COLO idea
> with QEMU-KVM.
> Steps using this patchset to get an overview of COLO:
> 1. configure
> 2. compile
> 3. just like QEMU's normal migration, run 2 QEMU VM:
> - Primary VM
> - Secondary VM with -incoming tcp:[IP]:[PORT] option
> 4. on Primary VM's QEMU monitor, run following command:
> migrate_set_capability colo on
> migrate tcp:[IP]:[PORT]
> 5. done
> you will see two runing VMs, whenever you make changes to PVM, SVM
> will be synced to PVM's state.
>
> TODO list:
> 1. failover (will require heartbeat module: http://www.linux-ha.org/wiki/Downloads)
> 2. disk replication[COLO Disk manager]
Hi all:
I will start to implement disk replication. Before doing this, I think we should decide
how to implement it.
I have two ideas about it:
1. implement it in qemu
Advantage: very easy, and don't take too much time
Disadvantage: the virtio disk with vhost is not supported, because the disk I/O
operations are not handled in qemu.
2. update drbd and make it support colo
Advantage: we can use it for both KVM and XEN.
Disadvantage: The implementation may be complex, and need too much time to
implement it.(I don't read the drbd's codes, and can't estimate the cost)
I think we can use 1 to implement it first.
If you have some other idea, please let me know.
Thanks
Wen Congyang
>
> Any comments/feedbacks are warmly welcomed.
>
> Thanks,
> Yang
>
>
> Dr. David Alan Gilbert (1):
> QEMUSizedBuffer/QEMUFile
>
> Yang Hongyang (22):
> configure: add CONFIG_COLO to switch COLO support
> COLO: introduce an api colo_supported() to indicate COLO support
> COLO migration: add a migration capability 'colo'
> COLO info: use colo info to tell migration target colo is enabled
> COLO save: integrate COLO checkpointed save into qemu migration
> COLO restore: integrate COLO checkpointed restore into qemu restore
> COLO: disable qdev hotplug
> COLO ctl: implement API's that communicate with colo agent
> COLO ctl: introduce is_slave() and is_master()
> COLO ctl: implement colo checkpoint protocol
> COLO ctl: add a RunState RUN_STATE_COLO
> COLO ctl: implement colo save
> COLO ctl: implement colo restore
> COLO save: reuse migration bitmap under colo checkpoint
> COLO ram cache: implement colo ram cache on slave
> HACK: trigger checkpoint every 500ms
> COLO nic: add command line switch
> COLO nic: init/remove colo nic devices when add/cleanup tap devices
> COLO nic: implement colo nic device interface support_colo()
> COLO nic: implement colo nic device interface configure()
> COLO nic: export colo nic APIs
> COLO nic: setup/teardown colo nic devices
>
> Makefile.objs | 2 +
> arch_init.c | 174 +++++++++++-
> configure | 14 +
> include/exec/cpu-all.h | 1 +
> include/migration/migration-colo.h | 36 +++
> include/migration/migration.h | 13 +
> include/migration/qemu-file.h | 28 ++
> include/net/colo-nic.h | 20 ++
> include/net/net.h | 4 +
> include/qemu/typedefs.h | 1 +
> migration-colo-comm.c | 78 ++++++
> migration-colo.c | 540 +++++++++++++++++++++++++++++++++++++
> migration.c | 47 ++--
> net/Makefile.objs | 1 +
> net/colo-nic.c | 227 ++++++++++++++++
> net/tap.c | 45 +++-
> network-colo | 194 +++++++++++++
> qapi-schema.json | 18 +-
> qemu-file.c | 410 ++++++++++++++++++++++++++++
> qemu-options.hx | 10 +-
> stubs/Makefile.objs | 1 +
> stubs/migration-colo.c | 34 +++
> vl.c | 12 +
> 23 files changed, 1879 insertions(+), 31 deletions(-)
> create mode 100644 include/migration/migration-colo.h
> create mode 100644 include/net/colo-nic.h
> create mode 100644 migration-colo-comm.c
> create mode 100644 migration-colo.c
> create mode 100644 net/colo-nic.c
> create mode 100755 network-colo
> create mode 100644 stubs/migration-colo.c
>
^ permalink raw reply [flat|nested] 36+ messages in thread* Re: [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service
2014-10-29 6:53 ` [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service Wen Congyang
@ 2014-10-29 9:34 ` Dr. David Alan Gilbert
2014-10-29 9:54 ` Wen Congyang
2014-10-29 10:19 ` Hongyang Yang
0 siblings, 2 replies; 36+ messages in thread
From: Dr. David Alan Gilbert @ 2014-10-29 9:34 UTC (permalink / raw)
To: Wen Congyang
Cc: Gui Jianfeng, Jiang Yunhong, Dong Eddie, qemu-devel,
Michael R. Hines, Paolo Bonzini, Walid Nouri, Yang Hongyang
* Wen Congyang (wency@cn.fujitsu.com) wrote:
<snip>
> Hi all:
>
> I will start to implement disk replication. Before doing this, I think we should decide
> how to implement it.
>
> I have two ideas about it:
> 1. implement it in qemu
> Advantage: very easy, and don't take too much time
> Disadvantage: the virtio disk with vhost is not supported, because the disk I/O
> operations are not handled in qemu.
>
> 2. update drbd and make it support colo
> Advantage: we can use it for both KVM and XEN.
> Disadvantage: The implementation may be complex, and need too much time to
> implement it.(I don't read the drbd's codes, and can't estimate the cost)
>
> I think we can use 1 to implement it first.
> If you have some other idea, please let me know.
For the COLO disk replication; are you talking here about 'local storage'
and treating it as 'internal state' or 'external state' (as described in the
first half of 4.4 in the original COLO paper)?
I know some groups would like to take advantage of facilities in the storage
layer to help; e.g. take snapshots/rollback etc - so I think it's best
to do (1) but make the interface clean so that other mechanisms could be added.
Similarly I guess things like scsi-xcopy might help for some people - I'm
not saying implement these, just if possible make an interface where it could
fit later.
It's probably best to check with the QEMU storage guys that you can reuse
anything they have; there was a discussion a few weeks back where I cc'd
Fam, Stefan and Kevin in).
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service
2014-10-29 9:34 ` Dr. David Alan Gilbert
@ 2014-10-29 9:54 ` Wen Congyang
2014-10-29 11:05 ` Dr. David Alan Gilbert
2014-10-29 17:19 ` Stefan Hajnoczi
2014-10-29 10:19 ` Hongyang Yang
1 sibling, 2 replies; 36+ messages in thread
From: Wen Congyang @ 2014-10-29 9:54 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Gui Jianfeng, Jiang Yunhong, Dong Eddie, qemu-devel,
Michael R. Hines, Paolo Bonzini, Walid Nouri, Yang Hongyang
On 10/29/2014 05:34 PM, Dr. David Alan Gilbert wrote:
> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>
> <snip>
>
>> Hi all:
>>
>> I will start to implement disk replication. Before doing this, I think we should decide
>> how to implement it.
>>
>> I have two ideas about it:
>> 1. implement it in qemu
>> Advantage: very easy, and don't take too much time
>> Disadvantage: the virtio disk with vhost is not supported, because the disk I/O
>> operations are not handled in qemu.
>>
>> 2. update drbd and make it support colo
>> Advantage: we can use it for both KVM and XEN.
>> Disadvantage: The implementation may be complex, and need too much time to
>> implement it.(I don't read the drbd's codes, and can't estimate the cost)
>>
>> I think we can use 1 to implement it first.
>> If you have some other idea, please let me know.
>
> For the COLO disk replication; are you talking here about 'local storage'
> and treating it as 'internal state' or 'external state' (as described in the
> first half of 4.4 in the original COLO paper)?
I don't know what is 'internal state' or 'external state'.
What I want to implement is:
1. forward primary vm's write operation(start sector, size, content) to secondary vm
2. Cache the primary vm's write operation in secondary vm's qemu
3. cache the secondary vm's write operation in secondary vm's qemu
4. flush primary vm's write operation, and drop secondary vm's write operation after a
checkpoint
5. drop primary vm's write operation and flush secondary vm's write operation when doing
failover.
>
> I know some groups would like to take advantage of facilities in the storage
> layer to help; e.g. take snapshots/rollback etc - so I think it's best
I don't use snapshots recently years. Is it still too slow? How much time to
take snapshot/rollback?
Thanks
Wen Congyang
> to do (1) but make the interface clean so that other mechanisms could be added.
> Similarly I guess things like scsi-xcopy might help for some people - I'm
> not saying implement these, just if possible make an interface where it could
> fit later.
>
> It's probably best to check with the QEMU storage guys that you can reuse
> anything they have; there was a discussion a few weeks back where I cc'd
> Fam, Stefan and Kevin in).
>
> Dave
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> .
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service
2014-10-29 9:54 ` Wen Congyang
@ 2014-10-29 11:05 ` Dr. David Alan Gilbert
2014-10-29 17:19 ` Stefan Hajnoczi
1 sibling, 0 replies; 36+ messages in thread
From: Dr. David Alan Gilbert @ 2014-10-29 11:05 UTC (permalink / raw)
To: Wen Congyang
Cc: Gui Jianfeng, Jiang Yunhong, Dong Eddie, qemu-devel,
Michael R. Hines, Paolo Bonzini, Walid Nouri, Yang Hongyang
* Wen Congyang (wency@cn.fujitsu.com) wrote:
> On 10/29/2014 05:34 PM, Dr. David Alan Gilbert wrote:
> > * Wen Congyang (wency@cn.fujitsu.com) wrote:
> >
> > <snip>
> >
> >> Hi all:
> >>
> >> I will start to implement disk replication. Before doing this, I think we should decide
> >> how to implement it.
> >>
> >> I have two ideas about it:
> >> 1. implement it in qemu
> >> Advantage: very easy, and don't take too much time
> >> Disadvantage: the virtio disk with vhost is not supported, because the disk I/O
> >> operations are not handled in qemu.
> >>
> >> 2. update drbd and make it support colo
> >> Advantage: we can use it for both KVM and XEN.
> >> Disadvantage: The implementation may be complex, and need too much time to
> >> implement it.(I don't read the drbd's codes, and can't estimate the cost)
> >>
> >> I think we can use 1 to implement it first.
> >> If you have some other idea, please let me know.
> >
> > For the COLO disk replication; are you talking here about 'local storage'
> > and treating it as 'internal state' or 'external state' (as described in the
> > first half of 4.4 in the original COLO paper)?
>
> I don't know what is 'internal state' or 'external state'.
It's OK, Yang clarified that; it's 'local storage/internal state'.
> What I want to implement is:
> 1. forward primary vm's write operation(start sector, size, content) to secondary vm
> 2. Cache the primary vm's write operation in secondary vm's qemu
> 3. cache the secondary vm's write operation in secondary vm's qemu
> 4. flush primary vm's write operation, and drop secondary vm's write operation after a
> checkpoint
> 5. drop primary vm's write operation and flush secondary vm's write operation when doing
> failover.
OK; and each QEMU has it's own disk image file that needs to be copied at the start?
> > I know some groups would like to take advantage of facilities in the storage
> > layer to help; e.g. take snapshots/rollback etc - so I think it's best
>
> I don't use snapshots recently years. Is it still too slow? How much time to
> take snapshot/rollback?
I don't know; I'd hope it could be fast if it's a COW underneath.
Dave
> Thanks
> Wen Congyang
>
> > to do (1) but make the interface clean so that other mechanisms could be added.
> > Similarly I guess things like scsi-xcopy might help for some people - I'm
> > not saying implement these, just if possible make an interface where it could
> > fit later.
> >
> > It's probably best to check with the QEMU storage guys that you can reuse
> > anything they have; there was a discussion a few weeks back where I cc'd
> > Fam, Stefan and Kevin in).
> >
> > Dave
> >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > .
> >
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service
2014-10-29 9:54 ` Wen Congyang
2014-10-29 11:05 ` Dr. David Alan Gilbert
@ 2014-10-29 17:19 ` Stefan Hajnoczi
1 sibling, 0 replies; 36+ messages in thread
From: Stefan Hajnoczi @ 2014-10-29 17:19 UTC (permalink / raw)
To: Wen Congyang
Cc: Gui Jianfeng, Jiang Yunhong, Dong Eddie, Dr. David Alan Gilbert,
Michael R. Hines, qemu-devel, Paolo Bonzini, Walid Nouri,
Yang Hongyang
[-- Attachment #1: Type: text/plain, Size: 2039 bytes --]
On Wed, Oct 29, 2014 at 05:54:00PM +0800, Wen Congyang wrote:
> On 10/29/2014 05:34 PM, Dr. David Alan Gilbert wrote:
> > * Wen Congyang (wency@cn.fujitsu.com) wrote:
> >
> > <snip>
> >
> >> Hi all:
> >>
> >> I will start to implement disk replication. Before doing this, I think we should decide
> >> how to implement it.
> >>
> >> I have two ideas about it:
> >> 1. implement it in qemu
> >> Advantage: very easy, and don't take too much time
> >> Disadvantage: the virtio disk with vhost is not supported, because the disk I/O
> >> operations are not handled in qemu.
> >>
> >> 2. update drbd and make it support colo
> >> Advantage: we can use it for both KVM and XEN.
> >> Disadvantage: The implementation may be complex, and need too much time to
> >> implement it.(I don't read the drbd's codes, and can't estimate the cost)
> >>
> >> I think we can use 1 to implement it first.
> >> If you have some other idea, please let me know.
> >
> > For the COLO disk replication; are you talking here about 'local storage'
> > and treating it as 'internal state' or 'external state' (as described in the
> > first half of 4.4 in the original COLO paper)?
>
> I don't know what is 'internal state' or 'external state'.
> What I want to implement is:
> 1. forward primary vm's write operation(start sector, size, content) to secondary vm
> 2. Cache the primary vm's write operation in secondary vm's qemu
> 3. cache the secondary vm's write operation in secondary vm's qemu
> 4. flush primary vm's write operation, and drop secondary vm's write operation after a
> checkpoint
> 5. drop primary vm's write operation and flush secondary vm's write operation when doing
> failover.
This is more complicated than it sounds.
I suggest writing a prototype in QEMU and sending RFC patches to the
mailing list.
After the error scenarios and problems have been fully thought through,
then it will be clearer whether to implement it in QEMU or the kernel.
Stefan
[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service
2014-10-29 9:34 ` Dr. David Alan Gilbert
2014-10-29 9:54 ` Wen Congyang
@ 2014-10-29 10:19 ` Hongyang Yang
2014-10-29 11:01 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 36+ messages in thread
From: Hongyang Yang @ 2014-10-29 10:19 UTC (permalink / raw)
To: Dr. David Alan Gilbert, Wen Congyang
Cc: Gui Jianfeng, Jiang Yunhong, Dong Eddie, qemu-devel,
Michael R. Hines, Paolo Bonzini, Walid Nouri
Hi Dave,
在 10/29/2014 05:34 PM, Dr. David Alan Gilbert 写道:
> * Wen Congyang (wency@cn.fujitsu.com) wrote:
>
> <snip>
>
>> Hi all:
>>
>> I will start to implement disk replication. Before doing this, I think we should decide
>> how to implement it.
>>
>> I have two ideas about it:
>> 1. implement it in qemu
>> Advantage: very easy, and don't take too much time
>> Disadvantage: the virtio disk with vhost is not supported, because the disk I/O
>> operations are not handled in qemu.
>>
>> 2. update drbd and make it support colo
>> Advantage: we can use it for both KVM and XEN.
>> Disadvantage: The implementation may be complex, and need too much time to
>> implement it.(I don't read the drbd's codes, and can't estimate the cost)
>>
>> I think we can use 1 to implement it first.
>> If you have some other idea, please let me know.
>
> For the COLO disk replication; are you talking here about 'local storage'
> and treating it as 'internal state' or 'external state' (as described in the
> first half of 4.4 in the original COLO paper)?
'local storage' and 'internal state'. This is what later half of 4.4 said:
'COLO considers the state of local storage device as an internal s-
tate of the guest, and snapshots the local storage state as part of the
VM checkpoint, and plans to explore the other solution in future.'
>
> I know some groups would like to take advantage of facilities in the storage
> layer to help; e.g. take snapshots/rollback etc - so I think it's best
> to do (1) but make the interface clean so that other mechanisms could be added.
> Similarly I guess things like scsi-xcopy might help for some people - I'm
> not saying implement these, just if possible make an interface where it could
> fit later.
>
> It's probably best to check with the QEMU storage guys that you can reuse
> anything they have; there was a discussion a few weeks back where I cc'd
> Fam, Stefan and Kevin in).
>
> Dave
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> .
>
--
Thanks,
Yang.
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [Qemu-devel] [RFC PATCH v2 00/23] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service
2014-10-29 10:19 ` Hongyang Yang
@ 2014-10-29 11:01 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 36+ messages in thread
From: Dr. David Alan Gilbert @ 2014-10-29 11:01 UTC (permalink / raw)
To: Hongyang Yang
Cc: Gui Jianfeng, Jiang Yunhong, Dong Eddie, qemu-devel,
Michael R. Hines, Paolo Bonzini, Walid Nouri
* Hongyang Yang (yanghy@cn.fujitsu.com) wrote:
> Hi Dave,
> >For the COLO disk replication; are you talking here about 'local storage'
> >and treating it as 'internal state' or 'external state' (as described in the
> >first half of 4.4 in the original COLO paper)?
>
> 'local storage' and 'internal state'. This is what later half of 4.4 said:
> 'COLO considers the state of local storage device as an internal s-
> tate of the guest, and snapshots the local storage state as part of the
> VM checkpoint, and plans to explore the other solution in future.'
Thanks for the clarification.
Dave
>
> >
> >I know some groups would like to take advantage of facilities in the storage
> >layer to help; e.g. take snapshots/rollback etc - so I think it's best
> >to do (1) but make the interface clean so that other mechanisms could be added.
> >Similarly I guess things like scsi-xcopy might help for some people - I'm
> >not saying implement these, just if possible make an interface where it could
> >fit later.
> >
> >It's probably best to check with the QEMU storage guys that you can reuse
> >anything they have; there was a discussion a few weeks back where I cc'd
> >Fam, Stefan and Kevin in).
> >
> >Dave
> >
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >.
> >
>
> --
> Thanks,
> Yang.
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 36+ messages in thread