[Qemu-devel] [PATCH v2 0/4] Coroutines for better asynchronous programming

[Qemu-devel] [PATCH v2 0/4] Coroutines for better asynchronous programming

[Stefan Hajnoczi]

QEMU is event-driven and suffers when blocking operations are performed because
VM execution may be stopped until the operation completes.  Therefore many
operations that could block are performed asynchronously and a callback is
invoked when the operation has completed.  This allows QEMU to continue
executing while the operation is pending.

The downside to callbacks is that they split up code into many smaller
functions, each of which is a single step in a state machine that quickly
becomes complex and hard to understand.  Callback functions also result in lots
of noise as variables are packed and unpacked into temporary structs that pass
state to the callback function.

This patch series introduces coroutines as a solution for writing asynchronous
code while still having a nice sequential control flow.  The semantics are
explained in the first patch.  The second patch adds automated tests.

A nice feature of coroutines is that it is relatively easy to take synchronous
code and lift it into a coroutine to make it asynchronous.  Work has been done
to move qcow2 request processing into coroutines and thereby make it
asynchronous (today qcow2 will perform synchronous metadata accesses).  This
qcow2 work is still ongoing and not quite ready for mainline yet.

Coroutines are also being used for virtfs (virtio-9p) so I have submitted this
patch now because virtfs patches that depend on coroutines will follow shortly.

Other areas of QEMU that could take advantage of coroutines include the VNC
server, migration, and qemu-tools.

v2:
 * Added ./check-coroutine --lifecycle-benchmark for performance measurement
 * Split pooling into a separate patch with performance justification
 * Set maximum pool size to prevent holding onto too many free coroutines
 * Added atexit(3) handler to free pool
 * Coding style cleanups

 Makefile             |    3 +-
 Makefile.objs        |    7 ++
 check-coroutine.c    |  238 ++++++++++++++++++++++++++++++++++++++++++++++++++
 coroutine-ucontext.c |   73 +++++++++++++++
 coroutine-win32.c    |   57 ++++++++++++
 qemu-coroutine-int.h |   55 ++++++++++++
 qemu-coroutine.c     |  168 +++++++++++++++++++++++++++++++++++
 qemu-coroutine.h     |   91 +++++++++++++++++++
 trace-events         |    5 +
 vl.c                 |    2 +
 10 files changed, 698 insertions(+), 1 deletions(-)

[Qemu-devel] [PATCH v2 1/4] coroutine: introduce coroutines

[Stefan Hajnoczi]

From: Kevin Wolf <kwolf@redhat.com>

Asynchronous code is becoming very complex.  At the same time
synchronous code is growing because it is convenient to write.
Sometimes duplicate code paths are even added, one synchronous and the
other asynchronous.  This patch introduces coroutines which allow code
that looks synchronous but is asynchronous under the covers.

A coroutine has its own stack and is therefore able to preserve state
across blocking operations, which traditionally require callback
functions and manual marshalling of parameters.

Creating and starting a coroutine is easy:

  coroutine = qemu_coroutine_create(my_coroutine);
  qemu_coroutine_enter(coroutine, my_data);

The coroutine then executes until it returns or yields:

  void coroutine_fn my_coroutine(void *opaque) {
      MyData *my_data = opaque;

      /* do some work */

      qemu_coroutine_yield();

      /* do some more work */
  }

Yielding switches control back to the caller of qemu_coroutine_enter().
This is typically used to switch back to the main thread's event loop
after issuing an asynchronous I/O request.  The request callback will
then invoke qemu_coroutine_enter() once more to switch back to the
coroutine.

Note that coroutines never execute concurrently and should only be used
from threads which hold the global mutex.  This restriction makes
programming with coroutines easier than with threads.  Race conditions
cannot occur since only one coroutine may be active at any time.  Other
coroutines can only run across yield.

This coroutines implementation is based on the gtk-vnc implementation
written by Anthony Liguori <anthony@codemonkey.ws> but it has been
significantly rewritten by Kevin Wolf <kwolf@redhat.com> to use
setjmp()/longjmp() instead of the more expensive swapcontext().

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 Makefile.objs        |    7 +++
 coroutine-ucontext.c |   73 +++++++++++++++++++++++++++++
 coroutine-win32.c    |   57 +++++++++++++++++++++++
 qemu-coroutine-int.h |   53 +++++++++++++++++++++
 qemu-coroutine.c     |  125 ++++++++++++++++++++++++++++++++++++++++++++++++++
 qemu-coroutine.h     |   82 +++++++++++++++++++++++++++++++++
 trace-events         |    5 ++
 7 files changed, 402 insertions(+), 0 deletions(-)
 create mode 100644 coroutine-ucontext.c
 create mode 100644 coroutine-win32.c
 create mode 100644 qemu-coroutine-int.h
 create mode 100644 qemu-coroutine.c
 create mode 100644 qemu-coroutine.h

diff --git a/Makefile.objs b/Makefile.objs
index 9d8851e..cba6c2b 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -11,6 +11,12 @@ oslib-obj-$(CONFIG_WIN32) += oslib-win32.o
 oslib-obj-$(CONFIG_POSIX) += oslib-posix.o
 
 #######################################################################
+# coroutines
+coroutine-obj-y = qemu-coroutine.o
+coroutine-obj-$(CONFIG_POSIX) += coroutine-ucontext.o
+coroutine-obj-$(CONFIG_WIN32) += coroutine-win32.o
+
+#######################################################################
 # block-obj-y is code used by both qemu system emulation and qemu-img
 
 block-obj-y = cutils.o cache-utils.o qemu-malloc.o qemu-option.o module.o async.o
@@ -67,6 +73,7 @@ common-obj-y += readline.o console.o cursor.o qemu-error.o
 common-obj-y += $(oslib-obj-y)
 common-obj-$(CONFIG_WIN32) += os-win32.o
 common-obj-$(CONFIG_POSIX) += os-posix.o
+common-obj-y += $(coroutine-obj-y)
 
 common-obj-y += tcg-runtime.o host-utils.o
 common-obj-y += irq.o ioport.o input.o
diff --git a/coroutine-ucontext.c b/coroutine-ucontext.c
new file mode 100644
index 0000000..3b14ebf
--- /dev/null
+++ b/coroutine-ucontext.c
@@ -0,0 +1,73 @@
+/*
+ * ucontext coroutine initialization code
+ *
+ * Copyright (C) 2006  Anthony Liguori <anthony@codemonkey.ws>
+ * Copyright (C) 2011  Kevin Wolf <kwolf@redhat.com>
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2.0 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
+ */
+
+/* XXX Is there a nicer way to disable glibc's stack check for longjmp? */
+#ifdef _FORTIFY_SOURCE
+#undef _FORTIFY_SOURCE
+#endif
+#include <setjmp.h>
+#include <stdint.h>
+#include <ucontext.h>
+#include "qemu-coroutine-int.h"
+
+static Coroutine *new_coroutine;
+
+static void continuation_trampoline(void)
+{
+    Coroutine *co = new_coroutine;
+
+    /* Initialize longjmp environment and switch back to
+     * qemu_coroutine_init_env() in the old ucontext. */
+    if (!setjmp(co->env)) {
+        return;
+    }
+
+    while (true) {
+        co->entry(co->data);
+        if (!setjmp(co->env)) {
+            longjmp(co->caller->env, COROUTINE_TERMINATE);
+        }
+    }
+}
+
+int qemu_coroutine_init_env(Coroutine *co, size_t stack_size)
+{
+    ucontext_t old_uc, uc;
+
+    /* Create a new ucontext for switching to the coroutine stack and setting
+     * up a longjmp environment. */
+    if (getcontext(&uc) == -1) {
+        return -errno;
+    }
+
+    uc.uc_link = &old_uc;
+    uc.uc_stack.ss_sp = co->stack;
+    uc.uc_stack.ss_size = stack_size;
+    uc.uc_stack.ss_flags = 0;
+
+    new_coroutine = co;
+    makecontext(&uc, (void *)continuation_trampoline, 0);
+
+    /* Initialize the longjmp environment */
+    swapcontext(&old_uc, &uc);
+
+    return 0;
+}
diff --git a/coroutine-win32.c b/coroutine-win32.c
new file mode 100644
index 0000000..99141dd
--- /dev/null
+++ b/coroutine-win32.c
@@ -0,0 +1,57 @@
+/*
+ * Win32 coroutine initialization code
+ *
+ * Copyright (c) 2011 Kevin Wolf <kwolf@redhat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu-coroutine-int.h"
+
+static void __attribute__((used)) trampoline(Coroutine *co)
+{
+    if (!setjmp(co->env)) {
+        return;
+    }
+
+    while (true) {
+        co->entry(co->data);
+        if (!setjmp(co->env)) {
+            longjmp(co->caller->env, COROUTINE_TERMINATE);
+        }
+    }
+}
+
+int qemu_coroutine_init_env(Coroutine *co, size_t stack_size)
+{
+#ifdef __i386__
+    asm volatile(
+        "mov %%esp, %%ebx;"
+        "mov %0, %%esp;"
+        "pushl %1;"
+        "call _trampoline;"
+        "mov %%ebx, %%esp;"
+        : : "r" (co->stack + stack_size), "r" (co) : "ebx"
+    );
+#else
+    #error This host architecture is not supported for win32
+#endif
+
+    return 0;
+}
diff --git a/qemu-coroutine-int.h b/qemu-coroutine-int.h
new file mode 100644
index 0000000..71c6ee9
--- /dev/null
+++ b/qemu-coroutine-int.h
@@ -0,0 +1,53 @@
+/*
+ * Coroutine internals
+ *
+ * Copyright (c) 2011 Kevin Wolf <kwolf@redhat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#ifndef QEMU_COROUTINE_INT_H
+#define QEMU_COROUTINE_INT_H
+
+#include <setjmp.h>
+#include "qemu-common.h"
+#include "qemu-queue.h"
+#include "qemu-coroutine.h"
+
+enum {
+    /* setjmp() return values */
+    COROUTINE_YIELD = 1,
+    COROUTINE_TERMINATE = 2,
+};
+
+struct Coroutine {
+    struct Coroutine *caller;
+
+    char *stack;
+
+    /* Used to pass arguments/return values for coroutines */
+    void *data;
+    CoroutineEntry *entry;
+
+    jmp_buf env;
+};
+
+int qemu_coroutine_init_env(Coroutine *co, size_t stack_size);
+
+#endif
diff --git a/qemu-coroutine.c b/qemu-coroutine.c
new file mode 100644
index 0000000..80bed14
--- /dev/null
+++ b/qemu-coroutine.c
@@ -0,0 +1,125 @@
+/*
+ * QEMU coroutines
+ *
+ * Copyright IBM, Corp. 2011
+ *
+ * Authors:
+ *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
+ *  Kevin Wolf         <kwolf@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+/* XXX Is there a nicer way to disable glibc's stack check for longjmp? */
+#ifdef _FORTIFY_SOURCE
+#undef _FORTIFY_SOURCE
+#endif
+#include <setjmp.h>
+
+#include "trace.h"
+#include "qemu-queue.h"
+#include "qemu-common.h"
+#include "qemu-coroutine.h"
+#include "qemu-coroutine-int.h"
+
+static __thread Coroutine leader;
+static __thread Coroutine *current;
+
+static void coroutine_terminate(Coroutine *co)
+{
+    trace_qemu_coroutine_terminate(co);
+    qemu_free(co->stack);
+    qemu_free(co);
+}
+
+static Coroutine *coroutine_new(void)
+{
+    const size_t stack_size = 4 << 20;
+    Coroutine *co;
+
+    co = qemu_mallocz(sizeof(*co));
+    co->stack = qemu_malloc(stack_size);
+    qemu_coroutine_init_env(co, stack_size);
+    return co;
+}
+
+Coroutine *qemu_coroutine_create(CoroutineEntry *entry)
+{
+    Coroutine *co;
+
+    co = coroutine_new();
+    co->entry = entry;
+
+    return co;
+}
+
+Coroutine *coroutine_fn qemu_coroutine_self(void)
+{
+    if (current == NULL) {
+        current = &leader;
+    }
+
+    return current;
+}
+
+bool qemu_in_coroutine(void)
+{
+    return qemu_coroutine_self() != &leader;
+}
+
+static void *coroutine_swap(Coroutine *from, Coroutine *to, void *opaque)
+{
+    int ret;
+    void *to_data;
+
+    to->data = opaque;
+
+    ret = setjmp(from->env);
+    switch (ret) {
+    case COROUTINE_YIELD:
+        return from->data;
+    case COROUTINE_TERMINATE:
+        current = to->caller;
+        to_data = to->data;
+        coroutine_terminate(to);
+        return to_data;
+    default:
+        /* Switch to called coroutine */
+        current = to;
+        longjmp(to->env, COROUTINE_YIELD);
+        return NULL;
+    }
+}
+
+void qemu_coroutine_enter(Coroutine *co, void *opaque)
+{
+    Coroutine *self = qemu_coroutine_self();
+
+    trace_qemu_coroutine_enter(self, co, opaque);
+
+    if (co->caller) {
+        fprintf(stderr, "Co-routine re-entered recursively\n");
+        abort();
+    }
+
+    co->caller = self;
+    coroutine_swap(self, co, opaque);
+}
+
+void *coroutine_fn qemu_coroutine_yield(void)
+{
+    Coroutine *self = qemu_coroutine_self();
+    Coroutine *to = self->caller;
+
+    trace_qemu_coroutine_yield(self, self->caller);
+
+    if (!to) {
+        fprintf(stderr, "Co-routine is yielding to no one\n");
+        abort();
+    }
+
+    self->caller = NULL;
+    return coroutine_swap(self, to, NULL);
+}
diff --git a/qemu-coroutine.h b/qemu-coroutine.h
new file mode 100644
index 0000000..b79b4bf
--- /dev/null
+++ b/qemu-coroutine.h
@@ -0,0 +1,82 @@
+/*
+ * QEMU coroutine implementation
+ *
+ * Copyright IBM, Corp. 2011
+ *
+ * Authors:
+ *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_COROUTINE_H
+#define QEMU_COROUTINE_H
+
+#include <stdbool.h>
+
+/**
+ * Mark a function that executes in coroutine context
+ *
+ * Functions that execute in coroutine context cannot be called directly from
+ * normal functions.  In the future it would be nice to enable compiler or
+ * static checker support for catching such errors.  This annotation might make
+ * it possible and in the meantime it serves as documentation.
+ *
+ * For example:
+ *
+ *   static void coroutine_fn foo(void) {
+ *       ....
+ *   }
+ */
+#define coroutine_fn
+
+typedef struct Coroutine Coroutine;
+
+/**
+ * Coroutine entry point
+ *
+ * When the coroutine is entered for the first time, opaque is passed in as an
+ * argument.
+ *
+ * When this function returns, the coroutine is destroyed automatically and
+ * execution continues in the caller who last entered the coroutine.
+ */
+typedef void coroutine_fn CoroutineEntry(void *opaque);
+
+/**
+ * Create a new coroutine
+ *
+ * Use qemu_coroutine_enter() to actually transfer control to the coroutine.
+ */
+Coroutine *qemu_coroutine_create(CoroutineEntry *entry);
+
+/**
+ * Transfer control to a coroutine
+ *
+ * The opaque argument is made available to the coroutine either as the entry
+ * function argument if this is the first time a new coroutine is entered, or
+ * as the return value from qemu_coroutine_yield().
+ */
+void qemu_coroutine_enter(Coroutine *coroutine, void *opaque);
+
+/**
+ * Transfer control back to a coroutine's caller
+ *
+ * The return value is the argument passed back in from the next
+ * qemu_coroutine_enter().
+ */
+void *coroutine_fn qemu_coroutine_yield(void);
+
+/**
+ * Get the currently executing coroutine
+ */
+Coroutine *coroutine_fn qemu_coroutine_self(void);
+
+/**
+ * Return whether or not currently inside a coroutine
+ */
+bool qemu_in_coroutine(void);
+
+#endif /* QEMU_COROUTINE_H */
diff --git a/trace-events b/trace-events
index 4f965e2..2d4db05 100644
--- a/trace-events
+++ b/trace-events
@@ -361,3 +361,8 @@ disable milkymist_uart_pulse_irq_tx(void) "Pulse IRQ TX"
 # hw/milkymist-vgafb.c
 disable milkymist_vgafb_memory_read(uint32_t addr, uint32_t value) "addr %08x value %08x"
 disable milkymist_vgafb_memory_write(uint32_t addr, uint32_t value) "addr %08x value %08x"
+
+# qemu-coroutine.c
+qemu_coroutine_enter(void *from, void *to, void *opaque) "from %p to %p opaque %p"
+qemu_coroutine_yield(void *from, void *to) "from %p to %p"
+qemu_coroutine_terminate(void *co) "self %p"
-- 
1.7.4.4

[Qemu-devel] [PATCH v2 2/4] coroutine: add check-coroutine automated tests

[Stefan Hajnoczi]

To run automated tests for coroutines:

  make check-coroutine
  ./check-coroutine

On success the program terminates with exit status 0.  On failure an
error message is written to stderr and the program exits with exit
status 1.

Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 Makefile          |    3 +-
 check-coroutine.c |  188 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 190 insertions(+), 1 deletions(-)
 create mode 100644 check-coroutine.c

diff --git a/Makefile b/Makefile
index 67c0268..27ec6a1 100644
--- a/Makefile
+++ b/Makefile
@@ -130,7 +130,7 @@ qemu-io$(EXESUF): qemu-io.o cmd.o qemu-tool.o qemu-error.o $(oslib-obj-y) $(trac
 qemu-img-cmds.h: $(SRC_PATH)/qemu-img-cmds.hx
 	$(call quiet-command,sh $(SRC_PATH)/scripts/hxtool -h < $< > $@,"  GEN   $@")
 
-check-qint.o check-qstring.o check-qdict.o check-qlist.o check-qfloat.o check-qjson.o: $(GENERATED_HEADERS)
+check-qint.o check-qstring.o check-qdict.o check-qlist.o check-qfloat.o check-qjson.o check-coroutine.o: $(GENERATED_HEADERS)
 
 CHECK_PROG_DEPS = qemu-malloc.o $(oslib-obj-y) $(trace-obj-y)
 
@@ -140,6 +140,7 @@ check-qdict: check-qdict.o qdict.o qfloat.o qint.o qstring.o qbool.o qlist.o $(C
 check-qlist: check-qlist.o qlist.o qint.o $(CHECK_PROG_DEPS)
 check-qfloat: check-qfloat.o qfloat.o $(CHECK_PROG_DEPS)
 check-qjson: check-qjson.o qfloat.o qint.o qdict.o qstring.o qlist.o qbool.o qjson.o json-streamer.o json-lexer.o json-parser.o $(CHECK_PROG_DEPS)
+check-coroutine: check-coroutine.o $(coroutine-obj-y) $(CHECK_PROG_DEPS)
 
 QEMULIBS=libhw32 libhw64 libuser libdis libdis-user
 
diff --git a/check-coroutine.c b/check-coroutine.c
new file mode 100644
index 0000000..f65ac2e
--- /dev/null
+++ b/check-coroutine.c
@@ -0,0 +1,188 @@
+/*
+ * Coroutine tests
+ *
+ * Copyright IBM, Corp. 2011
+ *
+ * Authors:
+ *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include "qemu-coroutine.h"
+
+static const char *cur_test_name;
+
+static void test_assert(bool condition, const char *msg)
+{
+    if (!condition) {
+        fprintf(stderr, "%s: %s\n", cur_test_name, msg);
+        exit(EXIT_FAILURE);
+    }
+}
+
+/*
+ * Check that qemu_in_coroutine() works
+ */
+
+static void coroutine_fn verify_in_coroutine(void *opaque)
+{
+    test_assert(qemu_in_coroutine(), "expected coroutine context");
+}
+
+static void test_in_coroutine(void)
+{
+    Coroutine *coroutine;
+
+    test_assert(!qemu_in_coroutine(), "expected no coroutine context");
+
+    coroutine = qemu_coroutine_create(verify_in_coroutine);
+    qemu_coroutine_enter(coroutine, NULL);
+}
+
+/*
+ * Check that qemu_coroutine_self() works
+ */
+
+static void coroutine_fn verify_self(void *opaque)
+{
+    test_assert(qemu_coroutine_self() == opaque,
+                "qemu_coroutine_self() did not return this coroutine");
+}
+
+static void test_self(void)
+{
+    Coroutine *coroutine;
+
+    coroutine = qemu_coroutine_create(verify_self);
+    qemu_coroutine_enter(coroutine, coroutine);
+}
+
+/*
+ * Check that coroutines may nest multiple levels
+ */
+
+typedef struct {
+    unsigned int n_enter;   /* num coroutines entered */
+    unsigned int n_return;  /* num coroutines returned */
+    unsigned int max;       /* maximum level of nesting */
+} NestData;
+
+static void coroutine_fn nest(void *opaque)
+{
+    NestData *nd = opaque;
+
+    nd->n_enter++;
+
+    if (nd->n_enter < nd->max) {
+        Coroutine *child;
+
+        child = qemu_coroutine_create(nest);
+        qemu_coroutine_enter(child, nd);
+    }
+
+    nd->n_return++;
+}
+
+static void test_nesting(void)
+{
+    Coroutine *root;
+    NestData nd = {
+        .n_enter  = 0,
+        .n_return = 0,
+        .max      = 1,
+    };
+
+    root = qemu_coroutine_create(nest);
+    qemu_coroutine_enter(root, &nd);
+
+    test_assert(nd.n_enter == nd.max,
+                "failed entering to max nesting level");
+    test_assert(nd.n_return == nd.max,
+                "failed returning from max nesting level");
+}
+
+/*
+ * Check that yield/enter transfer control correctly
+ */
+
+static void coroutine_fn yield_5_times(void *opaque)
+{
+    bool *done = opaque;
+    int i;
+
+    for (i = 0; i < 5; i++) {
+        qemu_coroutine_yield();
+    }
+    *done = true;
+}
+
+static void test_yield(void)
+{
+    Coroutine *coroutine;
+    bool done = false;
+    int i = -1; /* one extra time to return from coroutine */
+
+    coroutine = qemu_coroutine_create(yield_5_times);
+    while (!done) {
+        qemu_coroutine_enter(coroutine, &done);
+        i++;
+    }
+    test_assert(i == 5, "coroutine did not yield 5 times");
+}
+
+/*
+ * Check that creation, enter, and return work
+ */
+
+static void coroutine_fn set_and_exit(void *opaque)
+{
+    bool *done = opaque;
+
+    *done = true;
+}
+
+static void test_lifecycle(void)
+{
+    Coroutine *coroutine;
+    bool done = false;
+
+    /* Create, enter, and return from coroutine */
+    coroutine = qemu_coroutine_create(set_and_exit);
+    qemu_coroutine_enter(coroutine, &done);
+    test_assert(done, "expected done to be true (first time)");
+
+    /* Repeat to check that no state affects this test */
+    done = false;
+    coroutine = qemu_coroutine_create(set_and_exit);
+    qemu_coroutine_enter(coroutine, &done);
+    test_assert(done, "expected done to be true (second time)");
+}
+
+#define TESTCASE(fn) { #fn, fn }
+int main(int argc, char **argv)
+{
+    static struct {
+        const char *name;
+        void (*run)(void);
+    } testcases[] = {
+        TESTCASE(test_lifecycle),
+        TESTCASE(test_yield),
+        TESTCASE(test_nesting),
+        TESTCASE(test_self),
+        TESTCASE(test_in_coroutine),
+        {},
+    };
+    int i;
+
+    for (i = 0; testcases[i].name; i++) {
+        cur_test_name = testcases[i].name;
+        printf("%s\n", testcases[i].name);
+        testcases[i].run();
+    }
+    return EXIT_SUCCESS;
+}
-- 
1.7.4.4

[Qemu-devel] [PATCH v2 3/4] coroutine: add check-coroutine --benchmark-lifecycle

[Stefan Hajnoczi]

Add a microbenchmark for coroutine create, enter, and return (aka
lifecycle).  This is a useful benchmark because users are expected to
create many coroutines, one per I/O request for example, and we
therefore need to provide good performance in that scenario.

To run:

  make check-coroutine
  ./check-coroutine --benchmark-lifecycle 20000000

This will do 20,000,000 coroutine create, enter, return iterations and
print the resulting time.

Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 check-coroutine.c |   48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 48 insertions(+), 0 deletions(-)

diff --git a/check-coroutine.c b/check-coroutine.c
index f65ac2e..5a42c49 100644
--- a/check-coroutine.c
+++ b/check-coroutine.c
@@ -11,8 +11,10 @@
  *
  */
 
+#include <string.h>
 #include <stdlib.h>
 #include <stdio.h>
+#include <sys/time.h>
 #include "qemu-coroutine.h"
 
 static const char *cur_test_name;
@@ -163,6 +165,43 @@ static void test_lifecycle(void)
     test_assert(done, "expected done to be true (second time)");
 }
 
+/*
+ * Lifecycle benchmark
+ */
+
+static void coroutine_fn empty_coroutine(void *opaque)
+{
+    /* Do nothing */
+}
+
+static void benchmark_lifecycle(const char *iterations)
+{
+    Coroutine *coroutine;
+    unsigned int i, max;
+    struct timeval start, finish;
+    time_t dsec;
+    suseconds_t dusec;
+
+    max = atoi(iterations);
+
+    gettimeofday(&start, NULL);
+    for (i = 0; i < max; i++) {
+        coroutine = qemu_coroutine_create(empty_coroutine);
+        qemu_coroutine_enter(coroutine, NULL);
+    }
+    gettimeofday(&finish, NULL);
+
+    dsec = finish.tv_sec - start.tv_sec;
+    if (finish.tv_usec < start.tv_usec) {
+        dsec--;
+        dusec = finish.tv_usec + 1000000 - start.tv_usec;
+    } else {
+        dusec = finish.tv_usec - start.tv_usec;
+    }
+    printf("Lifecycle %u iterations: %lu sec %lu us\n",
+           max, dsec, dusec);
+}
+
 #define TESTCASE(fn) { #fn, fn }
 int main(int argc, char **argv)
 {
@@ -179,6 +218,15 @@ int main(int argc, char **argv)
     };
     int i;
 
+    if (argc == 3 && strcmp(argv[1], "--benchmark-lifecycle") == 0) {
+        benchmark_lifecycle(argv[2]);
+        return EXIT_SUCCESS;
+    } else if (argc != 1) {
+        fprintf(stderr, "usage: %s [--benchmark-lifecycle <iterations>]\n",
+                argv[0]);
+        return EXIT_FAILURE;
+    }
+
     for (i = 0; testcases[i].name; i++) {
         cur_test_name = testcases[i].name;
         printf("%s\n", testcases[i].name);
-- 
1.7.4.4

[Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Stefan Hajnoczi]

This patch speeds up coroutine creation by reusing freed coroutines.
When a coroutine terminates it is placed in the pool instead of having
its resources freed.  The next time a coroutine is created it can be
taken straight from the pool and requires no initialization.

Performance results on an Intel Core2 Duo T9400 (2.53GHz) for
./check-coroutine --benchmark-lifecycle 20000000:

  No pooling:    19.5 sec
  With pooling:   1.1 sec

Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
---
 check-coroutine.c    |    2 ++
 qemu-coroutine-int.h |    2 ++
 qemu-coroutine.c     |   49 ++++++++++++++++++++++++++++++++++++++++++++++---
 qemu-coroutine.h     |    9 +++++++++
 vl.c                 |    2 ++
 5 files changed, 61 insertions(+), 3 deletions(-)

diff --git a/check-coroutine.c b/check-coroutine.c
index 5a42c49..223c50c 100644
--- a/check-coroutine.c
+++ b/check-coroutine.c
@@ -218,6 +218,8 @@ int main(int argc, char **argv)
     };
     int i;
 
+    qemu_coroutine_init();
+
     if (argc == 3 && strcmp(argv[1], "--benchmark-lifecycle") == 0) {
         benchmark_lifecycle(argv[2]);
         return EXIT_SUCCESS;
diff --git a/qemu-coroutine-int.h b/qemu-coroutine-int.h
index 71c6ee9..b264881 100644
--- a/qemu-coroutine-int.h
+++ b/qemu-coroutine-int.h
@@ -45,6 +45,8 @@ struct Coroutine {
     void *data;
     CoroutineEntry *entry;
 
+    QLIST_ENTRY(Coroutine) pool_next;
+
     jmp_buf env;
 };
 
diff --git a/qemu-coroutine.c b/qemu-coroutine.c
index 80bed14..cff5a4f 100644
--- a/qemu-coroutine.c
+++ b/qemu-coroutine.c
@@ -24,16 +24,35 @@
 #include "qemu-coroutine.h"
 #include "qemu-coroutine-int.h"
 
+enum {
+    /* Maximum free pool size prevents holding too many freed coroutines */
+    POOL_MAX_SIZE = 64,
+};
+
+static QLIST_HEAD(, Coroutine) pool = QLIST_HEAD_INITIALIZER(&pool);
+static unsigned int pool_size;
 static __thread Coroutine leader;
 static __thread Coroutine *current;
 
-static void coroutine_terminate(Coroutine *co)
+static void coroutine_delete(Coroutine *co)
 {
-    trace_qemu_coroutine_terminate(co);
     qemu_free(co->stack);
     qemu_free(co);
 }
 
+static void coroutine_terminate(Coroutine *co)
+{
+    trace_qemu_coroutine_terminate(co);
+
+    if (pool_size < POOL_MAX_SIZE) {
+        QLIST_INSERT_HEAD(&pool, co, pool_next);
+        co->caller = NULL;
+        pool_size++;
+    } else {
+        coroutine_delete(co);
+    }
+}
+
 static Coroutine *coroutine_new(void)
 {
     const size_t stack_size = 4 << 20;
@@ -49,7 +68,15 @@ Coroutine *qemu_coroutine_create(CoroutineEntry *entry)
 {
     Coroutine *co;
 
-    co = coroutine_new();
+    co = QLIST_FIRST(&pool);
+
+    if (co) {
+        QLIST_REMOVE(co, pool_next);
+        pool_size--;
+    } else {
+        co = coroutine_new();
+    }
+
     co->entry = entry;
 
     return co;
@@ -123,3 +150,19 @@ void *coroutine_fn qemu_coroutine_yield(void)
     self->caller = NULL;
     return coroutine_swap(self, to, NULL);
 }
+
+static void coroutine_free_pool(void)
+{
+    Coroutine *co;
+    Coroutine *tmp;
+
+    QLIST_FOREACH_SAFE(co, &pool, pool_next, tmp) {
+        coroutine_delete(co);
+    }
+    pool_size = 0;
+}
+
+void qemu_coroutine_init(void)
+{
+    atexit(coroutine_free_pool);
+}
diff --git a/qemu-coroutine.h b/qemu-coroutine.h
index b79b4bf..0fae0a4 100644
--- a/qemu-coroutine.h
+++ b/qemu-coroutine.h
@@ -17,6 +17,15 @@
 #include <stdbool.h>
 
 /**
+ * Initialize coroutine implementation
+ *
+ * This function should be called when the program is starting up.  It installs
+ * an atexit(3) handler which will free pooled coroutines when the program
+ * exits.  This keeps valgrind output clean.
+ */
+void qemu_coroutine_init(void);
+
+/**
  * Mark a function that executes in coroutine context
  *
  * Functions that execute in coroutine context cannot be called directly from
diff --git a/vl.c b/vl.c
index 6b9a2f6..7092c9f 100644
--- a/vl.c
+++ b/vl.c
@@ -145,6 +145,7 @@ int main(int argc, char **argv)
 #include "qemu-config.h"
 #include "qemu-objects.h"
 #include "qemu-options.h"
+#include "qemu-coroutine.h"
 #ifdef CONFIG_VIRTFS
 #include "fsdev/qemu-fsdev.h"
 #endif
@@ -1975,6 +1976,7 @@ int main(int argc, char **argv, char **envp)
     init_clocks();
 
     qemu_cache_utils_init(envp);
+    qemu_coroutine_init();
 
     QLIST_INIT (&vm_change_state_head);
     os_setup_early_signal_handling();
-- 
1.7.4.4

Re: [Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Kevin Wolf]

terAm 12.05.2011 11:54, schrieb Stefan Hajnoczi:
> This patch speeds up coroutine creation by reusing freed coroutines.
> When a coroutine terminates it is placed in the pool instead of having
> its resources freed.  The next time a coroutine is created it can be
> taken straight from the pool and requires no initialization.
> 
> Performance results on an Intel Core2 Duo T9400 (2.53GHz) for
> ./check-coroutine --benchmark-lifecycle 20000000:
> 
>   No pooling:    19.5 sec
>   With pooling:   1.1 sec
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
> ---
>  check-coroutine.c    |    2 ++
>  qemu-coroutine-int.h |    2 ++
>  qemu-coroutine.c     |   49 ++++++++++++++++++++++++++++++++++++++++++++++---
>  qemu-coroutine.h     |    9 +++++++++
>  vl.c                 |    2 ++
>  5 files changed, 61 insertions(+), 3 deletions(-)
> 
> diff --git a/check-coroutine.c b/check-coroutine.c
> index 5a42c49..223c50c 100644
> --- a/check-coroutine.c
> +++ b/check-coroutine.c
> @@ -218,6 +218,8 @@ int main(int argc, char **argv)
>      };
>      int i;
>  
> +    qemu_coroutine_init();

Can we use module_init instead of adding an explicit call to main()?
This would prevent forgetting to add it in qemu-img and qemu-io like in
this patch.

Kevin

Re: [Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Stefan Hajnoczi]

On Thu, May 12, 2011 at 11:13 AM, Kevin Wolf <kwolf@redhat.com> wrote:
> terAm 12.05.2011 11:54, schrieb Stefan Hajnoczi:
>> This patch speeds up coroutine creation by reusing freed coroutines.
>> When a coroutine terminates it is placed in the pool instead of having
>> its resources freed.  The next time a coroutine is created it can be
>> taken straight from the pool and requires no initialization.
>>
>> Performance results on an Intel Core2 Duo T9400 (2.53GHz) for
>> ./check-coroutine --benchmark-lifecycle 20000000:
>>
>>   No pooling:    19.5 sec
>>   With pooling:   1.1 sec
>>
>> Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
>> ---
>>  check-coroutine.c    |    2 ++
>>  qemu-coroutine-int.h |    2 ++
>>  qemu-coroutine.c     |   49 ++++++++++++++++++++++++++++++++++++++++++++++---
>>  qemu-coroutine.h     |    9 +++++++++
>>  vl.c                 |    2 ++
>>  5 files changed, 61 insertions(+), 3 deletions(-)
>>
>> diff --git a/check-coroutine.c b/check-coroutine.c
>> index 5a42c49..223c50c 100644
>> --- a/check-coroutine.c
>> +++ b/check-coroutine.c
>> @@ -218,6 +218,8 @@ int main(int argc, char **argv)
>>      };
>>      int i;
>>
>> +    qemu_coroutine_init();
>
> Can we use module_init instead of adding an explicit call to main()?
> This would prevent forgetting to add it in qemu-img and qemu-io like in
> this patch.

module_init what? :)  qemu-img/qemu-io only init MODULE_INIT_BLOCK so
we'd have to modify them anyway.

I don't want to add qemu-img/qemu-io things yet because we don't have
a block layer user for coroutines yet.  The qcow2 patches should
contain these changes.

Stefan

Re: [Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Kevin Wolf]

Am 12.05.2011 12:22, schrieb Stefan Hajnoczi:
> On Thu, May 12, 2011 at 11:13 AM, Kevin Wolf <kwolf@redhat.com> wrote:
>> terAm 12.05.2011 11:54, schrieb Stefan Hajnoczi:
>>> This patch speeds up coroutine creation by reusing freed coroutines.
>>> When a coroutine terminates it is placed in the pool instead of having
>>> its resources freed.  The next time a coroutine is created it can be
>>> taken straight from the pool and requires no initialization.
>>>
>>> Performance results on an Intel Core2 Duo T9400 (2.53GHz) for
>>> ./check-coroutine --benchmark-lifecycle 20000000:
>>>
>>>   No pooling:    19.5 sec
>>>   With pooling:   1.1 sec
>>>
>>> Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
>>> ---
>>>  check-coroutine.c    |    2 ++
>>>  qemu-coroutine-int.h |    2 ++
>>>  qemu-coroutine.c     |   49 ++++++++++++++++++++++++++++++++++++++++++++++---
>>>  qemu-coroutine.h     |    9 +++++++++
>>>  vl.c                 |    2 ++
>>>  5 files changed, 61 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/check-coroutine.c b/check-coroutine.c
>>> index 5a42c49..223c50c 100644
>>> --- a/check-coroutine.c
>>> +++ b/check-coroutine.c
>>> @@ -218,6 +218,8 @@ int main(int argc, char **argv)
>>>      };
>>>      int i;
>>>
>>> +    qemu_coroutine_init();
>>
>> Can we use module_init instead of adding an explicit call to main()?
>> This would prevent forgetting to add it in qemu-img and qemu-io like in
>> this patch.
> 
> module_init what? :)  qemu-img/qemu-io only init MODULE_INIT_BLOCK so
> we'd have to modify them anyway.

Right... I was thinking of block, but in fact coroutines are not limited
to the block layer, so we would abuse it.

> I don't want to add qemu-img/qemu-io things yet because we don't have
> a block layer user for coroutines yet.  The qcow2 patches should
> contain these changes.

I hope we won't forget it. A missing atexit isn't a very obvious bug.

Kevin

Re: [Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Paolo Bonzini]

On 05/12/2011 12:38 PM, Kevin Wolf wrote:
>> I don't want to add qemu-img/qemu-io things yet because we don't have
>> a block layer user for coroutines yet.  The qcow2 patches should
>> contain these changes.
>
> I hope we won't forget it. A missing atexit isn't a very obvious bug.

I was going to reply that this atexit is actually useless, since memory 
is freed automatically at exit?

Paolo

Re: [Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Stefan Hajnoczi]

On Thu, May 12, 2011 at 12:12 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 05/12/2011 12:38 PM, Kevin Wolf wrote:
>>>
>>> I don't want to add qemu-img/qemu-io things yet because we don't have
>>> a block layer user for coroutines yet.  The qcow2 patches should
>>> contain these changes.
>>
>> I hope we won't forget it. A missing atexit isn't a very obvious bug.
>
> I was going to reply that this atexit is actually useless, since memory is
> freed automatically at exit?

It's just for completeness to make tools like valgrind happy.  Sure,
the kernel will reclaim memory and we're just burning CPU by freeing
this stuff ;).

Stefan

Re: [Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Paolo Bonzini]

On 05/12/2011 01:15 PM, Stefan Hajnoczi wrote:
> It's just for completeness to make tools like valgrind happy.  Sure,
> the kernel will reclaim memory and we're just burning CPU by freeing
> this stuff;).

But valgrind will not complain about reachable memory still allocated at 
exit, at least not with the default command-line options.

Paolo

Re: [Qemu-devel] [PATCH v2 4/4] coroutine: pool coroutines to speed up creation

[Stefan Hajnoczi]

On Thu, May 12, 2011 at 12:18 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 05/12/2011 01:15 PM, Stefan Hajnoczi wrote:
>>
>> It's just for completeness to make tools like valgrind happy.  Sure,
>> the kernel will reclaim memory and we're just burning CPU by freeing
>> this stuff;).
>
> But valgrind will not complain about reachable memory still allocated at
> exit, at least not with the default command-line options.

/me is outsmarted by valgrind

Then I will remove pool freeing in v3.

Stefan

Re: [Qemu-devel] [PATCH v2 1/4] coroutine: introduce coroutines

[Blue Swirl]

On Thu, May 12, 2011 at 12:54 PM, Stefan Hajnoczi
<stefanha@linux.vnet.ibm.com> wrote:
> From: Kevin Wolf <kwolf@redhat.com>
>
> Asynchronous code is becoming very complex.  At the same time
> synchronous code is growing because it is convenient to write.
> Sometimes duplicate code paths are even added, one synchronous and the
> other asynchronous.  This patch introduces coroutines which allow code
> that looks synchronous but is asynchronous under the covers.
>
> A coroutine has its own stack and is therefore able to preserve state
> across blocking operations, which traditionally require callback
> functions and manual marshalling of parameters.
>
> Creating and starting a coroutine is easy:
>
>  coroutine = qemu_coroutine_create(my_coroutine);
>  qemu_coroutine_enter(coroutine, my_data);
>
> The coroutine then executes until it returns or yields:
>
>  void coroutine_fn my_coroutine(void *opaque) {
>      MyData *my_data = opaque;
>
>      /* do some work */
>
>      qemu_coroutine_yield();
>
>      /* do some more work */
>  }
>
> Yielding switches control back to the caller of qemu_coroutine_enter().
> This is typically used to switch back to the main thread's event loop
> after issuing an asynchronous I/O request.  The request callback will
> then invoke qemu_coroutine_enter() once more to switch back to the
> coroutine.
>
> Note that coroutines never execute concurrently and should only be used
> from threads which hold the global mutex.  This restriction makes
> programming with coroutines easier than with threads.  Race conditions
> cannot occur since only one coroutine may be active at any time.  Other
> coroutines can only run across yield.
>
> This coroutines implementation is based on the gtk-vnc implementation
> written by Anthony Liguori <anthony@codemonkey.ws> but it has been
> significantly rewritten by Kevin Wolf <kwolf@redhat.com> to use
> setjmp()/longjmp() instead of the more expensive swapcontext().
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@linux.vnet.ibm.com>
> ---
>  Makefile.objs        |    7 +++
>  coroutine-ucontext.c |   73 +++++++++++++++++++++++++++++
>  coroutine-win32.c    |   57 +++++++++++++++++++++++
>  qemu-coroutine-int.h |   53 +++++++++++++++++++++
>  qemu-coroutine.c     |  125 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  qemu-coroutine.h     |   82 +++++++++++++++++++++++++++++++++
>  trace-events         |    5 ++
>  7 files changed, 402 insertions(+), 0 deletions(-)
>  create mode 100644 coroutine-ucontext.c
>  create mode 100644 coroutine-win32.c
>  create mode 100644 qemu-coroutine-int.h
>  create mode 100644 qemu-coroutine.c
>  create mode 100644 qemu-coroutine.h
>
> diff --git a/Makefile.objs b/Makefile.objs
> index 9d8851e..cba6c2b 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -11,6 +11,12 @@ oslib-obj-$(CONFIG_WIN32) += oslib-win32.o
>  oslib-obj-$(CONFIG_POSIX) += oslib-posix.o
>
>  #######################################################################
> +# coroutines
> +coroutine-obj-y = qemu-coroutine.o
> +coroutine-obj-$(CONFIG_POSIX) += coroutine-ucontext.o
> +coroutine-obj-$(CONFIG_WIN32) += coroutine-win32.o
> +
> +#######################################################################
>  # block-obj-y is code used by both qemu system emulation and qemu-img
>
>  block-obj-y = cutils.o cache-utils.o qemu-malloc.o qemu-option.o module.o async.o
> @@ -67,6 +73,7 @@ common-obj-y += readline.o console.o cursor.o qemu-error.o
>  common-obj-y += $(oslib-obj-y)
>  common-obj-$(CONFIG_WIN32) += os-win32.o
>  common-obj-$(CONFIG_POSIX) += os-posix.o
> +common-obj-y += $(coroutine-obj-y)
>
>  common-obj-y += tcg-runtime.o host-utils.o
>  common-obj-y += irq.o ioport.o input.o
> diff --git a/coroutine-ucontext.c b/coroutine-ucontext.c
> new file mode 100644
> index 0000000..3b14ebf
> --- /dev/null
> +++ b/coroutine-ucontext.c
> @@ -0,0 +1,73 @@
> +/*
> + * ucontext coroutine initialization code
> + *
> + * Copyright (C) 2006  Anthony Liguori <anthony@codemonkey.ws>
> + * Copyright (C) 2011  Kevin Wolf <kwolf@redhat.com>
> + *
> + * This library is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.0 of the License, or (at your option) any later version.
> + *
> + * This library is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with this library; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA

Please use the web link version.

> + */
> +
> +/* XXX Is there a nicer way to disable glibc's stack check for longjmp? */

What is the problem?

> +#ifdef _FORTIFY_SOURCE
> +#undef _FORTIFY_SOURCE
> +#endif
> +#include <setjmp.h>
> +#include <stdint.h>
> +#include <ucontext.h>
> +#include "qemu-coroutine-int.h"
> +
> +static Coroutine *new_coroutine;
> +
> +static void continuation_trampoline(void)
> +{
> +    Coroutine *co = new_coroutine;
> +
> +    /* Initialize longjmp environment and switch back to
> +     * qemu_coroutine_init_env() in the old ucontext. */
> +    if (!setjmp(co->env)) {
> +        return;
> +    }
> +
> +    while (true) {
> +        co->entry(co->data);
> +        if (!setjmp(co->env)) {
> +            longjmp(co->caller->env, COROUTINE_TERMINATE);
> +        }
> +    }
> +}
> +
> +int qemu_coroutine_init_env(Coroutine *co, size_t stack_size)
> +{
> +    ucontext_t old_uc, uc;
> +
> +    /* Create a new ucontext for switching to the coroutine stack and setting
> +     * up a longjmp environment. */
> +    if (getcontext(&uc) == -1) {
> +        return -errno;
> +    }
> +
> +    uc.uc_link = &old_uc;
> +    uc.uc_stack.ss_sp = co->stack;
> +    uc.uc_stack.ss_size = stack_size;
> +    uc.uc_stack.ss_flags = 0;
> +
> +    new_coroutine = co;
> +    makecontext(&uc, (void *)continuation_trampoline, 0);
> +
> +    /* Initialize the longjmp environment */
> +    swapcontext(&old_uc, &uc);
> +
> +    return 0;
> +}
> diff --git a/coroutine-win32.c b/coroutine-win32.c
> new file mode 100644
> index 0000000..99141dd
> --- /dev/null
> +++ b/coroutine-win32.c
> @@ -0,0 +1,57 @@
> +/*
> + * Win32 coroutine initialization code
> + *
> + * Copyright (c) 2011 Kevin Wolf <kwolf@redhat.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.
> + */
> +
> +#include "qemu-coroutine-int.h"
> +
> +static void __attribute__((used)) trampoline(Coroutine *co)
> +{
> +    if (!setjmp(co->env)) {
> +        return;
> +    }
> +
> +    while (true) {
> +        co->entry(co->data);
> +        if (!setjmp(co->env)) {
> +            longjmp(co->caller->env, COROUTINE_TERMINATE);
> +        }
> +    }
> +}
> +
> +int qemu_coroutine_init_env(Coroutine *co, size_t stack_size)
> +{
> +#ifdef __i386__
> +    asm volatile(
> +        "mov %%esp, %%ebx;"
> +        "mov %0, %%esp;"
> +        "pushl %1;"
> +        "call _trampoline;"
> +        "mov %%ebx, %%esp;"
> +        : : "r" (co->stack + stack_size), "r" (co) : "ebx"
> +    );
> +#else
> +    #error This host architecture is not supported for win32
> +#endif
> +
> +    return 0;
> +}
> diff --git a/qemu-coroutine-int.h b/qemu-coroutine-int.h
> new file mode 100644
> index 0000000..71c6ee9
> --- /dev/null
> +++ b/qemu-coroutine-int.h
> @@ -0,0 +1,53 @@
> +/*
> + * Coroutine internals
> + *
> + * Copyright (c) 2011 Kevin Wolf <kwolf@redhat.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.
> + */
> +
> +#ifndef QEMU_COROUTINE_INT_H
> +#define QEMU_COROUTINE_INT_H
> +
> +#include <setjmp.h>
> +#include "qemu-common.h"
> +#include "qemu-queue.h"
> +#include "qemu-coroutine.h"
> +
> +enum {
> +    /* setjmp() return values */
> +    COROUTINE_YIELD = 1,
> +    COROUTINE_TERMINATE = 2,
> +};
> +
> +struct Coroutine {
> +    struct Coroutine *caller;
> +
> +    char *stack;
> +
> +    /* Used to pass arguments/return values for coroutines */
> +    void *data;
> +    CoroutineEntry *entry;
> +
> +    jmp_buf env;
> +};
> +
> +int qemu_coroutine_init_env(Coroutine *co, size_t stack_size);
> +
> +#endif
> diff --git a/qemu-coroutine.c b/qemu-coroutine.c
> new file mode 100644
> index 0000000..80bed14
> --- /dev/null
> +++ b/qemu-coroutine.c
> @@ -0,0 +1,125 @@
> +/*
> + * QEMU coroutines
> + *
> + * Copyright IBM, Corp. 2011
> + *
> + * Authors:
> + *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
> + *  Kevin Wolf         <kwolf@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or later.
> + * See the COPYING.LIB file in the top-level directory.
> + *
> + */
> +
> +/* XXX Is there a nicer way to disable glibc's stack check for longjmp? */
> +#ifdef _FORTIFY_SOURCE
> +#undef _FORTIFY_SOURCE
> +#endif
> +#include <setjmp.h>
> +
> +#include "trace.h"
> +#include "qemu-queue.h"
> +#include "qemu-common.h"
> +#include "qemu-coroutine.h"
> +#include "qemu-coroutine-int.h"
> +
> +static __thread Coroutine leader;

$ cat thread.c
static __thread int sigusr1_wfd;
$ gcc thread.c -c -pthread
thread.c:1: error: thread-local storage not supported for this target
$ gcc -v
Reading specs from
/usr/bin/../lib/gcc-lib/sparc64-unknown-openbsd4.9/4.2.1/specs
Target: sparc64-unknown-openbsd4.9
Configured with: OpenBSD/sparc64 system compiler
Thread model: posix
gcc version 4.2.1 20070719

> +static __thread Coroutine *current;
> +
> +static void coroutine_terminate(Coroutine *co)
> +{
> +    trace_qemu_coroutine_terminate(co);
> +    qemu_free(co->stack);
> +    qemu_free(co);
> +}
> +
> +static Coroutine *coroutine_new(void)
> +{
> +    const size_t stack_size = 4 << 20;
> +    Coroutine *co;
> +
> +    co = qemu_mallocz(sizeof(*co));
> +    co->stack = qemu_malloc(stack_size);
> +    qemu_coroutine_init_env(co, stack_size);
> +    return co;
> +}
> +
> +Coroutine *qemu_coroutine_create(CoroutineEntry *entry)
> +{
> +    Coroutine *co;
> +
> +    co = coroutine_new();
> +    co->entry = entry;
> +
> +    return co;
> +}
> +
> +Coroutine *coroutine_fn qemu_coroutine_self(void)
> +{
> +    if (current == NULL) {
> +        current = &leader;
> +    }
> +
> +    return current;
> +}
> +
> +bool qemu_in_coroutine(void)
> +{
> +    return qemu_coroutine_self() != &leader;
> +}
> +
> +static void *coroutine_swap(Coroutine *from, Coroutine *to, void *opaque)
> +{
> +    int ret;
> +    void *to_data;
> +
> +    to->data = opaque;
> +
> +    ret = setjmp(from->env);
> +    switch (ret) {
> +    case COROUTINE_YIELD:
> +        return from->data;
> +    case COROUTINE_TERMINATE:
> +        current = to->caller;
> +        to_data = to->data;
> +        coroutine_terminate(to);
> +        return to_data;
> +    default:
> +        /* Switch to called coroutine */
> +        current = to;
> +        longjmp(to->env, COROUTINE_YIELD);
> +        return NULL;
> +    }
> +}
> +
> +void qemu_coroutine_enter(Coroutine *co, void *opaque)
> +{
> +    Coroutine *self = qemu_coroutine_self();
> +
> +    trace_qemu_coroutine_enter(self, co, opaque);
> +
> +    if (co->caller) {
> +        fprintf(stderr, "Co-routine re-entered recursively\n");
> +        abort();
> +    }
> +
> +    co->caller = self;
> +    coroutine_swap(self, co, opaque);
> +}
> +
> +void *coroutine_fn qemu_coroutine_yield(void)
> +{
> +    Coroutine *self = qemu_coroutine_self();
> +    Coroutine *to = self->caller;
> +
> +    trace_qemu_coroutine_yield(self, self->caller);
> +
> +    if (!to) {
> +        fprintf(stderr, "Co-routine is yielding to no one\n");
> +        abort();
> +    }
> +
> +    self->caller = NULL;
> +    return coroutine_swap(self, to, NULL);
> +}
> diff --git a/qemu-coroutine.h b/qemu-coroutine.h
> new file mode 100644
> index 0000000..b79b4bf
> --- /dev/null
> +++ b/qemu-coroutine.h
> @@ -0,0 +1,82 @@
> +/*
> + * QEMU coroutine implementation
> + *
> + * Copyright IBM, Corp. 2011
> + *
> + * Authors:
> + *  Stefan Hajnoczi    <stefanha@linux.vnet.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU LGPL, version 2 or later.
> + * See the COPYING.LIB file in the top-level directory.
> + *
> + */
> +
> +#ifndef QEMU_COROUTINE_H
> +#define QEMU_COROUTINE_H
> +
> +#include <stdbool.h>
> +
> +/**
> + * Mark a function that executes in coroutine context
> + *
> + * Functions that execute in coroutine context cannot be called directly from
> + * normal functions.  In the future it would be nice to enable compiler or
> + * static checker support for catching such errors.  This annotation might make
> + * it possible and in the meantime it serves as documentation.
> + *
> + * For example:
> + *
> + *   static void coroutine_fn foo(void) {
> + *       ....
> + *   }
> + */
> +#define coroutine_fn
> +
> +typedef struct Coroutine Coroutine;
> +
> +/**
> + * Coroutine entry point
> + *
> + * When the coroutine is entered for the first time, opaque is passed in as an
> + * argument.
> + *
> + * When this function returns, the coroutine is destroyed automatically and
> + * execution continues in the caller who last entered the coroutine.
> + */
> +typedef void coroutine_fn CoroutineEntry(void *opaque);
> +
> +/**
> + * Create a new coroutine
> + *
> + * Use qemu_coroutine_enter() to actually transfer control to the coroutine.
> + */
> +Coroutine *qemu_coroutine_create(CoroutineEntry *entry);
> +
> +/**
> + * Transfer control to a coroutine
> + *
> + * The opaque argument is made available to the coroutine either as the entry
> + * function argument if this is the first time a new coroutine is entered, or
> + * as the return value from qemu_coroutine_yield().
> + */
> +void qemu_coroutine_enter(Coroutine *coroutine, void *opaque);
> +
> +/**
> + * Transfer control back to a coroutine's caller
> + *
> + * The return value is the argument passed back in from the next
> + * qemu_coroutine_enter().
> + */
> +void *coroutine_fn qemu_coroutine_yield(void);
> +
> +/**
> + * Get the currently executing coroutine
> + */
> +Coroutine *coroutine_fn qemu_coroutine_self(void);
> +
> +/**
> + * Return whether or not currently inside a coroutine
> + */
> +bool qemu_in_coroutine(void);
> +
> +#endif /* QEMU_COROUTINE_H */
> diff --git a/trace-events b/trace-events
> index 4f965e2..2d4db05 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -361,3 +361,8 @@ disable milkymist_uart_pulse_irq_tx(void) "Pulse IRQ TX"
>  # hw/milkymist-vgafb.c
>  disable milkymist_vgafb_memory_read(uint32_t addr, uint32_t value) "addr %08x value %08x"
>  disable milkymist_vgafb_memory_write(uint32_t addr, uint32_t value) "addr %08x value %08x"
> +
> +# qemu-coroutine.c
> +qemu_coroutine_enter(void *from, void *to, void *opaque) "from %p to %p opaque %p"
> +qemu_coroutine_yield(void *from, void *to) "from %p to %p"
> +qemu_coroutine_terminate(void *co) "self %p"

Not disabled?

Re: [Qemu-devel] [PATCH v2 1/4] coroutine: introduce coroutines

[Stefan Hajnoczi]

On Thu, May 12, 2011 at 7:12 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
> On Thu, May 12, 2011 at 12:54 PM, Stefan Hajnoczi
> <stefanha@linux.vnet.ibm.com> wrote:
>> diff --git a/coroutine-ucontext.c b/coroutine-ucontext.c
>> new file mode 100644
>> index 0000000..3b14ebf
>> --- /dev/null
>> +++ b/coroutine-ucontext.c
>> @@ -0,0 +1,73 @@
>> +/*
>> + * ucontext coroutine initialization code
>> + *
>> + * Copyright (C) 2006  Anthony Liguori <anthony@codemonkey.ws>
>> + * Copyright (C) 2011  Kevin Wolf <kwolf@redhat.com>
>> + *
>> + * This library is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU Lesser General Public
>> + * License as published by the Free Software Foundation; either
>> + * version 2.0 of the License, or (at your option) any later version.
>> + *
>> + * This library is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + * Lesser General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU Lesser General Public
>> + * License along with this library; if not, write to the Free Software
>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
>
> Please use the web link version.

Will update in v3.

>> + */
>> +
>> +/* XXX Is there a nicer way to disable glibc's stack check for longjmp? */
>
> What is the problem?

Kevin?

>> +static __thread Coroutine leader;
>
> $ cat thread.c
> static __thread int sigusr1_wfd;
> $ gcc thread.c -c -pthread
> thread.c:1: error: thread-local storage not supported for this target
> $ gcc -v
> Reading specs from
> /usr/bin/../lib/gcc-lib/sparc64-unknown-openbsd4.9/4.2.1/specs
> Target: sparc64-unknown-openbsd4.9
> Configured with: OpenBSD/sparc64 system compiler
> Thread model: posix
> gcc version 4.2.1 20070719

We had the thread-local variables in as part of the gtk-vnc history of
this code.  In QEMU we don't need it:

"Note that coroutines never execute concurrently and should only be used
from threads which hold the global mutex."

So I will remove the thread-local attribute.  Thank you for testing
that platform.

>> diff --git a/trace-events b/trace-events
>> index 4f965e2..2d4db05 100644
>> --- a/trace-events
>> +++ b/trace-events
>> @@ -361,3 +361,8 @@ disable milkymist_uart_pulse_irq_tx(void) "Pulse IRQ TX"
>>  # hw/milkymist-vgafb.c
>>  disable milkymist_vgafb_memory_read(uint32_t addr, uint32_t value) "addr %08x value %08x"
>>  disable milkymist_vgafb_memory_write(uint32_t addr, uint32_t value) "addr %08x value %08x"
>> +
>> +# qemu-coroutine.c
>> +qemu_coroutine_enter(void *from, void *to, void *opaque) "from %p to %p opaque %p"
>> +qemu_coroutine_yield(void *from, void *to) "from %p to %p"
>> +qemu_coroutine_terminate(void *co) "self %p"
>
> Not disabled?

Good catch.

Stefan

Re: [Qemu-devel] [PATCH v2 1/4] coroutine: introduce coroutines

[Kevin Wolf]

Am 12.05.2011 21:22, schrieb Stefan Hajnoczi:
> On Thu, May 12, 2011 at 7:12 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
>> On Thu, May 12, 2011 at 12:54 PM, Stefan Hajnoczi
>> <stefanha@linux.vnet.ibm.com> wrote:
>>> diff --git a/coroutine-ucontext.c b/coroutine-ucontext.c
>>> new file mode 100644
>>> index 0000000..3b14ebf
>>> --- /dev/null
>>> +++ b/coroutine-ucontext.c
>>> @@ -0,0 +1,73 @@
>>> +/*
>>> + * ucontext coroutine initialization code
>>> + *
>>> + * Copyright (C) 2006  Anthony Liguori <anthony@codemonkey.ws>
>>> + * Copyright (C) 2011  Kevin Wolf <kwolf@redhat.com>
>>> + *
>>> + * This library is free software; you can redistribute it and/or
>>> + * modify it under the terms of the GNU Lesser General Public
>>> + * License as published by the Free Software Foundation; either
>>> + * version 2.0 of the License, or (at your option) any later version.
>>> + *
>>> + * This library is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> + * Lesser General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU Lesser General Public
>>> + * License along with this library; if not, write to the Free Software
>>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301 USA
>>
>> Please use the web link version.
> 
> Will update in v3.
> 
>>> + */
>>> +
>>> +/* XXX Is there a nicer way to disable glibc's stack check for longjmp? */
>>
>> What is the problem?
> 
> Kevin?

I don't remember the details of the mechanism, but with _FORTIFY_SOURCE
glibc has some check in longjmp that doesn't like stack switches. If you
uncomment the #undef, you should be able to reproduce the abort().

Kevin