[Qemu-devel] [PATCH 2/4] coroutine-lock: convert CoQueue to use QemuLockable

[Qemu-devel] [PATCH 2/4] coroutine-lock: convert CoQueue to use QemuLockable

[Paolo Bonzini]

There are cases in which a queued coroutine must be restarted from
non-coroutine context (with qemu_co_enter_next).  In this cases,
qemu_co_enter_next also needs to be thread-safe, but it cannot use
a CoMutex and so cannot qemu_co_queue_wait.  Use QemuLockable so
that the CoQueue can interchangeably use CoMutex or QemuMutex.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 include/qemu/coroutine.h   |  6 +++++-
 util/qemu-coroutine-lock.c | 12 +++++++-----
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h
index 8a5129741c..1e5f0957e6 100644
--- a/include/qemu/coroutine.h
+++ b/include/qemu/coroutine.h
@@ -183,7 +183,9 @@ void qemu_co_queue_init(CoQueue *queue);
  * caller of the coroutine.  The mutex is unlocked during the wait and
  * locked again afterwards.
  */
-void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex);
+#define qemu_co_queue_wait(queue, lock) \
+    qemu_co_queue_wait_impl(queue, QEMU_MAKE_LOCKABLE(lock))
+void coroutine_fn qemu_co_queue_wait_impl(CoQueue *queue, QemuLockable *lock);
 
 /**
  * Restarts the next coroutine in the CoQueue and removes it from the queue.
@@ -271,4 +273,6 @@ void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns);
  */
 void coroutine_fn yield_until_fd_readable(int fd);
 
+#include "qemu/lockable.h"
+
 #endif /* QEMU_COROUTINE_H */
diff --git a/util/qemu-coroutine-lock.c b/util/qemu-coroutine-lock.c
index 846ff9167f..2a66fc1467 100644
--- a/util/qemu-coroutine-lock.c
+++ b/util/qemu-coroutine-lock.c
@@ -40,13 +40,13 @@ void qemu_co_queue_init(CoQueue *queue)
     QSIMPLEQ_INIT(&queue->entries);
 }
 
-void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex)
+void coroutine_fn qemu_co_queue_wait_impl(CoQueue *queue, QemuLockable *lock)
 {
     Coroutine *self = qemu_coroutine_self();
     QSIMPLEQ_INSERT_TAIL(&queue->entries, self, co_queue_next);
 
-    if (mutex) {
-        qemu_co_mutex_unlock(mutex);
+    if (lock) {
+        qemu_lockable_unlock(lock);
     }
 
     /* There is no race condition here.  Other threads will call
@@ -60,9 +60,11 @@ void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex)
     /* TODO: OSv implements wait morphing here, where the wakeup
      * primitive automatically places the woken coroutine on the
      * mutex's queue.  This avoids the thundering herd effect.
+     * This could be implemented for CoMutexes, but not really for
+     * other cases of QemuLockable.
      */
-    if (mutex) {
-        qemu_co_mutex_lock(mutex);
+    if (lock) {
+        qemu_lockable_lock(lock);
     }
 }
 
-- 
2.14.3

[Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

[Paolo Bonzini]

There are cases in which a queued coroutine must be restarted from
non-coroutine context (with qemu_co_enter_next).  In this cases,
qemu_co_enter_next also needs to be thread-safe, but it cannot use a
CoMutex and so cannot qemu_co_queue_wait.  This happens in curl (which
right now is rolling its own list of Coroutines) and will happen in
Fam's NVMe driver as well.

This series extracts the idea of a polymorphic lockable object
from my "scoped lock guard" proposal, and applies it to CoQueue.
The implementation of QemuLockable is similar to C11 _Generic, but
redone using the preprocessor and GCC builtins for compatibility.

In general, while a bit on the esoteric side, the functionality used
to emulate _Generic is fairly old in GCC, and the builtins are already
used by include/qemu/atomic.h; the series was tested with Fedora 27 (boot
Damn Small Linux via http) and CentOS 6 (compiled only).

Paolo

v1->v2: fix typos and copyright year

Paolo Bonzini (4):
  lockable: add QemuLockable
  coroutine-lock: convert CoQueue to use QemuLockable
  coroutine-lock: make qemu_co_enter_next thread-safe
  curl: convert to CoQueue

 block/curl.c                | 20 +++---------
 fsdev/qemu-fsdev-throttle.c |  4 +--
 include/qemu/compiler.h     | 40 ++++++++++++++++++++++++
 include/qemu/coroutine.h    | 25 ++++++++++-----
 include/qemu/lockable.h     | 75 +++++++++++++++++++++++++++++++++++++++++++++
 include/qemu/thread.h       |  5 ++-
 include/qemu/typedefs.h     |  4 +++
 util/qemu-coroutine-lock.c  | 22 ++++++++-----
 8 files changed, 159 insertions(+), 36 deletions(-)
 create mode 100644 include/qemu/lockable.h

-- 
2.14.3

[Qemu-devel] [PATCH 1/4] lockable: add QemuLockable

[Paolo Bonzini]

QemuLockable is a polymorphic lock type that takes an object and
knows which function to use for locking and unlocking.  The
implementation could use C11 _Generic, but since the support is
not very widespread I am instead using __builtin_choose_expr and
__builtin_types_compatible_p, which are already used by
include/qemu/atomic.h.

QemuLockable can be used to implement lock guards (in which case the
indirect function calls should be inlined and devirtualized), or to
pass around a lock in such a way that a function can release it and
re-acquire it.  The next patch will do this for CoQueue.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 include/qemu/compiler.h  | 40 ++++++++++++++++++++++++++
 include/qemu/coroutine.h |  4 +--
 include/qemu/lockable.h  | 75 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/qemu/thread.h    |  5 ++--
 include/qemu/typedefs.h  |  4 +++
 5 files changed, 123 insertions(+), 5 deletions(-)
 create mode 100644 include/qemu/lockable.h

diff --git a/include/qemu/compiler.h b/include/qemu/compiler.h
index 340e5fdc09..5179bedb1e 100644
--- a/include/qemu/compiler.h
+++ b/include/qemu/compiler.h
@@ -111,4 +111,44 @@
 #define GCC_FMT_ATTR(n, m)
 #endif
 
+/* Implement C11 _Generic via GCC builtins.  Example:
+ *
+ *    QEMU_GENERIC(x, (float, sinf), (long double, sinl), sin) (x)
+ *
+ * The first argument is the discriminator.  The last is the default value.
+ * The middle ones are tuples in "(type, expansion)" format.
+ */
+
+/* First, find out the number of generic cases.  */
+#define QEMU_GENERIC(x, ...) \
+    QEMU_GENERIC_(typeof(x), __VA_ARGS__, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
+
+/* There will be extra arguments, but they are not used.  */
+#define QEMU_GENERIC_(x, a0, a1, a2, a3, a4, a5, a6, a7, a8, a9, count, ...) \
+    QEMU_GENERIC##count(x, a0, a1, a2, a3, a4, a5, a6, a7, a8, a9)
+
+/* Two more helper macros, this time to extract items from a parenthesized
+ * list.
+ */
+#define QEMU_FIRST_(a, b) a
+#define QEMU_SECOND_(a, b) b
+
+/* ... and a final one for the common part of the "recursion".  */
+#define QEMU_GENERIC_IF(x, type_then, else_)                                   \
+    __builtin_choose_expr(__builtin_types_compatible_p(x,                      \
+                                                       QEMU_FIRST_ type_then), \
+                          QEMU_SECOND_ type_then, else_)
+
+/* CPP poor man's "recursion".  */
+#define QEMU_GENERIC1(x, a0, ...) (a0)
+#define QEMU_GENERIC2(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC1(x, __VA_ARGS__))
+#define QEMU_GENERIC3(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC2(x, __VA_ARGS__))
+#define QEMU_GENERIC4(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC3(x, __VA_ARGS__))
+#define QEMU_GENERIC5(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC4(x, __VA_ARGS__))
+#define QEMU_GENERIC6(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC5(x, __VA_ARGS__))
+#define QEMU_GENERIC7(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC6(x, __VA_ARGS__))
+#define QEMU_GENERIC8(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC7(x, __VA_ARGS__))
+#define QEMU_GENERIC9(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC8(x, __VA_ARGS__))
+#define QEMU_GENERIC10(x, a0, ...) QEMU_GENERIC_IF(x, a0, QEMU_GENERIC9(x, __VA_ARGS__))
+
 #endif /* COMPILER_H */
diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h
index ce2eb73670..8a5129741c 100644
--- a/include/qemu/coroutine.h
+++ b/include/qemu/coroutine.h
@@ -121,7 +121,7 @@ bool qemu_coroutine_entered(Coroutine *co);
  * Provides a mutex that can be used to synchronise coroutines
  */
 struct CoWaitRecord;
-typedef struct CoMutex {
+struct CoMutex {
     /* Count of pending lockers; 0 for a free mutex, 1 for an
      * uncontended mutex.
      */
@@ -142,7 +142,7 @@ typedef struct CoMutex {
     unsigned handoff, sequence;
 
     Coroutine *holder;
-} CoMutex;
+};
 
 /**
  * Initialises a CoMutex. This must be called before any other operation is used
diff --git a/include/qemu/lockable.h b/include/qemu/lockable.h
new file mode 100644
index 0000000000..e280431290
--- /dev/null
+++ b/include/qemu/lockable.h
@@ -0,0 +1,75 @@
+/*
+ * Polymorphic locking functions (aka poor man templates)
+ *
+ * Copyright Red Hat, Inc. 2017-2018
+ *
+ * Author: Paolo Bonzini <pbonzini@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2 or later.
+ * See the COPYING.LIB file in the top-level directory.
+ *
+ */
+
+#ifndef QEMU_LOCKABLE_H
+#define QEMU_LOCKABLE_H
+
+#include "qemu/coroutine.h"
+#include "qemu/thread.h"
+
+typedef void QemuLockUnlockFunc(void *);
+
+struct QemuLockable {
+    void *object;
+    QemuLockUnlockFunc *lock;
+    QemuLockUnlockFunc *unlock;
+};
+
+/* This function gives link-time errors if an invalid, non-NULL
+ * pointer type is passed to QEMU_MAKE_LOCKABLE.
+ */
+void unknown_lock_type(void *);
+
+/* Auxiliary macros to simplify QEMU_MAKE_LOCKABLE.  */
+#define QEMU_LOCK_FUNC(x) ((QemuLockUnlockFunc *)    \
+    QEMU_GENERIC(x,                                  \
+                 (QemuMutex *, qemu_mutex_lock),     \
+                 (CoMutex *, qemu_co_mutex_lock),    \
+                 (QemuSpin *, qemu_spin_lock),       \
+                 ((x) ? unknown_lock_type : NULL)))
+
+#define QEMU_UNLOCK_FUNC(x) ((QemuLockUnlockFunc *)  \
+    QEMU_GENERIC(x,                                  \
+                 (QemuMutex *, qemu_mutex_unlock),   \
+                 (CoMutex *, qemu_co_mutex_unlock),  \
+                 (QemuSpin *, qemu_spin_unlock),     \
+                 ((x) ? unknown_lock_type : NULL)))
+
+#define QEMU_MAKE_LOCKABLE_(x) (&(QemuLockable) {    \
+        .object = (x),                               \
+        .lock = QEMU_LOCK_FUNC(x),                   \
+        .unlock = QEMU_UNLOCK_FUNC(x),               \
+    })
+
+/* QEMU_MAKE_LOCKABLE - Make a polymorphic QemuLockable
+ *
+ * @x: a lock object (currently one of QemuMutex, CoMutex, QemuSpin).
+ *
+ * Returns a QemuLockable object that can be passed around
+ * to a function that can operate with locks of any kind.
+ */
+#define QEMU_MAKE_LOCKABLE(x)                        \
+    QEMU_GENERIC(x,                                  \
+                 (QemuLockable *, (x)),              \
+                 QEMU_MAKE_LOCKABLE_(x))
+
+static inline void qemu_lockable_lock(QemuLockable *x)
+{
+    x->lock(x);
+}
+
+static inline void qemu_lockable_unlock(QemuLockable *x)
+{
+    x->unlock(x);
+}
+
+#endif
diff --git a/include/qemu/thread.h b/include/qemu/thread.h
index 9910f49b3a..a3bc056d89 100644
--- a/include/qemu/thread.h
+++ b/include/qemu/thread.h
@@ -4,7 +4,6 @@
 #include "qemu/processor.h"
 #include "qemu/atomic.h"
 
-typedef struct QemuMutex QemuMutex;
 typedef struct QemuCond QemuCond;
 typedef struct QemuSemaphore QemuSemaphore;
 typedef struct QemuEvent QemuEvent;
@@ -66,9 +65,9 @@ struct Notifier;
 void qemu_thread_atexit_add(struct Notifier *notifier);
 void qemu_thread_atexit_remove(struct Notifier *notifier);
 
-typedef struct QemuSpin {
+struct QemuSpin {
     int value;
-} QemuSpin;
+};
 
 static inline void qemu_spin_init(QemuSpin *spin)
 {
diff --git a/include/qemu/typedefs.h b/include/qemu/typedefs.h
index 9bd7a834ba..5923849cdd 100644
--- a/include/qemu/typedefs.h
+++ b/include/qemu/typedefs.h
@@ -19,6 +19,7 @@ typedef struct BusClass BusClass;
 typedef struct BusState BusState;
 typedef struct Chardev Chardev;
 typedef struct CompatProperty CompatProperty;
+typedef struct CoMutex CoMutex;
 typedef struct CPUAddressSpace CPUAddressSpace;
 typedef struct CPUState CPUState;
 typedef struct DeviceListener DeviceListener;
@@ -86,9 +87,12 @@ typedef struct QEMUBH QEMUBH;
 typedef struct QemuConsole QemuConsole;
 typedef struct QemuDmaBuf QemuDmaBuf;
 typedef struct QEMUFile QEMUFile;
+typedef struct QemuLockable QemuLockable;
+typedef struct QemuMutex QemuMutex;
 typedef struct QemuOpt QemuOpt;
 typedef struct QemuOpts QemuOpts;
 typedef struct QemuOptsList QemuOptsList;
+typedef struct QemuSpin QemuSpin;
 typedef struct QEMUSGList QEMUSGList;
 typedef struct QEMUTimer QEMUTimer;
 typedef struct QEMUTimerListGroup QEMUTimerListGroup;
-- 
2.14.3

[Qemu-devel] [PATCH 2/4] coroutine-lock: convert CoQueue to use QemuLockable

[Paolo Bonzini]

There are cases in which a queued coroutine must be restarted from
non-coroutine context (with qemu_co_enter_next).  In this cases,
qemu_co_enter_next also needs to be thread-safe, but it cannot use
a CoMutex and so cannot qemu_co_queue_wait.  Use QemuLockable so
that the CoQueue can interchangeably use CoMutex or QemuMutex.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 include/qemu/coroutine.h   |  6 +++++-
 util/qemu-coroutine-lock.c | 12 +++++++-----
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h
index 8a5129741c..1e5f0957e6 100644
--- a/include/qemu/coroutine.h
+++ b/include/qemu/coroutine.h
@@ -183,7 +183,9 @@ void qemu_co_queue_init(CoQueue *queue);
  * caller of the coroutine.  The mutex is unlocked during the wait and
  * locked again afterwards.
  */
-void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex);
+#define qemu_co_queue_wait(queue, lock) \
+    qemu_co_queue_wait_impl(queue, QEMU_MAKE_LOCKABLE(lock))
+void coroutine_fn qemu_co_queue_wait_impl(CoQueue *queue, QemuLockable *lock);
 
 /**
  * Restarts the next coroutine in the CoQueue and removes it from the queue.
@@ -271,4 +273,6 @@ void coroutine_fn qemu_co_sleep_ns(QEMUClockType type, int64_t ns);
  */
 void coroutine_fn yield_until_fd_readable(int fd);
 
+#include "qemu/lockable.h"
+
 #endif /* QEMU_COROUTINE_H */
diff --git a/util/qemu-coroutine-lock.c b/util/qemu-coroutine-lock.c
index 846ff9167f..2a66fc1467 100644
--- a/util/qemu-coroutine-lock.c
+++ b/util/qemu-coroutine-lock.c
@@ -40,13 +40,13 @@ void qemu_co_queue_init(CoQueue *queue)
     QSIMPLEQ_INIT(&queue->entries);
 }
 
-void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex)
+void coroutine_fn qemu_co_queue_wait_impl(CoQueue *queue, QemuLockable *lock)
 {
     Coroutine *self = qemu_coroutine_self();
     QSIMPLEQ_INSERT_TAIL(&queue->entries, self, co_queue_next);
 
-    if (mutex) {
-        qemu_co_mutex_unlock(mutex);
+    if (lock) {
+        qemu_lockable_unlock(lock);
     }
 
     /* There is no race condition here.  Other threads will call
@@ -60,9 +60,11 @@ void coroutine_fn qemu_co_queue_wait(CoQueue *queue, CoMutex *mutex)
     /* TODO: OSv implements wait morphing here, where the wakeup
      * primitive automatically places the woken coroutine on the
      * mutex's queue.  This avoids the thundering herd effect.
+     * This could be implemented for CoMutexes, but not really for
+     * other cases of QemuLockable.
      */
-    if (mutex) {
-        qemu_co_mutex_lock(mutex);
+    if (lock) {
+        qemu_lockable_lock(lock);
     }
 }
 
-- 
2.14.3

[Qemu-devel] [PATCH 3/4] coroutine-lock: make qemu_co_enter_next thread-safe

[Paolo Bonzini]

qemu_co_queue_next does not need to release and re-acquire the mutex,
because the queued coroutine does not run immediately.  However, this
does not hold for qemu_co_enter_next.  Now that qemu_co_queue_wait
can synchronize (via QemuLockable) with code that is not running in
coroutine context, it's important that code using qemu_co_enter_next
can easily use a standardized locking idiom.

First of all, qemu_co_enter_next must use aio_co_wake to restart the
coroutine.  Second, the function gains a second argument, a QemuLockable*,
and the comments of qemu_co_queue_next and qemu_co_queue_restart_all
are adjusted to clarify the difference.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 fsdev/qemu-fsdev-throttle.c |  4 ++--
 include/qemu/coroutine.h    | 15 ++++++++++-----
 util/qemu-coroutine-lock.c  | 10 ++++++++--
 3 files changed, 20 insertions(+), 9 deletions(-)

diff --git a/fsdev/qemu-fsdev-throttle.c b/fsdev/qemu-fsdev-throttle.c
index 49eebb5412..1dc07fbc12 100644
--- a/fsdev/qemu-fsdev-throttle.c
+++ b/fsdev/qemu-fsdev-throttle.c
@@ -20,13 +20,13 @@
 static void fsdev_throttle_read_timer_cb(void *opaque)
 {
     FsThrottle *fst = opaque;
-    qemu_co_enter_next(&fst->throttled_reqs[false]);
+    qemu_co_enter_next(&fst->throttled_reqs[false], NULL);
 }
 
 static void fsdev_throttle_write_timer_cb(void *opaque)
 {
     FsThrottle *fst = opaque;
-    qemu_co_enter_next(&fst->throttled_reqs[true]);
+    qemu_co_enter_next(&fst->throttled_reqs[true], NULL);
 }
 
 void fsdev_throttle_parse_opts(QemuOpts *opts, FsThrottle *fst, Error **errp)
diff --git a/include/qemu/coroutine.h b/include/qemu/coroutine.h
index 1e5f0957e6..6fdbc837c0 100644
--- a/include/qemu/coroutine.h
+++ b/include/qemu/coroutine.h
@@ -188,21 +188,26 @@ void qemu_co_queue_init(CoQueue *queue);
 void coroutine_fn qemu_co_queue_wait_impl(CoQueue *queue, QemuLockable *lock);
 
 /**
- * Restarts the next coroutine in the CoQueue and removes it from the queue.
+ * Removes the next coroutine from the CoQueue; it will run as soon as the
+ * current one yields.
  *
- * Returns true if a coroutine was restarted, false if the queue is empty.
+ * Returns true if a coroutine was removed, false if the queue is empty.
  */
 bool coroutine_fn qemu_co_queue_next(CoQueue *queue);
 
 /**
- * Restarts all coroutines in the CoQueue and leaves the queue empty.
+ * Empties the CoQueue; all coroutines in it will run in FIFO orer as soon
+ * as the current one yields.
  */
 void coroutine_fn qemu_co_queue_restart_all(CoQueue *queue);
 
 /**
- * Enter the next coroutine in the queue
+ * Immediately enter the next coroutine in the queue.  Release the mutex
+ * while it runs.
  */
-bool qemu_co_enter_next(CoQueue *queue);
+#define qemu_co_enter_next(queue, lock) \
+    qemu_co_enter_next_impl(queue, QEMU_MAKE_LOCKABLE(lock))
+bool qemu_co_enter_next_impl(CoQueue *queue, QemuLockable *lock);
 
 /**
  * Checks if the CoQueue is empty.
diff --git a/util/qemu-coroutine-lock.c b/util/qemu-coroutine-lock.c
index 2a66fc1467..78fb79acf8 100644
--- a/util/qemu-coroutine-lock.c
+++ b/util/qemu-coroutine-lock.c
@@ -132,7 +132,7 @@ void coroutine_fn qemu_co_queue_restart_all(CoQueue *queue)
     qemu_co_queue_do_restart(queue, false);
 }
 
-bool qemu_co_enter_next(CoQueue *queue)
+bool qemu_co_enter_next_impl(CoQueue *queue, QemuLockable *lock)
 {
     Coroutine *next;
 
@@ -142,7 +142,13 @@ bool qemu_co_enter_next(CoQueue *queue)
     }
 
     QSIMPLEQ_REMOVE_HEAD(&queue->entries, co_queue_next);
-    qemu_coroutine_enter(next);
+    if (lock) {
+        qemu_lockable_unlock(lock);
+    }
+    aio_co_wake(next);
+    if (lock) {
+        qemu_lockable_lock(lock);
+    }
     return true;
 }
 
-- 
2.14.3

[Qemu-devel] [PATCH 4/4] curl: convert to CoQueue

[Paolo Bonzini]

Now that CoQueues can use a QemuMutex for thread-safety, there is no
need for curl to roll its own coroutine queue.  Coroutines can be
placed directly on the queue instead of using a list of CURLAIOCBs.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 block/curl.c | 20 ++++----------------
 1 file changed, 4 insertions(+), 16 deletions(-)

diff --git a/block/curl.c b/block/curl.c
index 35cf417f59..cd578d3d14 100644
--- a/block/curl.c
+++ b/block/curl.c
@@ -101,8 +101,6 @@ typedef struct CURLAIOCB {
 
     size_t start;
     size_t end;
-
-    QSIMPLEQ_ENTRY(CURLAIOCB) next;
 } CURLAIOCB;
 
 typedef struct CURLSocket {
@@ -138,7 +136,7 @@ typedef struct BDRVCURLState {
     bool accept_range;
     AioContext *aio_context;
     QemuMutex mutex;
-    QSIMPLEQ_HEAD(, CURLAIOCB) free_state_waitq;
+    CoQueue free_state_waitq;
     char *username;
     char *password;
     char *proxyusername;
@@ -538,7 +536,6 @@ static int curl_init_state(BDRVCURLState *s, CURLState *state)
 /* Called with s->mutex held.  */
 static void curl_clean_state(CURLState *s)
 {
-    CURLAIOCB *next;
     int j;
     for (j = 0; j < CURL_NUM_ACB; j++) {
         assert(!s->acb[j]);
@@ -556,13 +553,7 @@ static void curl_clean_state(CURLState *s)
 
     s->in_use = 0;
 
-    next = QSIMPLEQ_FIRST(&s->s->free_state_waitq);
-    if (next) {
-        QSIMPLEQ_REMOVE_HEAD(&s->s->free_state_waitq, next);
-        qemu_mutex_unlock(&s->s->mutex);
-        aio_co_wake(next->co);
-        qemu_mutex_lock(&s->s->mutex);
-    }
+    qemu_co_enter_next(&s->s->free_state_waitq, &s->s->mutex);
 }
 
 static void curl_parse_filename(const char *filename, QDict *options,
@@ -784,7 +775,7 @@ static int curl_open(BlockDriverState *bs, QDict *options, int flags,
     }
 
     DPRINTF("CURL: Opening %s\n", file);
-    QSIMPLEQ_INIT(&s->free_state_waitq);
+    qemu_co_queue_init(&s->free_state_waitq);
     s->aio_context = bdrv_get_aio_context(bs);
     s->url = g_strdup(file);
     qemu_mutex_lock(&s->mutex);
@@ -888,10 +879,7 @@ static void curl_setup_preadv(BlockDriverState *bs, CURLAIOCB *acb)
         if (state) {
             break;
         }
-        QSIMPLEQ_INSERT_TAIL(&s->free_state_waitq, acb, next);
-        qemu_mutex_unlock(&s->mutex);
-        qemu_coroutine_yield();
-        qemu_mutex_lock(&s->mutex);
+        qemu_co_queue_wait(&s->free_state_waitq, &s->mutex);
     }
 
     if (curl_init_state(s, state) < 0) {
-- 
2.14.3

Re: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

[no-reply]

Hi,

This series failed docker-build@min-glib build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 20180116142316.30486-1-pbonzini@redhat.com
Subject: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-build@min-glib
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
ceb8579b71 curl: convert to CoQueue
deb8f34b74 coroutine-lock: make qemu_co_enter_next thread-safe
5a30f36fba coroutine-lock: convert CoQueue to use QemuLockable
215b4ef03d lockable: add QemuLockable

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-m03t9cw_/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   min-glib
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 407, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 404, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 287, in run
    dkr = Docker()
  File "./tests/docker/docker.py", line 133, in __init__
    self._command = _guess_docker_command()
  File "./tests/docker/docker.py", line 58, in _guess_docker_command
    commands_txt)
Exception: Cannot find working docker command. Tried:
  docker
  sudo -n docker
make: *** [tests/docker/Makefile.include:37: docker-image-min-glib] Error 1

real	0m0.649s
user	0m0.049s
sys	0m0.029s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

[Fam Zheng]

On 01/17/2018 02:04 PM, no-reply@patchew.org wrote:
>    BUILD   min-glib
> Traceback (most recent call last):
>    File "./tests/docker/docker.py", line 407, in <module>
>      sys.exit(main())
>    File "./tests/docker/docker.py", line 404, in main
>      return args.cmdobj.run(args, argv)
>    File "./tests/docker/docker.py", line 287, in run
>      dkr = Docker()
>    File "./tests/docker/docker.py", line 133, in __init__
>      self._command = _guess_docker_command()
>    File "./tests/docker/docker.py", line 58, in _guess_docker_command
>      commands_txt)
> Exception: Cannot find working docker command. Tried:
>    docker
>    sudo -n docker

Patchew.org was misconfigured to run this docker tests on a host that 
has no docker. Fixed now. Sorry for the noise.

Fam

Re: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

[Fam Zheng]

On Tue, Jan 16, 2018 at 10:23 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> There are cases in which a queued coroutine must be restarted from
> non-coroutine context (with qemu_co_enter_next).  In this cases,
> qemu_co_enter_next also needs to be thread-safe, but it cannot use a
> CoMutex and so cannot qemu_co_queue_wait.  This happens in curl (which
> right now is rolling its own list of Coroutines) and will happen in
> Fam's NVMe driver as well.
>
> This series extracts the idea of a polymorphic lockable object
> from my "scoped lock guard" proposal, and applies it to CoQueue.
> The implementation of QemuLockable is similar to C11 _Generic, but
> redone using the preprocessor and GCC builtins for compatibility.
>
> In general, while a bit on the esoteric side, the functionality used
> to emulate _Generic is fairly old in GCC, and the builtins are already
> used by include/qemu/atomic.h; the series was tested with Fedora 27 (boot
> Damn Small Linux via http) and CentOS 6 (compiled only).
>
> Paolo
>
> v1->v2: fix typos and copyright year

Reviewed-by: Fam Zheng <famz@redhat.com>

Should I include this series in my pull request for the NVMe driver?

Re: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

[Paolo Bonzini]

On 24/01/2018 04:58, Fam Zheng wrote:
> On Tue, Jan 16, 2018 at 10:23 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
>> There are cases in which a queued coroutine must be restarted from
>> non-coroutine context (with qemu_co_enter_next).  In this cases,
>> qemu_co_enter_next also needs to be thread-safe, but it cannot use a
>> CoMutex and so cannot qemu_co_queue_wait.  This happens in curl (which
>> right now is rolling its own list of Coroutines) and will happen in
>> Fam's NVMe driver as well.
>>
>> This series extracts the idea of a polymorphic lockable object
>> from my "scoped lock guard" proposal, and applies it to CoQueue.
>> The implementation of QemuLockable is similar to C11 _Generic, but
>> redone using the preprocessor and GCC builtins for compatibility.
>>
>> In general, while a bit on the esoteric side, the functionality used
>> to emulate _Generic is fairly old in GCC, and the builtins are already
>> used by include/qemu/atomic.h; the series was tested with Fedora 27 (boot
>> Damn Small Linux via http) and CentOS 6 (compiled only).
>>
>> Paolo
>>
>> v1->v2: fix typos and copyright year
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> 
> Should I include this series in my pull request for the NVMe driver?

Yes, please.  This is within block layer area, since CoQueue and
block/curl.c are still the only users.

Paolo

Re: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

[Fam Zheng]

On Tue, Jan 16, 2018 at 10:23 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> There are cases in which a queued coroutine must be restarted from
> non-coroutine context (with qemu_co_enter_next).  In this cases,
> qemu_co_enter_next also needs to be thread-safe, but it cannot use a
> CoMutex and so cannot qemu_co_queue_wait.  This happens in curl (which
> right now is rolling its own list of Coroutines) and will happen in
> Fam's NVMe driver as well.
>
> This series extracts the idea of a polymorphic lockable object
> from my "scoped lock guard" proposal, and applies it to CoQueue.
> The implementation of QemuLockable is similar to C11 _Generic, but
> redone using the preprocessor and GCC builtins for compatibility.
>
> In general, while a bit on the esoteric side, the functionality used
> to emulate _Generic is fairly old in GCC, and the builtins are already
> used by include/qemu/atomic.h; the series was tested with Fedora 27 (boot
> Damn Small Linux via http) and CentOS 6 (compiled only).

I'm seeing this crash with the series:

(gdb) bt
#0  0x00007ff76204d66b in raise () at /lib64/libc.so.6
#1  0x00007ff76204f381 in abort () at /lib64/libc.so.6
#2  0x00007ff7620458fa in __assert_fail_base () at /lib64/libc.so.6
#3  0x00007ff762045972 in  () at /lib64/libc.so.6
#4  0x000055eaab249c68 in qemu_co_mutex_unlock (mutex=0x7ff750bf7b40)
at /stor/work/qemu/util/qemu-coroutine-lock.c:320
#5  0x000055eaab249da3 in qemu_lockable_unlock (x=0x7ff750bf7b40) at
/stor/work/qemu/include/qemu/lockable.h:72
#6  0x000055eaab249da3 in qemu_co_queue_wait_impl
(queue=0x55eaaef41a08, lock=lock@entry=0x7ff750bf7b40) at
/stor/work/qemu/util/qemu-coroutine-lock.c:49
#7  0x000055eaab19f2b9 in handle_dependencies
(bs=bs@entry=0x55eaad9c6620, guest_offset=guest_offset@entry=1597440,
cur_bytes=cur_bytes@entry=0x7ff750bf7ba0, m=m@entry=0x7ff7
50bf7c58) at /stor/work/qemu/block/qcow2-cluster.c:1067
#8  0x000055eaab1a1b85 in qcow2_alloc_cluster_offset
(bs=bs@entry=0x55eaad9c6620, offset=offset@entry=1597440,
bytes=bytes@entry=0x7ff750bf7c4c, host_offset=host_offset@entry
=0x7ff750bf7c50, m=m@entry=0x7ff750bf7c58) at
/stor/work/qemu/block/qcow2-cluster.c:1497
#9  0x000055eaab19411e in qcow2_co_pwritev (bs=0x55eaad9c6620,
offset=1597440, bytes=8192, qiov=0x55eaaedb4880, flags=<optimized
out>) at /stor/work/qemu/block/qcow2.c:1896
#10 0x000055eaab1c2962 in bdrv_driver_pwritev
(bs=bs@entry=0x55eaad9c6620, offset=offset@entry=1597440,
bytes=bytes@entry=8192, qiov=qiov@entry=0x55eaaedb4880, flags=flags@en
try=0) at /stor/work/qemu/block/io.c:976
#11 0x000055eaab1c3985 in bdrv_aligned_pwritev
(child=child@entry=0x55eaad92bd00, req=req@entry=0x7ff750bf7e70,
offset=offset@entry=1597440, bytes=bytes@entry=8192, align=ali
gn@entry=1, qiov=qiov@entry=0x55eaaedb4880, flags=0) at
/stor/work/qemu/block/io.c:1534
#12 0x000055eaab1c4ca5 in bdrv_co_pwritev (child=0x55eaad92bd00,
offset=offset@entry=1597440, bytes=bytes@entry=8192,
qiov=qiov@entry=0x55eaaedb4880, flags=flags@entry=0)
   at /stor/work/qemu/block/io.c:1785
#13 0x000055eaab1b4f06 in blk_co_pwritev (blk=0x55eaad9c63c0,
offset=1597440, bytes=8192, qiov=0x55eaaedb4880, flags=0) at
/stor/work/qemu/block/block-backend.c:1135
#14 0x000055eaab1b4fff in blk_aio_write_entry (opaque=0x55eaaefc5eb0)
at /stor/work/qemu/block/block-backend.c:1326
#15 0x000055eaab24a77a in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>) at /stor/work/qemu/util/coroutine-ucontext.c:79
#16 0x00007ff762066bc0 in __start_context () at /lib64/libc.so.6
#17 0x00007ffdf69102d0 in  ()
#18 0x0000000000000000 in  ()


It's late today so I'll take a closer look tomorrow.

Fam

Re: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue

[Paolo Bonzini]

----- Original Message -----
> From: "Fam Zheng" <famz@redhat.com>
> To: "Paolo Bonzini" <pbonzini@redhat.com>
> Cc: "QEMU Developers" <qemu-devel@nongnu.org>
> Sent: Thursday, January 25, 2018 4:05:27 PM
> Subject: Re: [Qemu-devel] [PATCH v2 0/4] coroutine-lock: polymorphic CoQueue
> 
> On Tue, Jan 16, 2018 at 10:23 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> > There are cases in which a queued coroutine must be restarted from
> > non-coroutine context (with qemu_co_enter_next).  In this cases,
> > qemu_co_enter_next also needs to be thread-safe, but it cannot use a
> > CoMutex and so cannot qemu_co_queue_wait.  This happens in curl (which
> > right now is rolling its own list of Coroutines) and will happen in
> > Fam's NVMe driver as well.
> >
> > This series extracts the idea of a polymorphic lockable object
> > from my "scoped lock guard" proposal, and applies it to CoQueue.
> > The implementation of QemuLockable is similar to C11 _Generic, but
> > redone using the preprocessor and GCC builtins for compatibility.
> >
> > In general, while a bit on the esoteric side, the functionality used
> > to emulate _Generic is fairly old in GCC, and the builtins are already
> > used by include/qemu/atomic.h; the series was tested with Fedora 27 (boot
> > Damn Small Linux via http) and CentOS 6 (compiled only).
> 
> I'm seeing this crash with the series:
> 
> (gdb) bt
> #0  0x00007ff76204d66b in raise () at /lib64/libc.so.6
> #1  0x00007ff76204f381 in abort () at /lib64/libc.so.6
> #2  0x00007ff7620458fa in __assert_fail_base () at /lib64/libc.so.6
> #3  0x00007ff762045972 in  () at /lib64/libc.so.6
> #4  0x000055eaab249c68 in qemu_co_mutex_unlock (mutex=0x7ff750bf7b40)
> at /stor/work/qemu/util/qemu-coroutine-lock.c:320
> #5  0x000055eaab249da3 in qemu_lockable_unlock (x=0x7ff750bf7b40) at
> /stor/work/qemu/include/qemu/lockable.h:72
> #6  0x000055eaab249da3 in qemu_co_queue_wait_impl
> (queue=0x55eaaef41a08, lock=lock@entry=0x7ff750bf7b40) at
> /stor/work/qemu/util/qemu-coroutine-lock.c:49
> #7  0x000055eaab19f2b9 in handle_dependencies
> (bs=bs@entry=0x55eaad9c6620, guest_offset=guest_offset@entry=1597440,
> cur_bytes=cur_bytes@entry=0x7ff750bf7ba0, m=m@entry=0x7ff7
> 50bf7c58) at /stor/work/qemu/block/qcow2-cluster.c:1067
> #8  0x000055eaab1a1b85 in qcow2_alloc_cluster_offset
> (bs=bs@entry=0x55eaad9c6620, offset=offset@entry=1597440,
> bytes=bytes@entry=0x7ff750bf7c4c, host_offset=host_offset@entry
> =0x7ff750bf7c50, m=m@entry=0x7ff750bf7c58) at
> /stor/work/qemu/block/qcow2-cluster.c:1497
> #9  0x000055eaab19411e in qcow2_co_pwritev (bs=0x55eaad9c6620,
> offset=1597440, bytes=8192, qiov=0x55eaaedb4880, flags=<optimized
> out>) at /stor/work/qemu/block/qcow2.c:1896
> #10 0x000055eaab1c2962 in bdrv_driver_pwritev
> (bs=bs@entry=0x55eaad9c6620, offset=offset@entry=1597440,
> bytes=bytes@entry=8192, qiov=qiov@entry=0x55eaaedb4880, flags=flags@en
> try=0) at /stor/work/qemu/block/io.c:976
> #11 0x000055eaab1c3985 in bdrv_aligned_pwritev
> (child=child@entry=0x55eaad92bd00, req=req@entry=0x7ff750bf7e70,
> offset=offset@entry=1597440, bytes=bytes@entry=8192, align=ali
> gn@entry=1, qiov=qiov@entry=0x55eaaedb4880, flags=0) at
> /stor/work/qemu/block/io.c:1534
> #12 0x000055eaab1c4ca5 in bdrv_co_pwritev (child=0x55eaad92bd00,
> offset=offset@entry=1597440, bytes=bytes@entry=8192,
> qiov=qiov@entry=0x55eaaedb4880, flags=flags@entry=0)
>    at /stor/work/qemu/block/io.c:1785
> #13 0x000055eaab1b4f06 in blk_co_pwritev (blk=0x55eaad9c63c0,
> offset=1597440, bytes=8192, qiov=0x55eaaedb4880, flags=0) at
> /stor/work/qemu/block/block-backend.c:1135
> #14 0x000055eaab1b4fff in blk_aio_write_entry (opaque=0x55eaaefc5eb0)
> at /stor/work/qemu/block/block-backend.c:1326
> #15 0x000055eaab24a77a in coroutine_trampoline (i0=<optimized out>,
> i1=<optimized out>) at /stor/work/qemu/util/coroutine-ucontext.c:79
> #16 0x00007ff762066bc0 in __start_context () at /lib64/libc.so.6
> #17 0x00007ffdf69102d0 in  ()
> #18 0x0000000000000000 in  ()
> 
> It's late today so I'll take a closer look tomorrow.

Ouch.  /me bangs the head against the wall and runs writing testcases.
Sorry.

Paolo