[PATCH 00/20] qemu-kvm: Cleanup and switch to upstream - The Season Final

[PATCH 00/20] qemu-kvm: Cleanup and switch to upstream - The Season Final

[Jan Kiszka]

With this series applied, we are finally at a level of almost zero
redundancy between QEMU upstream and the qemu-kvm tree. The last major
duplication to be removed is the original io-thread implementation and
everything related to it:
 - locking
 - vcpu wakeup/kicking as well as suspend/resume
 - on-vcpu execution
 - io-thread loop
 - vcpu thread loop

The approach taken here is similar to what was done to morph KVM core
functions into upstream code: First all to be converted code is moved
over into cpus.c, then qemu-kvm's functions are gradually changed and
replaced with the upstream versions.

This means that we have to enable CONFIG_IOTHREAD which is so far
incompatible with qemu-kvm. This incompatibility causes a temporary
breakage of the TCG mode early in the series, but it is healed again
with the last patch applied.

To make it clear: Even with all this applied, there is still a lot to
do to turn upstream QEMU into the primary KVM platform. We need to
 - rework in-kernel IOAPIC/PIC/APIC/PIT support, namely
    - proper qdev modeling
    - new MSI hooking architecture (half-done)
    - VAPIC support (haven't looked at details yet, maybe just cleanups)
 - prepare device assignment for upstream (VFIO and/or KVM-based)
 - clean up remaining small deltas (including posix-aio-compat...)
 - get rid of legacy command line interfaces (e.g. via deprecation
   warning in release X and removal in X+n, n >= 1)

But then we are really done and can all retire. ;)

In the meantime, please review/merge these bits.

Jan Kiszka (20):
  qemu-kvm: Move thread-related code to cpus.c
  qemu-kvm: Enable CONFIG_IOTHREAD
  qemu-kvm: Switch to iothread version of qemu_notify_event
  qemu-kvm: Use upstream mutex and conds
  qemu-kvm: Restrict validity of cpu_single_env
  qemu-kvm: Use upstream qemu_mutex_lock/unlock_iothread
  qemu-kvm: Clean up kvm_update_interrupt_request
  qemu-kvm: Replace kvm_update_interrupt_request with qemu_cpu_kick
  qemu-kvm: Use upstream run_on_cpu and flush_queued_work
  qemu-kvm: Drop kvm_cond_wait
  qemu-kvm: Replace kvm_cpu_is_stopped with cpu_is_stopped
  qemu-kvm: Remove obsolete current_env
  qemu-kvm: Use upstream vcpu pause/resume
  qemu-kvm: Move main loop setup code
  qemu-kvm: Use upstream's way of signaling debug stops
  qemu-kvm: Use upstream main loop
  qemu-kvm: Use upstream main loop initialization
  qemu-kvm: Replace kvm_get_thread_id with qemu_get_thread_id
  qemu-kvm: Use upstream kvm vcpu initialization
  qemu-kvm: Use upstream vcpu loop

 configure            |    1 +
 cpu-defs.h           |   11 +-
 cpus.c               |   17 +--
 cpus.h               |    1 +
 kvm-all.c            |    9 +-
 qemu-kvm.c           |  498 +-------------------------------------------------
 qemu-kvm.h           |    7 -
 target-i386/helper.c |    4 -
 vl.c                 |   17 --
 9 files changed, 13 insertions(+), 552 deletions(-)

[PATCH 01/20] qemu-kvm: Move thread-related code to cpus.c

[Jan Kiszka]

As a first step towards unified threading support, move all the related
qemu-kvm-specific bits from qemu-kvm.c to cpus.c. This already allows to
drop 3 identical functions (sigbus_reraise, sigbus_handler,
sigfd_handler) and should provide the environment for consolidating the
rest.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c     |  425 +++++++++++++++++++++++++++++++++++++++++++++++++++
 cpus.h     |    1 +
 kvm-all.c  |    2 +-
 qemu-kvm.c |  494 +-----------------------------------------------------------
 qemu-kvm.h |    1 +
 5 files changed, 430 insertions(+), 493 deletions(-)

diff --git a/cpus.c b/cpus.c
index b64c29f..d209535 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1179,3 +1179,428 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
     cpu_list(f, cpu_fprintf); /* deprecated */
 #endif
 }
+
+
+
+/*
+ * qemu-kvm threading code, integration zone
+ */
+#ifdef CONFIG_KVM
+
+#include <sys/syscall.h>
+
+#include "qemu-thread.h"
+
+pthread_mutex_t qemu_mutex = PTHREAD_MUTEX_INITIALIZER;
+pthread_cond_t qemu_vcpu_cond = PTHREAD_COND_INITIALIZER;
+pthread_cond_t qemu_system_cond = PTHREAD_COND_INITIALIZER;
+pthread_cond_t qemu_pause_cond = PTHREAD_COND_INITIALIZER;
+pthread_cond_t qemu_work_cond = PTHREAD_COND_INITIALIZER;
+__thread CPUState *current_env;
+
+static int qemu_system_ready;
+
+static CPUState *kvm_debug_cpu_requested;
+
+unsigned long kvm_get_thread_id(void)
+{
+    return syscall(SYS_gettid);
+}
+
+static void kvm_cond_wait(pthread_cond_t *cond)
+{
+    CPUState *env = cpu_single_env;
+
+    pthread_cond_wait(cond, &qemu_mutex);
+    cpu_single_env = env;
+}
+
+static void sig_ipi_handler(int n)
+{
+}
+
+void on_vcpu(CPUState *env, void (*func)(void *data), void *data)
+{
+    struct qemu_work_item wi;
+
+    if (env == current_env) {
+        func(data);
+        return;
+    }
+
+    wi.func = func;
+    wi.data = data;
+    if (!env->kvm_cpu_state.queued_work_first) {
+        env->kvm_cpu_state.queued_work_first = &wi;
+    } else {
+        env->kvm_cpu_state.queued_work_last->next = &wi;
+    }
+    env->kvm_cpu_state.queued_work_last = &wi;
+    wi.next = NULL;
+    wi.done = false;
+
+    pthread_kill(env->thread->thread, SIG_IPI);
+    while (!wi.done) {
+        kvm_cond_wait(&qemu_work_cond);
+    }
+}
+
+void kvm_update_interrupt_request(CPUState *env)
+{
+    int signal = 0;
+
+    if (env) {
+        if (!current_env || !current_env->created) {
+            signal = 1;
+        }
+        /*
+         * Testing for created here is really redundant
+         */
+        if (current_env && current_env->created &&
+            env != current_env && !env->thread_kicked) {
+            signal = 1;
+        }
+
+        if (signal) {
+            env->thread_kicked = true;
+            if (env->thread) {
+                pthread_kill(env->thread->thread, SIG_IPI);
+            }
+        }
+    }
+}
+
+static int kvm_cpu_is_stopped(CPUState *env)
+{
+    return !vm_running || env->stopped;
+}
+
+static void flush_queued_work(CPUState *env)
+{
+    struct qemu_work_item *wi;
+
+    if (!env->kvm_cpu_state.queued_work_first) {
+        return;
+    }
+
+    while ((wi = env->kvm_cpu_state.queued_work_first)) {
+        env->kvm_cpu_state.queued_work_first = wi->next;
+        wi->func(wi->data);
+        wi->done = true;
+    }
+    env->kvm_cpu_state.queued_work_last = NULL;
+    pthread_cond_broadcast(&qemu_work_cond);
+}
+
+static void kvm_main_loop_wait(CPUState *env, int timeout)
+{
+    struct timespec ts;
+    int r, e;
+    siginfo_t siginfo;
+    sigset_t waitset;
+    sigset_t chkset;
+
+    ts.tv_sec = timeout / 1000;
+    ts.tv_nsec = (timeout % 1000) * 1000000;
+    sigemptyset(&waitset);
+    sigaddset(&waitset, SIG_IPI);
+    sigaddset(&waitset, SIGBUS);
+
+    do {
+        pthread_mutex_unlock(&qemu_mutex);
+
+        r = sigtimedwait(&waitset, &siginfo, &ts);
+        e = errno;
+
+        pthread_mutex_lock(&qemu_mutex);
+
+        if (r == -1 && !(e == EAGAIN || e == EINTR)) {
+            printf("sigtimedwait: %s\n", strerror(e));
+            exit(1);
+        }
+
+        switch (r) {
+        case SIGBUS:
+            if (kvm_on_sigbus_vcpu(env, siginfo.si_code, siginfo.si_addr))
+                sigbus_reraise();
+            break;
+        default:
+            break;
+        }
+
+        r = sigpending(&chkset);
+        if (r == -1) {
+            printf("sigpending: %s\n", strerror(e));
+            exit(1);
+        }
+    } while (sigismember(&chkset, SIG_IPI) || sigismember(&chkset, SIGBUS));
+
+    cpu_single_env = env;
+    flush_queued_work(env);
+
+    if (env->stop) {
+        env->stop = 0;
+        env->stopped = 1;
+        pthread_cond_signal(&qemu_pause_cond);
+    }
+
+    env->thread_kicked = false;
+}
+
+static int all_threads_paused(void)
+{
+    CPUState *penv = first_cpu;
+
+    while (penv) {
+        if (penv->stop) {
+            return 0;
+        }
+        penv = (CPUState *) penv->next_cpu;
+    }
+
+    return 1;
+}
+
+static void pause_all_threads(void)
+{
+    CPUState *penv = first_cpu;
+
+    while (penv) {
+        if (penv != cpu_single_env) {
+            penv->stop = 1;
+            pthread_kill(penv->thread->thread, SIG_IPI);
+        } else {
+            penv->stop = 0;
+            penv->stopped = 1;
+            cpu_exit(penv);
+        }
+        penv = (CPUState *) penv->next_cpu;
+    }
+
+    while (!all_threads_paused()) {
+        kvm_cond_wait(&qemu_pause_cond);
+    }
+}
+
+static void resume_all_threads(void)
+{
+    CPUState *penv = first_cpu;
+
+    assert(!cpu_single_env);
+
+    while (penv) {
+        penv->stop = 0;
+        penv->stopped = 0;
+        pthread_kill(penv->thread->thread, SIG_IPI);
+        penv = (CPUState *) penv->next_cpu;
+    }
+}
+
+static void kvm_vm_state_change_handler(void *context, int running, int reason)
+{
+    if (running) {
+        resume_all_threads();
+    } else {
+        pause_all_threads();
+    }
+}
+
+static void setup_kernel_sigmask(CPUState *env)
+{
+    sigset_t set;
+
+    sigemptyset(&set);
+    sigaddset(&set, SIGUSR2);
+    sigaddset(&set, SIGIO);
+    sigaddset(&set, SIGALRM);
+    sigprocmask(SIG_BLOCK, &set, NULL);
+
+    sigprocmask(SIG_BLOCK, NULL, &set);
+    sigdelset(&set, SIG_IPI);
+    sigdelset(&set, SIGBUS);
+
+    kvm_set_signal_mask(env, &set);
+}
+
+static void qemu_kvm_system_reset(void)
+{
+    pause_all_threads();
+
+    cpu_synchronize_all_states();
+    qemu_system_reset();
+
+    resume_all_threads();
+}
+
+static int kvm_main_loop_cpu(CPUState *env)
+{
+    while (1) {
+        int timeout = 1000;
+        if (!kvm_cpu_is_stopped(env)) {
+            switch (kvm_cpu_exec(env)) {
+            case EXCP_HLT:
+                break;
+            case EXCP_DEBUG:
+                kvm_debug_cpu_requested = env;
+                env->stopped = 1;
+                break;
+            default:
+                timeout = 0;
+                break;
+            }
+        }
+        kvm_main_loop_wait(env, timeout);
+    }
+    pthread_mutex_unlock(&qemu_mutex);
+    return 0;
+}
+
+static void *ap_main_loop(void *_env)
+{
+    CPUState *env = _env;
+
+    current_env = env;
+    env->thread_id = kvm_get_thread_id();
+
+    pthread_mutex_lock(&qemu_mutex);
+    cpu_single_env = env;
+
+    if (kvm_create_vcpu(env) < 0) {
+        abort();
+    }
+    setup_kernel_sigmask(env);
+
+    /* signal VCPU creation */
+    current_env->created = 1;
+    pthread_cond_signal(&qemu_vcpu_cond);
+
+    /* and wait for machine initialization */
+    while (!qemu_system_ready) {
+        kvm_cond_wait(&qemu_system_cond);
+    }
+
+    /* re-initialize cpu_single_env after re-acquiring qemu_mutex */
+    cpu_single_env = env;
+
+    kvm_main_loop_cpu(env);
+    return NULL;
+}
+
+int kvm_init_vcpu(CPUState *env)
+{
+    env->thread = qemu_mallocz(sizeof(QemuThread));
+    qemu_thread_create(env->thread, ap_main_loop, env);
+
+    while (env->created == 0) {
+        kvm_cond_wait(&qemu_vcpu_cond);
+    }
+
+    return 0;
+}
+
+int kvm_init_ap(void)
+{
+    struct sigaction action;
+
+    pthread_mutex_lock(&qemu_mutex);
+
+    qemu_add_vm_change_state_handler(kvm_vm_state_change_handler, NULL);
+
+    signal(SIG_IPI, sig_ipi_handler);
+
+    memset(&action, 0, sizeof(action));
+    action.sa_flags = SA_SIGINFO;
+    action.sa_sigaction = (void (*)(int, siginfo_t*, void*))sigbus_handler;
+    sigaction(SIGBUS, &action, NULL);
+    prctl(PR_MCE_KILL, 1, 1, 0, 0);
+    return 0;
+}
+
+bool qemu_system_is_ready(void)
+{
+    return qemu_system_ready;
+}
+
+int kvm_main_loop(void)
+{
+    sigset_t mask;
+    int sigfd;
+
+    qemu_system_ready = 1;
+
+    sigemptyset(&mask);
+    sigaddset(&mask, SIGIO);
+    sigaddset(&mask, SIGALRM);
+    sigaddset(&mask, SIGBUS);
+    sigprocmask(SIG_BLOCK, &mask, NULL);
+
+    sigfd = qemu_signalfd(&mask);
+    if (sigfd == -1) {
+        fprintf(stderr, "failed to create signalfd\n");
+        return -errno;
+    }
+
+    fcntl(sigfd, F_SETFL, O_NONBLOCK);
+
+    qemu_set_fd_handler2(sigfd, NULL, sigfd_handler, NULL,
+                         (void *)(unsigned long) sigfd);
+
+    pthread_cond_broadcast(&qemu_system_cond);
+
+    cpu_single_env = NULL;
+
+    while (1) {
+        main_loop_wait(0);
+        if (qemu_shutdown_requested()) {
+            monitor_protocol_event(QEVENT_SHUTDOWN, NULL);
+            if (qemu_no_shutdown()) {
+                vm_stop(VMSTOP_SHUTDOWN);
+            } else {
+                break;
+            }
+        } else if (qemu_powerdown_requested()) {
+            monitor_protocol_event(QEVENT_POWERDOWN, NULL);
+            qemu_irq_raise(qemu_system_powerdown);
+        } else if (qemu_reset_requested()) {
+            qemu_kvm_system_reset();
+        } else if (kvm_debug_cpu_requested) {
+            gdb_set_stop_cpu(kvm_debug_cpu_requested);
+            vm_stop(VMSTOP_DEBUG);
+            kvm_debug_cpu_requested = NULL;
+        }
+    }
+
+    bdrv_close_all();
+    pause_all_threads();
+    pthread_mutex_unlock(&qemu_mutex);
+
+    return 0;
+}
+
+static void kvm_mutex_unlock(void)
+{
+    assert(!cpu_single_env);
+    pthread_mutex_unlock(&qemu_mutex);
+}
+
+static void kvm_mutex_lock(void)
+{
+    pthread_mutex_lock(&qemu_mutex);
+    cpu_single_env = NULL;
+}
+
+void qemu_mutex_unlock_iothread(void)
+{
+    if (kvm_enabled()) {
+        kvm_mutex_unlock();
+    }
+}
+
+void qemu_mutex_lock_iothread(void)
+{
+    if (kvm_enabled()) {
+        kvm_mutex_lock();
+    }
+}
+
+#endif /* CONFIG_KVM */
diff --git a/cpus.h b/cpus.h
index 6fdeb0d..4838b16 100644
--- a/cpus.h
+++ b/cpus.h
@@ -7,6 +7,7 @@ void qemu_main_loop_start(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool qemu_system_is_ready(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
diff --git a/kvm-all.c b/kvm-all.c
index 88d0785..bd53422 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -208,7 +208,7 @@ int kvm_pit_in_kernel(void)
     return kvm_state->pit_in_kernel;
 }
 
-static int kvm_create_vcpu(CPUState *env)
+int kvm_create_vcpu(CPUState *env)
 {
     KVMState *s = kvm_state;
     long mmap_size;
diff --git a/qemu-kvm.c b/qemu-kvm.c
index 94e12f3..dfbf045 100644
--- a/qemu-kvm.c
+++ b/qemu-kvm.c
@@ -19,29 +19,9 @@
 #include "gdbstub.h"
 #include "monitor.h"
 #include "cpus.h"
-#include "qemu-thread.h"
 
 #include "qemu-kvm.h"
 
-#include <pthread.h>
-#include <sys/utsname.h>
-#include <sys/syscall.h>
-#include <sys/mman.h>
-#include <sys/ioctl.h>
-#include "compatfd.h"
-#include <sys/prctl.h>
-
-#ifndef PR_MCE_KILL
-#define PR_MCE_KILL 33
-#endif
-
-#ifndef BUS_MCEERR_AR
-#define BUS_MCEERR_AR 4
-#endif
-#ifndef BUS_MCEERR_AO
-#define BUS_MCEERR_AO 5
-#endif
-
 #define EXPECTED_KVM_API_VERSION 12
 
 #if EXPECTED_KVM_API_VERSION != KVM_API_VERSION
@@ -53,19 +33,6 @@ int kvm_pit = 1;
 int kvm_pit_reinject = 1;
 int kvm_nested = 0;
 
-pthread_mutex_t qemu_mutex = PTHREAD_MUTEX_INITIALIZER;
-pthread_cond_t qemu_vcpu_cond = PTHREAD_COND_INITIALIZER;
-pthread_cond_t qemu_system_cond = PTHREAD_COND_INITIALIZER;
-pthread_cond_t qemu_pause_cond = PTHREAD_COND_INITIALIZER;
-pthread_cond_t qemu_work_cond = PTHREAD_COND_INITIALIZER;
-__thread CPUState *current_env;
-
-static int qemu_system_ready;
-
-#define SIG_IPI (SIGRTMIN+4)
-
-CPUState *kvm_debug_cpu_requested;
-
 #define ALIGN(x, y) (((x)+(y)-1) & ~((y)-1))
 
 static inline void set_gsi(KVMState *s, unsigned int gsi)
@@ -571,326 +538,6 @@ int kvm_assign_set_msix_entry(KVMState *s,
 }
 #endif
 
-unsigned long kvm_get_thread_id(void)
-{
-    return syscall(SYS_gettid);
-}
-
-static void kvm_cond_wait(pthread_cond_t *cond)
-{
-    CPUState *env = cpu_single_env;
-
-    pthread_cond_wait(cond, &qemu_mutex);
-    cpu_single_env = env;
-}
-
-static void sig_ipi_handler(int n)
-{
-}
-
-static void sigbus_reraise(void)
-{
-    sigset_t set;
-    struct sigaction action;
-
-    memset(&action, 0, sizeof(action));
-    action.sa_handler = SIG_DFL;
-    if (!sigaction(SIGBUS, &action, NULL)) {
-        raise(SIGBUS);
-        sigemptyset(&set);
-        sigaddset(&set, SIGBUS);
-        sigprocmask(SIG_UNBLOCK, &set, NULL);
-    }
-    perror("Failed to re-raise SIGBUS!\n");
-    abort();
-}
-
-static void sigbus_handler(int n, struct qemu_signalfd_siginfo *siginfo,
-                           void *ctx)
-{
-    if (kvm_on_sigbus(siginfo->ssi_code, (void *)(intptr_t)siginfo->ssi_addr))
-        sigbus_reraise();
-}
-
-void on_vcpu(CPUState *env, void (*func)(void *data), void *data)
-{
-    struct qemu_work_item wi;
-
-    if (env == current_env) {
-        func(data);
-        return;
-    }
-
-    wi.func = func;
-    wi.data = data;
-    if (!env->kvm_cpu_state.queued_work_first) {
-        env->kvm_cpu_state.queued_work_first = &wi;
-    } else {
-        env->kvm_cpu_state.queued_work_last->next = &wi;
-    }
-    env->kvm_cpu_state.queued_work_last = &wi;
-    wi.next = NULL;
-    wi.done = false;
-
-    pthread_kill(env->thread->thread, SIG_IPI);
-    while (!wi.done) {
-        kvm_cond_wait(&qemu_work_cond);
-    }
-}
-
-void kvm_update_interrupt_request(CPUState *env)
-{
-    int signal = 0;
-
-    if (env) {
-        if (!current_env || !current_env->created) {
-            signal = 1;
-        }
-        /*
-         * Testing for created here is really redundant
-         */
-        if (current_env && current_env->created &&
-            env != current_env && !env->thread_kicked) {
-            signal = 1;
-        }
-
-        if (signal) {
-            env->thread_kicked = true;
-            if (env->thread) {
-                pthread_kill(env->thread->thread, SIG_IPI);
-            }
-        }
-    }
-}
-
-static int kvm_cpu_is_stopped(CPUState *env)
-{
-    return !vm_running || env->stopped;
-}
-
-static void flush_queued_work(CPUState *env)
-{
-    struct qemu_work_item *wi;
-
-    if (!env->kvm_cpu_state.queued_work_first) {
-        return;
-    }
-
-    while ((wi = env->kvm_cpu_state.queued_work_first)) {
-        env->kvm_cpu_state.queued_work_first = wi->next;
-        wi->func(wi->data);
-        wi->done = true;
-    }
-    env->kvm_cpu_state.queued_work_last = NULL;
-    pthread_cond_broadcast(&qemu_work_cond);
-}
-
-static void kvm_main_loop_wait(CPUState *env, int timeout)
-{
-    struct timespec ts;
-    int r, e;
-    siginfo_t siginfo;
-    sigset_t waitset;
-    sigset_t chkset;
-
-    ts.tv_sec = timeout / 1000;
-    ts.tv_nsec = (timeout % 1000) * 1000000;
-    sigemptyset(&waitset);
-    sigaddset(&waitset, SIG_IPI);
-    sigaddset(&waitset, SIGBUS);
-
-    do {
-        pthread_mutex_unlock(&qemu_mutex);
-
-        r = sigtimedwait(&waitset, &siginfo, &ts);
-        e = errno;
-
-        pthread_mutex_lock(&qemu_mutex);
-
-        if (r == -1 && !(e == EAGAIN || e == EINTR)) {
-            printf("sigtimedwait: %s\n", strerror(e));
-            exit(1);
-        }
-
-        switch (r) {
-        case SIGBUS:
-            if (kvm_on_sigbus_vcpu(env, siginfo.si_code, siginfo.si_addr))
-                sigbus_reraise();
-            break;
-        default:
-            break;
-        }
-
-        r = sigpending(&chkset);
-        if (r == -1) {
-            printf("sigpending: %s\n", strerror(e));
-            exit(1);
-        }
-    } while (sigismember(&chkset, SIG_IPI) || sigismember(&chkset, SIGBUS));
-
-    cpu_single_env = env;
-    flush_queued_work(env);
-
-    if (env->stop) {
-        env->stop = 0;
-        env->stopped = 1;
-        pthread_cond_signal(&qemu_pause_cond);
-    }
-
-    env->thread_kicked = false;
-}
-
-static int all_threads_paused(void)
-{
-    CPUState *penv = first_cpu;
-
-    while (penv) {
-        if (penv->stop) {
-            return 0;
-        }
-        penv = (CPUState *) penv->next_cpu;
-    }
-
-    return 1;
-}
-
-static void pause_all_threads(void)
-{
-    CPUState *penv = first_cpu;
-
-    while (penv) {
-        if (penv != cpu_single_env) {
-            penv->stop = 1;
-            pthread_kill(penv->thread->thread, SIG_IPI);
-        } else {
-            penv->stop = 0;
-            penv->stopped = 1;
-            cpu_exit(penv);
-        }
-        penv = (CPUState *) penv->next_cpu;
-    }
-
-    while (!all_threads_paused()) {
-        kvm_cond_wait(&qemu_pause_cond);
-    }
-}
-
-static void resume_all_threads(void)
-{
-    CPUState *penv = first_cpu;
-
-    assert(!cpu_single_env);
-
-    while (penv) {
-        penv->stop = 0;
-        penv->stopped = 0;
-        pthread_kill(penv->thread->thread, SIG_IPI);
-        penv = (CPUState *) penv->next_cpu;
-    }
-}
-
-static void kvm_vm_state_change_handler(void *context, int running, int reason)
-{
-    if (running) {
-        resume_all_threads();
-    } else {
-        pause_all_threads();
-    }
-}
-
-static void setup_kernel_sigmask(CPUState *env)
-{
-    sigset_t set;
-
-    sigemptyset(&set);
-    sigaddset(&set, SIGUSR2);
-    sigaddset(&set, SIGIO);
-    sigaddset(&set, SIGALRM);
-    sigprocmask(SIG_BLOCK, &set, NULL);
-
-    sigprocmask(SIG_BLOCK, NULL, &set);
-    sigdelset(&set, SIG_IPI);
-    sigdelset(&set, SIGBUS);
-
-    kvm_set_signal_mask(env, &set);
-}
-
-static void qemu_kvm_system_reset(void)
-{
-    pause_all_threads();
-
-    cpu_synchronize_all_states();
-    qemu_system_reset();
-
-    resume_all_threads();
-}
-
-static int kvm_main_loop_cpu(CPUState *env)
-{
-    while (1) {
-        int timeout = 1000;
-        if (!kvm_cpu_is_stopped(env)) {
-            switch (kvm_cpu_exec(env)) {
-            case EXCP_HLT:
-                break;
-            case EXCP_DEBUG:
-                kvm_debug_cpu_requested = env;
-                env->stopped = 1;
-                break;
-            default:
-                timeout = 0;
-                break;
-            }
-        }
-        kvm_main_loop_wait(env, timeout);
-    }
-    pthread_mutex_unlock(&qemu_mutex);
-    return 0;
-}
-
-static void *ap_main_loop(void *_env)
-{
-    CPUState *env = _env;
-
-    current_env = env;
-    env->thread_id = kvm_get_thread_id();
-
-    pthread_mutex_lock(&qemu_mutex);
-    cpu_single_env = env;
-
-    if (kvm_create_vcpu(env) < 0) {
-        abort();
-    }
-    setup_kernel_sigmask(env);
-
-    /* signal VCPU creation */
-    current_env->created = 1;
-    pthread_cond_signal(&qemu_vcpu_cond);
-
-    /* and wait for machine initialization */
-    while (!qemu_system_ready) {
-        kvm_cond_wait(&qemu_system_cond);
-    }
-
-    /* re-initialize cpu_single_env after re-acquiring qemu_mutex */
-    cpu_single_env = env;
-
-    kvm_main_loop_cpu(env);
-    return NULL;
-}
-
-int kvm_init_vcpu(CPUState *env)
-{
-    env->thread = qemu_mallocz(sizeof(QemuThread));
-    qemu_thread_create(env->thread, ap_main_loop, env);
-
-    while (env->created == 0) {
-        kvm_cond_wait(&qemu_vcpu_cond);
-    }
-
-    return 0;
-}
-
 #ifdef TARGET_I386
 void kvm_hpet_disable_kpit(void)
 {
@@ -911,116 +558,6 @@ void kvm_hpet_enable_kpit(void)
 }
 #endif
 
-int kvm_init_ap(void)
-{
-    struct sigaction action;
-
-    pthread_mutex_lock(&qemu_mutex);
-
-    qemu_add_vm_change_state_handler(kvm_vm_state_change_handler, NULL);
-
-    signal(SIG_IPI, sig_ipi_handler);
-
-    memset(&action, 0, sizeof(action));
-    action.sa_flags = SA_SIGINFO;
-    action.sa_sigaction = (void (*)(int, siginfo_t*, void*))sigbus_handler;
-    sigaction(SIGBUS, &action, NULL);
-    prctl(PR_MCE_KILL, 1, 1, 0, 0);
-    return 0;
-}
-
-/* If we have signalfd, we mask out the signals we want to handle and then
- * use signalfd to listen for them.  We rely on whatever the current signal
- * handler is to dispatch the signals when we receive them.
- */
-
-static void sigfd_handler(void *opaque)
-{
-    int fd = (unsigned long) opaque;
-    struct qemu_signalfd_siginfo info;
-    struct sigaction action;
-    ssize_t len;
-
-    while (1) {
-        do {
-            len = read(fd, &info, sizeof(info));
-        } while (len == -1 && errno == EINTR);
-
-        if (len == -1 && errno == EAGAIN) {
-            break;
-        }
-
-        if (len != sizeof(info)) {
-            printf("read from sigfd returned %zd: %m\n", len);
-            return;
-        }
-
-        sigaction(info.ssi_signo, NULL, &action);
-        if ((action.sa_flags & SA_SIGINFO) && action.sa_sigaction) {
-            action.sa_sigaction(info.ssi_signo,
-                                (siginfo_t *)&info, NULL);
-        } else if (action.sa_handler) {
-            action.sa_handler(info.ssi_signo);
-        }
-    }
-}
-
-int kvm_main_loop(void)
-{
-    sigset_t mask;
-    int sigfd;
-
-    qemu_system_ready = 1;
-
-    sigemptyset(&mask);
-    sigaddset(&mask, SIGIO);
-    sigaddset(&mask, SIGALRM);
-    sigaddset(&mask, SIGBUS);
-    sigprocmask(SIG_BLOCK, &mask, NULL);
-
-    sigfd = qemu_signalfd(&mask);
-    if (sigfd == -1) {
-        fprintf(stderr, "failed to create signalfd\n");
-        return -errno;
-    }
-
-    fcntl(sigfd, F_SETFL, O_NONBLOCK);
-
-    qemu_set_fd_handler2(sigfd, NULL, sigfd_handler, NULL,
-                         (void *)(unsigned long) sigfd);
-
-    pthread_cond_broadcast(&qemu_system_cond);
-
-    cpu_single_env = NULL;
-
-    while (1) {
-        main_loop_wait(0);
-        if (qemu_shutdown_requested()) {
-            monitor_protocol_event(QEVENT_SHUTDOWN, NULL);
-            if (qemu_no_shutdown()) {
-                vm_stop(VMSTOP_SHUTDOWN);
-            } else {
-                break;
-            }
-        } else if (qemu_powerdown_requested()) {
-            monitor_protocol_event(QEVENT_POWERDOWN, NULL);
-            qemu_irq_raise(qemu_system_powerdown);
-        } else if (qemu_reset_requested()) {
-            qemu_kvm_system_reset();
-        } else if (kvm_debug_cpu_requested) {
-            gdb_set_stop_cpu(kvm_debug_cpu_requested);
-            vm_stop(VMSTOP_DEBUG);
-            kvm_debug_cpu_requested = NULL;
-        }
-    }
-
-    bdrv_close_all();
-    pause_all_threads();
-    pthread_mutex_unlock(&qemu_mutex);
-
-    return 0;
-}
-
 #if !defined(TARGET_I386)
 int kvm_arch_init_irq_routing(void)
 {
@@ -1028,32 +565,6 @@ int kvm_arch_init_irq_routing(void)
 }
 #endif
 
-static void kvm_mutex_unlock(void)
-{
-    assert(!cpu_single_env);
-    pthread_mutex_unlock(&qemu_mutex);
-}
-
-static void kvm_mutex_lock(void)
-{
-    pthread_mutex_lock(&qemu_mutex);
-    cpu_single_env = NULL;
-}
-
-void qemu_mutex_unlock_iothread(void)
-{
-    if (kvm_enabled()) {
-        kvm_mutex_unlock();
-    }
-}
-
-void qemu_mutex_lock_iothread(void)
-{
-    if (kvm_enabled()) {
-        kvm_mutex_lock();
-    }
-}
-
 #ifdef CONFIG_KVM_DEVICE_ASSIGNMENT
 typedef struct KVMIOPortRegion {
     unsigned long start;
@@ -1089,7 +600,7 @@ int kvm_add_ioport_region(unsigned long start, unsigned long size)
     region->status = 1;
     QLIST_INSERT_HEAD(&ioport_regions, region, entry);
 
-    if (qemu_system_ready) {
+    if (qemu_system_is_ready()) {
         for (env = first_cpu; env != NULL; env = env->next_cpu) {
             on_vcpu(env, do_set_ioport_access, region);
             if (region->status < 0) {
@@ -1112,7 +623,7 @@ int kvm_remove_ioport_region(unsigned long start, unsigned long size)
         if (region->start == start && region->size == size) {
             region->status = 0;
         }
-        if (qemu_system_ready) {
+        if (qemu_system_is_ready()) {
             for (env = first_cpu; env != NULL; env = env->next_cpu) {
                 on_vcpu(env, do_set_ioport_access, region);
             }
@@ -1155,4 +666,3 @@ int kvm_set_boot_cpu_id(KVMState *s, uint32_t id)
 #endif
     return -ENOSYS;
 }
-
diff --git a/qemu-kvm.h b/qemu-kvm.h
index 094aef2..0ebd3eb 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -265,6 +265,7 @@ struct kvm_pit_state {
 
 #endif                          /* !CONFIG_KVM */
 
+int kvm_create_vcpu(CPUState *env);
 int kvm_main_loop(void);
 int kvm_init_ap(void);
 void kvm_save_lapic(CPUState *env);
-- 
1.7.1

[PATCH 02/20] qemu-kvm: Enable CONFIG_IOTHREAD

[Jan Kiszka]

So far, qemu-kvm's build was incompatible with --enable-io-thread. But
to consolidate both iothread versions, we need to start enabling
upstream code under this config option.

This patch force-enables CONFIG_IOTHREAD but still picks the !IOTHREAD
variant of those functions that are used by the qemu-kvm code paths.
This temporary hack breaks TCG until we can safely enable all upstream
iothread functions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch_init.c |    3 ++-
 configure   |    1 +
 cpus.c      |   37 +++++++++++++++++++++++--------------
 vl.c        |    5 ++++-
 4 files changed, 30 insertions(+), 16 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index 484b39d..ac67c3d 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -711,7 +711,8 @@ int audio_available(void)
 
 int tcg_available(void)
 {
-    return 1;
+    /* temporarily broken */
+    return 0;
 }
 
 int kvm_available(void)
diff --git a/configure b/configure
index b3a8c68..4c738b1 100755
--- a/configure
+++ b/configure
@@ -1866,6 +1866,7 @@ EOF
   kvm_cflags="$kvm_cflags -idirafter $source_path/compat"
   if compile_prog "$kvm_cflags" "" ; then
     kvm=yes
+    io_thread=yes
     cat > $TMPC <<EOF
 #include <linux/kvm_para.h>
 int main(void) { return 0; }
diff --git a/cpus.c b/cpus.c
index d209535..48bac70 100644
--- a/cpus.c
+++ b/cpus.c
@@ -529,7 +529,7 @@ static void qemu_tcg_init_cpu_signals(void)
 }
 #endif /* _WIN32 */
 
-#ifndef CONFIG_IOTHREAD
+/*#ifndef CONFIG_IOTHREAD*/
 int qemu_init_main_loop(void)
 {
     int ret;
@@ -567,6 +567,7 @@ void qemu_init_vcpu(void *_env)
         qemu_tcg_init_cpu_signals();
     }
 }
+#ifndef CONFIG_IOTHREAD
 
 int qemu_cpu_is_self(void *env)
 {
@@ -578,6 +579,7 @@ void run_on_cpu(CPUState *env, void (*func)(void *data), void *data)
     func(data);
 }
 
+#endif /* !IOTHREAD */
 void resume_all_vcpus(void)
 {
 }
@@ -589,8 +591,8 @@ void pause_all_vcpus(void)
 void qemu_cpu_kick(void *env)
 {
 }
+#ifndef CONFIG_IOTHREAD
 
-#ifdef UNUSED_IMPL
 void qemu_cpu_kick_self(void)
 {
 #ifndef _WIN32
@@ -601,8 +603,8 @@ void qemu_cpu_kick_self(void)
     abort();
 #endif
 }
-#endif
 
+#endif /* !IOTHREAD */
 void qemu_notify_event(void)
 {
     CPUState *env = cpu_single_env;
@@ -616,12 +618,12 @@ void qemu_notify_event(void)
     }
     exit_request = 1;
 }
+#ifndef CONFIG_IOTHREAD
 
-#if defined(OBSOLETE_KVM_IMPL) || !defined(CONFIG_KVM)
 void qemu_mutex_lock_iothread(void) {}
 void qemu_mutex_unlock_iothread(void) {}
-#endif
 
+#endif /* !IOTHREAD */
 void cpu_stop_current(void)
 {
 }
@@ -630,25 +632,31 @@ void vm_stop(int reason)
 {
     do_vm_stop(reason);
 }
+#ifndef CONFIG_IOTHREAD
 
 #else /* CONFIG_IOTHREAD */
 
 QemuMutex qemu_global_mutex;
+#ifdef UNUSED_IOTHREAD_IMPL
 static QemuMutex qemu_fair_mutex;
 
 static QemuThread io_thread;
 
 static QemuThread *tcg_cpu_thread;
 static QemuCond *tcg_halt_cond;
+#endif
 
 static int qemu_system_ready;
 /* cpu creation */
+#ifdef UNUSED_IOTHREAD_IMPL
 static QemuCond qemu_cpu_cond;
 /* system init */
 static QemuCond qemu_system_cond;
 static QemuCond qemu_pause_cond;
+#endif
 static QemuCond qemu_work_cond;
 
+#ifdef UNUSED_IOTHREAD_IMPL
 int qemu_init_main_loop(void)
 {
     int ret;
@@ -684,6 +692,7 @@ void qemu_main_loop_start(void)
     qemu_system_ready = 1;
     qemu_cond_broadcast(&qemu_system_cond);
 }
+#endif /* UNUSED_IOTHREAD_IMPL */
 
 void run_on_cpu(CPUState *env, void (*func)(void *data), void *data)
 {
@@ -714,6 +723,7 @@ void run_on_cpu(CPUState *env, void (*func)(void *data), void *data)
     }
 }
 
+#ifdef UNUSED_IOTHREAD_IMPL
 static void flush_queued_work(CPUState *env)
 {
     struct qemu_work_item *wi;
@@ -849,8 +859,8 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
 
     return NULL;
 }
+#endif /* UNUSED_IOTHREAD_IMPL */
 
-#endif
 static void qemu_cpu_kick_thread(CPUState *env)
 {
 #ifndef _WIN32
@@ -869,8 +879,8 @@ static void qemu_cpu_kick_thread(CPUState *env)
     }
 #endif
 }
-#ifdef CONFIG_IOTHREAD
 
+#ifdef UNUSED_IOTHREAD_IMPL
 void qemu_cpu_kick(void *_env)
 {
     CPUState *env = _env;
@@ -881,8 +891,8 @@ void qemu_cpu_kick(void *_env)
         env->thread_kicked = true;
     }
 }
+#endif /* UNUSED_IOTHREAD_IMPL */
 
-#endif
 void qemu_cpu_kick_self(void)
 {
 #ifndef _WIN32
@@ -896,7 +906,6 @@ void qemu_cpu_kick_self(void)
     abort();
 #endif
 }
-#ifdef CONFIG_IOTHREAD
 
 int qemu_cpu_is_self(void *_env)
 {
@@ -905,6 +914,7 @@ int qemu_cpu_is_self(void *_env)
     return qemu_thread_is_self(env->thread);
 }
 
+#ifdef UNUSED_IOTHREAD_IMPL
 void qemu_mutex_lock_iothread(void)
 {
     if (kvm_enabled()) {
@@ -1043,6 +1053,7 @@ void vm_stop(int reason)
     }
     do_vm_stop(reason);
 }
+#endif /* UNUSED_IOTHREAD_IMPL */
 
 #endif
 
@@ -1195,11 +1206,9 @@ pthread_mutex_t qemu_mutex = PTHREAD_MUTEX_INITIALIZER;
 pthread_cond_t qemu_vcpu_cond = PTHREAD_COND_INITIALIZER;
 pthread_cond_t qemu_system_cond = PTHREAD_COND_INITIALIZER;
 pthread_cond_t qemu_pause_cond = PTHREAD_COND_INITIALIZER;
-pthread_cond_t qemu_work_cond = PTHREAD_COND_INITIALIZER;
+pthread_cond_t qemu_kvm_work_cond = PTHREAD_COND_INITIALIZER;
 __thread CPUState *current_env;
 
-static int qemu_system_ready;
-
 static CPUState *kvm_debug_cpu_requested;
 
 unsigned long kvm_get_thread_id(void)
@@ -1241,7 +1250,7 @@ void on_vcpu(CPUState *env, void (*func)(void *data), void *data)
 
     pthread_kill(env->thread->thread, SIG_IPI);
     while (!wi.done) {
-        kvm_cond_wait(&qemu_work_cond);
+        kvm_cond_wait(&qemu_kvm_work_cond);
     }
 }
 
@@ -1289,7 +1298,7 @@ static void flush_queued_work(CPUState *env)
         wi->done = true;
     }
     env->kvm_cpu_state.queued_work_last = NULL;
-    pthread_cond_broadcast(&qemu_work_cond);
+    pthread_cond_broadcast(&qemu_kvm_work_cond);
 }
 
 static void kvm_main_loop_wait(CPUState *env, int timeout)
diff --git a/vl.c b/vl.c
index 73e147f..2587aae 100644
--- a/vl.c
+++ b/vl.c
@@ -1910,7 +1910,8 @@ static int debugcon_parse(const char *devname)
 
 static int tcg_init(void)
 {
-    return 0;
+    fprintf(stderr, "Emulation temporarily broken\n");
+    return -1;
 }
 
 static struct {
@@ -2699,6 +2700,8 @@ int main(int argc, char **argv, char **envp)
                 }
                 break;
 	    case QEMU_OPTION_no_kvm:
+                fprintf(stderr, "Emulation temporarily broken\n");
+                exit(1);
                 olist = qemu_find_opts("machine");
                 qemu_opts_reset(olist);
                 qemu_opts_parse(olist, "accel=tcg", 0);
-- 
1.7.1

[PATCH 03/20] qemu-kvm: Switch to iothread version of qemu_notify_event

[Jan Kiszka]

The differences do not matter for qemu-kvm's iothread code.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cpus.c b/cpus.c
index 48bac70..8b9b1f6 100644
--- a/cpus.c
+++ b/cpus.c
@@ -604,7 +604,6 @@ void qemu_cpu_kick_self(void)
 #endif
 }
 
-#endif /* !IOTHREAD */
 void qemu_notify_event(void)
 {
     CPUState *env = cpu_single_env;
@@ -618,7 +617,6 @@ void qemu_notify_event(void)
     }
     exit_request = 1;
 }
-#ifndef CONFIG_IOTHREAD
 
 void qemu_mutex_lock_iothread(void) {}
 void qemu_mutex_unlock_iothread(void) {}
@@ -1024,12 +1022,14 @@ void qemu_init_vcpu(void *_env)
         qemu_tcg_init_vcpu(env);
     }
 }
+#endif /* UNUSED_IOTHREAD_IMPL */
 
 void qemu_notify_event(void)
 {
     qemu_event_increment();
 }
 
+#ifdef UNUSED_IOTHREAD_IMPL
 void cpu_stop_current(void)
 {
     if (cpu_single_env) {
-- 
1.7.1

[PATCH 04/20] qemu-kvm: Use upstream mutex and conds

[Jan Kiszka]

This temporarily requires our own initialization service as we are still
using the !IOTHREAD version of qemu_init_main_loop.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |   57 +++++++++++++++++++++++++++++++--------------------------
 1 files changed, 31 insertions(+), 26 deletions(-)

diff --git a/cpus.c b/cpus.c
index 8b9b1f6..bdffd2b 100644
--- a/cpus.c
+++ b/cpus.c
@@ -530,6 +530,8 @@ static void qemu_tcg_init_cpu_signals(void)
 #endif /* _WIN32 */
 
 /*#ifndef CONFIG_IOTHREAD*/
+static void qemu_kvm_init_main_loop(void);
+
 int qemu_init_main_loop(void)
 {
     int ret;
@@ -538,6 +540,7 @@ int qemu_init_main_loop(void)
     if (ret) {
         return ret;
     }
+    qemu_kvm_init_main_loop();
 
     qemu_init_sigbus();
 
@@ -635,9 +638,9 @@ void vm_stop(int reason)
 #else /* CONFIG_IOTHREAD */
 
 QemuMutex qemu_global_mutex;
-#ifdef UNUSED_IOTHREAD_IMPL
 static QemuMutex qemu_fair_mutex;
 
+#ifdef UNUSED_IOTHREAD_IMPL
 static QemuThread io_thread;
 
 static QemuThread *tcg_cpu_thread;
@@ -646,12 +649,10 @@ static QemuCond *tcg_halt_cond;
 
 static int qemu_system_ready;
 /* cpu creation */
-#ifdef UNUSED_IOTHREAD_IMPL
 static QemuCond qemu_cpu_cond;
 /* system init */
 static QemuCond qemu_system_cond;
 static QemuCond qemu_pause_cond;
-#endif
 static QemuCond qemu_work_cond;
 
 #ifdef UNUSED_IOTHREAD_IMPL
@@ -1200,13 +1201,6 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
 
 #include <sys/syscall.h>
 
-#include "qemu-thread.h"
-
-pthread_mutex_t qemu_mutex = PTHREAD_MUTEX_INITIALIZER;
-pthread_cond_t qemu_vcpu_cond = PTHREAD_COND_INITIALIZER;
-pthread_cond_t qemu_system_cond = PTHREAD_COND_INITIALIZER;
-pthread_cond_t qemu_pause_cond = PTHREAD_COND_INITIALIZER;
-pthread_cond_t qemu_kvm_work_cond = PTHREAD_COND_INITIALIZER;
 __thread CPUState *current_env;
 
 static CPUState *kvm_debug_cpu_requested;
@@ -1216,11 +1210,11 @@ unsigned long kvm_get_thread_id(void)
     return syscall(SYS_gettid);
 }
 
-static void kvm_cond_wait(pthread_cond_t *cond)
+static void kvm_cond_wait(QemuCond *cond)
 {
     CPUState *env = cpu_single_env;
 
-    pthread_cond_wait(cond, &qemu_mutex);
+    qemu_cond_wait(cond, &qemu_global_mutex);
     cpu_single_env = env;
 }
 
@@ -1250,7 +1244,7 @@ void on_vcpu(CPUState *env, void (*func)(void *data), void *data)
 
     pthread_kill(env->thread->thread, SIG_IPI);
     while (!wi.done) {
-        kvm_cond_wait(&qemu_kvm_work_cond);
+        kvm_cond_wait(&qemu_work_cond);
     }
 }
 
@@ -1298,7 +1292,7 @@ static void flush_queued_work(CPUState *env)
         wi->done = true;
     }
     env->kvm_cpu_state.queued_work_last = NULL;
-    pthread_cond_broadcast(&qemu_kvm_work_cond);
+    qemu_cond_broadcast(&qemu_work_cond);
 }
 
 static void kvm_main_loop_wait(CPUState *env, int timeout)
@@ -1316,12 +1310,12 @@ static void kvm_main_loop_wait(CPUState *env, int timeout)
     sigaddset(&waitset, SIGBUS);
 
     do {
-        pthread_mutex_unlock(&qemu_mutex);
+        qemu_mutex_unlock(&qemu_global_mutex);
 
         r = sigtimedwait(&waitset, &siginfo, &ts);
         e = errno;
 
-        pthread_mutex_lock(&qemu_mutex);
+        qemu_mutex_lock(&qemu_global_mutex);
 
         if (r == -1 && !(e == EAGAIN || e == EINTR)) {
             printf("sigtimedwait: %s\n", strerror(e));
@@ -1350,7 +1344,7 @@ static void kvm_main_loop_wait(CPUState *env, int timeout)
     if (env->stop) {
         env->stop = 0;
         env->stopped = 1;
-        pthread_cond_signal(&qemu_pause_cond);
+        qemu_cond_signal(&qemu_pause_cond);
     }
 
     env->thread_kicked = false;
@@ -1460,7 +1454,7 @@ static int kvm_main_loop_cpu(CPUState *env)
         }
         kvm_main_loop_wait(env, timeout);
     }
-    pthread_mutex_unlock(&qemu_mutex);
+    qemu_mutex_unlock(&qemu_global_mutex);
     return 0;
 }
 
@@ -1471,7 +1465,7 @@ static void *ap_main_loop(void *_env)
     current_env = env;
     env->thread_id = kvm_get_thread_id();
 
-    pthread_mutex_lock(&qemu_mutex);
+    qemu_mutex_lock(&qemu_global_mutex);
     cpu_single_env = env;
 
     if (kvm_create_vcpu(env) < 0) {
@@ -1481,7 +1475,7 @@ static void *ap_main_loop(void *_env)
 
     /* signal VCPU creation */
     current_env->created = 1;
-    pthread_cond_signal(&qemu_vcpu_cond);
+    qemu_cond_signal(&qemu_cpu_cond);
 
     /* and wait for machine initialization */
     while (!qemu_system_ready) {
@@ -1501,7 +1495,7 @@ int kvm_init_vcpu(CPUState *env)
     qemu_thread_create(env->thread, ap_main_loop, env);
 
     while (env->created == 0) {
-        kvm_cond_wait(&qemu_vcpu_cond);
+        kvm_cond_wait(&qemu_cpu_cond);
     }
 
     return 0;
@@ -1511,7 +1505,7 @@ int kvm_init_ap(void)
 {
     struct sigaction action;
 
-    pthread_mutex_lock(&qemu_mutex);
+    qemu_mutex_lock(&qemu_global_mutex);
 
     qemu_add_vm_change_state_handler(kvm_vm_state_change_handler, NULL);
 
@@ -1554,7 +1548,7 @@ int kvm_main_loop(void)
     qemu_set_fd_handler2(sigfd, NULL, sigfd_handler, NULL,
                          (void *)(unsigned long) sigfd);
 
-    pthread_cond_broadcast(&qemu_system_cond);
+    qemu_cond_broadcast(&qemu_system_cond);
 
     cpu_single_env = NULL;
 
@@ -1581,7 +1575,7 @@ int kvm_main_loop(void)
 
     bdrv_close_all();
     pause_all_threads();
-    pthread_mutex_unlock(&qemu_mutex);
+    qemu_mutex_unlock(&qemu_global_mutex);
 
     return 0;
 }
@@ -1589,12 +1583,12 @@ int kvm_main_loop(void)
 static void kvm_mutex_unlock(void)
 {
     assert(!cpu_single_env);
-    pthread_mutex_unlock(&qemu_mutex);
+    qemu_mutex_unlock(&qemu_global_mutex);
 }
 
 static void kvm_mutex_lock(void)
 {
-    pthread_mutex_lock(&qemu_mutex);
+    qemu_mutex_lock(&qemu_global_mutex);
     cpu_single_env = NULL;
 }
 
@@ -1612,4 +1606,15 @@ void qemu_mutex_lock_iothread(void)
     }
 }
 
+static void qemu_kvm_init_main_loop(void)
+{
+    qemu_cond_init(&qemu_cpu_cond);
+    qemu_cond_init(&qemu_system_cond);
+    qemu_cond_init(&qemu_pause_cond);
+    qemu_cond_init(&qemu_work_cond);
+    qemu_mutex_init(&qemu_fair_mutex);
+    qemu_mutex_init(&qemu_global_mutex);
+    qemu_mutex_lock(&qemu_global_mutex);
+}
+
 #endif /* CONFIG_KVM */
-- 
1.7.1

[PATCH 05/20] qemu-kvm: Restrict validity of cpu_single_env

[Jan Kiszka]

Just like in upstream, assume that cpu_single_env is only non-NULL while
in kvm_cpu_exec. Use qemu_cpu_is_self in pause_all_threads instead of
cpu_single_env and additionally avoid duplicate kicks. Then drop all
related cpu_single_env initializations and assertions.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |   18 +++++-------------
 1 files changed, 5 insertions(+), 13 deletions(-)

diff --git a/cpus.c b/cpus.c
index bdffd2b..383d359 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1338,7 +1338,6 @@ static void kvm_main_loop_wait(CPUState *env, int timeout)
         }
     } while (sigismember(&chkset, SIG_IPI) || sigismember(&chkset, SIGBUS));
 
-    cpu_single_env = env;
     flush_queued_work(env);
 
     if (env->stop) {
@@ -1369,9 +1368,12 @@ static void pause_all_threads(void)
     CPUState *penv = first_cpu;
 
     while (penv) {
-        if (penv != cpu_single_env) {
+        if (!qemu_cpu_is_self(penv)) {
             penv->stop = 1;
-            pthread_kill(penv->thread->thread, SIG_IPI);
+            if (!penv->thread_kicked) {
+                pthread_kill(penv->thread->thread, SIG_IPI);
+                penv->thread_kicked = true;
+            }
         } else {
             penv->stop = 0;
             penv->stopped = 1;
@@ -1389,8 +1391,6 @@ static void resume_all_threads(void)
 {
     CPUState *penv = first_cpu;
 
-    assert(!cpu_single_env);
-
     while (penv) {
         penv->stop = 0;
         penv->stopped = 0;
@@ -1466,7 +1466,6 @@ static void *ap_main_loop(void *_env)
     env->thread_id = kvm_get_thread_id();
 
     qemu_mutex_lock(&qemu_global_mutex);
-    cpu_single_env = env;
 
     if (kvm_create_vcpu(env) < 0) {
         abort();
@@ -1482,9 +1481,6 @@ static void *ap_main_loop(void *_env)
         kvm_cond_wait(&qemu_system_cond);
     }
 
-    /* re-initialize cpu_single_env after re-acquiring qemu_mutex */
-    cpu_single_env = env;
-
     kvm_main_loop_cpu(env);
     return NULL;
 }
@@ -1550,8 +1546,6 @@ int kvm_main_loop(void)
 
     qemu_cond_broadcast(&qemu_system_cond);
 
-    cpu_single_env = NULL;
-
     while (1) {
         main_loop_wait(0);
         if (qemu_shutdown_requested()) {
@@ -1582,14 +1576,12 @@ int kvm_main_loop(void)
 
 static void kvm_mutex_unlock(void)
 {
-    assert(!cpu_single_env);
     qemu_mutex_unlock(&qemu_global_mutex);
 }
 
 static void kvm_mutex_lock(void)
 {
     qemu_mutex_lock(&qemu_global_mutex);
-    cpu_single_env = NULL;
 }
 
 void qemu_mutex_unlock_iothread(void)
-- 
1.7.1

[PATCH 06/20] qemu-kvm: Use upstream qemu_mutex_lock/unlock_iothread

[Jan Kiszka]

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |   26 +-------------------------
 1 files changed, 1 insertions(+), 25 deletions(-)

diff --git a/cpus.c b/cpus.c
index 383d359..bf666b0 100644
--- a/cpus.c
+++ b/cpus.c
@@ -913,7 +913,6 @@ int qemu_cpu_is_self(void *_env)
     return qemu_thread_is_self(env->thread);
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 void qemu_mutex_lock_iothread(void)
 {
     if (kvm_enabled()) {
@@ -933,6 +932,7 @@ void qemu_mutex_unlock_iothread(void)
     qemu_mutex_unlock(&qemu_global_mutex);
 }
 
+#ifdef UNUSED_IOTHREAD_IMPL
 static int all_vcpus_paused(void)
 {
     CPUState *penv = first_cpu;
@@ -1574,30 +1574,6 @@ int kvm_main_loop(void)
     return 0;
 }
 
-static void kvm_mutex_unlock(void)
-{
-    qemu_mutex_unlock(&qemu_global_mutex);
-}
-
-static void kvm_mutex_lock(void)
-{
-    qemu_mutex_lock(&qemu_global_mutex);
-}
-
-void qemu_mutex_unlock_iothread(void)
-{
-    if (kvm_enabled()) {
-        kvm_mutex_unlock();
-    }
-}
-
-void qemu_mutex_lock_iothread(void)
-{
-    if (kvm_enabled()) {
-        kvm_mutex_lock();
-    }
-}
-
 static void qemu_kvm_init_main_loop(void)
 {
     qemu_cond_init(&qemu_cpu_cond);
-- 
1.7.1

[PATCH 07/20] qemu-kvm: Clean up kvm_update_interrupt_request

[Jan Kiszka]

Most tests in kvm_update_interrupt_request are unneeded today:
 - env argument is always non-NULL (caller references it as well)
 - current_env must have been created when we get here
 - env->thread can't be zero (initialized early during cpu creation)

So simply avoid self signaling and multiple kicks, drop the rest.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |   23 +++--------------------
 1 files changed, 3 insertions(+), 20 deletions(-)

diff --git a/cpus.c b/cpus.c
index bf666b0..152b596 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1250,26 +1250,9 @@ void on_vcpu(CPUState *env, void (*func)(void *data), void *data)
 
 void kvm_update_interrupt_request(CPUState *env)
 {
-    int signal = 0;
-
-    if (env) {
-        if (!current_env || !current_env->created) {
-            signal = 1;
-        }
-        /*
-         * Testing for created here is really redundant
-         */
-        if (current_env && current_env->created &&
-            env != current_env && !env->thread_kicked) {
-            signal = 1;
-        }
-
-        if (signal) {
-            env->thread_kicked = true;
-            if (env->thread) {
-                pthread_kill(env->thread->thread, SIG_IPI);
-            }
-        }
+    if (!qemu_cpu_is_self(env) && !env->thread_kicked) {
+        env->thread_kicked = true;
+        pthread_kill(env->thread->thread, SIG_IPI);
     }
 }
 
-- 
1.7.1

[PATCH 08/20] qemu-kvm: Replace kvm_update_interrupt_request with qemu_cpu_kick

[Jan Kiszka]

Activate the iothread version of qemu_cpu_kick. We just need to
initialize the yet unused CPUState::halt_cond for it.

This finally obsoletes kvm_update_interrupt_request, so drop it.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c     |   15 ++++-----------
 kvm-all.c  |    1 -
 qemu-kvm.h |    1 -
 3 files changed, 4 insertions(+), 13 deletions(-)

diff --git a/cpus.c b/cpus.c
index 152b596..2f72243 100644
--- a/cpus.c
+++ b/cpus.c
@@ -590,11 +590,11 @@ void resume_all_vcpus(void)
 void pause_all_vcpus(void)
 {
 }
+#ifndef CONFIG_IOTHREAD
 
 void qemu_cpu_kick(void *env)
 {
 }
-#ifndef CONFIG_IOTHREAD
 
 void qemu_cpu_kick_self(void)
 {
@@ -879,7 +879,6 @@ static void qemu_cpu_kick_thread(CPUState *env)
 #endif
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 void qemu_cpu_kick(void *_env)
 {
     CPUState *env = _env;
@@ -890,7 +889,6 @@ void qemu_cpu_kick(void *_env)
         env->thread_kicked = true;
     }
 }
-#endif /* UNUSED_IOTHREAD_IMPL */
 
 void qemu_cpu_kick_self(void)
 {
@@ -1248,14 +1246,6 @@ void on_vcpu(CPUState *env, void (*func)(void *data), void *data)
     }
 }
 
-void kvm_update_interrupt_request(CPUState *env)
-{
-    if (!qemu_cpu_is_self(env) && !env->thread_kicked) {
-        env->thread_kicked = true;
-        pthread_kill(env->thread->thread, SIG_IPI);
-    }
-}
-
 static int kvm_cpu_is_stopped(CPUState *env)
 {
     return !vm_running || env->stopped;
@@ -1448,6 +1438,9 @@ static void *ap_main_loop(void *_env)
     current_env = env;
     env->thread_id = kvm_get_thread_id();
 
+    env->halt_cond = qemu_mallocz(sizeof(QemuCond));
+    qemu_cond_init(env->halt_cond);
+
     qemu_mutex_lock(&qemu_global_mutex);
 
     if (kvm_create_vcpu(env) < 0) {
diff --git a/kvm-all.c b/kvm-all.c
index bd53422..8bef14a 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -703,7 +703,6 @@ static void kvm_handle_interrupt(CPUState *env, int mask)
     if (!qemu_cpu_is_self(env)) {
         qemu_cpu_kick(env);
     }
-    kvm_update_interrupt_request(env);
 }
 
 int kvm_init(void)
diff --git a/qemu-kvm.h b/qemu-kvm.h
index 0ebd3eb..20e24e7 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -275,7 +275,6 @@ void kvm_hpet_enable_kpit(void);
 void kvm_hpet_disable_kpit(void);
 
 void on_vcpu(CPUState *env, void (*func)(void *data), void *data);
-void kvm_update_interrupt_request(CPUState *env);
 
 int kvm_set_boot_cpu_id(KVMState *s, uint32_t id);
 
-- 
1.7.1

[PATCH 09/20] qemu-kvm: Use upstream run_on_cpu and flush_queued_work

[Jan Kiszka]

Upstream is now identical to qemu-kvm's versions, and we are using the
same signaling mechanisms now.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpu-defs.h           |   11 +----------
 cpus.c               |   45 +--------------------------------------------
 kvm-all.c            |    4 ----
 qemu-kvm.c           |    4 ++--
 qemu-kvm.h           |    2 --
 target-i386/helper.c |    4 ----
 6 files changed, 4 insertions(+), 66 deletions(-)

diff --git a/cpu-defs.h b/cpu-defs.h
index 5a0f11d..db48a7a 100644
--- a/cpu-defs.h
+++ b/cpu-defs.h
@@ -27,7 +27,6 @@
 #include <setjmp.h>
 #include <inttypes.h>
 #include <signal.h>
-#include <pthread.h>
 #include "osdep.h"
 #include "qemu-queue.h"
 #include "targphys.h"
@@ -154,13 +153,6 @@ typedef struct CPUWatchpoint {
     QTAILQ_ENTRY(CPUWatchpoint) entry;
 } CPUWatchpoint;
 
-/* forward decleration */
-struct qemu_work_item;
-
-struct KVMCPUState {
-    struct qemu_work_item *queued_work_first, *queued_work_last;
-};
-
 #define CPU_TEMP_BUF_NLONGS 128
 #define CPU_COMMON                                                      \
     struct TranslationBlock *current_tb; /* currently executing TB  */  \
@@ -226,7 +218,6 @@ struct KVMCPUState {
     struct KVMState *kvm_state;                                         \
     struct kvm_run *kvm_run;                                            \
     int kvm_fd;                                                         \
-    int kvm_vcpu_dirty;                                                 \
-    struct KVMCPUState kvm_cpu_state;
+    int kvm_vcpu_dirty;
 
 #endif
diff --git a/cpus.c b/cpus.c
index 2f72243..fc5605d 100644
--- a/cpus.c
+++ b/cpus.c
@@ -722,7 +722,6 @@ void run_on_cpu(CPUState *env, void (*func)(void *data), void *data)
     }
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 static void flush_queued_work(CPUState *env)
 {
     struct qemu_work_item *wi;
@@ -740,6 +739,7 @@ static void flush_queued_work(CPUState *env)
     qemu_cond_broadcast(&qemu_work_cond);
 }
 
+#ifdef UNUSED_IOTHREAD_IMPL
 static void qemu_wait_io_event_common(CPUState *env)
 {
     if (env->stop) {
@@ -1220,54 +1220,11 @@ static void sig_ipi_handler(int n)
 {
 }
 
-void on_vcpu(CPUState *env, void (*func)(void *data), void *data)
-{
-    struct qemu_work_item wi;
-
-    if (env == current_env) {
-        func(data);
-        return;
-    }
-
-    wi.func = func;
-    wi.data = data;
-    if (!env->kvm_cpu_state.queued_work_first) {
-        env->kvm_cpu_state.queued_work_first = &wi;
-    } else {
-        env->kvm_cpu_state.queued_work_last->next = &wi;
-    }
-    env->kvm_cpu_state.queued_work_last = &wi;
-    wi.next = NULL;
-    wi.done = false;
-
-    pthread_kill(env->thread->thread, SIG_IPI);
-    while (!wi.done) {
-        kvm_cond_wait(&qemu_work_cond);
-    }
-}
-
 static int kvm_cpu_is_stopped(CPUState *env)
 {
     return !vm_running || env->stopped;
 }
 
-static void flush_queued_work(CPUState *env)
-{
-    struct qemu_work_item *wi;
-
-    if (!env->kvm_cpu_state.queued_work_first) {
-        return;
-    }
-
-    while ((wi = env->kvm_cpu_state.queued_work_first)) {
-        env->kvm_cpu_state.queued_work_first = wi->next;
-        wi->func(wi->data);
-        wi->done = true;
-    }
-    env->kvm_cpu_state.queued_work_last = NULL;
-    qemu_cond_broadcast(&qemu_work_cond);
-}
-
 static void kvm_main_loop_wait(CPUState *env, int timeout)
 {
     struct timespec ts;
diff --git a/kvm-all.c b/kvm-all.c
index 8bef14a..e6a2b65 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -33,10 +33,6 @@
 #include <sys/eventfd.h>
 #endif
 
-#ifndef OBSOLETE_KVM_IMPL
-#define run_on_cpu on_vcpu
-#endif /* !OBSOLETE_KVM_IMPL */
-
 /* KVM uses PAGE_SIZE in it's definition of COALESCED_MMIO_MAX */
 #define PAGE_SIZE TARGET_PAGE_SIZE
 
diff --git a/qemu-kvm.c b/qemu-kvm.c
index dfbf045..649af9c 100644
--- a/qemu-kvm.c
+++ b/qemu-kvm.c
@@ -602,7 +602,7 @@ int kvm_add_ioport_region(unsigned long start, unsigned long size)
 
     if (qemu_system_is_ready()) {
         for (env = first_cpu; env != NULL; env = env->next_cpu) {
-            on_vcpu(env, do_set_ioport_access, region);
+            run_on_cpu(env, do_set_ioport_access, region);
             if (region->status < 0) {
                 r = region->status;
                 kvm_remove_ioport_region(start, size);
@@ -625,7 +625,7 @@ int kvm_remove_ioport_region(unsigned long start, unsigned long size)
         }
         if (qemu_system_is_ready()) {
             for (env = first_cpu; env != NULL; env = env->next_cpu) {
-                on_vcpu(env, do_set_ioport_access, region);
+                run_on_cpu(env, do_set_ioport_access, region);
             }
         }
         QLIST_REMOVE(region, entry);
diff --git a/qemu-kvm.h b/qemu-kvm.h
index 20e24e7..a577eba 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -274,8 +274,6 @@ void kvm_load_lapic(CPUState *env);
 void kvm_hpet_enable_kpit(void);
 void kvm_hpet_disable_kpit(void);
 
-void on_vcpu(CPUState *env, void (*func)(void *data), void *data);
-
 int kvm_set_boot_cpu_id(KVMState *s, uint32_t id);
 
 void kvm_tpr_access_report(CPUState *env, uint64_t rip, int is_write);
diff --git a/target-i386/helper.c b/target-i386/helper.c
index 2315c84..89df997 100644
--- a/target-i386/helper.c
+++ b/target-i386/helper.c
@@ -32,10 +32,6 @@
 #include "monitor.h"
 #endif
 
-#ifndef OBSOLETE_KVM_IMPL
-#define run_on_cpu on_vcpu
-#endif /* !OBSOLETE_KVM_IMPL */
-
 //#define DEBUG_MMU
 
 /* NOTE: must be called outside the CPU execute loop */
-- 
1.7.1

[PATCH 10/20] qemu-kvm: Drop kvm_cond_wait

[Jan Kiszka]

No caller depends on cpu_single_env saving/restoring anymore, so we can
call qemu_cond_wait directly.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |   14 +++-----------
 1 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/cpus.c b/cpus.c
index fc5605d..4b5d187 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1208,14 +1208,6 @@ unsigned long kvm_get_thread_id(void)
     return syscall(SYS_gettid);
 }
 
-static void kvm_cond_wait(QemuCond *cond)
-{
-    CPUState *env = cpu_single_env;
-
-    qemu_cond_wait(cond, &qemu_global_mutex);
-    cpu_single_env = env;
-}
-
 static void sig_ipi_handler(int n)
 {
 }
@@ -1313,7 +1305,7 @@ static void pause_all_threads(void)
     }
 
     while (!all_threads_paused()) {
-        kvm_cond_wait(&qemu_pause_cond);
+        qemu_cond_wait(&qemu_pause_cond, &qemu_global_mutex);
     }
 }
 
@@ -1411,7 +1403,7 @@ static void *ap_main_loop(void *_env)
 
     /* and wait for machine initialization */
     while (!qemu_system_ready) {
-        kvm_cond_wait(&qemu_system_cond);
+        qemu_cond_wait(&qemu_system_cond, &qemu_global_mutex);
     }
 
     kvm_main_loop_cpu(env);
@@ -1424,7 +1416,7 @@ int kvm_init_vcpu(CPUState *env)
     qemu_thread_create(env->thread, ap_main_loop, env);
 
     while (env->created == 0) {
-        kvm_cond_wait(&qemu_cpu_cond);
+        qemu_cond_wait(&qemu_cpu_cond, &qemu_global_mutex);
     }
 
     return 0;
-- 
1.7.1

[PATCH 11/20] qemu-kvm: Replace kvm_cpu_is_stopped with cpu_is_stopped

[Jan Kiszka]

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |    7 +------
 1 files changed, 1 insertions(+), 6 deletions(-)

diff --git a/cpus.c b/cpus.c
index 4b5d187..c7a5dec 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1212,11 +1212,6 @@ static void sig_ipi_handler(int n)
 {
 }
 
-static int kvm_cpu_is_stopped(CPUState *env)
-{
-    return !vm_running || env->stopped;
-}
-
 static void kvm_main_loop_wait(CPUState *env, int timeout)
 {
     struct timespec ts;
@@ -1361,7 +1356,7 @@ static int kvm_main_loop_cpu(CPUState *env)
 {
     while (1) {
         int timeout = 1000;
-        if (!kvm_cpu_is_stopped(env)) {
+        if (!cpu_is_stopped(env)) {
             switch (kvm_cpu_exec(env)) {
             case EXCP_HLT:
                 break;
-- 
1.7.1

[PATCH 12/20] qemu-kvm: Remove obsolete current_env

[Jan Kiszka]

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |    5 +----
 1 files changed, 1 insertions(+), 4 deletions(-)

diff --git a/cpus.c b/cpus.c
index c7a5dec..9b3f218 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1199,8 +1199,6 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
 
 #include <sys/syscall.h>
 
-__thread CPUState *current_env;
-
 static CPUState *kvm_debug_cpu_requested;
 
 unsigned long kvm_get_thread_id(void)
@@ -1379,7 +1377,6 @@ static void *ap_main_loop(void *_env)
 {
     CPUState *env = _env;
 
-    current_env = env;
     env->thread_id = kvm_get_thread_id();
 
     env->halt_cond = qemu_mallocz(sizeof(QemuCond));
@@ -1393,7 +1390,7 @@ static void *ap_main_loop(void *_env)
     setup_kernel_sigmask(env);
 
     /* signal VCPU creation */
-    current_env->created = 1;
+    env->created = 1;
     qemu_cond_signal(&qemu_cpu_cond);
 
     /* and wait for machine initialization */
-- 
1.7.1

[PATCH 13/20] qemu-kvm: Use upstream vcpu pause/resume

[Jan Kiszka]

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c   |   84 ++++++++-----------------------------------------------------
 sysemu.h |    1 +
 vl.c     |    6 +---
 3 files changed, 13 insertions(+), 78 deletions(-)

diff --git a/cpus.c b/cpus.c
index 9b3f218..470ab00 100644
--- a/cpus.c
+++ b/cpus.c
@@ -582,7 +582,6 @@ void run_on_cpu(CPUState *env, void (*func)(void *data), void *data)
     func(data);
 }
 
-#endif /* !IOTHREAD */
 void resume_all_vcpus(void)
 {
 }
@@ -590,7 +589,6 @@ void resume_all_vcpus(void)
 void pause_all_vcpus(void)
 {
 }
-#ifndef CONFIG_IOTHREAD
 
 void qemu_cpu_kick(void *env)
 {
@@ -624,7 +622,6 @@ void qemu_notify_event(void)
 void qemu_mutex_lock_iothread(void) {}
 void qemu_mutex_unlock_iothread(void) {}
 
-#endif /* !IOTHREAD */
 void cpu_stop_current(void)
 {
 }
@@ -633,16 +630,15 @@ void vm_stop(int reason)
 {
     do_vm_stop(reason);
 }
-#ifndef CONFIG_IOTHREAD
 
 #else /* CONFIG_IOTHREAD */
 
 QemuMutex qemu_global_mutex;
 static QemuMutex qemu_fair_mutex;
 
-#ifdef UNUSED_IOTHREAD_IMPL
 static QemuThread io_thread;
 
+#ifdef UNUSED_IOTHREAD_IMPL
 static QemuThread *tcg_cpu_thread;
 static QemuCond *tcg_halt_cond;
 #endif
@@ -930,7 +926,6 @@ void qemu_mutex_unlock_iothread(void)
     qemu_mutex_unlock(&qemu_global_mutex);
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 static int all_vcpus_paused(void)
 {
     CPUState *penv = first_cpu;
@@ -977,6 +972,7 @@ void resume_all_vcpus(void)
     }
 }
 
+#ifdef UNUSED_IOTHREAD_IMPL
 static void qemu_tcg_init_vcpu(void *_env)
 {
     CPUState *env = _env;
@@ -1028,7 +1024,6 @@ void qemu_notify_event(void)
     qemu_event_increment();
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 void cpu_stop_current(void)
 {
     if (cpu_single_env) {
@@ -1052,7 +1047,6 @@ void vm_stop(int reason)
     }
     do_vm_stop(reason);
 }
-#endif /* UNUSED_IOTHREAD_IMPL */
 
 #endif
 
@@ -1264,65 +1258,6 @@ static void kvm_main_loop_wait(CPUState *env, int timeout)
     env->thread_kicked = false;
 }
 
-static int all_threads_paused(void)
-{
-    CPUState *penv = first_cpu;
-
-    while (penv) {
-        if (penv->stop) {
-            return 0;
-        }
-        penv = (CPUState *) penv->next_cpu;
-    }
-
-    return 1;
-}
-
-static void pause_all_threads(void)
-{
-    CPUState *penv = first_cpu;
-
-    while (penv) {
-        if (!qemu_cpu_is_self(penv)) {
-            penv->stop = 1;
-            if (!penv->thread_kicked) {
-                pthread_kill(penv->thread->thread, SIG_IPI);
-                penv->thread_kicked = true;
-            }
-        } else {
-            penv->stop = 0;
-            penv->stopped = 1;
-            cpu_exit(penv);
-        }
-        penv = (CPUState *) penv->next_cpu;
-    }
-
-    while (!all_threads_paused()) {
-        qemu_cond_wait(&qemu_pause_cond, &qemu_global_mutex);
-    }
-}
-
-static void resume_all_threads(void)
-{
-    CPUState *penv = first_cpu;
-
-    while (penv) {
-        penv->stop = 0;
-        penv->stopped = 0;
-        pthread_kill(penv->thread->thread, SIG_IPI);
-        penv = (CPUState *) penv->next_cpu;
-    }
-}
-
-static void kvm_vm_state_change_handler(void *context, int running, int reason)
-{
-    if (running) {
-        resume_all_threads();
-    } else {
-        pause_all_threads();
-    }
-}
-
 static void setup_kernel_sigmask(CPUState *env)
 {
     sigset_t set;
@@ -1342,12 +1277,12 @@ static void setup_kernel_sigmask(CPUState *env)
 
 static void qemu_kvm_system_reset(void)
 {
-    pause_all_threads();
+    pause_all_vcpus();
 
     cpu_synchronize_all_states();
     qemu_system_reset();
 
-    resume_all_threads();
+    resume_all_vcpus();
 }
 
 static int kvm_main_loop_cpu(CPUState *env)
@@ -1420,8 +1355,6 @@ int kvm_init_ap(void)
 
     qemu_mutex_lock(&qemu_global_mutex);
 
-    qemu_add_vm_change_state_handler(kvm_vm_state_change_handler, NULL);
-
     signal(SIG_IPI, sig_ipi_handler);
 
     memset(&action, 0, sizeof(action));
@@ -1440,7 +1373,7 @@ bool qemu_system_is_ready(void)
 int kvm_main_loop(void)
 {
     sigset_t mask;
-    int sigfd;
+    int sigfd, r;
 
     qemu_system_ready = 1;
 
@@ -1482,10 +1415,13 @@ int kvm_main_loop(void)
             vm_stop(VMSTOP_DEBUG);
             kvm_debug_cpu_requested = NULL;
         }
+        if ((r = qemu_vmstop_requested())) {
+            vm_stop(r);
+        }
     }
 
     bdrv_close_all();
-    pause_all_threads();
+    pause_all_vcpus();
     qemu_mutex_unlock(&qemu_global_mutex);
 
     return 0;
@@ -1500,6 +1436,8 @@ static void qemu_kvm_init_main_loop(void)
     qemu_mutex_init(&qemu_fair_mutex);
     qemu_mutex_init(&qemu_global_mutex);
     qemu_mutex_lock(&qemu_global_mutex);
+
+    qemu_thread_get_self(&io_thread);
 }
 
 #endif /* CONFIG_KVM */
diff --git a/sysemu.h b/sysemu.h
index 560210d..5cacb9e 100644
--- a/sysemu.h
+++ b/sysemu.h
@@ -48,6 +48,7 @@ int qemu_reset_requested_get(void);
 int qemu_shutdown_requested(void);
 int qemu_reset_requested(void);
 int qemu_powerdown_requested(void);
+int qemu_vmstop_requested(void);
 void qemu_system_killed(int signal, pid_t pid);
 void qemu_kill_report(void);
 extern qemu_irq qemu_system_powerdown;
diff --git a/vl.c b/vl.c
index 2587aae..6b5877b 100644
--- a/vl.c
+++ b/vl.c
@@ -1225,7 +1225,7 @@ static int qemu_debug_requested(void)
     return r;
 }
 
-static int qemu_vmstop_requested(void)
+int qemu_vmstop_requested(void)
 {
     int r = vmstop_requested;
     vmstop_requested = 0;
@@ -1273,10 +1273,6 @@ void qemu_system_reset_request(void)
     } else {
         reset_requested = 1;
     }
-    if (cpu_single_env) {
-        cpu_single_env->stopped = 1;
-        cpu_exit(cpu_single_env);
-    }
     cpu_stop_current();
     qemu_notify_event();
 }
-- 
1.7.1

[PATCH 14/20] qemu-kvm: Move main loop setup code

[Jan Kiszka]

To prepare using the upstream iothread main loop, push some
initialization from kvm_main_loop to qemu_kvm_init_main_loop.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c |   41 +++++++++++++++++++++--------------------
 1 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/cpus.c b/cpus.c
index 470ab00..0455481 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1372,28 +1372,9 @@ bool qemu_system_is_ready(void)
 
 int kvm_main_loop(void)
 {
-    sigset_t mask;
-    int sigfd, r;
+    int r;
 
     qemu_system_ready = 1;
-
-    sigemptyset(&mask);
-    sigaddset(&mask, SIGIO);
-    sigaddset(&mask, SIGALRM);
-    sigaddset(&mask, SIGBUS);
-    sigprocmask(SIG_BLOCK, &mask, NULL);
-
-    sigfd = qemu_signalfd(&mask);
-    if (sigfd == -1) {
-        fprintf(stderr, "failed to create signalfd\n");
-        return -errno;
-    }
-
-    fcntl(sigfd, F_SETFL, O_NONBLOCK);
-
-    qemu_set_fd_handler2(sigfd, NULL, sigfd_handler, NULL,
-                         (void *)(unsigned long) sigfd);
-
     qemu_cond_broadcast(&qemu_system_cond);
 
     while (1) {
@@ -1429,6 +1410,26 @@ int kvm_main_loop(void)
 
 static void qemu_kvm_init_main_loop(void)
 {
+    sigset_t mask;
+    int sigfd;
+
+    sigemptyset(&mask);
+    sigaddset(&mask, SIGIO);
+    sigaddset(&mask, SIGALRM);
+    sigaddset(&mask, SIGBUS);
+    sigprocmask(SIG_BLOCK, &mask, NULL);
+
+    sigfd = qemu_signalfd(&mask);
+    if (sigfd == -1) {
+        fprintf(stderr, "failed to create signalfd\n");
+        exit(1);
+    }
+
+    fcntl(sigfd, F_SETFL, O_NONBLOCK);
+
+    qemu_set_fd_handler2(sigfd, NULL, sigfd_handler, NULL,
+                         (void *)(unsigned long) sigfd);
+
     qemu_cond_init(&qemu_cpu_cond);
     qemu_cond_init(&qemu_system_cond);
     qemu_cond_init(&qemu_pause_cond);
-- 
1.7.1

[PATCH 15/20] qemu-kvm: Use upstream's way of signaling debug stops

[Jan Kiszka]

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c   |    9 ++-------
 sysemu.h |    1 +
 vl.c     |    2 +-
 3 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/cpus.c b/cpus.c
index 0455481..23c6ccd 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1193,8 +1193,6 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
 
 #include <sys/syscall.h>
 
-static CPUState *kvm_debug_cpu_requested;
-
 unsigned long kvm_get_thread_id(void)
 {
     return syscall(SYS_gettid);
@@ -1294,8 +1292,7 @@ static int kvm_main_loop_cpu(CPUState *env)
             case EXCP_HLT:
                 break;
             case EXCP_DEBUG:
-                kvm_debug_cpu_requested = env;
-                env->stopped = 1;
+                cpu_handle_guest_debug(env);
                 break;
             default:
                 timeout = 0;
@@ -1391,10 +1388,8 @@ int kvm_main_loop(void)
             qemu_irq_raise(qemu_system_powerdown);
         } else if (qemu_reset_requested()) {
             qemu_kvm_system_reset();
-        } else if (kvm_debug_cpu_requested) {
-            gdb_set_stop_cpu(kvm_debug_cpu_requested);
+        } else if (qemu_debug_requested()) {
             vm_stop(VMSTOP_DEBUG);
-            kvm_debug_cpu_requested = NULL;
         }
         if ((r = qemu_vmstop_requested())) {
             vm_stop(r);
diff --git a/sysemu.h b/sysemu.h
index 5cacb9e..0d4f923 100644
--- a/sysemu.h
+++ b/sysemu.h
@@ -48,6 +48,7 @@ int qemu_reset_requested_get(void);
 int qemu_shutdown_requested(void);
 int qemu_reset_requested(void);
 int qemu_powerdown_requested(void);
+int qemu_debug_requested(void);
 int qemu_vmstop_requested(void);
 void qemu_system_killed(int signal, pid_t pid);
 void qemu_kill_report(void);
diff --git a/vl.c b/vl.c
index 6b5877b..e17094a 100644
--- a/vl.c
+++ b/vl.c
@@ -1218,7 +1218,7 @@ int qemu_powerdown_requested(void)
     return r;
 }
 
-static int qemu_debug_requested(void)
+int qemu_debug_requested(void)
 {
     int r = debug_requested;
     debug_requested = 0;
-- 
1.7.1

[PATCH 16/20] qemu-kvm: Use upstream main loop

[Jan Kiszka]

This also means switching to the CONFIG_IOTHREAD version of
qemu_main_loop_start and clean up some related patches of upstream code.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c     |   50 +++-----------------------------------------------
 qemu-kvm.h |    1 -
 sysemu.h   |    2 --
 vl.c       |   17 ++---------------
 4 files changed, 5 insertions(+), 65 deletions(-)

diff --git a/cpus.c b/cpus.c
index 23c6ccd..a4bac1a 100644
--- a/cpus.c
+++ b/cpus.c
@@ -546,11 +546,13 @@ int qemu_init_main_loop(void)
 
     return qemu_event_init();
 }
+#ifndef CONFIG_IOTHREAD
 
 void qemu_main_loop_start(void)
 {
 }
 
+#endif /* !CONFIG_IOTHREAD */
 void qemu_init_vcpu(void *_env)
 {
     CPUState *env = _env;
@@ -681,13 +683,13 @@ int qemu_init_main_loop(void)
 
     return 0;
 }
+#endif /* UNUSED_IOTHREAD_IMPL */
 
 void qemu_main_loop_start(void)
 {
     qemu_system_ready = 1;
     qemu_cond_broadcast(&qemu_system_cond);
 }
-#endif /* UNUSED_IOTHREAD_IMPL */
 
 void run_on_cpu(CPUState *env, void (*func)(void *data), void *data)
 {
@@ -1273,16 +1275,6 @@ static void setup_kernel_sigmask(CPUState *env)
     kvm_set_signal_mask(env, &set);
 }
 
-static void qemu_kvm_system_reset(void)
-{
-    pause_all_vcpus();
-
-    cpu_synchronize_all_states();
-    qemu_system_reset();
-
-    resume_all_vcpus();
-}
-
 static int kvm_main_loop_cpu(CPUState *env)
 {
     while (1) {
@@ -1367,42 +1359,6 @@ bool qemu_system_is_ready(void)
     return qemu_system_ready;
 }
 
-int kvm_main_loop(void)
-{
-    int r;
-
-    qemu_system_ready = 1;
-    qemu_cond_broadcast(&qemu_system_cond);
-
-    while (1) {
-        main_loop_wait(0);
-        if (qemu_shutdown_requested()) {
-            monitor_protocol_event(QEVENT_SHUTDOWN, NULL);
-            if (qemu_no_shutdown()) {
-                vm_stop(VMSTOP_SHUTDOWN);
-            } else {
-                break;
-            }
-        } else if (qemu_powerdown_requested()) {
-            monitor_protocol_event(QEVENT_POWERDOWN, NULL);
-            qemu_irq_raise(qemu_system_powerdown);
-        } else if (qemu_reset_requested()) {
-            qemu_kvm_system_reset();
-        } else if (qemu_debug_requested()) {
-            vm_stop(VMSTOP_DEBUG);
-        }
-        if ((r = qemu_vmstop_requested())) {
-            vm_stop(r);
-        }
-    }
-
-    bdrv_close_all();
-    pause_all_vcpus();
-    qemu_mutex_unlock(&qemu_global_mutex);
-
-    return 0;
-}
-
 static void qemu_kvm_init_main_loop(void)
 {
     sigset_t mask;
diff --git a/qemu-kvm.h b/qemu-kvm.h
index a577eba..2420f82 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -266,7 +266,6 @@ struct kvm_pit_state {
 #endif                          /* !CONFIG_KVM */
 
 int kvm_create_vcpu(CPUState *env);
-int kvm_main_loop(void);
 int kvm_init_ap(void);
 void kvm_save_lapic(CPUState *env);
 void kvm_load_lapic(CPUState *env);
diff --git a/sysemu.h b/sysemu.h
index 0d4f923..560210d 100644
--- a/sysemu.h
+++ b/sysemu.h
@@ -48,8 +48,6 @@ int qemu_reset_requested_get(void);
 int qemu_shutdown_requested(void);
 int qemu_reset_requested(void);
 int qemu_powerdown_requested(void);
-int qemu_debug_requested(void);
-int qemu_vmstop_requested(void);
 void qemu_system_killed(int signal, pid_t pid);
 void qemu_kill_report(void);
 extern qemu_irq qemu_system_powerdown;
diff --git a/vl.c b/vl.c
index e17094a..0bbd4e4 100644
--- a/vl.c
+++ b/vl.c
@@ -1164,13 +1164,6 @@ static int powerdown_requested;
 static int debug_requested;
 static int vmstop_requested;
 
-int qemu_no_shutdown(void)
-{
-    int r = no_shutdown;
-    no_shutdown = 0;
-    return r;
-}
-
 int qemu_shutdown_requested_get(void)
 {
     return shutdown_requested;
@@ -1218,14 +1211,14 @@ int qemu_powerdown_requested(void)
     return r;
 }
 
-int qemu_debug_requested(void)
+static int qemu_debug_requested(void)
 {
     int r = debug_requested;
     debug_requested = 0;
     return r;
 }
 
-int qemu_vmstop_requested(void)
+static int qemu_vmstop_requested(void)
 {
     int r = vmstop_requested;
     vmstop_requested = 0;
@@ -1372,12 +1365,6 @@ static void main_loop(void)
 #endif
     int r;
 
-    if (kvm_enabled()) {
-        kvm_main_loop();
-        cpu_disable_ticks();
-        return;
-    }
-
     qemu_main_loop_start();
 
     for (;;) {
-- 
1.7.1

[PATCH 17/20] qemu-kvm: Use upstream main loop initialization

[Jan Kiszka]

Switch to CONFIG_IOTHREAD version of qemu_kvm_init_main_loop and drop
qemu_kvm_init_main_loop as well as kvm_init_ap.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c     |   62 +-----------------------------------------------------------
 kvm-all.c  |    2 -
 qemu-kvm.h |    1 -
 3 files changed, 1 insertions(+), 64 deletions(-)

diff --git a/cpus.c b/cpus.c
index a4bac1a..7bd888a 100644
--- a/cpus.c
+++ b/cpus.c
@@ -529,9 +529,7 @@ static void qemu_tcg_init_cpu_signals(void)
 }
 #endif /* _WIN32 */
 
-/*#ifndef CONFIG_IOTHREAD*/
-static void qemu_kvm_init_main_loop(void);
-
+#ifndef CONFIG_IOTHREAD
 int qemu_init_main_loop(void)
 {
     int ret;
@@ -540,13 +538,11 @@ int qemu_init_main_loop(void)
     if (ret) {
         return ret;
     }
-    qemu_kvm_init_main_loop();
 
     qemu_init_sigbus();
 
     return qemu_event_init();
 }
-#ifndef CONFIG_IOTHREAD
 
 void qemu_main_loop_start(void)
 {
@@ -653,7 +649,6 @@ static QemuCond qemu_system_cond;
 static QemuCond qemu_pause_cond;
 static QemuCond qemu_work_cond;
 
-#ifdef UNUSED_IOTHREAD_IMPL
 int qemu_init_main_loop(void)
 {
     int ret;
@@ -683,7 +678,6 @@ int qemu_init_main_loop(void)
 
     return 0;
 }
-#endif /* UNUSED_IOTHREAD_IMPL */
 
 void qemu_main_loop_start(void)
 {
@@ -1200,10 +1194,6 @@ unsigned long kvm_get_thread_id(void)
     return syscall(SYS_gettid);
 }
 
-static void sig_ipi_handler(int n)
-{
-}
-
 static void kvm_main_loop_wait(CPUState *env, int timeout)
 {
     struct timespec ts;
@@ -1338,58 +1328,8 @@ int kvm_init_vcpu(CPUState *env)
     return 0;
 }
 
-int kvm_init_ap(void)
-{
-    struct sigaction action;
-
-    qemu_mutex_lock(&qemu_global_mutex);
-
-    signal(SIG_IPI, sig_ipi_handler);
-
-    memset(&action, 0, sizeof(action));
-    action.sa_flags = SA_SIGINFO;
-    action.sa_sigaction = (void (*)(int, siginfo_t*, void*))sigbus_handler;
-    sigaction(SIGBUS, &action, NULL);
-    prctl(PR_MCE_KILL, 1, 1, 0, 0);
-    return 0;
-}
-
 bool qemu_system_is_ready(void)
 {
     return qemu_system_ready;
 }
-
-static void qemu_kvm_init_main_loop(void)
-{
-    sigset_t mask;
-    int sigfd;
-
-    sigemptyset(&mask);
-    sigaddset(&mask, SIGIO);
-    sigaddset(&mask, SIGALRM);
-    sigaddset(&mask, SIGBUS);
-    sigprocmask(SIG_BLOCK, &mask, NULL);
-
-    sigfd = qemu_signalfd(&mask);
-    if (sigfd == -1) {
-        fprintf(stderr, "failed to create signalfd\n");
-        exit(1);
-    }
-
-    fcntl(sigfd, F_SETFL, O_NONBLOCK);
-
-    qemu_set_fd_handler2(sigfd, NULL, sigfd_handler, NULL,
-                         (void *)(unsigned long) sigfd);
-
-    qemu_cond_init(&qemu_cpu_cond);
-    qemu_cond_init(&qemu_system_cond);
-    qemu_cond_init(&qemu_pause_cond);
-    qemu_cond_init(&qemu_work_cond);
-    qemu_mutex_init(&qemu_fair_mutex);
-    qemu_mutex_init(&qemu_global_mutex);
-    qemu_mutex_lock(&qemu_global_mutex);
-
-    qemu_thread_get_self(&io_thread);
-}
-
 #endif /* CONFIG_KVM */
diff --git a/kvm-all.c b/kvm-all.c
index e6a2b65..7b032ba 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -821,8 +821,6 @@ int kvm_init(void)
         return ret;
     }
 
-    kvm_init_ap();
-
     cpu_interrupt_handler = kvm_handle_interrupt;
 
     return 0;
diff --git a/qemu-kvm.h b/qemu-kvm.h
index 2420f82..a4c811c 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -266,7 +266,6 @@ struct kvm_pit_state {
 #endif                          /* !CONFIG_KVM */
 
 int kvm_create_vcpu(CPUState *env);
-int kvm_init_ap(void);
 void kvm_save_lapic(CPUState *env);
 void kvm_load_lapic(CPUState *env);
 
-- 
1.7.1

[PATCH 18/20] qemu-kvm: Replace kvm_get_thread_id with qemu_get_thread_id

[Jan Kiszka]

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c     |    9 +--------
 qemu-kvm.h |    2 --
 2 files changed, 1 insertions(+), 10 deletions(-)

diff --git a/cpus.c b/cpus.c
index 7bd888a..2cfaa0d 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1187,13 +1187,6 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
  */
 #ifdef CONFIG_KVM
 
-#include <sys/syscall.h>
-
-unsigned long kvm_get_thread_id(void)
-{
-    return syscall(SYS_gettid);
-}
-
 static void kvm_main_loop_wait(CPUState *env, int timeout)
 {
     struct timespec ts;
@@ -1291,7 +1284,7 @@ static void *ap_main_loop(void *_env)
 {
     CPUState *env = _env;
 
-    env->thread_id = kvm_get_thread_id();
+    env->thread_id = qemu_get_thread_id();
 
     env->halt_cond = qemu_mallocz(sizeof(QemuCond));
     qemu_cond_init(env->halt_cond);
diff --git a/qemu-kvm.h b/qemu-kvm.h
index a4c811c..18eaade 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -300,6 +300,4 @@ int kvm_handle_tpr_access(CPUState *env);
 
 int kvm_tpr_enable_vapic(CPUState *env);
 
-unsigned long kvm_get_thread_id(void);
-
 #endif
-- 
1.7.1

[PATCH 19/20] qemu-kvm: Use upstream kvm vcpu initialization

[Jan Kiszka]

This converts everything except for kvm_main_loop_cpu to the upstream
version, i.e. thread creation, signal setup, some further field
initializations, and completion signaling.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 cpus.c     |   89 ++++++++---------------------------------------------------
 kvm-all.c  |    2 +-
 qemu-kvm.h |    1 -
 3 files changed, 13 insertions(+), 79 deletions(-)

diff --git a/cpus.c b/cpus.c
index 2cfaa0d..30bbfa4 100644
--- a/cpus.c
+++ b/cpus.c
@@ -344,11 +344,9 @@ fail:
     return err;
 }
 
-#ifdef OBSOLETE_KVM_IMPL
 static void dummy_signal(int sig)
 {
 }
-#endif
 
 /* If we have signalfd, we mask out the signals we want to handle and then
  * use signalfd to listen for them.  We rely on whatever the current signal
@@ -431,7 +429,6 @@ static int qemu_signal_init(void)
 
 static void qemu_kvm_init_cpu_signals(CPUState *env)
 {
-#if 0 /* Causes regressions: autotest WinXP.64 migrate.tcp */
     int r;
     sigset_t set;
     struct sigaction sigact;
@@ -467,7 +464,6 @@ static void qemu_kvm_init_cpu_signals(CPUState *env)
         fprintf(stderr, "kvm_set_signal_mask: %s\n", strerror(-r));
         exit(1);
     }
-#endif
 }
 
 static void qemu_tcg_init_cpu_signals(void)
@@ -548,7 +544,6 @@ void qemu_main_loop_start(void)
 {
 }
 
-#endif /* !CONFIG_IOTHREAD */
 void qemu_init_vcpu(void *_env)
 {
     CPUState *env = _env;
@@ -568,7 +563,6 @@ void qemu_init_vcpu(void *_env)
         qemu_tcg_init_cpu_signals();
     }
 }
-#ifndef CONFIG_IOTHREAD
 
 int qemu_cpu_is_self(void *env)
 {
@@ -636,10 +630,8 @@ static QemuMutex qemu_fair_mutex;
 
 static QemuThread io_thread;
 
-#ifdef UNUSED_IOTHREAD_IMPL
 static QemuThread *tcg_cpu_thread;
 static QemuCond *tcg_halt_cond;
-#endif
 
 static int qemu_system_ready;
 /* cpu creation */
@@ -685,6 +677,11 @@ void qemu_main_loop_start(void)
     qemu_cond_broadcast(&qemu_system_cond);
 }
 
+bool qemu_system_is_ready(void)
+{
+    return qemu_system_ready;
+}
+
 void run_on_cpu(CPUState *env, void (*func)(void *data), void *data)
 {
     struct qemu_work_item wi;
@@ -731,7 +728,6 @@ static void flush_queued_work(CPUState *env)
     qemu_cond_broadcast(&qemu_work_cond);
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 static void qemu_wait_io_event_common(CPUState *env)
 {
     if (env->stop) {
@@ -771,6 +767,7 @@ static void qemu_tcg_wait_io_event(void)
     }
 }
 
+#ifdef UNUSED_IOTHREAD_IMPL
 static void qemu_kvm_wait_io_event(CPUState *env)
 {
     while (cpu_thread_is_idle(env)) {
@@ -780,6 +777,9 @@ static void qemu_kvm_wait_io_event(CPUState *env)
     qemu_kvm_eat_signals(env);
     qemu_wait_io_event_common(env);
 }
+#endif /* UNUSED_IOTHREAD_IMPL */
+
+static int kvm_main_loop_cpu(CPUState *env);
 
 static void *qemu_kvm_cpu_thread_fn(void *arg)
 {
@@ -807,6 +807,7 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
         qemu_cond_wait(&qemu_system_cond, &qemu_global_mutex);
     }
 
+#ifdef UNUSED_IOTHREAD_IMPL
     while (1) {
         if (cpu_can_run(env)) {
             r = kvm_cpu_exec(env);
@@ -816,6 +817,8 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
         }
         qemu_kvm_wait_io_event(env);
     }
+#endif /* UNUSED_IOTHREAD_IMPL */
+    kvm_main_loop_cpu(env);
 
     return NULL;
 }
@@ -850,7 +853,6 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
 
     return NULL;
 }
-#endif /* UNUSED_IOTHREAD_IMPL */
 
 static void qemu_cpu_kick_thread(CPUState *env)
 {
@@ -968,7 +970,6 @@ void resume_all_vcpus(void)
     }
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 static void qemu_tcg_init_vcpu(void *_env)
 {
     CPUState *env = _env;
@@ -1013,7 +1014,6 @@ void qemu_init_vcpu(void *_env)
         qemu_tcg_init_vcpu(env);
     }
 }
-#endif /* UNUSED_IOTHREAD_IMPL */
 
 void qemu_notify_event(void)
 {
@@ -1185,7 +1185,6 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
 /*
  * qemu-kvm threading code, integration zone
  */
-#ifdef CONFIG_KVM
 
 static void kvm_main_loop_wait(CPUState *env, int timeout)
 {
@@ -1241,23 +1240,6 @@ static void kvm_main_loop_wait(CPUState *env, int timeout)
     env->thread_kicked = false;
 }
 
-static void setup_kernel_sigmask(CPUState *env)
-{
-    sigset_t set;
-
-    sigemptyset(&set);
-    sigaddset(&set, SIGUSR2);
-    sigaddset(&set, SIGIO);
-    sigaddset(&set, SIGALRM);
-    sigprocmask(SIG_BLOCK, &set, NULL);
-
-    sigprocmask(SIG_BLOCK, NULL, &set);
-    sigdelset(&set, SIG_IPI);
-    sigdelset(&set, SIGBUS);
-
-    kvm_set_signal_mask(env, &set);
-}
-
 static int kvm_main_loop_cpu(CPUState *env)
 {
     while (1) {
@@ -1279,50 +1261,3 @@ static int kvm_main_loop_cpu(CPUState *env)
     qemu_mutex_unlock(&qemu_global_mutex);
     return 0;
 }
-
-static void *ap_main_loop(void *_env)
-{
-    CPUState *env = _env;
-
-    env->thread_id = qemu_get_thread_id();
-
-    env->halt_cond = qemu_mallocz(sizeof(QemuCond));
-    qemu_cond_init(env->halt_cond);
-
-    qemu_mutex_lock(&qemu_global_mutex);
-
-    if (kvm_create_vcpu(env) < 0) {
-        abort();
-    }
-    setup_kernel_sigmask(env);
-
-    /* signal VCPU creation */
-    env->created = 1;
-    qemu_cond_signal(&qemu_cpu_cond);
-
-    /* and wait for machine initialization */
-    while (!qemu_system_ready) {
-        qemu_cond_wait(&qemu_system_cond, &qemu_global_mutex);
-    }
-
-    kvm_main_loop_cpu(env);
-    return NULL;
-}
-
-int kvm_init_vcpu(CPUState *env)
-{
-    env->thread = qemu_mallocz(sizeof(QemuThread));
-    qemu_thread_create(env->thread, ap_main_loop, env);
-
-    while (env->created == 0) {
-        qemu_cond_wait(&qemu_cpu_cond, &qemu_global_mutex);
-    }
-
-    return 0;
-}
-
-bool qemu_system_is_ready(void)
-{
-    return qemu_system_ready;
-}
-#endif /* CONFIG_KVM */
diff --git a/kvm-all.c b/kvm-all.c
index 7b032ba..f3958e8 100644
--- a/kvm-all.c
+++ b/kvm-all.c
@@ -204,7 +204,7 @@ int kvm_pit_in_kernel(void)
     return kvm_state->pit_in_kernel;
 }
 
-int kvm_create_vcpu(CPUState *env)
+int kvm_init_vcpu(CPUState *env)
 {
     KVMState *s = kvm_state;
     long mmap_size;
diff --git a/qemu-kvm.h b/qemu-kvm.h
index 18eaade..0f833f1 100644
--- a/qemu-kvm.h
+++ b/qemu-kvm.h
@@ -265,7 +265,6 @@ struct kvm_pit_state {
 
 #endif                          /* !CONFIG_KVM */
 
-int kvm_create_vcpu(CPUState *env);
 void kvm_save_lapic(CPUState *env);
 void kvm_load_lapic(CPUState *env);
 
-- 
1.7.1

[PATCH 20/20] qemu-kvm: Use upstream vcpu loop

[Jan Kiszka]

With switching to upstream code, TCG mode becomes usable again.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 arch_init.c |    3 +-
 cpus.c      |   89 -----------------------------------------------------------
 vl.c        |    5 +--
 3 files changed, 2 insertions(+), 95 deletions(-)

diff --git a/arch_init.c b/arch_init.c
index ac67c3d..484b39d 100644
--- a/arch_init.c
+++ b/arch_init.c
@@ -711,8 +711,7 @@ int audio_available(void)
 
 int tcg_available(void)
 {
-    /* temporarily broken */
-    return 0;
+    return 1;
 }
 
 int kvm_available(void)
diff --git a/cpus.c b/cpus.c
index 30bbfa4..b31bcc3 100644
--- a/cpus.c
+++ b/cpus.c
@@ -767,7 +767,6 @@ static void qemu_tcg_wait_io_event(void)
     }
 }
 
-#ifdef UNUSED_IOTHREAD_IMPL
 static void qemu_kvm_wait_io_event(CPUState *env)
 {
     while (cpu_thread_is_idle(env)) {
@@ -777,9 +776,6 @@ static void qemu_kvm_wait_io_event(CPUState *env)
     qemu_kvm_eat_signals(env);
     qemu_wait_io_event_common(env);
 }
-#endif /* UNUSED_IOTHREAD_IMPL */
-
-static int kvm_main_loop_cpu(CPUState *env);
 
 static void *qemu_kvm_cpu_thread_fn(void *arg)
 {
@@ -807,7 +803,6 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
         qemu_cond_wait(&qemu_system_cond, &qemu_global_mutex);
     }
 
-#ifdef UNUSED_IOTHREAD_IMPL
     while (1) {
         if (cpu_can_run(env)) {
             r = kvm_cpu_exec(env);
@@ -817,8 +812,6 @@ static void *qemu_kvm_cpu_thread_fn(void *arg)
         }
         qemu_kvm_wait_io_event(env);
     }
-#endif /* UNUSED_IOTHREAD_IMPL */
-    kvm_main_loop_cpu(env);
 
     return NULL;
 }
@@ -1179,85 +1172,3 @@ void list_cpus(FILE *f, fprintf_function cpu_fprintf, const char *optarg)
     cpu_list(f, cpu_fprintf); /* deprecated */
 #endif
 }
-
-
-
-/*
- * qemu-kvm threading code, integration zone
- */
-
-static void kvm_main_loop_wait(CPUState *env, int timeout)
-{
-    struct timespec ts;
-    int r, e;
-    siginfo_t siginfo;
-    sigset_t waitset;
-    sigset_t chkset;
-
-    ts.tv_sec = timeout / 1000;
-    ts.tv_nsec = (timeout % 1000) * 1000000;
-    sigemptyset(&waitset);
-    sigaddset(&waitset, SIG_IPI);
-    sigaddset(&waitset, SIGBUS);
-
-    do {
-        qemu_mutex_unlock(&qemu_global_mutex);
-
-        r = sigtimedwait(&waitset, &siginfo, &ts);
-        e = errno;
-
-        qemu_mutex_lock(&qemu_global_mutex);
-
-        if (r == -1 && !(e == EAGAIN || e == EINTR)) {
-            printf("sigtimedwait: %s\n", strerror(e));
-            exit(1);
-        }
-
-        switch (r) {
-        case SIGBUS:
-            if (kvm_on_sigbus_vcpu(env, siginfo.si_code, siginfo.si_addr))
-                sigbus_reraise();
-            break;
-        default:
-            break;
-        }
-
-        r = sigpending(&chkset);
-        if (r == -1) {
-            printf("sigpending: %s\n", strerror(e));
-            exit(1);
-        }
-    } while (sigismember(&chkset, SIG_IPI) || sigismember(&chkset, SIGBUS));
-
-    flush_queued_work(env);
-
-    if (env->stop) {
-        env->stop = 0;
-        env->stopped = 1;
-        qemu_cond_signal(&qemu_pause_cond);
-    }
-
-    env->thread_kicked = false;
-}
-
-static int kvm_main_loop_cpu(CPUState *env)
-{
-    while (1) {
-        int timeout = 1000;
-        if (!cpu_is_stopped(env)) {
-            switch (kvm_cpu_exec(env)) {
-            case EXCP_HLT:
-                break;
-            case EXCP_DEBUG:
-                cpu_handle_guest_debug(env);
-                break;
-            default:
-                timeout = 0;
-                break;
-            }
-        }
-        kvm_main_loop_wait(env, timeout);
-    }
-    qemu_mutex_unlock(&qemu_global_mutex);
-    return 0;
-}
diff --git a/vl.c b/vl.c
index 0bbd4e4..01ca414 100644
--- a/vl.c
+++ b/vl.c
@@ -1893,8 +1893,7 @@ static int debugcon_parse(const char *devname)
 
 static int tcg_init(void)
 {
-    fprintf(stderr, "Emulation temporarily broken\n");
-    return -1;
+    return 0;
 }
 
 static struct {
@@ -2683,8 +2682,6 @@ int main(int argc, char **argv, char **envp)
                 }
                 break;
 	    case QEMU_OPTION_no_kvm:
-                fprintf(stderr, "Emulation temporarily broken\n");
-                exit(1);
                 olist = qemu_find_opts("machine");
                 qemu_opts_reset(olist);
                 qemu_opts_parse(olist, "accel=tcg", 0);
-- 
1.7.1

Re: [PATCH 07/20] qemu-kvm: Clean up kvm_update_interrupt_request

[Avi Kivity]

On 05/27/2011 03:19 PM, Jan Kiszka wrote:
> Most tests in kvm_update_interrupt_request are unneeded today:
>   - env argument is always non-NULL (caller references it as well)
>   - current_env must have been created when we get here
>   - env->thread can't be zero (initialized early during cpu creation)
>
> So simply avoid self signaling and multiple kicks, drop the rest.
>

Why must there be a current_env?

Although the logic you remove seems to be meaningless.  It folds back to 
the new code whether current_env is set or not.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 08/20] qemu-kvm: Replace kvm_update_interrupt_request with qemu_cpu_kick

[Avi Kivity]

On 05/27/2011 03:19 PM, Jan Kiszka wrote:
> Activate the iothread version of qemu_cpu_kick. We just need to
> initialize the yet unused CPUState::halt_cond for it.
>
> This finally obsoletes kvm_update_interrupt_request, so drop it.
>

void qemu_cpu_kick(void *_env)
{
     CPUState *env = _env;

     qemu_cond_broadcast(env->halt_cond);
     if (!env->thread_kicked) {
         qemu_cpu_kick_thread(env);
         env->thread_kicked = true;
     }
}

Seems to have redundancies - we're both signalling a condition variable 
and sending a signal.




-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 00/20] qemu-kvm: Cleanup and switch to upstream - The Season Final

[Avi Kivity]

On 05/27/2011 03:19 PM, Jan Kiszka wrote:
> With this series applied, we are finally at a level of almost zero
> redundancy between QEMU upstream and the qemu-kvm tree. The last major
> duplication to be removed is the original io-thread implementation and
> everything related to it:
>   - locking
>   - vcpu wakeup/kicking as well as suspend/resume
>   - on-vcpu execution
>   - io-thread loop
>   - vcpu thread loop
>
> The approach taken here is similar to what was done to morph KVM core
> functions into upstream code: First all to be converted code is moved
> over into cpus.c, then qemu-kvm's functions are gradually changed and
> replaced with the upstream versions.
>
> This means that we have to enable CONFIG_IOTHREAD which is so far
> incompatible with qemu-kvm. This incompatibility causes a temporary
> breakage of the TCG mode early in the series, but it is healed again
> with the last patch applied.
>
> To make it clear: Even with all this applied, there is still a lot to
> do to turn upstream QEMU into the primary KVM platform. We need to
>   - rework in-kernel IOAPIC/PIC/APIC/PIT support, namely
>      - proper qdev modeling
>      - new MSI hooking architecture (half-done)
>      - VAPIC support (haven't looked at details yet, maybe just cleanups)
>   - prepare device assignment for upstream (VFIO and/or KVM-based)
>   - clean up remaining small deltas (including posix-aio-compat...)
>   - get rid of legacy command line interfaces (e.g. via deprecation
>     warning in release X and removal in X+n, n>= 1)
>
> But then we are really done and can all retire. ;)
>
> In the meantime, please review/merge these bits.

Looks good.  I really appreciate this effort, it's removing a huge thorn 
in our feet.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 07/20] qemu-kvm: Clean up kvm_update_interrupt_request

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

On 2011-05-29 18:16, Avi Kivity wrote:
> On 05/27/2011 03:19 PM, Jan Kiszka wrote:
>> Most tests in kvm_update_interrupt_request are unneeded today:
>>   - env argument is always non-NULL (caller references it as well)
>>   - current_env must have been created when we get here
>>   - env->thread can't be zero (initialized early during cpu creation)
>>
>> So simply avoid self signaling and multiple kicks, drop the rest.
>>
> 
> Why must there be a current_env?
> 
> Although the logic you remove seems to be meaningless.  It folds back to
> the new code whether current_env is set or not.

Are you commenting on the removed code or on the new one now?

I assume to old complex current_env check was once added to work around
initialization ordering problems. I do not see them in the way qemu
starts up, but I also do not have a clear picture of the original issues
anymore.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [PATCH 07/20] qemu-kvm: Clean up kvm_update_interrupt_request

[Avi Kivity]

On 05/29/2011 07:37 PM, Jan Kiszka wrote:
> On 2011-05-29 18:16, Avi Kivity wrote:
> >  On 05/27/2011 03:19 PM, Jan Kiszka wrote:
> >>  Most tests in kvm_update_interrupt_request are unneeded today:
> >>    - env argument is always non-NULL (caller references it as well)
> >>    - current_env must have been created when we get here
> >>    - env->thread can't be zero (initialized early during cpu creation)
> >>
> >>  So simply avoid self signaling and multiple kicks, drop the rest.
> >>
> >
> >  Why must there be a current_env?
> >
> >  Although the logic you remove seems to be meaningless.  It folds back to
> >  the new code whether current_env is set or not.
>
> Are you commenting on the removed code or on the new one now?

The old code - it seems dead already.

> I assume to old complex current_env check was once added to work around
> initialization ordering problems. I do not see them in the way qemu
> starts up, but I also do not have a clear picture of the original issues
> anymore.

Yeah, it was probably a make-it-work hack.  I don't think we have to 
worry about anything there now, though.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 08/20] qemu-kvm: Replace kvm_update_interrupt_request with qemu_cpu_kick

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On 2011-05-29 18:19, Avi Kivity wrote:
> On 05/27/2011 03:19 PM, Jan Kiszka wrote:
>> Activate the iothread version of qemu_cpu_kick. We just need to
>> initialize the yet unused CPUState::halt_cond for it.
>>
>> This finally obsoletes kvm_update_interrupt_request, so drop it.
>>
> 
> void qemu_cpu_kick(void *_env)
> {
>     CPUState *env = _env;
> 
>     qemu_cond_broadcast(env->halt_cond);
>     if (!env->thread_kicked) {
>         qemu_cpu_kick_thread(env);
>         env->thread_kicked = true;
>     }
> }
> 
> Seems to have redundancies - we're both signalling a condition variable
> and sending a signal.
> 

The target may block on the halt condition or run in guest mode. I don't
think we can (and should) try to find out which wakeup call is sufficient.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [PATCH 08/20] qemu-kvm: Replace kvm_update_interrupt_request with qemu_cpu_kick

[Avi Kivity]

On 05/29/2011 07:41 PM, Jan Kiszka wrote:
> >
> >  void qemu_cpu_kick(void *_env)
> >  {
> >      CPUState *env = _env;
> >
> >      qemu_cond_broadcast(env->halt_cond);
> >      if (!env->thread_kicked) {
> >          qemu_cpu_kick_thread(env);
> >          env->thread_kicked = true;
> >      }
> >  }
> >
> >  Seems to have redundancies - we're both signalling a condition variable
> >  and sending a signal.
> >
>
> The target may block on the halt condition or run in guest mode. I don't
> think we can (and should) try to find out which wakeup call is sufficient.

What we could do is make both waiters wait for the same event.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 08/20] qemu-kvm: Replace kvm_update_interrupt_request with qemu_cpu_kick

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1006 bytes --]

On 2011-05-29 18:44, Avi Kivity wrote:
> On 05/29/2011 07:41 PM, Jan Kiszka wrote:
>> >
>> >  void qemu_cpu_kick(void *_env)
>> >  {
>> >      CPUState *env = _env;
>> >
>> >      qemu_cond_broadcast(env->halt_cond);
>> >      if (!env->thread_kicked) {
>> >          qemu_cpu_kick_thread(env);
>> >          env->thread_kicked = true;
>> >      }
>> >  }
>> >
>> >  Seems to have redundancies - we're both signalling a condition
>> variable
>> >  and sending a signal.
>> >
>>
>> The target may block on the halt condition or run in guest mode. I don't
>> think we can (and should) try to find out which wakeup call is
>> sufficient.
> 
> What we could do is make both waiters wait for the same event.

That may only work if we are able to switch halt_cond to a signal-based
mechanism. As usual, a portability issue. I don't think there is much to
win /wrt performance (cond_broadcast is fast). So the question is if
code complexity could actually benefit from this.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]

Re: [PATCH 08/20] qemu-kvm: Replace kvm_update_interrupt_request with qemu_cpu_kick

[Avi Kivity]

On 05/29/2011 08:55 PM, Jan Kiszka wrote:
> >  What we could do is make both waiters wait for the same event.
>
> That may only work if we are able to switch halt_cond to a signal-based
> mechanism. As usual, a portability issue. I don't think there is much to
> win /wrt performance (cond_broadcast is fast). So the question is if
> code complexity could actually benefit from this.
>

Unlikely; kicks should be rare these days.

-- 
error compiling committee.c: too many arguments to function

Re: [PATCH 00/20] qemu-kvm: Cleanup and switch to upstream - The Season Final

[Marcelo Tosatti]

On Fri, May 27, 2011 at 02:19:04PM +0200, Jan Kiszka wrote:
> With this series applied, we are finally at a level of almost zero
> redundancy between QEMU upstream and the qemu-kvm tree. The last major
> duplication to be removed is the original io-thread implementation and
> everything related to it:
>  - locking
>  - vcpu wakeup/kicking as well as suspend/resume
>  - on-vcpu execution
>  - io-thread loop
>  - vcpu thread loop
> 
> The approach taken here is similar to what was done to morph KVM core
> functions into upstream code: First all to be converted code is moved
> over into cpus.c, then qemu-kvm's functions are gradually changed and
> replaced with the upstream versions.
> 
> This means that we have to enable CONFIG_IOTHREAD which is so far
> incompatible with qemu-kvm. This incompatibility causes a temporary
> breakage of the TCG mode early in the series, but it is healed again
> with the last patch applied.
> 
> To make it clear: Even with all this applied, there is still a lot to
> do to turn upstream QEMU into the primary KVM platform. We need to
>  - rework in-kernel IOAPIC/PIC/APIC/PIT support, namely
>     - proper qdev modeling
>     - new MSI hooking architecture (half-done)
>     - VAPIC support (haven't looked at details yet, maybe just cleanups)
>  - prepare device assignment for upstream (VFIO and/or KVM-based)
>  - clean up remaining small deltas (including posix-aio-compat...)
>  - get rid of legacy command line interfaces (e.g. via deprecation
>    warning in release X and removal in X+n, n >= 1)
> 
> But then we are really done and can all retire. ;)
> 
> In the meantime, please review/merge these bits.

Applied, thanks.