* [PULL 2/5] os-posix: asynchronous teardown for shutdown on Linux
2022-11-02 11:41 [PULL 0/5] Misc patches for QEMU 7.2 soft freeze Paolo Bonzini
2022-11-02 11:41 ` [PULL 1/5] target/i386: Fix calculation of LOCK NEG eflags Paolo Bonzini
@ 2022-11-02 11:41 ` Paolo Bonzini
2022-11-02 11:41 ` [PULL 3/5] target/i386: Set maximum APIC ID to KVM prior to vCPU creation Paolo Bonzini
` (4 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Paolo Bonzini @ 2022-11-02 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Claudio Imbrenda, Murilo Opsfelder Araujo
From: Claudio Imbrenda <imbrenda@linux.ibm.com>
This patch adds support for asynchronously tearing down a VM on Linux.
When qemu terminates, either naturally or because of a fatal signal,
the VM is torn down. If the VM is huge, it can take a considerable
amount of time for it to be cleaned up. In case of a protected VM, it
might take even longer than a non-protected VM (this is the case on
s390x, for example).
Some users might want to shut down a VM and restart it immediately,
without having to wait. This is especially true if management
infrastructure like libvirt is used.
This patch implements a simple trick on Linux to allow qemu to return
immediately, with the teardown of the VM being performed
asynchronously.
If the new commandline option -async-teardown is used, a new process is
spawned from qemu at startup, using the clone syscall, in such way that
it will share its address space with qemu.The new process will have the
name "cleanup/<QEMU_PID>". It will wait until qemu terminates
completely, and then it will exit itself.
This allows qemu to terminate quickly, without having to wait for the
whole address space to be torn down. The cleanup process will exit
after qemu, so it will be the last user of the address space, and
therefore it will take care of the actual teardown. The cleanup
process will share the same cgroups as qemu, so both memory usage and
cpu time will be accounted properly.
If possible, close_range will be used in the cleanup process to close
all open file descriptors. If it is not available or if it fails, /proc
will be used to determine which file descriptors to close.
If the cleanup process is forcefully killed with SIGKILL before the
main qemu process has terminated completely, the mechanism is defeated
and the teardown will not be asynchronous.
This feature can already be used with libvirt by adding the following
to the XML domain definition to pass the parameter to qemu directly:
<commandline xmlns="http://libvirt.org/schemas/domain/qemu/1.0">
<arg value='-async-teardown'/>
</commandline>
Signed-off-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Reviewed-by: Murilo Opsfelder Araujo <muriloo@linux.ibm.com>
Tested-by: Murilo Opsfelder Araujo <muriloo@linux.ibm.com>
Message-Id: <20220812133453.82671-1-imbrenda@linux.ibm.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
include/qemu/async-teardown.h | 22 +++++
meson.build | 1 +
os-posix.c | 6 ++
qemu-options.hx | 19 +++++
util/async-teardown.c | 150 ++++++++++++++++++++++++++++++++++
util/meson.build | 1 +
6 files changed, 199 insertions(+)
create mode 100644 include/qemu/async-teardown.h
create mode 100644 util/async-teardown.c
diff --git a/include/qemu/async-teardown.h b/include/qemu/async-teardown.h
new file mode 100644
index 000000000000..092e7a37e7af
--- /dev/null
+++ b/include/qemu/async-teardown.h
@@ -0,0 +1,22 @@
+/*
+ * Asynchronous teardown
+ *
+ * Copyright IBM, Corp. 2022
+ *
+ * Authors:
+ * Claudio Imbrenda <imbrenda@linux.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at your
+ * option) any later version. See the COPYING file in the top-level directory.
+ *
+ */
+#ifndef QEMU_ASYNC_TEARDOWN_H
+#define QEMU_ASYNC_TEARDOWN_H
+
+#include "config-host.h"
+
+#ifdef CONFIG_LINUX
+void init_async_teardown(void);
+#endif
+
+#endif
diff --git a/meson.build b/meson.build
index 37737913df6a..17834b3c3def 100644
--- a/meson.build
+++ b/meson.build
@@ -1949,6 +1949,7 @@ if targetos == 'windows'
endif
# has_function
+config_host_data.set('CONFIG_CLOSE_RANGE', cc.has_function('close_range'))
config_host_data.set('CONFIG_ACCEPT4', cc.has_function('accept4'))
config_host_data.set('CONFIG_CLOCK_ADJTIME', cc.has_function('clock_adjtime'))
config_host_data.set('CONFIG_DUP3', cc.has_function('dup3'))
diff --git a/os-posix.c b/os-posix.c
index 321fc4bd13b8..4858650c3e38 100644
--- a/os-posix.c
+++ b/os-posix.c
@@ -39,6 +39,7 @@
#ifdef CONFIG_LINUX
#include <sys/prctl.h>
+#include "qemu/async-teardown.h"
#endif
/*
@@ -150,6 +151,11 @@ int os_parse_cmd_args(int index, const char *optarg)
case QEMU_OPTION_daemonize:
daemonize = 1;
break;
+#if defined(CONFIG_LINUX)
+ case QEMU_OPTION_asyncteardown:
+ init_async_teardown();
+ break;
+#endif
default:
return -1;
}
diff --git a/qemu-options.hx b/qemu-options.hx
index eb38e5dc40bc..e26d1dad3990 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -4774,6 +4774,25 @@ HXCOMM Internal use
DEF("qtest", HAS_ARG, QEMU_OPTION_qtest, "", QEMU_ARCH_ALL)
DEF("qtest-log", HAS_ARG, QEMU_OPTION_qtest_log, "", QEMU_ARCH_ALL)
+#ifdef __linux__
+DEF("async-teardown", 0, QEMU_OPTION_asyncteardown,
+ "-async-teardown enable asynchronous teardown\n",
+ QEMU_ARCH_ALL)
+#endif
+SRST
+``-async-teardown``
+ Enable asynchronous teardown. A new process called "cleanup/<QEMU_PID>"
+ will be created at startup sharing the address space with the main qemu
+ process, using clone. It will wait for the main qemu process to
+ terminate completely, and then exit.
+ This allows qemu to terminate very quickly even if the guest was
+ huge, leaving the teardown of the address space to the cleanup
+ process. Since the cleanup process shares the same cgroups as the
+ main qemu process, accounting is performed correctly. This only
+ works if the cleanup process is not forcefully killed with SIGKILL
+ before the main qemu process has terminated completely.
+ERST
+
DEF("msg", HAS_ARG, QEMU_OPTION_msg,
"-msg [timestamp[=on|off]][,guest-name=[on|off]]\n"
" control error message format\n"
diff --git a/util/async-teardown.c b/util/async-teardown.c
new file mode 100644
index 000000000000..62bfce1b3ca8
--- /dev/null
+++ b/util/async-teardown.c
@@ -0,0 +1,150 @@
+/*
+ * Asynchronous teardown
+ *
+ * Copyright IBM, Corp. 2022
+ *
+ * Authors:
+ * Claudio Imbrenda <imbrenda@linux.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at your
+ * option) any later version. See the COPYING file in the top-level directory.
+ *
+ */
+#include <stdlib.h>
+#include <stdio.h>
+#include <sys/types.h>
+#include <dirent.h>
+#include <sys/prctl.h>
+#include <signal.h>
+#include <sched.h>
+#include <unistd.h>
+
+#include "qemu/osdep.h"
+#include "qemu/async-teardown.h"
+
+#ifdef _SC_THREAD_STACK_MIN
+#define CLONE_STACK_SIZE sysconf(_SC_THREAD_STACK_MIN)
+#else
+#define CLONE_STACK_SIZE 16384
+#endif
+
+static pid_t the_ppid;
+
+/*
+ * Close all open file descriptors.
+ */
+static void close_all_open_fd(void)
+{
+ struct dirent *de;
+ int fd, dfd;
+ DIR *dir;
+
+#ifdef CONFIG_CLOSE_RANGE
+ int r = close_range(0, ~0U, 0);
+ if (!r) {
+ /* Success, no need to try other ways. */
+ return;
+ }
+#endif
+
+ dir = opendir("/proc/self/fd");
+ if (!dir) {
+ /* If /proc is not mounted, there is nothing that can be done. */
+ return;
+ }
+ /* Avoid closing the directory. */
+ dfd = dirfd(dir);
+
+ for (de = readdir(dir); de; de = readdir(dir)) {
+ fd = atoi(de->d_name);
+ if (fd != dfd) {
+ close(fd);
+ }
+ }
+ closedir(dir);
+}
+
+static void hup_handler(int signal)
+{
+ /* Check every second if this process has been reparented. */
+ while (the_ppid == getppid()) {
+ /* sleep() is safe to use in a signal handler. */
+ sleep(1);
+ }
+
+ /* At this point the parent process has terminated completely. */
+ _exit(0);
+}
+
+static int async_teardown_fn(void *arg)
+{
+ struct sigaction sa = { .sa_handler = hup_handler };
+ sigset_t hup_signal;
+ char name[16];
+
+ /* Set a meaningful name for this process. */
+ snprintf(name, 16, "cleanup/%d", the_ppid);
+ prctl(PR_SET_NAME, (unsigned long)name);
+
+ /*
+ * Close all file descriptors that might have been inherited from the
+ * main qemu process when doing clone, needed to make libvirt happy.
+ * Not using close_range for increased compatibility with older kernels.
+ */
+ close_all_open_fd();
+
+ /* Set up a handler for SIGHUP and unblock SIGHUP. */
+ sigaction(SIGHUP, &sa, NULL);
+ sigemptyset(&hup_signal);
+ sigaddset(&hup_signal, SIGHUP);
+ sigprocmask(SIG_UNBLOCK, &hup_signal, NULL);
+
+ /* Ask to receive SIGHUP when the parent dies. */
+ prctl(PR_SET_PDEATHSIG, SIGHUP);
+
+ /*
+ * Sleep forever, unless the parent process has already terminated. The
+ * only interruption can come from the SIGHUP signal, which in normal
+ * operation is received when the parent process dies.
+ */
+ if (the_ppid == getppid()) {
+ pause();
+ }
+
+ /* At this point the parent process has terminated completely. */
+ _exit(0);
+}
+
+/*
+ * Allocate a new stack of a reasonable size, and return a pointer to its top.
+ */
+static void *new_stack_for_clone(void)
+{
+ size_t stack_size = CLONE_STACK_SIZE;
+ char *stack_ptr;
+
+ /* Allocate a new stack and get a pointer to its top. */
+ stack_ptr = qemu_alloc_stack(&stack_size);
+#if !defined(HOST_HPPA)
+ /* The top is at the end of the area, except on HPPA. */
+ stack_ptr += stack_size;
+#endif
+
+ return stack_ptr;
+}
+
+/*
+ * Block all signals, start (clone) a new process sharing the address space
+ * with qemu (CLONE_VM), then restore signals.
+ */
+void init_async_teardown(void)
+{
+ sigset_t all_signals, old_signals;
+
+ the_ppid = getpid();
+
+ sigfillset(&all_signals);
+ sigprocmask(SIG_BLOCK, &all_signals, &old_signals);
+ clone(async_teardown_fn, new_stack_for_clone(), CLONE_VM, NULL);
+ sigprocmask(SIG_SETMASK, &old_signals, NULL);
+}
diff --git a/util/meson.build b/util/meson.build
index 5e282130df85..63acd59bb02c 100644
--- a/util/meson.build
+++ b/util/meson.build
@@ -2,6 +2,7 @@ util_ss.add(files('osdep.c', 'cutils.c', 'unicode.c', 'qemu-timer-common.c'))
if not config_host_data.get('CONFIG_ATOMIC64')
util_ss.add(files('atomic64.c'))
endif
+util_ss.add(when: 'CONFIG_LINUX', if_true: files('async-teardown.c'))
util_ss.add(when: 'CONFIG_POSIX', if_true: files('aio-posix.c'))
util_ss.add(when: 'CONFIG_POSIX', if_true: files('fdmon-poll.c'))
if config_host_data.get('CONFIG_EPOLL_CREATE1')
--
2.38.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PULL 3/5] target/i386: Set maximum APIC ID to KVM prior to vCPU creation
2022-11-02 11:41 [PULL 0/5] Misc patches for QEMU 7.2 soft freeze Paolo Bonzini
2022-11-02 11:41 ` [PULL 1/5] target/i386: Fix calculation of LOCK NEG eflags Paolo Bonzini
2022-11-02 11:41 ` [PULL 2/5] os-posix: asynchronous teardown for shutdown on Linux Paolo Bonzini
@ 2022-11-02 11:41 ` Paolo Bonzini
2022-11-02 11:41 ` [PULL 4/5] util/log: Close per-thread log file on thread termination Paolo Bonzini
` (3 subsequent siblings)
6 siblings, 0 replies; 9+ messages in thread
From: Paolo Bonzini @ 2022-11-02 11:41 UTC (permalink / raw)
To: qemu-devel; +Cc: Zeng Guang, Peter Xu, Michael S . Tsirkin
From: Zeng Guang <guang.zeng@intel.com>
Specify maximum possible APIC ID assigned for current VM session to KVM
prior to the creation of vCPUs. By this setting, KVM can set up VM-scoped
data structure indexed by the APIC ID, e.g. Posted-Interrupt Descriptor
pointer table to support Intel IPI virtualization, with the most optimal
memory footprint.
It can be achieved by calling KVM_ENABLE_CAP for KVM_CAP_MAX_VCPU_ID
capability once KVM has enabled it. Ignoring the return error if KVM
doesn't support this capability yet.
Signed-off-by: Zeng Guang <guang.zeng@intel.com>
Acked-by: Peter Xu <peterx@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Message-Id: <20220825025246.26618-1-guang.zeng@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
hw/i386/x86.c | 4 ++++
target/i386/kvm/kvm-stub.c | 5 +++++
target/i386/kvm/kvm.c | 5 +++++
target/i386/kvm/kvm_i386.h | 2 ++
4 files changed, 16 insertions(+)
diff --git a/hw/i386/x86.c b/hw/i386/x86.c
index bd50a064a36d..78cc131926c8 100644
--- a/hw/i386/x86.c
+++ b/hw/i386/x86.c
@@ -140,6 +140,10 @@ void x86_cpus_init(X86MachineState *x86ms, int default_cpu_version)
exit(EXIT_FAILURE);
}
+ if (kvm_enabled()) {
+ kvm_set_max_apic_id(x86ms->apic_id_limit);
+ }
+
possible_cpus = mc->possible_cpu_arch_ids(ms);
for (i = 0; i < ms->smp.cpus; i++) {
x86_cpu_new(x86ms, possible_cpus->cpus[i].arch_id, &error_fatal);
diff --git a/target/i386/kvm/kvm-stub.c b/target/i386/kvm/kvm-stub.c
index f6e7e4466e1a..e052f1c7b0ef 100644
--- a/target/i386/kvm/kvm-stub.c
+++ b/target/i386/kvm/kvm-stub.c
@@ -44,3 +44,8 @@ bool kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
{
abort();
}
+
+void kvm_set_max_apic_id(uint32_t max_apic_id)
+{
+ return;
+}
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 4df04280890a..a21320937943 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -5723,3 +5723,8 @@ void kvm_arch_accel_class_init(ObjectClass *oc)
"Clock cycles without an event window "
"after which a notification VM exit occurs");
}
+
+void kvm_set_max_apic_id(uint32_t max_apic_id)
+{
+ kvm_vm_enable_cap(kvm_state, KVM_CAP_MAX_VCPU_ID, 0, max_apic_id);
+}
diff --git a/target/i386/kvm/kvm_i386.h b/target/i386/kvm/kvm_i386.h
index b7c38ba2c462..6a5c24e3dc40 100644
--- a/target/i386/kvm/kvm_i386.h
+++ b/target/i386/kvm/kvm_i386.h
@@ -66,4 +66,6 @@ typedef struct kvm_msr_handlers {
bool kvm_filter_msr(KVMState *s, uint32_t msr, QEMURDMSRHandler *rdmsr,
QEMUWRMSRHandler *wrmsr);
+void kvm_set_max_apic_id(uint32_t max_apic_id);
+
#endif
--
2.38.1
^ permalink raw reply related [flat|nested] 9+ messages in thread