[PATCH v1 0/2] accel/kvm: notify management of vCPU run errors

[PATCH v1 0/2] accel/kvm: notify management of vCPU run errors

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Today, when a KVM vCPU exits with KVM_EXIT_INTERNAL_ERROR,
KVM_EXIT_UNKNOWN, or a failed KVM_RUN ioctl, QEMU logs to stderr and
calls vm_stop(RUN_STATE_INTERNAL_ERROR). The management layer then only
sees the derived STOP event; it cannot tell which vCPU failed nor obtain
the kernel suberror / extra data. This gap is even called out by a
FIXME in kvm_handle_internal_error().

This series adds a KVM_VCPU_ERROR QMP event, emitted before the VM is
stopped, that carries the failing vCPU (cpu-index, qom-path), an error
category (KvmVcpuErrorReason) and, for KVM_EXIT_INTERNAL_ERROR, the
kernel suberror and extra data words.

  Patch 1 defines the event and the KvmVcpuErrorReason enum in
          qapi/run-state.json.
  Patch 2 wires it up at the three unrecoverable-exit points in
          accel/kvm/kvm-all.c and removes the FIXME.

Tao Cui (2):
  qapi: add KVM_VCPU_ERROR event
  accel/kvm: emit KVM_VCPU_ERROR on unrecoverable vCPU exits

 accel/kvm/kvm-all.c | 46 +++++++++++++++++++++++++++++++++---
 qapi/run-state.json | 57 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 100 insertions(+), 3 deletions(-)

-- 
2.43.0

[PATCH v1 1/2] qapi: add KVM_VCPU_ERROR event

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

When a KVM vCPU exits with an unrecoverable error (KVM_EXIT_INTERNAL_ERROR,
KVM_EXIT_UNKNOWN, or a failed KVM_RUN ioctl), QEMU currently only logs to
stderr and stops the VM with run state "internal-error". The management
layer must infer what happened from the subsequent STOP event, with no
indication of which vCPU failed or why.

Add a KVM_VCPU_ERROR event, emitted before the VM is stopped, that carries
the failing vCPU (cpu-index, qom-path), an error category
(KvmVcpuErrorReason) and, for KVM_EXIT_INTERNAL_ERROR, the kernel suberror
and extra data words.

Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 qapi/run-state.json | 57 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/qapi/run-state.json b/qapi/run-state.json
index a5771ad468..c9993ad9ac 100644
--- a/qapi/run-state.json
+++ b/qapi/run-state.json
@@ -732,6 +732,63 @@
   'data': { 'action-required': 'bool',
             'recursive': 'bool'} }
 
+##
+# @KvmVcpuErrorReason:
+#
+# Categorizes a KVM vCPU run error reported through @KVM_VCPU_ERROR.
+#
+# @ioctl-failed: the KVM_RUN ioctl failed; see stderr for the errno.
+#
+# @unknown: KVM_EXIT_UNKNOWN; the hardware exit reason is logged to
+#     stderr.
+#
+# @internal-error: KVM_EXIT_INTERNAL_ERROR; see @suberror and
+#     @exit-data for details.
+#
+# Since: 11.2
+##
+{ 'enum': 'KvmVcpuErrorReason',
+  'data': [ 'ioctl-failed', 'unknown', 'internal-error' ] }
+
+##
+# @KVM_VCPU_ERROR:
+#
+# Emitted when a KVM vCPU exits with an unrecoverable error, before
+# the VM is stopped with run state ``internal-error``.  Lets the
+# management layer learn which vCPU failed and why, instead of
+# inferring it from the subsequent STOP event.
+#
+# @cpu-index: index of the vCPU that failed.
+#
+# @qom-path: the QOM path of the vCPU that failed.
+#
+# @reason: error category.
+#
+# @suberror: KVM internal suberror.  Present only when @reason is
+#     @internal-error.
+#
+# @exit-data: extra KVM data words.  Present only when @reason is
+#     @internal-error and the kernel provided extra data.
+#
+# Since: 11.2
+#
+# .. qmp-example::
+#
+#     <- { "event": "KVM_VCPU_ERROR",
+#          "data": { "cpu-index": 0,
+#                    "qom-path": "/machine/unattached/device[0]",
+#                    "reason": "internal-error",
+#                    "suberror": 1,
+#                    "exit-data": [ 0 ] },
+#          "timestamp": { "seconds": 1267061043, "microseconds": 959568 } }
+##
+{ 'event': 'KVM_VCPU_ERROR',
+  'data': { 'cpu-index': 'int',
+            'qom-path': 'str',
+            'reason': 'KvmVcpuErrorReason',
+            '*suberror': 'int',
+            '*exit-data': ['uint64'] } }
+
 ##
 # @NotifyVmexitOption:
 #
-- 
2.43.0

[PATCH v1 2/2] accel/kvm: emit KVM_VCPU_ERROR on unrecoverable vCPU exits

[Tao Cui]

From: Tao Cui <cuitao@kylinos.cn>

Emit the KVM_VCPU_ERROR event at the three points where a vCPU run
becomes unrecoverable:

  - KVM_RUN ioctl failure           (reason "ioctl-failed")
  - KVM_EXIT_UNKNOWN                (reason "unknown")
  - KVM_EXIT_INTERNAL_ERROR         (reason "internal-error",
                                     with suberror and exit-data)

The KVM_RUN loop in kvm_cpu_exec() runs without the BQL, but QMP event
emission goes through the monitor and requires it; take and release the
lock around the send, mirroring the KVM_SYSTEM_EVENT_CRASH path.

This resolves the long-standing FIXME in kvm_handle_internal_error().

Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
 accel/kvm/kvm-all.c | 46 ++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 43 insertions(+), 3 deletions(-)

diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
index 5d55cb45cf..4f68e3b82b 100644
--- a/accel/kvm/kvm-all.c
+++ b/accel/kvm/kvm-all.c
@@ -46,6 +46,7 @@
 #include "qapi/visitor.h"
 #include "qapi/qapi-types-common.h"
 #include "qapi/qapi-visit-common.h"
+#include "qapi/qapi-events-run-state.h"
 #include "system/reset.h"
 #include "qemu/guest-random.h"
 #include "system/hw_accel.h"
@@ -3140,6 +3141,38 @@ static void kvm_handle_io(uint16_t port, MemTxAttrs attrs, void *data, int direc
     }
 }
 
+/*
+ * Notify the management layer that a vCPU hit an unrecoverable KVM run
+ * error.  The KVM_RUN loop in kvm_cpu_exec() runs without the BQL, but
+ * emitting a QMP event goes through the monitor and therefore requires
+ * it; take/release the lock around the send.
+ */
+static void kvm_emit_vcpu_error(CPUState *cpu, KvmVcpuErrorReason reason,
+                                bool has_suberror, int64_t suberror,
+                                uint32_t ndata, const uint64_t *data)
+{
+    g_autofree char *qom_path = object_get_canonical_path(OBJECT(cpu));
+    uint64List *exit_data = NULL;
+    uint64List **tail = &exit_data;
+    uint32_t i;
+
+    for (i = 0; i < ndata; i++) {
+        uint64List *node = g_new(uint64List, 1);
+        node->value = data[i];
+        node->next = NULL;
+        *tail = node;
+        tail = &node->next;
+    }
+
+    bql_lock();
+    qapi_event_send_kvm_vcpu_error(cpu->cpu_index, qom_path, reason,
+                                   has_suberror, suberror,
+                                   !!ndata, exit_data);
+    bql_unlock();
+
+    qapi_free_uint64List(exit_data);
+}
+
 static int kvm_handle_internal_error(CPUState *cpu, struct kvm_run *run)
 {
     int i;
@@ -3151,6 +3184,12 @@ static int kvm_handle_internal_error(CPUState *cpu, struct kvm_run *run)
         fprintf(stderr, "extra data[%d]: 0x%016"PRIx64"\n",
                 i, (uint64_t)run->internal.data[i]);
     }
+
+    kvm_emit_vcpu_error(cpu, KVM_VCPU_ERROR_REASON_INTERNAL_ERROR,
+                        true, run->internal.suberror,
+                        run->internal.ndata,
+                        (const uint64_t *)run->internal.data);
+
     if (run->internal.suberror == KVM_INTERNAL_ERROR_EMULATION) {
         fprintf(stderr, "emulation failure\n");
         if (!kvm_arch_stop_on_emulation_error(cpu)) {
@@ -3158,9 +3197,6 @@ static int kvm_handle_internal_error(CPUState *cpu, struct kvm_run *run)
             return EXCP_INTERRUPT;
         }
     }
-    /* FIXME: Should trigger a qmp message to let management know
-     * something went wrong.
-     */
     return -1;
 }
 
@@ -3508,6 +3544,8 @@ int kvm_cpu_exec(CPUState *cpu)
                             "secondary threads offline.\n");
                 }
 #endif
+                kvm_emit_vcpu_error(cpu, KVM_VCPU_ERROR_REASON_IOCTL_FAILED,
+                                    false, 0, 0, NULL);
                 ret = -1;
                 break;
             }
@@ -3543,6 +3581,8 @@ int kvm_cpu_exec(CPUState *cpu)
         case KVM_EXIT_UNKNOWN:
             fprintf(stderr, "KVM: unknown exit, hardware reason %" PRIx64 "\n",
                     (uint64_t)run->hw.hardware_exit_reason);
+            kvm_emit_vcpu_error(cpu, KVM_VCPU_ERROR_REASON_UNKNOWN,
+                                false, 0, 0, NULL);
             ret = -1;
             break;
         case KVM_EXIT_INTERNAL_ERROR:
-- 
2.43.0