[PATCH 0/3] add MEMORY_FAILURE event

[PATCH 0/3] add MEMORY_FAILURE event

[zhenwei pi]

Although QEMU could catch signal BUS to handle hardware memory
corrupted event, sadly, QEMU just prints a little log and try to fix
it silently.

In these patches, introduce a 'MEMORY_FAILURE' event with 4 detailed
actions of QEMU, then uplayer could know what situaction QEMU hit and
did. And further step we can do: if a host server hits a 'hypervisor-ignore'
or 'guest-mce', scheduler could migrate VM to another host; if hitting
'hypervisor-stop' or 'guest-triple-fault', scheduler could select other
healthy servers to launch VM.

zhenwei pi (3):
  target-i386: seperate MCIP & MCE_MASK error reason
  iqapi/run-state.json: introduce memory failure event
  target-i386: post memory failure event to uplayer

 qapi/run-state.json  | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 target/i386/helper.c | 30 +++++++++++++++++++++++-------
 target/i386/kvm.c    |  5 ++++-
 3 files changed, 73 insertions(+), 8 deletions(-)

-- 
2.11.0

[PATCH 1/3] target-i386: seperate MCIP & MCE_MASK error reason

[zhenwei pi]

Previously we can only get a simple string "Triple fault" in qemu
log. Add detailed message for the two reasons to describe why qemu
has to reset the guest.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
 target/i386/helper.c | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/target/i386/helper.c b/target/i386/helper.c
index 70be53e2c3..0c7fd32491 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -857,6 +857,8 @@ static void do_inject_x86_mce(CPUState *cs, run_on_cpu_data data)
     X86CPU *cpu = X86_CPU(cs);
     CPUX86State *cenv = &cpu->env;
     uint64_t *banks = cenv->mce_banks + 4 * params->bank;
+    char msg[64];
+    bool need_reset = false;
 
     cpu_synchronize_state(cs);
 
@@ -894,16 +896,25 @@ static void do_inject_x86_mce(CPUState *cs, run_on_cpu_data data)
             return;
         }
 
-        if ((cenv->mcg_status & MCG_STATUS_MCIP) ||
-            !(cenv->cr[4] & CR4_MCE_MASK)) {
-            monitor_printf(params->mon,
-                           "CPU %d: Previous MCE still in progress, raising"
-                           " triple fault\n",
-                           cs->cpu_index);
-            qemu_log_mask(CPU_LOG_RESET, "Triple fault\n");
+        if (cenv->mcg_status & MCG_STATUS_MCIP) {
+            need_reset = true;
+            snprintf(msg, sizeof(msg), "CPU %d: Previous MCE still in progress,"
+                     " raising triple fault", cs->cpu_index);
+        }
+
+        if (!(cenv->cr[4] & CR4_MCE_MASK)) {
+            need_reset = true;
+            snprintf(msg, sizeof(msg), "CPU %d: MCE capability is not enabled,"
+                     " raising triple fault", cs->cpu_index);
+        }
+
+        if (need_reset) {
+            monitor_printf(params->mon, "%s", msg);
+            qemu_log_mask(CPU_LOG_RESET, "%s\n", msg);
             qemu_system_reset_request(SHUTDOWN_CAUSE_GUEST_RESET);
             return;
         }
+
         if (banks[1] & MCI_STATUS_VAL) {
             params->status |= MCI_STATUS_OVER;
         }
-- 
2.11.0

[PATCH 2/3] iqapi/run-state.json: introduce memory failure event

[zhenwei pi]

Introduce 4 memory failure events for a guest. Then uplayer could
know when/why/what happened to a guest during hitting a hardware
memory failure.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
 qapi/run-state.json | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/qapi/run-state.json b/qapi/run-state.json
index 7cc9f96a5b..fdc39ce262 100644
--- a/qapi/run-state.json
+++ b/qapi/run-state.json
@@ -475,3 +475,49 @@
            'psw-mask': 'uint64',
            'psw-addr': 'uint64',
            'reason': 'S390CrashReason' } }
+
+##
+# @MEMORY_FAILURE:
+#
+# Emitted when a memory failure occurs on host side.
+#
+# @action: action that has been taken. action is defined as @MemoryFailureAction.
+#
+# Since: 5.2
+#
+# Example:
+#
+# <- { "event": "MEMORY_FAILURE",
+#      "data": { "action": "guest-mce" } }
+#
+##
+{ 'event': 'MEMORY_FAILURE',
+  'data': { 'action': 'MemoryFailureAction'} }
+
+##
+# @MemoryFailureAction:
+#
+# Host memory failure occurs, handled by QEMU.
+#
+# @hypervisor-ignore: action optional memory failure at QEMU process
+#                     addressspace (none PC-RAM), QEMU could ignore this
+#                     hardware memory failure.
+#
+# @hypervisor-stop: action required memory failure at QEMU process address
+#                   space (none PC-RAM), QEMU has to stop itself.
+#
+# @guest-mce: action required memory failure at PC-RAM, and guest enables MCE
+#             handling, QEMU injects MCE to guest.
+#
+# @guest-triple-fault: action required memory failure at PC-RAM, but guest does
+#                      not enable MCE handling. QEMU raises triple fault and
+#                      shutdown/reset. Also see detailed info in QEMU log.
+#
+# Since: 5.2
+#
+##
+{ 'enum': 'MemoryFailureAction',
+  'data': [ 'hypervisor-ignore',
+            'hypervisor-stop',
+            'guest-mce',
+            'guest-triple-fault' ] }
-- 
2.11.0

Re: [PATCH 2/3] iqapi/run-state.json: introduce memory failure event

[Peter Maydell]

On Mon, 14 Sep 2020 at 14:53, zhenwei pi <pizhenwei@bytedance.com> wrote:
>
> Introduce 4 memory failure events for a guest. Then uplayer could
> know when/why/what happened to a guest during hitting a hardware
> memory failure.
>
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
> +##
> +# @MemoryFailureAction:
> +#
> +# Host memory failure occurs, handled by QEMU.
> +#
> +# @hypervisor-ignore: action optional memory failure at QEMU process
> +#                     addressspace (none PC-RAM), QEMU could ignore this
> +#                     hardware memory failure.
> +#
> +# @hypervisor-stop: action required memory failure at QEMU process address
> +#                   space (none PC-RAM), QEMU has to stop itself.

I'm not entirely clear what the descriptions here are trying to say.
These would be for memory failure events which are reported by the
host and which are not in guest RAM but are in the memory QEMU itself
is using ? ("PC-RAM" is a bit x86-specific.)

> +#
> +# @guest-mce: action required memory failure at PC-RAM, and guest enables MCE
> +#             handling, QEMU injects MCE to guest.
> +#
> +# @guest-triple-fault: action required memory failure at PC-RAM, but guest does
> +#                      not enable MCE handling. QEMU raises triple fault and
> +#                      shutdown/reset. Also see detailed info in QEMU log.

"triple fault" sounds rather x86-specific; other architectures
also have support for host memory failure notifications, so we
should design the QAPI events to have architecture-neutral
definitions and descriptions.

I think the four cases you're trying to distinguish here are:
 (1) action-optional memory failure in memory used by the hypervisor
     (which QEMU has ignored other than to report this event)
 (2) action-required memory failure in memory used by the hypervisor
     (QEMU is stopping)
 (3) action-required memory failure in guest memory, which QEMU
     has reported to the guest
 (4) action-required memory failure in guest memory, but the
     guest OS does not support a mechanism for reporting it

Is that right?

Anyway, I think we should try to find names for the failure
types that are not x86-specific.

thanks
-- PMM

Re: [External] Re: [PATCH 2/3] iqapi/run-state.json: introduce memory failure event

[zhenwei pi]

On 9/21/20 8:48 PM, Peter Maydell wrote:
> On Mon, 14 Sep 2020 at 14:53, zhenwei pi <pizhenwei@bytedance.com> wrote:
>>
>> Introduce 4 memory failure events for a guest. Then uplayer could
>> know when/why/what happened to a guest during hitting a hardware
>> memory failure.
>>
>> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
>> ---
>> +##
>> +# @MemoryFailureAction:
>> +#
>> +# Host memory failure occurs, handled by QEMU.
>> +#
>> +# @hypervisor-ignore: action optional memory failure at QEMU process
>> +#                     addressspace (none PC-RAM), QEMU could ignore this
>> +#                     hardware memory failure.
>> +#
>> +# @hypervisor-stop: action required memory failure at QEMU process address
>> +#                   space (none PC-RAM), QEMU has to stop itself.
> 
> I'm not entirely clear what the descriptions here are trying to say.
> These would be for memory failure events which are reported by the
> host and which are not in guest RAM but are in the memory QEMU itself
> is using ? ("PC-RAM" is a bit x86-specific.)
> 
>> +#
>> +# @guest-mce: action required memory failure at PC-RAM, and guest enables MCE
>> +#             handling, QEMU injects MCE to guest.
>> +#
>> +# @guest-triple-fault: action required memory failure at PC-RAM, but guest does
>> +#                      not enable MCE handling. QEMU raises triple fault and
>> +#                      shutdown/reset. Also see detailed info in QEMU log.
> 
> "triple fault" sounds rather x86-specific; other architectures
> also have support for host memory failure notifications, so we
> should design the QAPI events to have architecture-neutral
> definitions and descriptions.
> 
> I think the four cases you're trying to distinguish here are:
>   (1) action-optional memory failure in memory used by the hypervisor
>       (which QEMU has ignored other than to report this event)
>   (2) action-required memory failure in memory used by the hypervisor
>       (QEMU is stopping)
>   (3) action-required memory failure in guest memory, which QEMU
>       has reported to the guest
>   (4) action-required memory failure in guest memory, but the
>       guest OS does not support a mechanism for reporting it
> 
> Is that right?
> 
> Anyway, I think we should try to find names for the failure
> types that are not x86-specific.
> 
> thanks
> -- PMM
> 
Right, to make architecture-neutral, how about these changes:
'PC-RAM' -> 'guest-memory'
'guest-mce' -> 'guest-mce-inject'
'guest-triple-fault' -> 'guest-mce-fault'

-- 
zhenwei pi

Re: [External] Re: [PATCH 2/3] iqapi/run-state.json: introduce memory failure event

[Paolo Bonzini]

On 21/09/20 15:10, zhenwei pi wrote:
>>
> Right, to make architecture-neutral, how about these changes:
> 'PC-RAM' -> 'guest-memory'
> 'guest-mce' -> 'guest-mce-inject'
> 'guest-triple-fault' -> 'guest-mce-fault'

Perhaps we should have three fields

1) recipient: 'hypervisor' or 'guest'

2) action: 'ignore', 'inject', 'fatal'

3) kind: 'action-optional' or 'action-required'

And possibly:

4) recursive: true or false

On x86 "recursive" would be set if MCIP=1.

Paolo

[PATCH 3/3] target-i386: post memory failure event to uplayer

[zhenwei pi]

Post memory failure event to uplayer to handle hardware memory
corrupted event. Rather than simple QEMU log, QEMU could report more
effective message to uplayer. For example, guest crashes by MCE,
selecting another host server is a better choice.

Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
---
 target/i386/helper.c | 5 +++++
 target/i386/kvm.c    | 5 ++++-
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/target/i386/helper.c b/target/i386/helper.c
index 0c7fd32491..f7b2dbeec8 100644
--- a/target/i386/helper.c
+++ b/target/i386/helper.c
@@ -18,6 +18,7 @@
  */
 
 #include "qemu/osdep.h"
+#include "qapi/qapi-events-run-state.h"
 #include "cpu.h"
 #include "exec/exec-all.h"
 #include "qemu/qemu-print.h"
@@ -897,6 +898,8 @@ static void do_inject_x86_mce(CPUState *cs, run_on_cpu_data data)
         }
 
         if (cenv->mcg_status & MCG_STATUS_MCIP) {
+            qapi_event_send_memory_failure(
+                 MEMORY_FAILURE_ACTION_GUEST_TRIPLE_FAULT);
             need_reset = true;
             snprintf(msg, sizeof(msg), "CPU %d: Previous MCE still in progress,"
                      " raising triple fault", cs->cpu_index);
@@ -934,6 +937,8 @@ static void do_inject_x86_mce(CPUState *cs, run_on_cpu_data data)
     } else {
         banks[1] |= MCI_STATUS_OVER;
     }
+
+    qapi_event_send_memory_failure(MEMORY_FAILURE_ACTION_GUEST_MCE);
 }
 
 void cpu_x86_inject_mce(Monitor *mon, X86CPU *cpu, int bank,
diff --git a/target/i386/kvm.c b/target/i386/kvm.c
index 205b68bc0c..63a585cc64 100644
--- a/target/i386/kvm.c
+++ b/target/i386/kvm.c
@@ -14,6 +14,7 @@
 
 #include "qemu/osdep.h"
 #include "qapi/error.h"
+#include "qapi/qapi-events-run-state.h"
 #include <sys/ioctl.h>
 #include <sys/utsname.h>
 
@@ -577,6 +578,7 @@ static void kvm_mce_inject(X86CPU *cpu, hwaddr paddr, int code)
 
 static void hardware_memory_error(void *host_addr)
 {
+    qapi_event_send_memory_failure(MEMORY_FAILURE_ACTION_HYPERVISOR_STOP);
     error_report("QEMU got Hardware memory error at addr %p", host_addr);
     exit(1);
 }
@@ -631,7 +633,8 @@ void kvm_arch_on_sigbus_vcpu(CPUState *c, int code, void *addr)
         hardware_memory_error(addr);
     }
 
-    /* Hope we are lucky for AO MCE */
+    /* Hope we are lucky for AO MCE, just notify a event */
+    qapi_event_send_memory_failure(MEMORY_FAILURE_ACTION_HYPERVISOR_IGNORE);
 }
 
 static void kvm_reset_exception(CPUX86State *env)
-- 
2.11.0

ping: [PATCH 0/3] add MEMORY_FAILURE event

[zhenwei pi]

Hi,

A patchset about handling 'MCE' might have been ignored, can anyone tell 
me whether the purpose is reasonable?

https://patchwork.kernel.org/cover/11773795/

On 9/14/20 9:43 PM, zhenwei pi wrote:
> Although QEMU could catch signal BUS to handle hardware memory
> corrupted event, sadly, QEMU just prints a little log and try to fix
> it silently.
> 
> In these patches, introduce a 'MEMORY_FAILURE' event with 4 detailed
> actions of QEMU, then uplayer could know what situaction QEMU hit and
> did. And further step we can do: if a host server hits a 'hypervisor-ignore'
> or 'guest-mce', scheduler could migrate VM to another host; if hitting
> 'hypervisor-stop' or 'guest-triple-fault', scheduler could select other
> healthy servers to launch VM.
> 
> zhenwei pi (3):
>    target-i386: seperate MCIP & MCE_MASK error reason
>    iqapi/run-state.json: introduce memory failure event
>    target-i386: post memory failure event to uplayer
> 
>   qapi/run-state.json  | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>   target/i386/helper.c | 30 +++++++++++++++++++++++-------
>   target/i386/kvm.c    |  5 ++++-
>   3 files changed, 73 insertions(+), 8 deletions(-)
> 

-- 
zhenwei pi

Re: ping: [PATCH 0/3] add MEMORY_FAILURE event

[Paolo Bonzini]

On 21/09/20 04:22, zhenwei pi wrote:
> Hi,
> 
> A patchset about handling 'MCE' might have been ignored, can anyone tell
> me whether the purpose is reasonable?
> 
> https://patchwork.kernel.org/cover/11773795/

Yes, it's very useful.  Just one thing, "guest-mce" can be reported for
both AR and AO faults.  Is it worth adding a 'type' field to distinguish
the two?

Paolo

> On 9/14/20 9:43 PM, zhenwei pi wrote:
>> Although QEMU could catch signal BUS to handle hardware memory
>> corrupted event, sadly, QEMU just prints a little log and try to fix
>> it silently.
>>
>> In these patches, introduce a 'MEMORY_FAILURE' event with 4 detailed
>> actions of QEMU, then uplayer could know what situaction QEMU hit and
>> did. And further step we can do: if a host server hits a
>> 'hypervisor-ignore'
>> or 'guest-mce', scheduler could migrate VM to another host; if hitting
>> 'hypervisor-stop' or 'guest-triple-fault', scheduler could select other
>> healthy servers to launch VM.
>>
>> zhenwei pi (3):
>>    target-i386: seperate MCIP & MCE_MASK error reason
>>    iqapi/run-state.json: introduce memory failure event
>>    target-i386: post memory failure event to uplayer
>>
>>   qapi/run-state.json  | 46
>> ++++++++++++++++++++++++++++++++++++++++++++++
>>   target/i386/helper.c | 30 +++++++++++++++++++++++-------
>>   target/i386/kvm.c    |  5 ++++-
>>   3 files changed, 73 insertions(+), 8 deletions(-)
>>
> 

Re: [External] Re: ping: [PATCH 0/3] add MEMORY_FAILURE event

[zhenwei pi]

On 9/21/20 8:09 PM, Paolo Bonzini wrote:
> On 21/09/20 04:22, zhenwei pi wrote:
>> Hi,
>>
>> A patchset about handling 'MCE' might have been ignored, can anyone tell
>> me whether the purpose is reasonable?
>>
>> https://patchwork.kernel.org/cover/11773795/
> 
> Yes, it's very useful.  Just one thing, "guest-mce" can be reported for
> both AR and AO faults.  Is it worth adding a 'type' field to distinguish
> the two?
> 
> Paolo
> 
Sure. how about adding a 'flags' of a structure? and a field named 
'action-required' to describe AO or AR?
>> On 9/14/20 9:43 PM, zhenwei pi wrote:
>>> Although QEMU could catch signal BUS to handle hardware memory
>>> corrupted event, sadly, QEMU just prints a little log and try to fix
>>> it silently.
>>>
>>> In these patches, introduce a 'MEMORY_FAILURE' event with 4 detailed
>>> actions of QEMU, then uplayer could know what situaction QEMU hit and
>>> did. And further step we can do: if a host server hits a
>>> 'hypervisor-ignore'
>>> or 'guest-mce', scheduler could migrate VM to another host; if hitting
>>> 'hypervisor-stop' or 'guest-triple-fault', scheduler could select other
>>> healthy servers to launch VM.
>>>
>>> zhenwei pi (3):
>>>     target-i386: seperate MCIP & MCE_MASK error reason
>>>     iqapi/run-state.json: introduce memory failure event
>>>     target-i386: post memory failure event to uplayer
>>>
>>>    qapi/run-state.json  | 46
>>> ++++++++++++++++++++++++++++++++++++++++++++++
>>>    target/i386/helper.c | 30 +++++++++++++++++++++++-------
>>>    target/i386/kvm.c    |  5 ++++-
>>>    3 files changed, 73 insertions(+), 8 deletions(-)
>>>
>>
> 

-- 
zhenwei pi