[Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Damien Hedde]

Hi all,

This series adds a python framework aiming to provide some ways to do fault
injection in a running vm. In its current state, it allows to easily interact
with memory, change gpios and qom properties.

The framework consists in a python script based on the qmp existing module
which allows to interact with the vm.

The series introduces a QMP command to schedule some virtual-clock-time-based
notifications. The notification is sent back to the python framework and can
be used to build time-driven fault scenario.

Additionaly the series adds some new QMP commands:

Commands are added to read/write memory or memory-mapped registers. Arguments
are similar to the existing [p]memsave commands.

A command is added to set a value to a qdev gpio.

Here's is a simple visual example which inject characters on the uart output
of the zynq platform:
$ # qemu must have been launched with -qmp unix:/tmp/qmpsock,server
$ # create the python framework object
$ import fault_injection
$ inj = fault_injection.FaultInjectionFramework("/tmp/qmpsock", 0)
$
$ # function which display a 'X' to the first uart
$ # it access directly the register using the physical address
$ def cb():
$   inj.write_pmem(0xe0000030,4, 88)
$
$ # schedule the function on a notification in 10s
$ inj.notify(10 * 1000 * 1000 * 1000, cb, True)
$
$ # handle one notification
$ inj.run_once()

The framework has been tested using python 2, on qemu running xilinx_zynq or
virt arm machines.

The series is organised as follows. Patches 1 and 2 adds the memory qmp and
gpio commands. Patch 3 adds the notification mechanism. Patches 4 and 5 add
a python helper module and some documention.

Thanks to the Xilinx's QEMU team who sponsored this work.

Damien Hedde (5):
  introduce [p]mem(read|write) qmp commands
  introduce a qmp command to set gpios
  add qmp time-notify event triggering system
  fault_injection: introduce Python scripting framework
  docs: add fault injection framework documentation

 cpus.c                         | 126 +++++++++++++++
 docs/fault_injection.txt       | 149 ++++++++++++++++++
 monitor/Makefile.objs          |   1 +
 monitor/qmp-cmd-time-notify.c  | 116 ++++++++++++++
 monitor/qmp-cmds.c             |  30 ++++
 monitor/trace-events           |   4 +
 qapi/misc.json                 | 196 +++++++++++++++++++++++
 scripts/qmp/fault_injection.py | 278 +++++++++++++++++++++++++++++++++
 8 files changed, 900 insertions(+)
 create mode 100644 docs/fault_injection.txt
 create mode 100644 monitor/qmp-cmd-time-notify.c
 create mode 100644 scripts/qmp/fault_injection.py

-- 
2.22.0

[Qemu-devel] [RFC PATCH 1/5] introduce [p]mem(read|write) qmp commands

[Damien Hedde]

This introduces memread, memwrite, pmemread and pmemwrite qmp commands.

The memread and memwrite read virtual memory from a given cpu point of
view. If no cpu index is specified, the cpu-id 0 is used.

The pmemread and pmemwrite directly read physical memory.

The data is passed/returned in a list of bytes. The maximum length is set to 8, they can be used to 64bits.

This is based on the work of Frederic Konrad.

Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
---
 cpus.c         | 126 +++++++++++++++++++++++++++++++++++++++++++++++++
 qapi/misc.json | 119 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 245 insertions(+)

diff --git a/cpus.c b/cpus.c
index 1af51b73dd..7aae35c098 100644
--- a/cpus.c
+++ b/cpus.c
@@ -2484,3 +2484,129 @@ void dump_drift_info(void)
         qemu_printf("Max guest advance   NA\n");
     }
 }
+
+static Bytes *memread(int64_t addr, int64_t size, CPUState *cpu, Error **errp)
+{
+    uint32_t l = 0;
+    uint8List *prev = NULL;
+    Bytes *res;
+    uint8_t buf[8];
+
+    if (size <= 0 || size > sizeof(buf)) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "size",
+                   "out of range");
+        return NULL;
+    }
+
+    if (cpu) {
+        if (cpu_memory_rw_debug(cpu, addr, buf, size, 0) != 0) {
+            error_setg(errp, "Invalid addr 0x%016" PRIx64 "/size %" PRId64
+                             " specified", addr, size);
+            return NULL;
+        }
+    } else {
+        MemTxResult r = address_space_read(&address_space_memory, addr,
+                                           MEMTXATTRS_UNSPECIFIED, buf, l);
+        if (r != MEMTX_OK) {
+            error_setg(errp, "Invalid addr 0x%016" PRIx64 "/size %" PRId64
+                             " specified", addr, size);
+            return NULL;
+        }
+    }
+
+    res = g_new0(Bytes, 1);
+    while (l < size) {
+        uint8List *cur = g_new0(uint8List, 1);
+        cur->value = buf[l++];
+        if (!prev) {
+            res->bytes = cur;
+        } else {
+            prev->next = cur;
+        }
+        prev = cur;
+    }
+
+    return res;
+
+}
+
+Bytes *qmp_memread(int64_t addr, int64_t size,
+                   bool has_cpu, int64_t cpu_index, Error **errp)
+{
+    CPUState *cpu;
+
+    if (!has_cpu) {
+        cpu_index = 0;
+    }
+
+    cpu = qemu_get_cpu(cpu_index);
+    if (cpu == NULL) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "cpu-index",
+                   "a CPU number");
+        return NULL;
+    }
+
+    return memread(addr, size, cpu, errp);
+}
+
+Bytes *qmp_pmemread(int64_t addr, int64_t size, Error **errp)
+{
+    return memread(addr, size, NULL, errp);
+}
+
+static void memwrite(int64_t addr, uint8List *bytes, CPUState *cpu,
+                     Error **errp)
+{
+    uint32_t l = 0;
+    uint8_t buf[8];
+
+    while (bytes != NULL) {
+        if (l >= sizeof(buf)) {
+            error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "bytes",
+                       "too long");
+            return;
+        }
+        buf[l++] = bytes->value;
+        bytes = bytes->next;
+    }
+
+    if (cpu) {
+        if (cpu_memory_rw_debug(cpu, addr, buf, l, 1) != 0) {
+            error_setg(errp, "Invalid addr 0x%016" PRIx64 "/size %" PRIu32
+                             " specified", addr, l);
+            return;
+        }
+    } else {
+        MemTxResult r = address_space_write(&address_space_memory, addr,
+                                            MEMTXATTRS_UNSPECIFIED, buf, l);
+        if (r != MEMTX_OK) {
+            error_setg(errp, "Invalid addr 0x%016" PRIx64 "/size %" PRId64
+                             " specified", addr, size);
+            return;
+        }
+    }
+}
+
+void qmp_memwrite(int64_t addr, uint8List *bytes,
+                  bool has_cpu, int64_t cpu_index, Error **errp)
+{
+    CPUState *cpu;
+
+    if (!has_cpu) {
+        cpu_index = 0;
+    }
+
+    cpu = qemu_get_cpu(cpu_index);
+    if (cpu == NULL) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "cpu-index",
+                   "a CPU number");
+        return;
+    }
+
+    memwrite(addr, bytes, cpu, errp);
+}
+
+void qmp_pmemwrite(int64_t addr, uint8List *bytes, Error **errp)
+{
+    memwrite(addr, bytes, NULL, errp);
+}
diff --git a/qapi/misc.json b/qapi/misc.json
index dc4cf9da20..3aca91b4ac 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3047,3 +3047,122 @@
   'data': 'NumaOptions',
   'allow-preconfig': true
 }
+
+##
+# @Bytes:
+#
+# An array of bytes.
+#
+# @bytes: the list of bytes
+#
+# Since: 4.1
+##
+{ 'struct': 'Bytes', 'data': {'bytes': ['uint8'] } }
+
+##
+# @memread:
+#
+# Read a portion of guest memory.
+#
+# @addr: the virtual address of the guest to read from
+#
+# @size: the size of memory region to read (max is 8)
+#
+# @cpu-index: the index of the virtual CPU to use for translating the
+#                       virtual address (defaults to CPU 0)
+#
+# Returns: The read bytes
+#
+# Since: 4.1
+#
+# Example:
+#
+# -> { "execute": "memread",
+#      "arguments": { "addr": 10,
+#                     "size": 4 } }
+# <- { "return": { 'bytes' : [10, 78, 231, 7] } }
+#
+##
+{ 'command': 'memread',
+  'data': {'addr': 'int', 'size': 'int', '*cpu-index': 'int'},
+  'returns' : 'Bytes'
+}
+
+##
+# @memwrite:
+#
+# Write a portion of guest memory.
+#
+# @addr: the virtual address of the guest to write to
+#
+# @bytes: the bytes to write into memory region (max length is 8)
+#
+# @cpu-index: the index of the virtual CPU to use for translating the
+#                       virtual address (defaults to CPU 0)
+#
+# Since: 4.1
+#
+# Returns: nothing on success.
+#
+# Example:
+#
+# -> { "execute": "memread",
+#      "arguments": { "addr": 10,
+#                     "bytes": [10, 78, 231, 7] } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'memwrite',
+  'data': {'addr': 'int', 'bytes': ['uint8'], '*cpu-index': 'int'}
+}
+
+##
+# @pmemread:
+#
+# Read a portion of guest memory.
+#
+# @addr: the physical address of the guest to read from
+#
+# @size: the size of memory region to read (max is 8)
+#
+# Returns: The read bytes
+#
+# Since: 4.1
+#
+# Example:
+#
+# -> { "execute": "memread",
+#      "arguments": { "addr": 10,
+#                     "size": 4 } }
+# <- { "return": { 'bytes' : [10, 78, 231, 7] } }
+#
+##
+{ 'command': 'pmemread',
+  'data': {'addr': 'int', 'size': 'int'},
+  'returns' : 'Bytes'
+}
+
+##
+# @pmemwrite:
+#
+# Write a portion of guest memory.
+#
+# @addr: the physical address of the guest to write to
+#
+# @bytes: the bytes to write into memory region (max length is 8)
+#
+# Since: 4.1
+#
+# Returns: nothing on success.
+#
+# Example:
+#
+# -> { "execute": "memread",
+#      "arguments": { "addr": 10,
+#                     "bytes": [10, 78, 231, 7] } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'pmemwrite',
+  'data': {'addr': 'int', 'bytes': ['uint8']}
+}
-- 
2.22.0

[Qemu-devel] [RFC PATCH 2/5] introduce a qmp command to set gpios

[Damien Hedde]

This adds the "gpio-set" qmp command.

Taking the device path, the gpio name and number, the command set the
value (true or false) of the gpio.
It works only on gpio input line.

This is based on Frederic Konrad's work.

Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
---
 cpus.c             |  4 ++--
 monitor/qmp-cmds.c | 30 ++++++++++++++++++++++++++++++
 qapi/misc.json     | 29 +++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 2 deletions(-)

diff --git a/cpus.c b/cpus.c
index 7aae35c098..3688550c55 100644
--- a/cpus.c
+++ b/cpus.c
@@ -2580,8 +2580,8 @@ static void memwrite(int64_t addr, uint8List *bytes, CPUState *cpu,
         MemTxResult r = address_space_write(&address_space_memory, addr,
                                             MEMTXATTRS_UNSPECIFIED, buf, l);
         if (r != MEMTX_OK) {
-            error_setg(errp, "Invalid addr 0x%016" PRIx64 "/size %" PRId64
-                             " specified", addr, size);
+            error_setg(errp, "Invalid addr 0x%016" PRIx64 "/size %" PRIu32
+                             " specified", addr, l);
             return;
         }
     }
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index 01ce77e129..6bf0204edd 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -726,3 +726,33 @@ MemoryInfo *qmp_query_memory_size_summary(Error **errp)
 
     return mem_info;
 }
+
+void qmp_gpio_set(const char *path, bool has_gpio, const char *gpio,
+                  bool has_number, int64_t number, bool value, Error **errp)
+{
+    DeviceState *dev;
+    qemu_irq irq;
+
+    dev = DEVICE(object_resolve_path(path, NULL));
+    if (!dev) {
+        error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
+                  "Cannot find device '%s'", path);
+        return;
+    }
+
+    if (!has_gpio) {
+        gpio = NULL;
+    }
+    if (!has_number) {
+        number = 0;
+    }
+    irq = qdev_get_gpio_in_named(dev, gpio, number);
+    if (!irq) {
+        error_set(errp, ERROR_CLASS_GENERIC_ERROR,
+                  "GPIO input '%s[%"PRId64"]' does not exists",
+                  has_gpio ? gpio : "unnamed", number);
+        return;
+    }
+
+    qemu_set_irq(irq, value);
+}
diff --git a/qapi/misc.json b/qapi/misc.json
index 3aca91b4ac..255236b96f 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3166,3 +3166,32 @@
 { 'command': 'pmemwrite',
   'data': {'addr': 'int', 'bytes': ['uint8']}
 }
+
+##
+# @gpio-set:
+#
+# @path: Path to the device.
+#
+# @gpio: Name of the GPIO will be unnamed-gpio if omitted.
+#
+# @number: Number of the GPIO line, 0 if omitted.
+#
+# @value: Value (boolean) to be set for the GPIO.
+#
+# Returns: nothing in case of success
+#
+# Since 4.1
+#
+# Example:
+#
+# -> { "execute": "gpio-set",
+#      "arguments": { "path": "/machine/unattached/device[5]",
+#                     "gpio": "ssi-gpio-cs",
+#                     "number": 0,
+#                     "value": true } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'gpio-set',
+  'data': { 'path': 'str', '*gpio': 'str', '*number': 'int', 'value': 'bool' }
+}
-- 
2.22.0

[Qemu-devel] [RFC PATCH 3/5] add qmp time-notify event triggering system

[Damien Hedde]

This adds an event triggering mechanism composed of:
 + an event that is catchable by qmp clients
 + a command to create such events

When triggered the event TIME_NOTIFICATION is signaled. Optionnaly the
virtual machine is also paused (put in debug state). The virtual machine can
then be restarted by the _cont_ command.

To create an event, an id and the deadline in virtual clock nanoseconds
should be given to the "time-notify" qmp command. The event will be
triggered at the given time which may be absolute or relative to the
current virtual clock time.

This allows to write qmp clients that can pause the vm, do some actions
then restart the vm.

This is based on the work of Frederic Konrad.

Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
---
 monitor/Makefile.objs         |   1 +
 monitor/qmp-cmd-time-notify.c | 116 ++++++++++++++++++++++++++++++++++
 monitor/trace-events          |   4 ++
 qapi/misc.json                |  48 ++++++++++++++
 4 files changed, 169 insertions(+)
 create mode 100644 monitor/qmp-cmd-time-notify.c

diff --git a/monitor/Makefile.objs b/monitor/Makefile.objs
index e91a8581cd..f4c7293460 100644
--- a/monitor/Makefile.objs
+++ b/monitor/Makefile.objs
@@ -1,3 +1,4 @@
 obj-y += misc.o
 common-obj-y += monitor.o qmp.o hmp.o
 common-obj-y += qmp-cmds.o hmp-cmds.o
+common-obj-y += qmp-cmd-time-notify.o
diff --git a/monitor/qmp-cmd-time-notify.c b/monitor/qmp-cmd-time-notify.c
new file mode 100644
index 0000000000..de13b2d3a1
--- /dev/null
+++ b/monitor/qmp-cmd-time-notify.c
@@ -0,0 +1,116 @@
+/*
+ * qmp-cmd-time-notify.c
+ *
+ * Copyright (c) 2016,2019 GreenSocs SAS
+ *
+ * Authors:
+ *    Fred Konrad
+ *    Damien Hedde
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qapi/qmp/qerror.h"
+#include "qapi/qmp/qjson.h"
+#include "qapi/qapi-commands-misc.h"
+#include "qapi/qapi-events-misc.h"
+#include "qemu/timer.h"
+#include "qapi/error.h"
+#include "qemu/log.h"
+#include "sysemu/sysemu.h"
+#include "sysemu/cpus.h"
+#include "trace.h"
+
+typedef struct TimeNotifEntry TimeNotifEntry;
+static QLIST_HEAD(, TimeNotifEntry) events = QLIST_HEAD_INITIALIZER(events);
+static QEMUTimer *timer;
+
+struct TimeNotifEntry {
+    uint64_t time_ns;
+    int64_t id;
+    bool pause;
+    QLIST_ENTRY(TimeNotifEntry) node;
+};
+
+static void mod_next_event_timer(void)
+{
+    if (QLIST_EMPTY(&events)) {
+        return;
+    }
+
+    timer_mod(timer, QLIST_FIRST(&events)->time_ns);
+}
+
+static void trigger_notif(void *opaque)
+{
+    TimeNotifEntry *entry;
+    uint64_t current_time = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    bool do_stop;
+
+    entry = QLIST_FIRST(&events);
+    assert(entry && entry->time_ns <= current_time);
+    do_stop = entry->pause;
+
+    QLIST_REMOVE(entry, node);
+    qapi_event_send_time_notification(entry->id, current_time);
+    trace_qmp_time_notify_trigger(entry->id, current_time, entry->pause);
+    g_free(entry);
+
+    mod_next_event_timer();
+
+    if (do_stop) {
+        qemu_system_vmstop_request_prepare();
+        qemu_system_vmstop_request(RUN_STATE_DEBUG);
+        /*
+         * FIXME: should not return to device code in case
+         * vm_stop() has been requested.
+         */
+        cpu_stop_current();
+    }
+}
+
+void qmp_time_notify(int64_t event_id, int64_t time_ns,
+                     bool has_relative, bool relative,
+                     bool has_pause, bool pause,
+                     Error **errp)
+{
+    TimeNotifEntry *new_entry, *entry, *prev = NULL;
+
+    if (!timer) {
+        timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, trigger_notif, NULL);
+    }
+
+    if (time_ns < 0) {
+        error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "time_ns",
+                "must be positive");
+    }
+
+    new_entry = g_new0(TimeNotifEntry, 1);
+    new_entry->id = event_id;
+    new_entry->time_ns = time_ns;
+    new_entry->pause = has_pause && pause;
+    if (has_relative && relative) {
+        new_entry->time_ns += qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    }
+    trace_qmp_time_notify_schedule(new_entry->id, new_entry->time_ns,
+                                   new_entry->pause);
+
+    /* find the event just before the new one */
+    QLIST_FOREACH(entry, &events, node) {
+        if (entry->time_ns > new_entry->time_ns) {
+            break;
+        }
+        prev = entry;
+    }
+
+    /* then insert the new entry */
+    if (prev) {
+        QLIST_INSERT_AFTER(prev, new_entry, node);
+    } else {
+        QLIST_INSERT_HEAD(&events, new_entry, node);
+        mod_next_event_timer();
+    }
+}
diff --git a/monitor/trace-events b/monitor/trace-events
index 0365ac4d99..73f375db68 100644
--- a/monitor/trace-events
+++ b/monitor/trace-events
@@ -13,3 +13,7 @@ monitor_suspend(void *ptr, int cnt) "mon %p: %d"
 monitor_qmp_cmd_in_band(const char *id) "%s"
 monitor_qmp_cmd_out_of_band(const char *id) "%s"
 handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
+
+# qmp-cmd-time-notify.c
+qmp_time_notify_trigger(int64_t id, uint64_t time, int pause) "event #%"PRId64" at %"PRIu64" ns pause %d"
+qmp_time_notify_schedule(int64_t id, uint64_t time, int pause) "event #%"PRId64" at %"PRIu64" ns pause %d"
diff --git a/qapi/misc.json b/qapi/misc.json
index 255236b96f..8e84f4d4d3 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3195,3 +3195,51 @@
 { 'command': 'gpio-set',
   'data': { 'path': 'str', '*gpio': 'str', '*number': 'int', 'value': 'bool' }
 }
+
+##
+# @time-notify:
+#
+# Schedule a TIME_NOTIFICATION which will optionally stop qemu when triggered.
+#
+# @time_ns:  The virtual guest clock at which do the notification.
+#
+# @relative: Optional boolean telling if time_ns is relative to current time.
+#            Defaults to False.
+#
+# @event_id: An ID, to track the notification.
+#
+# @pause: Optional boolean telling whether to pause qemu when notification is
+#         hit. Defaults to False.
+#
+# Returns: nothing in case of success
+#
+# Since 4.1
+#
+# Example:
+#
+# -> { "execute": "time-notify",
+#      "arguments": { "event_id": 5,
+#                      "time_ns": 10000000,
+#                      "pause": true} }
+# <- { "return": {} }
+#
+##
+{ 'command': 'time-notify',
+  'data': {'event_id': 'int', 'time_ns': 'int', '*relative': 'bool',
+           '*pause': 'bool'}
+}
+
+##
+# @TIME_NOTIFICATION:
+#
+# Emitted when notification deadline is hit
+#
+# @time_ns:  The current virtual guest clock time.
+#
+# @event_id: The id given when setting up the notification.
+#
+# Since: 4.1
+#
+##
+{ 'event': 'TIME_NOTIFICATION',
+  'data': { 'event_id': 'int', 'time_ns': 'int'} }
-- 
2.22.0

[Qemu-devel] [RFC PATCH 4/5] fault_injection: introduce Python scripting framework

[Damien Hedde]

This is the actual Python framework.
It provides some wrappers:
  * which allow to notify a callback in a given qemu time.
  * read or write some memory location
  * read/write qom properties.
  * set a GPIO.

This is based on the work of Frederic Konrad

Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
---
 scripts/qmp/fault_injection.py | 278 +++++++++++++++++++++++++++++++++
 1 file changed, 278 insertions(+)
 create mode 100644 scripts/qmp/fault_injection.py

diff --git a/scripts/qmp/fault_injection.py b/scripts/qmp/fault_injection.py
new file mode 100644
index 0000000000..2d23e69d47
--- /dev/null
+++ b/scripts/qmp/fault_injection.py
@@ -0,0 +1,278 @@
+# Fault injection helper script based on top of QMP.
+#
+# Copyright (C) 2016,2019 GreenSocs SAS
+#
+# Authors:
+#   Frederic Konrad <fred.konrad@greensocs.com>
+#   Damien Hedde <damien.hedde@greensocs.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or later.
+# See the COPYING file in the top-level directory.
+#
+
+import json
+import ast
+import readline
+import sys
+import struct
+import os
+sys.path.append(os.path.join(os.path.dirname(__file__),
+                             '..', '..', 'python', 'qemu'))
+import qmp
+
+def die(cause):
+    print(sys.stderr.write('error: %s\n' % cause))
+    sys.exit(1)
+
+class FaultInjectionQMPError(Exception):
+    def __init__(self, rsp):
+        self.args = (rsp,)
+
+class FaultInjectionFramework(qmp.QEMUMonitorProtocol):
+    qemu_time = 0
+    verbose = 0
+    callback = {}
+    endianness = None
+
+    def print_v(self, msg, level):
+        if level <= self.verbose:
+            print(msg)
+
+    def print_qemu_version(self):
+        version = self._greeting['QMP']['version']['qemu']
+        print('Connected to QEMU %d.%d.%d' % (version['major'],
+                                                version['minor'],
+                                                version['micro']))
+
+    def __init__(self, address, verbose = 0):
+        self.verbose = verbose
+        qmp.QEMUMonitorProtocol.__init__(self, self.__get_address(address))
+
+        try:
+            self._greeting = qmp.QEMUMonitorProtocol.connect(self)
+        except qmp.QMPConnectError:
+            die('Didn\'t get QMP greeting message')
+        except qmp.QMPCapabilitiesError:
+            die('Could not negotiate capabilities')
+        except self.error:
+            die('Could not connect to %s' % address)
+
+        self.print_qemu_version()
+        self._completer = None
+        self._pretty = False
+        self._transmode = False
+        self._actions = list()
+
+    def set_endian_big(self):
+        self.endianness = 'big'
+
+    def set_endian_little(self):
+        self.endianness = 'little'
+
+    def time_print(self, arg):
+        self.print_v('%sns: %s' % (self.qemu_time, arg), 1)
+
+    def send(self, qmpcmd):
+        self.print_v(qmpcmd, 2)
+        resp = self.cmd_obj(qmpcmd)
+        if resp is None:
+            die('Disconnected')
+        self.print_v(resp, 2)
+        if 'error' in resp:
+            raise FaultInjectionQMPError(resp)
+        return resp
+
+    def cont(self):
+        qmpcmd = {'execute': 'cont', 'arguments': {}}
+        self.send(qmpcmd)
+
+    def run_once(self):
+        # RUN the simulation until one event is received
+        self.cont()
+        # Wait for an event to appear
+        done = False
+        while done == False:
+            for ev in self.get_events(True):
+                self.print_v(ev, 2)
+                if ev['event'] == 'TIME_NOTIFICATION':
+                    data = ev['data']
+                    self.qemu_time = data['time_ns'];
+                    self.callback[data['event_id']]()
+                    self.cont()
+                    done = True
+                elif ev['event'] == 'SHUTDOWN':
+                    self.clear_events()
+                    return True
+        self.clear_events()
+        return False
+
+    def run(self):
+        # RUN the simulation.
+        self.time_print('Simulation is now running')
+        # Wait for an event to appear
+        shutdown_evt = False
+        while shutdown_evt == False:
+            shutdown_evt = self.run_once()
+        self.close()
+
+    def notify(self, time_ns, cb, relative = False):
+        # Notify a callback at qemu time time_ns
+        next_index = len(self.callback)
+        elt = 0
+        for elt in range(0, next_index + 1):
+            if elt == next_index:
+                break
+            if self.callback[elt] == cb:
+                break
+
+        self.callback[elt] = cb
+        if relative:
+            self.time_print('Notify %s in %sns' % (cb, time_ns))
+        else:
+            self.time_print('Notify %s at %sns' % (cb, time_ns))
+        qmpcmd = {'execute': 'time-notify',
+                  'arguments': {'event_id': elt,
+                                'time_ns': time_ns,
+                                'pause' : True}}
+        if relative:
+            qmpcmd['arguments']['relative'] = True
+        self.send(qmpcmd)
+
+    def _pvmemwrite(self, virtual, address, value, cpu = None):
+        # write a value to a virtual or physical address
+        if type(value) is not list:
+            value = list(value)
+
+        self.time_print('write: @%s0x%08x size %d values ['
+                        % ('V' if virtual else 'P', address, len(value))
+                        + ','.join('%d' % i for i in value)
+                        + '] from cpu %s' % (cpu))
+        qmpcmd = {'execute': 'memwrite' if virtual else 'pmemwrite',
+                  'arguments': {'addr': address, 'bytes': value}}
+        if cpu is not None:
+            qmpcmd['arguments']['cpu'] = cpu;
+        rsp = self.send(qmpcmd)
+
+    def _pvmemread(self, virtual, address, size, cpu = None):
+        # read a value to a virtual or physical address
+
+        self.time_print('read: @%s0x%08x size %d values from cpu %s'
+                        % ('V' if virtual else 'P', address, size, cpu))
+        qmpcmd = {'execute': 'memread' if virtual else 'pmemread',
+                  'arguments': {'addr': address, 'size': size}}
+        if cpu is not None:
+            qmpcmd['arguments']['cpu'] = cpu;
+        rsp = self.send(qmpcmd)
+        value = bytearray(rsp['return']['bytes'])
+        return value
+
+    def _memstructformat(self, size):
+        if self.endianness == 'little':
+            s = '<'
+        elif self.endianness == 'big':
+            s = '>'
+        else:
+            #defaults to native
+            s = '='
+        s += {1:'B', 2:'H', 4:'I', 8:'Q'}[size]
+        return s
+
+    def read_mem(self, address, size, cpu = None):
+        fmt = self._memstructformat(size)
+        value = self._pvmemread(True, address, size, cpu)
+        return struct.unpack(fmt, value)[0]
+
+    def write_mem(self, address, size, value, cpu = None):
+        fmt = self._memstructformat(size)
+        value = bytearray(struct.pack(fmt, value))
+        self._pvmemwrite(True, address, value, cpu)
+
+    def read_pmem(self, address, size):
+        fmt = self._memstructformat(size)
+        value = self._pvmemread(False, address, size)
+        return struct.unpack(fmt, value)[0]
+
+    def write_pmem(self, address, size, value):
+        fmt = self._memstructformat(size)
+        value = bytearray(struct.pack(fmt, value))
+        self._pvmemwrite(False, address, value)
+
+    def get_qom_property(self, path, property):
+        # Get a QOM property
+        qmpcmd = {'execute': 'qom-get',
+                  'arguments': {'path': path,
+                                'property': property}}
+        value = self.send(qmpcmd)['return']
+        return value
+
+    def set_qom_property(self, path, property, value):
+        # Set a QOM property
+        qmpcmd = {'execute': 'qom-set',
+                  'arguments': {'path': path,
+                                'property': property,
+                                'value': value}}
+        self.send(qmpcmd)
+
+    def set_gpio(self, path, gpio, num, value):
+        # Set a GPIO
+        qmpcmd = {'execute': 'gpio-set',
+                  'arguments': {'path': path, 'value': value}}
+        if gpio is not None:
+            qmpcmd['arguments']['gpio'] = gpio
+        if num is not None:
+            qmpcmd['arguments']['number'] = num
+        self.send(qmpcmd)
+
+    def help(self):
+        msg = [
+            "Fault Injection Framework Commands",
+            "==================================\n",
+            "cont()",
+            " * Resume the simulation when the Virtual Machine is stopped.\n",
+            "run()",
+            " * Start the simulation when the notify are set.\n",
+            "notify(delta_ns, cb)",
+            " * Notify the callback cb in guest time delta_ns.\n",
+            "write_mem(address, size, value, cpu)",
+            " * write @value of size @size at virtual @address from @cpu.",
+            " * @size is in bytes, supported values are 1, 2, 4 and 8.",
+            " * @cpu is the cpu id.\n",
+            "read_mem(address, size, cpu)",
+            " * read a value of size @size at virtual @address from @cpu.",
+            " * @size is in bytes, supported values are 1, 2, 4 and 8.",
+            " * @cpu is the cpu id.",
+            " * returns the value.\n",
+            "write_pmem(address, size, value)",
+            " * write @value of size @size at physical @address.",
+            " * @size is in bytes, supported values are 1, 2, 4 and 8.\n",
+            "read_pmem(address, size)",
+            " * read a value of size @size at physical @address.",
+            " * @size is in bytes, supported values are 1, 2, 4 and 8.",
+            " * returns the value.\n",
+            "get_qom_property(path, property)",
+            " * Get a qom property.",
+            " * Returns the qom property named @property in @path.\n",
+            "set_qom_property(path, property, value)",
+            " * Set the property named @property in @path with @value.\n",
+            "set_gpio(path, gpio, num, value)",
+            " * Set the gpio named @gpio number @num in @path with the @val.",
+            " * @val is a boolean.\n"
+            ]
+        for m in msg:
+            print(m)
+
+    def __get_address(self, arg):
+        """
+        Figure out if the argument is in the port:host form, if it's not it's
+        probably a file path.
+        """
+        addr = arg.split(':')
+        if len(addr) == 2:
+            try:
+                port = int(addr[1])
+            except ValueError:
+                raise QMPShellBadPort
+            return ( addr[0], port )
+        # socket path
+        return arg
+
-- 
2.22.0

[Qemu-devel] [RFC PATCH 5/5] docs: add fault injection framework documentation

[Damien Hedde]

This adds some hints about how to use the fault injection framework.

This is based on Frederic Konrad's work.

Signed-off-by: Damien Hedde <damien.hedde@greensocs.com>
---
 docs/fault_injection.txt | 149 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 149 insertions(+)
 create mode 100644 docs/fault_injection.txt

diff --git a/docs/fault_injection.txt b/docs/fault_injection.txt
new file mode 100644
index 0000000000..3cb1c1fd1a
--- /dev/null
+++ b/docs/fault_injection.txt
@@ -0,0 +1,149 @@
+                           Fault Injection Framework
+                           =========================
+
+Introduction
+------------
+
+The fault injection framework allows users to write python scripts to inject
+faults through the QMP (QEMU Machine Protocol) during execution.
+
+Basically it's composed of a Python API which makes some QMP commands easy to
+send to the machine:
+  * read/write a virtual/memory memory location.
+  * set a GPIO line.
+  * get/set a QOM property.
+
+In addition it allows the Python script to be notified back by QEMU so it can
+do any of the previous commands at a given virtual clock time.
+
+Today the available function in the API are the following:
+  * notify(time_ns, cb, do_halt)
+  * write_mem(address, value, size, cpu)
+  * read_mem(address, size, cpu)
+  * write_pmem(address, value, size)
+  * read_pmem(address, size)
+  * get_qom_property(path, property)
+  * set_qom_property(path, property, value)
+  * set_gpio(path, gpio, num, value)
+
+Empty Example
+-------------
+
+This is an empty example to begin:
+
+import fault_injection
+import sys
+
+framework = None
+
+def main():
+    # The injection framework will parse the command line automatically
+    # (eg: the qmp socket/port.. etc)
+    sys.stdout.write('Fault Injection Example\n')
+    global framework
+    framework = fault_injection.FaultInjectionFramework(sys.argv[1])
+
+    framework.run()
+    sys.exit(1)
+
+if __name__ == '__main__':
+    main()
+
+To run the example just save the example in `script/qmp/example_scenario`
+Run qemu with the additional arguments: `-S -qmp unix:/path/to/qmp-sock,server`
+in order to wait for a qmp connection and stop the QEMU machine.
+Run the example with: `./example_scenario /path/to/qmp-sock`
+
+It will start the simulation inside QEMU and do nothing else.
+
+Adding a callback at a given time
+---------------------------------
+
+As described above a callback can be added in the python scenario.
+For example we can create the following callback which write 0xDEADBEEF @0 with
+a size of 4 from cpu 0 and then reads it back:
+
+def write_mem_callback():
+    print 'write_mem_callback()'
+    framework.write_mem(0x0, 0xDEADBEEF, 4, 0)
+    val = framework.read_mem(0x0, 4, 0)
+    print 'value read: 0x%8.8X' %val
+
+Then we can notify it in the main function before framework.run():
+`framework.notify(1000000000, write_mem_callback)`
+
+The script works as expected:
+
+write_mem_callback()
+value read: 0xDEADBEEF
+
+Using the python interpreter
+----------------------------
+
+The Python interpreter can be used to send the command above:
+For example to set the vinithi bit to 1 for the /rpu_cpu@0 the following
+commands can be done in the script/qmp directory:
+
+$ python
+>>> import fault_injection
+>>> inj=fault_injection.FaultInjectionFramework("../../qmp-sock", 0)
+Connected to QEMU 2.2.50
+
+>>> inj.help()
+
+Fault Injection Framework Commands
+==================================
+
+cont()
+ * Resume the simulation when the Virtual Machine is stopped.
+
+notify(time_ns, cb, do_pause)
+ * Notify the callback cb in guest time time_ns. Simulation is stopped only if
+   do_pause is set to True.
+
+run_once()
+ * Start the simulation and handle a callback set by notify. Return after
+   first callback is handled
+
+run()
+ * Start the simulation and handle the callbacks set by notify. Do not return
+   until the end of the simulation.
+
+write_mem(address, value, size, cpu)
+ * Write @value of size @size at virtual @address from @cpu.
+ * @cpu is the cpu id.
+
+write_pmem(address, value, size)
+ * Like write_mem, but on a physical @address.
+
+read_mem(address, size, cpu)
+ * Read a value of size @size at virtual @address from @cpu.
+ * @cpu is the cpu id.
+ * Returns the value.
+
+read_pmem(address, size)
+ * Like read_mem, but on a physical @address.
+
+get_qom_property(path, property)
+ * Get a qom property.
+ * Returns the qom property named @property in @path.
+
+set_qom_property(path, property, value)
+ * Set the property named @property in @path with @value.
+
+set_gpio(path, gpio, num, value)
+ * Set the gpio named @gpio number @num in @path with the @val.
+ * @val is a boolean.
+
+>>> inj.set_gpio('/rpu_cpu@0', 'vinithi', 0, 1)
+
+Notes
+-----
+
+The user can turn debug information on by passing a level to the framework
+constructor eg:
+"framework = fault_injection.FaultInjectionFramework(1)" will print timed traces
+such as write or read.
+"framework = fault_injection.FaultInjectionFramework(2)" will print the QMP
+commands as well.
+
-- 
2.22.0

Re: [Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

On Fri, Jun 28, 2019 at 02:45:29PM +0200, Damien Hedde wrote:
> This series adds a python framework aiming to provide some ways to do fault
> injection in a running vm. In its current state, it allows to easily interact
> with memory, change gpios and qom properties.
> 
> The framework consists in a python script based on the qmp existing module
> which allows to interact with the vm.

How does this compare to qtest?  There seems to be a lot of overlap
between them.

Why is it called "fault injection"?  The commands seem to be
general-purpose device testing functions (like qtest and libqos), not
functions for testing error code paths as would be expected from a fault
injection framework.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Philippe Mathieu-Daudé]

[-- Attachment #1.1: Type: text/plain, Size: 1501 bytes --]

On 7/1/19 10:37 AM, Stefan Hajnoczi wrote:
> On Fri, Jun 28, 2019 at 02:45:29PM +0200, Damien Hedde wrote:
>> This series adds a python framework aiming to provide some ways to do fault
>> injection in a running vm. In its current state, it allows to easily interact
>> with memory, change gpios and qom properties.
>>
>> The framework consists in a python script based on the qmp existing module
>> which allows to interact with the vm.
> 
> How does this compare to qtest?  There seems to be a lot of overlap
> between them.
> 
> Why is it called "fault injection"?  The commands seem to be
> general-purpose device testing functions (like qtest and libqos), not
> functions for testing error code paths as would be expected from a fault
> injection framework.

I understand qtest is to test QEMU, while this framework/command is to
test how the guest react to an hardware faults.

To use the qtest_mem commands you need to run QEMU with the qtest
chardev backend, while this series expose a QMP interface.

To avoid the overlap, a cleaner follow up might be to have qtest wrap
these QMP commands (mostly like HMP commands do).

Another note while looking at a glance, qtest uses the 1st cpu address
space view, this series allow to select a specific cpu.

It makes sense to me to be able to select address spaces by name (more
generic, not restricted to a cpu view, since one might want to inject
fault in a device ram not always mapped to a cpu: dma, emac desc).


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2301 bytes --]

On Mon, Jul 01, 2019 at 12:16:44PM +0200, Philippe Mathieu-Daudé wrote:
> On 7/1/19 10:37 AM, Stefan Hajnoczi wrote:
> > On Fri, Jun 28, 2019 at 02:45:29PM +0200, Damien Hedde wrote:
> >> This series adds a python framework aiming to provide some ways to do fault
> >> injection in a running vm. In its current state, it allows to easily interact
> >> with memory, change gpios and qom properties.
> >>
> >> The framework consists in a python script based on the qmp existing module
> >> which allows to interact with the vm.
> > 
> > How does this compare to qtest?  There seems to be a lot of overlap
> > between them.
> > 
> > Why is it called "fault injection"?  The commands seem to be
> > general-purpose device testing functions (like qtest and libqos), not
> > functions for testing error code paths as would be expected from a fault
> > injection framework.
> 
> I understand qtest is to test QEMU, while this framework/command is to
> test how the guest react to an hardware faults.

The commands seems to be equivalent to qtest commands, just implemented
as QMP commands.

Damien: Can you explain the use case more and show some example test
cases?

> To use the qtest_mem commands you need to run QEMU with the qtest
> chardev backend, while this series expose a QMP interface.
> 
> To avoid the overlap, a cleaner follow up might be to have qtest wrap
> these QMP commands (mostly like HMP commands do).
> 
> Another note while looking at a glance, qtest uses the 1st cpu address
> space view, this series allow to select a specific cpu.
> 
> It makes sense to me to be able to select address spaces by name (more
> generic, not restricted to a cpu view, since one might want to inject
> fault in a device ram not always mapped to a cpu: dma, emac desc).

The naming issue still stands: none of the commands actually perform
fault injection.  They can be used for other types of testing or even
non-testing purposes.

Fault injection commands would be "make the next watchdog expiry fail",
"return error code X on the next DMA request", "report an AHCI link
failure", etc.

These commands are lower-level.  Therefore, I think "fault injection
framework" is a misnomer and will age poorly if this API is extended in
the future.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Damien Hedde]

On 7/3/19 11:29 AM, Stefan Hajnoczi wrote:
> On Mon, Jul 01, 2019 at 12:16:44PM +0200, Philippe Mathieu-Daudé wrote:
>> On 7/1/19 10:37 AM, Stefan Hajnoczi wrote:
>>> On Fri, Jun 28, 2019 at 02:45:29PM +0200, Damien Hedde wrote:
>>>> This series adds a python framework aiming to provide some ways to do fault
>>>> injection in a running vm. In its current state, it allows to easily interact
>>>> with memory, change gpios and qom properties.
>>>>
>>>> The framework consists in a python script based on the qmp existing module
>>>> which allows to interact with the vm.
>>>
>>> How does this compare to qtest?  There seems to be a lot of overlap
>>> between them.
>>>
>>> Why is it called "fault injection"?  The commands seem to be
>>> general-purpose device testing functions (like qtest and libqos), not
>>> functions for testing error code paths as would be expected from a fault
>>> injection framework.
>>
>> I understand qtest is to test QEMU, while this framework/command is to
>> test how the guest react to an hardware faults.
> 
> The commands seems to be equivalent to qtest commands, just implemented
> as QMP commands.
> 
> Damien: Can you explain the use case more and show some example test
> cases?

The goal is to test and validate the software running on the vp. We want
to generate some fault to test if the software behave correctly. We
target corner cases that do not happen otherwise on the vp. Basically we
would like, using some scripts, to run some specific scenarios and check
that the expected behavior happens.

Regarding qtest, I was not aware that it provided such commands. I'm
sorry I've missed that. Just checked after reading your feedback,
commands seem indeed equivalent. I don't know if running the vp with
qtest enabled has some hidden drawbacks. But if that's not the case, we
can work to extend the existing qtest commands (or switch some of them
to QMP like Philippe proposed, I don't know what's best).

> 
>> To use the qtest_mem commands you need to run QEMU with the qtest
>> chardev backend, while this series expose a QMP interface.
>>
>> To avoid the overlap, a cleaner follow up might be to have qtest wrap
>> these QMP commands (mostly like HMP commands do).
>>
>> Another note while looking at a glance, qtest uses the 1st cpu address
>> space view, this series allow to select a specific cpu.
>>
>> It makes sense to me to be able to select address spaces by name (more
>> generic, not restricted to a cpu view, since one might want to inject
>> fault in a device ram not always mapped to a cpu: dma, emac desc).

Good point.

> 
> The naming issue still stands: none of the commands actually perform
> fault injection.  They can be used for other types of testing or even
> non-testing purposes.
> 
> Fault injection commands would be "make the next watchdog expiry fail",
> "return error code X on the next DMA request", "report an AHCI link
> failure", etc.
> 
> These commands are lower-level.  Therefore, I think "fault injection
> framework" is a misnomer and will age poorly if this API is extended in
> the future.

The only fault injection naming was for the python module. I suppose
that if we just extend qtest, there is no need for a new module or
documentation file.

Thanks,

Damien

Re: [Qemu-devel] [RFC PATCH 0/5] FAULT INJECTION FRAMEWORK

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2705 bytes --]

On Wed, Jul 03, 2019 at 05:47:47PM +0200, Damien Hedde wrote:
> On 7/3/19 11:29 AM, Stefan Hajnoczi wrote:
> > On Mon, Jul 01, 2019 at 12:16:44PM +0200, Philippe Mathieu-Daudé wrote:
> >> On 7/1/19 10:37 AM, Stefan Hajnoczi wrote:
> >>> On Fri, Jun 28, 2019 at 02:45:29PM +0200, Damien Hedde wrote:
> >>>> This series adds a python framework aiming to provide some ways to do fault
> >>>> injection in a running vm. In its current state, it allows to easily interact
> >>>> with memory, change gpios and qom properties.
> >>>>
> >>>> The framework consists in a python script based on the qmp existing module
> >>>> which allows to interact with the vm.
> >>>
> >>> How does this compare to qtest?  There seems to be a lot of overlap
> >>> between them.
> >>>
> >>> Why is it called "fault injection"?  The commands seem to be
> >>> general-purpose device testing functions (like qtest and libqos), not
> >>> functions for testing error code paths as would be expected from a fault
> >>> injection framework.
> >>
> >> I understand qtest is to test QEMU, while this framework/command is to
> >> test how the guest react to an hardware faults.
> > 
> > The commands seems to be equivalent to qtest commands, just implemented
> > as QMP commands.
> > 
> > Damien: Can you explain the use case more and show some example test
> > cases?
> 
> The goal is to test and validate the software running on the vp. We want
> to generate some fault to test if the software behave correctly. We
> target corner cases that do not happen otherwise on the vp. Basically we
> would like, using some scripts, to run some specific scenarios and check
> that the expected behavior happens.
> 
> Regarding qtest, I was not aware that it provided such commands. I'm
> sorry I've missed that. Just checked after reading your feedback,
> commands seem indeed equivalent. I don't know if running the vp with
> qtest enabled has some hidden drawbacks. But if that's not the case, we
> can work to extend the existing qtest commands (or switch some of them
> to QMP like Philippe proposed, I don't know what's best).

I'm not 100% sure that qtest is the right tool for the job.  Maybe you
really need to add QMP commands as you have done.

Could you share some test cases so reviewers have an idea of how these
new commands are used for fault injection?

qtest is special in that no guest code executes.  QEMU allocates guest
RAM and initializes devices as usual but TCG/KVM do not execute guest
CPU instructions.  Does your use case require guest execution?

Here is a presentation on qtest if you want to get an overview:
https://www.youtube.com/watch?v=4TSaMmrnHy8

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]