[Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

Hi everybody,

This is the sixth version of this RFC (see the changes below).

Those are the two first patch-set we have been worked on for reverse execution.

The first part is fully reviewed except the "icount: introduce icount timer"
patch maybe we can merge them?

The first series:
  icount: put icount variables into TimerState.
  migration: migrate icount fields.
  migration: make qemu_savevm_state public.
  icount: introduce icount timer.
  icount: check for icount clock deadline when cpu loop exits.
  icount: make icount extra computed on icount clock as well.
  timer: add cpu_icount_to_ns function.

are various preparation patches for reverse execution.

The last patches:
  trace-events: add reverse-execution events.
  introduce reverse execution mechanism.
  gdbstub: allow reverse execution in gdb stub.
  cpu-exec: trigger a debug request when rexec stops.
  rexec: synchronize icount on the next event.
  rexec: allow to enable reverse execution.

are reverse execution introduction.

They can be clone at: git://git.greensocs.com/qemu_cexe.git:cexe_2_3_v6

The third series will be sent as soon as possible and have some issues with
QEMU's thread as it use fork.

This implementation of reverse execution works with instruction counting:

A new clock is implemented which is icount clock. It grows each time an
instruction is executed and is totally independant of host clock.

Snapshots are taken regularly (based on icount clock) with help of migration
code and written on the disk.

When user wants to use reverse-stepi:
 * Last snapshot is reloaded.
 * A stop callback is created to be triggered at the previous instruction.

This stop callback generates a debug exception so QEMU stops in debug mode.

Command line:
 * rexec suboption is added to icount to enable reverse execution, it needs
   icount=N and doesn't support auto mode.

About non determinism in QEMU:
 * This implementation doesn't take IO in account so any IO will cause non
   determinism and break reverse execution.

 * The icount warp mechanism have been disabled when reverse execution is
   enabled so the time grow differently inside the VM.

Testing:
 * It has been tested on ARM without any IO such as network or asynchronous file
   access to keep the deterministic behaviour of icount.

Known issues:
 * On ARM stepi seems to do some additional steps which are added to icount
   counter so reverse-stepi just after stepi is broken.

 * The IO replay explained above.

Changes:
 v5 -> v6:
  * Boolize some variable (As suggested by Li Guang).
  * s/snap_timer/snapshot_timer/g (As suggested by Li Guang).
  * Choice a better name for the snapshot file. (As suggested by Li Guang).
  * Boolize stepping_back, etc.. (As suggested by Li Guang).
  * Cherry-pick icount: Add QemuOpts for icount. (As suggested by Paolo).
  * Making reverse execution option using QEMUOpts (As sugested by Paolo).
  * s/cexe/rexec/g (As nobody like cexe :)).
 
 v4 -> v5:
  * Fix a coding style issue (As suggested by Juan).

 v3 -> v4:
  * Fix icount_state_needed (As suggested by Amit).
  * Rebase.

 v2 -> v3:
  * Use trace instead of debug printfs (As suggested by Lluis).

 v1 -> v2:
  * Use subsection for icount migration (As suggested by Paolo).

  * Use with_bias parameters to get_icount instead of get_icount_wo_bias
    function (As suggested by Paolo).

KONRAD Frederic (13):
  icount: put icount variables into TimerState.
  migration: migrate icount fields.
  migration: make qemu_savevm_state public.
  icount: introduce icount timer.
  icount: check for icount clock deadline when cpu loop exits.
  icount: make icount extra computed on icount clock as well.
  timer: add cpu_icount_to_ns function.
  trace-events: add reverse-execution events.
  introduce reverse execution mechanism.
  gdbstub: allow reverse execution in gdb stub.
  cpu-exec: trigger a debug request when rexec stops.
  rexec: synchronize icount on the next event.
  rexec: allow to enable reverse execution.

Sebastian Tanase (1):
  icount: Add QemuOpts for icount

 Makefile.target                |   1 +
 cpu-exec.c                     |  13 ++
 cpus.c                         | 141 ++++++++++++++++---
 gdbstub.c                      |  31 ++++-
 include/qemu-common.h          |   3 +-
 include/qemu/timer.h           |  18 ++-
 include/reverse-execution.h    |  43 ++++++
 include/sysemu/sysemu.h        |   1 +
 main-loop.c                    |  10 ++
 qemu-options.hx                |   8 +-
 qemu-timer.c                   |   8 +-
 qtest.c                        |  13 +-
 reverse-execution.c            | 308 +++++++++++++++++++++++++++++++++++++++++
 savevm.c                       |   2 +-
 stubs/Makefile.objs            |   1 +
 stubs/cpu-get-icount.c         |  10 +-
 stubs/reverse-execution-stub.c |  32 +++++
 trace-events                   |   6 +
 vl.c                           |  48 +++++--
 19 files changed, 658 insertions(+), 39 deletions(-)
 create mode 100644 include/reverse-execution.h
 create mode 100644 reverse-execution.c
 create mode 100644 stubs/reverse-execution-stub.c

-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 01/14] icount: put icount variables into TimerState.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This puts qemu_icount and qemu_icount_bias into TimerState structure to allow
them to be migrated.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 cpus.c | 29 ++++++++++++++++-------------
 1 file changed, 16 insertions(+), 13 deletions(-)

diff --git a/cpus.c b/cpus.c
index 5e7f2cf..127de1c 100644
--- a/cpus.c
+++ b/cpus.c
@@ -102,17 +102,12 @@ static bool all_cpu_threads_idle(void)
 
 /* Protected by TimersState seqlock */
 
-/* Compensate for varying guest execution speed.  */
-static int64_t qemu_icount_bias;
 static int64_t vm_clock_warp_start;
 /* Conversion factor from emulated instructions to virtual clock ticks.  */
 static int icount_time_shift;
 /* Arbitrarily pick 1MIPS as the minimum allowable speed.  */
 #define MAX_ICOUNT_SHIFT 10
 
-/* Only written by TCG thread */
-static int64_t qemu_icount;
-
 static QEMUTimer *icount_rt_timer;
 static QEMUTimer *icount_vm_timer;
 static QEMUTimer *icount_warp_timer;
@@ -129,6 +124,11 @@ typedef struct TimersState {
     int64_t cpu_clock_offset;
     int32_t cpu_ticks_enabled;
     int64_t dummy;
+
+    /* Compensate for varying guest execution speed.  */
+    int64_t qemu_icount_bias;
+    /* Only written by TCG thread */
+    int64_t qemu_icount;
 } TimersState;
 
 static TimersState timers_state;
@@ -139,14 +139,14 @@ static int64_t cpu_get_icount_locked(void)
     int64_t icount;
     CPUState *cpu = current_cpu;
 
-    icount = qemu_icount;
+    icount = timers_state.qemu_icount;
     if (cpu) {
         if (!cpu_can_do_io(cpu)) {
             fprintf(stderr, "Bad clock read\n");
         }
         icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
     }
-    return qemu_icount_bias + (icount << icount_time_shift);
+    return timers_state.qemu_icount_bias + (icount << icount_time_shift);
 }
 
 int64_t cpu_get_icount(void)
@@ -284,7 +284,8 @@ static void icount_adjust(void)
         icount_time_shift++;
     }
     last_delta = delta;
-    qemu_icount_bias = cur_icount - (qemu_icount << icount_time_shift);
+    timers_state.qemu_icount_bias = cur_icount
+                              - (timers_state.qemu_icount << icount_time_shift);
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 }
 
@@ -333,7 +334,7 @@ static void icount_warp_rt(void *opaque)
             int64_t delta = cur_time - cur_icount;
             warp_delta = MIN(warp_delta, delta);
         }
-        qemu_icount_bias += warp_delta;
+        timers_state.qemu_icount_bias += warp_delta;
     }
     vm_clock_warp_start = -1;
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);
@@ -351,7 +352,7 @@ void qtest_clock_warp(int64_t dest)
         int64_t deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
         int64_t warp = qemu_soonest_timeout(dest - clock, deadline);
         seqlock_write_lock(&timers_state.vm_clock_seqlock);
-        qemu_icount_bias += warp;
+        timers_state.qemu_icount_bias += warp;
         seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 
         qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL);
@@ -1250,7 +1251,8 @@ static int tcg_cpu_exec(CPUArchState *env)
         int64_t count;
         int64_t deadline;
         int decr;
-        qemu_icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
+        timers_state.qemu_icount -= (cpu->icount_decr.u16.low
+                                    + cpu->icount_extra);
         cpu->icount_decr.u16.low = 0;
         cpu->icount_extra = 0;
         deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
@@ -1265,7 +1267,7 @@ static int tcg_cpu_exec(CPUArchState *env)
         }
 
         count = qemu_icount_round(deadline);
-        qemu_icount += count;
+        timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
         count -= decr;
         cpu->icount_decr.u16.low = decr;
@@ -1278,7 +1280,8 @@ static int tcg_cpu_exec(CPUArchState *env)
     if (use_icount) {
         /* Fold pending instructions back into the
            instruction counter, and clear the interrupt flag.  */
-        qemu_icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
+        timers_state.qemu_icount -= (cpu->icount_decr.u16.low
+                        + cpu->icount_extra);
         cpu->icount_decr.u32 = 0;
         cpu->icount_extra = 0;
     }
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 02/14] migration: migrate icount fields.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This fixes a bug where qemu_icount and qemu_icount_bias are not migrated.
It adds a subsection "timer/icount" to vmstate_timers so icount is migrated only
when needed.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Amit Shah <amit.shah@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 cpus.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/cpus.c b/cpus.c
index 127de1c..a6b6557 100644
--- a/cpus.c
+++ b/cpus.c
@@ -429,6 +429,25 @@ void qemu_clock_warp(QEMUClockType type)
     }
 }
 
+static bool icount_state_needed(void *opaque)
+{
+    return use_icount;
+}
+
+/*
+ * This is a subsection for icount migration.
+ */
+static const VMStateDescription icount_vmstate_timers = {
+    .name = "timer/icount",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_INT64(qemu_icount_bias, TimersState),
+        VMSTATE_INT64(qemu_icount, TimersState),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static const VMStateDescription vmstate_timers = {
     .name = "timer",
     .version_id = 2,
@@ -438,6 +457,14 @@ static const VMStateDescription vmstate_timers = {
         VMSTATE_INT64(dummy, TimersState),
         VMSTATE_INT64_V(cpu_clock_offset, TimersState, 2),
         VMSTATE_END_OF_LIST()
+    },
+    .subsections = (VMStateSubsection[]) {
+        {
+            .vmsd = &icount_vmstate_timers,
+            .needed = icount_state_needed,
+        }, {
+            /* empty */
+        }
     }
 };
 
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 03/14] migration: make qemu_savevm_state public.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This makes qemu_savevm_state public for reverse-execution.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Amit Shah <amit.shah@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
---
 include/sysemu/sysemu.h | 1 +
 savevm.c                | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index d8539fd..bc52525 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -81,6 +81,7 @@ void do_info_snapshots(Monitor *mon, const QDict *qdict);
 
 void qemu_announce_self(void);
 
+int qemu_savevm_state(QEMUFile *f);
 bool qemu_savevm_state_blocked(Error **errp);
 void qemu_savevm_state_begin(QEMUFile *f,
                              const MigrationParams *params);
diff --git a/savevm.c b/savevm.c
index e19ae0a..0286f9b 100644
--- a/savevm.c
+++ b/savevm.c
@@ -779,7 +779,7 @@ void qemu_savevm_state_cancel(void)
     }
 }
 
-static int qemu_savevm_state(QEMUFile *f)
+int qemu_savevm_state(QEMUFile *f)
 {
     int ret;
     MigrationParams params = {
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 04/14] icount: introduce icount timer.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This introduces a new timer based only on instruction counter and without any
compensation.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
---
 cpus.c                 | 19 ++++++++++++-------
 include/qemu/timer.h   |  9 ++++++++-
 qemu-timer.c           |  8 +++++++-
 stubs/cpu-get-icount.c |  2 +-
 4 files changed, 28 insertions(+), 10 deletions(-)

diff --git a/cpus.c b/cpus.c
index a6b6557..6bba30b 100644
--- a/cpus.c
+++ b/cpus.c
@@ -134,7 +134,7 @@ typedef struct TimersState {
 static TimersState timers_state;
 
 /* Return the virtual CPU time, based on the instruction counter.  */
-static int64_t cpu_get_icount_locked(void)
+static int64_t cpu_get_icount_locked(int with_bias)
 {
     int64_t icount;
     CPUState *cpu = current_cpu;
@@ -146,17 +146,22 @@ static int64_t cpu_get_icount_locked(void)
         }
         icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
     }
-    return timers_state.qemu_icount_bias + (icount << icount_time_shift);
+
+    if (with_bias) {
+        return timers_state.qemu_icount_bias + (icount << icount_time_shift);
+    } else {
+        return icount << icount_time_shift;
+    }
 }
 
-int64_t cpu_get_icount(void)
+int64_t cpu_get_icount(int with_bias)
 {
     int64_t icount;
     unsigned start;
 
     do {
         start = seqlock_read_begin(&timers_state.vm_clock_seqlock);
-        icount = cpu_get_icount_locked();
+        icount = cpu_get_icount_locked(with_bias);
     } while (seqlock_read_retry(&timers_state.vm_clock_seqlock, start));
 
     return icount;
@@ -169,7 +174,7 @@ int64_t cpu_get_ticks(void)
     int64_t ticks;
 
     if (use_icount) {
-        return cpu_get_icount();
+        return cpu_get_icount(true);
     }
 
     ticks = timers_state.cpu_ticks_offset;
@@ -267,7 +272,7 @@ static void icount_adjust(void)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     cur_time = cpu_get_clock_locked();
-    cur_icount = cpu_get_icount_locked();
+    cur_icount = cpu_get_icount_locked(true);
 
     delta = cur_icount - cur_time;
     /* FIXME: This is a very crude algorithm, somewhat prone to oscillation.  */
@@ -330,7 +335,7 @@ static void icount_warp_rt(void *opaque)
              * far ahead of real time.
              */
             int64_t cur_time = cpu_get_clock_locked();
-            int64_t cur_icount = cpu_get_icount_locked();
+            int64_t cur_icount = cpu_get_icount_locked(true);
             int64_t delta = cur_time - cur_icount;
             warp_delta = MIN(warp_delta, delta);
         }
diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 7f9a074..6204cab 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -36,12 +36,19 @@
  * is suspended, and it will reflect system time changes the host may
  * undergo (e.g. due to NTP). The host clock has the same precision as
  * the virtual clock.
+ *
+ * @QEMU_CLOCK_ICOUNT: icount clock
+ *
+ * The icount clock is based on instruction counter without any compensation for
+ * speed. It will run only when instruction are executed and make only sense in
+ * icount mode.
  */
 
 typedef enum {
     QEMU_CLOCK_REALTIME = 0,
     QEMU_CLOCK_VIRTUAL = 1,
     QEMU_CLOCK_HOST = 2,
+    QEMU_CLOCK_ICOUNT = 3,
     QEMU_CLOCK_MAX
 } QEMUClockType;
 
@@ -743,7 +750,7 @@ static inline int64_t get_clock(void)
 #endif
 
 /* icount */
-int64_t cpu_get_icount(void);
+int64_t cpu_get_icount(int with_bias);
 int64_t cpu_get_clock(void);
 
 /*******************************************/
diff --git a/qemu-timer.c b/qemu-timer.c
index 00a5d35..3be80b1 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -554,7 +554,7 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
     default:
     case QEMU_CLOCK_VIRTUAL:
         if (use_icount) {
-            return cpu_get_icount();
+            return cpu_get_icount(true);
         } else {
             return cpu_get_clock();
         }
@@ -566,6 +566,12 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
+    case QEMU_CLOCK_ICOUNT:
+        if (use_icount) {
+            return cpu_get_icount(false);
+        } else {
+            return -1;
+        }
     }
 }
 
diff --git a/stubs/cpu-get-icount.c b/stubs/cpu-get-icount.c
index d685859..1968de7 100644
--- a/stubs/cpu-get-icount.c
+++ b/stubs/cpu-get-icount.c
@@ -3,7 +3,7 @@
 
 int use_icount;
 
-int64_t cpu_get_icount(void)
+int64_t cpu_get_icount(int with_bias)
 {
     abort();
 }
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 05/14] icount: check for icount clock deadline when cpu loop exits.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

Notify events on icount clock when CPU loop exits.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 cpus.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/cpus.c b/cpus.c
index 6bba30b..dc2dc21 100644
--- a/cpus.c
+++ b/cpus.c
@@ -992,6 +992,11 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
             if (deadline == 0) {
                 qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
             }
+
+            deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_ICOUNT);
+            if (deadline == 0) {
+                qemu_clock_notify(QEMU_CLOCK_ICOUNT);
+            }
         }
         qemu_tcg_wait_io_event();
     }
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 06/14] icount: make icount extra computed on icount clock as well.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This takes icount clock in account for icount extra computation so icount
clock's timers will be triggered at the exact time.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 cpus.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/cpus.c b/cpus.c
index dc2dc21..e90b9ff 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1287,6 +1287,7 @@ static int tcg_cpu_exec(CPUArchState *env)
     if (use_icount) {
         int64_t count;
         int64_t deadline;
+        int64_t icount_deadline;
         int decr;
         timers_state.qemu_icount -= (cpu->icount_decr.u16.low
                                     + cpu->icount_extra);
@@ -1303,6 +1304,15 @@ static int tcg_cpu_exec(CPUArchState *env)
             deadline = INT32_MAX;
         }
 
+        /*
+         * Take icount clock deadline in account too, and keep the nearest
+         * deadline.
+         */
+        icount_deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_ICOUNT);
+        if ((icount_deadline >= 0) && (icount_deadline < deadline)) {
+            deadline = icount_deadline;
+        }
+
         count = qemu_icount_round(deadline);
         timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 07/14] timer: add cpu_icount_to_ns function.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This adds cpu_icount_to_ns function which is needed for reverse execution.

It returns the time for a specific instruction.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 cpus.c               | 9 +++++++--
 include/qemu/timer.h | 1 +
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/cpus.c b/cpus.c
index e90b9ff..a6f092a 100644
--- a/cpus.c
+++ b/cpus.c
@@ -148,9 +148,9 @@ static int64_t cpu_get_icount_locked(int with_bias)
     }
 
     if (with_bias) {
-        return timers_state.qemu_icount_bias + (icount << icount_time_shift);
+        return timers_state.qemu_icount_bias + cpu_icount_to_ns(icount);
     } else {
-        return icount << icount_time_shift;
+        return cpu_icount_to_ns(icount);
     }
 }
 
@@ -167,6 +167,11 @@ int64_t cpu_get_icount(int with_bias)
     return icount;
 }
 
+int64_t cpu_icount_to_ns(int64_t icount)
+{
+    return icount << icount_time_shift;
+}
+
 /* return the host CPU cycle counter and handle stop/restart */
 /* Caller must hold the BQL */
 int64_t cpu_get_ticks(void)
diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 6204cab..0ae7f28 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -752,6 +752,7 @@ static inline int64_t get_clock(void)
 /* icount */
 int64_t cpu_get_icount(int with_bias);
 int64_t cpu_get_clock(void);
+int64_t cpu_icount_to_ns(int64_t icount);
 
 /*******************************************/
 /* host CPU ticks (if available) */
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 08/14] icount: Add QemuOpts for icount

[fred.konrad]

From: Sebastian Tanase <sebastian.tanase@openwide.fr>

Make icount parameter use QemuOpts style options in order
to easily add other suboptions.

Signed-off-by: Sebastian Tanase <sebastian.tanase@openwide.fr>
Tested-by: Camille Bégué <camille.begue@openwide.fr>
---
 cpus.c                | 10 +++++++++-
 include/qemu-common.h |  3 ++-
 qemu-options.hx       |  4 ++--
 qtest.c               | 13 +++++++++++--
 vl.c                  | 35 ++++++++++++++++++++++++++++-------
 5 files changed, 52 insertions(+), 13 deletions(-)

diff --git a/cpus.c b/cpus.c
index a6f092a..9519711 100644
--- a/cpus.c
+++ b/cpus.c
@@ -478,13 +478,21 @@ static const VMStateDescription vmstate_timers = {
     }
 };
 
-void configure_icount(const char *option)
+void configure_icount(QemuOpts *opts, Error **errp)
 {
+    const char *option;
+
     seqlock_init(&timers_state.vm_clock_seqlock, NULL);
     vmstate_register(NULL, 0, &vmstate_timers, &timers_state);
+    option = qemu_opt_get(opts, "shift");
     if (!option) {
         return;
     }
+    /* When using -icount shift, the shift option will be
+       misinterpreted as a boolean */
+    if (strcmp(option, "on") == 0 || strcmp(option, "off") == 0) {
+        error_setg(errp, "The shift option must be a number or auto");
+    }
 
     icount_warp_timer = timer_new_ns(QEMU_CLOCK_REALTIME,
                                           icount_warp_rt, NULL);
diff --git a/include/qemu-common.h b/include/qemu-common.h
index ae76197..cc346ec 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -41,6 +41,7 @@
 #include <assert.h>
 #include <signal.h>
 #include "glib-compat.h"
+#include "qemu/option.h"
 
 #ifdef _WIN32
 #include "sysemu/os-win32.h"
@@ -105,7 +106,7 @@ static inline char *realpath(const char *path, char *resolved_path)
 #endif
 
 /* icount */
-void configure_icount(const char *option);
+void configure_icount(QemuOpts *opts, Error **errp);
 extern int use_icount;
 
 #include "qemu/osdep.h"
diff --git a/qemu-options.hx b/qemu-options.hx
index 9e54686..143def4 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3011,11 +3011,11 @@ re-inject them.
 ETEXI
 
 DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
-    "-icount [N|auto]\n" \
+    "-icount [shift=N|auto]\n" \
     "                enable virtual instruction counter with 2^N clock ticks per\n" \
     "                instruction\n", QEMU_ARCH_ALL)
 STEXI
-@item -icount [@var{N}|auto]
+@item -icount [shift=@var{N}|auto]
 @findex -icount
 Enable virtual instruction counter.  The virtual cpu will execute one
 instruction every 2^@var{N} ns of virtual time.  If @code{auto} is specified
diff --git a/qtest.c b/qtest.c
index 04a6dc1..ef0d991 100644
--- a/qtest.c
+++ b/qtest.c
@@ -19,6 +19,9 @@
 #include "hw/irq.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/cpus.h"
+#include "qemu/config-file.h"
+#include "qemu/option.h"
+#include "qemu/error-report.h"
 
 #define MAX_IRQ 256
 
@@ -509,10 +512,16 @@ static void qtest_event(void *opaque, int event)
     }
 }
 
-int qtest_init_accel(MachineClass *mc)
+static void configure_qtest_icount(const char *options)
 {
-    configure_icount("0");
+    QemuOpts *opts  = qemu_opts_parse(qemu_find_opts("icount"), options, 1);
+    configure_icount(opts, &error_abort);
+    qemu_opts_del(opts);
+}
 
+int qtest_init_accel(MachineClass *mc)
+{
+    configure_qtest_icount("0");
     return 0;
 }
 
diff --git a/vl.c b/vl.c
index 6e084c2..4fa732b 100644
--- a/vl.c
+++ b/vl.c
@@ -537,6 +537,20 @@ static QemuOptsList qemu_mem_opts = {
     },
 };
 
+static QemuOptsList qemu_icount_opts = {
+    .name = "icount",
+    .implied_opt_name = "shift",
+    .merge_lists = true,
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_icount_opts.head),
+    .desc = {
+        {
+            .name = "shift",
+            .type = QEMU_OPT_STRING,
+        },
+        { /* end of list */ }
+    },
+};
+
 /**
  * Get machine options
  *
@@ -2898,13 +2912,12 @@ int main(int argc, char **argv, char **envp)
 {
     int i;
     int snapshot, linux_boot;
-    const char *icount_option = NULL;
     const char *initrd_filename;
     const char *kernel_filename, *kernel_cmdline;
     const char *boot_order;
     DisplayState *ds;
     int cyls, heads, secs, translation;
-    QemuOpts *hda_opts = NULL, *opts, *machine_opts;
+    QemuOpts *hda_opts = NULL, *opts, *machine_opts, *icount_opts = NULL;
     QemuOptsList *olist;
     int optind;
     const char *optarg;
@@ -2969,6 +2982,7 @@ int main(int argc, char **argv, char **envp)
     qemu_add_opts(&qemu_msg_opts);
     qemu_add_opts(&qemu_name_opts);
     qemu_add_opts(&qemu_numa_opts);
+    qemu_add_opts(&qemu_icount_opts);
 
     runstate_init();
 
@@ -3819,7 +3833,11 @@ int main(int argc, char **argv, char **envp)
                 }
                 break;
             case QEMU_OPTION_icount:
-                icount_option = optarg;
+                icount_opts = qemu_opts_parse(qemu_find_opts("icount"),
+                                              optarg, 1);
+                if (!icount_opts) {
+                    exit(1);
+                }
                 break;
             case QEMU_OPTION_incoming:
                 incoming = optarg;
@@ -4295,11 +4313,14 @@ int main(int argc, char **argv, char **envp)
     qemu_spice_init();
 #endif
 
-    if (icount_option && (kvm_enabled() || xen_enabled())) {
-        fprintf(stderr, "-icount is not allowed with kvm or xen\n");
-        exit(1);
+    if (icount_opts) {
+        if (kvm_enabled() || xen_enabled()) {
+            fprintf(stderr, "-icount is not allowed with kvm or xen\n");
+            exit(1);
+        }
+        configure_icount(icount_opts, &error_abort);
+        qemu_opts_del(icount_opts);
     }
-    configure_icount(icount_option);
 
     /* clean up network at qemu process termination */
     atexit(&net_cleanup);
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 09/14] trace-events: add reverse-execution events.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This adds some trace-events for reverse execution.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
---
 trace-events | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/trace-events b/trace-events
index 11a17a8..66f9e08 100644
--- a/trace-events
+++ b/trace-events
@@ -1301,3 +1301,9 @@ mhp_pc_dimm_assigned_address(uint64_t addr) "0x%"PRIx64
 # target-s390x/kvm.c
 kvm_enable_cmma(int rc) "CMMA: enabling with result code %d"
 kvm_clear_cmma(int rc) "CMMA: clearing with result code %d"
+
+# reverse-execution.c
+snap_callback(uint64_t time, const char *filename) "snapshot @%li -> %s"
+stop_callback(uint64_t time) "stopping @%li"
+rexec_stop_reverse_continue(void) "stop reverse continue."
+rexec_step_backward(uint64_t time, uint64_t stop) "stepping backward @%li stop @%li"
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 10/14] introduce reverse execution mechanism.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This introduces the basic reverse-execution mechanism.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
---
 Makefile.target             |   1 +
 cpus.c                      |   6 +
 include/reverse-execution.h |  41 ++++++
 reverse-execution.c         | 308 ++++++++++++++++++++++++++++++++++++++++++++
 vl.c                        |   7 +-
 5 files changed, 362 insertions(+), 1 deletion(-)
 create mode 100644 include/reverse-execution.h
 create mode 100644 reverse-execution.c

diff --git a/Makefile.target b/Makefile.target
index 137d0b0..a2cb224 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -120,6 +120,7 @@ endif #CONFIG_BSD_USER
 # System emulator target
 ifdef CONFIG_SOFTMMU
 obj-y += arch_init.o cpus.o monitor.o gdbstub.o balloon.o ioport.o numa.o
+obj-y += reverse-execution.o
 obj-y += qtest.o
 obj-y += hw/
 obj-$(CONFIG_FDT) += device_tree.o
diff --git a/cpus.c b/cpus.c
index 9519711..f812564 100644
--- a/cpus.c
+++ b/cpus.c
@@ -63,6 +63,8 @@
 
 #endif /* CONFIG_LINUX */
 
+#include "reverse-execution.h"
+
 static CPUState *next_cpu;
 
 bool cpu_is_stopped(CPUState *cpu)
@@ -600,7 +602,11 @@ static bool cpu_can_run(CPUState *cpu)
 
 static void cpu_handle_guest_debug(CPUState *cpu)
 {
+    if (rexec_is_continuing_backward()) {
+        rexec_step_done();
+    }
     gdb_set_stop_cpu(cpu);
+    rexec_stop_stepping_back_mode();
     qemu_system_debug_request();
     cpu->stopped = true;
 }
diff --git a/include/reverse-execution.h b/include/reverse-execution.h
new file mode 100644
index 0000000..9951549
--- /dev/null
+++ b/include/reverse-execution.h
@@ -0,0 +1,41 @@
+/*
+ *  reverse execution.
+ *
+ *  Copyright (C) 2014 : GreenSocs Ltd
+ *      http://www.greensocs.com/ , email: info@greensocs.com
+ *
+ *  Developed by :
+ *  Frederic Konrad   <fred.konrad@greensocs.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#ifndef REVERSE_EXECUTION
+#define REVERSE_EXECUTION
+
+void rexec_setup(void);
+void rexec_step_backward(CPUState *cpu, uint64_t steps);
+void rexec_stop_stepping_back_mode(void);
+void rexec_continue_backward(CPUState *cpu);
+int rexec_is_continuing_backward(void);
+void rexec_next_reverse_continue_step(void);
+void rexec_stop_reverse_continue(void);
+void rexec_step_done(void);
+bool rexec_is_step_done(void);
+bool rexec_is_enabled(void);
+void rexec_cleanup(void);
+bool rexec_dbg_requested(void);
+
+#endif /* REVERSE_EXECUTION */
diff --git a/reverse-execution.c b/reverse-execution.c
new file mode 100644
index 0000000..4c97f39
--- /dev/null
+++ b/reverse-execution.c
@@ -0,0 +1,308 @@
+/*
+ *  reverse execution.
+ *
+ *  Copyright (C) 2014 : GreenSocs Ltd
+ *      http://www.greensocs.com/ , email: info@greensocs.com
+ *
+ *  Developed by :
+ *  Frederic Konrad   <fred.konrad@greensocs.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#include "qemu/timer.h"
+#include "sysemu/sysemu.h"
+#include "migration/qemu-file.h"
+
+#include "reverse-execution.h"
+
+#include "trace.h"  /* needed for trace event prototype */
+
+#define SNAPSHOT_FILENAME "snapshot-step"
+
+typedef struct snapshot_entry {
+    uint32_t id;
+    int64_t time;
+    QLIST_ENTRY(snapshot_entry) next;
+} snapshot_entry;
+
+static QLIST_HEAD(, snapshot_entry) snapshot = QLIST_HEAD_INITIALIZER(snapshot);
+
+QEMUTimer *snapshot_timer;
+QEMUTimer *stop_timer;
+
+struct rexec_state {
+    bool stepping_back;
+    bool continue_backward_mode;
+    bool singlestep_was_enabled;
+    bool step_done;
+    bool stop_requested;
+};
+
+static bool rexec_enabled;
+struct rexec_state rexec_state;
+
+static snapshot_entry *new_snapshot(void)
+{
+    snapshot_entry *snap = NULL;
+    snap = g_malloc(sizeof(snapshot_entry));
+    assert(snap);
+
+    if (QLIST_FIRST(&snapshot) != NULL) {
+        snap->id = QLIST_FIRST(&snapshot)->id + 1;
+    } else {
+        snap->id = 0;
+    }
+
+    QLIST_INSERT_HEAD(&snapshot, snap, next);
+    return snap;
+}
+
+/*
+ * Timer callback called when a snapshot must be done.
+ */
+static void snap_callback(void *opaque)
+{
+    QEMUFile *file = NULL;
+    int saved_vm_running;
+    snapshot_entry *snap = NULL;
+    CPUArchState *cpu = NULL;
+    char filename[20];
+
+    cpu = qemu_get_cpu(0)->env_ptr;
+    assert(cpu != NULL);
+
+    if (!rexec_state.stepping_back) {
+        snap = new_snapshot();
+
+        saved_vm_running = runstate_is_running();
+        vm_stop(RUN_STATE_SAVE_VM);
+        snap->time = qemu_clock_get_ns(QEMU_CLOCK_ICOUNT);
+        sprintf(filename, "%s%04u", SNAPSHOT_FILENAME, snap->id);
+
+        trace_snap_callback(snap->time, filename);
+
+        file = qemu_fopen(filename, "wb");
+        qemu_savevm_state(file);
+        qemu_fclose(file);
+
+        if (saved_vm_running) {
+            vm_start();
+        }
+        timer_mod_ns(snapshot_timer, snap->time + 100000000);
+    }
+}
+
+/*
+ * Timer callback called when the VM have to stop.
+ */
+static void stop_callback(void *opaque)
+{
+    trace_stop_callback(qemu_clock_get_ns(QEMU_CLOCK_ICOUNT));
+    rexec_state.stop_requested = true;
+}
+
+void rexec_setup(void)
+{
+    snapshot_timer = timer_new_ns(QEMU_CLOCK_ICOUNT, snap_callback, NULL);
+    stop_timer = timer_new_ns(QEMU_CLOCK_ICOUNT, stop_callback, NULL);
+
+    timer_mod_ns(snapshot_timer, qemu_clock_get_ns(QEMU_CLOCK_ICOUNT));
+    rexec_enabled = true;
+    rexec_state.stepping_back = false;
+    rexec_state.continue_backward_mode = false;
+    rexec_state.stop_requested = false;
+}
+
+void rexec_stop_stepping_back_mode(void)
+{
+    if (rexec_state.stepping_back) {
+        singlestep = rexec_state.singlestep_was_enabled;
+        rexec_state.stepping_back = false;
+    }
+    rexec_state.stop_requested = false;
+}
+
+static void rexec_start_stepping_back_mode(CPUState *cpu)
+{
+    assert(!rexec_state.stepping_back);
+    /*
+     * Flushing tb.
+     * FIXME: might not be necessary with counter.
+     */
+    tb_flush(cpu->env_ptr);
+
+    /*
+     * Single step to the right PC.
+     */
+    rexec_state.singlestep_was_enabled = singlestep;
+    singlestep = 1;
+
+    rexec_state.stepping_back = true;
+}
+
+/**
+ * \func rexec_step_backward
+ * \param cpu GDBStub's cpu.
+ * \param steps Number of steps to step back.
+ * \brief Steps backward: "reverse-step" in GDB.
+ *
+ */
+void rexec_step_backward(CPUState *cpu, uint64_t steps)
+{
+    QEMUFile *file = NULL;
+    char filename[256];
+    snapshot_entry *snap = QLIST_FIRST(&snapshot);
+
+    int64_t stop_time = qemu_clock_get_ns(QEMU_CLOCK_ICOUNT)
+                      - cpu_icount_to_ns(steps);
+
+    /*
+     * FIXME: Remove the file?
+     */
+    while ((stop_time > 0) && ((snap = QLIST_FIRST(&snapshot)) != NULL)
+                           && (snap->time >= stop_time)) {
+        /*
+         * Remove the snapshot from the list and mod the snapshot timer to its
+         * time. This will cause the snapshot to be taken at the same value in
+         * case of a forward execution.
+         */
+        QLIST_REMOVE(snap, next);
+        timer_mod_ns(snapshot_timer, snap->time);
+        g_free(snap);
+    }
+
+    if ((stop_time <= 0) || (snap == NULL)) {
+        /*
+         * This happens when an instruction behind the first snapshot is asked.
+         * Just trigger a debug event so it won't move.
+         */
+        rexec_state.stop_requested = true;
+        vm_start();
+        return;
+    }
+
+    sprintf(filename, "%s%04u", SNAPSHOT_FILENAME, snap->id);
+
+    /*
+     * Load the previous state.
+     */
+    vm_stop(RUN_STATE_RESTORE_VM);
+
+    trace_rexec_step_backward(qemu_clock_get_ns(QEMU_CLOCK_ICOUNT), stop_time);
+
+    file = qemu_fopen(filename, "rb");
+    qemu_loadvm_state(file);
+    qemu_fclose(file);
+
+    /*
+     * Mod the timer so it will stop at the exact instruction.
+     */
+    timer_mod_ns(stop_timer, stop_time);
+
+    rexec_start_stepping_back_mode(cpu);
+    /*
+     * Restart the vm.
+     */
+    vm_start();
+}
+
+/**
+ * \func rexec_continue_backward
+ * \brief Continue execution backward.
+ * \param cpu GDB's stub cpu.
+ *
+ */
+void rexec_continue_backward(CPUState *cpu)
+{
+    rexec_state.continue_backward_mode = true;
+    rexec_state.step_done = false;
+    rexec_step_backward(cpu, 1);
+}
+
+/**
+ * \func rexec_is_continuing_backward
+ * \brief Check if we are continuing backward.
+ * \return Return true if we are continuing backward.
+ *
+ */
+int rexec_is_continuing_backward(void)
+{
+    return rexec_state.continue_backward_mode;
+}
+
+void rexec_next_reverse_continue_step(void)
+{
+    CPUState *cpu = qemu_get_cpu(0);
+
+    assert(cpu != NULL);
+    rexec_state.step_done = false;
+
+    /*
+     * FIXME:
+     *         - Stop at breakpoint in reverse order.
+     *         - The reverse execution speed is not constant as the snapshot
+     *           replay is not constant.
+     */
+    rexec_step_backward(cpu, 10000000);
+}
+
+void rexec_stop_reverse_continue(void)
+{
+    if (rexec_state.continue_backward_mode) {
+        trace_rexec_stop_reverse_continue();
+        rexec_state.continue_backward_mode = false;
+        rexec_state.step_done = false;
+        rexec_stop_stepping_back_mode();
+    }
+}
+
+void rexec_step_done(void)
+{
+    rexec_state.step_done = true;
+}
+
+bool rexec_is_step_done(void)
+{
+    return rexec_state.step_done;
+}
+
+bool rexec_is_enabled(void)
+{
+    return rexec_enabled;
+}
+
+void rexec_cleanup(void)
+{
+    snapshot_entry *snap = QLIST_FIRST(&snapshot);
+
+    /*
+     * FIXME: Remove the file?
+     */
+    while ((snap = QLIST_FIRST(&snapshot)) != NULL) {
+        /*
+         * Remove the snapshot from the list and mod the snapshot timer to its
+         * time. This will cause the snapshot to be taken at the same value in
+         * case of a forward execution.
+         */
+        QLIST_REMOVE(snap, next);
+        g_free(snap);
+    }
+}
+
+bool rexec_dbg_requested(void)
+{
+    return rexec_state.stop_requested;
+}
diff --git a/vl.c b/vl.c
index 4fa732b..18df07c 100644
--- a/vl.c
+++ b/vl.c
@@ -119,6 +119,8 @@ int main(int argc, char **argv)
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
 
+#include "reverse-execution.h"
+
 #define DEFAULT_RAM_SIZE 128
 
 #define MAX_VIRTIO_CONSOLES 1
@@ -1967,7 +1969,7 @@ void qemu_system_debug_request(void)
 static bool main_loop_should_exit(void)
 {
     RunState r;
-    if (qemu_debug_requested()) {
+    if (qemu_debug_requested() && !rexec_is_continuing_backward()) {
         vm_stop(RUN_STATE_DEBUG);
     }
     if (qemu_suspend_requested()) {
@@ -2017,6 +2019,9 @@ static void main_loop(void)
     int64_t ti;
 #endif
     do {
+        if (rexec_is_continuing_backward() && rexec_is_step_done()) {
+            rexec_next_reverse_continue_step();
+        }
         nonblocking = !kvm_enabled() && !xen_enabled() && last_io > 0;
 #ifdef CONFIG_PROFILER
         ti = profile_getclock();
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 11/14] gdbstub: allow reverse execution in gdb stub.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This allows gdb to reverse step QEMU: reverse-stepi and reverse-cont commands
are allowed.

When step_backward is called, QEMU restores a snapshot before the actual
instruction and stops (with a debug exit) when the previous instruction is
reached.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
---
 gdbstub.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)

diff --git a/gdbstub.c b/gdbstub.c
index 8afe0b7..4564988 100644
--- a/gdbstub.c
+++ b/gdbstub.c
@@ -33,6 +33,7 @@
 #include "sysemu/char.h"
 #include "sysemu/sysemu.h"
 #include "exec/gdbstub.h"
+#include "reverse-execution.h"
 #endif
 
 #define MAX_PACKET_LENGTH 4096
@@ -1113,6 +1114,17 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
             if (cc->gdb_core_xml_file != NULL) {
                 pstrcat(buf, sizeof(buf), ";qXfer:features:read+");
             }
+
+            #ifndef CONFIG_USER_ONLY
+            /*
+             * When reverse execution is enabled those additional features must
+             * be set so GDB allows reverse-stepi and reverse-continue command.
+             */
+            if (rexec_is_enabled()) {
+                pstrcat(buf, sizeof(buf), ";ReverseStep+;ReverseContinue+");
+            }
+            #endif /* !CONFIG_USER_ONLY */
+
             put_packet(s, buf);
             break;
         }
@@ -1161,7 +1173,23 @@ static int gdb_handle_packet(GDBState *s, const char *line_buf)
         }
         /* Unrecognised 'q' command.  */
         goto unknown_command;
-
+    #ifndef CONFIG_USER_ONLY
+    case 'b':
+        /* Reverse execution. */
+        switch (*p) {
+        case 's':
+            rexec_step_backward(s->c_cpu, 1);
+            break;
+        case 'c':
+            rexec_continue_backward(s->c_cpu);
+            break;
+        default:
+            buf[0] = '\0';
+            put_packet(s, buf);
+            break;
+        }
+        break;
+    #endif /* !CONFIG_USER_ONLY */
     default:
     unknown_command:
         /* put empty packet */
@@ -1221,6 +1249,7 @@ static void gdb_vm_state_change(void *opaque, int running, RunState state)
         ret = GDB_SIGNAL_TRAP;
         break;
     case RUN_STATE_PAUSED:
+        rexec_stop_reverse_continue();
         ret = GDB_SIGNAL_INT;
         break;
     case RUN_STATE_SHUTDOWN:
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 12/14] cpu-exec: trigger a debug request when rexec stops.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This allows QEMU to trigger a debug exception when rexec_dbg_requested
is set.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
---
 cpu-exec.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/cpu-exec.c b/cpu-exec.c
index 38e5f02..82e2568 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -22,6 +22,7 @@
 #include "tcg.h"
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
+#include "reverse-execution.h"
 
 void cpu_loop_exit(CPUState *cpu)
 {
@@ -230,6 +231,18 @@ int cpu_exec(CPUArchState *env)
     /* This must be volatile so it is not trashed by longjmp() */
     volatile bool have_tb_lock = false;
 
+    #ifndef CONFIG_USER_ONLY
+    if (rexec_is_enabled() && rexec_dbg_requested()) {
+        /*
+         * Reverse execution need to stop right now.
+         * So just generate a EXCP_DEBUG.
+         */
+        cpu->exception_index = EXCP_DEBUG;
+        cpu_handle_debug_exception(env);
+        return EXCP_DEBUG;
+    }
+    #endif /* !CONFIG_USER_ONLY */
+
     if (cpu->halted) {
         if (!cpu_has_work(cpu)) {
             return EXCP_HALTED;
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 13/14] rexec: synchronize icount on the next event.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

We don't want to warp on host clock as it is not deterministic for replay.
So this patch warp icount on the next QEMU_VIRTUAL_CLOCK event if reverse
execution is enabled.

The normal behaviour is kept when reverse execution is disabled.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
---
 cpus.c                         | 19 +++++++++++++++++--
 include/qemu/timer.h           |  8 ++++++++
 include/reverse-execution.h    |  2 ++
 main-loop.c                    | 10 ++++++++++
 stubs/Makefile.objs            |  1 +
 stubs/cpu-get-icount.c         |  8 ++++++++
 stubs/reverse-execution-stub.c | 32 ++++++++++++++++++++++++++++++++
 7 files changed, 78 insertions(+), 2 deletions(-)
 create mode 100644 stubs/reverse-execution-stub.c

diff --git a/cpus.c b/cpus.c
index f812564..740c6e5 100644
--- a/cpus.c
+++ b/cpus.c
@@ -321,8 +321,10 @@ static int64_t qemu_icount_round(int64_t count)
     return (count + (1 << icount_time_shift) - 1) >> icount_time_shift;
 }
 
-static void icount_warp_rt(void *opaque)
+void icount_warp_rt(void *opaque)
 {
+    int64_t next_vm_deadline = -1;
+
     /* The icount_warp_timer is rescheduled soon after vm_clock_warp_start
      * changes from -1 to another value, so the race here is okay.
      */
@@ -330,6 +332,13 @@ static void icount_warp_rt(void *opaque)
         return;
     }
 
+    if (rexec_is_enabled()) {
+        /*
+         * We need this because the standard warp_delta is not deterministic.
+         */
+        next_vm_deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
+    }
+
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
         int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
@@ -346,7 +355,13 @@ static void icount_warp_rt(void *opaque)
             int64_t delta = cur_time - cur_icount;
             warp_delta = MIN(warp_delta, delta);
         }
-        timers_state.qemu_icount_bias += warp_delta;
+        if (rexec_is_enabled()) {
+            if (next_vm_deadline > 0) {
+                timers_state.qemu_icount_bias += next_vm_deadline;
+            }
+        } else {
+            timers_state.qemu_icount_bias += warp_delta;
+        }
     }
     vm_clock_warp_start = -1;
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);
diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 0ae7f28..de2641a 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -754,6 +754,14 @@ int64_t cpu_get_icount(int with_bias);
 int64_t cpu_get_clock(void);
 int64_t cpu_icount_to_ns(int64_t icount);
 
+/**
+ * void icount_warp_rt:
+ *
+ * Move icount to the realtime clock or to the next QEMU_VIRTUAL_CLOCK event
+ * when reverse execution is enabled.
+ */
+void icount_warp_rt(void *opaque);
+
 /*******************************************/
 /* host CPU ticks (if available) */
 
diff --git a/include/reverse-execution.h b/include/reverse-execution.h
index 9951549..871d5ee 100644
--- a/include/reverse-execution.h
+++ b/include/reverse-execution.h
@@ -25,6 +25,8 @@
 #ifndef REVERSE_EXECUTION
 #define REVERSE_EXECUTION
 
+#include "qom/cpu.h"
+
 void rexec_setup(void);
 void rexec_step_backward(CPUState *cpu, uint64_t steps);
 void rexec_stop_stepping_back_mode(void);
diff --git a/main-loop.c b/main-loop.c
index 8a85493..0e4404b 100644
--- a/main-loop.c
+++ b/main-loop.c
@@ -34,6 +34,8 @@
 
 #include "qemu/compatfd.h"
 
+#include "reverse-execution.h"
+
 /* If we have signalfd, we mask out the signals we want to handle and then
  * use signalfd to listen for them.  We rely on whatever the current signal
  * handler is to dispatch the signals when we receive them.
@@ -489,6 +491,14 @@ int main_loop_wait(int nonblocking)
 
     qemu_clock_run_all_timers();
 
+    /*
+     * Sometimes deadlock can appears because there is no pending event on
+     * virtual clock.
+     */
+    if (rexec_is_enabled()) {
+        icount_warp_rt(NULL);
+    }
+
     return ret;
 }
 
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 528e161..2192e45 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -39,3 +39,4 @@ stub-obj-$(CONFIG_WIN32) += fd-register.o
 stub-obj-y += cpus.o
 stub-obj-y += kvm.o
 stub-obj-y += qmp_pc_dimm_device_list.o
+stub-obj-y += reverse-execution-stub.o
diff --git a/stubs/cpu-get-icount.c b/stubs/cpu-get-icount.c
index 1968de7..d35948f 100644
--- a/stubs/cpu-get-icount.c
+++ b/stubs/cpu-get-icount.c
@@ -7,3 +7,11 @@ int64_t cpu_get_icount(int with_bias)
 {
     abort();
 }
+
+void icount_warp_rt(void *opaque)
+{
+    /*
+     * Should not happen, as rexec_is_enabled() always return false.
+     */
+    abort();
+}
diff --git a/stubs/reverse-execution-stub.c b/stubs/reverse-execution-stub.c
new file mode 100644
index 0000000..e417d31
--- /dev/null
+++ b/stubs/reverse-execution-stub.c
@@ -0,0 +1,32 @@
+/*
+ *  reverse-execution-stub.c
+ *
+ *  Copyright (C) 2014 : GreenSocs Ltd
+ *      http://www.greensocs.com/ , email: info@greensocs.com
+ *
+ *  Developed by :
+ *  Frederic Konrad   <fred.konrad@greensocs.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+#include <stdbool.h>
+
+bool rexec_is_enabled(void);
+
+bool rexec_is_enabled(void)
+{
+    return false;
+}
-- 
1.9.0

[Qemu-devel] [RFC PATCH v6 14/14] rexec: allow to enable reverse execution.

[fred.konrad]

From: KONRAD Frederic <fred.konrad@greensocs.com>

This creates QEMU options for reverse execution.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
---
 cpus.c          | 15 +++++++++++++++
 qemu-options.hx |  6 +++++-
 vl.c            |  6 ++++++
 3 files changed, 26 insertions(+), 1 deletion(-)

diff --git a/cpus.c b/cpus.c
index 740c6e5..ba5f8f5 100644
--- a/cpus.c
+++ b/cpus.c
@@ -498,6 +498,7 @@ static const VMStateDescription vmstate_timers = {
 void configure_icount(QemuOpts *opts, Error **errp)
 {
     const char *option;
+    bool enable_rexec;
 
     seqlock_init(&timers_state.vm_clock_seqlock, NULL);
     vmstate_register(NULL, 0, &vmstate_timers, &timers_state);
@@ -513,12 +514,26 @@ void configure_icount(QemuOpts *opts, Error **errp)
 
     icount_warp_timer = timer_new_ns(QEMU_CLOCK_REALTIME,
                                           icount_warp_rt, NULL);
+
+    enable_rexec = qemu_opt_get_bool(opts, "rexec", false);
+
     if (strcmp(option, "auto") != 0) {
         icount_time_shift = strtol(option, NULL, 0);
         use_icount = 1;
+
+        if (enable_rexec) {
+            rexec_setup();
+        }
         return;
     }
 
+    /*
+     * We don't allow to run reverse execution with use_icount != 1.
+     */
+    if (enable_rexec) {
+        error_setg(errp, "Reverse execution requires icount in fixed mode.");
+    }
+
     use_icount = 2;
 
     /* 125MIPS seems a reasonable initial guess at the guest speed.
diff --git a/qemu-options.hx b/qemu-options.hx
index 143def4..ee1fdf4 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3011,7 +3011,7 @@ re-inject them.
 ETEXI
 
 DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
-    "-icount [shift=N|auto]\n" \
+    "-icount [shift=N|auto][,rexec=on|off]\n" \
     "                enable virtual instruction counter with 2^N clock ticks per\n" \
     "                instruction\n", QEMU_ARCH_ALL)
 STEXI
@@ -3026,6 +3026,10 @@ Note that while this option can give deterministic behavior, it does not
 provide cycle accurate emulation.  Modern CPUs contain superscalar out of
 order cores with complex cache hierarchies.  The number of instructions
 executed often has little or no correlation with actual performance.
+
+@option{rexec=on} will enable reverse execution. A snapshot is taken regularly
+and will allow to replay the execution when gdb reverse-step or reverse-cont
+command are used. This requires shift!=auto.
 ETEXI
 
 DEF("watchdog", HAS_ARG, QEMU_OPTION_watchdog, \
diff --git a/vl.c b/vl.c
index 18df07c..69924cd 100644
--- a/vl.c
+++ b/vl.c
@@ -548,6 +548,9 @@ static QemuOptsList qemu_icount_opts = {
         {
             .name = "shift",
             .type = QEMU_OPT_STRING,
+        }, {
+            .name = "rexec",
+            .type = QEMU_OPT_BOOL,
         },
         { /* end of list */ }
     },
@@ -4561,5 +4564,8 @@ int main(int argc, char **argv, char **envp)
     tpm_cleanup();
 #endif
 
+    if (rexec_is_enabled()) {
+        rexec_cleanup();
+    }
     return 0;
 }
-- 
1.9.0

Re: [Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Paolo Bonzini]

Il 03/07/2014 16:33, fred.konrad@greensocs.com ha scritto:
> From: KONRAD Frederic <fred.konrad@greensocs.com>
> 
> Hi everybody,
> 
> This is the sixth version of this RFC (see the changes below).
> 
> Those are the two first patch-set we have been worked on for reverse execution.
> 
> The first part is fully reviewed except the "icount: introduce icount timer"
> patch maybe we can merge them?
> 
> The first series:
>   icount: put icount variables into TimerState.
>   migration: migrate icount fields.
>   migration: make qemu_savevm_state public.
>   icount: introduce icount timer.
>   icount: check for icount clock deadline when cpu loop exits.
>   icount: make icount extra computed on icount clock as well.
>   timer: add cpu_icount_to_ns function.
> 
> are various preparation patches for reverse execution.
> 
> The last patches:
>   trace-events: add reverse-execution events.
>   introduce reverse execution mechanism.
>   gdbstub: allow reverse execution in gdb stub.
>   cpu-exec: trigger a debug request when rexec stops.
>   rexec: synchronize icount on the next event.
>   rexec: allow to enable reverse execution.
> 
> are reverse execution introduction.
> 
> They can be clone at: git://git.greensocs.com/qemu_cexe.git:cexe_2_3_v6
> 
> The third series will be sent as soon as possible and have some issues with
> QEMU's thread as it use fork.
> 
> This implementation of reverse execution works with instruction counting:
> 
> A new clock is implemented which is icount clock. It grows each time an
> instruction is executed and is totally independant of host clock.
> 
> Snapshots are taken regularly (based on icount clock) with help of migration
> code and written on the disk.
> 
> When user wants to use reverse-stepi:
>  * Last snapshot is reloaded.
>  * A stop callback is created to be triggered at the previous instruction.
> 
> This stop callback generates a debug exception so QEMU stops in debug mode.
> 
> Command line:
>  * rexec suboption is added to icount to enable reverse execution, it needs
>    icount=N and doesn't support auto mode.
> 
> About non determinism in QEMU:
>  * This implementation doesn't take IO in account so any IO will cause non
>    determinism and break reverse execution.
> 
>  * The icount warp mechanism have been disabled when reverse execution is
>    enabled so the time grow differently inside the VM.
> 
> Testing:
>  * It has been tested on ARM without any IO such as network or asynchronous file
>    access to keep the deterministic behaviour of icount.
> 
> Known issues:
>  * On ARM stepi seems to do some additional steps which are added to icount
>    counter so reverse-stepi just after stepi is broken.
> 
>  * The IO replay explained above.

Hi, can you rebase and repost these patches?  It would be nice to have a
discussion of the different approaches to record/replay in your patches
and Pavel's (perhaps before Pavel presents at KVM Forum).

Paolo

Re: [Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Frederic Konrad]

On 01/09/2014 18:22, Paolo Bonzini wrote:
> Il 03/07/2014 16:33, fred.konrad@greensocs.com ha scritto:
>> From: KONRAD Frederic <fred.konrad@greensocs.com>
>>
>> Hi everybody,
>>
>> This is the sixth version of this RFC (see the changes below).
>>
>> Those are the two first patch-set we have been worked on for reverse execution.
>>
>> The first part is fully reviewed except the "icount: introduce icount timer"
>> patch maybe we can merge them?
>>
>> The first series:
>>    icount: put icount variables into TimerState.
>>    migration: migrate icount fields.
>>    migration: make qemu_savevm_state public.
>>    icount: introduce icount timer.
>>    icount: check for icount clock deadline when cpu loop exits.
>>    icount: make icount extra computed on icount clock as well.
>>    timer: add cpu_icount_to_ns function.
>>
>> are various preparation patches for reverse execution.
>>
>> The last patches:
>>    trace-events: add reverse-execution events.
>>    introduce reverse execution mechanism.
>>    gdbstub: allow reverse execution in gdb stub.
>>    cpu-exec: trigger a debug request when rexec stops.
>>    rexec: synchronize icount on the next event.
>>    rexec: allow to enable reverse execution.
>>
>> are reverse execution introduction.
>>
>> They can be clone at: git://git.greensocs.com/qemu_cexe.git:cexe_2_3_v6
>>
>> The third series will be sent as soon as possible and have some issues with
>> QEMU's thread as it use fork.
>>
>> This implementation of reverse execution works with instruction counting:
>>
>> A new clock is implemented which is icount clock. It grows each time an
>> instruction is executed and is totally independant of host clock.
>>
>> Snapshots are taken regularly (based on icount clock) with help of migration
>> code and written on the disk.
>>
>> When user wants to use reverse-stepi:
>>   * Last snapshot is reloaded.
>>   * A stop callback is created to be triggered at the previous instruction.
>>
>> This stop callback generates a debug exception so QEMU stops in debug mode.
>>
>> Command line:
>>   * rexec suboption is added to icount to enable reverse execution, it needs
>>     icount=N and doesn't support auto mode.
>>
>> About non determinism in QEMU:
>>   * This implementation doesn't take IO in account so any IO will cause non
>>     determinism and break reverse execution.
>>
>>   * The icount warp mechanism have been disabled when reverse execution is
>>     enabled so the time grow differently inside the VM.
>>
>> Testing:
>>   * It has been tested on ARM without any IO such as network or asynchronous file
>>     access to keep the deterministic behaviour of icount.
>>
>> Known issues:
>>   * On ARM stepi seems to do some additional steps which are added to icount
>>     counter so reverse-stepi just after stepi is broken.
>>
>>   * The IO replay explained above.
> Hi, can you rebase and repost these patches?  It would be nice to have a
> discussion of the different approaches to record/replay in your patches
> and Pavel's (perhaps before Pavel presents at KVM Forum).
>
> Paolo
Hi Paolo,

Sorry for the delay.
I'll rebase and resend these patches this week.

Thanks,
Fred

Re: [Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Frederic Konrad]

On 08/09/2014 09:57, Frederic Konrad wrote:
> On 01/09/2014 18:22, Paolo Bonzini wrote:
>> Il 03/07/2014 16:33, fred.konrad@greensocs.com ha scritto:
>>> From: KONRAD Frederic <fred.konrad@greensocs.com>
>>>
>>> Hi everybody,
>>>
>>> This is the sixth version of this RFC (see the changes below).
>>>
>>> Those are the two first patch-set we have been worked on for reverse 
>>> execution.
>>>
>>> The first part is fully reviewed except the "icount: introduce 
>>> icount timer"
>>> patch maybe we can merge them?
>>>
>>> The first series:
>>>    icount: put icount variables into TimerState.
>>>    migration: migrate icount fields.
>>>    migration: make qemu_savevm_state public.
>>>    icount: introduce icount timer.
>>>    icount: check for icount clock deadline when cpu loop exits.
>>>    icount: make icount extra computed on icount clock as well.
>>>    timer: add cpu_icount_to_ns function.
>>>
>>> are various preparation patches for reverse execution.
>>>
>>> The last patches:
>>>    trace-events: add reverse-execution events.
>>>    introduce reverse execution mechanism.
>>>    gdbstub: allow reverse execution in gdb stub.
>>>    cpu-exec: trigger a debug request when rexec stops.
>>>    rexec: synchronize icount on the next event.
>>>    rexec: allow to enable reverse execution.
>>>
>>> are reverse execution introduction.
>>>
>>> They can be clone at: git://git.greensocs.com/qemu_cexe.git:cexe_2_3_v6
>>>
>>> The third series will be sent as soon as possible and have some 
>>> issues with
>>> QEMU's thread as it use fork.
>>>
>>> This implementation of reverse execution works with instruction 
>>> counting:
>>>
>>> A new clock is implemented which is icount clock. It grows each time an
>>> instruction is executed and is totally independant of host clock.
>>>
>>> Snapshots are taken regularly (based on icount clock) with help of 
>>> migration
>>> code and written on the disk.
>>>
>>> When user wants to use reverse-stepi:
>>>   * Last snapshot is reloaded.
>>>   * A stop callback is created to be triggered at the previous 
>>> instruction.
>>>
>>> This stop callback generates a debug exception so QEMU stops in 
>>> debug mode.
>>>
>>> Command line:
>>>   * rexec suboption is added to icount to enable reverse execution, 
>>> it needs
>>>     icount=N and doesn't support auto mode.
>>>
>>> About non determinism in QEMU:
>>>   * This implementation doesn't take IO in account so any IO will 
>>> cause non
>>>     determinism and break reverse execution.
>>>
>>>   * The icount warp mechanism have been disabled when reverse 
>>> execution is
>>>     enabled so the time grow differently inside the VM.
>>>
>>> Testing:
>>>   * It has been tested on ARM without any IO such as network or 
>>> asynchronous file
>>>     access to keep the deterministic behaviour of icount.
>>>
>>> Known issues:
>>>   * On ARM stepi seems to do some additional steps which are added 
>>> to icount
>>>     counter so reverse-stepi just after stepi is broken.
>>>
>>>   * The IO replay explained above.
>> Hi, can you rebase and repost these patches?  It would be nice to have a
>> discussion of the different approaches to record/replay in your patches
>> and Pavel's (perhaps before Pavel presents at KVM Forum).
>>
>> Paolo
> Hi Paolo,
>
> Sorry for the delay.
> I'll rebase and resend these patches this week.
>
> Thanks,
> Fred
>
By the way how do you want to have this discussion?

At the KVM forum? Or by phone on KVM phone call?

Fred

Re: [Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Paolo Bonzini]

Il 08/09/2014 10:09, Frederic Konrad ha scritto:
>>
> By the way how do you want to have this discussion?
> 
> At the KVM forum? Or by phone on KVM phone call?

Or both. :)

Seriously, Pavel is presenting on record/replay at KVM Forum.  I guess
there will be some discussion after his presentation, but anything that
happens _before_ could make the presentation more interesting.

Paolo

Re: [Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Frederic Konrad]

On 08/09/2014 10:29, Paolo Bonzini wrote:
> Il 08/09/2014 10:09, Frederic Konrad ha scritto:
>> By the way how do you want to have this discussion?
>>
>> At the KVM forum? Or by phone on KVM phone call?
> Or both. :)
>
> Seriously, Pavel is presenting on record/replay at KVM Forum.  I guess
> there will be some discussion after his presentation, but anything that
> happens _before_ could make the presentation more interesting.
>
> Paolo
Ok fine with us,

So is that possible to allocate time for this on the next KVM phone 
calls I think it's
next week?
If that suits to you Pavel?

Fred

Re: [Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Pavel Dovgaluk]

> From: Frederic Konrad [mailto:fred.konrad@greensocs.com]
> On 08/09/2014 10:29, Paolo Bonzini wrote:
> > Il 08/09/2014 10:09, Frederic Konrad ha scritto:
> >> By the way how do you want to have this discussion?
> >>
> >> At the KVM forum? Or by phone on KVM phone call?
> > Or both. :)
> >
> > Seriously, Pavel is presenting on record/replay at KVM Forum.  I guess
> > there will be some discussion after his presentation, but anything that
> > happens _before_ could make the presentation more interesting.
> >
> > Paolo
> Ok fine with us,
> 
> So is that possible to allocate time for this on the next KVM phone
> calls I think it's
> next week?
> If that suits to you Pavel?

Yes, I can take part in phone calls next week.
What should I do for that?

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v6 00/14] Reverse execution.

[Frederic Konrad]

On 09/09/2014 08:30, Pavel Dovgaluk wrote:
>> From: Frederic Konrad [mailto:fred.konrad@greensocs.com]
>> On 08/09/2014 10:29, Paolo Bonzini wrote:
>>> Il 08/09/2014 10:09, Frederic Konrad ha scritto:
>>>> By the way how do you want to have this discussion?
>>>>
>>>> At the KVM forum? Or by phone on KVM phone call?
>>> Or both. :)
>>>
>>> Seriously, Pavel is presenting on record/replay at KVM Forum.  I guess
>>> there will be some discussion after his presentation, but anything that
>>> happens _before_ could make the presentation more interesting.
>>>
>>> Paolo
>> Ok fine with us,
>>
>> So is that possible to allocate time for this on the next KVM phone
>> calls I think it's
>> next week?
>> If that suits to you Pavel?
> Yes, I can take part in phone calls next week.
> What should I do for that?
>
> Pavel Dovgalyuk
>
>
Hi,

Ok good, is that good for you too Paolo?

We should reply to the agenda I haven't see any email yet?

Thanks,
Fred