[Qemu-devel] [RFC PATCH v4 00/25] Deterministic replay and reverse execution

[Qemu-devel] [RFC PATCH v4 00/25] Deterministic replay and reverse execution

[Pavel Dovgalyuk]

This set of patches is related to the reverse execution and deterministic 
replay of qemu execution  Our implementation of deterministic replay can 
be used for deterministic and reverse debugging of guest code through gdb 
remote interface.

Execution recording writes non-deterministic events log, which can be later 
used for replaying the execution anywhere and for unlimited number of times. 
It also supports checkpointing for faster rewinding during reverse debugging. 
Execution replaying reads the log and replays all non-deterministic events 
including external input, hardware clocks, and interrupts.

Reverse execution has the following features:
 * Deterministically replays whole system execution and all contents of the memory,
   state of the hadrware devices, clocks, and screen of the VM.
 * Writes execution log into the file for latter replaying for multiple times 
   on different machines.
 * Supports i386, x86_64, and ARM hardware platforms.
 * Performs deterministic replay of all operations with keyboard, mouse, network adapters,
   audio devices, serial interfaces, and physical USB devices connected to the emulator.
 * Provides support for gdb reverse debugging commands like reverse-step and reverse-continue.
 * Supports auto-checkpointing for convenient reverse debugging.

Usage of the record/replay:
 * First, record the execution, by adding '-record fname=replay.bin' to the
   command line.
 * Then you can replay it for the multiple times by using another command
   line option: '-replay fname=replay.bin'
 * Virtual machine should have at least one virtual disk, which is used to
   store checkpoints. If you want to enable automatic checkpointing, simply
   add ',period=XX' to record options, where XX is the checkpointing period
   in seconds.
 * Using of the network adapters in record/replay mode is possible with 
   the following command-line options:
   - '-net user' (or another host adapter) in record mode
   - '-net replay' in replay mode. Every host network adapter should be
     replaced by 'replay' when replaying the execution.
 * Reverse debugging can be used through gdb remote interface.
   reverse-stepi and reverse-continue commands are supported. Other reverse
   commands should also work, because they reuse these ones.
 * Monitor is extended by the following commands:
   - replay_info - prints information about replay mode and current step
     (number of instructions executed)
   - replay_break - sets "breakpoint" at the specified instructions count.
   - replay_seek - rewinds (using the checkpoints, if possible) to the
     specified step of replay log.

Paper with short description of deterministic replay implementation:
http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html

Modifications of qemu include:
 * adding missed fields of the virtual devices' states to the vmstate 
   structures to allow deterministic saving and restoring the VM state
 * adding virtual clock-based timers to vmstate structures, because virtual 
   clock is the part of the virtual machine state
 * modification of block layer to support automatic creation of the overlay
   files to store the changes and snapshots while recording
 * disabling of system reset while loading VM state to avoid generating of
   interrupts by reset handlers
 * adding warpers for clock and time functions to save their return
   values in the log
 * saving different asynchronous events (e.g. system shutdown) into the log
 * synchronization of the bottom halves execution
 * synchronization of the threads from thread pool
 * recording/replaying user input (mouse and keyboard), input from virtual
   serial ports, incoming network packets, input from connected USB devices
 * adding HMP/QMP commands to monitor for controlling replay execution

v4 changes:
 * Updated block drivers to support new bdrv_open interface.
 * Moved migration patches into separate series (as suggested by Paolo Bonzini)
 * Fixed a bug in replay_break operation.
 * Fixed rtl8139 migration for replay.
 * Fixed 'period' parameter processing for record mode.
 * Fixed bug in 'reverse-stepi' implementation.
 * Fixed replay without making any snapshots (even the starting one).
 * Moved core replay patches into the separate series.
 * Fixed reverse step and reverse continue support.

v3 changes:
 * Fixed bug with replay of the aio write operations.
 * Added virtual clock based on replay icount.
 * Removed duplicated saving of interrupt_request CPU field.
 * Fixed some coding style issues.
 * Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
 * Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
 * Added description and example for replay-info command (as suggested by Eric Blake)
 * Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
 * Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
 * Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
 * New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
 * Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
 * Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)

v2 changes:
 * Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
 * Added QMP versions of replay commands (as suggested by Eric Blake)
 * Removed some optional features of replay to make patches cleaner
 * Minor changes and code cleanup were made

---

Pavel Dovgalyuk (25):
      acpi: accurate overflow check
      mc146818rtc: add missed field to vmstate
      replay: global variables and function stubs
      sysemu: system functions for replay
      replay: internal functions for replay log
      cpu-exec: reset exception_index correctly
      icount: implement icount requesting
      icount: improve enable/disable ticks
      replay: introduce icount event
      i386: do not cross the pages boundaries in replay mode
      cpu-exec: allow temporary disabling icount
      replay: interrupts and exceptions
      replay: asynchronous events infrastructure
      cpu: replay instructions sequence
      replay: recording and replaying clock ticks
      replay: recording and replaying different timers
      cpus: make icount warp deterministic in replay mode
      replay: shutdown event
      replay: checkpoints
      replay: bottom halves
      replay: replay aio requests
      replay: thread pool
      replay: initialization and deinitialization
      replay: command line options
      replay: recording of the user input


 Makefile.target                |    1 
 async.c                        |   46 +++++-
 block.c                        |   92 +++++++++++-
 block/block-backend.c          |   30 ++++
 block/qcow2.c                  |    4 +
 block/raw-posix.c              |    6 +
 block/raw-win32.c              |    4 -
 cpu-exec.c                     |   37 +++--
 cpus.c                         |  115 +++++++++++----
 dma-helpers.c                  |   10 +
 exec.c                         |    1 
 hw/acpi/core.c                 |    7 +
 hw/block/virtio-blk.c          |   10 +
 hw/ide/ahci.c                  |    4 -
 hw/ide/atapi.c                 |   10 +
 hw/ide/core.c                  |   18 ++
 hw/timer/arm_timer.c           |    2 
 hw/timer/mc146818rtc.c         |   11 +
 hw/timer/pl031.c               |   10 +
 hw/usb/hcd-uhci.c              |    2 
 include/block/aio.h            |   18 ++
 include/block/block.h          |   15 ++
 include/block/thread-pool.h    |    4 -
 include/exec/exec-all.h        |    8 +
 include/qemu-common.h          |    3 
 include/qemu/main-loop.h       |    1 
 include/qemu/timer.h           |   27 +++
 include/qom/cpu.h              |   10 +
 include/sysemu/block-backend.h |   10 +
 include/sysemu/cpus.h          |    1 
 include/sysemu/sysemu.h        |    2 
 include/ui/input.h             |    2 
 main-loop.c                    |    5 +
 qapi-schema.json               |   32 ++++
 qemu-io-cmds.c                 |    2 
 qemu-options.hx                |   27 +++
 qemu-timer.c                   |   44 ++++--
 replay/Makefile.objs           |    5 +
 replay/replay-events.c         |  292 +++++++++++++++++++++++++++++++++++++
 replay/replay-input.c          |  108 ++++++++++++++
 replay/replay-internal.c       |  155 ++++++++++++++++++++
 replay/replay-internal.h       |  132 +++++++++++++++++
 replay/replay-time.c           |  191 ++++++++++++++++++++++++
 replay/replay.c                |  314 ++++++++++++++++++++++++++++++++++++++++
 replay/replay.h                |  117 +++++++++++++++
 savevm.c                       |   25 +++
 stubs/Makefile.objs            |    1 
 stubs/replay.c                 |   42 +++++
 target-i386/cpu.h              |    7 +
 target-i386/translate.c        |   14 ++
 tests/test-thread-pool.c       |    7 +
 thread-pool.c                  |   49 ++++--
 trace-events                   |    2 
 translate-all.c                |   30 ++++
 ui/input.c                     |   80 ++++++++--
 util/iov.c                     |    4 +
 vl.c                           |  112 +++++++++++++-
 57 files changed, 2162 insertions(+), 156 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay-events.c
 create mode 100755 replay/replay-input.c
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h
 create mode 100755 replay/replay-time.c
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

-- 
Pavel Dovgalyuk

[Qemu-devel] [RFC PATCH v4 01/25] acpi: accurate overflow check

[Pavel Dovgalyuk]

Compare clock in ns, because acpi_pm_tmr_update uses rounded
to ns value instead of ticks.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 hw/acpi/core.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/hw/acpi/core.c b/hw/acpi/core.c
index a7368fb..51913d6 100644
--- a/hw/acpi/core.c
+++ b/hw/acpi/core.c
@@ -376,8 +376,11 @@ static void acpi_notify_wakeup(Notifier *notifier, void *data)
 /* ACPI PM1a EVT */
 uint16_t acpi_pm1_evt_get_sts(ACPIREGS *ar)
 {
-    int64_t d = acpi_pm_tmr_get_clock();
-    if (d >= ar->tmr.overflow_time) {
+    /* Compare ns-clock, not PM timer ticks, because
+       acpi_pm_tmr_update function uses ns for setting the timer. */
+    int64_t d = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+    if (d >= muldiv64(ar->tmr.overflow_time,
+                      get_ticks_per_sec(), PM_TIMER_FREQUENCY)) {
         ar->pm1.evt.sts |= ACPI_BITMASK_TIMER_STATUS;
     }
     return ar->pm1.evt.sts;

Re: [Qemu-devel] [RFC PATCH v4 01/25] acpi: accurate overflow check

[Paolo Bonzini]

On 07/11/2014 11:31, Pavel Dovgalyuk wrote:
> Compare clock in ns, because acpi_pm_tmr_update uses rounded
> to ns value instead of ticks.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  hw/acpi/core.c |    7 +++++--
>  1 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/hw/acpi/core.c b/hw/acpi/core.c
> index a7368fb..51913d6 100644
> --- a/hw/acpi/core.c
> +++ b/hw/acpi/core.c
> @@ -376,8 +376,11 @@ static void acpi_notify_wakeup(Notifier *notifier, void *data)
>  /* ACPI PM1a EVT */
>  uint16_t acpi_pm1_evt_get_sts(ACPIREGS *ar)
>  {
> -    int64_t d = acpi_pm_tmr_get_clock();
> -    if (d >= ar->tmr.overflow_time) {
> +    /* Compare ns-clock, not PM timer ticks, because
> +       acpi_pm_tmr_update function uses ns for setting the timer. */
> +    int64_t d = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
> +    if (d >= muldiv64(ar->tmr.overflow_time,
> +                      get_ticks_per_sec(), PM_TIMER_FREQUENCY)) {
>          ar->pm1.evt.sts |= ACPI_BITMASK_TIMER_STATUS;
>      }
>      return ar->pm1.evt.sts;
> 

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

The same muldiv64 is in acpi_pm_tmr_update, it could be extracted to a
separate function if you so wish.

Paolo

[Qemu-devel] [RFC PATCH v4 02/25] mc146818rtc: add missed field to vmstate

[Pavel Dovgalyuk]

This patch adds irq_reinject_on_ack_count field to VMState to allow correct
saving/loading the state of MC146818 RTC.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 hw/timer/mc146818rtc.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
index f18d128..2c4b650 100644
--- a/hw/timer/mc146818rtc.c
+++ b/hw/timer/mc146818rtc.c
@@ -809,6 +809,7 @@ static void rtc_reset(void *opaque)
 
     s->cmos_data[RTC_REG_B] &= ~(REG_B_PIE | REG_B_AIE | REG_B_SQWE);
     s->cmos_data[RTC_REG_C] &= ~(REG_C_UF | REG_C_IRQF | REG_C_PF | REG_C_AF);
+    s->irq_reinject_on_ack_count = 0;
     check_update_timer(s);
 
     qemu_irq_lower(s->irq);

Re: [Qemu-devel] [RFC PATCH v4 02/25] mc146818rtc: add missed field to vmstate

[Paolo Bonzini]

On 07/11/2014 11:31, Pavel Dovgalyuk wrote:
> This patch adds irq_reinject_on_ack_count field to VMState to allow correct
> saving/loading the state of MC146818 RTC.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  hw/timer/mc146818rtc.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
> 
> diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
> index f18d128..2c4b650 100644
> --- a/hw/timer/mc146818rtc.c
> +++ b/hw/timer/mc146818rtc.c
> @@ -809,6 +809,7 @@ static void rtc_reset(void *opaque)
>  
>      s->cmos_data[RTC_REG_B] &= ~(REG_B_PIE | REG_B_AIE | REG_B_SQWE);
>      s->cmos_data[RTC_REG_C] &= ~(REG_C_UF | REG_C_IRQF | REG_C_PF | REG_C_AF);
> +    s->irq_reinject_on_ack_count = 0;
>      check_update_timer(s);
>  
>      qemu_irq_lower(s->irq);
> 

This is already set below.  If you need it here, please justify it in
the commit message and move up the code that already sets it.

Paolo

[Qemu-devel] [RFC PATCH v4 03/25] replay: global variables and function stubs

[Pavel Dovgalyuk]

This patch adds global variables, defines, functions declarations,
and function stubs for deterministic VM replay used by external modules.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 Makefile.target      |    1 +
 qapi-schema.json     |   32 ++++++++++++++++++++++++++++++++
 replay/Makefile.objs |    1 +
 replay/replay.c      |   25 +++++++++++++++++++++++++
 replay/replay.h      |   23 +++++++++++++++++++++++
 stubs/Makefile.objs  |    1 +
 stubs/replay.c       |    8 ++++++++
 7 files changed, 91 insertions(+), 0 deletions(-)
 create mode 100755 replay/Makefile.objs
 create mode 100755 replay/replay.c
 create mode 100755 replay/replay.h
 create mode 100755 stubs/replay.c

diff --git a/Makefile.target b/Makefile.target
index e9ff1ee..a45378f 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -83,6 +83,7 @@ all: $(PROGS) stap
 #########################################################
 # cpu emulator library
 obj-y = exec.o translate-all.o cpu-exec.o
+obj-y += replay/
 obj-y += tcg/tcg.o tcg/optimize.o
 obj-$(CONFIG_TCG_INTERPRETER) += tci.o
 obj-$(CONFIG_TCG_INTERPRETER) += disas/tci.o
diff --git a/qapi-schema.json b/qapi-schema.json
index 24379ab..49d61c5 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3508,3 +3508,35 @@
 # Since: 2.1
 ##
 { 'command': 'rtc-reset-reinjection' }
+
+##
+# ReplayMode:
+#
+# Mode of the replay subsystem.
+#
+# @none: normal execution mode. Replay or record are not enabled.
+#
+# @record: record mode. All non-deterministic data is written into the
+#          replay log.
+#
+# @play: replay mode. Non-deterministic data required for system execution
+#        is read from the log.
+#
+# Since: 2.2
+##
+{ 'enum': 'ReplayMode',
+  'data': [ 'none', 'record', 'play' ] }
+
+##
+# ReplaySubmode:
+#
+# Submode of the replay subsystem.
+#
+# @unknown: used for modes different from play.
+#
+# @normal: normal replay mode.
+#
+# Since: 2.2
+##
+{ 'enum': 'ReplaySubmode',
+  'data': [ 'unknown', 'normal' ] }
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
new file mode 100755
index 0000000..7ea860f
--- /dev/null
+++ b/replay/Makefile.objs
@@ -0,0 +1 @@
+obj-y += replay.o
diff --git a/replay/replay.c b/replay/replay.c
new file mode 100755
index 0000000..ac976b2
--- /dev/null
+++ b/replay/replay.c
@@ -0,0 +1,25 @@
+/*
+ * replay.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "replay.h"
+
+ReplayMode replay_mode = REPLAY_MODE_NONE;
+/*! Stores current submode for PLAY mode */
+ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;
+
+/* Suffix for the disk images filenames */
+char *replay_image_suffix;
+
+
+ReplaySubmode replay_get_play_submode(void)
+{
+    return play_submode;
+}
diff --git a/replay/replay.h b/replay/replay.h
new file mode 100755
index 0000000..51a18fe
--- /dev/null
+++ b/replay/replay.h
@@ -0,0 +1,23 @@
+#ifndef REPLAY_H
+#define REPLAY_H
+
+/*
+ * replay.h
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qapi-types.h"
+
+extern ReplayMode replay_mode;
+extern char *replay_image_suffix;
+
+/*! Returns replay play submode */
+ReplaySubmode replay_get_play_submode(void);
+
+#endif
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 5e347d0..45a6c71 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -27,6 +27,7 @@ stub-obj-y += notify-event.o
 stub-obj-y += pci-drive-hot-add.o
 stub-obj-$(CONFIG_SPICE) += qemu-chr-open-spice.o
 stub-obj-y += qtest.o
+stub-obj-y += replay.o
 stub-obj-y += reset.o
 stub-obj-y += runstate-check.o
 stub-obj-y += set-fd-handler.o
diff --git a/stubs/replay.c b/stubs/replay.c
new file mode 100755
index 0000000..b146d55
--- /dev/null
+++ b/stubs/replay.c
@@ -0,0 +1,8 @@
+#include "replay/replay.h"
+
+ReplayMode replay_mode;
+
+ReplaySubmode replay_get_play_submode(void)
+{
+    return 0;
+}

Re: [Qemu-devel] [RFC PATCH v4 03/25] replay: global variables and function stubs

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1366 bytes --]

On 11/07/2014 11:31 AM, Pavel Dovgalyuk wrote:
> This patch adds global variables, defines, functions declarations,
> and function stubs for deterministic VM replay used by external modules.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---

Just focusing on interface review:

> +++ b/qapi-schema.json
> @@ -3508,3 +3508,35 @@
>  # Since: 2.1
>  ##
>  { 'command': 'rtc-reset-reinjection' }
> +
> +##
> +# ReplayMode:
> +#
> +# Mode of the replay subsystem.
> +#
> +# @none: normal execution mode. Replay or record are not enabled.
> +#
> +# @record: record mode. All non-deterministic data is written into the
> +#          replay log.
> +#
> +# @play: replay mode. Non-deterministic data required for system execution
> +#        is read from the log.
> +#
> +# Since: 2.2

You've missed 2.2 freeze; this will have to be 2.3.

> +##
> +{ 'enum': 'ReplayMode',
> +  'data': [ 'none', 'record', 'play' ] }
> +
> +##
> +# ReplaySubmode:
> +#
> +# Submode of the replay subsystem.
> +#
> +# @unknown: used for modes different from play.
> +#
> +# @normal: normal replay mode.
> +#
> +# Since: 2.2

Again, 2.3

> +##
> +{ 'enum': 'ReplaySubmode',
> +  'data': [ 'unknown', 'normal' ] }

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]

[Qemu-devel] [RFC PATCH v4 04/25] sysemu: system functions for replay

[Pavel Dovgalyuk]

This patch removes "static" specifier from several qemu function to make
them visible to the replay module. It also invents several system functions
that will be used by replay.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c                  |    4 ++--
 include/exec/exec-all.h |    1 +
 include/qom/cpu.h       |   10 ++++++++++
 include/sysemu/cpus.h   |    1 +
 include/sysemu/sysemu.h |    1 +
 savevm.c                |   25 ++++++++++++++++++++-----
 translate-all.c         |    8 ++++++++
 7 files changed, 43 insertions(+), 7 deletions(-)

diff --git a/cpus.c b/cpus.c
index 0c33458..e53d605 100644
--- a/cpus.c
+++ b/cpus.c
@@ -88,7 +88,7 @@ static bool cpu_thread_is_idle(CPUState *cpu)
     return true;
 }
 
-static bool all_cpu_threads_idle(void)
+bool all_cpu_threads_idle(void)
 {
     CPUState *cpu;
 
@@ -1112,7 +1112,7 @@ bool qemu_cpu_is_self(CPUState *cpu)
     return qemu_thread_is_self(cpu->thread);
 }
 
-static bool qemu_in_vcpu_thread(void)
+bool qemu_in_vcpu_thread(void)
 {
     return current_cpu && qemu_cpu_is_self(current_cpu);
 }
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index ab956f2..1d17f75 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -212,6 +212,7 @@ static inline unsigned int tb_phys_hash_func(tb_page_addr_t pc)
 
 void tb_free(TranslationBlock *tb);
 void tb_flush(CPUArchState *env);
+void tb_flush_all(void);
 void tb_phys_invalidate(TranslationBlock *tb, tb_page_addr_t page_addr);
 
 #if defined(USE_DIRECT_JUMP)
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index 2098f1c..5afb44c 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -480,6 +480,16 @@ static inline bool cpu_has_work(CPUState *cpu)
 bool qemu_cpu_is_self(CPUState *cpu);
 
 /**
+ * qemu_in_vcpu_thread:
+ *
+ * Checks whether the caller is executing on the vCPU thread
+ * of the current vCPU.
+ *
+ * Returns: %true if called from vCPU's thread, %false otherwise.
+ */
+bool qemu_in_vcpu_thread(void);
+
+/**
  * qemu_cpu_kick:
  * @cpu: The vCPU to kick.
  *
diff --git a/include/sysemu/cpus.h b/include/sysemu/cpus.h
index 3f162a9..86ae556 100644
--- a/include/sysemu/cpus.h
+++ b/include/sysemu/cpus.h
@@ -6,6 +6,7 @@ void qemu_init_cpu_loop(void);
 void resume_all_vcpus(void);
 void pause_all_vcpus(void);
 void cpu_stop_current(void);
+bool all_cpu_threads_idle(void);
 
 void cpu_synchronize_all_states(void);
 void cpu_synchronize_all_post_reset(void);
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 0037a69..72bc093 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -76,6 +76,7 @@ void qemu_add_machine_init_done_notifier(Notifier *notify);
 
 void do_savevm(Monitor *mon, const QDict *qdict);
 int load_vmstate(const char *name);
+int save_vmstate(Monitor *mon, const char *name);
 void do_delvm(Monitor *mon, const QDict *qdict);
 void do_info_snapshots(Monitor *mon, const QDict *qdict);
 
diff --git a/savevm.c b/savevm.c
index 2d8eb96..b4fe8d3 100644
--- a/savevm.c
+++ b/savevm.c
@@ -42,7 +42,7 @@
 #include "qemu/iov.h"
 #include "block/snapshot.h"
 #include "block/qapi.h"
-
+#include "replay/replay.h"
 
 #ifndef ETH_P_RARP
 #define ETH_P_RARP 0x8035
@@ -1039,7 +1039,7 @@ static int del_existing_snapshots(Monitor *mon, const char *name)
     return 0;
 }
 
-void do_savevm(Monitor *mon, const QDict *qdict)
+int save_vmstate(Monitor *mon, const char *name)
 {
     BlockDriverState *bs, *bs1;
     QEMUSnapshotInfo sn1, *sn = &sn1, old_sn1, *old_sn = &old_sn1;
@@ -1049,7 +1049,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
     uint64_t vm_state_size;
     qemu_timeval tv;
     struct tm tm;
-    const char *name = qdict_get_try_str(qdict, "name");
+    int success = 0;
 
     /* Verify if there is a device that doesn't support snapshots and is writable */
     bs = NULL;
@@ -1062,14 +1062,19 @@ void do_savevm(Monitor *mon, const QDict *qdict)
         if (!bdrv_can_snapshot(bs)) {
             monitor_printf(mon, "Device '%s' is writable but does not support snapshots.\n",
                                bdrv_get_device_name(bs));
-            return;
+            return success;
         }
     }
 
     bs = find_vmstate_bs();
     if (!bs) {
         monitor_printf(mon, "No block device can accept snapshots\n");
-        return;
+        if (replay_mode != REPLAY_MODE_NONE) {
+            fprintf(stderr,
+                    "At least one hdd should be attached to QEMU for replay\n");
+            exit(1);
+        }
+        return success;
     }
 
     saved_vm_running = runstate_is_running();
@@ -1118,6 +1123,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
 
     /* create the snapshots */
 
+    success = 1;
     bs1 = NULL;
     while ((bs1 = bdrv_next(bs1))) {
         if (bdrv_can_snapshot(bs1)) {
@@ -1127,6 +1133,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
             if (ret < 0) {
                 monitor_printf(mon, "Error while creating snapshot on '%s'\n",
                                bdrv_get_device_name(bs1));
+                success = 0;
             }
         }
     }
@@ -1135,6 +1142,14 @@ void do_savevm(Monitor *mon, const QDict *qdict)
     if (saved_vm_running) {
         vm_start();
     }
+
+    return success;
+}
+
+void do_savevm(Monitor *mon, const QDict *qdict)
+{
+    const char *name = qdict_get_try_str(qdict, "name");
+    save_vmstate(mon, name);
 }
 
 void qmp_xen_save_devices_state(const char *filename, Error **errp)
diff --git a/translate-all.c b/translate-all.c
index ba5c840..7177b71 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -806,6 +806,14 @@ void tb_flush(CPUArchState *env1)
     tcg_ctx.tb_ctx.tb_flush_count++;
 }
 
+void tb_flush_all(void)
+{
+    CPUState *cpu;
+    for (cpu = first_cpu ; cpu != NULL ; cpu = CPU_NEXT(cpu)) {
+        tb_flush(cpu->env_ptr);
+    }
+}
+
 #ifdef DEBUG_TB_CHECK
 
 static void tb_invalidate_check(target_ulong address)

Re: [Qemu-devel] [RFC PATCH v4 04/25] sysemu: system functions for replay

[Alex Bennée]

Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru> writes:

> This patch removes "static" specifier from several qemu function to make
> them visible to the replay module. It also invents several system functions
> that will be used by replay.
>
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
<snip>
>  
>  void do_savevm(Monitor *mon, const QDict *qdict);
>  int load_vmstate(const char *name);
> +int save_vmstate(Monitor *mon, const char *name);
>  void do_delvm(Monitor *mon, const QDict *qdict);
>  void do_info_snapshots(Monitor *mon, const QDict *qdict);
<snip>
>  
> -void do_savevm(Monitor *mon, const QDict *qdict)
> +int save_vmstate(Monitor *mon, const char *name)
>  {
>      BlockDriverState *bs, *bs1;
>      QEMUSnapshotInfo sn1, *sn = &sn1, old_sn1, *old_sn = &old_sn1;
> @@ -1049,7 +1049,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
>      uint64_t vm_state_size;
>      qemu_timeval tv;
>      struct tm tm;
> -    const char *name = qdict_get_try_str(qdict, "name");
> +    int success = 0;
>  
>      /* Verify if there is a device that doesn't support snapshots and is writable */
>      bs = NULL;
> @@ -1062,14 +1062,19 @@ void do_savevm(Monitor *mon, const QDict *qdict)
>          if (!bdrv_can_snapshot(bs)) {
>              monitor_printf(mon, "Device '%s' is writable but does not support snapshots.\n",
>                                 bdrv_get_device_name(bs));
> -            return;
> +            return success;
>          }
>      }
>  
>      bs = find_vmstate_bs();
>      if (!bs) {
>          monitor_printf(mon, "No block device can accept snapshots\n");
> -        return;
> +        if (replay_mode != REPLAY_MODE_NONE) {
> +            fprintf(stderr,
> +                    "At least one hdd should be attached to QEMU for replay\n");
> +            exit(1);

Perhaps we should be doing a proper error_report here?

> +        }
> +        return success;
>      }
>  
>      saved_vm_running = runstate_is_running();
> @@ -1118,6 +1123,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
>  
>      /* create the snapshots */
>  
> +    success = 1;
>      bs1 = NULL;
>      while ((bs1 = bdrv_next(bs1))) {
>          if (bdrv_can_snapshot(bs1)) {
> @@ -1127,6 +1133,7 @@ void do_savevm(Monitor *mon, const QDict *qdict)
>              if (ret < 0) {
>                  monitor_printf(mon, "Error while creating snapshot on '%s'\n",
>                                 bdrv_get_device_name(bs1));
> +                success = 0;
>              }
>          }
>      }
> @@ -1135,6 +1142,14 @@ void do_savevm(Monitor *mon, const QDict *qdict)
>      if (saved_vm_running) {
>          vm_start();
>      }
> +
> +    return success;
> +}
> +
> +void do_savevm(Monitor *mon, const QDict *qdict)
> +{
> +    const char *name = qdict_get_try_str(qdict, "name");
> +    save_vmstate(mon, name);
>  }

You've re-factored do_savevm() and added a success/fail parameter which
isn't used by the caller. A bit of documentation on what success means
in this context would be useful for the function.

My personal preference is for success/fail returns is to use unambiguous
bool types but we don't mandate that. 

-- 
Alex Bennée

[Qemu-devel] [RFC PATCH v4 05/25] replay: internal functions for replay log

[Pavel Dovgalyuk]

This patch adds functions to perform read and write operations
with replay log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/Makefile.objs     |    1 
 replay/replay-internal.c |  141 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   50 ++++++++++++++++
 3 files changed, 192 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-internal.c
 create mode 100755 replay/replay-internal.h

diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 7ea860f..1148f45 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1 +1,2 @@
 obj-y += replay.o
+obj-y += replay-internal.o
diff --git a/replay/replay-internal.c b/replay/replay-internal.c
new file mode 100755
index 0000000..29f995d
--- /dev/null
+++ b/replay/replay-internal.c
@@ -0,0 +1,141 @@
+/*
+ * replay-internal.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay-internal.h"
+
+volatile unsigned int replay_data_kind = -1;
+volatile unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+FILE *replay_file;
+
+void replay_put_byte(unsigned char byte)
+{
+    if (replay_file) {
+        fwrite(&byte, sizeof(byte), 1, replay_file);
+    }
+}
+
+void replay_put_event(unsigned char event)
+{
+    replay_put_byte(event);
+}
+
+
+void replay_put_word(uint16_t word)
+{
+    if (replay_file) {
+        fwrite(&word, sizeof(word), 1, replay_file);
+    }
+}
+
+void replay_put_dword(unsigned int dword)
+{
+    if (replay_file) {
+        fwrite(&dword, sizeof(dword), 1, replay_file);
+    }
+}
+
+void replay_put_qword(int64_t qword)
+{
+    if (replay_file) {
+        fwrite(&qword, sizeof(qword), 1, replay_file);
+    }
+}
+
+void replay_put_array(const uint8_t *buf, size_t size)
+{
+    if (replay_file) {
+        fwrite(&size, sizeof(size), 1, replay_file);
+        fwrite(buf, 1, size, replay_file);
+    }
+}
+
+unsigned char replay_get_byte(void)
+{
+    unsigned char byte;
+    if (replay_file) {
+        fread(&byte, sizeof(byte), 1, replay_file);
+    }
+    return byte;
+}
+
+uint16_t replay_get_word(void)
+{
+    uint16_t word;
+    if (replay_file) {
+        fread(&word, sizeof(word), 1, replay_file);
+    }
+
+    return word;
+}
+
+unsigned int replay_get_dword(void)
+{
+    unsigned int dword;
+    if (replay_file) {
+        fread(&dword, sizeof(dword), 1, replay_file);
+    }
+
+    return dword;
+}
+
+int64_t replay_get_qword(void)
+{
+    int64_t qword;
+    if (replay_file) {
+        fread(&qword, sizeof(qword), 1, replay_file);
+    }
+
+    return qword;
+}
+
+void replay_get_array(uint8_t *buf, size_t *size)
+{
+    if (replay_file) {
+        fread(size, sizeof(*size), 1, replay_file);
+        fread(buf, 1, *size, replay_file);
+    }
+}
+
+void replay_get_array_alloc(uint8_t **buf, size_t *size)
+{
+    if (replay_file) {
+        fread(size, sizeof(*size), 1, replay_file);
+        *buf = g_malloc(*size);
+        fread(*buf, 1, *size, replay_file);
+    }
+}
+
+void replay_check_error(void)
+{
+    if (replay_file) {
+        if (feof(replay_file)) {
+            fprintf(stderr, "replay file is over\n");
+            exit(1);
+        } else if (ferror(replay_file)) {
+            fprintf(stderr, "replay file is over or something goes wrong\n");
+            exit(1);
+        }
+    }
+}
+
+void replay_fetch_data_kind(void)
+{
+    if (replay_file) {
+        if (!replay_has_unread_data) {
+            replay_data_kind = replay_get_byte();
+            replay_check_error();
+            replay_has_unread_data = 1;
+        }
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
new file mode 100755
index 0000000..45bb344
--- /dev/null
+++ b/replay/replay-internal.h
@@ -0,0 +1,50 @@
+#ifndef REPLAY_INTERNAL_H
+#define REPLAY_INTERNAL_H
+
+/*
+ * replay-internal.h
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <stdio.h>
+
+extern volatile unsigned int replay_data_kind;
+extern volatile unsigned int replay_has_unread_data;
+
+/* File for replay writing */
+extern FILE *replay_file;
+
+void replay_put_byte(unsigned char byte);
+void replay_put_event(unsigned char event);
+void replay_put_word(uint16_t word);
+void replay_put_dword(unsigned int dword);
+void replay_put_qword(int64_t qword);
+void replay_put_array(const uint8_t *buf, size_t size);
+
+unsigned char replay_get_byte(void);
+uint16_t replay_get_word(void);
+unsigned int replay_get_dword(void);
+int64_t replay_get_qword(void);
+void replay_get_array(uint8_t *buf, size_t *size);
+void replay_get_array_alloc(uint8_t **buf, size_t *size);
+
+/*! Checks error status of the file. */
+void replay_check_error(void);
+
+/*! Reads data type from the file and stores it in the
+    replay_data_kind variable. */
+void replay_fetch_data_kind(void);
+
+/*! Saves queued events (like instructions and sound). */
+void replay_save_instructions(void);
+/*! Checks that the next data is corresponding to the desired kind.
+    Terminates the program in case of error. */
+void validate_data_kind(int kind);
+
+#endif

Re: [Qemu-devel] [RFC PATCH v4 05/25] replay: internal functions for replay log

[Alex Bennée]

Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru> writes:

> This patch adds functions to perform read and write operations
> with replay log.
>
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
<snip>
> +
> +/* File for replay writing */
> +FILE *replay_file;
> +
> +void replay_put_byte(unsigned char byte)
> +{
> +    if (replay_file) {
> +        fwrite(&byte, sizeof(byte), 1, replay_file);
> +    }
> +}
> +
> +void replay_put_event(unsigned char event)
> +{
> +    replay_put_byte(event);
> +}
> +
> +
<snip>
> +void replay_put_dword(unsigned int dword)
> +{
> +    if (replay_file) {
> +        fwrite(&dword, sizeof(dword), 1, replay_file);
> +    }
> +}

Given you use the unambiguous types bellow shouldn't you use uint8_t and
uint32_t here to be safe?

<snip>
> +
> +unsigned char replay_get_byte(void)
> +{
> +    unsigned char byte;
> +    if (replay_file) {
> +        fread(&byte, sizeof(byte), 1, replay_file);
> +    }
> +    return byte;
> +}
> +
> +uint16_t replay_get_word(void)
> +{
> +    uint16_t word;
> +    if (replay_file) {
> +        fread(&word, sizeof(word), 1, replay_file);
> +    }
> +
> +    return word;
> +}
> +
> +unsigned int replay_get_dword(void)
> +{
> +    unsigned int dword;
> +    if (replay_file) {
> +        fread(&dword, sizeof(dword), 1, replay_file);
> +    }
> +
> +    return dword;
> +}

Ditto for the fetchers
<snip>

-- 
Alex Bennée

[Qemu-devel] [RFC PATCH v4 06/25] cpu-exec: reset exception_index correctly

[Pavel Dovgalyuk]

Exception index is reset at every entry at every entry into cpu_exec()
function. This may cause missing the exceptions while replaying them.
This patch moves exception_index reset to the locations where they are
processed.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c |    2 +-
 cpus.c     |    1 +
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 8830255..011f51f 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -358,7 +358,6 @@ int cpu_exec(CPUArchState *env)
     }
 
     cc->cpu_exec_enter(cpu);
-    cpu->exception_index = -1;
 
     /* Calculate difference between guest clock and host clock.
      * This delay includes the delay of the last cycle, so
@@ -378,6 +377,7 @@ int cpu_exec(CPUArchState *env)
                     if (ret == EXCP_DEBUG) {
                         cpu_handle_debug_exception(env);
                     }
+                    cpu->exception_index = -1;
                     break;
                 } else {
 #if defined(CONFIG_USER_ONLY)
diff --git a/cpus.c b/cpus.c
index e53d605..7e8c507 100644
--- a/cpus.c
+++ b/cpus.c
@@ -1016,6 +1016,7 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
     CPU_FOREACH(cpu) {
         cpu->thread_id = qemu_get_thread_id();
         cpu->created = true;
+        cpu->exception_index = -1;
     }
     qemu_cond_signal(&qemu_cpu_cond);
 

Re: [Qemu-devel] [RFC PATCH v4 06/25] cpu-exec: reset exception_index correctly

[Paolo Bonzini]

On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> Exception index is reset at every entry at every entry into cpu_exec()
> function. This may cause missing the exceptions while replaying them.
> This patch moves exception_index reset to the locations where they are
> processed.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpu-exec.c |    2 +-
>  cpus.c     |    1 +
>  2 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/cpu-exec.c b/cpu-exec.c
> index 8830255..011f51f 100644
> --- a/cpu-exec.c
> +++ b/cpu-exec.c
> @@ -358,7 +358,6 @@ int cpu_exec(CPUArchState *env)
>      }
>  
>      cc->cpu_exec_enter(cpu);
> -    cpu->exception_index = -1;
>  
>      /* Calculate difference between guest clock and host clock.
>       * This delay includes the delay of the last cycle, so
> @@ -378,6 +377,7 @@ int cpu_exec(CPUArchState *env)
>                      if (ret == EXCP_DEBUG) {
>                          cpu_handle_debug_exception(env);
>                      }
> +                    cpu->exception_index = -1;
>                      break;
>                  } else {
>  #if defined(CONFIG_USER_ONLY)
> diff --git a/cpus.c b/cpus.c
> index e53d605..7e8c507 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -1016,6 +1016,7 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
>      CPU_FOREACH(cpu) {
>          cpu->thread_id = qemu_get_thread_id();
>          cpu->created = true;
> +        cpu->exception_index = -1;
>      }
>      qemu_cond_signal(&qemu_cpu_cond);
>  
> 

Thanks, I queued this for 2.3 already.

Paolo

Re: [Qemu-devel] [RFC PATCH v4 06/25] cpu-exec: reset exception_index correctly

[Paolo Bonzini]

On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> Exception index is reset at every entry at every entry into cpu_exec()
> function. This may cause missing the exceptions while replaying them.
> This patch moves exception_index reset to the locations where they are
> processed.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpu-exec.c |    2 +-
>  cpus.c     |    1 +
>  2 files changed, 2 insertions(+), 1 deletions(-)
> 
> diff --git a/cpu-exec.c b/cpu-exec.c
> index 8830255..011f51f 100644
> --- a/cpu-exec.c
> +++ b/cpu-exec.c
> @@ -358,7 +358,6 @@ int cpu_exec(CPUArchState *env)
>      }
>  
>      cc->cpu_exec_enter(cpu);
> -    cpu->exception_index = -1;
>  
>      /* Calculate difference between guest clock and host clock.
>       * This delay includes the delay of the last cycle, so
> @@ -378,6 +377,7 @@ int cpu_exec(CPUArchState *env)
>                      if (ret == EXCP_DEBUG) {
>                          cpu_handle_debug_exception(env);
>                      }
> +                    cpu->exception_index = -1;
>                      break;
>                  } else {
>  #if defined(CONFIG_USER_ONLY)
> diff --git a/cpus.c b/cpus.c
> index e53d605..7e8c507 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -1016,6 +1016,7 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
>      CPU_FOREACH(cpu) {
>          cpu->thread_id = qemu_get_thread_id();
>          cpu->created = true;
> +        cpu->exception_index = -1;
>      }
>      qemu_cond_signal(&qemu_cpu_cond);
>  
> 
> 
> 

What about user-mode emulation?  Do you need to reset
cpu->exception_index for it too?

Paolo

Re: [Qemu-devel] [RFC PATCH v4 06/25] cpu-exec: reset exception_index correctly

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
> On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> > Exception index is reset at every entry at every entry into cpu_exec()
> > function. This may cause missing the exceptions while replaying them.
> > This patch moves exception_index reset to the locations where they are
> > processed.
> >
> > Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> > ---
> >  cpu-exec.c |    2 +-
> >  cpus.c     |    1 +
> >  2 files changed, 2 insertions(+), 1 deletions(-)
> >
> > diff --git a/cpu-exec.c b/cpu-exec.c
> > index 8830255..011f51f 100644
> > --- a/cpu-exec.c
> > +++ b/cpu-exec.c
> > @@ -358,7 +358,6 @@ int cpu_exec(CPUArchState *env)
> >      }
> >
> >      cc->cpu_exec_enter(cpu);
> > -    cpu->exception_index = -1;
> >
> >      /* Calculate difference between guest clock and host clock.
> >       * This delay includes the delay of the last cycle, so
> > @@ -378,6 +377,7 @@ int cpu_exec(CPUArchState *env)
> >                      if (ret == EXCP_DEBUG) {
> >                          cpu_handle_debug_exception(env);
> >                      }
> > +                    cpu->exception_index = -1;
> >                      break;
> >                  } else {
> >  #if defined(CONFIG_USER_ONLY)
> > diff --git a/cpus.c b/cpus.c
> > index e53d605..7e8c507 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -1016,6 +1016,7 @@ static void *qemu_tcg_cpu_thread_fn(void *arg)
> >      CPU_FOREACH(cpu) {
> >          cpu->thread_id = qemu_get_thread_id();
> >          cpu->created = true;
> > +        cpu->exception_index = -1;
> >      }
> >      qemu_cond_signal(&qemu_cpu_cond);
> >
> >
> >
> >
> 
> What about user-mode emulation?  Do you need to reset
> cpu->exception_index for it too?

Thank you, I missed that.

Pavel Dovgalyuk

[Qemu-devel] [RFC PATCH v4 07/25] icount: implement icount requesting

[Pavel Dovgalyuk]

Replay uses number of executed instructions to determine corrent events
injection moments. This patch introduces new function for querying the
instructions counter.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c               |   26 +++++++++++++++++++++++---
 include/qemu/timer.h |    1 +
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/cpus.c b/cpus.c
index 7e8c507..2ec6d75 100644
--- a/cpus.c
+++ b/cpus.c
@@ -136,8 +136,7 @@ typedef struct TimersState {
 
 static TimersState timers_state;
 
-/* Return the virtual CPU time, based on the instruction counter.  */
-static int64_t cpu_get_icount_locked(void)
+static int64_t cpu_get_instructions_counter_locked(void)
 {
     int64_t icount;
     CPUState *cpu = current_cpu;
@@ -145,10 +144,31 @@ static int64_t cpu_get_icount_locked(void)
     icount = timers_state.qemu_icount;
     if (cpu) {
         if (!cpu_can_do_io(cpu)) {
-            fprintf(stderr, "Bad clock read\n");
+            fprintf(stderr, "Bad icount read\n");
+            exit(1);
         }
         icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
     }
+    return icount;
+}
+
+int64_t cpu_get_instructions_counter(void)
+{
+    /* This function calls are synchnonized to timer changes,
+       calling cpu_get_instructions_counter_locked without lock is safe */
+    int64_t icount = timers_state.qemu_icount;
+    CPUState *cpu = current_cpu;
+
+    if (cpu) {
+        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
+    }
+    return icount;
+}
+
+/* Return the virtual CPU time, based on the instruction counter.  */
+static int64_t cpu_get_icount_locked(void)
+{
+    int64_t icount = cpu_get_instructions_counter_locked();
     return timers_state.qemu_icount_bias + cpu_icount_to_ns(icount);
 }
 
diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 5f5210d..38a02c5 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -743,6 +743,7 @@ static inline int64_t get_clock(void)
 #endif
 
 /* icount */
+int64_t cpu_get_instructions_counter(void);
 int64_t cpu_get_icount(void);
 int64_t cpu_get_clock(void);
 int64_t cpu_get_clock_offset(void);

Re: [Qemu-devel] [RFC PATCH v4 07/25] icount: implement icount requesting

[Paolo Bonzini]

On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> Replay uses number of executed instructions to determine corrent events
> injection moments. This patch introduces new function for querying the
> instructions counter.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpus.c               |   26 +++++++++++++++++++++++---
>  include/qemu/timer.h |    1 +
>  2 files changed, 24 insertions(+), 3 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 7e8c507..2ec6d75 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -136,8 +136,7 @@ typedef struct TimersState {
>  
>  static TimersState timers_state;
>  
> -/* Return the virtual CPU time, based on the instruction counter.  */
> -static int64_t cpu_get_icount_locked(void)
> +static int64_t cpu_get_instructions_counter_locked(void)
>  {
>      int64_t icount;
>      CPUState *cpu = current_cpu;
> @@ -145,10 +144,31 @@ static int64_t cpu_get_icount_locked(void)
>      icount = timers_state.qemu_icount;
>      if (cpu) {
>          if (!cpu_can_do_io(cpu)) {
> -            fprintf(stderr, "Bad clock read\n");
> +            fprintf(stderr, "Bad icount read\n");
> +            exit(1);
>          }
>          icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
>      }
> +    return icount;
> +}
> +
> +int64_t cpu_get_instructions_counter(void)
> +{
> +    /* This function calls are synchnonized to timer changes,
> +       calling cpu_get_instructions_counter_locked without lock is safe */
> +    int64_t icount = timers_state.qemu_icount;
> +    CPUState *cpu = current_cpu;
> +
> +    if (cpu) {
> +        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
> +    }
> +    return icount;
> +}

This is the same as Frederic's QEMU_CLOCK_ICOUNT clock, right?  Any
reason why one is better than the other?

Paolo

> +/* Return the virtual CPU time, based on the instruction counter.  */
> +static int64_t cpu_get_icount_locked(void)
> +{
> +    int64_t icount = cpu_get_instructions_counter_locked();
>      return timers_state.qemu_icount_bias + cpu_icount_to_ns(icount);
>  }
>  
> diff --git a/include/qemu/timer.h b/include/qemu/timer.h
> index 5f5210d..38a02c5 100644
> --- a/include/qemu/timer.h
> +++ b/include/qemu/timer.h
> @@ -743,6 +743,7 @@ static inline int64_t get_clock(void)
>  #endif
>  
>  /* icount */
> +int64_t cpu_get_instructions_counter(void);
>  int64_t cpu_get_icount(void);
>  int64_t cpu_get_clock(void);
>  int64_t cpu_get_clock_offset(void);
> 

Re: [Qemu-devel] [RFC PATCH v4 07/25] icount: implement icount requesting

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> > Replay uses number of executed instructions to determine corrent events
> > injection moments. This patch introduces new function for querying the
> > instructions counter.
> >
> > Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> > ---
> >  cpus.c               |   26 +++++++++++++++++++++++---
> >  include/qemu/timer.h |    1 +
> >  2 files changed, 24 insertions(+), 3 deletions(-)
> >
> > diff --git a/cpus.c b/cpus.c
> > index 7e8c507..2ec6d75 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -136,8 +136,7 @@ typedef struct TimersState {
> >
> >  static TimersState timers_state;
> >
> > -/* Return the virtual CPU time, based on the instruction counter.  */
> > -static int64_t cpu_get_icount_locked(void)
> > +static int64_t cpu_get_instructions_counter_locked(void)
> >  {
> >      int64_t icount;
> >      CPUState *cpu = current_cpu;
> > @@ -145,10 +144,31 @@ static int64_t cpu_get_icount_locked(void)
> >      icount = timers_state.qemu_icount;
> >      if (cpu) {
> >          if (!cpu_can_do_io(cpu)) {
> > -            fprintf(stderr, "Bad clock read\n");
> > +            fprintf(stderr, "Bad icount read\n");
> > +            exit(1);
> >          }
> >          icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
> >      }
> > +    return icount;
> > +}
> > +
> > +int64_t cpu_get_instructions_counter(void)
> > +{
> > +    /* This function calls are synchnonized to timer changes,
> > +       calling cpu_get_instructions_counter_locked without lock is safe */
> > +    int64_t icount = timers_state.qemu_icount;
> > +    CPUState *cpu = current_cpu;
> > +
> > +    if (cpu) {
> > +        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
> > +    }
> > +    return icount;
> > +}
> 
> This is the same as Frederic's QEMU_CLOCK_ICOUNT clock, right?  Any
> reason why one is better than the other?

Not really. Frederic's QEMU_CLOCK_ICOUNT is measured in nanoseconds.
And this counter is measured in instructions.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v4 07/25] icount: implement icount requesting

[Frederic Konrad]

On 07/11/2014 12:36, Pavel Dovgaluk wrote:
>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>> On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
>>> Replay uses number of executed instructions to determine corrent events
>>> injection moments. This patch introduces new function for querying the
>>> instructions counter.
>>>
>>> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
>>> ---
>>>   cpus.c               |   26 +++++++++++++++++++++++---
>>>   include/qemu/timer.h |    1 +
>>>   2 files changed, 24 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/cpus.c b/cpus.c
>>> index 7e8c507..2ec6d75 100644
>>> --- a/cpus.c
>>> +++ b/cpus.c
>>> @@ -136,8 +136,7 @@ typedef struct TimersState {
>>>
>>>   static TimersState timers_state;
>>>
>>> -/* Return the virtual CPU time, based on the instruction counter.  */
>>> -static int64_t cpu_get_icount_locked(void)
>>> +static int64_t cpu_get_instructions_counter_locked(void)
>>>   {
>>>       int64_t icount;
>>>       CPUState *cpu = current_cpu;
>>> @@ -145,10 +144,31 @@ static int64_t cpu_get_icount_locked(void)
>>>       icount = timers_state.qemu_icount;
>>>       if (cpu) {
>>>           if (!cpu_can_do_io(cpu)) {
>>> -            fprintf(stderr, "Bad clock read\n");
>>> +            fprintf(stderr, "Bad icount read\n");
>>> +            exit(1);
>>>           }
>>>           icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
>>>       }
>>> +    return icount;
>>> +}
>>> +
>>> +int64_t cpu_get_instructions_counter(void)
>>> +{
>>> +    /* This function calls are synchnonized to timer changes,
>>> +       calling cpu_get_instructions_counter_locked without lock is safe */
>>> +    int64_t icount = timers_state.qemu_icount;
>>> +    CPUState *cpu = current_cpu;
>>> +
>>> +    if (cpu) {
>>> +        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
>>> +    }
>>> +    return icount;
>>> +}
>> This is the same as Frederic's QEMU_CLOCK_ICOUNT clock, right?  Any
>> reason why one is better than the other?
> Not really. Frederic's QEMU_CLOCK_ICOUNT is measured in nanoseconds.
> And this counter is measured in instructions.
>
> Pavel Dovgalyuk
>
So you transformed your implementation to use icount?
I didn't saw that in the cover letter.

I guess you replay event's according to the current instruction?
I implemented icount clock with the QEMUTimer API so a callback can be
notified during the replay at the right instruction.

By the way do you have a git tree somewhere where I can get the patches?

Fred

Re: [Qemu-devel] [RFC PATCH v4 07/25] icount: implement icount requesting

[Pavel Dovgaluk]

> From: Frederic Konrad [mailto:fred.konrad@greensocs.com]
> On 07/11/2014 12:36, Pavel Dovgaluk wrote:
> >> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> >> On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> >>> Replay uses number of executed instructions to determine corrent events
> >>> injection moments. This patch introduces new function for querying the
> >>> instructions counter.
> >>>
> >>> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> >>> ---
> >>>   cpus.c               |   26 +++++++++++++++++++++++---
> >>>   include/qemu/timer.h |    1 +
> >>>   2 files changed, 24 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/cpus.c b/cpus.c
> >>> index 7e8c507..2ec6d75 100644
> >>> --- a/cpus.c
> >>> +++ b/cpus.c
> >>> @@ -136,8 +136,7 @@ typedef struct TimersState {
> >>>
> >>>   static TimersState timers_state;
> >>>
> >>> -/* Return the virtual CPU time, based on the instruction counter.  */
> >>> -static int64_t cpu_get_icount_locked(void)
> >>> +static int64_t cpu_get_instructions_counter_locked(void)
> >>>   {
> >>>       int64_t icount;
> >>>       CPUState *cpu = current_cpu;
> >>> @@ -145,10 +144,31 @@ static int64_t cpu_get_icount_locked(void)
> >>>       icount = timers_state.qemu_icount;
> >>>       if (cpu) {
> >>>           if (!cpu_can_do_io(cpu)) {
> >>> -            fprintf(stderr, "Bad clock read\n");
> >>> +            fprintf(stderr, "Bad icount read\n");
> >>> +            exit(1);
> >>>           }
> >>>           icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
> >>>       }
> >>> +    return icount;
> >>> +}
> >>> +
> >>> +int64_t cpu_get_instructions_counter(void)
> >>> +{
> >>> +    /* This function calls are synchnonized to timer changes,
> >>> +       calling cpu_get_instructions_counter_locked without lock is safe */
> >>> +    int64_t icount = timers_state.qemu_icount;
> >>> +    CPUState *cpu = current_cpu;
> >>> +
> >>> +    if (cpu) {
> >>> +        icount -= (cpu->icount_decr.u16.low + cpu->icount_extra);
> >>> +    }
> >>> +    return icount;
> >>> +}
> >> This is the same as Frederic's QEMU_CLOCK_ICOUNT clock, right?  Any
> >> reason why one is better than the other?
> > Not really. Frederic's QEMU_CLOCK_ICOUNT is measured in nanoseconds.
> > And this counter is measured in instructions.
> >
> So you transformed your implementation to use icount?

Right, new implementation uses icount.

> I guess you replay event's according to the current instruction?
> I implemented icount clock with the QEMUTimer API so a callback can be
> notified during the replay at the right instruction.

We cannot use timers, because there are synchronization events 
(like exceptions, interrupts, timer reads) that control the execution correctness.

> By the way do you have a git tree somewhere where I can get the patches?

No, there is no public repository. There are only the patches in the mail list.

Pavel Dovgalyuk

[Qemu-devel] [RFC PATCH v4 08/25] icount: improve enable/disable ticks

[Pavel Dovgalyuk]

This patch eliminates call of the cpu_get_real_ticks while enabling
or disabling the virtual timer in icount mode. These calls are used
for cpu_ticks_offset which is not needed in this mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c |   12 ++++++++----
 1 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/cpus.c b/cpus.c
index 2ec6d75..15ac3a1 100644
--- a/cpus.c
+++ b/cpus.c
@@ -267,8 +267,10 @@ void cpu_enable_ticks(void)
     /* Here, the really thing protected by seqlock is cpu_clock_offset. */
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (!timers_state.cpu_ticks_enabled) {
-        timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
-        timers_state.cpu_clock_offset -= get_clock();
+        if (!use_icount) {
+            timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
+            timers_state.cpu_clock_offset -= get_clock();
+        }
         timers_state.cpu_ticks_enabled = 1;
     }
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);
@@ -283,8 +285,10 @@ void cpu_disable_ticks(void)
     /* Here, the really thing protected by seqlock is cpu_clock_offset. */
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (timers_state.cpu_ticks_enabled) {
-        timers_state.cpu_ticks_offset += cpu_get_real_ticks();
-        timers_state.cpu_clock_offset = cpu_get_clock_locked();
+        if (!use_icount) {
+            timers_state.cpu_ticks_offset += cpu_get_real_ticks();
+            timers_state.cpu_clock_offset = cpu_get_clock_locked();
+        }
         timers_state.cpu_ticks_enabled = 0;
     }
     seqlock_write_unlock(&timers_state.vm_clock_seqlock);

Re: [Qemu-devel] [RFC PATCH v4 08/25] icount: improve enable/disable ticks

[Paolo Bonzini]

On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> This patch eliminates call of the cpu_get_real_ticks while enabling
> or disabling the virtual timer in icount mode. These calls are used
> for cpu_ticks_offset which is not needed in this mode.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpus.c |   12 ++++++++----
>  1 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index 2ec6d75..15ac3a1 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -267,8 +267,10 @@ void cpu_enable_ticks(void)
>      /* Here, the really thing protected by seqlock is cpu_clock_offset. */
>      seqlock_write_lock(&timers_state.vm_clock_seqlock);
>      if (!timers_state.cpu_ticks_enabled) {
> -        timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
> -        timers_state.cpu_clock_offset -= get_clock();
> +        if (!use_icount) {
> +            timers_state.cpu_ticks_offset -= cpu_get_real_ticks();
> +            timers_state.cpu_clock_offset -= get_clock();
> +        }
>          timers_state.cpu_ticks_enabled = 1;
>      }
>      seqlock_write_unlock(&timers_state.vm_clock_seqlock);
> @@ -283,8 +285,10 @@ void cpu_disable_ticks(void)
>      /* Here, the really thing protected by seqlock is cpu_clock_offset. */
>      seqlock_write_lock(&timers_state.vm_clock_seqlock);
>      if (timers_state.cpu_ticks_enabled) {
> -        timers_state.cpu_ticks_offset += cpu_get_real_ticks();
> -        timers_state.cpu_clock_offset = cpu_get_clock_locked();
> +        if (!use_icount) {
> +            timers_state.cpu_ticks_offset += cpu_get_real_ticks();
> +            timers_state.cpu_clock_offset = cpu_get_clock_locked();
> +        }
>          timers_state.cpu_ticks_enabled = 0;
>      }
>      seqlock_write_unlock(&timers_state.vm_clock_seqlock);
> 

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

[Qemu-devel] [RFC PATCH v4 09/25] replay: introduce icount event

[Pavel Dovgalyuk]

This patch adds icount event to the replay subsystem. This event corresponds
to execution of several instructions and used to synchronize input events
in the replay phase.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/replay-internal.c |   14 ++++++++++++++
 replay/replay-internal.h |   18 ++++++++++++++++++
 replay/replay.c          |   45 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    7 +++++++
 4 files changed, 84 insertions(+), 0 deletions(-)

diff --git a/replay/replay-internal.c b/replay/replay-internal.c
index 29f995d..14881b7 100755
--- a/replay/replay-internal.c
+++ b/replay/replay-internal.c
@@ -10,6 +10,7 @@
  */
 
 #include "qemu-common.h"
+#include "replay.h"
 #include "replay-internal.h"
 
 volatile unsigned int replay_data_kind = -1;
@@ -139,3 +140,16 @@ void replay_fetch_data_kind(void)
         }
     }
 }
+
+/*! Saves cached instructions. */
+void replay_save_instructions(void)
+{
+    if (replay_file && replay_mode == REPLAY_MODE_RECORD) {
+        int diff = (int)(replay_get_current_step() - replay_state.current_step);
+        if (first_cpu != NULL && diff > 0) {
+            replay_put_event(EVENT_INSTRUCTION);
+            replay_put_dword(diff);
+            replay_state.current_step += diff;
+        }
+    }
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 45bb344..577d1e9 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -14,6 +14,17 @@
 
 #include <stdio.h>
 
+/* for instruction event */
+#define EVENT_INSTRUCTION           32
+
+typedef struct ReplayState {
+    /*! Current step - number of processed instructions and timer events. */
+    uint64_t current_step;
+    /*! Number of instructions to be executed before other events happen. */
+    int instructions_count;
+} ReplayState;
+extern ReplayState replay_state;
+
 extern volatile unsigned int replay_data_kind;
 extern volatile unsigned int replay_has_unread_data;
 
@@ -47,4 +58,11 @@ void replay_save_instructions(void);
     Terminates the program in case of error. */
 void validate_data_kind(int kind);
 
+/*! Skips async events until some sync event will be found. */
+bool skip_async_events(int stop_event);
+/*! Skips async events invocations from the input,
+    until required data kind is found. If the requested data is not found
+    reports an error and stops the execution. */
+void skip_async_events_until(unsigned int kind);
+
 #endif
diff --git a/replay/replay.c b/replay/replay.c
index ac976b2..c305e0c 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -9,7 +9,10 @@
  *
  */
 
+#include "qemu-common.h"
 #include "replay.h"
+#include "replay-internal.h"
+#include "qemu/timer.h"
 
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 /*! Stores current submode for PLAY mode */
@@ -18,8 +21,50 @@ ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;
 /* Suffix for the disk images filenames */
 char *replay_image_suffix;
 
+ReplayState replay_state;
 
 ReplaySubmode replay_get_play_submode(void)
 {
     return play_submode;
 }
+
+bool skip_async_events(int stop_event)
+{
+    /* nothing to skip - not all instructions used */
+    if (replay_state.instructions_count != 0
+        && replay_has_unread_data) {
+        return stop_event == EVENT_INSTRUCTION;
+    }
+
+    bool res = false;
+    while (true) {
+        replay_fetch_data_kind();
+        if (stop_event == replay_data_kind) {
+            res = true;
+        }
+        switch (replay_data_kind) {
+        case EVENT_INSTRUCTION:
+            replay_state.instructions_count = replay_get_dword();
+            return res;
+        default:
+            /* clock, time_t, checkpoint and other events */
+            return res;
+        }
+    }
+
+    return res;
+}
+
+void skip_async_events_until(unsigned int kind)
+{
+    if (!skip_async_events(kind)) {
+        fprintf(stderr, "%"PRId64": Read data kind %d instead of expected %d\n",
+            replay_get_current_step(), replay_data_kind, kind);
+        exit(1);
+    }
+}
+
+uint64_t replay_get_current_step(void)
+{
+    return cpu_get_instructions_counter();
+}
diff --git a/replay/replay.h b/replay/replay.h
index 51a18fe..e40daf5 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -12,6 +12,8 @@
  *
  */
 
+#include <stdbool.h>
+#include <stdint.h>
 #include "qapi-types.h"
 
 extern ReplayMode replay_mode;
@@ -20,4 +22,9 @@ extern char *replay_image_suffix;
 /*! Returns replay play submode */
 ReplaySubmode replay_get_play_submode(void);
 
+/* Processing the instructions */
+
+/*! Returns number of executed instructions. */
+uint64_t replay_get_current_step(void);
+
 #endif

[Qemu-devel] [RFC PATCH v4 10/25] i386: do not cross the pages boundaries in replay mode

[Pavel Dovgalyuk]

This patch denies crossing the boundary of the pages in the replay mode,
because it can cause an exception. Do it only when boundary is
crossed by the first instruction in the block.
If current instruction already crossed the bound - it's ok,
because an exception hasn't stopped this code.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 target-i386/cpu.h       |    7 +++++++
 target-i386/translate.c |   14 ++++++++++++++
 2 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index 2968749..bc3f9f5 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -28,6 +28,13 @@
 #define TARGET_LONG_BITS 32
 #endif
 
+/* Maximum instruction code size */
+#ifdef TARGET_X86_64
+#define TARGET_MAX_INSN_SIZE 16
+#else
+#define TARGET_MAX_INSN_SIZE 16
+#endif
+
 /* target supports implicit self modifying code */
 #define TARGET_HAS_SMC
 /* support for self modifying code even if the modified instruction is
diff --git a/target-i386/translate.c b/target-i386/translate.c
index 4d5dfb3..a264908 100644
--- a/target-i386/translate.c
+++ b/target-i386/translate.c
@@ -8035,6 +8035,20 @@ static inline void gen_intermediate_code_internal(X86CPU *cpu,
             gen_eob(dc);
             break;
         }
+        /* Do not cross the boundary of the pages in icount mode,
+           it can cause an exception. Do it only when boundary is
+           crossed by the first instruction in the block.
+           If current instruction already crossed the bound - it's ok,
+           because an exception hasn't stopped this code.
+         */
+        if (use_icount
+            && ((pc_ptr & TARGET_PAGE_MASK)
+                != ((pc_ptr + TARGET_MAX_INSN_SIZE - 1) & TARGET_PAGE_MASK)
+                || (pc_ptr & ~TARGET_PAGE_MASK) == 0)) {
+            gen_jmp_im(pc_ptr - dc->cs_base);
+            gen_eob(dc);
+            break;
+        }
         /* if too long translation, stop generation too */
         if (tcg_ctx.gen_opc_ptr >= gen_opc_end ||
             (pc_ptr - pc_start) >= (TARGET_PAGE_SIZE - 32) ||

Re: [Qemu-devel] [RFC PATCH v4 10/25] i386: do not cross the pages boundaries in replay mode

[Paolo Bonzini]

On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> This patch denies crossing the boundary of the pages in the replay mode,
> because it can cause an exception. Do it only when boundary is
> crossed by the first instruction in the block.
> If current instruction already crossed the bound - it's ok,
> because an exception hasn't stopped this code.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  target-i386/cpu.h       |    7 +++++++
>  target-i386/translate.c |   14 ++++++++++++++
>  2 files changed, 21 insertions(+), 0 deletions(-)
> 
> diff --git a/target-i386/cpu.h b/target-i386/cpu.h
> index 2968749..bc3f9f5 100644
> --- a/target-i386/cpu.h
> +++ b/target-i386/cpu.h
> @@ -28,6 +28,13 @@
>  #define TARGET_LONG_BITS 32
>  #endif
>  
> +/* Maximum instruction code size */
> +#ifdef TARGET_X86_64
> +#define TARGET_MAX_INSN_SIZE 16
> +#else
> +#define TARGET_MAX_INSN_SIZE 16
> +#endif
> +
>  /* target supports implicit self modifying code */
>  #define TARGET_HAS_SMC
>  /* support for self modifying code even if the modified instruction is
> diff --git a/target-i386/translate.c b/target-i386/translate.c
> index 4d5dfb3..a264908 100644
> --- a/target-i386/translate.c
> +++ b/target-i386/translate.c
> @@ -8035,6 +8035,20 @@ static inline void gen_intermediate_code_internal(X86CPU *cpu,
>              gen_eob(dc);
>              break;
>          }
> +        /* Do not cross the boundary of the pages in icount mode,
> +           it can cause an exception. Do it only when boundary is
> +           crossed by the first instruction in the block.
> +           If current instruction already crossed the bound - it's ok,
> +           because an exception hasn't stopped this code.
> +         */
> +        if (use_icount
> +            && ((pc_ptr & TARGET_PAGE_MASK)
> +                != ((pc_ptr + TARGET_MAX_INSN_SIZE - 1) & TARGET_PAGE_MASK)
> +                || (pc_ptr & ~TARGET_PAGE_MASK) == 0)) {
> +            gen_jmp_im(pc_ptr - dc->cs_base);
> +            gen_eob(dc);
> +            break;
> +        }
>          /* if too long translation, stop generation too */
>          if (tcg_ctx.gen_opc_ptr >= gen_opc_end ||
>              (pc_ptr - pc_start) >= (TARGET_PAGE_SIZE - 32) ||
> 

Why only in icount mode?  Does it have a sensible performance problem?

Paolo

Re: [Qemu-devel] [RFC PATCH v4 10/25] i386: do not cross the pages boundaries in replay mode

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> > This patch denies crossing the boundary of the pages in the replay mode,
> > because it can cause an exception. Do it only when boundary is
> > crossed by the first instruction in the block.
> > If current instruction already crossed the bound - it's ok,
> > because an exception hasn't stopped this code.
> >
> > Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> > ---
> >  target-i386/cpu.h       |    7 +++++++
> >  target-i386/translate.c |   14 ++++++++++++++
> >  2 files changed, 21 insertions(+), 0 deletions(-)
> >
> > diff --git a/target-i386/cpu.h b/target-i386/cpu.h
> > index 2968749..bc3f9f5 100644
> > --- a/target-i386/cpu.h
> > +++ b/target-i386/cpu.h
> > @@ -28,6 +28,13 @@
> >  #define TARGET_LONG_BITS 32
> >  #endif
> >
> > +/* Maximum instruction code size */
> > +#ifdef TARGET_X86_64
> > +#define TARGET_MAX_INSN_SIZE 16
> > +#else
> > +#define TARGET_MAX_INSN_SIZE 16
> > +#endif
> > +
> >  /* target supports implicit self modifying code */
> >  #define TARGET_HAS_SMC
> >  /* support for self modifying code even if the modified instruction is
> > diff --git a/target-i386/translate.c b/target-i386/translate.c
> > index 4d5dfb3..a264908 100644
> > --- a/target-i386/translate.c
> > +++ b/target-i386/translate.c
> > @@ -8035,6 +8035,20 @@ static inline void gen_intermediate_code_internal(X86CPU *cpu,
> >              gen_eob(dc);
> >              break;
> >          }
> > +        /* Do not cross the boundary of the pages in icount mode,
> > +           it can cause an exception. Do it only when boundary is
> > +           crossed by the first instruction in the block.
> > +           If current instruction already crossed the bound - it's ok,
> > +           because an exception hasn't stopped this code.
> > +         */
> > +        if (use_icount
> > +            && ((pc_ptr & TARGET_PAGE_MASK)
> > +                != ((pc_ptr + TARGET_MAX_INSN_SIZE - 1) & TARGET_PAGE_MASK)
> > +                || (pc_ptr & ~TARGET_PAGE_MASK) == 0)) {
> > +            gen_jmp_im(pc_ptr - dc->cs_base);
> > +            gen_eob(dc);
> > +            break;
> > +        }
> >          /* if too long translation, stop generation too */
> >          if (tcg_ctx.gen_opc_ptr >= gen_opc_end ||
> >              (pc_ptr - pc_start) >= (TARGET_PAGE_SIZE - 32) ||
> >
> 
> Why only in icount mode?  Does it have a sensible performance problem?

Maybe. This is the same problem, which was discussed here for ARM: http://lists.gnu.org/archive/html/qemu-devel/2014-10/msg02232.html
But I haven't figured out what I have to do with it.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v4 10/25] i386: do not cross the pages boundaries in replay mode

[Andreas Färber]

Am 07.11.2014 um 11:32 schrieb Pavel Dovgalyuk:
> This patch denies crossing the boundary of the pages in the replay mode,
> because it can cause an exception. Do it only when boundary is
> crossed by the first instruction in the block.
> If current instruction already crossed the bound - it's ok,
> because an exception hasn't stopped this code.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  target-i386/cpu.h       |    7 +++++++
>  target-i386/translate.c |   14 ++++++++++++++
>  2 files changed, 21 insertions(+), 0 deletions(-)
> 
> diff --git a/target-i386/cpu.h b/target-i386/cpu.h
> index 2968749..bc3f9f5 100644
> --- a/target-i386/cpu.h
> +++ b/target-i386/cpu.h
> @@ -28,6 +28,13 @@
>  #define TARGET_LONG_BITS 32
>  #endif
>  
> +/* Maximum instruction code size */
> +#ifdef TARGET_X86_64
> +#define TARGET_MAX_INSN_SIZE 16
> +#else
> +#define TARGET_MAX_INSN_SIZE 16
> +#endif

Is this a spot-the-difference game? ;)
Seriously, if they're the same values, just drop the #ifdef.

> +
>  /* target supports implicit self modifying code */
>  #define TARGET_HAS_SMC
>  /* support for self modifying code even if the modified instruction is
[snip]

Regards,
Andreas

-- 
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 21284 AG Nürnberg

[Qemu-devel] [RFC PATCH v4 11/25] cpu-exec: allow temporary disabling icount

[Pavel Dovgalyuk]

This patch is required for deterministic replay to generate an exception
by trying executing an instruction without changing icount.
It adds new flag to TB for disabling icount while translating it.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c              |    6 +++---
 include/exec/exec-all.h |    7 ++++---
 translate-all.c         |   22 ++++++++++++++++++++--
 3 files changed, 27 insertions(+), 8 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index 011f51f..e7206ac 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -198,7 +198,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
 /* Execute the code without caching the generated code. An interpreter
    could be used if available. */
 static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
-                             TranslationBlock *orig_tb)
+                             TranslationBlock *orig_tb, bool ignore_icount)
 {
     CPUState *cpu = ENV_GET_CPU(env);
     TranslationBlock *tb;
@@ -214,7 +214,7 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
     /* tb_gen_code can flush our orig_tb, invalidate it now */
     tb_phys_invalidate(orig_tb, -1);
     tb = tb_gen_code(cpu, pc, cs_base, flags,
-                     max_cycles);
+                     max_cycles | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
     cpu->current_tb = tb;
     /* execute the generated code */
     trace_exec_tb_nocache(tb, tb->pc);
@@ -516,7 +516,7 @@ int cpu_exec(CPUArchState *env)
                         } else {
                             if (insns_left > 0) {
                                 /* Execute remaining instructions.  */
-                                cpu_exec_nocache(env, insns_left, tb);
+                                cpu_exec_nocache(env, insns_left, tb, false);
                                 align_clocks(&sc, cpu);
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 1d17f75..052e8ef 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -142,9 +142,10 @@ struct TranslationBlock {
     uint64_t flags; /* flags defining in which context the code was generated */
     uint16_t size;      /* size of target code for this block (1 <=
                            size <= TARGET_PAGE_SIZE) */
-    uint16_t cflags;    /* compile flags */
-#define CF_COUNT_MASK  0x7fff
-#define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
+    uint32_t cflags;    /* compile flags */
+#define CF_COUNT_MASK    0x7fff
+#define CF_LAST_IO       0x8000 /* Last insn may be an IO access.  */
+#define CF_IGNORE_ICOUNT 0x10000 /* Do not generate icount code */
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
diff --git a/translate-all.c b/translate-all.c
index 7177b71..182ea7d 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -207,6 +207,7 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
     TCGContext *s = &tcg_ctx;
     int j;
     uintptr_t tc_ptr;
+    int old_icount = use_icount;
 #ifdef CONFIG_PROFILER
     int64_t ti;
 #endif
@@ -218,6 +219,10 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
 
     gen_intermediate_code_pc(env, tb);
 
+    if (tb->cflags & CF_IGNORE_ICOUNT) {
+        use_icount = 0;
+    }
+
     if (use_icount) {
         /* Reset the cycle counter to the start of the block.  */
         cpu->icount_decr.u16.low += tb->icount;
@@ -227,8 +232,10 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
 
     /* find opc index corresponding to search_pc */
     tc_ptr = (uintptr_t)tb->tc_ptr;
-    if (searched_pc < tc_ptr)
+    if (searched_pc < tc_ptr) {
+        use_icount = old_icount;
         return -1;
+    }
 
     s->tb_next_offset = tb->tb_next_offset;
 #ifdef USE_DIRECT_JUMP
@@ -240,8 +247,10 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
 #endif
     j = tcg_gen_code_search_pc(s, (tcg_insn_unit *)tc_ptr,
                                searched_pc - tc_ptr);
-    if (j < 0)
+    if (j < 0) {
+        use_icount = old_icount;
         return -1;
+    }
     /* now find start of instruction before */
     while (s->gen_opc_instr_start[j] == 0) {
         j--;
@@ -254,6 +263,8 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
     s->restore_time += profile_getclock() - ti;
     s->restore_count++;
 #endif
+    use_icount = old_icount;
+
     return 0;
 }
 
@@ -264,6 +275,8 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t retaddr)
     tb = tb_find_pc(retaddr);
     if (tb) {
         cpu_restore_state_from_tb(cpu, tb, retaddr);
+        /* tb could be temporary, generated by exec nocache */
+        tb_phys_invalidate(tb, -1);
         return true;
     }
     return false;
@@ -1045,6 +1058,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
     tb_page_addr_t phys_pc, phys_page2;
     target_ulong virt_page2;
     int code_gen_size;
+    int old_icount = use_icount;
 
     phys_pc = get_page_addr_code(env, pc);
     tb = tb_alloc(pc);
@@ -1060,9 +1074,13 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
     tb->cs_base = cs_base;
     tb->flags = flags;
     tb->cflags = cflags;
+    if (cflags & CF_IGNORE_ICOUNT) {
+        use_icount = 0;
+    }
     cpu_gen_code(env, tb, &code_gen_size);
     tcg_ctx.code_gen_ptr = (void *)(((uintptr_t)tcg_ctx.code_gen_ptr +
             code_gen_size + CODE_GEN_ALIGN - 1) & ~(CODE_GEN_ALIGN - 1));
+    use_icount = old_icount;
 
     /* check next page if needed */
     virt_page2 = (pc + tb->size - 1) & TARGET_PAGE_MASK;

Re: [Qemu-devel] [RFC PATCH v4 11/25] cpu-exec: allow temporary disabling icount

[Paolo Bonzini]

On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> This patch is required for deterministic replay to generate an exception
> by trying executing an instruction without changing icount.
> It adds new flag to TB for disabling icount while translating it.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpu-exec.c              |    6 +++---
>  include/exec/exec-all.h |    7 ++++---
>  translate-all.c         |   22 ++++++++++++++++++++--
>  3 files changed, 27 insertions(+), 8 deletions(-)
> 
> diff --git a/cpu-exec.c b/cpu-exec.c
> index 011f51f..e7206ac 100644
> --- a/cpu-exec.c
> +++ b/cpu-exec.c
> @@ -198,7 +198,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t *tb_ptr)
>  /* Execute the code without caching the generated code. An interpreter
>     could be used if available. */
>  static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
> -                             TranslationBlock *orig_tb)
> +                             TranslationBlock *orig_tb, bool ignore_icount)
>  {
>      CPUState *cpu = ENV_GET_CPU(env);
>      TranslationBlock *tb;
> @@ -214,7 +214,7 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
>      /* tb_gen_code can flush our orig_tb, invalidate it now */
>      tb_phys_invalidate(orig_tb, -1);
>      tb = tb_gen_code(cpu, pc, cs_base, flags,
> -                     max_cycles);
> +                     max_cycles | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
>      cpu->current_tb = tb;
>      /* execute the generated code */
>      trace_exec_tb_nocache(tb, tb->pc);
> @@ -516,7 +516,7 @@ int cpu_exec(CPUArchState *env)
>                          } else {
>                              if (insns_left > 0) {
>                                  /* Execute remaining instructions.  */
> -                                cpu_exec_nocache(env, insns_left, tb);
> +                                cpu_exec_nocache(env, insns_left, tb, false);
>                                  align_clocks(&sc, cpu);
>                              }
>                              cpu->exception_index = EXCP_INTERRUPT;
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 1d17f75..052e8ef 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -142,9 +142,10 @@ struct TranslationBlock {
>      uint64_t flags; /* flags defining in which context the code was generated */
>      uint16_t size;      /* size of target code for this block (1 <=
>                             size <= TARGET_PAGE_SIZE) */
> -    uint16_t cflags;    /* compile flags */
> -#define CF_COUNT_MASK  0x7fff
> -#define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
> +    uint32_t cflags;    /* compile flags */
> +#define CF_COUNT_MASK    0x7fff
> +#define CF_LAST_IO       0x8000 /* Last insn may be an IO access.  */
> +#define CF_IGNORE_ICOUNT 0x10000 /* Do not generate icount code */
>  
>      void *tc_ptr;    /* pointer to the translated code */
>      /* next matching tb for physical address. */
> diff --git a/translate-all.c b/translate-all.c
> index 7177b71..182ea7d 100644
> --- a/translate-all.c
> +++ b/translate-all.c
> @@ -207,6 +207,7 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
>      TCGContext *s = &tcg_ctx;
>      int j;
>      uintptr_t tc_ptr;
> +    int old_icount = use_icount;
>  #ifdef CONFIG_PROFILER
>      int64_t ti;
>  #endif
> @@ -218,6 +219,10 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
>  
>      gen_intermediate_code_pc(env, tb);
>  
> +    if (tb->cflags & CF_IGNORE_ICOUNT) {

Would it be possible to instead have a CF_USE_ICOUNT tbflag, and check
it when applicable instead of use_icount?  It seems cleaner, because it
would avoid playing with global variables, but perhaps it's hard to do it.

Paolo

> +        use_icount = 0;
> +    }
> +
>      if (use_icount) {
>          /* Reset the cycle counter to the start of the block.  */
>          cpu->icount_decr.u16.low += tb->icount;
> @@ -227,8 +232,10 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
>  
>      /* find opc index corresponding to search_pc */
>      tc_ptr = (uintptr_t)tb->tc_ptr;
> -    if (searched_pc < tc_ptr)
> +    if (searched_pc < tc_ptr) {
> +        use_icount = old_icount;
>          return -1;
> +    }
>  
>      s->tb_next_offset = tb->tb_next_offset;
>  #ifdef USE_DIRECT_JUMP
> @@ -240,8 +247,10 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
>  #endif
>      j = tcg_gen_code_search_pc(s, (tcg_insn_unit *)tc_ptr,
>                                 searched_pc - tc_ptr);
> -    if (j < 0)
> +    if (j < 0) {
> +        use_icount = old_icount;
>          return -1;
> +    }
>      /* now find start of instruction before */
>      while (s->gen_opc_instr_start[j] == 0) {
>          j--;
> @@ -254,6 +263,8 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock *tb,
>      s->restore_time += profile_getclock() - ti;
>      s->restore_count++;
>  #endif
> +    use_icount = old_icount;
> +
>      return 0;
>  }
>  
> @@ -264,6 +275,8 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t retaddr)
>      tb = tb_find_pc(retaddr);
>      if (tb) {
>          cpu_restore_state_from_tb(cpu, tb, retaddr);
> +        /* tb could be temporary, generated by exec nocache */
> +        tb_phys_invalidate(tb, -1);
>          return true;
>      }
>      return false;
> @@ -1045,6 +1058,7 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
>      tb_page_addr_t phys_pc, phys_page2;
>      target_ulong virt_page2;
>      int code_gen_size;
> +    int old_icount = use_icount;
>  
>      phys_pc = get_page_addr_code(env, pc);
>      tb = tb_alloc(pc);
> @@ -1060,9 +1074,13 @@ TranslationBlock *tb_gen_code(CPUState *cpu,
>      tb->cs_base = cs_base;
>      tb->flags = flags;
>      tb->cflags = cflags;
> +    if (cflags & CF_IGNORE_ICOUNT) {
> +        use_icount = 0;
> +    }
>      cpu_gen_code(env, tb, &code_gen_size);
>      tcg_ctx.code_gen_ptr = (void *)(((uintptr_t)tcg_ctx.code_gen_ptr +
>              code_gen_size + CODE_GEN_ALIGN - 1) & ~(CODE_GEN_ALIGN - 1));
> +    use_icount = old_icount;
>  
>      /* check next page if needed */
>      virt_page2 = (pc + tb->size - 1) & TARGET_PAGE_MASK;
> 

Re: [Qemu-devel] [RFC PATCH v4 11/25] cpu-exec: allow temporary disabling icount

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> > This patch is required for deterministic replay to generate an exception
> > by trying executing an instruction without changing icount.
> > It adds new flag to TB for disabling icount while translating it.
> >
> > Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> > ---
> >  cpu-exec.c              |    6 +++---
> >  include/exec/exec-all.h |    7 ++++---
> >  translate-all.c         |   22 ++++++++++++++++++++--
> >  3 files changed, 27 insertions(+), 8 deletions(-)
> >
> > diff --git a/cpu-exec.c b/cpu-exec.c
> > index 011f51f..e7206ac 100644
> > --- a/cpu-exec.c
> > +++ b/cpu-exec.c
> > @@ -198,7 +198,7 @@ static inline tcg_target_ulong cpu_tb_exec(CPUState *cpu, uint8_t
> *tb_ptr)
> >  /* Execute the code without caching the generated code. An interpreter
> >     could be used if available. */
> >  static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
> > -                             TranslationBlock *orig_tb)
> > +                             TranslationBlock *orig_tb, bool ignore_icount)
> >  {
> >      CPUState *cpu = ENV_GET_CPU(env);
> >      TranslationBlock *tb;
> > @@ -214,7 +214,7 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
> >      /* tb_gen_code can flush our orig_tb, invalidate it now */
> >      tb_phys_invalidate(orig_tb, -1);
> >      tb = tb_gen_code(cpu, pc, cs_base, flags,
> > -                     max_cycles);
> > +                     max_cycles | (ignore_icount ? CF_IGNORE_ICOUNT : 0));
> >      cpu->current_tb = tb;
> >      /* execute the generated code */
> >      trace_exec_tb_nocache(tb, tb->pc);
> > @@ -516,7 +516,7 @@ int cpu_exec(CPUArchState *env)
> >                          } else {
> >                              if (insns_left > 0) {
> >                                  /* Execute remaining instructions.  */
> > -                                cpu_exec_nocache(env, insns_left, tb);
> > +                                cpu_exec_nocache(env, insns_left, tb, false);
> >                                  align_clocks(&sc, cpu);
> >                              }
> >                              cpu->exception_index = EXCP_INTERRUPT;
> > diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> > index 1d17f75..052e8ef 100644
> > --- a/include/exec/exec-all.h
> > +++ b/include/exec/exec-all.h
> > @@ -142,9 +142,10 @@ struct TranslationBlock {
> >      uint64_t flags; /* flags defining in which context the code was generated */
> >      uint16_t size;      /* size of target code for this block (1 <=
> >                             size <= TARGET_PAGE_SIZE) */
> > -    uint16_t cflags;    /* compile flags */
> > -#define CF_COUNT_MASK  0x7fff
> > -#define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
> > +    uint32_t cflags;    /* compile flags */
> > +#define CF_COUNT_MASK    0x7fff
> > +#define CF_LAST_IO       0x8000 /* Last insn may be an IO access.  */
> > +#define CF_IGNORE_ICOUNT 0x10000 /* Do not generate icount code */
> >
> >      void *tc_ptr;    /* pointer to the translated code */
> >      /* next matching tb for physical address. */
> > diff --git a/translate-all.c b/translate-all.c
> > index 7177b71..182ea7d 100644
> > --- a/translate-all.c
> > +++ b/translate-all.c
> > @@ -207,6 +207,7 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock
> *tb,
> >      TCGContext *s = &tcg_ctx;
> >      int j;
> >      uintptr_t tc_ptr;
> > +    int old_icount = use_icount;
> >  #ifdef CONFIG_PROFILER
> >      int64_t ti;
> >  #endif
> > @@ -218,6 +219,10 @@ static int cpu_restore_state_from_tb(CPUState *cpu, TranslationBlock
> *tb,
> >
> >      gen_intermediate_code_pc(env, tb);
> >
> > +    if (tb->cflags & CF_IGNORE_ICOUNT) {
> 
> Would it be possible to instead have a CF_USE_ICOUNT tbflag, and check
> it when applicable instead of use_icount?  It seems cleaner, because it
> would avoid playing with global variables, but perhaps it's hard to do it.

Using context-specific flag is better than using global variable like use_icount.
But I could not test such changes because they will affect all of the platforms
(and will be also hard to implement for i386).

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v4 11/25] cpu-exec: allow temporary disabling icount

[Paolo Bonzini]

On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
>          cpu_restore_state_from_tb(cpu, tb, retaddr);
> +        /* tb could be temporary, generated by exec nocache */
> +        tb_phys_invalidate(tb, -1);

Would you need to do the same sequence as cpu_exec_nocache in this case?

    cpu->current_tb = NULL;
    tb_phys_invalidate(tb, -1);
    tb_free(tb);

In this case, perhaps something like this would be better:

diff --git a/cpu-exec.c b/cpu-exec.c
index 10c0f42..faf8041 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -209,7 +209,7 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
         max_cycles = CF_COUNT_MASK;
 
     tb = tb_gen_code(cpu, orig_tb->pc, orig_tb->cs_base, orig_tb->flags,
-                     max_cycles);
+                     max_cycles | CF_NOCACHE);
     cpu->current_tb = tb;
     /* execute the generated code */
     trace_exec_tb_nocache(tb, tb->pc);
diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
index 9b67d16..41464f3 100644
--- a/include/exec/exec-all.h
+++ b/include/exec/exec-all.h
@@ -148,6 +148,7 @@ struct TranslationBlock {
 #define CF_COUNT_MASK  0x7fff
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
 #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
+#define CF_NOCACHE     0x40000 /* To be freed after execution */
 
     void *tc_ptr;    /* pointer to the translated code */
     /* next matching tb for physical address. */
diff --git a/translate-all.c b/translate-all.c
index 1a27df7..6ea4da4 100644
--- a/translate-all.c
+++ b/translate-all.c
@@ -264,6 +264,12 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t retaddr)
     tb = tb_find_pc(retaddr);
     if (tb) {
         cpu_restore_state_from_tb(cpu, tb, retaddr);
+        if (tb->cflags & CF_NOCACHE) {
+            /* one-shot translation, invalidate it immediately */
+            cpu->current_tb = NULL;
+            tb_phys_invalidate(tb, -1);
+            tb_free(tb);
+        }
         return true;
     }
     return false;


I pushed it to an icount branch on my github repository, together with
other patches that introduce CF_USE_ICOUNT instead of touching the
use_icount global variable.  Can you please take a look and, if it
works, incorporate it in your patch series?

Thanks,

Paolo

Re: [Qemu-devel] [RFC PATCH v4 11/25] cpu-exec: allow temporary disabling icount

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:paolo.bonzini@gmail.com] On Behalf Of Paolo Bonzini
> On 07/11/2014 11:32, Pavel Dovgalyuk wrote:
> >          cpu_restore_state_from_tb(cpu, tb, retaddr);
> > +        /* tb could be temporary, generated by exec nocache */
> > +        tb_phys_invalidate(tb, -1);
> 
> Would you need to do the same sequence as cpu_exec_nocache in this case?

Right.

>     cpu->current_tb = NULL;
>     tb_phys_invalidate(tb, -1);
>     tb_free(tb);
> 
> In this case, perhaps something like this would be better:
> 
> diff --git a/cpu-exec.c b/cpu-exec.c
> index 10c0f42..faf8041 100644
> --- a/cpu-exec.c
> +++ b/cpu-exec.c
> @@ -209,7 +209,7 @@ static void cpu_exec_nocache(CPUArchState *env, int max_cycles,
>          max_cycles = CF_COUNT_MASK;
> 
>      tb = tb_gen_code(cpu, orig_tb->pc, orig_tb->cs_base, orig_tb->flags,
> -                     max_cycles);
> +                     max_cycles | CF_NOCACHE);
>      cpu->current_tb = tb;
>      /* execute the generated code */
>      trace_exec_tb_nocache(tb, tb->pc);
> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
> index 9b67d16..41464f3 100644
> --- a/include/exec/exec-all.h
> +++ b/include/exec/exec-all.h
> @@ -148,6 +148,7 @@ struct TranslationBlock {
>  #define CF_COUNT_MASK  0x7fff
>  #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
>  #define CF_LAST_IO     0x8000 /* Last insn may be an IO access.  */
> +#define CF_NOCACHE     0x40000 /* To be freed after execution */
> 
>      void *tc_ptr;    /* pointer to the translated code */
>      /* next matching tb for physical address. */
> diff --git a/translate-all.c b/translate-all.c
> index 1a27df7..6ea4da4 100644
> --- a/translate-all.c
> +++ b/translate-all.c
> @@ -264,6 +264,12 @@ bool cpu_restore_state(CPUState *cpu, uintptr_t retaddr)
>      tb = tb_find_pc(retaddr);
>      if (tb) {
>          cpu_restore_state_from_tb(cpu, tb, retaddr);
> +        if (tb->cflags & CF_NOCACHE) {
> +            /* one-shot translation, invalidate it immediately */
> +            cpu->current_tb = NULL;
> +            tb_phys_invalidate(tb, -1);
> +            tb_free(tb);
> +        }
>          return true;
>      }
>      return false;
> 
> 
> I pushed it to an icount branch on my github repository, together with
> other patches that introduce CF_USE_ICOUNT instead of touching the
> use_icount global variable.  Can you please take a look and, if it
> works, incorporate it in your patch series?

Ok, I'll check it.
I noticed that you missed this patch in your branch: http://lists.nongnu.org/archive/html/qemu-devel/2014-10/msg04034.html

Pavel Dovgalyuk

[Qemu-devel] [RFC PATCH v4 12/25] replay: interrupts and exceptions

[Pavel Dovgalyuk]

This patch includes modifications of common cpu files. All interrupts and
exceptions occured during recording are written into the replay log.
These events allow correct replaying the execution by kicking cpu thread
when one of these events is found in the log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c               |   28 +++++++++++++++++++-----
 replay/replay-internal.h |    4 +++
 replay/replay.c          |   53 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   17 +++++++++++++++
 4 files changed, 96 insertions(+), 6 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index e7206ac..fa136d5 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -24,6 +24,7 @@
 #include "qemu/atomic.h"
 #include "sysemu/qtest.h"
 #include "qemu/timer.h"
+#include "replay/replay.h"
 
 /* -icount align implementation. */
 
@@ -390,10 +391,21 @@ int cpu_exec(CPUArchState *env)
                     ret = cpu->exception_index;
                     break;
 #else
-                    cc->do_interrupt(cpu);
-                    cpu->exception_index = -1;
+                    if (replay_exception()) {
+                        cc->do_interrupt(cpu);
+                        cpu->exception_index = -1;
+                    } else if (!replay_has_interrupt()) {
+                        /* give a chance to iothread in replay mode */
+                        ret = EXCP_INTERRUPT;
+                        break;
+                    }
 #endif
                 }
+            } else if (replay_has_exception()
+                       && cpu->icount_decr.u16.low + cpu->icount_extra == 0) {
+                /* try to cause an exception pending in the log */
+                cpu_exec_nocache(env, 1, tb_find_fast(env), true);
+                break;
             }
 
             next_tb = 0; /* force lookup of first TB */
@@ -409,21 +421,24 @@ int cpu_exec(CPUArchState *env)
                         cpu->exception_index = EXCP_DEBUG;
                         cpu_loop_exit(cpu);
                     }
-                    if (interrupt_request & CPU_INTERRUPT_HALT) {
+                    if ((interrupt_request & CPU_INTERRUPT_HALT)
+                        && replay_interrupt()) {
                         cpu->interrupt_request &= ~CPU_INTERRUPT_HALT;
                         cpu->halted = 1;
                         cpu->exception_index = EXCP_HLT;
                         cpu_loop_exit(cpu);
                     }
 #if defined(TARGET_I386)
-                    if (interrupt_request & CPU_INTERRUPT_INIT) {
+                    if ((interrupt_request & CPU_INTERRUPT_INIT)
+                        && replay_interrupt()) {
                         cpu_svm_check_intercept_param(env, SVM_EXIT_INIT, 0);
                         do_cpu_init(x86_cpu);
                         cpu->exception_index = EXCP_HALTED;
                         cpu_loop_exit(cpu);
                     }
 #else
-                    if (interrupt_request & CPU_INTERRUPT_RESET) {
+                    if ((interrupt_request & CPU_INTERRUPT_RESET)
+                        && replay_interrupt()) {
                         cpu_reset(cpu);
                     }
 #endif
@@ -431,7 +446,8 @@ int cpu_exec(CPUArchState *env)
                        False when the interrupt isn't processed,
                        True when it is, and we should restart on a new TB,
                        and via longjmp via cpu_loop_exit.  */
-                    if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
+                    if (replay_interrupt()
+                        && cc->cpu_exec_interrupt(cpu, interrupt_request)) {
                         next_tb = 0;
                     }
                     /* Don't use the cached interrupt_request value,
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 577d1e9..3588385 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -14,6 +14,10 @@
 
 #include <stdio.h>
 
+/* for software interrupt */
+#define EVENT_INTERRUPT             15
+/* for emulated exceptions */
+#define EVENT_EXCEPTION             23
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 
diff --git a/replay/replay.c b/replay/replay.c
index c305e0c..c275794 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -68,3 +68,56 @@ uint64_t replay_get_current_step(void)
 {
     return cpu_get_instructions_counter();
 }
+
+bool replay_exception(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_put_event(EVENT_EXCEPTION);
+        return true;
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        if (skip_async_events(EVENT_EXCEPTION)) {
+            replay_has_unread_data = 0;
+            return true;
+        }
+        return false;
+    }
+
+    return true;
+}
+
+bool replay_has_exception(void)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        if (skip_async_events(EVENT_EXCEPTION)) {
+            return true;
+        }
+    }
+
+    return false;
+}
+
+bool replay_interrupt(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_save_instructions();
+        replay_put_event(EVENT_INTERRUPT);
+        return true;
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        if (skip_async_events(EVENT_INTERRUPT)) {
+            replay_has_unread_data = 0;
+            return true;
+        }
+        return false;
+    }
+
+    return true;
+}
+
+bool replay_has_interrupt(void)
+{
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        return skip_async_events(EVENT_INTERRUPT);
+    }
+    return false;
+}
diff --git a/replay/replay.h b/replay/replay.h
index e40daf5..d7abaee 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -27,4 +27,21 @@ ReplaySubmode replay_get_play_submode(void);
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
 
+/* Interrupts and exceptions */
+
+/*! Called by exception handler to write or read
+    exception processing events. */
+bool replay_exception(void);
+/*! Used to determine that exception is pending.
+    Does not proceed to the next event in the log. */
+bool replay_has_exception(void);
+/*! Called by interrupt handlers to write or read
+    interrupt processing events.
+    \return true if interrupt should be processed */
+bool replay_interrupt(void);
+/*! Tries to read interrupt event from the file.
+    Returns true, when interrupt request is pending */
+bool replay_has_interrupt(void);
+
+
 #endif

[Qemu-devel] [RFC PATCH v4 13/25] replay: asynchronous events infrastructure

[Pavel Dovgalyuk]

This patch adds module for saving and replaying asynchronous events.
These events include network packets, keyboard and mouse input,
USB packets, thread pool and bottom halves callbacks.
All events are stored in the queue to be processed at synchronization points
such as beginning of TB execution, or checkpoint in the iothread.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |  217 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   27 ++++++
 replay/replay.h          |    4 +
 4 files changed, 249 insertions(+), 0 deletions(-)
 create mode 100755 replay/replay-events.c

diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 1148f45..7b0c8ed 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1,2 +1,3 @@
 obj-y += replay.o
 obj-y += replay-internal.o
+obj-y += replay-events.o
\ No newline at end of file
diff --git a/replay/replay-events.c b/replay/replay-events.c
new file mode 100755
index 0000000..f3c9b16
--- /dev/null
+++ b/replay/replay-events.c
@@ -0,0 +1,217 @@
+/*
+ * replay-events.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+typedef struct Event {
+    int event_kind;
+    void *opaque;
+    void *opaque2;
+    uint64_t id;
+
+    QTAILQ_ENTRY(Event) events;
+} Event;
+
+static QTAILQ_HEAD(, Event) events_list = QTAILQ_HEAD_INITIALIZER(events_list);
+
+static QemuMutex lock;
+static unsigned int read_event_kind = -1;
+static uint64_t read_id = -1;
+static int read_opt = -1;
+
+static bool replay_events_enabled = false;
+
+/* Functions */
+
+static void replay_run_event(Event *event)
+{
+    switch (event->event_kind) {
+    default:
+        fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
+                event->event_kind);
+        exit(1);
+        break;
+    }
+}
+
+void replay_enable_events(void)
+{
+    replay_events_enabled = true;
+}
+
+bool replay_has_events(void)
+{
+    return !QTAILQ_EMPTY(&events_list);
+}
+
+void replay_flush_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        replay_run_event(event);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_disable_events(void)
+{
+    replay_events_enabled = false;
+    /* Flush events queue before waiting of completion */
+    replay_flush_events();
+}
+
+void replay_clear_events(void)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        QTAILQ_REMOVE(&events_list, event, events);
+
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+static void replay_add_event_internal(int event_kind, void *opaque,
+                                      void *opaque2, uint64_t id)
+{
+    if (event_kind >= REPLAY_ASYNC_COUNT) {
+        fprintf(stderr, "Replay: invalid async event ID (%d)\n", event_kind);
+        exit(1);
+    }
+    if (!replay_file || replay_mode == REPLAY_MODE_NONE
+        || !replay_events_enabled) {
+        Event e;
+        e.event_kind = event_kind;
+        e.opaque = opaque;
+        e.opaque2 = opaque2;
+        e.id = id;
+        replay_run_event(&e);
+        return;
+    }
+
+    Event *event = g_malloc0(sizeof(Event));
+    event->event_kind = event_kind;
+    event->opaque = opaque;
+    event->opaque2 = opaque2;
+    event->id = id;
+
+    qemu_mutex_lock(&lock);
+    QTAILQ_INSERT_TAIL(&events_list, event, events);
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_add_event(int event_kind, void *opaque)
+{
+    replay_add_event_internal(event_kind, opaque, NULL, 0);
+}
+
+void replay_save_events(int opt)
+{
+    qemu_mutex_lock(&lock);
+    while (!QTAILQ_EMPTY(&events_list)) {
+        Event *event = QTAILQ_FIRST(&events_list);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            /* put the event into the file */
+            replay_put_event(EVENT_ASYNC_OPT);
+            replay_put_byte(opt);
+            replay_put_byte(event->event_kind);
+
+            /* save event-specific data */
+            switch (event->event_kind) {
+            }
+        }
+
+        replay_run_event(event);
+        QTAILQ_REMOVE(&events_list, event, events);
+        g_free(event);
+    }
+    qemu_mutex_unlock(&lock);
+}
+
+void replay_read_events(int opt)
+{
+    replay_fetch_data_kind();
+    while (replay_data_kind == EVENT_ASYNC_OPT) {
+        if (read_event_kind == -1) {
+            read_opt = replay_get_byte();
+            read_event_kind = replay_get_byte();
+            read_id = -1;
+            replay_check_error();
+        }
+
+        if (opt != read_opt) {
+            break;
+        }
+        /* Execute some events without searching them in the queue */
+        switch (read_event_kind) {
+        default:
+            fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
+            exit(1);
+            break;
+        }
+
+        qemu_mutex_lock(&lock);
+
+        Event *event = NULL;
+        Event *curr = NULL;
+        QTAILQ_FOREACH(curr, &events_list, events) {
+            if (curr->event_kind == read_event_kind
+                && (read_id == -1 || read_id == curr->id)) {
+                event = curr;
+                break;
+            }
+        }
+
+        if (event) {
+            /* read event-specific reading data */
+
+            QTAILQ_REMOVE(&events_list, event, events);
+
+            qemu_mutex_unlock(&lock);
+
+            /* reset unread data and other parameters to allow
+               reading other data from the log while
+               running the event */
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_id = -1;
+            read_opt = -1;
+
+            replay_run_event(event);
+            g_free(event);
+
+            replay_fetch_data_kind();
+        } else {
+            qemu_mutex_unlock(&lock);
+            /* No such event found in the queue */
+            break;
+        }
+    }
+}
+
+void replay_init_events(void)
+{
+    read_event_kind = -1;
+    qemu_mutex_init(&lock);
+}
+
+void replay_finish_events(void)
+{
+    replay_events_enabled = false;
+    replay_clear_events();
+    qemu_mutex_destroy(&lock);
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 3588385..b3139fa 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -18,9 +18,15 @@
 #define EVENT_INTERRUPT             15
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
+/* for async events */
+#define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
 
+/* Asynchronous events IDs */
+
+#define REPLAY_ASYNC_COUNT             0
+
 typedef struct ReplayState {
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
@@ -69,4 +75,25 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/* Asynchronous events queue */
+
+/*! Initializes events' processing internals */
+void replay_init_events(void);
+/*! Clears internal data structures for events handling */
+void replay_finish_events(void);
+/*! Enables storing events in the queue */
+void replay_enable_events(void);
+/*! Flushes events queue */
+void replay_flush_events(void);
+/*! Clears events list before loading new VM state */
+void replay_clear_events(void);
+/*! Returns true if there are any unsaved events in the queue */
+bool replay_has_events(void);
+/*! Saves events from queue into the file */
+void replay_save_events(int opt);
+/*! Read events from the file into the input queue */
+void replay_read_events(int opt);
+/*! Adds specified async event to the queue */
+void replay_add_event(int event_id, void *opaque);
+
 #endif
diff --git a/replay/replay.h b/replay/replay.h
index d7abaee..0cfd71a 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -43,5 +43,9 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Asynchronous events queue */
+
+/*! Disables storing events in the queue */
+void replay_disable_events(void);
 
 #endif

Re: [Qemu-devel] [RFC PATCH v4 13/25] replay: asynchronous events infrastructure

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]

On 11/07/2014 11:32 AM, Pavel Dovgalyuk wrote:
> This patch adds module for saving and replaying asynchronous events.
> These events include network packets, keyboard and mouse input,
> USB packets, thread pool and bottom halves callbacks.
> All events are stored in the queue to be processed at synchronization points
> such as beginning of TB execution, or checkpoint in the iothread.
> 
> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  replay/Makefile.objs     |    1 
>  replay/replay-events.c   |  217 ++++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-internal.h |   27 ++++++
>  replay/replay.h          |    4 +
>  4 files changed, 249 insertions(+), 0 deletions(-)
>  create mode 100755 replay/replay-events.c
> 
> diff --git a/replay/Makefile.objs b/replay/Makefile.objs
> index 1148f45..7b0c8ed 100755
> --- a/replay/Makefile.objs
> +++ b/replay/Makefile.objs
> @@ -1,2 +1,3 @@
>  obj-y += replay.o
>  obj-y += replay-internal.o
> +obj-y += replay-events.o
> \ No newline at end of file

Please ensure that text files always end in newline.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 539 bytes --]

[Qemu-devel] [RFC PATCH v4 14/25] cpu: replay instructions sequence

[Pavel Dovgalyuk]

This patch adds calls to replay functions into the icount setup block.
In record mode number of executed instructions is written to the log.
In replay mode number of istructions to execute is taken from the replay log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpu-exec.c      |    1 +
 cpus.c          |   28 ++++++++++++++++++----------
 replay/replay.c |   18 ++++++++++++++++++
 replay/replay.h |    4 ++++
 4 files changed, 41 insertions(+), 10 deletions(-)

diff --git a/cpu-exec.c b/cpu-exec.c
index fa136d5..06ea5cb 100644
--- a/cpu-exec.c
+++ b/cpu-exec.c
@@ -537,6 +537,7 @@ int cpu_exec(CPUArchState *env)
                             }
                             cpu->exception_index = EXCP_INTERRUPT;
                             next_tb = 0;
+                            qemu_notify_event();
                             cpu_loop_exit(cpu);
                         }
                         break;
diff --git a/cpus.c b/cpus.c
index 15ac3a1..b41baf6 100644
--- a/cpus.c
+++ b/cpus.c
@@ -41,6 +41,7 @@
 #include "qemu/seqlock.h"
 #include "qapi-event.h"
 #include "hw/nmi.h"
+#include "replay/replay.h"
 
 #ifndef _WIN32
 #include "qemu/compatfd.h"
@@ -1358,18 +1359,22 @@ static int tcg_cpu_exec(CPUArchState *env)
                                     + cpu->icount_extra);
         cpu->icount_decr.u16.low = 0;
         cpu->icount_extra = 0;
-        deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
+        if (replay_mode != REPLAY_MODE_PLAY) {
+            deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
 
-        /* Maintain prior (possibly buggy) behaviour where if no deadline
-         * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
-         * INT32_MAX nanoseconds ahead, we still use INT32_MAX
-         * nanoseconds.
-         */
-        if ((deadline < 0) || (deadline > INT32_MAX)) {
-            deadline = INT32_MAX;
-        }
+            /* Maintain prior (possibly buggy) behaviour where if no deadline
+             * was set (as there is no QEMU_CLOCK_VIRTUAL timer) or it is more than
+             * INT32_MAX nanoseconds ahead, we still use INT32_MAX
+             * nanoseconds.
+             */
+            if ((deadline < 0) || (deadline > INT32_MAX)) {
+                deadline = INT32_MAX;
+            }
 
-        count = qemu_icount_round(deadline);
+            count = qemu_icount_round(deadline);
+        } else {
+            count = replay_get_instructions();
+        }
         timers_state.qemu_icount += count;
         decr = (count > 0xffff) ? 0xffff : count;
         count -= decr;
@@ -1387,6 +1392,9 @@ static int tcg_cpu_exec(CPUArchState *env)
                         + cpu->icount_extra);
         cpu->icount_decr.u32 = 0;
         cpu->icount_extra = 0;
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            replay_exec_instructions();
+        }
     }
     return ret;
 }
diff --git a/replay/replay.c b/replay/replay.c
index c275794..a6de6a1 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -69,6 +69,24 @@ uint64_t replay_get_current_step(void)
     return cpu_get_instructions_counter();
 }
 
+int replay_get_instructions(void)
+{
+    if (skip_async_events(EVENT_INSTRUCTION)) {
+        return replay_state.instructions_count;
+    }
+    return 0;
+}
+
+void replay_exec_instructions(void)
+{
+    int count = (int)(replay_get_current_step() - replay_state.current_step);
+    replay_state.instructions_count -= count;
+    replay_state.current_step += count;
+    if (replay_state.instructions_count == 0 && count != 0) {
+        replay_has_unread_data = 0;
+    }
+}
+
 bool replay_exception(void)
 {
     if (replay_mode == REPLAY_MODE_RECORD) {
diff --git a/replay/replay.h b/replay/replay.h
index 0cfd71a..90a949b 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -26,6 +26,10 @@ ReplaySubmode replay_get_play_submode(void);
 
 /*! Returns number of executed instructions. */
 uint64_t replay_get_current_step(void);
+/*! Returns number of instructions to execute in replay mode. */
+int replay_get_instructions(void);
+/*! Updates instructions counter in replay mode. */
+void replay_exec_instructions(void);
 
 /* Interrupts and exceptions */
 

[Qemu-devel] [RFC PATCH v4 15/25] replay: recording and replaying clock ticks

[Pavel Dovgalyuk]

Clock ticks are considered as the sources of non-deterministic data for
virtual machine. This patch implements saving the clock values when they
are acquired (virtual, host clock, rdtsc, and some other timers).
When replaying the execution corresponding values are read from log and
transfered to the module, which wants to read the values.
Such a design required the clock polling to be synchronized. Sometimes
it is not true - e.g. when timeouts for timer lists are checked. In this case
we use a cached value of the clock, passing it to the client code.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/qemu/timer.h     |   20 ++++++++++--
 qemu-timer.c             |    3 +-
 replay/Makefile.objs     |    3 +-
 replay/replay-internal.h |   11 ++++++
 replay/replay-time.c     |   79 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   21 ++++++++++++
 stubs/replay.c           |    9 +++++
 7 files changed, 142 insertions(+), 4 deletions(-)
 create mode 100755 replay/replay-time.c

diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 38a02c5..7b43331 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -4,6 +4,7 @@
 #include "qemu/typedefs.h"
 #include "qemu-common.h"
 #include "qemu/notify.h"
+#include "replay/replay.h"
 
 /* timers */
 
@@ -699,8 +700,8 @@ static inline int64_t get_ticks_per_sec(void)
  * Low level clock functions
  */
 
-/* real time host monotonic timer */
-static inline int64_t get_clock_realtime(void)
+/* real time host monotonic timer implementation */
+static inline int64_t get_clock_realtime_impl(void)
 {
     struct timeval tv;
 
@@ -708,6 +709,12 @@ static inline int64_t get_clock_realtime(void)
     return tv.tv_sec * 1000000000LL + (tv.tv_usec * 1000);
 }
 
+/* real time host monotonic timer interface */
+static inline int64_t get_clock_realtime(void)
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_HOST, get_clock_realtime_impl());
+}
+
 /* Warning: don't insert tracepoints into these functions, they are
    also used by simpletrace backend and tracepoints would cause
    an infinite recursion! */
@@ -752,6 +759,8 @@ int64_t cpu_icount_to_ns(int64_t icount);
 /*******************************************/
 /* host CPU ticks (if available) */
 
+#define cpu_get_real_ticks cpu_get_real_ticks_impl
+
 #if defined(_ARCH_PPC)
 
 static inline int64_t cpu_get_real_ticks(void)
@@ -905,6 +914,13 @@ static inline int64_t cpu_get_real_ticks (void)
 }
 #endif
 
+#undef cpu_get_real_ticks
+
+static inline int64_t cpu_get_real_ticks(void)
+{
+    return REPLAY_CLOCK(REPLAY_CLOCK_REAL_TICKS, cpu_get_real_ticks_impl());
+}
+
 #ifdef CONFIG_PROFILER
 static inline int64_t profile_getclock(void)
 {
diff --git a/qemu-timer.c b/qemu-timer.c
index 00a5d35..8307913 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -25,6 +25,7 @@
 #include "sysemu/sysemu.h"
 #include "monitor/monitor.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 #include "hw/hw.h"
 
@@ -562,7 +563,7 @@ int64_t qemu_clock_get_ns(QEMUClockType type)
         now = get_clock_realtime();
         last = clock->last;
         clock->last = now;
-        if (now < last) {
+        if (now < last && replay_mode == REPLAY_MODE_NONE) {
             notifier_list_notify(&clock->reset_notifiers, &now);
         }
         return now;
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 7b0c8ed..257c320 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -1,3 +1,4 @@
 obj-y += replay.o
 obj-y += replay-internal.o
-obj-y += replay-events.o
\ No newline at end of file
+obj-y += replay-events.o
+obj-y += replay-time.o
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index b3139fa..d2b2bdf 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -22,12 +22,17 @@
 #define EVENT_ASYNC_OPT             25
 /* for instruction event */
 #define EVENT_INSTRUCTION           32
+/* for clock read/writes */
+#define EVENT_CLOCK                 64
+/* some of grteater codes are reserved for clocks */
 
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
 
 typedef struct ReplayState {
+    /*! Cached clock values. */
+    int64_t cached_clock[REPLAY_CLOCK_COUNT];
     /*! Current step - number of processed instructions and timer events. */
     uint64_t current_step;
     /*! Number of instructions to be executed before other events happen. */
@@ -75,6 +80,12 @@ bool skip_async_events(int stop_event);
     reports an error and stops the execution. */
 void skip_async_events_until(unsigned int kind);
 
+/*! Reads next clock value from the file.
+    If clock kind read from the file is different from the parameter,
+    the value is not used.
+    If the parameter is -1, the clock value is read to the cache anyway. */
+void replay_read_next_clock(unsigned int kind);
+
 /* Asynchronous events queue */
 
 /*! Initializes events' processing internals */
diff --git a/replay/replay-time.c b/replay/replay-time.c
new file mode 100755
index 0000000..3f94f4e
--- /dev/null
+++ b/replay/replay-time.c
@@ -0,0 +1,79 @@
+/*
+ * replay-time.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+    replay_save_instructions();
+
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    if (replay_file) {
+        replay_put_event(EVENT_CLOCK + kind);
+        replay_put_qword(clock);
+    }
+
+    return clock;
+}
+
+void replay_read_next_clock(unsigned int kind)
+{
+    replay_fetch_data_kind();
+    if (replay_file) {
+        unsigned int read_kind = replay_data_kind - EVENT_CLOCK;
+
+        if (kind != -1 && read_kind != kind) {
+            return;
+        }
+        if (read_kind >= REPLAY_CLOCK_COUNT) {
+            fprintf(stderr,
+                    "invalid clock ID %d was read from replay\n", read_kind);
+            exit(1);
+        }
+
+        int64_t clock = replay_get_qword();
+
+        replay_check_error();
+        replay_has_unread_data = 0;
+
+        replay_state.cached_clock[read_kind] = clock;
+    }
+}
+
+/*! Reads next clock event from the input. */
+int64_t replay_read_clock(unsigned int kind)
+{
+    if (kind >= REPLAY_CLOCK_COUNT) {
+        fprintf(stderr, "invalid clock ID %d for replay\n", kind);
+        exit(1);
+    }
+
+    replay_exec_instructions();
+
+    if (replay_file) {
+        if (skip_async_events(EVENT_CLOCK + kind)) {
+            replay_read_next_clock(kind);
+        }
+        int64_t ret = replay_state.cached_clock[kind];
+
+        return ret;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
diff --git a/replay/replay.h b/replay/replay.h
index 90a949b..cba3b18 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -16,6 +16,14 @@
 #include <stdint.h>
 #include "qapi-types.h"
 
+/* replay clock kinds */
+/* rdtsc */
+#define REPLAY_CLOCK_REAL_TICKS 0
+/* host_clock */
+#define REPLAY_CLOCK_HOST       1
+
+#define REPLAY_CLOCK_COUNT      2
+
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
 
@@ -47,6 +55,19 @@ bool replay_interrupt(void);
     Returns true, when interrupt request is pending */
 bool replay_has_interrupt(void);
 
+/* Processing clocks and other time sources */
+
+/*! Save the specified clock */
+int64_t replay_save_clock(unsigned int kind, int64_t clock);
+/*! Read the specified clock from the log or return cached data */
+int64_t replay_read_clock(unsigned int kind);
+/*! Saves or reads the clock depending on the current replay mode. */
+#define REPLAY_CLOCK(clock, value)                                      \
+    (replay_mode == REPLAY_MODE_PLAY ? replay_read_clock((clock))       \
+        : replay_mode == REPLAY_MODE_RECORD                             \
+            ? replay_save_clock((clock), (value))                       \
+        : (value))
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
diff --git a/stubs/replay.c b/stubs/replay.c
index b146d55..f3f60fe 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -6,3 +6,12 @@ ReplaySubmode replay_get_play_submode(void)
 {
     return 0;
 }
+
+int64_t replay_save_clock(unsigned int kind, int64_t clock)
+{
+}
+
+int64_t replay_read_clock(unsigned int kind)
+{
+    return 0;
+}

[Qemu-devel] [RFC PATCH v4 16/25] replay: recording and replaying different timers

[Pavel Dovgalyuk]

This patch introduces functions for recording and replaying realtime sources,
that do not use qemu-clock interface. These include return value of time()
function in time_t and struct tm forms. Patch also adds warning to
get_timedate function to prevent its usage in recording mode, because it may
lead to non-determinism.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 hw/timer/mc146818rtc.c   |   10 ++++
 hw/timer/pl031.c         |   10 ++++
 include/qemu-common.h    |    1 
 replay/replay-internal.h |    4 ++
 replay/replay-time.c     |  112 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |    8 +++
 vl.c                     |   17 ++++++-
 7 files changed, 157 insertions(+), 5 deletions(-)

diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
index 2c4b650..7ae5898 100644
--- a/hw/timer/mc146818rtc.c
+++ b/hw/timer/mc146818rtc.c
@@ -28,6 +28,7 @@
 #include "qapi/visitor.h"
 #include "qapi-event.h"
 #include "qmp-commands.h"
+#include "replay/replay.h"
 
 #ifdef TARGET_I386
 #include "hw/i386/apic.h"
@@ -703,7 +704,14 @@ static void rtc_set_date_from_host(ISADevice *dev)
     RTCState *s = MC146818_RTC(dev);
     struct tm tm;
 
-    qemu_get_timedate(&tm, 0);
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }
 
     s->base_rtc = mktimegm(&tm);
     s->last_update = qemu_clock_get_ns(rtc_clock);
diff --git a/hw/timer/pl031.c b/hw/timer/pl031.c
index 34d9b44..19264f2 100644
--- a/hw/timer/pl031.c
+++ b/hw/timer/pl031.c
@@ -14,6 +14,7 @@
 #include "hw/sysbus.h"
 #include "qemu/timer.h"
 #include "sysemu/sysemu.h"
+#include "replay/replay.h"
 
 //#define DEBUG_PL031
 
@@ -200,7 +201,14 @@ static int pl031_init(SysBusDevice *dev)
     sysbus_init_mmio(dev, &s->iomem);
 
     sysbus_init_irq(dev, &s->irq);
-    qemu_get_timedate(&tm, 0);
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        qemu_get_timedate_no_warning(&tm, 0);
+        replay_save_tm(&tm);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        replay_read_tm(&tm);
+    } else {
+        qemu_get_timedate_no_warning(&tm, 0);
+    }
     s->tick_offset = mktimegm(&tm) -
         qemu_clock_get_ns(rtc_clock) / get_ticks_per_sec();
 
diff --git a/include/qemu-common.h b/include/qemu-common.h
index b87e9c2..d18f033 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -129,6 +129,7 @@ void dump_drift_info(FILE *f, fprintf_function cpu_fprintf);
 int qemu_main(int argc, char **argv, char **envp);
 #endif
 
+void qemu_get_timedate_no_warning(struct tm *tm, int offset);
 void qemu_get_timedate(struct tm *tm, int offset);
 int qemu_timedate_diff(struct tm *tm);
 
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index d2b2bdf..367d345 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -14,6 +14,10 @@
 
 #include <stdio.h>
 
+/* for time_t event */
+#define EVENT_TIME_T                1
+/* for tm event */
+#define EVENT_TM                    2
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
 /* for emulated exceptions */
diff --git a/replay/replay-time.c b/replay/replay-time.c
index 3f94f4e..5d944d3 100755
--- a/replay/replay-time.c
+++ b/replay/replay-time.c
@@ -77,3 +77,115 @@ int64_t replay_read_clock(unsigned int kind)
     fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
     exit(1);
 }
+
+/*! Saves time_t value to the log */
+static void replay_save_time_t(time_t tm)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        replay_put_event(EVENT_TIME_T);
+        if (sizeof(tm) == 4) {
+            replay_put_dword(tm);
+        } else if (sizeof(tm) == 8) {
+            replay_put_qword(tm);
+        } else {
+            fprintf(stderr, "invalid time_t sizeof: %u\n",
+                    (unsigned)sizeof(tm));
+            exit(1);
+        }
+    }
+}
+
+/*! Reads time_t value from the log. Stops execution in case of error */
+static time_t replay_read_time_t(void)
+{
+    replay_exec_instructions();
+
+    if (replay_file) {
+        time_t tm;
+
+        skip_async_events_until(EVENT_TIME_T);
+
+        if (sizeof(tm) == 4) {
+            tm = replay_get_dword();
+        } else if (sizeof(tm) == 8) {
+            tm = replay_get_qword();
+        } else {
+            fprintf(stderr, "invalid time_t sizeof: %u\n",
+                    (unsigned)sizeof(tm));
+            exit(1);
+        }
+
+        replay_check_error();
+
+        replay_has_unread_data = 0;
+
+        return tm;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
+
+void replay_save_tm(struct tm *tm)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        replay_put_event(EVENT_TM);
+
+        replay_put_dword(tm->tm_sec);
+        replay_put_dword(tm->tm_min);
+        replay_put_dword(tm->tm_hour);
+        replay_put_dword(tm->tm_mday);
+        replay_put_dword(tm->tm_mon);
+        replay_put_dword(tm->tm_year);
+        replay_put_dword(tm->tm_wday);
+        replay_put_dword(tm->tm_yday);
+        replay_put_dword(tm->tm_isdst);
+    }
+}
+
+void replay_read_tm(struct tm *tm)
+{
+    replay_exec_instructions();
+
+    if (replay_file) {
+        skip_async_events_until(EVENT_TM);
+
+        tm->tm_sec = replay_get_dword();
+        tm->tm_min = replay_get_dword();
+        tm->tm_hour = replay_get_dword();
+        tm->tm_mday = replay_get_dword();
+        tm->tm_mon = replay_get_dword();
+        tm->tm_year = replay_get_dword();
+        tm->tm_wday = replay_get_dword();
+        tm->tm_yday = replay_get_dword();
+        tm->tm_isdst = replay_get_dword();
+
+        replay_check_error();
+        replay_has_unread_data = 0;
+
+        return;
+    }
+
+    fprintf(stderr, "REPLAY INTERNAL ERROR %d\n", __LINE__);
+    exit(1);
+}
+
+time_t replay_time(void)
+{
+    time_t systime;
+
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        systime = time(NULL);
+        replay_save_time_t(systime);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        systime = replay_read_time_t();
+    } else {
+        systime = time(NULL);
+    }
+
+    return systime;
+}
diff --git a/replay/replay.h b/replay/replay.h
index cba3b18..143fe85 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -14,6 +14,7 @@
 
 #include <stdbool.h>
 #include <stdint.h>
+#include <time.h>
 #include "qapi-types.h"
 
 /* replay clock kinds */
@@ -67,6 +68,13 @@ int64_t replay_read_clock(unsigned int kind);
         : replay_mode == REPLAY_MODE_RECORD                             \
             ? replay_save_clock((clock), (value))                       \
         : (value))
+/*! Returns result of time() function execution in normal and record modes.
+    In play mode returns value read from the log. */
+time_t replay_time(void);
+/*! Saves struct tm value to the log */
+void replay_save_tm(struct tm *tm);
+/*! Reads struct tm value from the log. Stops execution in case of error */
+void replay_read_tm(struct tm *tm);
 
 /* Asynchronous events queue */
 
diff --git a/vl.c b/vl.c
index f6b3546..37c6616 100644
--- a/vl.c
+++ b/vl.c
@@ -118,6 +118,7 @@ int main(int argc, char **argv)
 #include "qapi/opts-visitor.h"
 #include "qom/object_interfaces.h"
 #include "qapi-event.h"
+#include "replay/replay.h"
 
 #define DEFAULT_RAM_SIZE 128
 
@@ -767,7 +768,7 @@ void vm_start(void)
 
 /***********************************************************/
 /* host time/date access */
-void qemu_get_timedate(struct tm *tm, int offset)
+void qemu_get_timedate_no_warning(struct tm *tm, int offset)
 {
     time_t ti;
 
@@ -784,6 +785,16 @@ void qemu_get_timedate(struct tm *tm, int offset)
     }
 }
 
+/* host time/date access with replay warning */
+void qemu_get_timedate(struct tm *tm, int offset)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fprintf(stderr, "REPLAY WARNING! qemu_get_timedate "
+                        "function may lead to non-determinism\n");
+    }
+    qemu_get_timedate_no_warning(tm, offset);
+}
+
 int qemu_timedate_diff(struct tm *tm)
 {
     time_t seconds;
@@ -799,7 +810,7 @@ int qemu_timedate_diff(struct tm *tm)
     else
         seconds = mktimegm(tm) + rtc_date_offset;
 
-    return seconds - time(NULL);
+    return seconds - replay_time();
 }
 
 static void configure_rtc_date_offset(const char *startdate, int legacy)
@@ -837,7 +848,7 @@ static void configure_rtc_date_offset(const char *startdate, int legacy)
                             "'2006-06-17T16:01:21' or '2006-06-17'\n");
             exit(1);
         }
-        rtc_date_offset = time(NULL) - rtc_start_date;
+        rtc_date_offset = replay_time() - rtc_start_date;
     }
 }
 

[Qemu-devel] [RFC PATCH v4 17/25] cpus: make icount warp deterministic in replay mode

[Pavel Dovgalyuk]

This patch adds saving and replaying warping parameters in record and replay
modes. These parameters affect on virtual clock values and therefore should
be deterministic.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c |   24 +++++++++++++++++-------
 1 files changed, 17 insertions(+), 7 deletions(-)

diff --git a/cpus.c b/cpus.c
index b41baf6..dac222b 100644
--- a/cpus.c
+++ b/cpus.c
@@ -370,7 +370,9 @@ static void icount_warp_rt(void *opaque)
 
     seqlock_write_lock(&timers_state.vm_clock_seqlock);
     if (runstate_is_running()) {
-        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
+        int64_t clock = qemu_clock_get_ns(replay_mode == REPLAY_MODE_NONE
+                                          ? QEMU_CLOCK_REALTIME
+                                          : QEMU_CLOCK_HOST);
         int64_t warp_delta;
 
         warp_delta = clock - vm_clock_warp_start;
@@ -444,7 +446,9 @@ void qemu_clock_warp(QEMUClockType type)
     }
 
     /* We want to use the earliest deadline from ALL vm_clocks */
-    clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
+    clock = qemu_clock_get_ns(replay_mode == REPLAY_MODE_NONE
+                              ? QEMU_CLOCK_REALTIME
+                              : QEMU_CLOCK_HOST);
     deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
     if (deadline < 0) {
         return;
@@ -537,8 +541,10 @@ void configure_icount(QemuOpts *opts, Error **errp)
         return;
     }
     icount_align_option = qemu_opt_get_bool(opts, "align", false);
-    icount_warp_timer = timer_new_ns(QEMU_CLOCK_REALTIME,
-                                          icount_warp_rt, NULL);
+    icount_warp_timer = timer_new_ns(replay_mode == REPLAY_MODE_NONE
+                                     ? QEMU_CLOCK_REALTIME
+                                     : QEMU_CLOCK_HOST,
+                                     icount_warp_rt, NULL);
     if (strcmp(option, "auto") != 0) {
         errno = 0;
         icount_time_shift = strtol(option, &rem_str, 0);
@@ -562,10 +568,14 @@ void configure_icount(QemuOpts *opts, Error **errp)
        the virtual time trigger catches emulated time passing too fast.
        Realtime triggers occur even when idle, so use them less frequently
        than VM triggers.  */
-    icount_rt_timer = timer_new_ms(QEMU_CLOCK_REALTIME,
-                                        icount_adjust_rt, NULL);
+    icount_rt_timer = timer_new_ms(replay_mode == REPLAY_MODE_NONE
+                                   ? QEMU_CLOCK_REALTIME
+                                   : QEMU_CLOCK_HOST,
+                                   icount_adjust_rt, NULL);
     timer_mod(icount_rt_timer,
-                   qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1000);
+                   qemu_clock_get_ms(replay_mode == REPLAY_MODE_NONE
+                                     ? QEMU_CLOCK_REALTIME
+                                     : QEMU_CLOCK_HOST) + 1000);
     icount_vm_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
                                         icount_adjust_vm, NULL);
     timer_mod(icount_vm_timer,

Re: [Qemu-devel] [RFC PATCH v4 17/25] cpus: make icount warp deterministic in replay mode

[Paolo Bonzini]

On 07/11/2014 11:33, Pavel Dovgalyuk wrote:
> This patch adds saving and replaying warping parameters in record and replay
> modes. These parameters affect on virtual clock values and therefore should
> be deterministic.

Why are QEMU_CLOCK_REALTIME timers not recorded/replayed like
QEMU_CLOCK_HOST is?

Paolo

> Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> ---
>  cpus.c |   24 +++++++++++++++++-------
>  1 files changed, 17 insertions(+), 7 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index b41baf6..dac222b 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -370,7 +370,9 @@ static void icount_warp_rt(void *opaque)
>  
>      seqlock_write_lock(&timers_state.vm_clock_seqlock);
>      if (runstate_is_running()) {
> -        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
> +        int64_t clock = qemu_clock_get_ns(replay_mode == REPLAY_MODE_NONE
> +                                          ? QEMU_CLOCK_REALTIME
> +                                          : QEMU_CLOCK_HOST);
>          int64_t warp_delta;
>  
>          warp_delta = clock - vm_clock_warp_start;
> @@ -444,7 +446,9 @@ void qemu_clock_warp(QEMUClockType type)
>      }
>  
>      /* We want to use the earliest deadline from ALL vm_clocks */
> -    clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
> +    clock = qemu_clock_get_ns(replay_mode == REPLAY_MODE_NONE
> +                              ? QEMU_CLOCK_REALTIME
> +                              : QEMU_CLOCK_HOST);
>      deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
>      if (deadline < 0) {
>          return;
> @@ -537,8 +541,10 @@ void configure_icount(QemuOpts *opts, Error **errp)
>          return;
>      }
>      icount_align_option = qemu_opt_get_bool(opts, "align", false);
> -    icount_warp_timer = timer_new_ns(QEMU_CLOCK_REALTIME,
> -                                          icount_warp_rt, NULL);
> +    icount_warp_timer = timer_new_ns(replay_mode == REPLAY_MODE_NONE
> +                                     ? QEMU_CLOCK_REALTIME
> +                                     : QEMU_CLOCK_HOST,
> +                                     icount_warp_rt, NULL);
>      if (strcmp(option, "auto") != 0) {
>          errno = 0;
>          icount_time_shift = strtol(option, &rem_str, 0);
> @@ -562,10 +568,14 @@ void configure_icount(QemuOpts *opts, Error **errp)
>         the virtual time trigger catches emulated time passing too fast.
>         Realtime triggers occur even when idle, so use them less frequently
>         than VM triggers.  */
> -    icount_rt_timer = timer_new_ms(QEMU_CLOCK_REALTIME,
> -                                        icount_adjust_rt, NULL);
> +    icount_rt_timer = timer_new_ms(replay_mode == REPLAY_MODE_NONE
> +                                   ? QEMU_CLOCK_REALTIME
> +                                   : QEMU_CLOCK_HOST,
> +                                   icount_adjust_rt, NULL);
>      timer_mod(icount_rt_timer,
> -                   qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1000);
> +                   qemu_clock_get_ms(replay_mode == REPLAY_MODE_NONE
> +                                     ? QEMU_CLOCK_REALTIME
> +                                     : QEMU_CLOCK_HOST) + 1000);
>      icount_vm_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
>                                          icount_adjust_vm, NULL);
>      timer_mod(icount_vm_timer,
> 

Re: [Qemu-devel] [RFC PATCH v4 17/25] cpus: make icount warp deterministic in replay mode

[Pavel Dovgaluk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 07/11/2014 11:33, Pavel Dovgalyuk wrote:
> > This patch adds saving and replaying warping parameters in record and replay
> > modes. These parameters affect on virtual clock values and therefore should
> > be deterministic.
> 
> Why are QEMU_CLOCK_REALTIME timers not recorded/replayed like
> QEMU_CLOCK_HOST is?

We cannot record all QEMU_CLOCK_REALTIME timers, because they are simulator-related timers,
not the VM-related ones. If we'll record them then we'll have to replay simulator
execution. And we want just to replay VM allowing different realtime events to be
non-replayable (like screen refresh, monitor events handling, and so on).

Pavel Dovgalyuk
> 
> > Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
> > ---
> >  cpus.c |   24 +++++++++++++++++-------
> >  1 files changed, 17 insertions(+), 7 deletions(-)
> >
> > diff --git a/cpus.c b/cpus.c
> > index b41baf6..dac222b 100644
> > --- a/cpus.c
> > +++ b/cpus.c
> > @@ -370,7 +370,9 @@ static void icount_warp_rt(void *opaque)
> >
> >      seqlock_write_lock(&timers_state.vm_clock_seqlock);
> >      if (runstate_is_running()) {
> > -        int64_t clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
> > +        int64_t clock = qemu_clock_get_ns(replay_mode == REPLAY_MODE_NONE
> > +                                          ? QEMU_CLOCK_REALTIME
> > +                                          : QEMU_CLOCK_HOST);
> >          int64_t warp_delta;
> >
> >          warp_delta = clock - vm_clock_warp_start;
> > @@ -444,7 +446,9 @@ void qemu_clock_warp(QEMUClockType type)
> >      }
> >
> >      /* We want to use the earliest deadline from ALL vm_clocks */
> > -    clock = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
> > +    clock = qemu_clock_get_ns(replay_mode == REPLAY_MODE_NONE
> > +                              ? QEMU_CLOCK_REALTIME
> > +                              : QEMU_CLOCK_HOST);
> >      deadline = qemu_clock_deadline_ns_all(QEMU_CLOCK_VIRTUAL);
> >      if (deadline < 0) {
> >          return;
> > @@ -537,8 +541,10 @@ void configure_icount(QemuOpts *opts, Error **errp)
> >          return;
> >      }
> >      icount_align_option = qemu_opt_get_bool(opts, "align", false);
> > -    icount_warp_timer = timer_new_ns(QEMU_CLOCK_REALTIME,
> > -                                          icount_warp_rt, NULL);
> > +    icount_warp_timer = timer_new_ns(replay_mode == REPLAY_MODE_NONE
> > +                                     ? QEMU_CLOCK_REALTIME
> > +                                     : QEMU_CLOCK_HOST,
> > +                                     icount_warp_rt, NULL);
> >      if (strcmp(option, "auto") != 0) {
> >          errno = 0;
> >          icount_time_shift = strtol(option, &rem_str, 0);
> > @@ -562,10 +568,14 @@ void configure_icount(QemuOpts *opts, Error **errp)
> >         the virtual time trigger catches emulated time passing too fast.
> >         Realtime triggers occur even when idle, so use them less frequently
> >         than VM triggers.  */
> > -    icount_rt_timer = timer_new_ms(QEMU_CLOCK_REALTIME,
> > -                                        icount_adjust_rt, NULL);
> > +    icount_rt_timer = timer_new_ms(replay_mode == REPLAY_MODE_NONE
> > +                                   ? QEMU_CLOCK_REALTIME
> > +                                   : QEMU_CLOCK_HOST,
> > +                                   icount_adjust_rt, NULL);
> >      timer_mod(icount_rt_timer,
> > -                   qemu_clock_get_ms(QEMU_CLOCK_REALTIME) + 1000);
> > +                   qemu_clock_get_ms(replay_mode == REPLAY_MODE_NONE
> > +                                     ? QEMU_CLOCK_REALTIME
> > +                                     : QEMU_CLOCK_HOST) + 1000);
> >      icount_vm_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL,
> >                                          icount_adjust_vm, NULL);
> >      timer_mod(icount_vm_timer,
> >

Re: [Qemu-devel] [RFC PATCH v4 17/25] cpus: make icount warp deterministic in replay mode

[Paolo Bonzini]

On 07/11/2014 12:45, Pavel Dovgaluk wrote:
>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>> On 07/11/2014 11:33, Pavel Dovgalyuk wrote:
>>> This patch adds saving and replaying warping parameters in record and replay
>>> modes. These parameters affect on virtual clock values and therefore should
>>> be deterministic.
>>
>> Why are QEMU_CLOCK_REALTIME timers not recorded/replayed like
>> QEMU_CLOCK_HOST is?
> 
> We cannot record all QEMU_CLOCK_REALTIME timers, because they are simulator-related timers,
> not the VM-related ones. If we'll record them then we'll have to replay simulator
> execution. And we want just to replay VM allowing different realtime events to be
> non-replayable (like screen refresh, monitor events handling, and so on).

Hmm... I think you want the icount_rt timer to use what
QEMU_CLOCK_VIRTUAL does in !use_icount mode.  Maybe you can add a fourth
clock, QEMU_CLOCK_VIRTUAL_RT and use this one independent of the replay
mode.  This clock can then be traced like QEMU_CLOCK_HOST.

The problem is that QEMU_CLOCK_HOST can move backwards or forwards,
sometimes by as much as 1 hour, so it's not the best match for this case.

Paolo

[Qemu-devel] [RFC PATCH v4 18/25] replay: shutdown event

[Pavel Dovgalyuk]

This patch records and replays simulator shutdown event.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/sysemu/sysemu.h  |    1 +
 replay/replay-internal.h |    2 ++
 replay/replay.c          |   11 +++++++++++
 replay/replay.h          |    5 +++++
 vl.c                     |    8 +++++++-
 5 files changed, 26 insertions(+), 1 deletions(-)

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 72bc093..621154f 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -57,6 +57,7 @@ void qemu_register_suspend_notifier(Notifier *notifier);
 void qemu_system_wakeup_request(WakeupReason reason);
 void qemu_system_wakeup_enable(WakeupReason reason, bool enabled);
 void qemu_register_wakeup_notifier(Notifier *notifier);
+void qemu_system_shutdown_request_impl(void);
 void qemu_system_shutdown_request(void);
 void qemu_system_powerdown_request(void);
 void qemu_register_powerdown_notifier(Notifier *notifier);
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 367d345..94487fb 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -20,6 +20,8 @@
 #define EVENT_TM                    2
 /* for software interrupt */
 #define EVENT_INTERRUPT             15
+/* for shutdown request */
+#define EVENT_SHUTDOWN              20
 /* for emulated exceptions */
 #define EVENT_EXCEPTION             23
 /* for async events */
diff --git a/replay/replay.c b/replay/replay.c
index a6de6a1..c118f62 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -43,6 +43,10 @@ bool skip_async_events(int stop_event)
             res = true;
         }
         switch (replay_data_kind) {
+        case EVENT_SHUTDOWN:
+            replay_has_unread_data = 0;
+            qemu_system_shutdown_request_impl();
+            break;
         case EVENT_INSTRUCTION:
             replay_state.instructions_count = replay_get_dword();
             return res;
@@ -139,3 +143,10 @@ bool replay_has_interrupt(void)
     }
     return false;
 }
+
+void replay_shutdown_request(void)
+{
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_put_event(EVENT_SHUTDOWN);
+    }
+}
diff --git a/replay/replay.h b/replay/replay.h
index 143fe85..8963e4e 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -76,6 +76,11 @@ void replay_save_tm(struct tm *tm);
 /*! Reads struct tm value from the log. Stops execution in case of error */
 void replay_read_tm(struct tm *tm);
 
+/* Events */
+
+/*! Called when qemu shutdown is requested. */
+void replay_shutdown_request(void);
+
 /* Asynchronous events queue */
 
 /*! Disables storing events in the queue */
diff --git a/vl.c b/vl.c
index 37c6616..4155342 100644
--- a/vl.c
+++ b/vl.c
@@ -1792,13 +1792,19 @@ void qemu_system_killed(int signal, pid_t pid)
     qemu_system_shutdown_request();
 }
 
-void qemu_system_shutdown_request(void)
+void qemu_system_shutdown_request_impl(void)
 {
     trace_qemu_system_shutdown_request();
     shutdown_requested = 1;
     qemu_notify_event();
 }
 
+void qemu_system_shutdown_request(void)
+{
+    replay_shutdown_request();
+    qemu_system_shutdown_request_impl();
+}
+
 static void qemu_system_powerdown(void)
 {
     qapi_event_send_powerdown(&error_abort);

[Qemu-devel] [RFC PATCH v4 19/25] replay: checkpoints

[Pavel Dovgalyuk]

This patch introduces checkpoints that synchronize cpu thread and iothread.
When checkpoint is met in the code all asynchronous events from the queue
are executed.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                  |   11 +++++++++++
 cpus.c                   |    2 +-
 include/qemu/timer.h     |    6 ++++--
 qemu-timer.c             |   41 +++++++++++++++++++++++++++++++++--------
 replay/replay-internal.h |    3 +++
 replay/replay.c          |   28 ++++++++++++++++++++++++++++
 replay/replay.h          |    6 ++++++
 stubs/replay.c           |   11 +++++++++++
 vl.c                     |    3 ++-
 9 files changed, 99 insertions(+), 12 deletions(-)

diff --git a/block.c b/block.c
index 88f6d9b..cc84050 100644
--- a/block.c
+++ b/block.c
@@ -1919,6 +1919,11 @@ void bdrv_drain_all(void)
     BlockDriverState *bs;
 
     while (busy) {
+        if (!replay_checkpoint(8)) {
+            /* Do not wait anymore, we stopped at some place in
+               the middle of execution during replay */
+            return;
+        }
         busy = false;
 
         QTAILQ_FOREACH(bs, &bdrv_states, device_list) {
@@ -1935,6 +1940,12 @@ void bdrv_drain_all(void)
             busy |= bs_busy;
         }
     }
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Skip checkpoints from the log */
+        while (replay_checkpoint(8)) {
+            /* Nothing */
+        }
+    }
 }
 
 /* make a BlockDriverState anonymous by removing from bdrv_state and
diff --git a/cpus.c b/cpus.c
index b7a25fd..6adff2b 100644
--- a/cpus.c
+++ b/cpus.c
@@ -406,7 +406,7 @@ void qtest_clock_warp(int64_t dest)
         timers_state.qemu_icount_bias += warp;
         seqlock_write_unlock(&timers_state.vm_clock_seqlock);
 
-        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL);
+        qemu_clock_run_timers(QEMU_CLOCK_VIRTUAL, false);
         clock = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
     }
     qemu_clock_notify(QEMU_CLOCK_VIRTUAL);
diff --git a/include/qemu/timer.h b/include/qemu/timer.h
index 7b43331..5eaf77a 100644
--- a/include/qemu/timer.h
+++ b/include/qemu/timer.h
@@ -232,13 +232,14 @@ void qemu_clock_unregister_reset_notifier(QEMUClockType type,
 /**
  * qemu_clock_run_timers:
  * @type: clock on which to operate
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Run all the timers associated with the default timer list
  * of a clock.
  *
  * Returns: true if any timer ran.
  */
-bool qemu_clock_run_timers(QEMUClockType type);
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all);
 
 /**
  * qemu_clock_run_all_timers:
@@ -329,12 +330,13 @@ QEMUClockType timerlist_get_clock(QEMUTimerList *timer_list);
 /**
  * timerlist_run_timers:
  * @timer_list: the timer list to use
+ * @run_all: true, when called from qemu_clock_run_all_timers
  *
  * Call all expired timers associated with the timer list.
  *
  * Returns: true if any timer expired
  */
-bool timerlist_run_timers(QEMUTimerList *timer_list);
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all);
 
 /**
  * timerlist_notify:
diff --git a/qemu-timer.c b/qemu-timer.c
index 8307913..b06aa4a 100644
--- a/qemu-timer.c
+++ b/qemu-timer.c
@@ -458,7 +458,7 @@ bool timer_expired(QEMUTimer *timer_head, int64_t current_time)
     return timer_expired_ns(timer_head, current_time * timer_head->scale);
 }
 
-bool timerlist_run_timers(QEMUTimerList *timer_list)
+bool timerlist_run_timers(QEMUTimerList *timer_list, bool run_all)
 {
     QEMUTimer *ts;
     int64_t current_time;
@@ -466,6 +466,24 @@ bool timerlist_run_timers(QEMUTimerList *timer_list)
     QEMUTimerCB *cb;
     void *opaque;
 
+    switch (timer_list->clock->type) {
+    case QEMU_CLOCK_REALTIME:
+        break;
+    default:
+    case QEMU_CLOCK_VIRTUAL:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 2 : 3)) {
+            return false;
+        }
+        break;
+    case QEMU_CLOCK_HOST:
+        if ((replay_mode != REPLAY_MODE_NONE && !runstate_is_running())
+            || !replay_checkpoint(run_all ? 5 : 6)) {
+            return false;
+        }
+        break;
+    }
+
     qemu_event_reset(&timer_list->timers_done_ev);
     if (!timer_list->clock->enabled) {
         goto out;
@@ -498,9 +516,9 @@ out:
     return progress;
 }
 
-bool qemu_clock_run_timers(QEMUClockType type)
+bool qemu_clock_run_timers(QEMUClockType type, bool run_all)
 {
-    return timerlist_run_timers(main_loop_tlg.tl[type]);
+    return timerlist_run_timers(main_loop_tlg.tl[type], run_all);
 }
 
 void timerlistgroup_init(QEMUTimerListGroup *tlg,
@@ -525,7 +543,7 @@ bool timerlistgroup_run_timers(QEMUTimerListGroup *tlg)
     QEMUClockType type;
     bool progress = false;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
-        progress |= timerlist_run_timers(tlg->tl[type]);
+        progress |= timerlist_run_timers(tlg->tl[type], false);
     }
     return progress;
 }
@@ -534,11 +552,18 @@ int64_t timerlistgroup_deadline_ns(QEMUTimerListGroup *tlg)
 {
     int64_t deadline = -1;
     QEMUClockType type;
+    bool play = replay_mode == REPLAY_MODE_PLAY;
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
         if (qemu_clock_use_for_deadline(tlg->tl[type]->clock->type)) {
-            deadline = qemu_soonest_timeout(deadline,
-                                            timerlist_deadline_ns(
-                                                tlg->tl[type]));
+            if (!play || tlg->tl[type]->clock->type == QEMU_CLOCK_REALTIME) {
+                deadline = qemu_soonest_timeout(deadline,
+                                                timerlist_deadline_ns(
+                                                    tlg->tl[type]));
+            } else {
+                /* Read clock from the replay file and
+                   do not calculate the deadline, based on virtual clock. */
+                qemu_clock_get_ns(tlg->tl[type]->clock->type);
+            }
         }
     }
     return deadline;
@@ -606,7 +631,7 @@ bool qemu_clock_run_all_timers(void)
     QEMUClockType type;
 
     for (type = 0; type < QEMU_CLOCK_MAX; type++) {
-        progress |= qemu_clock_run_timers(type);
+        progress |= qemu_clock_run_timers(type, true);
     }
 
     return progress;
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 94487fb..e73fa7a 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -32,6 +32,9 @@
 #define EVENT_CLOCK                 64
 /* some of grteater codes are reserved for clocks */
 
+/* for checkpoint event */
+#define EVENT_CHECKPOINT            96
+
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_COUNT             0
diff --git a/replay/replay.c b/replay/replay.c
index c118f62..54e1c4e 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -150,3 +150,31 @@ void replay_shutdown_request(void)
         replay_put_event(EVENT_SHUTDOWN);
     }
 }
+
+/* Used checkpoints: 2 3 5 6 7 8 9 */
+int replay_checkpoint(unsigned int checkpoint)
+{
+    replay_save_instructions();
+
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_PLAY) {
+            if (!skip_async_events(EVENT_CHECKPOINT + checkpoint)) {
+                if (replay_data_kind == EVENT_ASYNC_OPT) {
+                    replay_read_events(checkpoint);
+                    replay_fetch_data_kind();
+                    return replay_data_kind != EVENT_ASYNC_OPT;
+                }
+                return 0;
+            }
+            replay_has_unread_data = 0;
+            replay_read_events(checkpoint);
+            replay_fetch_data_kind();
+            return replay_data_kind != EVENT_ASYNC_OPT;
+        } else if (replay_mode == REPLAY_MODE_RECORD) {
+            replay_put_event(EVENT_CHECKPOINT + checkpoint);
+            replay_save_events(checkpoint);
+        }
+    }
+
+    return 1;
+}
diff --git a/replay/replay.h b/replay/replay.h
index 8963e4e..e76c541 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -80,6 +80,12 @@ void replay_read_tm(struct tm *tm);
 
 /*! Called when qemu shutdown is requested. */
 void replay_shutdown_request(void);
+/*! Should be called at check points in the execution.
+    These check points are skipped, if they were not met.
+    Saves checkpoint in the SAVE mode and validates in the PLAY mode.
+    Returns 0 in PLAY mode if checkpoint was not found.
+    Returns 1 in all other cases. */
+int replay_checkpoint(unsigned int checkpoint);
 
 /* Asynchronous events queue */
 
diff --git a/stubs/replay.c b/stubs/replay.c
index f3f60fe..ab9ede9 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -1,4 +1,5 @@
 #include "replay/replay.h"
+#include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
 
@@ -15,3 +16,13 @@ int64_t replay_read_clock(unsigned int kind)
 {
     return 0;
 }
+
+int replay_checkpoint(unsigned int checkpoint)
+{
+    return 0;
+}
+
+int runstate_is_running(void)
+{
+    return 0;
+}
diff --git a/vl.c b/vl.c
index 4155342..717d67e 100644
--- a/vl.c
+++ b/vl.c
@@ -1847,7 +1847,8 @@ static bool main_loop_should_exit(void)
             return true;
         }
     }
-    if (qemu_reset_requested()) {
+    if (qemu_reset_requested_get() && replay_checkpoint(7)) {
+        qemu_reset_requested();
         pause_all_vcpus();
         cpu_synchronize_all_states();
         qemu_system_reset(VMRESET_REPORT);

[Qemu-devel] [RFC PATCH v4 20/25] replay: bottom halves

[Pavel Dovgalyuk]

This patch introduces bottom half event for replay queue. It saves the events
into the queue and process them at the checkpoints and instructions execution.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 async.c                  |   46 ++++++++++++++++++++++++++++++++++++++++------
 dma-helpers.c            |    4 +++-
 hw/ide/ahci.c            |    4 +++-
 hw/ide/core.c            |    4 +++-
 hw/timer/arm_timer.c     |    2 +-
 hw/usb/hcd-uhci.c        |    2 +-
 include/block/aio.h      |   18 ++++++++++++++++++
 include/qemu/main-loop.h |    1 +
 main-loop.c              |    5 +++++
 replay/replay-events.c   |   16 ++++++++++++++++
 replay/replay-internal.h |    3 ++-
 replay/replay.h          |    2 ++
 stubs/replay.c           |    4 ++++
 13 files changed, 99 insertions(+), 12 deletions(-)

diff --git a/async.c b/async.c
index 6e1b282..97111c0 100644
--- a/async.c
+++ b/async.c
@@ -27,6 +27,7 @@
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
 #include "qemu/atomic.h"
+#include "replay/replay.h"
 
 /***********************************************************/
 /* bottom halves (can be seen as timers which expire ASAP) */
@@ -39,24 +40,53 @@ struct QEMUBH {
     bool scheduled;
     bool idle;
     bool deleted;
+    bool replay;
+    uint64_t id;
 };
 
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque)
 {
-    QEMUBH *bh;
+    QEMUBH *bh, **last;
     bh = g_malloc0(sizeof(QEMUBH));
     bh->ctx = ctx;
     bh->cb = cb;
     bh->opaque = opaque;
     qemu_mutex_lock(&ctx->bh_lock);
-    bh->next = ctx->first_bh;
-    /* Make sure that the members are ready before putting bh into list */
-    smp_wmb();
-    ctx->first_bh = bh;
+    if (replay_mode != REPLAY_MODE_NONE) {
+        /* Slower way, but this is a queue and not a stack.
+           Replay will process the BH in the same order they
+           came into the queue. */
+        last = &ctx->first_bh;
+        while (*last) {
+            last = &(*last)->next;
+        }
+        smp_wmb();
+        *last = bh;
+    } else {
+        bh->next = ctx->first_bh;
+        /* Make sure that the members are ready before putting bh into list */
+        smp_wmb();
+        ctx->first_bh = bh;
+    }
     qemu_mutex_unlock(&ctx->bh_lock);
     return bh;
 }
 
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id)
+{
+    QEMUBH *bh = aio_bh_new(ctx, cb, opaque);
+    bh->replay = true;
+    bh->id = id;
+    return bh;
+}
+
+void aio_bh_call(void *opaque)
+{
+    QEMUBH *bh = (QEMUBH *)opaque;
+    bh->cb(bh->opaque);
+}
+
 /* Multiple occurrences of aio_bh_poll cannot be called concurrently */
 int aio_bh_poll(AioContext *ctx)
 {
@@ -79,7 +109,11 @@ int aio_bh_poll(AioContext *ctx)
             if (!bh->idle)
                 ret = 1;
             bh->idle = 0;
-            bh->cb(bh->opaque);
+            if (!bh->replay) {
+                aio_bh_call(bh);
+            } else {
+                replay_add_bh_event(bh, bh->id);
+            }
         }
     }
 
diff --git a/dma-helpers.c b/dma-helpers.c
index 6918572..357d7e9 100644
--- a/dma-helpers.c
+++ b/dma-helpers.c
@@ -13,6 +13,7 @@
 #include "qemu/range.h"
 #include "qemu/thread.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 /* #define DEBUG_IOMMU */
 
@@ -96,7 +97,8 @@ static void continue_after_map_failure(void *opaque)
 {
     DMAAIOCB *dbs = (DMAAIOCB *)opaque;
 
-    dbs->bh = qemu_bh_new(reschedule_dma, dbs);
+    dbs->bh = qemu_bh_new_replay(reschedule_dma, dbs,
+                                 replay_get_current_step());
     qemu_bh_schedule(dbs->bh);
 }
 
diff --git a/hw/ide/ahci.c b/hw/ide/ahci.c
index 70958e3..fbefc52 100644
--- a/hw/ide/ahci.c
+++ b/hw/ide/ahci.c
@@ -33,6 +33,7 @@
 #include "internal.h"
 #include <hw/ide/pci.h>
 #include <hw/ide/ahci.h>
+#include "replay/replay.h"
 
 /* #define DEBUG_AHCI */
 
@@ -1192,7 +1193,8 @@ static void ahci_cmd_done(IDEDMA *dma)
 
     if (!ad->check_bh) {
         /* maybe we still have something to process, check later */
-        ad->check_bh = qemu_bh_new(ahci_check_cmd_bh, ad);
+        ad->check_bh = qemu_bh_new_replay(ahci_check_cmd_bh, ad,
+                                          replay_get_current_step());
         qemu_bh_schedule(ad->check_bh);
     }
 }
diff --git a/hw/ide/core.c b/hw/ide/core.c
index 44e3d50..b622aae 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -32,6 +32,7 @@
 #include "sysemu/dma.h"
 #include "hw/block/block.h"
 #include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 #include <hw/ide/internal.h>
 
@@ -448,7 +449,8 @@ BlockAIOCB *ide_issue_trim(BlockBackend *blk,
 
     iocb = blk_aio_get(&trim_aiocb_info, blk, cb, opaque);
     iocb->blk = blk;
-    iocb->bh = qemu_bh_new(ide_trim_bh_cb, iocb);
+    iocb->bh = qemu_bh_new_replay(ide_trim_bh_cb, iocb,
+                                  replay_get_current_step());
     iocb->ret = 0;
     iocb->qiov = qiov;
     iocb->i = -1;
diff --git a/hw/timer/arm_timer.c b/hw/timer/arm_timer.c
index 1452910..97784a0 100644
--- a/hw/timer/arm_timer.c
+++ b/hw/timer/arm_timer.c
@@ -168,7 +168,7 @@ static arm_timer_state *arm_timer_init(uint32_t freq)
     s->freq = freq;
     s->control = TIMER_CTRL_IE;
 
-    bh = qemu_bh_new(arm_timer_tick, s);
+    bh = qemu_bh_new_replay(arm_timer_tick, s, 0);
     s->timer = ptimer_init(bh);
     vmstate_register(NULL, -1, &vmstate_arm_timer, s);
     return s;
diff --git a/hw/usb/hcd-uhci.c b/hw/usb/hcd-uhci.c
index 5b88f30..dc17e5f 100644
--- a/hw/usb/hcd-uhci.c
+++ b/hw/usb/hcd-uhci.c
@@ -1224,7 +1224,7 @@ static int usb_uhci_common_initfn(PCIDevice *dev)
                               USB_SPEED_MASK_LOW | USB_SPEED_MASK_FULL);
         }
     }
-    s->bh = qemu_bh_new(uhci_bh, s);
+    s->bh = qemu_bh_new_replay(uhci_bh, s, 0);
     s->frame_timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, uhci_frame_timer, s);
     s->num_ports_vmstate = NB_PORTS;
     QTAILQ_INIT(&s->queues);
diff --git a/include/block/aio.h b/include/block/aio.h
index 6bf0e04..5a77431 100644
--- a/include/block/aio.h
+++ b/include/block/aio.h
@@ -35,6 +35,8 @@ struct BlockAIOCB {
     const AIOCBInfo *aiocb_info;
     BlockDriverState *bs;
     BlockCompletionFunc *cb;
+    bool replay;
+    uint64_t replay_step;
     void *opaque;
     int refcnt;
 };
@@ -144,6 +146,17 @@ void aio_context_release(AioContext *ctx);
 QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 
 /**
+ * aio_bh_new_replay: Allocate a new bottom half structure for replay.
+ *
+ * This function calls aio_bh_new function and also fills replay parameters
+ * of the BH structure. BH created with this function in record/replay mode
+ * are executed through the replay queue only at checkpoints and instructions
+ * executions.
+ */
+QEMUBH *aio_bh_new_replay(AioContext *ctx, QEMUBHFunc *cb, void *opaque,
+                          uint64_t id);
+
+/**
  * aio_notify: Force processing of pending events.
  *
  * Similar to signaling a condition variable, aio_notify forces
@@ -159,6 +172,11 @@ QEMUBH *aio_bh_new(AioContext *ctx, QEMUBHFunc *cb, void *opaque);
 void aio_notify(AioContext *ctx);
 
 /**
+ * aio_bh_call: Executes callback function of the specified BH.
+ */
+void aio_bh_call(void *opaque);
+
+/**
  * aio_bh_poll: Poll bottom halves for an AioContext.
  *
  * These are internal functions used by the QEMU main loop.
diff --git a/include/qemu/main-loop.h b/include/qemu/main-loop.h
index 62c68c0..f5a98fe 100644
--- a/include/qemu/main-loop.h
+++ b/include/qemu/main-loop.h
@@ -306,6 +306,7 @@ void qemu_iohandler_fill(GArray *pollfds);
 void qemu_iohandler_poll(GArray *pollfds, int rc);
 
 QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque);
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id);
 void qemu_bh_schedule_idle(QEMUBH *bh);
 
 #endif
diff --git a/main-loop.c b/main-loop.c
index 981bcb5..342aeef 100644
--- a/main-loop.c
+++ b/main-loop.c
@@ -508,3 +508,8 @@ QEMUBH *qemu_bh_new(QEMUBHFunc *cb, void *opaque)
 {
     return aio_bh_new(qemu_aio_context, cb, opaque);
 }
+
+QEMUBH *qemu_bh_new_replay(QEMUBHFunc *cb, void *opaque, uint64_t id)
+{
+    return aio_bh_new_replay(qemu_aio_context, cb, opaque, id);
+}
diff --git a/replay/replay-events.c b/replay/replay-events.c
index f3c9b16..1aee0a4 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -36,6 +36,9 @@ static bool replay_events_enabled = false;
 static void replay_run_event(Event *event)
 {
     switch (event->event_kind) {
+    case REPLAY_ASYNC_EVENT_BH:
+        aio_bh_call(event->opaque);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -119,6 +122,11 @@ void replay_add_event(int event_kind, void *opaque)
     replay_add_event_internal(event_kind, opaque, NULL, 0);
 }
 
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -132,6 +140,9 @@ void replay_save_events(int opt)
 
             /* save event-specific data */
             switch (event->event_kind) {
+            case REPLAY_ASYNC_EVENT_BH:
+                replay_put_qword(event->id);
+                break;
             }
         }
 
@@ -158,6 +169,11 @@ void replay_read_events(int opt)
         }
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
+        case REPLAY_ASYNC_EVENT_BH:
+            if (read_id == -1) {
+                read_id = replay_get_qword();
+            }
+            break;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index e73fa7a..ec36f44 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -37,7 +37,8 @@
 
 /* Asynchronous events IDs */
 
-#define REPLAY_ASYNC_COUNT             0
+#define REPLAY_ASYNC_EVENT_BH          0
+#define REPLAY_ASYNC_COUNT             1
 
 typedef struct ReplayState {
     /*! Cached clock values. */
diff --git a/replay/replay.h b/replay/replay.h
index e76c541..ad8bdd3 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -91,5 +91,7 @@ int replay_checkpoint(unsigned int checkpoint);
 
 /*! Disables storing events in the queue */
 void replay_disable_events(void);
+/*! Adds BH event to the queue */
+void replay_add_bh_event(void *bh, uint64_t id);
 
 #endif
diff --git a/stubs/replay.c b/stubs/replay.c
index ab9ede9..6d6c2eb 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -26,3 +26,7 @@ int runstate_is_running(void)
 {
     return 0;
 }
+
+void replay_add_bh_event(void *bh, uint64_t id)
+{
+}

[Qemu-devel] [RFC PATCH v4 21/25] replay: replay aio requests

[Pavel Dovgalyuk]

This patch adds identifier to aio requests. ID is used for creating bottom
halves and identifying them while replaying.
The patch also introduces several functions that make possible replaying
of the aio requests.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                        |   81 ++++++++++++++++++++++++++++++++++++----
 block/block-backend.c          |   30 ++++++++++++++-
 block/qcow2.c                  |    4 ++
 dma-helpers.c                  |    6 ++-
 hw/block/virtio-blk.c          |   10 ++---
 hw/ide/atapi.c                 |   10 +++--
 hw/ide/core.c                  |   14 ++++---
 include/block/block.h          |   15 +++++++
 include/qemu-common.h          |    2 +
 include/sysemu/block-backend.h |   10 +++++
 qemu-io-cmds.c                 |    2 -
 stubs/replay.c                 |    5 ++
 trace-events                   |    2 +
 util/iov.c                     |    4 ++
 14 files changed, 167 insertions(+), 28 deletions(-)

diff --git a/block.c b/block.c
index cc84050..02c6a78 100644
--- a/block.c
+++ b/block.c
@@ -83,7 +83,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write);
+                                         bool is_write,
+                                         bool aio_replay);
 static void coroutine_fn bdrv_co_do_rw(void *opaque);
 static int coroutine_fn bdrv_co_do_write_zeroes(BlockDriverState *bs,
     int64_t sector_num, int nb_sectors, BdrvRequestFlags flags);
@@ -4342,7 +4343,19 @@ BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_readv(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, false);
+                                 cb, opaque, false, false);
+}
+
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *qiov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque)
+{
+    trace_bdrv_aio_readv_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, false, true);
 }
 
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
@@ -4352,7 +4365,19 @@ BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
     trace_bdrv_aio_writev(bs, sector_num, nb_sectors, opaque);
 
     return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
-                                 cb, opaque, true);
+                                 cb, opaque, true, false);
+}
+
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *qiov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque)
+{
+    trace_bdrv_aio_writev_replay(bs, sector_num, nb_sectors, opaque);
+
+    return bdrv_co_aio_rw_vector(bs, sector_num, qiov, nb_sectors, 0,
+                                 cb, opaque, true, true);
 }
 
 BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
@@ -4363,7 +4388,7 @@ BlockAIOCB *bdrv_aio_write_zeroes(BlockDriverState *bs,
 
     return bdrv_co_aio_rw_vector(bs, sector_num, NULL, nb_sectors,
                                  BDRV_REQ_ZERO_WRITE | flags,
-                                 cb, opaque, true);
+                                 cb, opaque, true, true);
 }
 
 
@@ -4505,7 +4530,8 @@ static int multiwrite_merge(BlockDriverState *bs, BlockRequest *reqs,
  * requests. However, the fields opaque and error are left unmodified as they
  * are used to signal failure for a single request to the caller.
  */
-int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
+int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs,
+                        bool replay)
 {
     MultiwriteCB *mcb;
     int i;
@@ -4543,7 +4569,7 @@ int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs, int num_reqs)
         bdrv_co_aio_rw_vector(bs, reqs[i].sector, reqs[i].qiov,
                               reqs[i].nb_sectors, reqs[i].flags,
                               multiwrite_cb, mcb,
-                              true);
+                              true, replay);
     }
 
     return 0;
@@ -4688,7 +4714,12 @@ static void coroutine_fn bdrv_co_do_rw(void *opaque)
             acb->req.nb_sectors, acb->req.qiov, acb->req.flags);
     }
 
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4699,7 +4730,8 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
                                          BdrvRequestFlags flags,
                                          BlockCompletionFunc *cb,
                                          void *opaque,
-                                         bool is_write)
+                                         bool is_write,
+                                         bool aio_replay)
 {
     Coroutine *co;
     BlockAIOCBCoroutine *acb;
@@ -4710,6 +4742,11 @@ static BlockAIOCB *bdrv_co_aio_rw_vector(BlockDriverState *bs,
     acb->req.qiov = qiov;
     acb->req.flags = flags;
     acb->is_write = is_write;
+    acb->done = NULL;
+    acb->common.replay = aio_replay;
+    if (aio_replay) {
+        acb->common.replay_step = replay_get_current_step();
+    }
 
     co = qemu_coroutine_create(bdrv_co_do_rw);
     qemu_coroutine_enter(co, acb);
@@ -4723,7 +4760,12 @@ static void coroutine_fn bdrv_aio_flush_co_entry(void *opaque)
     BlockDriverState *bs = acb->common.bs;
 
     acb->req.error = bdrv_co_flush(bs);
-    acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    if (acb->common.replay) {
+        acb->bh = aio_bh_new_replay(bdrv_get_aio_context(bs), bdrv_co_em_bh,
+                                    acb, acb->common.replay_step);
+    } else {
+        acb->bh = aio_bh_new(bdrv_get_aio_context(bs), bdrv_co_em_bh, acb);
+    }
     qemu_bh_schedule(acb->bh);
 }
 
@@ -4743,6 +4785,25 @@ BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
     return &acb->common;
 }
 
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+        BlockCompletionFunc *cb, void *opaque)
+{
+    trace_bdrv_aio_flush(bs, opaque);
+
+    Coroutine *co;
+    BlockAIOCBCoroutine *acb;
+
+    acb = qemu_aio_get(&bdrv_em_co_aiocb_info, bs, cb, opaque);
+    acb->done = NULL;
+    acb->common.replay = true;
+    acb->common.replay_step = replay_get_current_step();
+
+    co = qemu_coroutine_create(bdrv_aio_flush_co_entry);
+    qemu_coroutine_enter(co, acb);
+
+    return &acb->common;
+}
+
 static void coroutine_fn bdrv_aio_discard_co_entry(void *opaque)
 {
     BlockAIOCBCoroutine *acb = opaque;
@@ -4793,6 +4854,8 @@ void *qemu_aio_get(const AIOCBInfo *aiocb_info, BlockDriverState *bs,
     acb->cb = cb;
     acb->opaque = opaque;
     acb->refcnt = 1;
+    acb->replay_step = 0;
+    acb->replay = false;
     return acb;
 }
 
diff --git a/block/block-backend.c b/block/block-backend.c
index d0692b1..a3f9e67 100644
--- a/block/block-backend.c
+++ b/block/block-backend.c
@@ -451,6 +451,14 @@ BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_readv(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_readv_replay(blk->bs, sector_num, iov, nb_sectors,
+                                 cb, opaque);
+}
+
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque)
@@ -458,12 +466,26 @@ BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
     return bdrv_aio_writev(blk->bs, sector_num, iov, nb_sectors, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_writev_replay(blk->bs, sector_num, iov, nb_sectors,
+                                  cb, opaque);
+}
+
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque)
 {
     return bdrv_aio_flush(blk->bs, cb, opaque);
 }
 
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque)
+{
+    return bdrv_aio_flush_replay(blk->bs, cb, opaque);
+}
+
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque)
@@ -483,7 +505,13 @@ void blk_aio_cancel_async(BlockAIOCB *acb)
 
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs)
 {
-    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs);
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, false);
+}
+
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs)
+{
+    return bdrv_aio_multiwrite(blk->bs, reqs, num_reqs, true);
 }
 
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf)
diff --git a/block/qcow2.c b/block/qcow2.c
index d031515..96a607e 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1140,6 +1140,8 @@ static coroutine_fn int qcow2_co_readv(BlockDriverState *bs, int64_t sector_num,
     uint8_t *cluster_data = NULL;
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     qemu_co_mutex_lock(&s->lock);
 
@@ -1297,6 +1299,8 @@ static coroutine_fn int qcow2_co_writev(BlockDriverState *bs,
                                  remaining_sectors);
 
     qemu_iovec_init(&hd_qiov, qiov->niov);
+    hd_qiov.replay = qiov->replay;
+    hd_qiov.replay_step = qiov->replay_step;
 
     s->cluster_cache_offset = -1; /* disable compressed cache */
 
diff --git a/dma-helpers.c b/dma-helpers.c
index 357d7e9..4faf6d3 100644
--- a/dma-helpers.c
+++ b/dma-helpers.c
@@ -212,6 +212,8 @@ BlockAIOCB *dma_blk_io(
     dbs->io_func = io_func;
     dbs->bh = NULL;
     qemu_iovec_init(&dbs->iov, sg->nsg);
+    dbs->iov.replay = true;
+    dbs->iov.replay_step = replay_get_current_step();
     dma_blk_cb(dbs, 0);
     return &dbs->common;
 }
@@ -221,7 +223,7 @@ BlockAIOCB *dma_blk_read(BlockBackend *blk,
                          QEMUSGList *sg, uint64_t sector,
                          void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_readv, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_readv_replay, cb, opaque,
                       DMA_DIRECTION_FROM_DEVICE);
 }
 
@@ -229,7 +231,7 @@ BlockAIOCB *dma_blk_write(BlockBackend *blk,
                           QEMUSGList *sg, uint64_t sector,
                           void (*cb)(void *opaque, int ret), void *opaque)
 {
-    return dma_blk_io(blk, sg, sector, blk_aio_writev, cb, opaque,
+    return dma_blk_io(blk, sg, sector, blk_aio_writev_replay, cb, opaque,
                       DMA_DIRECTION_TO_DEVICE);
 }
 
diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c
index b19b102..2413cbe 100644
--- a/hw/block/virtio-blk.c
+++ b/hw/block/virtio-blk.c
@@ -265,7 +265,7 @@ void virtio_submit_multiwrite(BlockBackend *blk, MultiReqBuffer *mrb)
         return;
     }
 
-    ret = blk_aio_multiwrite(blk, mrb->blkreq, mrb->num_writes);
+    ret = blk_aio_multiwrite_replay(blk, mrb->blkreq, mrb->num_writes);
     if (ret != 0) {
         for (i = 0; i < mrb->num_writes; i++) {
             if (mrb->blkreq[i].error) {
@@ -286,7 +286,7 @@ static void virtio_blk_handle_flush(VirtIOBlockReq *req, MultiReqBuffer *mrb)
      * Make sure all outstanding writes are posted to the backing device.
      */
     virtio_submit_multiwrite(req->dev->blk, mrb);
-    blk_aio_flush(req->dev->blk, virtio_blk_flush_complete, req);
+    blk_aio_flush_replay(req->dev->blk, virtio_blk_flush_complete, req);
 }
 
 static bool virtio_blk_sect_range_ok(VirtIOBlock *dev,
@@ -357,9 +357,9 @@ static void virtio_blk_handle_read(VirtIOBlockReq *req)
 
     block_acct_start(blk_get_stats(req->dev->blk), &req->acct, req->qiov.size,
                      BLOCK_ACCT_READ);
-    blk_aio_readv(req->dev->blk, sector, &req->qiov,
-                  req->qiov.size / BDRV_SECTOR_SIZE,
-                  virtio_blk_rw_complete, req);
+    blk_aio_readv_replay(req->dev->blk, sector, &req->qiov,
+                          req->qiov.size / BDRV_SECTOR_SIZE,
+                          virtio_blk_rw_complete, req);
 }
 
 void virtio_blk_handle_request(VirtIOBlockReq *req, MultiReqBuffer *mrb)
diff --git a/hw/ide/atapi.c b/hw/ide/atapi.c
index c63b7e5..02b11df 100644
--- a/hw/ide/atapi.c
+++ b/hw/ide/atapi.c
@@ -25,7 +25,7 @@
 
 #include "hw/ide/internal.h"
 #include "hw/scsi/scsi.h"
-#include "sysemu/block-backend.h"
+#include "replay/replay.h"
 
 static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret);
 
@@ -350,10 +350,12 @@ static void ide_atapi_cmd_read_dma_cb(void *opaque, int ret)
     s->bus->dma->iov.iov_base = (void *)(s->io_buffer + data_offset);
     s->bus->dma->iov.iov_len = n * 4 * 512;
     qemu_iovec_init_external(&s->bus->dma->qiov, &s->bus->dma->iov, 1);
+    s->bus->dma->qiov.replay = true;
+    s->bus->dma->qiov.replay_step = replay_get_current_step();
 
-    s->bus->dma->aiocb = blk_aio_readv(s->blk, (int64_t)s->lba << 2,
-                                       &s->bus->dma->qiov, n * 4,
-                                       ide_atapi_cmd_read_dma_cb, s);
+    s->bus->dma->aiocb = blk_aio_readv_replay(s->blk, (int64_t)s->lba << 2,
+                                              &s->bus->dma->qiov, n * 4,
+                                              ide_atapi_cmd_read_dma_cb, s);
     return;
 
 eot:
diff --git a/hw/ide/core.c b/hw/ide/core.c
index b622aae..1886364 100644
--- a/hw/ide/core.c
+++ b/hw/ide/core.c
@@ -629,11 +629,13 @@ void ide_sector_read(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_readv(s->blk, sector_num, &s->qiov, n,
-                                 ide_sector_read_cb, s);
+    s->pio_aiocb = blk_aio_readv_replay(s->blk, sector_num, &s->qiov, n,
+                                        ide_sector_read_cb, s);
 }
 
 static void dma_buf_commit(IDEState *s)
@@ -881,11 +883,13 @@ void ide_sector_write(IDEState *s)
     s->iov.iov_base = s->io_buffer;
     s->iov.iov_len  = n * BDRV_SECTOR_SIZE;
     qemu_iovec_init_external(&s->qiov, &s->iov, 1);
+    s->qiov.replay = true;
+    s->qiov.replay_step = replay_get_current_step();
 
     block_acct_start(blk_get_stats(s->blk), &s->acct,
                      n * BDRV_SECTOR_SIZE, BLOCK_ACCT_READ);
-    s->pio_aiocb = blk_aio_writev(s->blk, sector_num, &s->qiov, n,
-                                  ide_sector_write_cb, s);
+    s->pio_aiocb = blk_aio_writev_replay(s->blk, sector_num, &s->qiov, n,
+                                         ide_sector_write_cb, s);
 }
 
 static void ide_flush_cb(void *opaque, int ret)
@@ -921,7 +925,7 @@ void ide_flush_cache(IDEState *s)
 
     s->status |= BUSY_STAT;
     block_acct_start(blk_get_stats(s->blk), &s->acct, 0, BLOCK_ACCT_FLUSH);
-    s->pio_aiocb = blk_aio_flush(s->blk, ide_flush_cb, s);
+    s->pio_aiocb = blk_aio_flush_replay(s->blk, ide_flush_cb, s);
 }
 
 static void ide_cfata_metadata_inquiry(IDEState *s)
diff --git a/include/block/block.h b/include/block/block.h
index 341054d..17211ff 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -284,11 +284,24 @@ typedef void BlockDriverDirtyHandler(BlockDriverState *bs, int64_t sector,
 BlockAIOCB *bdrv_aio_readv(BlockDriverState *bs, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_readv_replay(BlockDriverState *bs,
+                                  int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_writev(BlockDriverState *bs, int64_t sector_num,
                             QEMUIOVector *iov, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_writev_replay(BlockDriverState *bs,
+                                   int64_t sector_num,
+                                   QEMUIOVector *iov, int nb_sectors,
+                                   BlockCompletionFunc *cb,
+                                   void *opaque);
 BlockAIOCB *bdrv_aio_flush(BlockDriverState *bs,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *bdrv_aio_flush_replay(BlockDriverState *bs,
+                                  BlockCompletionFunc *cb,
+                                  void *opaque);
 BlockAIOCB *bdrv_aio_discard(BlockDriverState *bs,
                              int64_t sector_num, int nb_sectors,
                              BlockCompletionFunc *cb, void *opaque);
@@ -309,7 +322,7 @@ typedef struct BlockRequest {
 } BlockRequest;
 
 int bdrv_aio_multiwrite(BlockDriverState *bs, BlockRequest *reqs,
-    int num_reqs);
+                        int num_reqs, bool replay);
 
 /* sg packet commands */
 int bdrv_ioctl(BlockDriverState *bs, unsigned long int req, void *buf);
diff --git a/include/qemu-common.h b/include/qemu-common.h
index d18f033..1fa65d0 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -320,6 +320,8 @@ typedef struct QEMUIOVector {
     int niov;
     int nalloc;
     size_t size;
+    bool replay;
+    uint64_t replay_step;
 } QEMUIOVector;
 
 void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint);
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index 52d13c1..1f8a7c3 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -94,17 +94,27 @@ void blk_get_geometry(BlockBackend *blk, uint64_t *nb_sectors_ptr);
 BlockAIOCB *blk_aio_readv(BlockBackend *blk, int64_t sector_num,
                           QEMUIOVector *iov, int nb_sectors,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_readv_replay(BlockBackend *blk, int64_t sector_num,
+                                 QEMUIOVector *iov, int nb_sectors,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_writev(BlockBackend *blk, int64_t sector_num,
                            QEMUIOVector *iov, int nb_sectors,
                            BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_writev_replay(BlockBackend *blk, int64_t sector_num,
+                                  QEMUIOVector *iov, int nb_sectors,
+                                  BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_flush(BlockBackend *blk,
                           BlockCompletionFunc *cb, void *opaque);
+BlockAIOCB *blk_aio_flush_replay(BlockBackend *blk,
+                                 BlockCompletionFunc *cb, void *opaque);
 BlockAIOCB *blk_aio_discard(BlockBackend *blk,
                             int64_t sector_num, int nb_sectors,
                             BlockCompletionFunc *cb, void *opaque);
 void blk_aio_cancel(BlockAIOCB *acb);
 void blk_aio_cancel_async(BlockAIOCB *acb);
 int blk_aio_multiwrite(BlockBackend *blk, BlockRequest *reqs, int num_reqs);
+int blk_aio_multiwrite_replay(BlockBackend *blk, BlockRequest *reqs,
+                              int num_reqs);
 int blk_ioctl(BlockBackend *blk, unsigned long int req, void *buf);
 BlockAIOCB *blk_aio_ioctl(BlockBackend *blk, unsigned long int req, void *buf,
                           BlockCompletionFunc *cb, void *opaque);
diff --git a/qemu-io-cmds.c b/qemu-io-cmds.c
index d94fb1e..b4e484c 100644
--- a/qemu-io-cmds.c
+++ b/qemu-io-cmds.c
@@ -583,7 +583,7 @@ static int do_aio_multiwrite(BlockDriverState *bs, BlockRequest* reqs,
         *total += reqs[i].qiov->size;
     }
 
-    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs);
+    ret = bdrv_aio_multiwrite(bs, reqs, num_reqs, false);
     if (ret < 0) {
         return ret;
     }
diff --git a/stubs/replay.c b/stubs/replay.c
index 6d6c2eb..507ea32 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -30,3 +30,8 @@ int runstate_is_running(void)
 void replay_add_bh_event(void *bh, uint64_t id)
 {
 }
+
+uint64_t replay_get_current_step(void)
+{
+    return 0;
+}
diff --git a/trace-events b/trace-events
index 6c3a400..7c9828c 100644
--- a/trace-events
+++ b/trace-events
@@ -64,7 +64,9 @@ bdrv_aio_multiwrite(void *mcb, int num_callbacks, int num_reqs) "mcb %p num_call
 bdrv_aio_discard(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_flush(void *bs, void *opaque) "bs %p opaque %p"
 bdrv_aio_readv(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_readv_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_writev(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
+bdrv_aio_writev_replay(void *bs, int64_t sector_num, int nb_sectors, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d opaque %p"
 bdrv_aio_write_zeroes(void *bs, int64_t sector_num, int nb_sectors, int flags, void *opaque) "bs %p sector_num %"PRId64" nb_sectors %d flags %#x opaque %p"
 bdrv_lock_medium(void *bs, bool locked) "bs %p locked %d"
 bdrv_co_readv(void *bs, int64_t sector_num, int nb_sector) "bs %p sector_num %"PRId64" nb_sectors %d"
diff --git a/util/iov.c b/util/iov.c
index 24566c8..b03f83f 100644
--- a/util/iov.c
+++ b/util/iov.c
@@ -257,6 +257,8 @@ void qemu_iovec_init(QEMUIOVector *qiov, int alloc_hint)
     qiov->niov = 0;
     qiov->nalloc = alloc_hint;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
 }
 
 void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
@@ -267,6 +269,8 @@ void qemu_iovec_init_external(QEMUIOVector *qiov, struct iovec *iov, int niov)
     qiov->niov = niov;
     qiov->nalloc = -1;
     qiov->size = 0;
+    qiov->replay = false;
+    qiov->replay_step = 0;
     for (i = 0; i < niov; i++)
         qiov->size += iov[i].iov_len;
 }

[Qemu-devel] [RFC PATCH v4 22/25] replay: thread pool

[Pavel Dovgalyuk]

This patch modifies thread pool to allow replaying asynchronous thread tasks
synchronously in replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block/raw-posix.c           |    6 ++++-
 block/raw-win32.c           |    4 +++-
 include/block/thread-pool.h |    4 +++-
 replay/replay-events.c      |   11 ++++++++++
 replay/replay-internal.h    |    3 ++-
 replay/replay.h             |    2 ++
 stubs/replay.c              |    4 ++++
 tests/test-thread-pool.c    |    7 ++++--
 thread-pool.c               |   49 ++++++++++++++++++++++++++++++-------------
 9 files changed, 67 insertions(+), 23 deletions(-)

diff --git a/block/raw-posix.c b/block/raw-posix.c
index 475cf74..5733f4f 100644
--- a/block/raw-posix.c
+++ b/block/raw-posix.c
@@ -1078,7 +1078,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, int fd,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 static BlockAIOCB *raw_aio_submit(BlockDriverState *bs,
@@ -1970,7 +1972,7 @@ static BlockAIOCB *hdev_aio_ioctl(BlockDriverState *bs,
     acb->aio_ioctl_buf = buf;
     acb->aio_ioctl_cmd = req;
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque, false, 0);
 }
 
 #elif defined(__FreeBSD__) || defined(__FreeBSD_kernel__)
diff --git a/block/raw-win32.c b/block/raw-win32.c
index 7b58881..39f2fa0 100644
--- a/block/raw-win32.c
+++ b/block/raw-win32.c
@@ -158,7 +158,9 @@ static BlockAIOCB *paio_submit(BlockDriverState *bs, HANDLE hfile,
 
     trace_paio_submit(acb, opaque, sector_num, nb_sectors, type);
     pool = aio_get_thread_pool(bdrv_get_aio_context(bs));
-    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque);
+    return thread_pool_submit_aio(pool, aio_worker, acb, cb, opaque,
+                                  qiov ? qiov->replay : false,
+                                  qiov ? qiov->replay_step : 0);
 }
 
 int qemu_ftruncate64(int fd, int64_t length)
diff --git a/include/block/thread-pool.h b/include/block/thread-pool.h
index 42eb5e8..801ac00 100644
--- a/include/block/thread-pool.h
+++ b/include/block/thread-pool.h
@@ -29,9 +29,11 @@ void thread_pool_free(ThreadPool *pool);
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque);
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step);
 int coroutine_fn thread_pool_submit_co(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg);
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg);
+void thread_pool_work(ThreadPool *pool, void *r);
 
 #endif
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 1aee0a4..4da5de0 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -12,6 +12,7 @@
 #include "qemu-common.h"
 #include "replay.h"
 #include "replay-internal.h"
+#include "block/thread-pool.h"
 
 typedef struct Event {
     int event_kind;
@@ -39,6 +40,9 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_BH:
         aio_bh_call(event->opaque);
         break;
+    case REPLAY_ASYNC_EVENT_THREAD:
+        thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -127,6 +131,11 @@ void replay_add_bh_event(void *bh, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_BH, bh, NULL, id);
 }
 
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -141,6 +150,7 @@ void replay_save_events(int opt)
             /* save event-specific data */
             switch (event->event_kind) {
             case REPLAY_ASYNC_EVENT_BH:
+            case REPLAY_ASYNC_EVENT_THREAD:
                 replay_put_qword(event->id);
                 break;
             }
@@ -170,6 +180,7 @@ void replay_read_events(int opt)
         /* Execute some events without searching them in the queue */
         switch (read_event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
+        case REPLAY_ASYNC_EVENT_THREAD:
             if (read_id == -1) {
                 read_id = replay_get_qword();
             }
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index ec36f44..3654aad 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -38,7 +38,8 @@
 /* Asynchronous events IDs */
 
 #define REPLAY_ASYNC_EVENT_BH          0
-#define REPLAY_ASYNC_COUNT             1
+#define REPLAY_ASYNC_EVENT_THREAD      1
+#define REPLAY_ASYNC_COUNT             2
 
 typedef struct ReplayState {
     /*! Cached clock values. */
diff --git a/replay/replay.h b/replay/replay.h
index ad8bdd3..78b6779 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -93,5 +93,7 @@ int replay_checkpoint(unsigned int checkpoint);
 void replay_disable_events(void);
 /*! Adds BH event to the queue */
 void replay_add_bh_event(void *bh, uint64_t id);
+/*! Adds thread event to the queue */
+void replay_add_thread_event(void *pool, void *req, uint64_t id);
 
 #endif
diff --git a/stubs/replay.c b/stubs/replay.c
index 507ea32..f2d4ed6 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -35,3 +35,7 @@ uint64_t replay_get_current_step(void)
 {
     return 0;
 }
+
+void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
+{
+}
diff --git a/tests/test-thread-pool.c b/tests/test-thread-pool.c
index 6a0b981..f32594c 100644
--- a/tests/test-thread-pool.c
+++ b/tests/test-thread-pool.c
@@ -56,7 +56,7 @@ static void test_submit_aio(void)
 {
     WorkerTestData data = { .n = 0, .ret = -EINPROGRESS };
     data.aiocb = thread_pool_submit_aio(pool, worker_cb, &data,
-                                        done_cb, &data);
+                                        done_cb, &data, false, 0);
 
     /* The callbacks are not called until after the first wait.  */
     active = 1;
@@ -120,7 +120,8 @@ static void test_submit_many(void)
     for (i = 0; i < 100; i++) {
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
-        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i]);
+        thread_pool_submit_aio(pool, worker_cb, &data[i], done_cb, &data[i],
+                               false, 0);
     }
 
     active = 100;
@@ -149,7 +150,7 @@ static void do_test_cancel(bool sync)
         data[i].n = 0;
         data[i].ret = -EINPROGRESS;
         data[i].aiocb = thread_pool_submit_aio(pool, long_cb, &data[i],
-                                               done_cb, &data[i]);
+                                               done_cb, &data[i], false, 0);
     }
 
     /* Starting the threads may be left to a bottom half.  Let it
diff --git a/thread-pool.c b/thread-pool.c
index e2cac8e..f5a4dac 100644
--- a/thread-pool.c
+++ b/thread-pool.c
@@ -22,6 +22,7 @@
 #include "trace.h"
 #include "block/thread-pool.h"
 #include "qemu/main-loop.h"
+#include "replay/replay.h"
 
 static void do_spawn_thread(ThreadPool *pool);
 
@@ -74,6 +75,27 @@ struct ThreadPool {
     bool stopping;
 };
 
+void thread_pool_work(ThreadPool *pool, void *r)
+{
+    ThreadPoolElement *req = (ThreadPoolElement *)r;
+    int ret;
+    if (replay_mode == REPLAY_MODE_NONE) {
+        qemu_mutex_unlock(&pool->lock);
+    }
+
+    ret = req->func(req->arg);
+    req->ret = ret;
+    /* Write ret before state.  */
+    smp_wmb();
+    req->state = THREAD_DONE;
+
+    if (replay_mode == REPLAY_MODE_NONE) {
+        qemu_mutex_lock(&pool->lock);
+    }
+
+    qemu_bh_schedule(pool->completion_bh);
+}
+
 static void *worker_thread(void *opaque)
 {
     ThreadPool *pool = opaque;
@@ -100,18 +122,12 @@ static void *worker_thread(void *opaque)
         req = QTAILQ_FIRST(&pool->request_list);
         QTAILQ_REMOVE(&pool->request_list, req, reqs);
         req->state = THREAD_ACTIVE;
-        qemu_mutex_unlock(&pool->lock);
-
-        ret = req->func(req->arg);
-
-        req->ret = ret;
-        /* Write ret before state.  */
-        smp_wmb();
-        req->state = THREAD_DONE;
-
-        qemu_mutex_lock(&pool->lock);
 
-        qemu_bh_schedule(pool->completion_bh);
+        if (replay_mode != REPLAY_MODE_NONE && req->common.replay) {
+            replay_add_thread_event(pool, req, req->common.replay_step);
+        } else {
+            thread_pool_work(pool, req);
+        }
     }
 
     pool->cur_threads--;
@@ -235,7 +251,8 @@ static const AIOCBInfo thread_pool_aiocb_info = {
 
 BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         ThreadPoolFunc *func, void *arg,
-        BlockCompletionFunc *cb, void *opaque)
+        BlockCompletionFunc *cb, void *opaque,
+        bool replay, uint64_t replay_step)
 {
     ThreadPoolElement *req;
 
@@ -244,6 +261,8 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
     req->arg = arg;
     req->state = THREAD_QUEUED;
     req->pool = pool;
+    req->common.replay = replay;
+    req->common.replay_step = replay_step;
 
     QLIST_INSERT_HEAD(&pool->head, req, all);
 
@@ -254,8 +273,8 @@ BlockAIOCB *thread_pool_submit_aio(ThreadPool *pool,
         spawn_thread(pool);
     }
     QTAILQ_INSERT_TAIL(&pool->request_list, req, reqs);
-    qemu_mutex_unlock(&pool->lock);
     qemu_sem_post(&pool->sem);
+    qemu_mutex_unlock(&pool->lock);
     return &req->common;
 }
 
@@ -277,14 +296,14 @@ int coroutine_fn thread_pool_submit_co(ThreadPool *pool, ThreadPoolFunc *func,
 {
     ThreadPoolCo tpc = { .co = qemu_coroutine_self(), .ret = -EINPROGRESS };
     assert(qemu_in_coroutine());
-    thread_pool_submit_aio(pool, func, arg, thread_pool_co_cb, &tpc);
+    thread_pool_submit_aio(pool, func, arg, thread_pool_co_cb, &tpc, false, 0);
     qemu_coroutine_yield();
     return tpc.ret;
 }
 
 void thread_pool_submit(ThreadPool *pool, ThreadPoolFunc *func, void *arg)
 {
-    thread_pool_submit_aio(pool, func, arg, NULL, NULL);
+    thread_pool_submit_aio(pool, func, arg, NULL, NULL, false, 0);
 }
 
 static void thread_pool_init_one(ThreadPool *pool, AioContext *ctx)

[Qemu-devel] [RFC PATCH v4 23/25] replay: initialization and deinitialization

[Pavel Dovgalyuk]

This patch introduces the functions for enabling the record/replay and for
freeing the resources when simulator closes.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 block.c                  |    2 -
 exec.c                   |    1 
 replay/replay-internal.h |    2 +
 replay/replay.c          |  134 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay.h          |   13 ++++
 stubs/replay.c           |    1 
 vl.c                     |    5 ++
 7 files changed, 157 insertions(+), 1 deletions(-)

diff --git a/block.c b/block.c
index 02c6a78..08b6b3f 100644
--- a/block.c
+++ b/block.c
@@ -1941,7 +1941,7 @@ void bdrv_drain_all(void)
             busy |= bs_busy;
         }
     }
-    if (replay_mode == REPLAY_MODE_PLAY) {
+    if (replay_mode == REPLAY_MODE_PLAY && replay_checkpoints) {
         /* Skip checkpoints from the log */
         while (replay_checkpoint(8)) {
             /* Nothing */
diff --git a/exec.c b/exec.c
index 759055d..2215984 100644
--- a/exec.c
+++ b/exec.c
@@ -794,6 +794,7 @@ void cpu_abort(CPUState *cpu, const char *fmt, ...)
     }
     va_end(ap2);
     va_end(ap);
+    replay_finish();
 #if defined(CONFIG_USER_ONLY)
     {
         struct sigaction act;
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 3654aad..52aef3f 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -34,6 +34,8 @@
 
 /* for checkpoint event */
 #define EVENT_CHECKPOINT            96
+/* end of log event */
+#define EVENT_END                   127
 
 /* Asynchronous events IDs */
 
diff --git a/replay/replay.c b/replay/replay.c
index 54e1c4e..63498b2 100755
--- a/replay/replay.c
+++ b/replay/replay.c
@@ -14,14 +14,23 @@
 #include "replay-internal.h"
 #include "qemu/timer.h"
 
+/* Current version of the replay mechanism.
+   Increase it when file format changes. */
+#define REPLAY_VERSION              0xe02002
+/* Size of replay log header */
+#define HEADER_SIZE                 (sizeof(uint32_t) + sizeof(uint64_t))
+
 ReplayMode replay_mode = REPLAY_MODE_NONE;
 /*! Stores current submode for PLAY mode */
 ReplaySubmode play_submode = REPLAY_SUBMODE_UNKNOWN;
 
+/* Name of replay file  */
+static char *replay_filename;
 /* Suffix for the disk images filenames */
 char *replay_image_suffix;
 
 ReplayState replay_state;
+bool replay_checkpoints;
 
 ReplaySubmode replay_get_play_submode(void)
 {
@@ -154,6 +163,10 @@ void replay_shutdown_request(void)
 /* Used checkpoints: 2 3 5 6 7 8 9 */
 int replay_checkpoint(unsigned int checkpoint)
 {
+    if (!replay_checkpoints) {
+        return 1;
+    }
+
     replay_save_instructions();
 
     if (replay_file) {
@@ -178,3 +191,124 @@ int replay_checkpoint(unsigned int checkpoint)
 
     return 1;
 }
+
+static void replay_enable(const char *fname, int mode)
+{
+    const char *fmode = NULL;
+    if (replay_file) {
+        fprintf(stderr,
+                "Replay: some record/replay operation is already started\n");
+        return;
+    }
+
+    switch (mode) {
+    case REPLAY_MODE_RECORD:
+        fmode = "wb";
+        break;
+    case REPLAY_MODE_PLAY:
+        fmode = "rb";
+        play_submode = REPLAY_SUBMODE_NORMAL;
+        break;
+    default:
+        fprintf(stderr, "Replay: internal error: invalid replay mode\n");
+        exit(1);
+    }
+
+    atexit(replay_finish);
+
+    replay_file = fopen(fname, fmode);
+    if (replay_file == NULL) {
+        fprintf(stderr, "Replay: open %s: %s\n", fname, strerror(errno));
+        exit(1);
+    }
+
+    replay_filename = g_strdup(fname);
+
+    replay_mode = mode;
+    replay_has_unread_data = 0;
+    replay_data_kind = -1;
+    replay_state.instructions_count = 0;
+    replay_state.current_step = 0;
+
+    /* skip file header for RECORD and check it for PLAY */
+    if (replay_mode == REPLAY_MODE_RECORD) {
+        fseek(replay_file, HEADER_SIZE, SEEK_SET);
+    } else if (replay_mode == REPLAY_MODE_PLAY) {
+        unsigned int version = replay_get_dword();
+        uint64_t offset = replay_get_qword();
+        if (version != REPLAY_VERSION) {
+            fprintf(stderr, "Replay: invalid input log file version\n");
+            exit(1);
+        }
+        /* go to the beginning */
+        fseek(replay_file, 12, SEEK_SET);
+    }
+
+    replay_init_events();
+}
+
+void replay_configure(QemuOpts *opts, int mode)
+{
+    const char *fname;
+
+    fname = qemu_opt_get(opts, "fname");
+    if (!fname) {
+        fprintf(stderr, "File name not specified for replay\n");
+        exit(1);
+    }
+
+    const char *suffix = qemu_opt_get(opts, "suffix");
+    if (suffix) {
+        replay_image_suffix = g_strdup(suffix);
+    } else {
+        replay_image_suffix = g_strdup("replay_qcow");
+    }
+
+    replay_enable(fname, mode);
+}
+
+void replay_init_timer(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    replay_checkpoints = true;
+    replay_enable_events();
+}
+
+void replay_finish(void)
+{
+    if (replay_mode == REPLAY_MODE_NONE) {
+        return;
+    }
+
+    replay_save_instructions();
+
+    /* finalize the file */
+    if (replay_file) {
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            uint64_t offset = 0;
+            /* write end event */
+            replay_put_event(EVENT_END);
+
+            /* write header */
+            fseek(replay_file, 0, SEEK_SET);
+            replay_put_dword(REPLAY_VERSION);
+            replay_put_qword(offset);
+        }
+
+        fclose(replay_file);
+        replay_file = NULL;
+    }
+    if (replay_filename) {
+        g_free(replay_filename);
+        replay_filename = NULL;
+    }
+    if (replay_image_suffix) {
+        g_free(replay_image_suffix);
+        replay_image_suffix = NULL;
+    }
+
+    replay_finish_events();
+}
diff --git a/replay/replay.h b/replay/replay.h
index 78b6779..c9f9ae4 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -17,6 +17,8 @@
 #include <time.h>
 #include "qapi-types.h"
 
+struct QemuOpts;
+
 /* replay clock kinds */
 /* rdtsc */
 #define REPLAY_CLOCK_REAL_TICKS 0
@@ -27,10 +29,21 @@
 
 extern ReplayMode replay_mode;
 extern char *replay_image_suffix;
+/*! Is set to true after finishing initialization */
+extern bool replay_checkpoints;
 
 /*! Returns replay play submode */
 ReplaySubmode replay_get_play_submode(void);
 
+/* Replay process control functions */
+
+/*! Enables recording or saving event log with specified parameters */
+void replay_configure(struct QemuOpts *opts, int mode);
+/*! Initializes timers used for snapshotting and enables events recording */
+void replay_init_timer(void);
+/*! Closes replay log file and frees other resources. */
+void replay_finish(void);
+
 /* Processing the instructions */
 
 /*! Returns number of executed instructions. */
diff --git a/stubs/replay.c b/stubs/replay.c
index f2d4ed6..4827ef1 100755
--- a/stubs/replay.c
+++ b/stubs/replay.c
@@ -2,6 +2,7 @@
 #include "sysemu/sysemu.h"
 
 ReplayMode replay_mode;
+bool replay_checkpoints;
 
 ReplaySubmode replay_get_play_submode(void)
 {
diff --git a/vl.c b/vl.c
index 717d67e..877a77c 100644
--- a/vl.c
+++ b/vl.c
@@ -4363,7 +4363,12 @@ int main(int argc, char **argv, char **envp)
         }
     }
 
+    replay_init_timer();
+
     main_loop();
+    if (replay_mode != REPLAY_MODE_NONE) {
+        replay_disable_events();
+    }
     bdrv_close_all();
     pause_all_vcpus();
     res_free();

[Qemu-devel] [RFC PATCH v4 24/25] replay: command line options

[Pavel Dovgalyuk]

This patch introduces command line options for enabling recording or replaying
virtual machine behavior. "-record" option starts recording of the execution
and saves it into the log, specified with "fname" parameter. "-replay" option
is intended for replaying previously saved log.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 cpus.c          |   15 ++++++++--
 qemu-options.hx |   27 +++++++++++++++++++
 vl.c            |   79 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 116 insertions(+), 5 deletions(-)

diff --git a/cpus.c b/cpus.c
index 6adff2b..de8276b 100644
--- a/cpus.c
+++ b/cpus.c
@@ -933,12 +933,21 @@ static void qemu_wait_io_event_common(CPUState *cpu)
 static void qemu_tcg_wait_io_event(void)
 {
     CPUState *cpu;
+    GMainContext *context = g_main_context_default();
 
-    while (all_cpu_threads_idle()) {
-       /* Start accounting real time to the virtual clock if the CPUs
-          are idle.  */
+    if (replay_mode == REPLAY_MODE_PLAY
+        && all_cpu_threads_idle() && first_cpu->halted) {
+        /* wakeup iothread when there is no code to execute in replay mode */
         qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+        g_main_context_wakeup(context);
         qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex);
+    } else {
+        while (all_cpu_threads_idle()) {
+            /* Start accounting real time to the virtual clock if the CPUs
+              are idle.  */
+            qemu_clock_warp(QEMU_CLOCK_VIRTUAL);
+            qemu_cond_wait(tcg_halt_cond, &qemu_global_mutex);
+        }
     }
 
     while (iothread_requesting_mutex) {
diff --git a/qemu-options.hx b/qemu-options.hx
index 22cf3b9..44ad8fc 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -3380,6 +3380,33 @@ Dump json-encoded vmstate information for current machine type to file
 in @var{file}
 ETEXI
 
+DEF("record", HAS_ARG, QEMU_OPTION_record,
+    "-record fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                writes replay file for latter replaying\n",
+    QEMU_ARCH_ALL)
+STEXI
+@item -record fname=@var{file}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Writes compact execution trace into @var{file}.
+Changes for disk images are written
+into separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
+DEF("replay", HAS_ARG, QEMU_OPTION_replay,
+    "-replay fname=<filename>[,suffix=<suffix>,snapshot=<on/off>]\n"
+    "                plays saved replay file\n", QEMU_ARCH_ALL)
+STEXI
+@item -replay fname=@var{filename}[,suffix=@var{suffix},snapshot=@var{snapshot}]
+Plays compact execution trace from @var{filename}.
+Changes for disk images and VM states are read
+from separate files with @var{suffix} added. If no @var{suffix} is
+specified, "replay_qcow" is used as suffix.
+If @var{snapshot} parameter is set as off, then original disk image will be
+modified. Default value is on.
+ETEXI
+
 HXCOMM This is the last statement. Insert new options before this line!
 STEXI
 @end table
diff --git a/vl.c b/vl.c
index 877a77c..fb5003e 100644
--- a/vl.c
+++ b/vl.c
@@ -548,6 +548,42 @@ static QemuOptsList qemu_icount_opts = {
     },
 };
 
+static QemuOptsList qemu_record_opts = {
+    .name = "record",
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_record_opts.head),
+    .desc = {
+        {
+            .name = "fname",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "suffix",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "snapshot",
+            .type = QEMU_OPT_BOOL,
+        },
+        { /* end of list */ }
+    },
+};
+
+static QemuOptsList qemu_replay_opts = {
+    .name = "replay",
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_replay_opts.head),
+    .desc = {
+        {
+            .name = "fname",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "suffix",
+            .type = QEMU_OPT_STRING,
+        },{
+            .name = "snapshot",
+            .type = QEMU_OPT_BOOL,
+        },
+        { /* end of list */ }
+    },
+};
+
 /**
  * Get machine options
  *
@@ -2711,7 +2747,9 @@ out:
 int main(int argc, char **argv, char **envp)
 {
     int i;
-    int snapshot, linux_boot;
+    int snapshot, linux_boot, replay_snapshot;
+    int not_compatible_replay_param = 0;
+    const char *icount_option = NULL;
     const char *initrd_filename;
     const char *kernel_filename, *kernel_cmdline;
     const char *boot_order;
@@ -2784,6 +2822,8 @@ int main(int argc, char **argv, char **envp)
     qemu_add_opts(&qemu_name_opts);
     qemu_add_opts(&qemu_numa_opts);
     qemu_add_opts(&qemu_icount_opts);
+    qemu_add_opts(&qemu_replay_opts);
+    qemu_add_opts(&qemu_record_opts);
 
     runstate_init();
 
@@ -2797,6 +2837,7 @@ int main(int argc, char **argv, char **envp)
     cpu_model = NULL;
     ram_size = default_ram_size;
     snapshot = 0;
+    replay_snapshot = 1;
     cyls = heads = secs = 0;
     translation = BIOS_ATA_TRANSLATION_AUTO;
 
@@ -2914,6 +2955,7 @@ int main(int argc, char **argv, char **envp)
                 break;
             case QEMU_OPTION_pflash:
                 drive_add(IF_PFLASH, -1, optarg, PFLASH_OPTS);
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_snapshot:
                 snapshot = 1;
@@ -3070,6 +3112,7 @@ int main(int argc, char **argv, char **envp)
 #endif
             case QEMU_OPTION_bt:
                 add_device_config(DEV_BT, optarg);
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_audio_help:
                 AUD_help ();
@@ -3284,6 +3327,7 @@ int main(int argc, char **argv, char **envp)
                 if (!opts) {
                     exit(1);
                 }
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_fsdev:
                 olist = qemu_find_opts("fsdev");
@@ -3412,6 +3456,7 @@ int main(int argc, char **argv, char **envp)
                 if (strncmp(optarg, "mon:", 4) == 0) {
                     default_monitor = 0;
                 }
+                not_compatible_replay_param++;
                 break;
             case QEMU_OPTION_debugcon:
                 add_device_config(DEV_DEBUGCON, optarg);
@@ -3532,6 +3577,7 @@ int main(int argc, char **argv, char **envp)
                 if (!qemu_opts_parse(qemu_find_opts("smp-opts"), optarg, 1)) {
                     exit(1);
                 }
+                not_compatible_replay_param++;
                 break;
 	    case QEMU_OPTION_vnc:
 #ifdef CONFIG_VNC
@@ -3770,6 +3816,24 @@ int main(int argc, char **argv, char **envp)
                     exit(1);
                 }
                 break;
+            case QEMU_OPTION_record:
+                opts = qemu_opts_parse(qemu_find_opts("record"), optarg, 0);
+                if (!opts) {
+                    fprintf(stderr, "Invalid record options: %s\n", optarg);
+                    exit(1);
+                }
+                replay_configure(opts, REPLAY_MODE_RECORD);
+                replay_snapshot = qemu_opt_get_bool(opts, "snapshot", 1);
+                break;
+            case QEMU_OPTION_replay:
+                opts = qemu_opts_parse(qemu_find_opts("replay"), optarg, 0);
+                if (!opts) {
+                    fprintf(stderr, "Invalid replay options: %s\n", optarg);
+                    exit(1);
+                }
+                replay_configure(opts, REPLAY_MODE_PLAY);
+                replay_snapshot = qemu_opt_get_bool(opts, "snapshot", 1);
+                break;
             default:
                 os_parse_cmd_args(popt->index, optarg);
             }
@@ -3784,6 +3848,12 @@ int main(int argc, char **argv, char **envp)
         exit(1);
     }
 
+    if (not_compatible_replay_param && (replay_mode != REPLAY_MODE_NONE)) {
+        fprintf(stderr, "options -smp, -pflash, -chardev, -bt, -parallel "
+                        "are not compatible with record/replay\n");
+        exit(1);
+    }
+
     if (qemu_opts_foreach(qemu_find_opts("sandbox"), parse_sandbox, NULL, 0)) {
         exit(1);
     }
@@ -4125,6 +4195,11 @@ int main(int argc, char **argv, char **envp)
         qemu_opts_del(icount_opts);
     }
 
+    if (replay_mode != REPLAY_MODE_NONE && !use_icount) {
+        fprintf(stderr, "Please enable icount to use record/replay\n");
+        exit(1);
+    }
+
     /* clean up network at qemu process termination */
     atexit(&net_cleanup);
 
@@ -4161,7 +4236,7 @@ int main(int argc, char **argv, char **envp)
     }
 
     /* open the virtual block devices */
-    if (snapshot)
+    if (snapshot || (replay_mode != REPLAY_MODE_NONE && replay_snapshot))
         qemu_opts_foreach(qemu_find_opts("drive"), drive_enable_snapshot, NULL, 0);
     if (qemu_opts_foreach(qemu_find_opts("drive"), drive_init_func,
                           &machine_class->block_default_type, 1) != 0) {

[Qemu-devel] [RFC PATCH v4 25/25] replay: recording of the user input

[Pavel Dovgalyuk]

This records user input (keyboard and mouse events) in record mode and replays
these input events in replay mode.

Signed-off-by: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
---
 include/ui/input.h       |    2 +
 replay/Makefile.objs     |    1 
 replay/replay-events.c   |   48 ++++++++++++++++++++
 replay/replay-input.c    |  108 ++++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-internal.h |   11 ++++-
 replay/replay.h          |    5 ++
 ui/input.c               |   80 ++++++++++++++++++++++++++--------
 7 files changed, 235 insertions(+), 20 deletions(-)
 create mode 100755 replay/replay-input.c

diff --git a/include/ui/input.h b/include/ui/input.h
index 5d5ac00..d06a12d 100644
--- a/include/ui/input.h
+++ b/include/ui/input.h
@@ -33,7 +33,9 @@ void qemu_input_handler_bind(QemuInputHandlerState *s,
                              const char *device_id, int head,
                              Error **errp);
 void qemu_input_event_send(QemuConsole *src, InputEvent *evt);
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt);
 void qemu_input_event_sync(void);
+void qemu_input_event_sync_impl(void);
 
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down);
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down);
diff --git a/replay/Makefile.objs b/replay/Makefile.objs
index 257c320..3936296 100755
--- a/replay/Makefile.objs
+++ b/replay/Makefile.objs
@@ -2,3 +2,4 @@ obj-y += replay.o
 obj-y += replay-internal.o
 obj-y += replay-events.o
 obj-y += replay-time.o
+obj-y += replay-input.o
diff --git a/replay/replay-events.c b/replay/replay-events.c
index 4da5de0..308186b 100755
--- a/replay/replay-events.c
+++ b/replay/replay-events.c
@@ -13,6 +13,7 @@
 #include "replay.h"
 #include "replay-internal.h"
 #include "block/thread-pool.h"
+#include "ui/input.h"
 
 typedef struct Event {
     int event_kind;
@@ -43,6 +44,16 @@ static void replay_run_event(Event *event)
     case REPLAY_ASYNC_EVENT_THREAD:
         thread_pool_work((ThreadPool *)event->opaque, event->opaque2);
         break;
+    case REPLAY_ASYNC_EVENT_INPUT:
+        qemu_input_event_send_impl(NULL, (InputEvent *)event->opaque);
+        /* Using local variables, when replaying. Do not free them. */
+        if (replay_mode == REPLAY_MODE_RECORD) {
+            qapi_free_InputEvent((InputEvent *)event->opaque);
+        }
+        break;
+    case REPLAY_ASYNC_EVENT_INPUT_SYNC:
+        qemu_input_event_sync_impl();
+        break;
     default:
         fprintf(stderr, "Replay: invalid async event ID (%d) in the queue\n",
                 event->event_kind);
@@ -136,6 +147,16 @@ void replay_add_thread_event(void *opaque, void *opaque2, uint64_t id)
     replay_add_event_internal(REPLAY_ASYNC_EVENT_THREAD, opaque, opaque2, id);
 }
 
+void replay_add_input_event(struct InputEvent *event)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_INPUT, event, NULL, 0);
+}
+
+void replay_add_input_sync_event(void)
+{
+    replay_add_event_internal(REPLAY_ASYNC_EVENT_INPUT_SYNC, NULL, NULL, 0);
+}
+
 void replay_save_events(int opt)
 {
     qemu_mutex_lock(&lock);
@@ -153,6 +174,9 @@ void replay_save_events(int opt)
             case REPLAY_ASYNC_EVENT_THREAD:
                 replay_put_qword(event->id);
                 break;
+            case REPLAY_ASYNC_EVENT_INPUT:
+                replay_save_input_event(event->opaque);
+                break;
             }
         }
 
@@ -178,6 +202,7 @@ void replay_read_events(int opt)
             break;
         }
         /* Execute some events without searching them in the queue */
+        Event e;
         switch (read_event_kind) {
         case REPLAY_ASYNC_EVENT_BH:
         case REPLAY_ASYNC_EVENT_THREAD:
@@ -185,6 +210,29 @@ void replay_read_events(int opt)
                 read_id = replay_get_qword();
             }
             break;
+        case REPLAY_ASYNC_EVENT_INPUT:
+            e.event_kind = read_event_kind;
+            e.opaque = replay_read_input_event();
+
+            replay_run_event(&e);
+
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_opt = -1;
+            replay_fetch_data_kind();
+            /* continue with the next event */
+            continue;
+        case REPLAY_ASYNC_EVENT_INPUT_SYNC:
+            e.event_kind = read_event_kind;
+            e.opaque = 0;
+            replay_run_event(&e);
+
+            replay_has_unread_data = 0;
+            read_event_kind = -1;
+            read_opt = -1;
+            replay_fetch_data_kind();
+            /* continue with the next event */
+            continue;
         default:
             fprintf(stderr, "Unknown ID %d of replay event\n", read_event_kind);
             exit(1);
diff --git a/replay/replay-input.c b/replay/replay-input.c
new file mode 100755
index 0000000..f5d1482
--- /dev/null
+++ b/replay/replay-input.c
@@ -0,0 +1,108 @@
+/*
+ * replay-input.c
+ *
+ * Copyright (c) 2010-2014 Institute for System Programming
+ *                         of the Russian Academy of Sciences.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *
+ */
+
+#include "qemu-common.h"
+#include "replay.h"
+#include "replay-internal.h"
+#include "ui/input.h"
+
+void replay_save_input_event(InputEvent *evt)
+{
+    replay_put_dword(evt->kind);
+
+    switch (evt->kind) {
+    case INPUT_EVENT_KIND_KEY:
+        replay_put_dword(evt->key->key->kind);
+
+        switch (evt->key->key->kind) {
+        case KEY_VALUE_KIND_NUMBER:
+            replay_put_qword(evt->key->key->number);
+            replay_put_byte(evt->key->down);
+            break;
+        case KEY_VALUE_KIND_QCODE:
+            replay_put_dword(evt->key->key->qcode);
+            replay_put_byte(evt->key->down);
+            break;
+        case KEY_VALUE_KIND_MAX:
+            /* keep gcc happy */
+            break;
+        }
+        break;
+    case INPUT_EVENT_KIND_BTN:
+        replay_put_dword(evt->btn->button);
+        replay_put_byte(evt->btn->down);
+        break;
+    case INPUT_EVENT_KIND_REL:
+        replay_put_dword(evt->rel->axis);
+        replay_put_qword(evt->rel->value);
+        break;
+    case INPUT_EVENT_KIND_ABS:
+        replay_put_dword(evt->abs->axis);
+        replay_put_qword(evt->abs->value);
+        break;
+    case INPUT_EVENT_KIND_MAX:
+        /* keep gcc happy */
+        break;
+    }
+}
+
+InputEvent *replay_read_input_event(void)
+{
+    static InputEvent evt;
+    static KeyValue keyValue;
+    static InputKeyEvent key;
+    key.key = &keyValue;
+    static InputBtnEvent btn;
+    static InputMoveEvent rel;
+    static InputMoveEvent abs;
+
+    evt.kind = replay_get_dword();
+    switch (evt.kind) {
+    case INPUT_EVENT_KIND_KEY:
+        evt.key = &key;
+        evt.key->key->kind = replay_get_dword();
+
+        switch (evt.key->key->kind) {
+        case KEY_VALUE_KIND_NUMBER:
+            evt.key->key->number = replay_get_qword();
+            evt.key->down = replay_get_byte();
+            break;
+        case KEY_VALUE_KIND_QCODE:
+            evt.key->key->qcode = (QKeyCode)replay_get_dword();
+            evt.key->down = replay_get_byte();
+            break;
+        case KEY_VALUE_KIND_MAX:
+            /* keep gcc happy */
+            break;
+        }
+        break;
+    case INPUT_EVENT_KIND_BTN:
+        evt.btn = &btn;
+        evt.btn->button = (InputButton)replay_get_dword();
+        evt.btn->down = replay_get_byte();
+        break;
+    case INPUT_EVENT_KIND_REL:
+        evt.rel = &rel;
+        evt.rel->axis = (InputAxis)replay_get_dword();
+        evt.rel->value = replay_get_qword();
+        break;
+    case INPUT_EVENT_KIND_ABS:
+        evt.abs = &abs;
+        evt.abs->axis = (InputAxis)replay_get_dword();
+        evt.abs->value = replay_get_qword();
+        break;
+    case INPUT_EVENT_KIND_MAX:
+        /* keep gcc happy */
+        break;
+    }
+
+    return &evt;
+}
diff --git a/replay/replay-internal.h b/replay/replay-internal.h
index 52aef3f..4dc4db5 100755
--- a/replay/replay-internal.h
+++ b/replay/replay-internal.h
@@ -41,7 +41,9 @@
 
 #define REPLAY_ASYNC_EVENT_BH          0
 #define REPLAY_ASYNC_EVENT_THREAD      1
-#define REPLAY_ASYNC_COUNT             2
+#define REPLAY_ASYNC_EVENT_INPUT       2
+#define REPLAY_ASYNC_EVENT_INPUT_SYNC  3
+#define REPLAY_ASYNC_COUNT             4
 
 typedef struct ReplayState {
     /*! Cached clock values. */
@@ -120,4 +122,11 @@ void replay_read_events(int opt);
 /*! Adds specified async event to the queue */
 void replay_add_event(int event_id, void *opaque);
 
+/* Input events */
+
+/*! Saves input event to the log */
+void replay_save_input_event(InputEvent *evt);
+/*! Reads input event from the log */
+InputEvent *replay_read_input_event(void);
+
 #endif
diff --git a/replay/replay.h b/replay/replay.h
index c9f9ae4..7e5c2e1 100755
--- a/replay/replay.h
+++ b/replay/replay.h
@@ -18,6 +18,7 @@
 #include "qapi-types.h"
 
 struct QemuOpts;
+struct InputEvent;
 
 /* replay clock kinds */
 /* rdtsc */
@@ -108,5 +109,9 @@ void replay_disable_events(void);
 void replay_add_bh_event(void *bh, uint64_t id);
 /*! Adds thread event to the queue */
 void replay_add_thread_event(void *pool, void *req, uint64_t id);
+/*! Adds input event to the queue */
+void replay_add_input_event(struct InputEvent *event);
+/*! Adds input sync event to the queue */
+void replay_add_input_sync_event(void);
 
 #endif
diff --git a/ui/input.c b/ui/input.c
index 002831e..4e49fe5 100644
--- a/ui/input.c
+++ b/ui/input.c
@@ -5,6 +5,7 @@
 #include "trace.h"
 #include "ui/input.h"
 #include "ui/console.h"
+#include "replay/replay.h"
 
 struct QemuInputHandlerState {
     DeviceState       *dev;
@@ -295,14 +296,10 @@ static void qemu_input_queue_sync(struct QemuInputEventQueueHead *queue)
     QTAILQ_INSERT_TAIL(queue, item, node);
 }
 
-void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
+void qemu_input_event_send_impl(QemuConsole *src, InputEvent *evt)
 {
     QemuInputHandlerState *s;
 
-    if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
-        return;
-    }
-
     qemu_input_event_trace(src, evt);
 
     /* pre processing */
@@ -319,14 +316,25 @@ void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
     s->events++;
 }
 
-void qemu_input_event_sync(void)
+void qemu_input_event_send(QemuConsole *src, InputEvent *evt)
 {
-    QemuInputHandlerState *s;
-
     if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
         return;
     }
 
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_event(evt);
+    } else {
+        qemu_input_event_send_impl(src, evt);
+    }
+}
+
+void qemu_input_event_sync_impl(void)
+{
+    QemuInputHandlerState *s;
+
     trace_input_event_sync();
 
     QTAILQ_FOREACH(s, &handlers, node) {
@@ -340,6 +348,21 @@ void qemu_input_event_sync(void)
     }
 }
 
+void qemu_input_event_sync(void)
+{
+    if (!runstate_is_running() && !runstate_check(RUN_STATE_SUSPENDED)) {
+        return;
+    }
+
+    if (replay_mode == REPLAY_MODE_PLAY) {
+        /* Nothing */
+    } else if (replay_mode == REPLAY_MODE_RECORD) {
+        replay_add_input_sync_event();
+    } else {
+        qemu_input_event_sync_impl();
+    }
+}
+
 InputEvent *qemu_input_event_new_key(KeyValue *key, bool down)
 {
     InputEvent *evt = g_new0(InputEvent, 1);
@@ -353,14 +376,23 @@ InputEvent *qemu_input_event_new_key(KeyValue *key, bool down)
 void qemu_input_event_send_key(QemuConsole *src, KeyValue *key, bool down)
 {
     InputEvent *evt;
-    evt = qemu_input_event_new_key(key, down);
-    if (QTAILQ_EMPTY(&kbd_queue)) {
-        qemu_input_event_send(src, evt);
-        qemu_input_event_sync();
-        qapi_free_InputEvent(evt);
-    } else {
-        qemu_input_queue_event(&kbd_queue, src, evt);
-        qemu_input_queue_sync(&kbd_queue);
+    if (replay_mode != REPLAY_MODE_PLAY) {
+        evt = qemu_input_event_new_key(key, down);
+        if (QTAILQ_EMPTY(&kbd_queue)) {
+            qemu_input_event_send(src, evt);
+            qemu_input_event_sync();
+            if (replay_mode != REPLAY_MODE_RECORD) {
+                qapi_free_InputEvent(evt);
+            }
+        } else {
+            if (replay_mode != REPLAY_MODE_NONE) {
+                fprintf(stderr, "Input queue is not supported "
+                                "in record/replay mode\n");
+                exit(1);
+            }
+            qemu_input_queue_event(&kbd_queue, src, evt);
+            qemu_input_queue_sync(&kbd_queue);
+        }
     }
 }
 
@@ -386,6 +418,10 @@ void qemu_input_event_send_key_delay(uint32_t delay_ms)
         kbd_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, qemu_input_queue_process,
                                  &kbd_queue);
     }
+    if (replay_mode != REPLAY_MODE_NONE) {
+        fprintf(stderr, "Input queue is not supported in record/replay mode\n");
+        exit(1);
+    }
     qemu_input_queue_delay(&kbd_queue, kbd_timer,
                            delay_ms ? delay_ms : kbd_default_delay_ms);
 }
@@ -405,7 +441,9 @@ void qemu_input_queue_btn(QemuConsole *src, InputButton btn, bool down)
     InputEvent *evt;
     evt = qemu_input_event_new_btn(btn, down);
     qemu_input_event_send(src, evt);
-    qapi_free_InputEvent(evt);
+    if (replay_mode != REPLAY_MODE_RECORD) {
+        qapi_free_InputEvent(evt);
+    }
 }
 
 void qemu_input_update_buttons(QemuConsole *src, uint32_t *button_map,
@@ -458,7 +496,9 @@ void qemu_input_queue_rel(QemuConsole *src, InputAxis axis, int value)
     InputEvent *evt;
     evt = qemu_input_event_new_move(INPUT_EVENT_KIND_REL, axis, value);
     qemu_input_event_send(src, evt);
-    qapi_free_InputEvent(evt);
+    if (replay_mode != REPLAY_MODE_RECORD) {
+        qapi_free_InputEvent(evt);
+    }
 }
 
 void qemu_input_queue_abs(QemuConsole *src, InputAxis axis, int value, int size)
@@ -467,7 +507,9 @@ void qemu_input_queue_abs(QemuConsole *src, InputAxis axis, int value, int size)
     int scaled = qemu_input_scale_axis(value, size, INPUT_EVENT_ABS_SIZE);
     evt = qemu_input_event_new_move(INPUT_EVENT_KIND_ABS, axis, scaled);
     qemu_input_event_send(src, evt);
-    qapi_free_InputEvent(evt);
+    if (replay_mode != REPLAY_MODE_RECORD) {
+        qapi_free_InputEvent(evt);
+    }
 }
 
 void qemu_input_check_mode_change(void)