Re: [Qemu-devel] [RFC PATCH v9 00/23] Deterministic replay core

[Paolo Bonzini <pbonzini@redhat.com>]

On 18/02/2015 12:55, Pavel Dovgalyuk wrote:
> This set of patches is related to the reverse execution and deterministic 
> replay of qemu execution. This implementation of deterministic replay can 
> be used for deterministic debugging of guest code through gdb remote
> interface.
> 
> These patches include only core function of the replay,
> excluding the support for replaying serial, audio, network, and USB devices'
> operations. Reverse debugging and monitor commands were also excluded to
> be submitted later as separate patches.
> 
> Execution recording writes non-deterministic events log, which can be later 
> used for replaying the execution anywhere and for unlimited number of times. 
> It also supports checkpointing for faster rewinding during reverse debugging. 
> Execution replaying reads the log and replays all non-deterministic events 
> including external input, hardware clocks, and interrupts.
> 
> Deterministic replay has the following features:
>  * Deterministically replays whole system execution and all contents of the memory,
>    state of the hadrware devices, clocks, and screen of the VM.
>  * Writes execution log into the file for latter replaying for multiple times 
>    on different machines.
>  * Supports i386, x86_64, and ARM hardware platforms.
>  * Performs deterministic replay of all operations with keyboard and mouse
>    input devices.
>  * Supports auto-checkpointing for convenient reverse debugging.
> 
> Usage of the record/replay:
>  * First, record the execution, by adding the following string to the command line:
>    '-icount shift=7,rr=record,rrfile=replay.bin -net none'. 
>    Block devices' images are not actually changed in the recording mode, 
>    because all of the changes are written to the temporary overlay file.
>  * Then you can replay it for the multiple times by using another command
>    line option: '-icount shift=7,rr=replay,rrfile=replay.bin -net none'
>  * '-net none' option should also be specified if network replay patches
>    are not applied.
> 
> Paper with short description of deterministic replay implementation:
> http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-abs.html
> 
> Modifications of qemu include:
>  * wrappers for clock and time functions to save their return values in the log
>  * saving different asynchronous events (e.g. system shutdown) into the log
>  * synchronization of the bottom halves execution
>  * synchronization of the threads from thread pool
>  * recording/replaying user input (mouse and keyboard)
>  * adding internal events for cpu and io synchronization
> 
> v9 changes:
>  * Replaced fwrite/fread with putc/getc (as suggested by Paolo Bonzini)
>  * Stopping virtual machine in case of replay file end (as suggested by Paolo Bonzini)
>  * Removed one of the replay mutexes (as suggested by Paolo Bonzini)
>  * Fixed RCU queue for bottom halves (as suggested by Paolo Bonzini)
>  * Updated command line options' names (as suggested by Paolo Bonzini)
>  * Added design document for record/replay (as suggested by Paolo Bonzini)
>  * Simplified checkpoints for the timers
>  * Added cloning InputEvent objects for replay (as suggested by Paolo Bonzini)
>  * Added replay blockers instead of checking the command line (as suggested by Paolo Bonzini)
>  * Some functions renaming and extracting.

I haven't yet reviewed patch 23 completely, and I have to think (a lot
:)) more about block devices.  In the meanwhile I understand the replay
code much better so I had some suggestions.

In general, the handling of replay_has_unread_data / replay_data_kind is
a bit messy.  It would be nice if you could call
replay_fetch_data_kind() only when replay_has_unread_data == 0.  Or,
even, remove replay_has_unread_data altogether: just call
replay_fetch_data_kind() when you'd set it to zero.  That would simplify
a lot the code for readers.

Paolo

> v8 changes:
>  * Simplified processing of the shutdown event (as suggested by Paolo Bonzini)
>  * Replaced stack of bottom halves in AIO context with QSIMPLEQ (as suggested by Paolo Bonzini)
>  * Moved replay_submode out of the series (as suggested by Paolo Bonzini)
>  * Moved suffix option out of the series
>  * Converted some of the defines into enums (as suggested by Paolo Bonzini)
>  * Encapsulated save_tm/read_tm calls into the single function (as suggested by Paolo Bonzini)
>  * Moved record/replay options to icount group (as suggested by Paolo Bonzini)
>  * Updated mutex protection for the events queue (as suggested by Paolo Bonzini)
>  * Added mutex to protect replay log file (as suggested by Paolo Bonzini)
>  * Minor cleanups
> 
> v7 changes:
>  * Removed patches that were applied to upstream.
> 
> v6 changes:
>  * Fixed replay stub return value (as suggested by Eric Blake)
>  * Fixed icount warping.
>  * Virtual rt clock now uses cpu_get_clock() (as suggested by Paolo Bonzini)
>  * Replated get_clock_realtime and get_clock calls with qemu clock requests (as suggested by Paolo Bonzini)
>  * Modified can_do_io logic to allow requesting icount from cpu_exec function (as suggested by Paolo Bonzini)
>  * Removed applied patches.
> 
> v5 changes:
>  * Minor changes.
>  * Used fixed-width integer types for read/write functions (as suggested by Alex Bennee)
>  * Moved savevm-related code out of the core.
>  * Added new traced clock for deterministic virtual clock warping (as suggested by Paolo Bonzini)
>  * Fixed exception_index reset for user mode (as suggested by Paolo Bonzini)
>  * Adopted Paolo's icount patches
>  * Fixed hardware interrupts replaying
> 
> v4 changes:
>  * Updated block drivers to support new bdrv_open interface.
>  * Moved migration patches into separate series (as suggested by Paolo Bonzini)
>  * Fixed a bug in replay_break operation.
>  * Fixed rtl8139 migration for replay.
>  * Fixed 'period' parameter processing for record mode.
>  * Fixed bug in 'reverse-stepi' implementation.
>  * Fixed replay without making any snapshots (even the starting one).
>  * Moved core replay patches into the separate series.
>  * Fixed reverse step and reverse continue support.
>  * Fixed several bugs in icount subsystem.
>  * Reusing native qemu icount for replay instructions counting.
>  * Separated core patches into their own series.
> 
> v3 changes:
>  * Fixed bug with replay of the aio write operations.
>  * Added virtual clock based on replay icount.
>  * Removed duplicated saving of interrupt_request CPU field.
>  * Fixed some coding style issues.
>  * Renamed QMP commands for controlling reverse execution (as suggested by Eric Blake)
>  * Replay mode and submode implemented as QAPI enumerations (as suggested by Eric Blake)
>  * Added description and example for replay-info command (as suggested by Eric Blake)
>  * Added information about the current breakpoint to the output of replay-info (as suggested by Eric Blake)
>  * Updated version id for HPET vmstate (as suggested by Paolo Bonzini)
>  * Removed static fields from parallel vmstate (as suggested by Paolo Bonzini)
>  * New vmstate fields for mc146818rtc, pckbd, kvmapic, serial, fdc, rtl8139 moved to subsection (as suggested by Paolo Bonzini)
>  * Disabled textmode cursor blinking, when virtual machine is stopped (as suggested by Paolo Bonzini)
>  * Extracted saving of exception_index to separate patch (as suggested by Paolo Bonzini)
> 
> v2 changes:
>  * Patches are split to be reviewable and bisectable (as suggested by Kirill Batuzov)
>  * Added QMP versions of replay commands (as suggested by Eric Blake)
>  * Removed some optional features of replay to make patches cleaner
>  * Minor changes and code cleanup were made
> 
> ---
> 
> Pavel Dovgalyuk (23):
>       i386: partial revert of interrupt poll fix
>       replay: global variables and function stubs
>       sysemu: system functions for replay
>       replay: internal functions for replay log
>       replay: introduce mutex to protect the replay log
>       replay: introduce icount event
>       cpu-exec: allow temporary disabling icount
>       cpu: replay instructions sequence
>       replay: interrupts and exceptions
>       replay: asynchronous events infrastructure
>       replay: recording and replaying clock ticks
>       timer: replace time() with QEMU_CLOCK_HOST
>       replay: shutdown event
>       replay: checkpoints
>       aio: replace stack of bottom halves with queue
>       replay: bottom halves
>       replay: replay aio requests
>       replay: thread pool
>       typedef: add typedef for QemuOpts
>       replay: initialization and deinitialization
>       replay: replay blockers for devices
>       replay: command line options
>       replay: recording of the user input
> 
> 
>  Makefile.target                |    1 
>  async.c                        |   49 ++++--
>  block.c                        |   93 ++++++++++-
>  block/block-backend.c          |   30 +++
>  block/qcow2.c                  |    4 
>  block/raw-posix.c              |    6 -
>  block/raw-win32.c              |    4 
>  cpu-exec.c                     |   59 +++++--
>  cpus.c                         |   51 ++++--
>  dma-helpers.c                  |   10 +
>  docs/replay.txt                |  161 ++++++++++++++++++
>  exec.c                         |    1 
>  hw/block/virtio-blk.c          |   10 +
>  hw/bt/hci.c                    |    7 +
>  hw/ide/ahci.c                  |    4 
>  hw/ide/atapi.c                 |   10 +
>  hw/ide/core.c                  |   18 +-
>  hw/timer/arm_timer.c           |    2 
>  hw/usb/hcd-uhci.c              |    2 
>  include/block/aio.h            |   22 ++-
>  include/block/block.h          |   15 ++
>  include/block/thread-pool.h    |    4 
>  include/exec/exec-all.h        |    2 
>  include/qapi/qmp/qerror.h      |    3 
>  include/qemu-common.h          |    2 
>  include/qemu/main-loop.h       |    1 
>  include/qemu/option.h          |    5 -
>  include/qemu/queue.h           |    7 +
>  include/qemu/typedefs.h        |    3 
>  include/qom/cpu.h              |   10 +
>  include/sysemu/block-backend.h |   10 +
>  include/sysemu/cpus.h          |    1 
>  include/ui/input.h             |    2 
>  main-loop.c                    |   11 +
>  qapi-schema.json               |   18 ++
>  qemu-io-cmds.c                 |    2 
>  qemu-options.hx                |    8 +
>  qemu-timer.c                   |   45 ++++-
>  replay/Makefile.objs           |    5 +
>  replay/replay-events.c         |  287 +++++++++++++++++++++++++++++++++
>  replay/replay-input.c          |  159 ++++++++++++++++++
>  replay/replay-internal.c       |  191 ++++++++++++++++++++++
>  replay/replay-internal.h       |  144 ++++++++++++++++
>  replay/replay-time.c           |   84 ++++++++++
>  replay/replay.c                |  351 ++++++++++++++++++++++++++++++++++++++++
>  replay/replay.h                |  120 ++++++++++++++
>  stubs/Makefile.objs            |    1 
>  stubs/replay.c                 |   41 +++++
>  target-i386/cpu.c              |   10 -
>  target-i386/seg_helper.c       |    7 +
>  tests/test-thread-pool.c       |    7 -
>  thread-pool.c                  |   49 ++++--
>  trace-events                   |    2 
>  translate-all.c                |   10 +
>  ui/input.c                     |   27 ++-
>  util/iov.c                     |    4 
>  vl.c                           |   47 +++++
>  57 files changed, 2100 insertions(+), 139 deletions(-)
>  create mode 100755 docs/replay.txt
>  create mode 100755 replay/Makefile.objs
>  create mode 100755 replay/replay-events.c
>  create mode 100755 replay/replay-input.c
>  create mode 100755 replay/replay-internal.c
>  create mode 100755 replay/replay-internal.h
>  create mode 100755 replay/replay-time.c
>  create mode 100755 replay/replay.c
>  create mode 100755 replay/replay.h
>  create mode 100755 stubs/replay.c
>