From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:60185) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YWUvt-0001Bj-AD for qemu-devel@nongnu.org; Fri, 13 Mar 2015 15:07:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YWUvp-0004p0-4w for qemu-devel@nongnu.org; Fri, 13 Mar 2015 15:07:41 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55792) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YWUvo-0004od-Pw for qemu-devel@nongnu.org; Fri, 13 Mar 2015 15:07:37 -0400 Message-ID: <55033569.20205@redhat.com> Date: Fri, 13 Mar 2015 13:07:21 -0600 From: Eric Blake MIME-Version: 1.0 References: <20150227130939.11912.50660.stgit@PASHA-ISP> <20150227130958.11912.56622.stgit@PASHA-ISP> In-Reply-To: <20150227130958.11912.56622.stgit@PASHA-ISP> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="61DIDrxIsA5n9F95NkSpbNr4jLEJp643x" Subject: Re: [Qemu-devel] [RFC PATCH v10 02/24] replay: global variables and function stubs List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Pavel Dovgalyuk , qemu-devel@nongnu.org Cc: peter.maydell@linaro.org, peter.crosthwaite@xilinx.com, mark.burton@greensocs.com, real@ispras.ru, batuzovk@ispras.ru, maria.klimushenkova@ispras.ru, pbonzini@redhat.com, alex.bennee@linaro.org, afaerber@suse.de, fred.konrad@greensocs.com This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --61DIDrxIsA5n9F95NkSpbNr4jLEJp643x Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 02/27/2015 06:09 AM, Pavel Dovgalyuk wrote: > This patch adds global variables, defines, functions declarations, s/functions/function/ > and function stubs for deterministic VM replay used by external modules= =2E >=20 > Reviewed-by: Paolo Bonzini > Reviewed-by: Eric Blake >=20 > Signed-off-by: Pavel Dovgalyuk > --- > +++ b/docs/replay.txt > @@ -0,0 +1,161 @@ > +Record/replay > +------------- It might be nice to include an explicit copyright and license statement at the top of this file (without one, you have inherited the top-level default of GPLv2+). > + > +Record/replay functions are used for the reverse execution and determi= nistic=20 > +replay of qemu execution. This implementation of deterministic replay = can=20 > +be used for deterministic debugging of guest code through gdb remote s/through/through a/ > +interface. > + > +Execution recording writes non-deterministic events log, which can be = later=20 s/writes/writes a/ > +used for replaying the execution anywhere and for unlimited number of = times.=20 > +It also supports checkpointing for faster rewinding during reverse deb= ugging.=20 > +Execution replaying reads the log and replays all non-deterministic ev= ents=20 > +including external input, hardware clocks, and interrupts. > + > +Deterministic replay has the following features: > + * Deterministically replays whole system execution and all contents o= f=20 > + the memory, state of the hadrware devices, clocks, and screen of th= e VM. s/hadrware/hardware/ > + * Writes execution log into the file for latter replaying for multipl= e times=20 s/latter/later/ > + on different machines. > + * Supports i386, x86_64, and ARM hardware platforms. > + * Performs deterministic replay of all operations with keyboard and m= ouse > + input devices. > + > +Usage of the record/replay: > + * First, record the execution, by adding the following string to the = command line: s/string/arguments/ (it is not 1 string argument with 3 embedded spaces, but four separate arguments) > + '-icount shift=3D7,rr=3Drecord,rrfile=3Dreplay.bin -net none'.=20 > + Block devices' images are not actually changed in the recording mod= e,=20 > + because all of the changes are written to the temporary overlay fil= e. > + * Then you can replay it for the multiple times by using another comm= and s/for the multiple times// > + line option: '-icount shift=3D7,rr=3Dreplay,rrfile=3Dreplay.bin -ne= t none' > + * '-net none' option should also be specified if network replay patch= es > + are not applied. > + > +Paper with short description of deterministic replay implementation: > +http://www.computer.org/csdl/proceedings/csmr/2012/4666/00/4666a553-ab= s.html > + > +Modifications of qemu include: > + * wrappers for clock and time functions to save their return values i= n the log > + * saving different asynchronous events (e.g. system shutdown) into th= e log > + * synchronization of the bottom halves execution > + * synchronization of the threads from thread pool > + * recording/replaying user input (mouse and keyboard) > + * adding internal checkpoints for cpu and io synchronization > + > +Non-deterministic events > +------------------------ > + > +Our record/replay system is based on saving and replaying non-determin= istic=20 > +events (e.g. keyboard input) and simulating deterministic ones (e.g. r= eading=20 > +from HDD or memory of the VM). Saving only non-deterministic events ma= kes=20 > +log file smaller, simulation faster, and allows using reverse debuggin= g even=20 > +for realtime applications.=20 > + > +The following non-deterministic data from peripheral devices is saved = into=20 > +the log: mouse and keyboard input, network packets, audio controller i= nput,=20 > +USB packets, serial port input, and hardware clocks (they are non-dete= rministic=20 > +too, because their values are taken from the host machine). Inputs fro= m=20 > +simulated hardware, memory of VM, software interrupts, and execution o= f=20 > +instructions are not saved into the log, because they are deterministi= c and=20 > +can be replayed by simulating the behavior of virtual machine starting= from=20 > +initial state. > + > +We had to solve three tasks to implement deterministic replay: recordi= ng=20 > +non-deterministic events, replaying non-deterministic events, and chec= king=20 > +that there is no divergence between record and replay modes. > + > +We changed several parts of QEMU to make event log recording and repla= ying. > +Devices' models that have non-deterministic input from external device= s were=20 > +changed to write every external event into the execution log immediate= ly.=20 > +E.g. network packets are written into the log when they arrive into th= e virtual=20 > +network adapter. > + > +All non-deterministic events are coming from these devices. But to=20 > +replay them we need to know at which moments they occur. We specify=20 > +these moments by counting the number of instructions executed between = > +every pair of consecutive events. > + > +Instructions counting s/Instructions/Instruction/ > +--------------------- > + > +QEMU should work in icount mode to use record/replay feature. icount w= as > +designed to allow deterministic execution in absense of external input= s s/absense/absence/ > +of the virtual machine. We also use icount to control the occurence of= the s/occurence/occurrence/ > +non-deterministic events. Number of instruction passed from the last e= vent s/Number/The number/ s/instruction passed/instructions elapsed/ > +is written to the log while recording the execution. In replay mode we= > +can predict when to inject that event using the instructions counter. s/instructions/instruction/ > + > +Timers > +------ > + > +Timers are used to execute callbacks from different subsystems of QEMU= > +at the specified moments of time. There are several kinds of timers:=20 > + * Real time clock. Based on host time and used only for callbacks tha= t=20 > + do not change the virtual machine state. For this reason real time > + clock and timers does not affect deterministic replay at all; s/;/./ > + * Virtual clock. These timers run only during the emulation. In icoun= t > + mode virtual clock value is calculated using executed instructions = counter. > + That is why it is completely deterministic and does not have to be = recorded; > + * Host clock. This clock is used by device models that simulate real = time > + sources (e.g. real time clock chip). Host clock is the one of the s= ources > + of non-determinism. Host clock read operations should be logged to > + make the execution deterministic. > + * Real time clock for icount. This clock is similar to real time cloc= k but > + it is used only for increasing virtual clock while virtual machine = is > + sleeping. Due to its nature it is also non-deterministic as the hos= t clock > + and has to be logged too. > + > +Checkpoints > +----------- > + > +Replaying of the execution of virtual machine is bound by sources of > +non-determinism. These are inputs from clock and peripheral devices, > +and QEMU thread scheduling. Thread scheduling affect on processing eve= nts > +from timers, asynchronous input-output, and bottom halves. > + > +Invocations of timers are coupled with clock reads and changing the st= ate=20 > +of the virtual machine. Reads produce non-deterministic data taken fro= m > +host clock. And VM state changes should preserve their order. Their re= lative > +order in replay mode must replicate the order of callbacks in record m= ode. > +To preserve this order we use checkpoints. When specific clock is proc= essed s/When/When a/ > +in record mode we save to the log special ''checkpoint`` event. s/``/''/ or even s/''\(.*\)``/"\1"/ > +Checkpoints here do not refer to virtual machine snapshots. They are j= ust > +record/replay events used for synchronization. > + > +QEMU in replay mode will try to invoke timers processing in random mom= ent=20 > +of time. That's why we do not process group of timers until the checkp= oint s/group/a group/ > +event will be read from the log. Such an event allows synchronizing CP= U=20 > +execution and timer events. > + > +Another checkpoints application in record/replay is instructions count= ing s/instructions/instruction/ > +while the virtual machine is idle. This function (qemu_clock_warp) is = called > +from the wait loop. It changes virtual machine state and must be deter= ministic > +then. That is why we added checkpoint to this function to prevent its = > +operation in replay mode when it does not correspond to record mode. > + > +Bottom halves > +------------- > + > +Disk I/O events are completely deterministic in our model, because=20 > +in both record and replay modes we start virtual machine from the same= =20 > +disk state. But callbacks that virtual disk controller uses for readin= g and > +writing the disk may occur at different moments of time in record and = replay > +modes. > + > +Reading and writing requests are created by CPU thread of QEMU. Later = these > +requests proceed to block layer which creates ''bottom halves``. Botto= m Another instance of your odd quoting style. Make it consistent with the solution you chose above. > +halves consist of callback and its parameters. They are processed when= > +main loop locks the global mutex. These locking are not synchronized w= ith s/locking/locks/ > +replaying process because main loop also processes the events that do = not > +affect the virtual machine state (like user interation with monitor). s/interation/interaction/ > + > +That is why we had to implement saving and replaying bottom halves cal= lbacks > +synchronously to the CPU execution. When the callback is about to exec= ute > +it is added to the queue in the replay module. This queue is written t= o the > +log when its callbacks are executed. In replay mode callbacks do not p= rocessed s/do/are/ > +until the corresponding event is read from the events log file. > + > +Sometimes block layer uses asynchronous callbacks for its internal pur= poses=20 s/block/the block/ > +(like reading or writing VM snapshots or disk image cluster tables). I= n this > +case bottom halves are not marked as ''replayable`` and do not saved=20 No trailing whitespace. Another instance of odd quoting style. > +into the log. --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --61DIDrxIsA5n9F95NkSpbNr4jLEJp643x Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJVAzVpAAoJEKeha0olJ0Nq8REIAKhSkuzmQwlaik33BE+TXAUR 14GjGD8d5BfhEo4cBFc1l4+R4ucz/7V6QIORPchNOtOfg7DSCHL4CyZOJ8bZUsAK IAXwZqj1ZbVRstupGQxpg77c98X/T0zqpL11wz/O0hWxknqKRplwmZV+o67VT8eQ 26t2HwNNI+jo30gMmxIO47Z7ctWLoJovWue3wJr13IkoWGX8Jf/BrMpMeVSytWOc fQMKntsbI4cSuCORYgQ++xkeARivBlrPx2aG4AVcuzSjka4lLzI79iBgGzmE7ubd MMZY8vCench0fwNATMXW482tiHQQk3O3OLDawfLQj5bymBWtvihOEmmf7TySUCs= =xJw/ -----END PGP SIGNATURE----- --61DIDrxIsA5n9F95NkSpbNr4jLEJp643x--