From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40748) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bmMvI-0004bx-70 for qemu-devel@nongnu.org; Tue, 20 Sep 2016 11:25:33 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bmMvD-0007Uc-5g for qemu-devel@nongnu.org; Tue, 20 Sep 2016 11:25:28 -0400 Received: from mta02.ornl.gov ([128.219.177.136]:51120) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bmMvC-0007UO-Rk for qemu-devel@nongnu.org; Tue, 20 Sep 2016 11:25:23 -0400 From: "Nutaro, James J." Date: Tue, 20 Sep 2016 15:22:42 +0000 Message-ID: <1474384962709.82066@ornl.gov> References: <1473270798-12193-1-git-send-email-nutarojj@ornl.gov>, <20160907203343.5019ee0b@bahia> In-Reply-To: <20160907203343.5019ee0b@bahia> Content-Language: en-US Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [Qemu-devel] [PATCH v3] qqq: module for synchronizing with a simulation List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Greg Kurz Cc: "qemu-devel@nongnu.org" I realizes my reply wasn't sent to the forum. Thanks Greg for the feedback.= I'll keep this in mind for the next revision.=0A= =0A= Jim=0A= ________________________________________=0A= From: Greg Kurz =0A= Sent: Wednesday, September 07, 2016 2:33 PM=0A= To: Nutaro, James J.=0A= Cc: qemu-devel@nongnu.org=0A= Subject: Re: [Qemu-devel] [PATCH v3] qqq: module for synchronizing with a s= imulation=0A= =0A= On Wed, 7 Sep 2016 13:53:18 -0400=0A= "James J. Nutaro" wrote:=0A= =0A= > This patch adds an interface for pacing the execution of=0A= > QEMU to match its rtc with an external simulation clock.=0A= > Its aim is to permit QEMU to be used as a module within a=0A= > larger simulation system. This revision (v3) corrects=0A= > formatting errors detected by Patchew.=0A= >=0A= =0A= The changelog should only mention what the patch actually does. If you have= =0A= to re-post newer versions before the patch is finally accepted, changes sho= uld=0A= be mentioned...=0A= =0A= > Signed-off-by: James J. Nutaro =0A= > ---=0A= =0A= ... here, below the ---, so they don't appear in the commit changelog.=0A= =0A= Cheers.=0A= =0A= --=0A= Greg=0A= =0A= > Makefile.target | 3 ++=0A= > docs/simulation-sync.txt | 60 ++++++++++++++++++++++++++=0A= > qemu-options.hx | 16 +++++++=0A= > qqq.c | 109 +++++++++++++++++++++++++++++++++++++++++= ++++++=0A= > qqq.h | 35 +++++++++++++++=0A= > vl.c | 32 ++++++++++++++=0A= > 6 files changed, 255 insertions(+)=0A= > create mode 100644 docs/simulation-sync.txt=0A= > create mode 100644 qqq.c=0A= > create mode 100644 qqq.h=0A= >=0A= > diff --git a/Makefile.target b/Makefile.target=0A= > index 8c7a072..b7f6c4c 100644=0A= > --- a/Makefile.target=0A= > +++ b/Makefile.target=0A= > @@ -103,6 +103,9 @@ obj-$(CONFIG_LIBDECNUMBER) +=3D libdecnumber/dpd/deci= mal32.o=0A= > obj-$(CONFIG_LIBDECNUMBER) +=3D libdecnumber/dpd/decimal64.o=0A= > obj-$(CONFIG_LIBDECNUMBER) +=3D libdecnumber/dpd/decimal128.o=0A= >=0A= > +# qqq=0A= > +obj-y +=3D qqq.o=0A= > +=0A= > #########################################################=0A= > # Linux user emulator target=0A= >=0A= > diff --git a/docs/simulation-sync.txt b/docs/simulation-sync.txt=0A= > new file mode 100644=0A= > index 0000000..2133093=0A= > --- /dev/null=0A= > +++ b/docs/simulation-sync.txt=0A= > @@ -0,0 +1,60 @@=0A= > +=3D Synchronizing the virtual clock with an external source =3D=0A= > +=0A= > +QEMU has a protocol for synchronizing its virtual clock=0A= > +with the clock of a simulator in which QEMU is embedded=0A= > +as a component. This options is enabled with the -qqq=0A= > +argument, and it should generally be accompanied by the=0A= > +following additional command line arguments:=0A= > +=0A= > +-icount 1,sleep=3Doff -rtc clock=3Dvm=0A= > +=0A= > +The -qqq argument is used to supply file descriptors=0A= > +for two Unix pipes. The read pipe is used by QEMU to=0A= > +receive synchronization data from the external simulator.=0A= > +The write pipe is used by QEMU to supply synchronization=0A= > +data to the external emulator. The typical procedure for=0A= > +launching QEMU in is synchronization mode has three steps:=0A= > +=0A= > +(1) Create two pairs of pipes with the Linux pipe function.=0A= > + The code segment that does this might look like=0A= > +=0A= > + int pipefd1[2];=0A= > + int pipefd2[2];=0A= > + pipe(pipefd1);=0A= > + pipe(pipefd2);=0A= > +=0A= > +(2) Fork QEMU with the appropriate command line arguments.=0A= > + The -qqq part of the argument will look something like=0A= > +=0A= > + -qqq write=3Dpipefd1[1],read=3Dpipefd2[0]=0A= > +=0A= > +(3) After forking QEMU, close pipefd1[1] and pipefd2[0].=0A= > + Retain the other pair of pipes for communicating with QEMU.=0A= > +=0A= > +The synchronization protocol is very simple. To start, the=0A= > +external simulator writes an integer to its write pipe with=0A= > +the amount of time in microseconds that QEMU is allowed to=0A= > +advance. The code segment that does this might look like:=0A= > +=0A= > + int ta =3D 1000; // Advance by 1 millisecond=0A= > + write(pipefd2[1],&ta,sizeof(int));=0A= > +=0A= > +The external simulator can then advance its clock by this=0A= > +same amount. During this time, QEMU and the external simulator=0A= > +will be executing in parallel. When the external simulator=0A= > +completes its time advance, it waits for QEMU by reading from=0A= > +its read pipe. The value read will be the actual number of=0A= > +virtual microseconds by which QEMU has advanced its virtual clock.=0A= > +This will be greater than or equal to the requested advance.=0A= > +The code that does this might look like:=0A= > +=0A= > + read(pipefd1[0],&ta,sizeof(int));=0A= > +=0A= > +These steps are repeated until either (1) the external simulator=0A= > +closes its pipes thereby causing QEMU to terminate or (2) QEMU=0A= > +stops executing (e.g., if the emulated computer is shutdown) and=0A= > +causes SIGPIPE to be generated by the closing of its pipes.=0A= > +=0A= > +You can find an example of a simulator using this protocol in=0A= > +the adevs simulation package at http://sourceforge.net/projects/adevs/= =0A= > +=0A= > diff --git a/qemu-options.hx b/qemu-options.hx=0A= > index a71aaf8..d52cc9c 100644=0A= > --- a/qemu-options.hx=0A= > +++ b/qemu-options.hx=0A= > @@ -3359,6 +3359,22 @@ many timer interrupts were not processed by the Wi= ndows guest and will=0A= > re-inject them.=0A= > ETEXI=0A= >=0A= > +DEF("qqq", HAS_ARG, QEMU_OPTION_qqq, \=0A= > + "-qqq read=3Dfd,write=3Dfd\n" \=0A= > + " enable synchronization of the virtual clock \n" \= =0A= > + " with an external simulation clock\n", QEMU_ARCH_ALL= )=0A= > +STEXI=0A= > +@item -qqq read=3D@var{fd0},write=3D@var{fd1}=0A= > +@findex -qqq=0A= > +Qemu will use the supplied pipes to synchronize its virtual clock with= =0A= > +an external simulation clock. Qemu will wait until a time slice size in= =0A= > +microseconds is supplied on the read pipe. Then it will execute for at= =0A= > +least that number of virtual microseconds before writing the actual=0A= > +virtual time that has elapsed in microseconds to the write pipe. This=0A= > +cycle will repeat until a zero is elaspsed time is requested, which=0A= > +will cause qemu to exit.=0A= > +ETEXI=0A= > +=0A= > DEF("icount", HAS_ARG, QEMU_OPTION_icount, \=0A= > "-icount [shift=3DN|auto][,align=3Don|off][,sleep=3Don|off,rr=3Dreco= rd|replay,rrfile=3D]\n" \=0A= > " enable virtual instruction counter with 2^N clock t= icks per\n" \=0A= > diff --git a/qqq.c b/qqq.c=0A= > new file mode 100644=0A= > index 0000000..f94711f=0A= > --- /dev/null=0A= > +++ b/qqq.c=0A= > @@ -0,0 +1,109 @@=0A= > +=0A= > +#include "qemu/osdep.h"=0A= > +#include "qemu/timer.h"=0A= > +#include "qemu/main-loop.h"=0A= > +#include "sysemu/cpus.h"=0A= > +#include "qqq.h"=0A= > +#include =0A= > +=0A= > +/* This is a Linux only feature */=0A= > +=0A= > +#ifndef _WIN32=0A= > +=0A= > +#include =0A= > +#include =0A= > +=0A= > +static int elapsed;=0A= > +static int time_advance =3D -1;=0A= > +static int read_fd =3D -1, write_fd =3D -1;=0A= > +static int64_t t;=0A= > +static QEMUTimer *sync_timer;=0A= > +=0A= > +static void cleanup_and_exit(void)=0A= > +{=0A= > + close(read_fd);=0A= > + close(write_fd);=0A= > + exit(0);=0A= > +}=0A= > +=0A= > +static void write_mem_value(int val)=0A= > +{=0A= > + if (write(write_fd, &val, sizeof(int)) !=3D sizeof(int)) {=0A= > + /* If the pipe is no good, then assume this is an=0A= > + * indication that we should exit.=0A= > + */=0A= > + cleanup_and_exit();=0A= > + }=0A= > +}=0A= > +=0A= > +static int read_mem_value(void)=0A= > +{=0A= > + int tmp;=0A= > + if (read(read_fd, &tmp, sizeof(int)) !=3D sizeof(int)) {=0A= > + /* If the pipe is no good, then assume this is an=0A= > + * indication that we should exit.=0A= > + */=0A= > + cleanup_and_exit();=0A= > + }=0A= > + return tmp;=0A= > +}=0A= > +=0A= > +static void schedule_next_event(void)=0A= > +{=0A= > + /* If we got the time advance in fd_read, then don't do it=0A= > + * again here. */=0A= > + if (time_advance < 0) {=0A= > + /* Otherwise read the value from the pipe */=0A= > + time_advance =3D read_mem_value();=0A= > + }=0A= > + /* Schedule the next synchronization point */=0A= > + timer_mod(sync_timer, t + time_advance);=0A= > + /* Note that we need to read the time advance again on the next pass= */=0A= > + time_advance =3D -1;=0A= > +}=0A= > +=0A= > +static void sync_func(void *data)=0A= > +{=0A= > + /* Report the actual elapsed time. */=0A= > + int64_t tnow =3D qemu_clock_get_us(QEMU_CLOCK_VIRTUAL);=0A= > + elapsed =3D tnow - t;=0A= > + write_mem_value(elapsed);=0A= > + /* Update our time of last event */=0A= > + t =3D tnow;=0A= > + /* Schedule the next event */=0A= > + schedule_next_event();=0A= > +}=0A= > +=0A= > +static void fd_read(void *opaque)=0A= > +{=0A= > + /* Read the time advance if its becomes available=0A= > + * before our timer expires */=0A= > + time_advance =3D read_mem_value();=0A= > +}=0A= > +=0A= > +void setup_qqq(QemuOpts *opts)=0A= > +{=0A= > + /* Initialize the simulation clock */=0A= > + t =3D 0;=0A= > + /* Get the communication pipes */=0A= > + read_fd =3D qemu_opt_get_number(opts, "read", 0);=0A= > + write_fd =3D qemu_opt_get_number(opts, "write", 0);=0A= > + /* Start the timer to ensure time warps advance the clock */=0A= > + sync_timer =3D timer_new_us(QEMU_CLOCK_VIRTUAL, sync_func, NULL);=0A= > + /* Get the time advance that is requested by the simulation */=0A= > + schedule_next_event();=0A= > + /* Register the file descriptor with qemu. This should ensure=0A= > + * the emulator doesn't pause for lack of I/O and thereby=0A= > + * cause the attached simulator to pause with it. */=0A= > + qemu_set_fd_handler(read_fd, fd_read, NULL, NULL);=0A= > +}=0A= > +=0A= > +#else=0A= > +=0A= > +void setup_qqq(QemuOpts *opts)=0A= > +{=0A= > + fprintf(stderr, "-qqq is not supported on Windows, exiting\n");=0A= > + exit(0);=0A= > +}=0A= > +=0A= > +#endif=0A= > diff --git a/qqq.h b/qqq.h=0A= > new file mode 100644=0A= > index 0000000..40a24e3=0A= > --- /dev/null=0A= > +++ b/qqq.h=0A= > @@ -0,0 +1,35 @@=0A= > +/*=0A= > + * This work is licensed under the terms of the GNU GPL=0A= > + * version 2. Seethe COPYING file in the top-level directory.=0A= > + *=0A= > + * A module for pacing the rate of advance of the computer=0A= > + * clock in reference to an external simulation clock. The=0A= > + * basic approach used here is adapted from QBox from Green=0A= > + * Socs. The mode of operation is as follows:=0A= > + *=0A= > + * The simulator uses pipes to exchange time advance data.=0A= > + * The external simulator starts the exchange by forking a=0A= > + * QEMU process and passing is descriptors for a read and=0A= > + * write pipe. Then the external simulator writes an integer=0A= > + * (native endian) to the pipe to indicate the number of=0A= > + * microseconds that QEMU should advance. QEMU advances its=0A= > + * virtual clock by this amount and writes to its write pipe=0A= > + * the actual number of microseconds that have advanced.=0A= > + * This process continues until a pipe on either side is=0A= > + * closed, which will either cause QEMU to exit (if the=0A= > + * external simulator closes a pipe) or raise SIGPIPE in the=0A= > + * external simulator (if QEMU closes a pipe).=0A= > + *=0A= > + * Authors:=0A= > + * James Nutaro =0A= > + *=0A= > + */=0A= > +#ifndef QQQ_H=0A= > +#define QQQ_H=0A= > +=0A= > +#include "qemu/osdep.h"=0A= > +#include "qemu-options.h"=0A= > +=0A= > +void setup_qqq(QemuOpts *opts);=0A= > +=0A= > +#endif=0A= > diff --git a/vl.c b/vl.c=0A= > index ee557a1..fe72990 100644=0A= > --- a/vl.c=0A= > +++ b/vl.c=0A= > @@ -122,6 +122,8 @@ int main(int argc, char **argv)=0A= > #include "sysemu/replay.h"=0A= > #include "qapi/qmp/qerror.h"=0A= >=0A= > +#include "qqq.h"=0A= > +=0A= > #define MAX_VIRTIO_CONSOLES 1=0A= > #define MAX_SCLP_CONSOLES 1=0A= >=0A= > @@ -231,6 +233,23 @@ static struct {=0A= > { .driver =3D "virtio-vga", .flag =3D &default_vga }= ,=0A= > };=0A= >=0A= > +static QemuOptsList qemu_qqq_opts =3D {=0A= > + .name =3D "qqq",=0A= > + .implied_opt_name =3D "",=0A= > + .merge_lists =3D true,=0A= > + .head =3D QTAILQ_HEAD_INITIALIZER(qemu_qqq_opts.head),=0A= > + .desc =3D {=0A= > + {=0A= > + .name =3D "read",=0A= > + .type =3D QEMU_OPT_NUMBER,=0A= > + }, {=0A= > + .name =3D "write",=0A= > + .type =3D QEMU_OPT_NUMBER,=0A= > + },=0A= > + { /* end of list */ }=0A= > + },=0A= > +};=0A= > +=0A= > static QemuOptsList qemu_rtc_opts =3D {=0A= > .name =3D "rtc",=0A= > .head =3D QTAILQ_HEAD_INITIALIZER(qemu_rtc_opts.head),=0A= > @@ -2952,6 +2971,7 @@ int main(int argc, char **argv, char **envp)=0A= > DisplayState *ds;=0A= > int cyls, heads, secs, translation;=0A= > QemuOpts *hda_opts =3D NULL, *opts, *machine_opts, *icount_opts =3D = NULL;=0A= > + QemuOpts *qqq_opts =3D NULL;=0A= > QemuOptsList *olist;=0A= > int optind;=0A= > const char *optarg;=0A= > @@ -2987,6 +3007,7 @@ int main(int argc, char **argv, char **envp)=0A= >=0A= > module_call_init(MODULE_INIT_QOM);=0A= >=0A= > + qemu_add_opts(&qemu_qqq_opts);=0A= > qemu_add_opts(&qemu_drive_opts);=0A= > qemu_add_drive_opts(&qemu_legacy_drive_opts);=0A= > qemu_add_drive_opts(&qemu_common_drive_opts);=0A= > @@ -3847,6 +3868,13 @@ int main(int argc, char **argv, char **envp)=0A= > exit(1);=0A= > }=0A= > break;=0A= > + case QEMU_OPTION_qqq:=0A= > + qqq_opts =3D qemu_opts_parse_noisily(qemu_find_opts("qqq= "),=0A= > + optarg, true);=0A= > + if (!qqq_opts) {=0A= > + exit(1);=0A= > + }=0A= > + break;=0A= > case QEMU_OPTION_incoming:=0A= > if (!incoming) {=0A= > runstate_set(RUN_STATE_INMIGRATE);=0A= > @@ -4350,6 +4378,10 @@ int main(int argc, char **argv, char **envp)=0A= > /* spice needs the timers to be initialized by this point */=0A= > qemu_spice_init();=0A= >=0A= > + if (qqq_opts) {=0A= > + setup_qqq(qqq_opts);=0A= > + }=0A= > +=0A= > cpu_ticks_init();=0A= > if (icount_opts) {=0A= > if (kvm_enabled() || xen_enabled()) {=0A= =0A= =0A=