[Qemu-devel] Running Qemu in discrete time/step by step

[Qemu-devel] Running Qemu in discrete time/step by step

[Matt]

Hi,

[Qemu is evoked starting in paragraph 3]
I am interested in increasing the ease of
debugging/reproducibility/validity of network simulations.
Discrete time simulations is the solution that appeals the most to me
because then the final results of the simulation don't depend on the
CPU anymore; another slower computer should reach the same results as
your fast computer. The "perfect" reproducibility aspect makes
debuging a lot easier in my experience too (Especially true for kernel
stacks); indeed rerunning a simulation will exert the same code paths.

I've been using with some success [DCE] (Direct Code Execution) which
allows to run real applications (iperf/vlc) over the linux kernel
compiled as a library (aka "libOS" see
https://lwn.net/Articles/637658/).
The network between the kernels is simulated by [ns-3] (the discrete
time Network Simulator 3). Overall using DCE allows me to reach every
previously mentioned point; every scenario is perfectly reproducible
which proved invaluable when I worked on linux kernel aspects. I
believe its results are also quite credible in some scenarii
(https://hal.inria.fr/hal-00880870/file/con013-hal.pdf).

DCE looks pretty good except it's really a nightmare to maintain. It
partially reimplements glibc so that real applications can run on top
of ns3 (through a hackish gcc extension that breaks clang and gcc old
linker) and the interface to replace the ns3 TCP model with the linux
stack is also hard to maintain.
Thus I've been looking at running Qemu in discrete time and interface
it  with ns3 (since both are FOSS I am familiar with).

- VMSimint does nearly that, it runs Qemu in discrete time but
interface it with a JAVA simulator
http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/We_SIMUTools_2014_40209.pdf
(with the code http://www.ikr.uni-stuttgart.de/Content/IKRSimLib/Download/)
- http://web.ornl.gov/~nutarojj/adevs/ does sthg similar too

My questions would be:
1/ do you know of any other related work ?
2/ I believe there is interest from the research side but would it be
possible to merge either approach or a similar one (adevs patch
doesn't seem too big ~500 lines), would that be of interest for the
Qemu comminity too ?
3/ if yes to 2. How to proceed, which one would be favorite ? if no,
what should be improved ? or would that be a definitive no ?

Best regards

Matt

[ns3] https://www.nsnam.org/ns-3-27/
[DCE] https://www.nsnam.org/overview/projects/direct-code-execution/
more details in https://hal.inria.fr/hal-00880870/file/con013-hal.pdf

Re: [Qemu-devel] Running Qemu in discrete time/step by step

[Emilio G. Cota]

On Thu, Oct 19, 2017 at 14:14:12 +0900, Matt wrote:
(snip)
> - VMSimint does nearly that, it runs Qemu in discrete time but
> interface it with a JAVA simulator
> http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/We_SIMUTools_2014_40209.pdf
> (with the code http://www.ikr.uni-stuttgart.de/Content/IKRSimLib/Download/)
> - http://web.ornl.gov/~nutarojj/adevs/ does sthg similar too

Nutaro's work to interface with QEMU has been posted on the list:
  https://patchwork.kernel.org/patch/9572497/
I'm Cc'ing him in case he's not subscribed to the list.


> My questions would be:
> 1/ do you know of any other related work ?

Is QEMU's record/replay mode of any use to you? Note that as is the
case with Nutaro's patch, you'll need icount mode (i.e. single-core)
enabled.

> 2/ I believe there is interest from the research side but would it be
> possible to merge either approach or a similar one (adevs patch
> doesn't seem too big ~500 lines), would that be of interest for the
> Qemu comminity too ?
> 3/ if yes to 2. How to proceed, which one would be favorite ? if no,
> what should be improved ? or would that be a definitive no ?

I think the adevs approach is reasonable. The patch hasn't gotten much
attention I guess because not many people care about this feature.
But if you could review the patch and certify that it works for you
(i.e. it works with simulators other than adevs), that could only help
the patch getting in.

That said, I make no merging decisions so take this as just my opinion.

Cheers,

		Emilio

Re: [Qemu-devel] Running Qemu in discrete time/step by step

[Nutaro, James J.]

Thanks for taking a look at the patch. The most recent version of the patch will also work with KVM if you need to speed things up. You can find it here:

https://patchwork.kernel.org/patch/9960369/

Somehow I messed up that post and it shows V8 when it should be V9.

You can get the simulator that it works with at 

https://sourceforge.net/projects/adevs/

If nothing else, this simulator would give you a blue print for integration with another tool.

Jim

-----Original Message-----
From: Emilio G. Cota [mailto:cota@braap.org] 
Sent: Thursday, October 19, 2017 4:54 PM
To: Matt
Cc: qemu-devel@nongnu.org; Hajime Tazaki; Nutaro, James J.
Subject: Re: [Qemu-devel] Running Qemu in discrete time/step by step

On Thu, Oct 19, 2017 at 14:14:12 +0900, Matt wrote:
(snip)
> - VMSimint does nearly that, it runs Qemu in discrete time but
> interface it with a JAVA simulator
> http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/We_SIMUTools_2014_40209.pdf
> (with the code http://www.ikr.uni-stuttgart.de/Content/IKRSimLib/Download/)
> - http://web.ornl.gov/~nutarojj/adevs/ does sthg similar too

Nutaro's work to interface with QEMU has been posted on the list:
  https://patchwork.kernel.org/patch/9572497/
I'm Cc'ing him in case he's not subscribed to the list.


> My questions would be:
> 1/ do you know of any other related work ?

Is QEMU's record/replay mode of any use to you? Note that as is the
case with Nutaro's patch, you'll need icount mode (i.e. single-core)
enabled.

> 2/ I believe there is interest from the research side but would it be
> possible to merge either approach or a similar one (adevs patch
> doesn't seem too big ~500 lines), would that be of interest for the
> Qemu comminity too ?
> 3/ if yes to 2. How to proceed, which one would be favorite ? if no,
> what should be improved ? or would that be a definitive no ?

I think the adevs approach is reasonable. The patch hasn't gotten much
attention I guess because not many people care about this feature.
But if you could review the patch and certify that it works for you
(i.e. it works with simulators other than adevs), that could only help
the patch getting in.

That said, I make no merging decisions so take this as just my opinion.

Cheers,

		Emilio

Re: [Qemu-devel] Running Qemu in discrete time/step by step

[Matt]

I did not know about Qemu's record/replay mode which is an interesting
feature but unneeded if a  2nd qemu run with the same inputs generated
the same outputs (as it's the case with DCE).
@nutaro Thanks for the updated patch. I will have a look at adevs. On
a related note https://github.com/qemu/qemu/blob/master/docs/replay.txt
enumerates different sources of indeterminism
==
Replaying of the execution of virtual machine is bound by sources of
non-determinism. These are inputs from clock and peripheral devices,
and QEMU thread scheduling. Thread scheduling affect on processing events
from timers, asynchronous input-output, and bottom halves.
==

Do you manage to achieve perfect reproducibility with adevs + qemu ?
If yes, is there any publication describing how you achieve this ?

Best regards
matt

2017-10-21 0:02 GMT+09:00 Nutaro, James J. <nutarojj@ornl.gov>:
> Thanks for taking a look at the patch. The most recent version of the patch will also work with KVM if you need to speed things up. You can find it here:
>
> https://patchwork.kernel.org/patch/9960369/
>
> Somehow I messed up that post and it shows V8 when it should be V9.
>
> You can get the simulator that it works with at
>
> https://sourceforge.net/projects/adevs/
>
> If nothing else, this simulator would give you a blue print for integration with another tool.
>
> Jim
>
> -----Original Message-----
> From: Emilio G. Cota [mailto:cota@braap.org]
> Sent: Thursday, October 19, 2017 4:54 PM
> To: Matt
> Cc: qemu-devel@nongnu.org; Hajime Tazaki; Nutaro, James J.
> Subject: Re: [Qemu-devel] Running Qemu in discrete time/step by step
>
> On Thu, Oct 19, 2017 at 14:14:12 +0900, Matt wrote:
> (snip)
>> - VMSimint does nearly that, it runs Qemu in discrete time but
>> interface it with a JAVA simulator
>> http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/We_SIMUTools_2014_40209.pdf
>> (with the code http://www.ikr.uni-stuttgart.de/Content/IKRSimLib/Download/)
>> - http://web.ornl.gov/~nutarojj/adevs/ does sthg similar too
>
> Nutaro's work to interface with QEMU has been posted on the list:
>   https://patchwork.kernel.org/patch/9572497/
> I'm Cc'ing him in case he's not subscribed to the list.
>
>
>> My questions would be:
>> 1/ do you know of any other related work ?
>
> Is QEMU's record/replay mode of any use to you? Note that as is the
> case with Nutaro's patch, you'll need icount mode (i.e. single-core)
> enabled.
>
>> 2/ I believe there is interest from the research side but would it be
>> possible to merge either approach or a similar one (adevs patch
>> doesn't seem too big ~500 lines), would that be of interest for the
>> Qemu comminity too ?
>> 3/ if yes to 2. How to proceed, which one would be favorite ? if no,
>> what should be improved ? or would that be a definitive no ?
>
> I think the adevs approach is reasonable. The patch hasn't gotten much
> attention I guess because not many people care about this feature.
> But if you could review the patch and certify that it works for you
> (i.e. it works with simulators other than adevs), that could only help
> the patch getting in.
>
> That said, I make no merging decisions so take this as just my opinion.
>
> Cheers,
>
>                 Emilio
>

Re: [Qemu-devel] Running Qemu in discrete time/step by step

[Peter Maydell]

On 23 October 2017 at 10:38, Matt <mattator@gmail.com> wrote:
> I did not know about Qemu's record/replay mode which is an interesting
> feature but unneeded if a  2nd qemu run with the same inputs generated
> the same outputs (as it's the case with DCE).

Depends what you're trying to do. Deterministic execution is
a prerequisite for record-and-replay, and sometimes determinism
is all you need, but record-and-replay can provide more than that.

thanks
-- PMM

Re: [Qemu-devel] Running Qemu in discrete time/step by step

[Nutaro, James J.]

I don't expect exact repeatability, but I haven't tested for it either. The machinery that prevents repeatability with replay will probably have the same effect in the adevs/qemu simulator.

-----Original Message-----
From: Matt [mailto:mattator@gmail.com] 
Sent: Monday, October 23, 2017 5:38 AM
To: Nutaro, James J.
Cc: Emilio G. Cota; qemu-devel@nongnu.org; Hajime Tazaki
Subject: Re: [Qemu-devel] Running Qemu in discrete time/step by step

I did not know about Qemu's record/replay mode which is an interesting
feature but unneeded if a  2nd qemu run with the same inputs generated
the same outputs (as it's the case with DCE).
@nutaro Thanks for the updated patch. I will have a look at adevs. On
a related note https://github.com/qemu/qemu/blob/master/docs/replay.txt
enumerates different sources of indeterminism
==
Replaying of the execution of virtual machine is bound by sources of
non-determinism. These are inputs from clock and peripheral devices,
and QEMU thread scheduling. Thread scheduling affect on processing events
from timers, asynchronous input-output, and bottom halves.
==

Do you manage to achieve perfect reproducibility with adevs + qemu ?
If yes, is there any publication describing how you achieve this ?

Best regards
matt

2017-10-21 0:02 GMT+09:00 Nutaro, James J. <nutarojj@ornl.gov>:
> Thanks for taking a look at the patch. The most recent version of the patch will also work with KVM if you need to speed things up. You can find it here:
>
> https://patchwork.kernel.org/patch/9960369/
>
> Somehow I messed up that post and it shows V8 when it should be V9.
>
> You can get the simulator that it works with at
>
> https://sourceforge.net/projects/adevs/
>
> If nothing else, this simulator would give you a blue print for integration with another tool.
>
> Jim
>
> -----Original Message-----
> From: Emilio G. Cota [mailto:cota@braap.org]
> Sent: Thursday, October 19, 2017 4:54 PM
> To: Matt
> Cc: qemu-devel@nongnu.org; Hajime Tazaki; Nutaro, James J.
> Subject: Re: [Qemu-devel] Running Qemu in discrete time/step by step
>
> On Thu, Oct 19, 2017 at 14:14:12 +0900, Matt wrote:
> (snip)
>> - VMSimint does nearly that, it runs Qemu in discrete time but
>> interface it with a JAVA simulator
>> http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/We_SIMUTools_2014_40209.pdf
>> (with the code http://www.ikr.uni-stuttgart.de/Content/IKRSimLib/Download/)
>> - http://web.ornl.gov/~nutarojj/adevs/ does sthg similar too
>
> Nutaro's work to interface with QEMU has been posted on the list:
>   https://patchwork.kernel.org/patch/9572497/
> I'm Cc'ing him in case he's not subscribed to the list.
>
>
>> My questions would be:
>> 1/ do you know of any other related work ?
>
> Is QEMU's record/replay mode of any use to you? Note that as is the
> case with Nutaro's patch, you'll need icount mode (i.e. single-core)
> enabled.
>
>> 2/ I believe there is interest from the research side but would it be
>> possible to merge either approach or a similar one (adevs patch
>> doesn't seem too big ~500 lines), would that be of interest for the
>> Qemu comminity too ?
>> 3/ if yes to 2. How to proceed, which one would be favorite ? if no,
>> what should be improved ? or would that be a definitive no ?
>
> I think the adevs approach is reasonable. The patch hasn't gotten much
> attention I guess because not many people care about this feature.
> But if you could review the patch and certify that it works for you
> (i.e. it works with simulators other than adevs), that could only help
> the patch getting in.
>
> That said, I make no merging decisions so take this as just my opinion.
>
> Cheers,
>
>                 Emilio
>

Re: [Qemu-devel] Running Qemu in discrete time/step by step

[Matt]

Within the context of a simulator, qemu runs with a limited set of
devices so that removes some indeterminism but not all I guess.

VMsimint boasts exact reproducibility because they control all
outputs/inputs. I tried to look into their patch but the patch is
empty and instead there seems to be the partial (full ?) tree of qemu
1.6.
I suspect this might be quite some work to port this to the latest
qemu. It's sad because I really like the approach :)
(qemu changes in the latest archive of
http://www.ikr.uni-stuttgart.de/Content/IKRSimLib/Download/)

I guess I will stick to DCE with the hope that qemu considers the
usecase and make it easier.

Cheers

2017-10-24 6:33 GMT+09:00 Nutaro, James J. <nutarojj@ornl.gov>:
> I don't expect exact repeatability, but I haven't tested for it either. The machinery that prevents repeatability with replay will probably have the same effect in the adevs/qemu simulator.
>
> -----Original Message-----
> From: Matt [mailto:mattator@gmail.com]
> Sent: Monday, October 23, 2017 5:38 AM
> To: Nutaro, James J.
> Cc: Emilio G. Cota; qemu-devel@nongnu.org; Hajime Tazaki
> Subject: Re: [Qemu-devel] Running Qemu in discrete time/step by step
>
> I did not know about Qemu's record/replay mode which is an interesting
> feature but unneeded if a  2nd qemu run with the same inputs generated
> the same outputs (as it's the case with DCE).
> @nutaro Thanks for the updated patch. I will have a look at adevs. On
> a related note https://github.com/qemu/qemu/blob/master/docs/replay.txt
> enumerates different sources of indeterminism
> ==
> Replaying of the execution of virtual machine is bound by sources of
> non-determinism. These are inputs from clock and peripheral devices,
> and QEMU thread scheduling. Thread scheduling affect on processing events
> from timers, asynchronous input-output, and bottom halves.
> ==
>
> Do you manage to achieve perfect reproducibility with adevs + qemu ?
> If yes, is there any publication describing how you achieve this ?
>
> Best regards
> matt
>
> 2017-10-21 0:02 GMT+09:00 Nutaro, James J. <nutarojj@ornl.gov>:
>> Thanks for taking a look at the patch. The most recent version of the patch will also work with KVM if you need to speed things up. You can find it here:
>>
>> https://patchwork.kernel.org/patch/9960369/
>>
>> Somehow I messed up that post and it shows V8 when it should be V9.
>>
>> You can get the simulator that it works with at
>>
>> https://sourceforge.net/projects/adevs/
>>
>> If nothing else, this simulator would give you a blue print for integration with another tool.
>>
>> Jim
>>
>> -----Original Message-----
>> From: Emilio G. Cota [mailto:cota@braap.org]
>> Sent: Thursday, October 19, 2017 4:54 PM
>> To: Matt
>> Cc: qemu-devel@nongnu.org; Hajime Tazaki; Nutaro, James J.
>> Subject: Re: [Qemu-devel] Running Qemu in discrete time/step by step
>>
>> On Thu, Oct 19, 2017 at 14:14:12 +0900, Matt wrote:
>> (snip)
>>> - VMSimint does nearly that, it runs Qemu in discrete time but
>>> interface it with a JAVA simulator
>>> http://www.ikr.uni-stuttgart.de/Content/Publications/Archive/We_SIMUTools_2014_40209.pdf
>>> (with the code http://www.ikr.uni-stuttgart.de/Content/IKRSimLib/Download/)
>>> - http://web.ornl.gov/~nutarojj/adevs/ does sthg similar too
>>
>> Nutaro's work to interface with QEMU has been posted on the list:
>>   https://patchwork.kernel.org/patch/9572497/
>> I'm Cc'ing him in case he's not subscribed to the list.
>>
>>
>>> My questions would be:
>>> 1/ do you know of any other related work ?
>>
>> Is QEMU's record/replay mode of any use to you? Note that as is the
>> case with Nutaro's patch, you'll need icount mode (i.e. single-core)
>> enabled.
>>
>>> 2/ I believe there is interest from the research side but would it be
>>> possible to merge either approach or a similar one (adevs patch
>>> doesn't seem too big ~500 lines), would that be of interest for the
>>> Qemu comminity too ?
>>> 3/ if yes to 2. How to proceed, which one would be favorite ? if no,
>>> what should be improved ? or would that be a definitive no ?
>>
>> I think the adevs approach is reasonable. The patch hasn't gotten much
>> attention I guess because not many people care about this feature.
>> But if you could review the patch and certify that it works for you
>> (i.e. it works with simulators other than adevs), that could only help
>> the patch getting in.
>>
>> That said, I make no merging decisions so take this as just my opinion.
>>
>> Cheers,
>>
>>                 Emilio
>>
>