Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Nutaro, James J.]

My original problem seems to stem from something that changed in the way that device emulation and instruction execution interact (I'm guessing). To reproduce the issue, I started a linux image with

qemu-system-i386 -rtc clock=vm -monitor none -icount 1,sleep=off jack.img

After booting, I run

ping localhost

The first round trip time reports look reasonable, being in the millisecond to sub-millisecond range. These occur while the emulator is running slower than real time. 

After a bit, the emulator speeds up (skipping over idle periods during I/O?) and the round trip times jump to hundreds of milliseconds, which I had not expected.

If you want to try this experiment yourself, you can get the disk image that I used from here:

http://www.ornl.gov/~1qn/qemu-images/jack.img


Jim

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Frédéric Konrad]

Hi Jim,

I think Alex and Paolo worked on an mttcg/icount issue.
Not sure if its related or not.

Fred

On 03/03/2017 08:38 PM, Nutaro, James J. wrote:
> My original problem seems to stem from something that changed in the way that device emulation and instruction execution interact (I'm guessing). To reproduce the issue, I started a linux image with
> 
> qemu-system-i386 -rtc clock=vm -monitor none -icount 1,sleep=off jack.img
> 
> After booting, I run
> 
> ping localhost
> 
> The first round trip time reports look reasonable, being in the millisecond to sub-millisecond range. These occur while the emulator is running slower than real time. 
> 
> After a bit, the emulator speeds up (skipping over idle periods during I/O?) and the round trip times jump to hundreds of milliseconds, which I had not expected.
> 
> If you want to try this experiment yourself, you can get the disk image that I used from here:
> 
> http://www.ornl.gov/~1qn/qemu-images/jack.img
> 
> 
> Jim
> 

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Alex Bennée]

Frédéric Konrad <konrad.frederic@yahoo.fr> writes:

> Hi Jim,
>
> I think Alex and Paolo worked on an mttcg/icount issue.
> Not sure if its related or not.

If you are running a tree post-MTTCG merge you will need to add Paolo's
patches:

  Subject: [PATCH 0/6] tcg: fix icount super slowdown
  Date: Fri,  3 Mar 2017 14:11:08 +0100
  Message-Id: <20170303131113.25898-1-pbonzini@redhat.com>

>
> Fred
>
> On 03/03/2017 08:38 PM, Nutaro, James J. wrote:
>> My original problem seems to stem from something that changed in the way that device emulation and instruction execution interact (I'm guessing). To reproduce the issue, I started a linux image with
>>
>> qemu-system-i386 -rtc clock=vm -monitor none -icount 1,sleep=off jack.img
>>
>> After booting, I run
>>
>> ping localhost
>>
>> The first round trip time reports look reasonable, being in the millisecond to sub-millisecond range. These occur while the emulator is running slower than real time.
>>
>> After a bit, the emulator speeds up (skipping over idle periods during I/O?) and the round trip times jump to hundreds of milliseconds, which I had not expected.
>>
>> If you want to try this experiment yourself, you can get the disk image that I used from here:
>>
>> http://www.ornl.gov/~1qn/qemu-images/jack.img
>>
>>
>> Jim
>>


--
Alex Bennée

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Pavel Dovgalyuk]

I also encountered icount problems with new MTTCG patches.

Record/replay now cannot work, because iothread requests timers
without kicking the CPU. And cpu thread updates icount (that
are used for the clock).

Therefore invocation of iothread uses incorrect clock and
all virtual timers behave incorrectly.

Record/replay is also broken because current icount is requested
from iothread where current_cpu (and icount progress stored in icount_extra)
is unavailable.

Pavel Dovgalyuk

> -----Original Message-----
> From: mttcg-request@listserver.greensocs.com [mailto:mttcg-request@listserver.greensocs.com]
> On Behalf Of Nutaro, James J.
> Sent: Friday, March 03, 2017 10:39 PM
> To: 'mttcg@listserver.greensocs.com'
> Cc: 'qemu-devel@nongnu.org'
> Subject: RE: -rtc clock=vm with -icount 1,sleep=off introduces unexpected delays in device
> interactions
> 
> My original problem seems to stem from something that changed in the way that device emulation
> and instruction execution interact (I'm guessing). To reproduce the issue, I started a linux
> image with
> 
> qemu-system-i386 -rtc clock=vm -monitor none -icount 1,sleep=off jack.img
> 
> After booting, I run
> 
> ping localhost
> 
> The first round trip time reports look reasonable, being in the millisecond to sub-millisecond
> range. These occur while the emulator is running slower than real time.
> 
> After a bit, the emulator speeds up (skipping over idle periods during I/O?) and the round
> trip times jump to hundreds of milliseconds, which I had not expected.
> 
> If you want to try this experiment yourself, you can get the disk image that I used from here:
> 
> http://www.ornl.gov/~1qn/qemu-images/jack.img
> 
> 
> Jim

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Alex Bennée]

Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:

> I also encountered icount problems with new MTTCG patches.
>
> Record/replay now cannot work, because iothread requests timers
> without kicking the CPU. And cpu thread updates icount (that
> are used for the clock).

The interaction of kicking the vCPU while grabbing the BQL was a
side-effect. This is now done explicitly for single-threaded emulation
by (6546706d28):

  tcg: add kick timer for single-threaded vCPU emulation

> Therefore invocation of iothread uses incorrect clock and
> all virtual timers behave incorrectly.
>
> Record/replay is also broken because current icount is requested
> from iothread where current_cpu (and icount progress stored in icount_extra)
> is unavailable.

I'm working through Paolo's series now but I don't see it as
insurmountable. The aim of keeping current_cpu set only during the
cpu-exec loop was intentional because the system as a whole can't make
assumptions about it always being valid.

>
> Pavel Dovgalyuk
>
>> -----Original Message-----
>> From: mttcg-request@listserver.greensocs.com [mailto:mttcg-request@listserver.greensocs.com]
>> On Behalf Of Nutaro, James J.
>> Sent: Friday, March 03, 2017 10:39 PM
>> To: 'mttcg@listserver.greensocs.com'
>> Cc: 'qemu-devel@nongnu.org'
>> Subject: RE: -rtc clock=vm with -icount 1,sleep=off introduces unexpected delays in device
>> interactions
>>
>> My original problem seems to stem from something that changed in the way that device emulation
>> and instruction execution interact (I'm guessing). To reproduce the issue, I started a linux
>> image with
>>
>> qemu-system-i386 -rtc clock=vm -monitor none -icount 1,sleep=off jack.img
>>
>> After booting, I run
>>
>> ping localhost
>>
>> The first round trip time reports look reasonable, being in the millisecond to sub-millisecond
>> range. These occur while the emulator is running slower than real time.
>>
>> After a bit, the emulator speeds up (skipping over idle periods during I/O?) and the round
>> trip times jump to hundreds of milliseconds, which I had not expected.
>>
>> If you want to try this experiment yourself, you can get the disk image that I used from here:
>>
>> http://www.ornl.gov/~1qn/qemu-images/jack.img
>>
>>
>> Jim


--
Alex Bennée

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Pavel Dovgalyuk]

> From: Alex Bennée [mailto:alex.bennee@linaro.org]
> > I also encountered icount problems with new MTTCG patches.
> >
> > Record/replay now cannot work, because iothread requests timers
> > without kicking the CPU. And cpu thread updates icount (that
> > are used for the clock).
> 
> The interaction of kicking the vCPU while grabbing the BQL was a
> side-effect. This is now done explicitly for single-threaded emulation
> by (6546706d28):
> 
>   tcg: add kick timer for single-threaded vCPU emulation

This is not the same.
BQL helped in making execution deterministic - no io and timer
callbacks were performed while CPU is executing.
Now iothread and CPU thread work simultaneously and timers can't
query virtual time correctly, because (if we could query number of executed
instructions) it may have different values on different runs.

I guess you'll have to bring kicking CPU back on iothread invocation
to make execution deterministic in icount mode.

> 
> > Therefore invocation of iothread uses incorrect clock and
> > all virtual timers behave incorrectly.
> >
> > Record/replay is also broken because current icount is requested
> > from iothread where current_cpu (and icount progress stored in icount_extra)
> > is unavailable.
> 
> I'm working through Paolo's series now but I don't see it as
> insurmountable. The aim of keeping current_cpu set only during the
> cpu-exec loop was intentional because the system as a whole can't make
> assumptions about it always being valid.


Pavel Dovgalyuk

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Paolo Bonzini]

On 14/03/2017 13:10, Pavel Dovgalyuk wrote:
> This is not the same.
> BQL helped in making execution deterministic - no io and timer
> callbacks were performed while CPU is executing.
> Now iothread and CPU thread work simultaneously and timers can't
> query virtual time correctly, because (if we could query number of executed
> instructions) it may have different values on different runs.
> 
> I guess you'll have to bring kicking CPU back on iothread invocation
> to make execution deterministic in icount mode.

Timers are fixed by my patch.  I/O should also work if its effects are
only apparent at given QEMU_CLOCK_VIRTUAL values.

Paolo

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> On 14/03/2017 13:10, Pavel Dovgalyuk wrote:
> > This is not the same.
> > BQL helped in making execution deterministic - no io and timer
> > callbacks were performed while CPU is executing.
> > Now iothread and CPU thread work simultaneously and timers can't
> > query virtual time correctly, because (if we could query number of executed
> > instructions) it may have different values on different runs.
> >
> > I guess you'll have to bring kicking CPU back on iothread invocation
> > to make execution deterministic in icount mode.
> 
> Timers are fixed by my patch.  I/O should also work if its effects are
> only apparent at given QEMU_CLOCK_VIRTUAL values.

Can you point me on that patch?

Pavel Dovgalyuk

Re: [Qemu-devel] -rtc clock=vm with -icount 1, sleep=off introduces unexpected delays in device interactions

[Paolo Bonzini]

On 14/03/2017 13:18, Pavel Dovgalyuk wrote:
>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>> On 14/03/2017 13:10, Pavel Dovgalyuk wrote:
>>> This is not the same.
>>> BQL helped in making execution deterministic - no io and timer
>>> callbacks were performed while CPU is executing.
>>> Now iothread and CPU thread work simultaneously and timers can't
>>> query virtual time correctly, because (if we could query number of executed
>>> instructions) it may have different values on different runs.
>>>
>>> I guess you'll have to bring kicking CPU back on iothread invocation
>>> to make execution deterministic in icount mode.
>>
>> Timers are fixed by my patch.  I/O should also work if its effects are
>> only apparent at given QEMU_CLOCK_VIRTUAL values.
> 
> Can you point me on that patch?

I sent a pull request now, but it doesn't fix record/replay.

https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg02621.html

Paolo