[Qemu-devel] Slowdowns comparing qemu-kvm.git to qemu.git: vcpu/thread scheduling differences

[Qemu-devel] Slowdowns comparing qemu-kvm.git to qemu.git: vcpu/thread scheduling differences

[Amit Shah]

Hello,

In my testing of virtio-console, I found qemu-kvm.git introduces a lot
of overhead in thread scheduling compared to qemu.git.

My test sends a 260M file from the host to a guest via a virtio-console
port and then computes the sha1sum of the file on the host as well as on
the guest, compares the checksum and declares the result based on the
checksum match. The test passes in all the scenarios listed below,
indicating there's no unsafe data transfer.


Repo               Time taken
-----              ----------
qemu.git             < 1 m (typically 30s)
qemu-kvm.git         > 16m
qemu-iothread        ~ 5m


The guest remains the same in all cases.

I went back upto Oct 2009 in the git history to check if this was a
regression; it is not. The qemu-kvm.git results are reproducible on
qemu-kvm-0.12.git as well.

The slowdown with qemu-iothread was reproduced by virtio-net as well:
netperf results were better on qemu.git with iothread disabled than on
qemu.git with iothread enabled.


The virtio-console code currently puts only one buffer in the in_vq,
meaning the host can only transfer 4k at a time before a guest consumes
the data and makes the buffer available for subsequent transfers.
Increasing the number of buffers available to 128 though "fixes" this --
qemu-kvm.git performance is much better in this case.


I have some gprof results comparing qemu.git and qemu-kvm.git (30% of
time in qemu-kvm.git is spent in main_loop_wait()), which I'm still
analyzing and also going over the code paths that are different.

I just thought I'd put out the observations out here so that others who
have an idea can chime in or validate these results with other
workloads.

		Amit

[Qemu-devel] Re: Slowdowns comparing qemu-kvm.git to qemu.git: vcpu/thread scheduling differences

[Anthony Liguori]

On 02/08/2010 07:46 AM, Amit Shah wrote:
> Hello,
>
> In my testing of virtio-console, I found qemu-kvm.git introduces a lot
> of overhead in thread scheduling compared to qemu.git.
>
> My test sends a 260M file from the host to a guest via a virtio-console
> port and then computes the sha1sum of the file on the host as well as on
> the guest, compares the checksum and declares the result based on the
> checksum match. The test passes in all the scenarios listed below,
> indicating there's no unsafe data transfer.
>
>
> Repo               Time taken
> -----              ----------
> qemu.git<  1 m (typically 30s)
> qemu-kvm.git>  16m
> qemu-iothread        ~ 5m
>    

That very likely suggests that there are missing qemu_notify_events() in 
qemu-kvm.git and you're getting blocked waiting for the next timer event 
to fire.

IOW, I assume that during the qemu-kvm.git run, the CPU isn't pegged at 
100% whereas it is in qemu.git.

Regards,

Anthony Liguori

[Qemu-devel] Re: Slowdowns comparing qemu-kvm.git to qemu.git: vcpu/thread scheduling differences

[Amit Shah]

On (Mon) Feb 08 2010 [08:57:05], Anthony Liguori wrote:
> On 02/08/2010 07:46 AM, Amit Shah wrote:
>> Hello,
>>
>> In my testing of virtio-console, I found qemu-kvm.git introduces a lot
>> of overhead in thread scheduling compared to qemu.git.
>>
>> My test sends a 260M file from the host to a guest via a virtio-console
>> port and then computes the sha1sum of the file on the host as well as on
>> the guest, compares the checksum and declares the result based on the
>> checksum match. The test passes in all the scenarios listed below,
>> indicating there's no unsafe data transfer.
>>
>>
>> Repo               Time taken
>> -----              ----------
>> qemu.git<  1 m (typically 30s)
>> qemu-kvm.git>  16m
>> qemu-iothread        ~ 5m
>>    
>
> That very likely suggests that there are missing qemu_notify_events() in  
> qemu-kvm.git and you're getting blocked waiting for the next timer event  
> to fire.

Hm, if that's the case, should virtio_notify() have a call to
qemu_notify_event()?

I added a qemu_notify_event() as the last statement in write_to_port()
but that didn't help.

> IOW, I assume that during the qemu-kvm.git run, the CPU isn't pegged at  
> 100% whereas it is in qemu.git.

I think you're right; for qemu-kvm.git, I have seen various numbers:
usage at an avg. of 2% and also at an avg. of 20% in different runs.

I just did another run before writing this: 2% for qemu-kvm.git and 100%
for qemu.git.

		Amit

[Qemu-devel] Re: Slowdowns comparing qemu-kvm.git to qemu.git: vcpu/thread scheduling differences

[Anthony Liguori]

On 02/08/2010 11:35 AM, Amit Shah wrote:
> On (Mon) Feb 08 2010 [08:57:05], Anthony Liguori wrote:
>    
>> On 02/08/2010 07:46 AM, Amit Shah wrote:
>>      
>>> Hello,
>>>
>>> In my testing of virtio-console, I found qemu-kvm.git introduces a lot
>>> of overhead in thread scheduling compared to qemu.git.
>>>
>>> My test sends a 260M file from the host to a guest via a virtio-console
>>> port and then computes the sha1sum of the file on the host as well as on
>>> the guest, compares the checksum and declares the result based on the
>>> checksum match. The test passes in all the scenarios listed below,
>>> indicating there's no unsafe data transfer.
>>>
>>>
>>> Repo               Time taken
>>> -----              ----------
>>> qemu.git<   1 m (typically 30s)
>>> qemu-kvm.git>   16m
>>> qemu-iothread        ~ 5m
>>>
>>>        
>> That very likely suggests that there are missing qemu_notify_events() in
>> qemu-kvm.git and you're getting blocked waiting for the next timer event
>> to fire.
>>      
> Hm, if that's the case, should virtio_notify() have a call to
> qemu_notify_event()?
>    

No, basically, the problem will boil down to, the IO thread is 
select()'d waiting for an event to occur.  However, you've done 
something in the VCPU thread that requires the IO thread to run it's 
main loop.  You need to use qemu_notify_event() to force the IO thread 
to break out of select().

Debugging these problems are very difficult and the complexity here is 
the main reason the IO thread still hasn't been enabled by default in 
qemu.git.

I'd suggest looking at the main loop in qemu-kvm, seeing what isn't 
processed as a result of fd becoming readable/writable, and then making 
sure that any of those mechanisms you're relying on in virtio-console 
have the appropriate tie-ins to qemu_notify_event().

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: Slowdowns comparing qemu-kvm.git to qemu.git: vcpu/thread scheduling differences

[Jamie Lokier]

Anthony Liguori wrote:
> No, basically, the problem will boil down to, the IO thread is 
> select()'d waiting for an event to occur.  However, you've done 
> something in the VCPU thread that requires the IO thread to run it's 
> main loop.  You need to use qemu_notify_event() to force the IO thread 
> to break out of select().
> 
> Debugging these problems are very difficult and the complexity here is 
> the main reason the IO thread still hasn't been enabled by default in 
> qemu.git.

It is difficult.  One approach to debugging them, in general, is to
have a special debugging mode where the select() loop wakes up
repeatedly at high speed and checks all the conditions it's supposed
to to be waiting for that _should_ have triggered a wakeup, and if any
of them are asserted for two timed wakeups (or some sufficient duration,
checked by gettimeofday), emit a bug message because it probably is one.

I don't know if that could be applied in qemu's event loop.

-- Jamie