[Qemu-devel] KVM brokenness due to IO thread changes

[Qemu-devel] KVM brokenness due to IO thread changes

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 649 bytes --]

Hi,

this is a heads-up, maybe someone has some time to look into this over
the day: I seems like the IO thread changes caused a few regressions to
the KVM mode.

When I keep this feature disabled, I see strange hick-ups of the event
delivery mechanism, and the guest stops once in a while for a second or
so. Attaching strace makes the whole process terminate early (looks like
it triggers a race in the signal handling). And when I enable the IO
thread, I immediately get a deadlock on qemu_global_mutex.

Moreover:
  CC    i386-softmmu/vl.o
/data/qemu/vl.c:4380: warning: ‘qemu_calculate_timeout’ defined but not used

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] KVM brokenness due to IO thread changes

[Marcelo Tosatti]

Jan,

On Tue, Apr 28, 2009 at 09:40:01AM +0200, Jan Kiszka wrote:
> Hi,
> 
> this is a heads-up, maybe someone has some time to look into this over
> the day: I seems like the IO thread changes caused a few regressions to
> the KVM mode.
> 
> When I keep this feature disabled, I see strange hick-ups of the event
> delivery mechanism, and the guest stops once in a while for a second or
> so. Attaching strace makes the whole process terminate early (looks like
> it triggers a race in the signal handling). And when I enable the IO
> thread, I immediately get a deadlock on qemu_global_mutex.

Yes its borked. The iothread should signal the vcpu thread whenever it
wants to grab the mutex lock, because unlike kvm-userspace it does not
drop the global mutex when entering guest mode (VCPU_RUN ioctl).

Anthony will commit patches to fix that soon.

> 
> Moreover:
>   CC    i386-softmmu/vl.o
> /data/qemu/vl.c:4380: warning: ‘qemu_calculate_timeout’ defined but not used
> 
> Jan

BTW, can you please review the changes made to the debugging logic in
the main loop sometime? (there's a debug_requested global now).

Re: [Qemu-devel] KVM brokenness due to IO thread changes

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1354 bytes --]

Marcelo Tosatti wrote:
> Jan,
> 
> On Tue, Apr 28, 2009 at 09:40:01AM +0200, Jan Kiszka wrote:
>> Hi,
>>
>> this is a heads-up, maybe someone has some time to look into this over
>> the day: I seems like the IO thread changes caused a few regressions to
>> the KVM mode.
>>
>> When I keep this feature disabled, I see strange hick-ups of the event
>> delivery mechanism, and the guest stops once in a while for a second or
>> so. Attaching strace makes the whole process terminate early (looks like
>> it triggers a race in the signal handling). And when I enable the IO
>> thread, I immediately get a deadlock on qemu_global_mutex.
> 
> Yes its borked. The iothread should signal the vcpu thread whenever it
> wants to grab the mutex lock, because unlike kvm-userspace it does not
> drop the global mutex when entering guest mode (VCPU_RUN ioctl).
> 
> Anthony will commit patches to fix that soon.

Looking forward. It's far more efficient to test my infrastructure
changes against the KVM mode.

> 
>> Moreover:
>>   CC    i386-softmmu/vl.o
>> /data/qemu/vl.c:4380: warning: ‘qemu_calculate_timeout’ defined but not used
>>
>> Jan
> 
> BTW, can you please review the changes made to the debugging logic in
> the main loop sometime? (there's a debug_requested global now).
> 

Looks OK and works fine.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Qemu-devel] KVM brokenness due to IO thread changes

[Anthony Liguori]

Jan Kiszka wrote:
> Marcelo Tosatti wrote:
>   
>> Jan,
>>
>> On Tue, Apr 28, 2009 at 09:40:01AM +0200, Jan Kiszka wrote:
>>     
>>> Hi,
>>>
>>> this is a heads-up, maybe someone has some time to look into this over
>>> the day: I seems like the IO thread changes caused a few regressions to
>>> the KVM mode.
>>>
>>> When I keep this feature disabled, I see strange hick-ups of the event
>>> delivery mechanism, and the guest stops once in a while for a second or
>>> so. Attaching strace makes the whole process terminate early (looks like
>>> it triggers a race in the signal handling). And when I enable the IO
>>> thread, I immediately get a deadlock on qemu_global_mutex.
>>>       
>> Yes its borked. The iothread should signal the vcpu thread whenever it
>> wants to grab the mutex lock, because unlike kvm-userspace it does not
>> drop the global mutex when entering guest mode (VCPU_RUN ioctl).
>>
>> Anthony will commit patches to fix that soon.
>>     
>
> Looking forward. It's far more efficient to test my infrastructure
> changes against the KVM mode.
>   

N.B. he's seeing issues with the IO thread disabled.

I've not seen this myself and my patch doesn't fix that problem.

Regards,

Anthony Liguori

Re: [Qemu-devel] KVM brokenness due to IO thread changes

[Marcelo Tosatti]

On Tue, Apr 28, 2009 at 03:09:57PM -0500, Anthony Liguori wrote:
>>>> this is a heads-up, maybe someone has some time to look into this over
>>>> the day: I seems like the IO thread changes caused a few regressions to
>>>> the KVM mode.
>>>>
>>>> When I keep this feature disabled, I see strange hick-ups of the event
>>>> delivery mechanism, and the guest stops once in a while for a second or
>>>> so. Attaching strace makes the whole process terminate early (looks like
>>>> it triggers a race in the signal handling). And when I enable the IO
>>>> thread, I immediately get a deadlock on qemu_global_mutex.
>>>>       
>>> Yes its borked. The iothread should signal the vcpu thread whenever it
>>> wants to grab the mutex lock, because unlike kvm-userspace it does not
>>> drop the global mutex when entering guest mode (VCPU_RUN ioctl).
>>>
>>> Anthony will commit patches to fix that soon.
>>>     
>>
>> Looking forward. It's far more efficient to test my infrastructure
>> changes against the KVM mode.
>>   
>
> N.B. he's seeing issues with the IO thread disabled.

Jan, 

Can you please provide more details on how to reproduce it? You're using
-nographic?

All I see, with ping -c 0.1, is an eventual ~= 25ms in response time:

64 bytes from 192.168.122.140: icmp_seq=80 ttl=64 time=0.008 ms
64 bytes from 192.168.122.140: icmp_seq=81 ttl=64 time=0.008 ms
64 bytes from 192.168.122.140: icmp_seq=82 ttl=64 time=0.008 ms
64 bytes from 192.168.122.140: icmp_seq=83 ttl=64 time=0.023 ms
64 bytes from 192.168.122.140: icmp_seq=84 ttl=64 time=0.008 ms
64 bytes from 192.168.122.140: icmp_seq=85 ttl=64 time=0.008 ms
64 bytes from 192.168.122.140: icmp_seq=86 ttl=64 time=0.009 ms

Which is also seen before the patches. 

Or even better, if you can bisect it...

> I've not seen this myself and my patch doesn't fix that problem.
>
> Regards,
>
> Anthony Liguori