[Qemu-devel] Possible bug in Qemu tcp migration

[Qemu-devel] Possible bug in Qemu tcp migration

[Chris Lalancette]

Anthony,
    I was just finishing implementing Qemu/KVM live migration support in libvirt
(posting here:
https://www.redhat.com/archives/libvir-list/2008-November/msg00087.html).  It
works for the most part, except for one bug which I *think* is a bug in the Qemu
live migration support.  Here's the scenario:

1.  start up the guest on the source side (virsh start myguest)
2.  migrate the guest to the destination (virsh migrate qemu+tcp://remote/system)
    a.  The virsh process on the source side sends a command to the destination
libvirtd, and basically tells it to start the qemu container with -incoming.
    b.  The source side performs the migrate via the "migrate tcp:remote:4444"
monitor command
    c.  Once the "migrate" monitor command completes without error, the source
side kills the source qemu process.

The problem seems to happen between steps b. and c. above.  If I let that
sequence rip as-is, then all of the memory from the source side is transferred
over to the destination, and the "migrate" monitor command returns without
error.  However, the guest on the destination never starts; it's there, and all
of the memory is there, it just won't execute at all.

If, instead, I add a 5 second sleep in between steps b. and c. on the source
side, then the migration completes as expected.  It seems that the "migrate"
monitor command is actually returning before everything is complete, so killing
off the guest on the source side makes the destination wait around forever.
Unfortunately, I haven't yet had time to look at it in any detail to see what's
going on in the Qemu side, but I thought I would give you a heads up, and maybe
you have an idea of where to look.

Thanks,
-- 
Chris Lalancette

[Qemu-devel] Re: Possible bug in Qemu tcp migration

[Anthony Liguori]

Chris Lalancette wrote:
> Anthony,
>
> If, instead, I add a 5 second sleep in between steps b. and c. on the source
> side, then the migration completes as expected.  It seems that the "migrate"
> monitor command is actually returning before everything is complete, so killing
> off the guest on the source side makes the destination wait around forever.
> Unfortunately, I haven't yet had time to look at it in any detail to see what's
> going on in the Qemu side, but I thought I would give you a heads up, and maybe
> you have an idea of where to look.
>   

Sounds like maybe the buffer isn't getting completely flushed before 
migration returns successfully.

Regards,

Anthony Liguori

> Thanks,
>   

[Qemu-devel] Re: Possible bug in Qemu tcp migration

[Anthony Liguori]

Chris Lalancette wrote:
> Anthony,
>
> If, instead, I add a 5 second sleep in between steps b. and c. on the source
> side, then the migration completes as expected.  It seems that the "migrate"
> monitor command is actually returning before everything is complete, so killing
> off the guest on the source side makes the destination wait around forever.
> Unfortunately, I haven't yet had time to look at it in any detail to see what's
> going on in the Qemu side, but I thought I would give you a heads up, and maybe
> you have an idea of where to look.
>   

Can you try the latest SVN?  Specifically, r5640.

Regards,

Anthony Liguori

> Thanks,
>   

Re: [Qemu-devel] Re: Possible bug in Qemu tcp migration

[Chris Lalancette]

Anthony Liguori wrote:
> Chris Lalancette wrote:
>> Anthony,
>>
>> If, instead, I add a 5 second sleep in between steps b. and c. on the source
>> side, then the migration completes as expected.  It seems that the "migrate"
>> monitor command is actually returning before everything is complete, so killing
>> off the guest on the source side makes the destination wait around forever.
>> Unfortunately, I haven't yet had time to look at it in any detail to see what's
>> going on in the Qemu side, but I thought I would give you a heads up, and maybe
>> you have an idea of where to look.
>>   
> 
> Can you try the latest SVN?  Specifically, r5640.

I took just that patch, applied it to my copy of KVM (KVM-78), and tried it out,
and it seems to have done it.  Thanks Anthony!

-- 
Chris Lalancette

Re: [Qemu-devel] Re: Possible bug in Qemu tcp migration

[Anthony Liguori]

Chris Lalancette wrote:
> Anthony Liguori wrote:
>   
>> Chris Lalancette wrote:
>>     
>>> Anthony,
>>>
>>> If, instead, I add a 5 second sleep in between steps b. and c. on the source
>>> side, then the migration completes as expected.  It seems that the "migrate"
>>> monitor command is actually returning before everything is complete, so killing
>>> off the guest on the source side makes the destination wait around forever.
>>> Unfortunately, I haven't yet had time to look at it in any detail to see what's
>>> going on in the Qemu side, but I thought I would give you a heads up, and maybe
>>> you have an idea of where to look.
>>>   
>>>       
>> Can you try the latest SVN?  Specifically, r5640.
>>     
>
> I took just that patch, applied it to my copy of KVM (KVM-78), and tried it out,
> and it seems to have done it.  Thanks Anthony!
>   

Good, I honestly don't understand why it works ;-)

Resuming the monitor should not allow monitor commands to execute until 
you get to the main loop.  I looked through the readline code and that 
seems to confirm that, but perhaps you had sent monitor input already 
and it was buffered?  So that it was just awaiting the opportunity to 
execute the command (no IO needed).

Regards,

Anthony Liguori