[Qemu-devel] race condition when exec'ing "qemu -incoming" followed by monitor "cont"

[Qemu-devel] race condition when exec'ing "qemu -incoming" followed by monitor "cont"

[Laine Stump]

(Please forgive (and correct!) any inaccuracies in my description of 
qemu's workings - I've only recently started looking at it directly, 
rather than through the lens of libvirt)

libvirt implements a "domain restore" operation by:

0) start with a previously saved domain image in a file

1) open the domain image, and connect it to a pipe

2) fork, connect the pipe to stdin, and exec qemu with "-incoming exec:cat"

3) execute "cont" in that qemu's monitor.

(for those familiar with the code, you can look at the 
src/qemu/eqmu_driver.c:qemudDomainRestore() in the libvirt source).

Although this works successfully for most people, I'm consistently 
seeing a problem on my particular hardware (Intel Core 2 Duo 2.2Ghz) 
that causes this domain restore to fail. It seems that the "cont" 
command takes effect before the restore is completed (possibly/probably 
before it even starts?) resulting in a failed restore - the domain is 
left in some random state, sometimes rebooting spontaneously, sometimes 
just hung.

If I insert a usleep(250 * 1000) between starting up qemu with 
"-incoming exec:cat" and issuing "cont" to start the CPUs, the restore 
is successful 100% of the time.

I've been told that once the incoming migration starts, the monitor will 
be non-responsive until it is complete. This should mean that as long as 
the "cont" isn't issued until after the migration starts, it will be 
blocked until the migration is complete, thus protecting us from the 
race; for this reason (along with the fact that a 250msec sleep is 
enough to cure the problem) I'm thinking it's likely the "cont" happens 
before the migration starts.

There is, of course, an "info migrate" command in the monitor that could 
be used to assure the migration had completed before issuing "cont", but 
that command only works for outgoing migrations, not incoming 
(presumably if it was available, checking the info prior to the 
migration starting would return "not started" (or something similar), 
and once it had started, the entire monitor interface would block until 
the migrate was completed).

Can someone provide any insight on why it is possible to start the CPUs 
in the domain before the incoming migration is complete, and what we can 
do (other than blindly sleeping) to prevent this?

[Qemu-devel] Re: race condition when exec'ing "qemu -incoming" followed by monitor "cont"

[Paolo Bonzini]

On 04/09/2010 06:03 PM, Laine Stump wrote:
>
> Can someone provide any insight on why it is possible to start the CPUs
> in the domain before the incoming migration is complete, and what we can
> do (other than blindly sleeping) to prevent this?

I would say it's a bug in QEMU, and it has to be fixed there.

Paolo

Re: [Qemu-devel] race condition when exec'ing "qemu -incoming" followed by monitor "cont"

[Daniel P. Berrange]

On Fri, Apr 09, 2010 at 12:03:54PM -0400, Laine Stump wrote:
> (Please forgive (and correct!) any inaccuracies in my description of 
> qemu's workings - I've only recently started looking at it directly, 
> rather than through the lens of libvirt)
> 
> libvirt implements a "domain restore" operation by:
> 
> 0) start with a previously saved domain image in a file
> 
> 1) open the domain image, and connect it to a pipe
> 
> 2) fork, connect the pipe to stdin, and exec qemu with "-incoming exec:cat"
> 
> 3) execute "cont" in that qemu's monitor.
> 
> (for those familiar with the code, you can look at the 
> src/qemu/eqmu_driver.c:qemudDomainRestore() in the libvirt source).
> 
> Although this works successfully for most people, I'm consistently 
> seeing a problem on my particular hardware (Intel Core 2 Duo 2.2Ghz) 
> that causes this domain restore to fail. It seems that the "cont" 
> command takes effect before the restore is completed (possibly/probably 
> before it even starts?) resulting in a failed restore - the domain is 
> left in some random state, sometimes rebooting spontaneously, sometimes 
> just hung.
> 
> If I insert a usleep(250 * 1000) between starting up qemu with 
> "-incoming exec:cat" and issuing "cont" to start the CPUs, the restore 
> is successful 100% of the time.
> 
> I've been told that once the incoming migration starts, the monitor will 
> be non-responsive until it is complete. This should mean that as long as 
> the "cont" isn't issued until after the migration starts, it will be 
> blocked until the migration is complete, thus protecting us from the 
> race; for this reason (along with the fact that a 250msec sleep is 
> enough to cure the problem) I'm thinking it's likely the "cont" happens 
> before the migration starts.
> 
> There is, of course, an "info migrate" command in the monitor that could 
> be used to assure the migration had completed before issuing "cont", but 
> that command only works for outgoing migrations, not incoming 
> (presumably if it was available, checking the info prior to the 
> migration starting would return "not started" (or something similar), 
> and once it had started, the entire monitor interface would block until 
> the migrate was completed).

Yep, I'd really like to see a 'info migrate' or equivalent that works
for incoming migration, even if it can't give us progress info, just
the status report is important to detect failure & completion.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] race condition when exec'ing "qemu -incoming" followed by monitor "cont"

[Laine Stump]

On 04/09/2010 12:03 PM, Laine Stump wrote:
> (Please forgive (and correct!) any inaccuracies in my description of 
> qemu's workings - I've only recently started looking at it directly, 
> rather than through the lens of libvirt)
>
> libvirt implements a "domain restore" operation by:
>
> 0) start with a previously saved domain image in a file
>
> 1) open the domain image, and connect it to a pipe
>
> 2) fork, connect the pipe to stdin, and exec qemu with "-incoming 
> exec:cat"
>
> 3) execute "cont" in that qemu's monitor.

I realized after sending that I was too hasty in my reading of the code, 
and interpreted it incorrectly. The file containing the domain image is 
not sent into a pipe, but merely dup2'ed to be stdin in the child 
process. So the correct sequence of events is:

0) start with a previously saved domain image in a file

1) open the domain image

2) fork, dup2 the domain image's fd as stdin,  and exec qemu with 
"-incoming exec:cat"

3) execute "cont" in that qemu's monitor.


On 04/09/2010 12:29 PM, Paolo Bonzini wrote:
> On 04/09/2010 06:03 PM, Laine Stump wrote:
>>
>> Can someone provide any insight on why it is possible to start the CPUs
>> in the domain before the incoming migration is complete, and what we can
>> do (other than blindly sleeping) to prevent this?
>
> I would say it's a bug in QEMU, and it has to be fixed there.

My assumption would have been that, if -incoming is specified on the 
commandline, the monitor should not even be started up until the 
incoming migration is complete. Being new to qemu, though, I'm hoping 
there's something I'm missing.