Re: [Qemu-devel] Re: [patch 2/3] Add support for live block copy

[Anthony Liguori <anthony@codemonkey.ws>]

On 03/01/2011 03:59 AM, Dor Laor wrote:
> On 02/28/2011 08:12 PM, Anthony Liguori wrote:
>>
>> On Feb 28, 2011 11:47 AM, "Avi Kivity" <avi@redhat.com
>> <mailto:avi@redhat.com>> wrote:
>> >
>> > On 02/28/2011 07:33 PM, Anthony Liguori wrote:
>> >>
>> >>
>> >> >
>> >> > You're just ignoring what I've written.
>> >>
>> >> No, you're just impervious to my subtle attempt to refocus the
>> discussion on solving a practical problem.
>> >>
>> >> There's a lot of good, reasonably straight forward changes we can
>> make that have a high return on investment.
>> >>
>> >
>> > Is making qemu the authoritative source of configuration information
>> a straightforward change?  Is the return on it high?  Is the 
>> investment low?
>>
>> I think this is where we fundamentally disagree.  My position is that
>> QEMU is already the authoritative source.  Having a state file doesn't
>> change anything.
>>
>> Do a hot unplug of a network device with upstream libvirt with acpiphp
>> unloaded, consult libvirt and then consult the monitor to see who has
>> the right view of the guests config.
>>
>> To me, that's the definition of authoritative.
>>
>> > "No" to all three (ignoring for the moment whether it is good or not,
>> which we were debating).
>> >
>> >
>> >> The only suggestion I'm making beyond Marcelo's original patch is
>> that we use a structured format and that we make it possible to use the
>> same file to solve this problem in multiple places.
>> >>
>> >
>> > No, you're suggesting a lot more than that.
>>
>> That's exactly what I'm suggesting from a technical perspective.
>>
>> >> I don't think this creates a fundamental break in how management
>> tools interact with QEMU.  I don't think introducing RAID support in the
>> block layer is a reasonable alternative.
>> >>
>> >>
>> >
>> > Why not?
>>
>> Because its a lot of complexity and code that can go wrong while only
>> solving the race for one specific case.  Not to mention that we double
>> the iop rate.
>>
>> > Something that avoids the whole state thing altogether:
>> >
>> > - instead of atomically switching when live copy is done, keep on
>> issuing writes to both the origin and the live copy
>> > - issue a notification to management
>> > - management receives the notification, and issues an atomic blockdev
>> switch command
>>
>> > this is really the RAID-1 solution but without the state file (credit
>> Dor).  An advantage is that there is no additional latency when trying
>> to catch up to the dirty bitmap.
>>
>> It still suffers from the two generals problem.  You cannot solve this
>> without making one node reliable and that takes us back to it being
>> either QEMU (posted event and state file) or the management tool (sync
>> event).
>
> It is safe w/o a state file by changing the basic live copy algorithm:
>
> 1. Live copy in progress stage
>    Once live copy command is issued, a dirty bit map is created for
>    tracking. There is a single pass over the entire image where we copy
>    blocks from the src to the dst.
>
>    Write commands for blocks that were already copied will be done
>    twice for the src and dst.
>
>    Once the full copy single pass ends, we trigger a QMP event that
>    this stage can end.
>
>    The live copy stage keeps running till the management issue a switch
>    command. When it will happen, the switch is immediate and no need to
>    copy additional blocks (but flush pending IOs).
>
> 2. Management sends a switch command.
>    Qemu stops the doubling the IO and switches to the destination.
>    End.

Here is where your race is:

2. Management sends a switch command

3. QEMU receives switch command

4. QEMU stops doubling IO and switches to the destination

5. QEMU sends acknowledgement of switch command

6. Management receives acknowledge of switch command

7. Management changes internal state definition to reflect the new 
destination

If QEMU or the management tool crashes after step 4 and before step 6, 
when the management tool restarts QEMU with the source image, data loss 
will have occurred (and potentially corruption if a flush had happened).

This all boils down to the Two Generals Problem[1].  It's simply not 
fixable without making one end reliable and that means that someone 
needs to fsync() something *after* the switchover happens but before the 
first write happens.  That can be QEMU (Avi's RAID proposal and my state 
file proposal) or it can be the management tool (if we introduce 
synchronous events).

[1] http://en.wikipedia.org/wiki/Two_Generals%27_Problem

Regards,

Anthony Liguori