Re: [Qemu-devel] [PATCH v2 0/6] external backup api

["Denis V. Lunev" <den-lists@parallels.com>]

On 02/09/2016 10:25 PM, Denis V. Lunev wrote:
> On 02/09/2016 09:12 PM, John Snow wrote:
>>
>> On 02/09/2016 11:58 AM, Denis V. Lunev wrote:
>>> On 02/09/2016 07:49 PM, John Snow wrote:
>>>> On 02/09/2016 09:37 AM, Denis V. Lunev wrote:
>>>>> On 02/09/2016 05:21 PM, Stefan Hajnoczi wrote:
>>>>>> On Fri, Feb 05, 2016 at 11:28:42AM +0300, Denis V. Lunev wrote:
>>>>>>> On 02/03/2016 11:14 AM, Fam Zheng wrote:
>>>>>>>> On Sat, 01/30 13:56, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> These series which aims to add external backup api. This is 
>>>>>>>>> needed
>>>>>>>>> to allow
>>>>>>>>> backup software use our dirty bitmaps.
>>>>>>>>>
>>>>>>>>> Vmware and Parallels Cloud Server have this feature.
>>>>>>>> What is the advantage of this appraoch over "drive-backup
>>>>>>>> sync=incremental
>>>>>>>> ..."?
>>>>>>> This will allow third-party vendors to backup QEMU VMs into
>>>>>>> their own formats or to the cloud etc.
>>>>>> Backup software can implement NBD to receive the incremental blocks
>>>>>> from
>>>>>> QEMU.  Use drive-backup sync=incremental and the backup appliance as
>>>>>> the
>>>>>> NBD target.
>>>>>>
>>>>>> It's more complicated to add this QMP command flow to backup 
>>>>>> software
>>>>>> than to implement NBD.
>>>>>>
>>>>>> Stefan
>>>>> it can, but this is a matter of problem due to the nature of
>>>>> how this software is implemented. Usually it is written
>>>>> in a semi-standard way and it uses "plugins" to actually
>>>>> collect the data, i.e. the code is written in standard
>>>>> interface/real implementation pattern and interfaces are
>>>>> basically the same.
>>>>>
>>>>> With this standard approach backup software is working
>>>>> as an active side of the process, i.e. it performs operations
>>>>> and controls the flow.
>>>>>
>>>>> This means that "non-standard" QEMU technology will be
>>>>> pain here.
>>>>>
>>>>> Den
>>>> Am I to understand that for e.g. VMWare the backup appliance is
>>>> literally reading the disk image from storage directly while the VM is
>>>> running?
>>>>
>>>> I'd be a bit surprised if that were true.
>>> I think that backup software is asking alive VM about the data.
>>>
>>>> My biggest concern here is that there is not a safe way, today, to 
>>>> read
>>>> from a disk image atomically while the VM is running. I think that'd
>>>> take a lot of work to do and you'll not find a lot of support in
>>>> implementing it.
>>>>
>>>> Of course, while the VM is paused/off is a different story, but for 
>>>> now
>>>> I still feel like NBD is the right answer for getting block data from
>>>> QEMU.
>>>>
>>>> What am I missing?
>>>>
>>>> --js
>>> In general, in Parallels Server the backup was created using the
>>> following approach:
>>> - create external snapshot. In this case the base image (backing store
>>> in QEMU terminology)
>>>    will be READ-ONLY and could be safely read by any entity
>>> - backup that read-only disk image (any way)
>>> - merge snapshots
>>>
>> I see.
>>
>>> In this process backup software is active while PCS is passive.
>> PCS?
> Parallels Cloud Server. Sorry for abbreviation :(
>
>
>>> With QEMU the approach looks the same to me:
>>> - start a backup
>>> - ask QEMU to give a data to be backuped (using NBD server in QEMU
>>>    with Fam's image fleecing)
>>> - finish backup
>>>
>>> Important bit here is that dirty bitmap should be provided
>>> by QEMU by request. This dirty bitmap will be read-only at that
>>> moment, current active dirty bitmap should be replaced with
>>> new at backup start operation.
>>>
>>> Den
>> I don't have any problems providing the bitmap data through an external
>> API, but the part I want to be 100% clear on before I ACK it is the API
>> portion where we allow an external client to split or merge bitmaps
>> externally -- that's functionality you don't need if you query the data
>> from QEMU itself.
> That is fine :)
OOPS. This could be mistake!

The client should not perform operations on the bitmap manually,
but it should instruct QEMU to do this job for him. Bitmap child
should be created by request of backup software and its existence
should also be controlled by the backup software. This could be either
hidden or not, but the state should be controlled.

Lets again discuss the sequence:
1. call guest-fsfreeze-freeze to make consistent backup. This means
     that the guest ensures that all its journals are committed and
     there are no pending writes. Filesystem is in a good state.

2. start backup. At this moment original dirty bitmap child should
   be created to track new writes. There are no writes from the
   guest thanks to step (1). Thus we could avoid to move QEMU
   to VM_PAUSED state to perform the operation. This requires to
   2.1 start image fleecing
   2.2 create bitmap child (successor)

3. call guest-fsfreeze-thaw to unfreeze the guest

4. get dirty bitmap (parent). It is read-only and could be safely extracted

5. collect data from QEMU using NBD server inside QEMU (image
     fleecing)

6a. finish backup successfully. This implies the following ops:
    6a.1  remove fleecing image
    6a.2 drop old dirty (parent) bitmap (abdicate)

6b. finish backup unsuccessfully:
    6b.1 remove fleecing image
    6b.2(a) merge child to parent (reclaim)
or alternatively
    6b.2(b) drop both child and parent dirty bitmaps and stop tracking
                 to facilitate new full backup next round

Constraints:
- step (4) is slow. We do not want to do it before step (3)
- paired operations aka (6.1)/(6.2) and (2.1)/(2.2) could be
   done either by the single QMP command or by two different
   commands, we do not care much on this

There are also troubles on guest snapshot operations. For now we
drop dirty bitmaps in Parallels Server on switch-to-snapshot
operations as it is unclear how to properly make dirty bitmap for
the case.

Den