Re: [Qemu-devel] [PATCH RFC] external backup api

[Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>]

On 05.02.2016 22:48, John Snow wrote:
>
> On 01/22/2016 12:07 PM, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all.
>>
>> This is the early begin of the series which aims to add external backup
>> api. This is needed to allow backup software use our dirty bitmaps.
>>
>> Vmware and Parallels Cloud Server have this feature.
>>
> Have a link to the equivalent feature that VMWare exposes? (Or Parallels
> Cloud Server) ... I'm curious about what the API there looks like.

For VMware you need their Virtual Disk Api Programming Guide
http://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/vddk60_programming.pdf
Look at Changed Block Tracking (CBT) , Backup and Restore.

For PCS here is part of SDK header, related to the topic:

====================================
/*
  * Builds a map of the disk contents changes between 2 PITs.
    Parameters
    hDisk :       A handle of type PHT_VIRTUAL_DISK identifying
                  the virtual disk.
    sPit1Uuid :   Uuid of the older PIT.
    sPit2Uuid :   Uuid of the later PIT.
    phMap :       A pointer to a variable which receives the
          result (a handle of type PHT_VIRTUAL_DISK_MAP).
    Returns
    PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                 PrlDisk_GetChangesMap_Local, (
         PRL_HANDLE hDisk,
         PRL_CONST_STR sPit1Uuid,
         PRL_CONST_STR sPit2Uuid,
         PRL_HANDLE_PTR phMap) );

/*
  * Reports the number of significant bits in the map.
    Parameters
    hMap :        A handle of type PHT_VIRTUAL_DISK_MAP identifying
                  the changes map.
    phSize :      A pointer to a variable which receives the
          result.
    Returns
    PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                 PrlDiskMap_GetSize, (
         PRL_HANDLE hMap,
         PRL_UINT32_PTR pnSize) );

/*
  * Reports the size (in bytes) of a block mapped by a single bit
  * in the map.
    Parameters
    hMap :        A handle of type PHT_VIRTUAL_DISK_MAP identifying
                  the changes map.
    phSize :      A pointer to a variable which receives the
          result.
    Returns
    PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                 PrlDiskMap_GetGranularity, (
         PRL_HANDLE hMap,
         PRL_UINT32_PTR pnSize) );

/*
  * Returns bits from the blocks map.
    Parameters
    hMap :        A handle of type PHT_VIRTUAL_DISK_MAP identifying
                  the changes map.
    pBuffer :     A pointer to a store.
    pnCapacity :  A pointer to a variable holding the size
          of the buffer and receiving the number of
          bytes actually written.
    Returns
    PRL_RESULT.
*/
PRL_METHOD_DECL( PARALLELS_API_VER_5,
                 PrlDiskMap_Read, (
         PRL_HANDLE hMap,
         PRL_VOID_PTR pBuffer,
         PRL_UINT32_PTR pnCapacity) );

=======================================


>
>> There is only one patch here, about querying dirty bitmap from qemu by
>> qmp command. It is just an updated and clipped (hmp command removed) old
>> my patch "[PATCH RFC v3 01/14] qmp: add query-block-dirty-bitmap".
>>
>> Before writing the whole thing I'd like to discuss the details. Or, may
>> be there are existing plans on this topic, or may be someone already
>> works on it?
>>
>> I see it like this:
>>
>> =====
>>
>> - add qmp commands for dirty-bitmap functions: create_successor, abdicate,
>> reclaime.
> Hm, why do we need such low-level control over splitting and merging
> bitmaps from an external client?
>
>> - make create-successor command transaction-able
>> - add query-block-dirty-bitmap qmp command
>>
>> then, external backup:
>>
>> qmp transaction {
>>      external-snapshot
>>      bitmap-create-successor
>> }
>>
>> qmp query frozen bitmap, not acquiring aio context.
>>
>> do external backup, using snapshot and bitmap
>>
>> if (success backup)
>>      qmp bitmap-abdicate
>> else
>>      qmp bitmap-reclaime
>>
>> qmp merge snapshot
>> =====
>>
> Hm, I see -- so you're hoping to manage the backup *entirely*
> externally, so you want to be able to reach inside of QEMU and control
> some status conditions to guarantee it'll be safe.
>
> I'm not convinced QEMU can guarantee such things -- due to various flush
> properties, race conditions on write, etc. QEMU handles all of this
> internally in a non-public way at the moment.

Hm, can you be more concrete? What operations are dangerous? We can do 
them in paused state for example.

>
>> In the following patch query-bitmap acquires aio context. This must be
>> ofcourse dropped for frozen bitmap.
>> But to make it in true way, I think, I should check somehow that this is
>> not just frozen bitmap, but the bitmap frozen by qmp command, to avoid
>> incorrect quering of bitmap frozen by internal backup (or other
>> mechanizm).. May be, it is not necessary.
>>
>>
>>
>
>


-- 
Best regards,
Vladimir