Re: [Qemu-devel] [PATCH v3 RFC 0/17] block: persistent dirty bitmaps

[Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>]

Add feature:

read-only mode.

For now bitmaps are loaded only from writable bs's: in_use should be set 
in the image.

Should we automatically load bitmaps in read-only mode if their storage 
is readonly? Hmm. I think it should be checked that bs_for - the disk, 
the bitmap is loading for is readonly too. Or may be there should be 
explicit cmd option for read-only loading.

Also, there may be needed read-only flag in Dirty Bitmap Directory 
Entry. Is it?


On 05.09.2015 19:43, Vladimir Sementsov-Ogievskiy wrote:
> v3:
>
> Hi all! After long delay I am starting a new thread about persistent
> dirty bitmaps. Not all ideas/requests are done from v2 thread, but I
> need to start somewhere. Also, there are a lot of code changes to be
> considered before everything is finished. Also, v2 thread is too large
> and it is hard to handle it.
>
> Here I'll summarize, what was changed. All ideas/comments, that was not
> done from v2 thread I'll add as replays to this cover letter. If I miss
> something, please add.
>
> 01-02: new patches, necessary preparation
>
> 03: docs
> - add constraints
> - document auto clear bit
> - renames
>      L1 table -> Dirty Bitmap Table
>      Dirty Bitmaps Table -> Dirty Bitmap Directory
> - remove Standard cluster with its zero bit,
>
> struct changes:
>
> extension header:
>    add 32bit dirty_bitmap_directory_size
>    it is needed to read the whole Dirty Bitmap Directory.
>    nb_dirty_bitmaps becomes excessive (it may be calculated by looking
>    through Dirty Bitmap Directory).
>    RFC: is it bad? should I drop nb_dirty_bitmaps?
>
> directory entry:
> - bitmap size is renamed to 'number of virtual bits'
> - flags added. (self/aut/read_only are not reliazed in code for now)
> - granularity is now represented by granularity_bits
>   
>
> 04: structs and consts
> Structures changed, accordingly to spec.
> QCowDirtyBitmap now only contains name and offset. Offset is offset in
> bytes in Dirty Bitmap Directory, so, bitmap header can be accessed:
> cached version: s->dirty_bitmap_directory + offset
> in image file: s->dirty_bitmap_directory_offset + offset
>
> == dirty-bitmaps feature ==
>
> initial patch [PATCH 2/8] qcow2: add dirty-bitmaps feature
> is splitted out to 5 parts. The first with license notice and
> constants is in 04. Then, there are:
> 05: read Dirty Bitmap Directory
> 06: load dirty bitmap
> 07: store dirty bitmap
> 08: add dirty bitmap extension
>
> The code is significantly changed and reorganized. For your comments:
> - fixes about flushes
> - g_try_... functions are used. But I left the old g_malloc/g_new for
> things with constant or strictly limited size (dirty bitmap header,
> with its name, name size < 1024, one cluster, bitmap name (< 1024))
> - input validation added.
> - dirty bitmap directory is read/written wholly, not by parts.
> - load/store through bdrv_dirty_bitmap_[de]serialize_part
> - 'delete dirty bitmap' function is removed. This function is not used
> in v3. May be this functionality should appear in qemu_img, or with qmp
> series.
> - qcow2_write_dirty_bitamps function is removed. This functionality is
> rewritten, now Bitmap Directory is rewritten from
> s->dirty_bitmap_directory, (directory_write()), and not generated from
> s->dirty_bitmaps
>
> 09: load check
> - new functionality, just check existance of dirty bitmap by name
>
> 10: store block dirty bitmap
> - bdrv_store_dirty_bitmap simplified, as bitmap is simply passed to
> qcow2_dirty_bitmap_store
>
> 11: load block dirty bitmap
> - bdrv_load_dirty_bitmap simplified, as bitmap is created by
> qcow2_dirty_bitmap_load. Also, granularity parameter is removed
>
> patch [[PATCH 5/8] qcow2: add qcow2_dirty_bitmap_delete_all] is
> removed. This function is not used in v3. May be this functionality
> should appear in qemu_img, or with qmp series.
>
> 12: autoclear bit
> - if it is not set, but bitmap extension is found, it just skipped and
> warning printed. Dirty bitmaps are not cleread from the image as in v2
> - handling in qcow2-dirty-bitmap.c is changed, the bit is just set when
> adding entry to Dirty Bitmap Directory in the image.
>
> 13: cmd line
> create=on|off flag added. Now, the bitmap is not auto-created.
> By default flag is off.
> If on, then new bitmap will be created in the image, if the bitmap with
> same name is already exists an error will be generated.
> If off, then the bitmap will be loaded from the image, if there is no
> one an error will be generated.
> If create=off and granularity is specified then granularity will be
> checked for loaded bitmap and if not match an error will be generated.
>
> also, s/drive/node
>
> also, dirty_bitmap_func():
>      - add errp parameter and handle this in main()
>      - open image with BDRV_O_RDWR for loading bitmaps
>
> 14-16: new patches
>
> 17: test
> - add three test cases.
>
>
> v2:
>   - rebase on my 'Dirty bitmaps migration' series
>   - remove 'print dirty bitmap', 'query-dirty-bitmap' and use md5 for
>     testing like with dirty bitmaps migration
>   - autoclean features
>
> v1:
>
> The bitmaps are saved into qcow2 file format. It provides both
> 'internal' and 'external' dirty bitmaps feature:
>   - for qcow2 drives we can store bitmaps in the same file
>   - for other formats we can store bitmaps in the separate qcow2 file
>
> QCow2 header is extended by fields 'nb_dirty_bitmaps' and
> 'dirty_bitmaps_offset' like with snapshots.
>
> Proposed command line syntax is the following:
>
> -dirty-bitmap [option1=val1][,option2=val2]...
>      Available options are:
>      name         The name for the bitmap (necessary).
>
>      file         The file to load the bitmap from.
>
>      file_id      When specified with 'file' option, then this file will
>                   be available through this id for other -dirty-bitmap
>                   options when specified without 'file' option, then it
>                   is a reference to 'file', specified with another
>                   -dirty-bitmap option, and it will be used to load the
>                   bitmap from.
>
>      drive        The drive to bind the bitmap to. It should be specified
>                   as 'id' suboption of one of -drive options. If nor
>                   'file' neither 'file_id' are specified, then the bitmap
>                   will be loaded from that drive (internal dirty bitmap).
>
>      granularity  The granularity for the bitmap. Not necessary, the
>                   default value may be used.
>
>      enabled      on|off. Default is 'on'. Disabled bitmaps are not
>                   changing regardless of writes to corresponding drive.
>
> Examples:
>
> qemu -drive file=a.qcow2,id=disk -dirty-bitmap name=b,drive=disk
> qemu -drive file=a.raw,id=disk \
>       -dirty-bitmap name=b,drive=disk,file=b.qcow2,enabled=off
>
> Vladimir Sementsov-Ogievskiy (8):
>    spec: add qcow2-dirty-bitmaps specification
>    qcow2: add dirty-bitmaps feature
>    block: store persistent dirty bitmaps
>    block: add bdrv_load_dirty_bitmap
>    qcow2: add qcow2_dirty_bitmap_delete_all
>    qcow2: add autoclear bit for dirty bitmaps
>    qemu: command line option for dirty bitmaps
>    iotests: test internal persistent dirty bitmap
>
>   block.c                       |  82 +++++++
>   block/Makefile.objs           |   2 +-
>   block/qcow2-dirty-bitmap.c    | 537 ++++++++++++++++++++++++++++++++++++++++++
>   block/qcow2.c                 |  69 +++++-
>   block/qcow2.h                 |  61 +++++
>   blockdev.c                    |  38 +++
>   docs/specs/qcow2.txt          |  66 ++++++
>   include/block/block.h         |   9 +
>   include/block/block_int.h     |  10 +
>   include/sysemu/blockdev.h     |   1 +
>   include/sysemu/sysemu.h       |   1 +
>   qemu-options.hx               |  37 +++
>   tests/qemu-iotests/118        |  83 +++++++
>   tests/qemu-iotests/118.out    |   5 +
>   tests/qemu-iotests/group      |   1 +
>   tests/qemu-iotests/iotests.py |   6 +
>   vl.c                          | 100 ++++++++
>   17 files changed, 1105 insertions(+), 3 deletions(-)
>   create mode 100644 block/qcow2-dirty-bitmap.c
>   create mode 100755 tests/qemu-iotests/118
>   create mode 100644 tests/qemu-iotests/118.out
>


-- 
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.