From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:58481) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZYHIk-0007hM-FZ for qemu-devel@nongnu.org; Sat, 05 Sep 2015 13:30:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZYHIg-0007qd-3W for qemu-devel@nongnu.org; Sat, 05 Sep 2015 13:30:54 -0400 Received: from relay.parallels.com ([195.214.232.42]:54519) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZYHIf-0007qK-Lz for qemu-devel@nongnu.org; Sat, 05 Sep 2015 13:30:50 -0400 Message-ID: <55EB26B7.6090501@virtuozzo.com> Date: Sat, 5 Sep 2015 20:30:31 +0300 From: Vladimir Sementsov-Ogievskiy MIME-Version: 1.0 References: <1441471439-6157-1-git-send-email-vsementsov@virtuozzo.com> In-Reply-To: <1441471439-6157-1-git-send-email-vsementsov@virtuozzo.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH v3 RFC 0/17] block: persistent dirty bitmaps List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: kwolf@redhat.com, stefanha@redhat.com, pbonzini@redhat.com, den@openvz.org, jsnow@redhat.com Add feature: read-only mode. For now bitmaps are loaded only from writable bs's: in_use should be set in the image. Should we automatically load bitmaps in read-only mode if their storage is readonly? Hmm. I think it should be checked that bs_for - the disk, the bitmap is loading for is readonly too. Or may be there should be explicit cmd option for read-only loading. Also, there may be needed read-only flag in Dirty Bitmap Directory Entry. Is it? On 05.09.2015 19:43, Vladimir Sementsov-Ogievskiy wrote: > v3: > > Hi all! After long delay I am starting a new thread about persistent > dirty bitmaps. Not all ideas/requests are done from v2 thread, but I > need to start somewhere. Also, there are a lot of code changes to be > considered before everything is finished. Also, v2 thread is too large > and it is hard to handle it. > > Here I'll summarize, what was changed. All ideas/comments, that was not > done from v2 thread I'll add as replays to this cover letter. If I miss > something, please add. > > 01-02: new patches, necessary preparation > > 03: docs > - add constraints > - document auto clear bit > - renames > L1 table -> Dirty Bitmap Table > Dirty Bitmaps Table -> Dirty Bitmap Directory > - remove Standard cluster with its zero bit, > > struct changes: > > extension header: > add 32bit dirty_bitmap_directory_size > it is needed to read the whole Dirty Bitmap Directory. > nb_dirty_bitmaps becomes excessive (it may be calculated by looking > through Dirty Bitmap Directory). > RFC: is it bad? should I drop nb_dirty_bitmaps? > > directory entry: > - bitmap size is renamed to 'number of virtual bits' > - flags added. (self/aut/read_only are not reliazed in code for now) > - granularity is now represented by granularity_bits > > > 04: structs and consts > Structures changed, accordingly to spec. > QCowDirtyBitmap now only contains name and offset. Offset is offset in > bytes in Dirty Bitmap Directory, so, bitmap header can be accessed: > cached version: s->dirty_bitmap_directory + offset > in image file: s->dirty_bitmap_directory_offset + offset > > == dirty-bitmaps feature == > > initial patch [PATCH 2/8] qcow2: add dirty-bitmaps feature > is splitted out to 5 parts. The first with license notice and > constants is in 04. Then, there are: > 05: read Dirty Bitmap Directory > 06: load dirty bitmap > 07: store dirty bitmap > 08: add dirty bitmap extension > > The code is significantly changed and reorganized. For your comments: > - fixes about flushes > - g_try_... functions are used. But I left the old g_malloc/g_new for > things with constant or strictly limited size (dirty bitmap header, > with its name, name size < 1024, one cluster, bitmap name (< 1024)) > - input validation added. > - dirty bitmap directory is read/written wholly, not by parts. > - load/store through bdrv_dirty_bitmap_[de]serialize_part > - 'delete dirty bitmap' function is removed. This function is not used > in v3. May be this functionality should appear in qemu_img, or with qmp > series. > - qcow2_write_dirty_bitamps function is removed. This functionality is > rewritten, now Bitmap Directory is rewritten from > s->dirty_bitmap_directory, (directory_write()), and not generated from > s->dirty_bitmaps > > 09: load check > - new functionality, just check existance of dirty bitmap by name > > 10: store block dirty bitmap > - bdrv_store_dirty_bitmap simplified, as bitmap is simply passed to > qcow2_dirty_bitmap_store > > 11: load block dirty bitmap > - bdrv_load_dirty_bitmap simplified, as bitmap is created by > qcow2_dirty_bitmap_load. Also, granularity parameter is removed > > patch [[PATCH 5/8] qcow2: add qcow2_dirty_bitmap_delete_all] is > removed. This function is not used in v3. May be this functionality > should appear in qemu_img, or with qmp series. > > 12: autoclear bit > - if it is not set, but bitmap extension is found, it just skipped and > warning printed. Dirty bitmaps are not cleread from the image as in v2 > - handling in qcow2-dirty-bitmap.c is changed, the bit is just set when > adding entry to Dirty Bitmap Directory in the image. > > 13: cmd line > create=on|off flag added. Now, the bitmap is not auto-created. > By default flag is off. > If on, then new bitmap will be created in the image, if the bitmap with > same name is already exists an error will be generated. > If off, then the bitmap will be loaded from the image, if there is no > one an error will be generated. > If create=off and granularity is specified then granularity will be > checked for loaded bitmap and if not match an error will be generated. > > also, s/drive/node > > also, dirty_bitmap_func(): > - add errp parameter and handle this in main() > - open image with BDRV_O_RDWR for loading bitmaps > > 14-16: new patches > > 17: test > - add three test cases. > > > v2: > - rebase on my 'Dirty bitmaps migration' series > - remove 'print dirty bitmap', 'query-dirty-bitmap' and use md5 for > testing like with dirty bitmaps migration > - autoclean features > > v1: > > The bitmaps are saved into qcow2 file format. It provides both > 'internal' and 'external' dirty bitmaps feature: > - for qcow2 drives we can store bitmaps in the same file > - for other formats we can store bitmaps in the separate qcow2 file > > QCow2 header is extended by fields 'nb_dirty_bitmaps' and > 'dirty_bitmaps_offset' like with snapshots. > > Proposed command line syntax is the following: > > -dirty-bitmap [option1=val1][,option2=val2]... > Available options are: > name The name for the bitmap (necessary). > > file The file to load the bitmap from. > > file_id When specified with 'file' option, then this file will > be available through this id for other -dirty-bitmap > options when specified without 'file' option, then it > is a reference to 'file', specified with another > -dirty-bitmap option, and it will be used to load the > bitmap from. > > drive The drive to bind the bitmap to. It should be specified > as 'id' suboption of one of -drive options. If nor > 'file' neither 'file_id' are specified, then the bitmap > will be loaded from that drive (internal dirty bitmap). > > granularity The granularity for the bitmap. Not necessary, the > default value may be used. > > enabled on|off. Default is 'on'. Disabled bitmaps are not > changing regardless of writes to corresponding drive. > > Examples: > > qemu -drive file=a.qcow2,id=disk -dirty-bitmap name=b,drive=disk > qemu -drive file=a.raw,id=disk \ > -dirty-bitmap name=b,drive=disk,file=b.qcow2,enabled=off > > Vladimir Sementsov-Ogievskiy (8): > spec: add qcow2-dirty-bitmaps specification > qcow2: add dirty-bitmaps feature > block: store persistent dirty bitmaps > block: add bdrv_load_dirty_bitmap > qcow2: add qcow2_dirty_bitmap_delete_all > qcow2: add autoclear bit for dirty bitmaps > qemu: command line option for dirty bitmaps > iotests: test internal persistent dirty bitmap > > block.c | 82 +++++++ > block/Makefile.objs | 2 +- > block/qcow2-dirty-bitmap.c | 537 ++++++++++++++++++++++++++++++++++++++++++ > block/qcow2.c | 69 +++++- > block/qcow2.h | 61 +++++ > blockdev.c | 38 +++ > docs/specs/qcow2.txt | 66 ++++++ > include/block/block.h | 9 + > include/block/block_int.h | 10 + > include/sysemu/blockdev.h | 1 + > include/sysemu/sysemu.h | 1 + > qemu-options.hx | 37 +++ > tests/qemu-iotests/118 | 83 +++++++ > tests/qemu-iotests/118.out | 5 + > tests/qemu-iotests/group | 1 + > tests/qemu-iotests/iotests.py | 6 + > vl.c | 100 ++++++++ > 17 files changed, 1105 insertions(+), 3 deletions(-) > create mode 100644 block/qcow2-dirty-bitmap.c > create mode 100755 tests/qemu-iotests/118 > create mode 100644 tests/qemu-iotests/118.out > -- Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.