Re: [Qemu-devel] [PATCH v7 0/5] virtio-balloon: free page hint reporting support

["Michael S. Tsirkin" <mst@redhat.com>]

On Tue, May 29, 2018 at 11:00:21PM +0800, Hailiang Zhang wrote:
> On 2018/4/24 14:13, Wei Wang wrote:
> > This is the deivce part implementation to add a new feature,
> > VIRTIO_BALLOON_F_FREE_PAGE_HINT to the virtio-balloon device. The device
> > receives the guest free page hints from the driver and clears the
> > corresponding bits in the dirty bitmap, so that those free pages are
> > not transferred by the migration thread to the destination.
> > 
> > - Test Environment
> >      Host: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz
> >      Guest: 8G RAM, 4 vCPU
> >      Migration setup: migrate_set_speed 100G, migrate_set_downtime 2 second
> > 
> > - Test Results
> >      - Idle Guest Live Migration Time (results are averaged over 10 runs):
> >          - Optimization v.s. Legacy = 271ms vs 1769ms --> ~86% reduction
> >      - Guest with Linux Compilation Workload (make bzImage -j4):
> >          - Live Migration Time (average)
> >            Optimization v.s. Legacy = 1265ms v.s. 2634ms --> ~51% reduction
> >          - Linux Compilation Time
> >            Optimization v.s. Legacy = 4min56s v.s. 5min3s
> >            --> no obvious difference
> > 
> > - Source Code
> >      - QEMU:  https://github.com/wei-w-wang/qemu-free-page-lm.git
> >      - Linux: https://github.com/wei-w-wang/linux-free-page-lm.git
> > 
> > ChangeLog:
> > v6->v7:
> >        virtio-balloon/virtio_balloo_poll_free_page_hints:
> >            - add virtio_notify() at the end to notify the driver that
> >              the optimization is done, which indicates that the entries
> >              have all been put back to the vq and ready to detach them.
> > v5->v6:
> >        virtio-balloon: use iothread to get free page hint
> > v4->v5:
> >      1) migration:
> >          - bitmap_clear_dirty: update the dirty bitmap and dirty page
> >            count under the bitmap mutex as what other functions are doing;
> >          - qemu_guest_free_page_hint:
> >              - add comments for this function;
> >              - check the !block case;
> >              - check "offset > block->used_length" before proceed;
> >              - assign used_len inside the for{} body;
> >              - update the dirty bitmap and dirty page counter under the
> >                bitmap mutex;
> >          - ram_state_reset:
> >              - rs->free_page_support: && with use "migrate_postcopy"
> >                instead of migration_in_postcopy;
> >              - clear the ram_bulk_stage flag if free_page_support is true;
> >      2) balloon:
> >           - add the usage documentation of balloon_free_page_start and
> >             balloon_free_page_stop in code;
> >           - the optimization thread is named "balloon_fpo" to meet the
> >             requirement of "less than 14 characters";
> >           - virtio_balloon_poll_free_page_hints:
> >               - run on condition when runstate_is_running() is true;
> >               - add a qemu spin lock to synchronize accesses to the free
> >                 page reporting related fields shared among the migration
> >                 thread and the optimization thread;
> >            - virtio_balloon_free_page_start: just return if
> >              runstate_is_running is false;
> >            - virtio_balloon_free_page_stop: access to the free page
> >              reporting related fields under a qemu spin lock;
> >            - virtio_balloon_device_unrealize/reset: call
> >              virtio_balloon_free_page_stop is the free page hint feature is
> >              used;
> >            - virtio_balloon_set_status: call irtio_balloon_free_page_stop
> >              in case the guest is stopped by qmp when the optimization is
> >              running;
> > v3->v4:
> >      1) bitmap: add a new API to count 1s starting from an offset of a
> >         bitmap
> >      2) migration:
> >          - qemu_guest_free_page_hint: calculate
> >            ram_state->migration_dirty_pages by counting how many bits of
> >            free pages are truely cleared. If some of the bits were
> >            already 0, they shouldn't be deducted by
> >            ram_state->migration_dirty_pages. This wasn't needed for
> >            previous versions since we optimized bulk stage only,
> >            where all bits are guaranteed to be set. It's needed now
> >            because we extened the usage of this optimizaton to all stages
> >            except the last stop&copy stage. From 2nd stage onward, there
> >            are possibilities that some bits of free pages are already 0.
> >       3) virtio-balloon:
> >           - virtio_balloon_free_page_report_status: introduce a new status,
> >             FREE_PAGE_REPORT_S_EXIT. This status indicates that the
> >             optimization thread has exited. FREE_PAGE_REPORT_S_STOP means
> >             the reporting is stopped, but the optimization thread still needs
> >             to be joined by the migration thread.
> > v2->v3:
> >      1) virtio-balloon
> >          - virtio_balloon_free_page_start: poll the hints using a new
> >            thread;
> >          - use cmd id between [0x80000000, UINT_MAX];
> >          - virtio_balloon_poll_free_page_hints:
> >              - stop the optimization only when it has started;
> >              - don't skip free pages when !poison_val;
> >          - add poison_val to vmsd to migrate;
> >          - virtio_balloon_get_features: add the F_PAGE_POISON feature when
> >            host has F_FREE_PAGE_HINT;
> >          - remove the timer patch which is not needed now.
> >      2) migration
> >         - new api, qemu_guest_free_page_hint;
> >         - rs->free_page_support set only in the precopy case;
> >         - use the new balloon APIs.
> > v1->v2:
> >      1) virtio-balloon
> >          - use subsections to save free_page_report_cmd_id;
> >          - poll the free page vq after sending a cmd id to the driver;
> >          - change the free page vq size to VIRTQUEUE_MAX_SIZE;
> >          - virtio_balloon_poll_free_page_hints: handle the corner case
> >            that the free page block reported from the driver may cross
> >            the RAMBlock boundary.
> >      2) migration/ram.c
> >          - use balloon_free_page_poll to start the optimization
> > 
> > 
> > Wei Wang (5):
> >    bitmap: bitmap_count_one_with_offset
> >    migration: use bitmap_mutex in migration_bitmap_clear_dirty
> >    migration: API to clear bits of guest free pages from the dirty bitmap
> >    virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT
> >    migration: use the free page hint feature from balloon
> > 
> >   balloon.c                                       |  58 +++++-
> >   hw/virtio/virtio-balloon.c                      | 241 ++++++++++++++++++++++--
> >   include/hw/virtio/virtio-balloon.h              |  27 ++-
> >   include/migration/misc.h                        |   2 +
> >   include/qemu/bitmap.h                           |  13 ++
> >   include/standard-headers/linux/virtio_balloon.h |   7 +
> >   include/sysemu/balloon.h                        |  15 +-
> >   migration/ram.c                                 |  73 ++++++-
> >   8 files changed, 406 insertions(+), 30 deletions(-)
> 
> Nice optimization, for the first stage of  current migration method, we need to migrate all the pages of
> VM to destination,  with this capability, we can reduce lots of unnecessary pages migrating.
> 
> Just a small piece of advice, it is better to split the fourth patch into small ones, to make it more easy
> for reviewing. Besides, should we make this capability an optional one, just like other migration capabilities do ?

That's already the case, one has to enable it in the balloon, and set
the iothread.

-- 
MST