Re: [Qemu-devel] [PATCH v3 0/4] migration: skip scanning and migrating ram pages released by virtio-balloon driver.

[David Hildenbrand <david@redhat.com>]

Am 23.12.2016 um 03:50 schrieb Li, Liang Z:
>> While measuring live migration performance for qemu/kvm guest, it was
>> observed that the qemu doesn’t maintain any intelligence for the guest ram
>> pages released by the guest balloon driver and treat such pages as any other
>> normal guest ram pages. This has direct impact on overall migration time for
>> the guest which has released (ballooned out) memory to the host.
>>
>> In case of large systems, where we can configure large guests with 1TB and
>> with considerable amount of memory released by balloon driver to the host,
>> the migration time gets worse.
>>
>> The solution proposed below is local to qemu (and does not require any
>> modification to Linux kernel or any guest driver). We have verified the fix for
>> large guests =1TB on HPE Superdome X (which can support up to 240 cores
>> and 12TB of memory).
>>
>> During live migration, as part of first iteration in ram_save_iterate() ->
>> ram_find_and_save_block () will try to migrate ram pages even if they are
>> released by vitrio-balloon driver (balloon inflate). Although these pages
>> which are returned to the host by virtio-balloon driver are zero pages, the
>> migration algorithm will still end up scanning the entire page
>> ram_find_and_save_block() ->
>> ram_save_page()/ram_save_compressed_page() ->
>> save_zero_page() -> is_zero_range(). We also end-up sending header
>> information over network for these pages during migration. This adds to the
>> total migration time.
>>
>> The solution creates a balloon bitmap ramblock as a part of virtio-balloon
>> device initialization. The bits in the balloon bitmap represent a guest ram
>> page of size 1UL << VIRTIO_BALLOON_PFN_SHIFT or 4K. If
>> TARGET_PAGE_BITS <= VIRTIO_BALLOON_PFN_SHIFT, ram_addr offset for
>> the dirty page which is used by dirty page bitmap during migration is checked
>> against the balloon bitmap as is, if the bit is set in the balloon bitmap, the
>> corresponding ram page will be excluded from scanning and sending header
>> information during migration. In case TARGET_PAGE_BITS >
>> VIRTIO_BALLOON_PFN_SHIFT for a given dirty page ram_addr, all sub-pages
>> of 1UL << VIRTIO_BALLOON_PFN_SHIFT size should be ballooned out to
>> avoid zero page scan and sending header information.
>>
>> The bitmap represents entire guest ram memory till max configured memory.
>> Guest ram pages claimed by the virtio-balloon driver will be represented by 1
>> in the bitmap. Since the bitmap is maintained as a ramblock, it’s migrated to
>> target as part migration’s ram iterative and ram complete phase. So that
>> substituent migrations from the target can continue to use optimization.
>>
>> A new migration capability called skip-balloon is introduced. The user can
>> disable the capability in cases where user does not expect much benefit or in
>> case the migration is from an older version.
>>
>> During live migration setup the optimization can be set to disabled state if .
>> no virtio-balloon device is initialized.
>> . skip-balloon migration capability is disabled.
>> . If the guest virtio-balloon driver has not set
>> VIRTIO_BALLOON_F_MUST_TELL_HOST
>>   flag. Which means the guest may start using a ram pages freed by guest
>> balloon
>>   driver, even before the host/qemu is aware of it. In such case, the
>>   optimization is disabled so that the ram pages that are being used by the
>>   guest will continue to be scanned and migrated.
>>
>> Balloon bitmap ramblock size is set to zero if the optimization is disabled, to
>> avoid overhead of migrating the bitmap. If the bitmap is not migrated to the
>> target, the destination starts with a fresh bitmap and tracks the ballooning
>> operation thereafter.
>>
>
> I have a better way to get rid of the bitmap.
> We should not maintain the inflating pages in the bitmap, instead, we
> can get them from the guest if it's needed, just like what we did for the guest's
> unused pages. Then we can combine the inflating page info with the unused page
> info together, and skip them during live migration.
>

If we want to actually host enforce (disallow) access to inflated pages, 
having such a bitmap in QEMU is required. Getting them from the guest 
doesn't make sense in that context anymore.

Thanks,
David

> Thanks!
> Liang
>
>


-- 

David